Assets and file system dependencies for pipelines

[jordibares]

Let's say I have an asset I have created with a number of dependencies to the files system, from ready flies, to caches, to creating point clouds and a not trivial (aka linear) approach in which the asset, based on certain conditions or user selection does one thing or another.

If I want to publish this assets and all its dependencies to a repository I will have to traverse this, potentially, huge network with subnetworks inside (think Chops inside Sops for example) and rewire all these paths, move files to the repository and then export the asset.

My feeling is that this particular task is kind of done in the Render Pre-flight but I wonder if it is accessible and more important, is this the best way to address a more compartmentalised pipeline?

How are you guys approaching this? any suggestion?

thanks in advance and hope this thread becomes a source of inspiration for everybody.

[Mandrake0]

sadly i can't give you much information because i'm also looking for a pipeline guide. but what i find there is no good manuals public available or i haven't found it yet. :-)

for accessing the pre-flight data you can use python the values are stored in a tuple.

hou.fileReferences()

http://archive.sidefx.com/docs/houdini15.5/hom/hou/fileReferences

 

when you have heard the last webinar there is something with git and H16 or later. 

 

[jordibares]

1 hour ago, Mandrake0 said:

sadly i can't give you much information because i'm also looking for a pipeline guide. but what i find there is no good manuals public available or i haven't found it yet. :-)

for accessing the pre-flight data you can use python the values are stored in a tuple.

hou.fileReferences()

http://archive.sidefx.com/docs/houdini15.5/hom/hou/fileReferences

 

when you have heard the last webinar there is something with git and H16 or later. 

 

Thank you, i have look at it and my issue is with large binary files with git, otherwise looks terrific. I have also explored special tools on top of git to address the big file problem but I wonder if this is the route that has been followed on big productions.

I do wonder how bigger companies scale up a production from a few FX guys to a truly parallel setup... anyone from SideFX willing to suggest things?  ;-)

Can you point me to that webinar? is it public?

thanks

[michael]

the likely reason you'll have a hard time finding information is that everybody seems to roll their own - and so what little info they are able to make public is often far too specific to their pipeline and don't work in other environments...

something to think about...

use your environment - Houdini can use environment variables so assets can have tons of stuff referenced deep inside but as long as you have environment variables set it'll find what it needs.

 

[jordibares]

Thanks Michael, would be nice for SideFX to suggest approaches specially with regards with publishing assets to repositories and the like.

At the moment I am using environment variables so yes, that seems to be the path of less friction but I do wonder if the same issues I am finding regarding general style when building assets is something others use.

For example, I tend to embed all the necessary small file in the asset itself so I don't have to deal with dependencies on disk but sometimes is just impossible.

Should I always have a button on the asset to force the regeneration of intermediate file out caches? seems like the only way to guarantee the asset is now fully in a repository.

 

Thanks

[michael]

I agree that there could be some recommendations from SideFX - gathered from our chats with customers etc...

I'll kick it around and see what we can do.

[jordibares]

That would be fantastic, Thanks Michael.

Feel free to contact me privately or phone in case you need any extra info.

[pezetko]

I would love to hear some ideas and recommendations from devs too.

Maybe masterclass?

Thank you!

[jordibares]

22 hours ago, pezetko said:

I would love to hear some ideas and recommendations from devs too.

Maybe masterclass?

Thank you!

I would love that too... may be focusing on;

- Asset creation style (naming, storage strategy, version management, deployment to teams, etc..)

- Publishing approaches to assets (development vs production states), consolidating material (textures maps, geo, etc...) for truly standalone assets

- Discuss possible asset library, best ways to use GIT to do asset management considering there are big files involved.

- Best practices on asset creation for efficient pipelines where you have hundreds of them (external scripts? load on demand?)

 

hope this gives some ideas to such an amazing topic.

 

IMHO any other 3D app leaves it all to the user/company and the solutions are pretty personal so I am kind of craving for a proper software engineering approach.

 

jb

[davpe]

same here. would love to hear some ideas for asset workflows and mixed pipeline integration. From conversation with different people I came to the conclusion that everybody using Houdini as a main 3d software is having a hard time figuring this out. The result is usually inventing some weird workarounds and hacks that are easy to break (small shops) or completely custom solutions that replace whatever Houdini offers out of the box (large companies).