designing a digital asset workflow

[cam]

Hi guys,

One of the great features of houdini is it's openess, this allows you to tackle a problem in a variety of ways. I was hoping to throw open the door to the discussion: 'designing workflow with digital assets'. I would like to hear about things that have worked, cautionary tales and other battle tested tips from real world production settings.

Consider this example:

1. We model a bunny in Maya

2. We animated the bunny in Maya

3. We bring the resulting bunny model and vertex cache into houdini

4. We blow up the bunny!

5. Export bunny bits (geo + particles + other) back to maya

6. Render - Looks great but needs to be gooier!

7. Add some gooey controls to the explodeBunnyOtl and repeat!

Here are the general parameters:

1. Hybrid environment:

Modelling, Rigging and Animation will happen in Maya. Vertex caches and animation curves will be brought into houdini.

2. Translation tools inhouse:

Converting vertex caches and animation curves will be handled by in house tools. Consider it not an issue.

3. Version control of takes happens outside of houdini

Assume we have a database which figures out where files go. This info gets passed to houdini (no heavily lifting needed from within houdini).

4. Everything is a locked OTL

This is not cast in stone but it seemed to work well in a feature animation setting and the approach does have some advantages, which are appealing. Lets call it a goal rather than a rule. I would be interested in the counter argument to this if anyone has one.

5. Ninjas make the tools, more novice users will use them

The idea is that the more experienced houdini users will create template tools to be used in shots. The analogy would be similar to riggers and animators. As such we would like to use OTLs to hide some of the complexity, reduce human error and achieve continuity. Also this allows us to leverage our escape licenses. The resulting tool should have an input (in our example a bunny), some parameters (in our example gooiness, explodiness) and some outputs (in our example bunny geo, bunny particles). The end user may likely be a Maya artist who has never used Houdini before.

Questions and talking points:

- Storage of OTL instances (shot specific settings applied to an OTL)

- Publishing and storing OTL definitions? where? how? cvs? svn?

- Upgrading OTL, without breaking everything:

....- I want to add a parameter that wasn't there before

....- I want to remove a parameter that was there before

....- I want to rename an existing parameter

....- I want to leave the parameters alone but change the inner logic

....- Advanced: find all shots with an old version, replace with the new version and redo processing

....- Advanced: Wedges and takes, send to the farm small, medium and large explosion

- Things that should be included/NOT included in an OTL: for example should a geometry file be a parameter of an OTL or an input?

- Any tips or considerations? Whatever you do, DON'T.......

- Any wars stories or interesting tales loosely related to pipeline design.

Many thanks, looking forward to hearing your thoughts.

cl

Edited August 18, 2009 by cam

[michael]

do you remember nothing from The Wild?

[Marc]

Hey Cam

Seems like a good data pipeline (except for the rendering in Maya bit, but that's another topic ). Unfortunately in my experience it's been easier to move data around from package to package than manage the versioning and all that good stuff within Houdini. Versioning otls is a perennial problem and from what I can tell there's no really elegant solution. We recently went through this whole process here and it was fairly painful, so I'll share some of what we decided on. If anyone else has anything to add then that would be great.

- Storage of OTL instances (shot specific settings applied to an OTL)

This is on the cards for somewhere down the line in our pipeline. Our focus on it would be shot and sequence specific setups. For instance all our lighting and shading is handled inside houdini, so scene lighting lookdev as well as sequence specific shaders would be included in this. As for how to do it... well I've been putting it off .

- Publishing and storing OTL definitions? where? how? cvs? svn?

As far as I can tell it doesn't really matter much. We're partial to git over here, although we haven't used it for binary data yet. It should handle it fairly well, but that's another thing that's on our roadmap. I've been in a facility that uses svn and it worked fairly well. The hardest part of using a system like this is handling the checkin/out process in a way that makes sense to people who don't deal with it on a daily basis. That, plus building a system where artists can develop or modify otls in a way that doesn't affect everyone else using the tool. So checked out otls would have to be copied locally (presumably) and renamed so there are no conflicts with published otls; once they're finished being modified there should be a way to rename them and move them to the correct location for the show as well as checking in to the versioning system.

- Upgrading OTL, without breaking everything:

....- I want to add a parameter that wasn't there before

....- I want to remove a parameter that was there before

....- I want to rename an existing parameter

....- I want to leave the parameters alone but change the inner logic

....- Advanced: find all shots with an old version, replace with the new version and redo processing

Now this is where things start getting a little tricky. The system we use currently works fairly well, although how well it's going to scale remains to be seen. Basically we hardcode version numbers into the filename and the name of the otl. All versions are kept on disk and old ones are removed from the tab menus using ophide in the OPcustomize script. This is a fairly transparent system in that older hipfiles will still work with the version of the otl that they were using when saved and won't automatically version up when opened again. Of course it makes versioning up more of an explicit process, but in my experience there have been very few times when I would've wanted this to be automated so it hasn't been a problem for me.

....- Advanced: Wedges and takes, send to the farm small, medium and large explosion

No idea . I'm not much of a takes person to be honest. After opening one too many hipfiles with complete take overload and having to take the time to sift through them and figure out what was going on I've sworn off them. Probably the easy way out, but I'm too busy to deal with that mess.

- Things that should be included/NOT included in an OTL: for example should a geometry file be a parameter of an OTL or an input?

I'd prefer parameter, but I suppose it could be a case by case basis.

- Any tips or considerations? Whatever you do, DON'T.......

Render in Maya?

- Any wars stories or interesting tales loosely related to pipeline design.

I recently discovered this (probably waaay later than others, but better late than never). I think it holds true for pipeline design too :

The Zen of Python:

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Complex is better than complicated.

Flat is better than nested.

Sparse is better than dense.

Readability counts.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

Errors should never pass silently.

Unless explicitly silenced.

In the face of ambiguity, refuse the temptation to guess.

There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.

If the implementation is easy to explain, it may be a good idea.

Namespaces are one honking great idea -- let's do more of those!

ok not specifically related, but I like it

...er.. ok, my longest post in years. tiring..

M

[cam]

Hey Marc,

Great post lots of good info. The zen of python bit is cool, applicable beyond coding. The version numbers in the otl names + ophide is exactly the type of info I am looking for. And to Mr.Goldfarb, yes i think I have forgotten it all. I am actually back in next week and may try to rally the troops, I'll send you a mail.

cheers,

cl

[sam.h]

- Publishing and storing OTL definitions? where? how? cvs? svn?

As far as I can tell it doesn't really matter much. We're partial to git over here, although we haven't used it for binary data yet. It should handle it fairly well, but that's another thing that's on our roadmap. I've been in a facility that uses svn and it worked fairly well. The hardest part of using a system like this is handling the checkin/out process in a way that makes sense to people who don't deal with it on a daily basis. That, plus building a system where artists can develop or modify otls in a way that doesn't affect everyone else using the tool. So checked out otls would have to be copied locally (presumably) and renamed so there are no conflicts with published otls; once they're finished being modified there should be a way to rename them and move them to the correct location for the show as well as checking in to the versioning system.

- Upgrading OTL, without breaking everything:

....- I want to add a parameter that wasn't there before

....- I want to remove a parameter that was there before

....- I want to rename an existing parameter

....- I want to leave the parameters alone but change the inner logic

....- Advanced: find all shots with an old version, replace with the new version and redo processing

Now this is where things start getting a little tricky. The system we use currently works fairly well, although how well it's going to scale remains to be seen. Basically we hardcode version numbers into the filename and the name of the otl. All versions are kept on disk and old ones are removed from the tab menus using ophide in the OPcustomize script. This is a fairly transparent system in that older hipfiles will still work with the version of the otl that they were using when saved and won't automatically version up when opened again. Of course it makes versioning up more of an explicit process, but in my experience there have been very few times when I would've wanted this to be automated so it hasn't been a problem for me.

My current thinking is that we can use the system we have in place for publishing. Any asset that can be installed is locked forever, and artists would need to explicitly up version. This means older scenes would work since the otl that the artist was using would be guaranteed to be the same. Different versions are just in different directories ...

When it comes to up versioning time, the new version of the otl will replace all of the instances ... doing it this way might be a bit funny that you couldn't have two versions of the same otl in a scene. this simplifies things though, since Houdini can handle the parameter 'porting'. I did a few tests and this was predictable in most cases (removing a parm was a bit funny, but there isn't really a case for that, it can just be hidden)

What do you think about doing it that way?

The way you are describing seems to mean that any parameters would have to be copied between version manually or using a script?

- Any tips or considerations? Whatever you do, DON'T.......

Render in Maya?

[Marc]

When it comes to up versioning time, the new version of the otl will replace all of the instances ... doing it this way might be a bit funny that you couldn't have two versions of the same otl in a scene. this simplifies things though, since Houdini can handle the parameter 'porting'. I did a few tests and this was predictable in most cases (removing a parm was a bit funny, but there isn't really a case for that, it can just be hidden)

What do you think about doing it that way?

The way you are describing seems to mean that any parameters would have to be copied between version manually or using a script?

Are you talking about the artist versioning up an otl? The way you describe sounds like the way to go. Having more than one version of an otl in a hipfile at any one time sounds like a recipe for utter confusion if you ask me. Versioning up and letting Houdini handle the parameter conversion should work just fine in most cases. Persumably things aren't going to change that much once it's in production anyway. Progressive enhancement more than complete rewirtes .

M

[sam.h]

Are you talking about the artist versioning up an otl? The way you describe sounds like the way to go. Having more than one version of an otl in a hipfile at any one time sounds like a recipe for utter confusion if you ask me. Versioning up and letting Houdini handle the parameter conversion should work just fine in most cases. Persumably things aren't going to change that much once it's in production anyway. Progressive enhancement more than complete rewirtes .

M

Yeah, the artist would be upgrading to the latest version of an asset. Maybe just a little script that loads the new otl and removes the old otl. Having two versions of an otl is a bit silly, cause if we want variations then it should be built into the asset anyway .

This all sounds a lot simpler than I thought it would be ...

Edited August 19, 2009 by sam.h

[michael]

there have been a number of discussions about this in the past...

and it's always interesting...

in general, the way we did this on The Wild (and to an extent today) was as follows

way up in the pipeline we had a publishing/version control system that would read a hip file and look for:

in two different streams (asset and shot)

in asset >

'publish ready' assets (otls) - this was an asset that the version control system knew about, just added to the DB

it would copy this asset to a protected part of the server

using the comment of the node it would place metadata - asset type, name, version, unique publish code #, date, user comment etc

this was all done in a simple web based app.

in shot >

an artist would select the assets from the web app and Houdini would load them in and place instances in the scene

these would be arranged and animated etc

additional nodes (nulls etc) that were needed to complete the shot would not be 'publish ready', so the web app would simply opscript them to a cmd file

when the artist published the shot the web app would do the following things:

- record the list of published assets and their version numbers

- opscript ALL of those assets' top level parameters to cmd files

- opscript any additional nodes as above (extra nulls, blends etc)

down the pipe, in lighting etc, and artist could then go to a shot, open the web app, and select the individual assets (characters, sets, props, camera etc) and load then into the scene, the published assets would be loaded form the list and would be locked and read only, any cmd files associated with the asset would be run to set the parameters/animation, additional cmd files would be run to add the other stuff - nulls, dynamic parenting etc. This made it possible to load JUST the assets needed to do their work, she could then add assets (lights for eg.) and publish JUST these additions and/or the modifications to the already existing assets.

to update an asset an artist simply loaded the currently published asset for the shot from the web app, selected the new version from the published asset list and load it in, then republish the asset for that shot.

we ALWAYS did a 'pull' rather than a 'push' of new assets

- Storage of OTL instances (shot specific settings applied to an OTL)

as above with scripted cmd files that were published in the shot each with a unique publish code #

it's possible now (the Wild was H6x) to add parameters to any node - so I guess these would be scripted out also.

- Publishing and storing OTL definitions? where? how? cvs? svn?

python has all the tools needed to build a very powerful yet simple database front end to do this...

store the otls in a read only section of a server(s), a link n the DB

Houdini just needs the path to the otl file

- Upgrading OTL, without breaking everything:

this is where the real effort lives...

I can't count the number of hours I've spent talking about this (maybe thekenny knows )

building assets that play nicely in a pipeline is serious business

some things like props, set elements etc are pretty easy, characters and effects tools are much harder

one way to simplify things is to design assets that contain other more simple assets

for example a character rig asset might contain an asset inside that holds the mesh and the shaders, this way the rig and the model/shading can be modified and updated separately.

or a set asset might just be a container for a bunch of smaller assets - cityBlock1.otl contains buildingA.otl, buildingB.otl, buildingC.otl etc

....- I want to add a parameter that wasn't there before

easy, just add it

loading this asset into a file containing an earlier version will be just fine

....- I want to remove a parameter that was there before

Don't.

disable the parameter so it has no effect and make it invisible.

....- I want to rename an existing parameter

Don't.

you can change the Label but if you change the parameter name the internal functionality will still work but any existing channels will break

you can fix this with scripting but it can be a PIA

....- I want to leave the parameters alone but change the inner logic

easy, just do it

loading this asset into a file containing an earlier version will be just fine - but will change whatever it's doing, be careful when updating old files.

....- Advanced: find all shots with an old version, replace with the new version and redo processing

if you use a DB to store the info for each shot you can just use scripting to read that info, find the stuff you're looking for, edit this info = done

....- Advanced: Wedges and takes, send to the farm small, medium and large explosion

haven't done much with takes but I can't think of any real issues.

the wedge rop is just a node with parameters...easy to script out and control

- Things that should be included/NOT included in an OTL: for example should a geometry file be a parameter of an OTL or an input?

this all comes down to how you like to work

for me, I like parameters more than imputs

- Any tips or considerations? Whatever you do, DON'T.......

SIMPLE, SIMPLE, SIMPLE.

break things down into their smallest components

avoid building huge assets that you expect to do everything and that can accommodate every need - it's imposible (jeffW has some good thoughts on this here: Digital Asset Design (4 parts) http://www.sidefx.com/index.php?option=com_content&task=view&id=559&Itemid=216 )

effects tools are a hard one because even shots that seem the same will aways need different tweaks...

- Any wars stories or interesting tales loosely related to pipeline design.

for that I need booze...see you at Fionn's

[cam]

Hi Micheal,

Thanks for the detailed post. Some of the concepts are returning to my brain. I have to jump on a plane soon, so I can only touch on one issue. Back on the wild there was a lot of h-script and cmd files. Has the recommended workflow changed with the addition of python in houdini? I recall there was talk of supporting an xml (or perhaps yaml) format for otl instance information. Did this ever get any traction? I know there is collada, but this seems more like fbx, an exchange format. Opening a list of items in a shot in a webapp, AND being able to see (and possibly edit) parameters outside of Houdini is very appealing.

Thanks,

cl

[sam.h]

Hi Micheal,

Thanks for the detailed post. Some of the concepts are returning to my brain. I have to jump on a plane soon, so I can only touch on one issue. Back on the wild there was a lot of h-script and cmd files. Has the recommended workflow changed with the addition of python in houdini? I recall there was talk of supporting an xml (or perhaps yaml) format for otl instance information. Did this ever get any traction? I know there is collada, but this seems more like fbx, an exchange format. Opening a list of items in a shot in a webapp, AND being able to see (and possibly edit) parameters outside of Houdini is very appealing.

Thanks,

cl

I was talking to Dan about this, I can see the benefit with updating to a new version of an otl without opening a GUI session, but isn't changing parameters what the GUI is for ... ?

[dan wills]

I was talking to Dan about this, I can see the benefit with updating to a new version of an otl without opening a GUI session, but isn't changing parameters what the GUI is for ... ?

Yeah I can't really see the benefit of tweaking things outside the app - how would you see the effect of the change? I definitely do like the idea of automated batch scene assembly though.

About the other question re: saving settings/animation etc to apply to an asset, Sam and I think that this could be achieved in a neat way by the python "asCode" method. It basically turns a node and all its params, channels, expressions, spare params etc into a python script. The script usually creates the node too but with a small amount of mashing it could be made to work on an existing node as well.

Just doing a couple of tests it seems that the script it generates can't just be pasted into the python shell, but if you make it a shelf button it does work, so I think it's something to do with the python formatting - maybe line breaks or tabs or something.

Just a thought we'll need to do some more investigation on how well that could work.

cya : ) ,

dan

[michael]

@cam:

not much has changed...some things are better and others haven't seen much RnD love in a while...

@sam.h

with the otls listed in DB you can very quickly edit this data to change the versions in the scene...we had a script on the wild that allowed us to do this on a shot by shot basis or an entire seq or the entire film! - very dangerous

but this wouldn't allow for parameter changes...in the system described you'd have to edit the individual .cmd files stored for each otl instance, easy with scripting but yeah, you'd never see the effect of the change...unless you know exactly what the change is - yay proceduralism!

@dan wills

automated batch scene assembly is GOLD!

this can be extended to every stage of the pipeline...layout, anim, lighting, comp...

imagine having to make a change in animation after a shot has already been comped...if set up right you can run a signle little script and the whole shot (or single element) can be sent back through the pipe and out without any user interaction...this is the way things should be all the time.

re: saving settings/animation etc to apply to an asset

this can be done in hscript with opscript, and in python...it's super easy and fast, the bonus is that you've separated the asset version from it's parameters, so you can change both independently.