How to proceed to overide default HDA in a clean way ?

[sebkaine]

Hi guys,

 

sorry for the noob question , but 

 

I would like to know what is the recommended workflow to overide default digital asset in a clean way.

For exemple on windows i want to overide  :

 

The Stereo Camera rig definition contain here 

C:/Program Files/Side Effects Software/Houdini 14.0.474/houdini/otls/OPlibObject.hda 

 

The Mantra Rop definition contain here 

C:/Program Files/Side Effects Software/Houdini 14.0.474/OPlibScripted.hda

 

How would you proceed to be able to use :

- your own custom definition of those default assets

- every machines of the network when houdini is started will use those custom definitions

- your assets will be store on a share folder

- none of the default assets store in the houdini install file will be broken / erase / corrupted 

- no bugs / weird surprise can occur because you have done things in a clean way

 

Thanks for your lights !

 

Cheers 

 

E

Edited November 2, 2015 by sebkaine

[lukeiamyourfather]

No. Just no. Create new digital assets with alternate names. Thing like shelf tools can be duplicated and customized for your needs too. Environment variables can be used to point clients to a shared directory with custom digital assets and tools. There are no good reasons to override stock nodes and tools.

[fathom]

yeah, i have to agree -- don't customize built-in nodes.  create work-alikes or whatever, but always keep the vanilla version available and clean.

 

for custom nodes, all you need to do is define a few env vars.  on windows, it might be easier to create a python wrapper that handles the env vars internally for you rather than having to f-around with windows env vars.  HOUDINI_OTLSCAN_PATH will let you adjust where otls are located in a semicolon separated list of paths.  be sure to include "&" as one of the paths so it will grab the sesi otls.

[sebkaine]

Many thanks for your advise guys ! i will make myself the economy of destroying my H install and do as you said.

- copy assets

- rename them

- use env var to point directly to the share folder

 

The workflow to copy / rename some assets is still not crystal sharp in my mind.

If we take the exemple of the Mantra Rop

 

It's definition is stored in

C:/Program Files/Side Effects Software/Houdini 14.0.474/OPlibScripted.hda

 

The OPlibScripted.hda contains a lot of other assets definitions.(cf img)

 

How would you proceed to cleanly :

- isolate only the Driver/ifd definition

- copy it in W:/BDD/OTLS

- rename it like something "mantra_pbr" to avoid any conflicts with the default definition

- reimport it and be able to see it in the Render tab not the Digital assets tab

 

Is it possible to do this ?

Is it a good idea ?

 

There are tons of parameters that i would like to offuscate in Lights / Mantra Rop / Camera and other nodes to simplify the PBR stuff.

not for me but in order to help lighter to not run away in front of H ...

 

would it be even possible to do the same things for

- Lights

- Env Lights

- mantra rop

- mantra shaders

- camera

- stereo camera

do the exact same operation copy/rename , but to pack all those new definitions inside one compilation OpLibPBR.hda

and thus instead of having to load an .hda for each assets i only have to load the compilation located in :

W:/BDD/OTLS/OpLibPBR.hda ?

 

Thanks again for your time !

 

Cheers

 

E

Edited November 3, 2015 by sebkaine

[acey195]

if you do not need to change the insides of the node, but just the interface,

you could also just put the native node in an subnet,

promote all the useful parameters and save that as a new hda.

that way it would also be patched automatically with new Houdini versions.

[f1480187]

Process is straightforward for standard OPlibs. Just create an asset using existing type name and place it in your public houdini folder. No need to override whole libraries, use an arbitrary name (babel_tower.hda for example). Houdini will use all standard nodes from libraries excluding overridden, using overrides instead. Tools may break, because they manipulate old definition's data such as parameters which no longer exists on new assets, you may want to rewrite and override scripts also. You can also override built-in nodes, but I don't know how to do it smoothly. There will be script errors on startup. Overriding $HH/OPbindings and $HH/OPcustomize files (and maybe something else) may be the proper way. Then you need to update it every build.

 

 

There are tons of parameters that i would like to offuscate in Lights / Mantra Rop / Camera and other nodes to simplify the PBR stuff

 

I believe that "Save as Permanent Defaults" was created for this. All you need is to store it in your_public_dir/presets/Driver/ifd.idx

[sebkaine]

Thanks for your answer guys !

 

In fact the "Save as Permanent Defaults" is exactly what is use right now it's a cool stuff but it is save in pref and doesn't open the door to inner modification.

The idea of building an empty subnet and promotte things is good but it introduce an extra level of hierarchie to access the node, and i think i don't want that.

I find that defining your own version of the assets that you rename <my>_<assetname>.hda that you can version / store in one central place more elegant. 

 

The thing i was searching ... and now i feel a little stupid  since i find it is     :

- select the assets you want to copy

- left mouse > operator type manager

- left mouse > copy 

- set the new name / set the path and voila ...

 

Now 2 grey area remain in my mind 

- what is the difference beetween Copy ...  and Duplicate ...

 

the doc say :

 

Copy  = Copies between .otl files, which keeps the last path.

 

Duplicate = Makes a duplicate within an .otl, which defaults to the type’s current path

 

What is the best one ?

- i would say copy for copy the assets 

- duplicate to create a new version 

But i'm not 100% sure 

 

And as you both have highlighted how you make this works with new version of houdini ... cause spending weeks on buildings assets and not being 

able to open them in H16 would be kinda counter - productive at the end

Edited November 3, 2015 by sebkaine

[sebkaine]

i forward this one here , must read :

http://forums.odforce.net/topic/9650-designing-a-digital-asset-workflow/?hl=%20rename%20%20node%20%20input#entry63377

[f1480187]

For me it seems that "Duplicate" automatically fills path field with hda's path in dialog window.
 

And as you both have highlighted how you make this works with new version of houdini

If you use asset with standard locked node inside, this node will be most recent ("would be patched automatically"). If you fork a subnet asset, it won't be updated anymore, you'll have to maintain network and parameter interface like in any normal user asset ("you need to update it every build").

 

 

 

 

My library isn't incredibly large and, mostly, code-based, I just spend a half of a day every major release, since I rarely modify existing assets. Maintaining large library of constantly updating nodes is potentially painful. For example, mantra node's defaults/copies need to be thoroughly reviewed or recreated on each release due to changes in interface. I found it is quite tricky to track it. But I believe most of simpler nodes can be maintained without doing anything.

[sebkaine]

Thanks again for your feeback !

 

so at the end

- i have done a copy of the stereorigcamera in the operator type manager ( RMB Menu )

- put that copy in W:/BDD/HDA/f_stereorigcamera.hda

- i name the first version f_stereorigcamera_001

- i edit my houdini.env with the following line :

HOUDINI_OTLSCAN_PATH = W:/BDD/HDA;&

- and i am versionning by saving version under f_stereorigcamera.hda by iterating HDA name like this f_stereorigcamera_002 ...

- i hide the previous definition of the assets in the TAB menu by adding in :

 

C:\Program Files\Side Effects Software\Houdini 14.0.474\houdini\OPcustomize

 

the following code :

ophide Object stereocamrig_f001

- Hope this can be helpful ...

- If  you see any big mistake here that would be cool to give your feedback.

 

Thanks for all your helpful answer, things start to enlighten ...  

Edited November 3, 2015 by sebkaine

[fathom]

is your houdini.env file the same for every machine on your net?  and for every user?

 

if i'm following your method here, don't do your versioning like that.  you want to retain the same name for what should be a compatible node so that you can fix things or add functionality without having to replace that node everywhere.  when you make a compatibility breaking change, then update to a new name (or use the version token thing -- but i'm vague on that one.  you can look at the scatter vs scatter::2.0 as an example of how that might work).

 

in terms of where stuff lands on disk, i keep minor versions in a dev folder and then copy the released (hopefully latest) version to a single common directory.

 

when i work on an otl, i check it out to a local dir and load it directly by name (that dir is also in my otl path).  that way i can modify it and test it out before versioning up and releasing it to others.  by keeping it the same name, everybody gets the update automatically.  this kind of based on how things were done at r&h.

 

as things grow, you have to start considering multiple versions of houdini and how you'll handle incompatibilities.  should you have different branches for each major release of houdini?  most stuff works across releases, but some stuff does not.  do only those particular items get entries in a release specific folder (that overrides the standard folder).

 

lots of questions to be answered that you won't really appreciate until you've f'd it up once or twice.

[sebkaine]

Many Thanks for your very helpful post Miles !

 

is your houdini.env file the same for every machine on your net?  and for every user?

 

I think a correct answer to this question should be no, so i go back to your original suggestion that sound better.

- Best way is to set env var at OS level in windows or linux

- Python is your friend for this on both system

 

 

if i'm following your method here, don't do your versioning like that. 

you want to retain the same name for what should be a compatible node so that you can fix things or add functionality without having to replace that node everywhere. 

when you make a compatibility breaking change, then update to a new name (or use the version token thing -- but i'm vague on that one.  you can look at the scatter vs scatter::2.0 as an example of how that might work).

 

Now that you've said it it's crystal clear i'm doing thing wrong. In fact with my system to update a version of the assets you have to overwrite the same file people are accessing all over the network.

To be sure that nothing will break.

- you have to set one file per version and increment 

- each past version stay in place and all scene done with this past version will still work

- you have to send a warning that a more recent version of the asset is available

- the user update manually if things doesn't explode, but can go back to previous version if things does explode ...

 

So new problem must be solve for me , you have to build for each asset you load ;

- a clean versioning UI  

- a routine that check if no new version is available 

- a way to send a warning in the node to signal that an update exist 

 

I tend to agree with you that an old school system base only on name :

asset_foo_v001.hda

asset_foo_v002.hda 

instead of using the SESI versionning system might be the most simple way to do things.

 

 

in terms of where stuff lands on disk, i keep minor versions in a dev folder and then copy the released (hopefully latest) version to a single common directory.

 

well i don't believe in latest version so their can't be a asset_foo_final.hda > final doesn't exist in VFX

 

my problem is that i would like to have an organization like this :

HDA / rendering / asset_light / asset_light_v001.hda

HDA / rendering / asset_light / asset_light_v002.hda

 

Basically each asset has a folder with his name that contain all the version :

asset_light_v001.hda

...

asset_light_v00n.hda

 

But also like you mention a dev folder that contain all the wips :

HDA / rendering / asset_light / dev /

dev_asset_light_v001_t001.hda

...

dev_asset_light_v001_t00n.hda

 

The problem is that i don't know how to force Houdini to search for HDA inside a folder recursively ?

If you set :

HOUDINI_OTLSCAN_PATH = W:/BDD/HDA;&

Houdini will search asset only in HDA folder where there only his family folder : rendering / layout / fx / modeling / anim / compo ....

if i want to make things works the way i want i have to set :

HOUDINI_OTLSCAN_PATH = W:/BDD/HDA/rendering/asset_light; W:/BDD/HDA/layout/stereocam ......... <this_for_each_asset>  ; &

And i ended with a huge OTLSCAN var ...

I also need to hide all version that are not activated by accessing OPcustomize and ophide ... 

 

That would be far easier to scan a folder recursively ?

 

when i work on an otl, i check it out to a local dir and load it directly by name (that dir is also in my otl path).  that way i can modify it and test it out before versioning up and releasing it to others.  by keeping it the same name, everybody gets the update automatically.  this kind of based on how things were done at r&h.

 

I agree that's the way to go for clean workflow in big companys:

- you do your stuff on your PC locally 

- when your stuff is good enough you publish 

- and it copy your stuff on the server and increment the version number 

 

But that's quite some work to make it work smoothly and for small facilities i find it easier to do all on the server, so i guess i'm going to stick with that for the moment

 

 

as things grow, you have to start considering multiple versions of houdini and how you'll handle incompatibilities.  should you have different branches for each major release of houdini?  most stuff works across releases, but some stuff does not.  do only those particular items get entries in a release specific folder (that overrides the standard folder).

 

lots of questions to be answered that you won't really appreciate until you've f'd it up once or twice.

 

Yes the more i see it the more i realise that you can build yourself some serious rubiscube challenge for your brain by trying to run a Houdini pipeline smoothly ...

An you are absolutely right no better way to f'd up once in a while to correct the trajectory ...

 

Thanks again for your extremely helpful post !

 

Cheers 

 

E

Edited November 4, 2015 by sebkaine

[fathom]

happy to be of assistance.  my method (based on r&h's) is to add a few items to the right-click context menu to handle all the tool stuff.

 

if the item is not a currently an otl located in the otl tool dir, you can create a new tool, which creates an otl in a local tool dir (local to the shot you're working on)

 

if it's already a tool and is located in the release dir, then you can check it out to edit (current version or any previous released version).  this copies the tool to your local shot dir and locks it in the release dir so nobody else can check it out.

 

if it's a tool and located locally (ie, you've check it out already) you can either minor revision it (same name, new revision goes to a dev folder and then the release folder as well), major revision it (new name because it breaks backwards compatibility), or discard your edit and revert to the released version.

 

 

with that system, i only need a single released directory where everything is always the latest version to be used.  i've got subdirs for all the dev stuff -- each node gets its own dir with subdirs for each version that includes otls and txt files for descriptions of what was done.  the dev stuff isn't ever touched by houdini scan path, so it doesn't need any special handling.  i do have an additional scan path that's added when you work on a shot in the shot structure, so that's where "custom" otls can be placed that are only picked up when working on a specific shot.  it doubles as a place to work on an update to a tool in a realworld setup.  when the edits are done, the otl gets removed and placed in the release dir (and the dev dir as well).

 

 

once it's up and going, it's very easy to check things out and edit them and make sure they work before releasing them back to the public.  ideally it's a silent update that won't break things, but if it does break things, you can always revert to a previous version.

 

 

another thing to consider is the ability to manually scan your own otl paths by using a 123/456.py file.  again, you have to figure out how to get everybody the same file to source in, but once you do, you can do your own otl loading for any number of directories if you so chose.

[sebkaine]

Thanks again for your great input Miles, this is extremely helpful !

 

I must confess that i will need a little more time to digest all this and see how i can implement some part of this.

I will come back when i will have figure how to blend this in my workflow ... 

 

Cheers 

 

E

Edited November 9, 2015 by sebkaine