Pipeline Q: Swapping Otl Definitions Reliably

[Jason]

Hi there,

Part of our pipeline involves moving OTLs around the search-path; something in common practise in any pipelined facility.

I've found it pretty tricky to get 100% reliable behaviour when switching definitions and I'd like to know if anyone has any experience/advice/change of strategy that I can lever?

(This variable success rate has fluctuated through the life of H7 and H8, btw; so my methods have evolved but I haven't tried going full-circle with my techniques yet.)

The most common case where its failing is when we move an OTL up the OTL_PATH in a method where the artist is releasing his OTL for public consumption. In an ideal world we'd just mv the OTL and ask Houdini to rediscover the OTL along by examining the path again (Refresh the OTL). .. and I think it should work this way. Or can it? Anyone?

However, this cant work this way, Houdini just doesn't like it and doing a "Refresh All OPlibraries" can take minutes on a simple scene. It's MUCH quicker to quit and reload a scene than to Refresh All. So we must refresh ONLY the OTL/definitions we are interested in.

The method I have now works 90% of the time but for 10% it throws that "Retry/Embed/Quit" message and requires a restart of Houdini. Here is the pseudo-code for this:

# $asset_file is the name of the source
# $published_asset_file is the name of the destination
#
unix cp  $asset_file $published_asset_file

# switch definitions so that i can unload the old one
#
otprefer $asset_type $published_asset_file
updateui

otunload $asset_file

# sleep just in case the network slowness is causing OTLs to be mis-read
#   ...but it doesnt seem to help, damn..
unix sleep 1

# finally, remove the original and clear overridden definition trusting we're not using it anymore due to the otprefer.
#
unix rm $asset_file
otprefer -c
updateui

# all done
#

Anybody have similar issues? Alternate strategies?

I have tried "otload" on the destination asset after the copy but before the delete, but I believe that will add it to the "Current HipFile" section in the hipfile and that asset will no longer dynamically respond to changes in the OTLs in the OTL_PATH. Or am I mistaken here?

95% of the time we have ONE hda definition per OTL file. BUT, I've just noticed that perhaps I should be doing an otprefer in a loop to make sure we've swapped ALL our definitions to the published file. Before I try that, any other suggestions?

Thanks!

Jason

[sibarrick]

Well the way I do it is just to do a copy (otcopy) to a new location from within the file currently using it. This updates the hip file to look at the new one at the same time it publishes to a new location. I tend to leave the old one in the file too, but I don't see any reason why you couldn't remove it with otunload.

Does that achieve the same result you are after?

[Jason]

Well the way I do it is just to do a copy (otcopy) to a new location from within the file currently using it. This updates the hip file to look at the new one at the same time it publishes to a new location. I tend to leave the old one in the file too, but I don't see any reason why you couldn't remove it with otunload.

Does that achieve the same result you are after?

26685[/snapback]

This strategy requires that you loop through the OTL and otcopy each defintion out to the new OTL, right? Also, does it add any pointers to the new OTL location to the Current HipFile section of the hipfile in the Manager? I just need to ensure that no paths are baked into the hipfile; paths to the OTLs on disk MUST remain dynamic or the design crashes down.

Is there a way to do an effective "otlcopy" as opposed to "otcopy"? - kinda like a "otcopy source.otl */* destination.otl" if otcopy supported wildcards for the operator type. And thats why I think that that should be the same as doing a unix copy.

I'm just to keep the whole process short and sweet and remove any potential errors. Quite frankly I think that doing a unix copy/move should be the quickest and cleanest and really doesn't even require houdini knowing what is inside those OTLs at all - and thus should help in removing any possibility that Houdini itself might mess with any of the definitions on the way.

[michael]

something I did on the wild and posted about at the SESI forums went something like this:

then otwrite to write that new definition to disc

then otload to bring that new definition into your session

then otprefer to change the definition in your session to the new file

then otsync to match the definition up again

so you code would look like this:

# $asset_file is the name of the source
# $published_asset_file is the name of the destination
#
unix cp  $asset_file $published_asset_file

#NEW#
otload $published_asset_file

# switch definitions so that i can unload the old one
#
otprefer $asset_type $published_asset_file

#NEW#
otsync # the nodes using the new definition
otunload $asset_file

unix rm $asset_file

# all done
#

I can't remember what this does in terms of the Operator Type Manager sticking the new definition into the HIP or not...since we never dealt with hip files I didn't have to worry about it

HTH

[sibarrick]

I thought you might say that, we don't really ever need to do this on mass, so it doesn't really scale up to your situation.

I notice there is an otcontentls - so that could give a route to createing a loop. But it sounds very much like the method you have already should work it just needs Sesi to take a look at it to make sure it happens 100% of the time in a reliable way.

[Jason]

I thought you might say that, we don't really ever need to do this on mass, so it doesn't really scale up to your situation.

I notice there is an otcontentls - so that could give a route to createing a loop. But it sounds very much like the method you have already should work it just needs Sesi to take a look at it to make sure it happens 100% of the time in a reliable way.

26695[/snapback]

I agree; but I have no real idea why I get these problems only SOME of the time. I was hoping for a magic bullet, like 'Oh! You just need to "touch" the file, or run otrefresh.' or something.

[edward]

I've never tried this but looking at the online help, have you tried combinations of otmerge and/or otrefresh -r?

[mtucker]

In your script, you never actually otload the $published_asset_file. You do an otprefer, but otprefer doesn't load the OTL, it just says that if that OTL contains a definition for the listed optype, give preference to it. I can't think of any reason why the following wouldn't work:

unix cp  $asset_file $published_asset_file

# Use this line if your OPlibraries files contain wild-cards like "*.otl"
otrefresh -r $HOME/houdini8.0/OPlibraries
# Use this line if your OPlibraries files have explicit lists of OTLs
otload $published_asset_file $HOME/houdini8.0/OPlibraries

otunload $asset_file
unix rm $asset_file

There should be no need to otsync, otprefer, or do anything in a loop. If the above script doesn't work all the time, there's almost definitely a bug somewhere... The "$HOME..." argument to otload and otrefresh would need to change depending on whether you want the OTL published site-wide, shot-wide, show-wide, or whatever.

Mark

[Jason]

Thanks for the info, Mark.

One thing I would love to know is if its possible to do an otload without specifying an OPlibraries file and NOT have the path added to the Current Hipfile.

For example, just say I have this in a READONLY OPlibaries

/show/otls/*.otl
/show/shot/otls/*.otl

.. and I am moving an otl from /show/shot/ to /show. In this eg, you can see that any otl in the shot level will take preference.

In you example I'd have to otload the /show otl right? If I use otload then I'll be hardcoding the path to /show in the hipfile; so if the otl appears again on the shot level (someone else promoted it) then the current hipfile with continue to access the show level, and that is BAD. This makes me mortally afraid of otload, especially since we have a strictly read-only master OPlibraries file. Our pipeline 100% depends on cascading paths set by environment variables and so wish never to store a path to an OTL internally in a scene file, lest it break the beauty of dynamic referencing.

SO: Is there a way we can ask to otload an otl but if the otloaded OTL happens to satisfy the current OPlibraries then it could skip the adding-toCurrent-Hipfile-or-adding-to-OPlibraries step?

We'd have to continuously scrub hipfiles of everything in the Current Hipfile in the 456.cmd. We never want those in there in the first place. Is there a way to do this, by the way?

Or, I'd hope for a "otrehash" which would briskly rehash the OTL_PATH, scan all OTLs as if you'd opened Houdini for the first time. Maybe this should/could be the intended usage of otrefresh?

Also, I cannot bear to use "otrefresh -r" - even on an empty scene with 3 small custom OTLs in it, it takes about 2 minutes. ouch! I can't imagine using it on a moderately complex scene.

Thanks,

Jason

[mtucker]

One thing I would love to know is if its possible to do an otload without specifying an OPlibraries file and NOT have the path added to the Current Hipfile.

No, I'm afraid not.

We'd have to continuously scrub hipfiles of everything in the Current Hipfile in the 456.cmd. We never want those in there in the first place. Is there a way to do this, by the way?

Yes, it's possible, but a little ugly. If you run "otls" you'll get a list of all installed OTLs and the OPlibraries files that loaded them. The ones installed to the hip file will appear under the name "Current HIP File". You could parse this returned string using arg() to extract OTL file names off the end of the list and moving backward until you hit the string "File" (which marks the end of the "Current HIP File" part of the list). For each OTL, do an 'otunload otlfile "Current HIP File"'. And you could skip the loop entirely if the phrase "Current HIP File" doesn't show up in the otls results at all.

Or, I'd hope for a "otrehash" which would briskly rehash the OTL_PATH, scan all OTLs as if you'd opened Houdini for the first time. Maybe this should/could be the intended usage of otrefresh?

Yes, this is how "otrefresh -r" is supposed to work. It shouldn't cause any library to be reloaded that hasn't changed.

Also, I cannot bear to use "otrefresh -r" - even on an empty scene with 3 small custom OTLs in it, it takes about 2 minutes. ouch! I can't imagine using it on a moderately complex scene.

Well, that would be the bug here. I figured there had to be one that was stopping this from working in a sensible way. Searching through our bug database, it looks like this has even been submitted already. I've bumped up the priority on it. There is no good reason why otrefresh should take so long.

For now, if you add the script to otunload anything loaded by "Current HIP File", there should be no harm in simply doing:

otload $published_asset_file

In place of the otrefresh or otload commands in the script I posted. Not ideal obviously (because it relies on the 456.cmd stuff above), but it might work until the otrefresh bug gets fixed...

Mark

[Jason]

Thanks Mark, that certainly answers a lot of vague areas for me.

It really seems like "otrefresh -r" should be the ticket for the whole situation. One more question really just to confirm things: would doing an "otrefresh -r" refresh all currently loaded OTLs and load any new OTLS discovered along the paths described in OPlibraries? If so, this seems like it might be the perfect for us to do something as simple as:

unix  mv $asset_file $published_asset_file
otrefresh -r

..and it should gracefully handle whether the OTL has been moved (or copied) anywhere along the paths in OPlibraries, right? i.e. As long as the definition for a node can be found ANYWHERE in any OTL along the paths in OPlibraries we should be able to avoid that "Retry/Discard&Quit/Embed" message, correct?

If yes, , this would save us some stress and danger.. squash that otrefresh bug soon if you can!

[mtucker]

In theory, yes, that should be all you need. But I would still advise taking the "long" route of copy, refresh, unload, delete. The problem with the short version is that for a brief time Houdini is in a situation where it will not be able to find definitions for the assets in the OTL. As soon as you do the otrefresh, we should get out of that state and everything should be happy again, but I would still worry that some copy of the asset would try to do something between those two commands and wreck the whole process. With the long version, you are never in this state because you go from having one definition for the assets to two definitions, and back to one. So you are guaranteed to be safe.

Mark

[Jason]

Great, Mark; very helpful.

[sibarrick]

Well, that would be the bug here. I figured there had to be one that was stopping this from working in a sensible way. Searching through our bug database, it looks like this has even been submitted already. I've bumped up the priority on it. There is no good reason why otrefresh should take so long.

he,he I think that might have been me complaining about it last year, I'd really like to see this fixed too. If it wasn't me add my name to the request for it to be looked at. Rather than using refresh I often just stick down a new instance of the hda or shader and that seems to update just the one otl in a speedy fashion. Of course if you have lots of things changing that isn't practical.

Jason do you ever find open hip files crashing when you update an otl definition? Shader code seems to be ok but changing sop otls will cause hip files to bail out it there are more than a couple of instances of the hda being used.

[Jason]

Jason do you ever find open hip files crashing when you update an otl definition? Shader code seems to be ok but changing sop otls will cause hip files to bail out it there are more than a couple of instances of the hda being used.

Thats not a problem I've had, to be honest. Here we forbid "Save Operator Definitions to Hip File" and if you don't set this (in your OTL Configuration) then I can imagine that this where the problem might lie? Maybe deleting the stored definition could be an additional point of instability...

[sibarrick]

No it's not that, we don't save definitions to hip files either, this is strictly when we update on disk otls. Might be a windows thing i guess.

[Jason]

Might be a windows thing i guess.

26780[/snapback]

Ah, probably.

[mtucker]

As of 8.0.585, the speed of otrefresh -r should be greatly improved. Please let me know if there are still any problems with it after this version.

As for Houdini crashing when refreshing an OTL with several instantiated copies of a SOP asset, please send a hip file to support (if you haven't already). This is not a Windows issue, nor is it something you should have to live with

Thanks,

Mark

[sibarrick]

There should be one there already, but if i get time i'll send a reminder one...

[Jason]

As of 8.0.585, the speed of otrefresh -r should be greatly improved. Please let me know if there are still any problems with it after this version.

Woo!

Thanks Mark! I'll d/l and test this immediately!