Creating HDA node in python "very" slow

[RayFoundry]

Hi there,

I have a python script which traverses a subnetworks GEO nodes and adds a new node (which is a HDA) via createOutputNode. The added node itself is bypassed, so it should not cook (yet). The problem I'm having is that the creation of the HDA nodes takes 0.5 seconds each and the subnetwork (created from a CAD file in another step before) can have several thousand GEO nodes. This results in extreme loading times and very slugish UI performance even with 64 GB of memory.

If I leave out the creation of the HDA at each GEO node of the subnetwork, the issues with performance and crashiness are gone. (Please note, that the HDA node is bypassed in the above case and will not cook).

The reason why I'm adding the HDA into the GEO nodes of the subnetwork is that I need to export the subnetwork later on via a FBX ROP for example with all the NULL nodes (e.g. transform hierarchy) still intact. Adding the HDA at the "end" of the GEOs node pipeline allows me to do this. Basically the HDA processed incoming geo and adds  UVs, polyreduces and bakes textures for the individual GEO nodes of the network.

I already tried to rewrite/rewire the HDA to be able to iterate over primitive attributes such as "path" and have the whole content of the subnetwork processed from a single HDA instance. This works, but I'm loosing the link to the subnetwork and have to write out many individual FBX files etc. 

Now my question: Is it possible to speed up the creation of the HDA node somehow? I already tried setting the updatemode to manual, which did not change the creation time. Copying instead of creating maybe?

 

[lukeiamyourfather]

I would use the Performance Monitor window to monitor exactly what is making it take so long. There might be low hanging fruit to address to improve the performance of the HDA or how the HDA gets deployed. I've seen things as simple as an if statement in an expression causing an order of magnitude slowdown because it was causing other networks to cook unknowingly.

[Stalkerx777]

It's pretty uncommon task to create a node inside thousands of other nodes. The issue easily might be nothing to do with your Python code or HDA itself but simply the Houdini bottleneck. 

What I would try:

•  Profile your code with cProfile. This might give you an idea what takes time. Performance monitor doesn't help you in this case.
  You might be querying some Houdini parameter or geometry which makes Houdini cook it.
•  Run your script in Hython instead.
•  Try to do this in Hscript, although it's very unlikely you see any difference.
• Ask SideFx for support if you have a commercial license.
• Go crazy and write it in HDK

 

 

Edited February 14, 2018 by Stalkerx777

[RayFoundry]

Thank you for your feedback @Stalkerx777 and @lukeiamyourfather. I had already profiled the Object and SOP node creation. After doing it again, I realized that the Color SOP takes some more time than other nodes to create. I replaced those with a VEX wrange a la @Cd = {0, 1, 0} and it helped some. But my HDA was still the main culprit. I didn't profile the python code because the python nodes were not cooked at all (because the hda was bypassed already on creation).

In the end I'm now having one instance of the hda processing all the geometry. There was some scripting and code required to retain the hierarchy but it all worked out in the end. The execution times went from literally hours (and crashes) down to minutes. It seems like Houdini is just not designed for having complex OBJ node hierarchies with many leaf items that include hda nodes. Every creation of the hda via python took 0.2 seconds. With 5000 nodes this adds up ;o) I wonder if there's a way to speed up the creation of subsequent hda instances after a hda instance has been placed in the scene.

I'm using Houdini as a "geometry operating system" and from a design standpoint it would have been nice to have the functionality directly at the data/context. Now I'm having some controller style hda which collects data from the hierarchy and places the results back into the hierarchy after it's processed.

Problem solved though. Thanks again!

Edited February 15, 2018 by RayFoundry

[lukeiamyourfather]

You might could save the Houdini file as text and insert your stuff where it needs to go that way.