File Cache SOP / checking for "dirtiness"

[ikoon]

Please, is there any kind of File Cache SOP manager? I use the filecache more and more in my setups and I had an idea to create a script, which will:
- create some kind of CRC for all the filecache's upstream (or referenced) nodes and parameters
- check if saved cache's CRC is the same as actual CRC in the scene
- colour the filecache node if its cache is obsolete, disable "load from disk", print it to console, etc.

It feels almost the same as if Houdini is checking for dirty nodes. But I am not that skilled to use HDK. I can do just quite simple python.

Maybe there already is a tool for the Cache management? Could you direct me, please?

[pezetko]

Not sure what is the workflow you are after. Cache files are there to avoid computation on heavy graphs. As hip scene files are tiny usually compared to the size of cached data it's not a problem to save unique (locked) source scene file for each cache version in the case of need to return to the previous version.

I tried to create a quick prototype that computes a checksum from the input graph and save it in the data block and then compares it against the current graph. It's done on Windows (10) but it has an issue with the way how Houdini evaluates nodes.

 

pz_input_hash.hip

[ikoon]

Petr this looks amazing!!! You did exactly what I wanted! We should have "donate" button here, not just emoticons

I will dig into it. I want to use it in cases, when I build an effect in sops (motion design, videomapping to music), when I branch the SOPs a lot. For example I animate geo, I use it for sourcing particles. I also convert the geo to VDB, and morph it slowly to another shape. Calculate velocity field, swirling around the morphing shape. Then volume vop the velocity field to music... etc. and I still emit particles on each music beat from the initial animated geo.

I have filecaches along those branches, to be able to preview and iterate. But if I have to update some "far" upstream parameter ... then I get lost what filecache should I recompute.

You really REALLY help me to save a lot of time and nerves  I will PM you!

[CinnamonMetal]

@pezetkomind sharing your code; and where it is run as I see there is a new parameter inside the file cache node ?

[pezetko]

Hi Christopher,

everything is in the hip file attached under the video in my previous post. Python code is in the extra parameters on the nodes. Attrib create has the code to compute the checksum, it's stored as detail in the info block. File cache node has code for reading it from info block and comparing it against the current value.

 

Everything is evaluated when the nodes are cooked. This is also the issue that the file cache node is colored only when it's cooked and it requires it's input to be cooked too to get the correct current value. It is just a prototype. For a production, I would probably put all the code into separate python module or directly into HDA and load it from there instead. Event callback could be more resilient to the propagation of changes in the node tree too, but I didn't try that.

[anim]

I like the idea of getting checksum from the input graph instead of from the contained geometry itself

but if the comparison script needs to cook the live input geo to retrieve checksum detail attrib, then what's the point of having the geo cached? Especially for heavy sequences.

would maybe be worth of generating input checksum on the filecache node itself for comparison so that the input graph is not cooked

EDIT: just read your final post, where you mentioned exactly that, ignore me then

Edited September 9, 2018 by anim