A few questions about cache

[Juraj]

Hi,

I have several questions about correct caching approach in Houdini.

In DOP Network node's settings there is Cache Simulation parameter. Disabling this parameter results in less memory usage, which is advantage for heavy simulations, however I noticed some strange particle motion in FLIPs simulated that way. When I enabled Cache Simulation, everything was ok. When I simulated Pyro with Caching disabled I didn't notice any problems.

 

So perhaps FLIPs works best with Caching enabled. But, when Houdini reaches memory limit, it starts to cache to disk, probably do Swap partition which slows simulating really a lot. What minimum Cache Memory limit should I set? I have 8 gig ram, but I noticed that when I set limit to 6,5 gigs, RAM would go full anyway. Wouldn't very small Cache Memory limit cause artifacts visible when Caching is disabled?

 

Then there is Explicit Cache option. I understand that it is good to enable it and save 3 last sim files in case of OS / Houdini crash. However I read somewhere on this forum, that it is better for heavy simulations as there is more memory left for simulation instead of cache.

 

What is the best approach for caching heavy simulations on PCs with smaller RAM? I want to focuse my few CPUs on simulating, not to wait till cache is written to slow HDD.

My usual approach is to save simulated geometry in bgeos after DOP I/O node, usually throught Hscript.

Hope you understand my questions

[danw]

I think FLIP sims require DOPs caching to function correctly, so no, don't turn it off from there.

Go to the FLIP Object node inside DOPs, under the Creation tab, and untick "Allow Caching" there instead.  That will cause the FLIP sim to unload any unrequired data as it goes, without messing up what it actually needs for the sim to function.

[danw]

Additionally, if you want to really squeeze what you can out of RAM... pre-bake all of your source SOPs to disk, and just reference File SOPs for any sourcing you do into DOPs.  Then look into running the sim scene from the command line so you don't need to actually load the Houdini interface.

 

Leave the DOPs cache size at default, as it should only use as much as it needs to function properly with the FLIP caching disabled.

 

You may be able to get a little bit of extra RAM back by checking the "Unload" flag on any input or output SOPs in your scene.  They'll still cook just fine.  (Avoid unloading anything that is an input for more than 1 downstream node, or is an input for a ForEach node, otherwise you'll end up recooking the graph multiple times per frame)

 

 

Lastly... go get more RAM.

8GB just isn't enough for FLIP... 16 really is a bare minimum... 64 is better :-P

[Juraj]

Thanks for response,

I didn't notice that caching option in FLIP Object.

Rendering source geometry is good idea and I'm already rendering it in Hscript from command line.

I'll have a look on Unload flag. I've never used it and it is on every node .

 

I agree that more RAM is better but anyway I am interested in optimizing my simulations.

Do you have any exprience with Explicit cache option in DOP network? Do you think that this option has any impact on performance or memory usage?

 

And one more question - does caching have an impact on pyro/smoke simulation's result? I haven't noticed any but anyway . If I am rendering pyro sim into BGEOs from command line than caching should be turned off, shouldn't it?

[danw]

I'm not certain what effect Explicit Cache has, as I tend to be even more "explicit" and output to disk from my own ROP inside the DOP network (well, actually a Null inside the DOP network, and a Dynamics ROP in the /out context that points to it)

 

That way it's definitely just outputting to disk, and shouldn't be keeping anything unnecessary around in RAM.

 

 

I haven't done much with smoke/pyro, but the Smoke Object has the same Allow Caching option on it.  I presume you use that as a base object for both smoke and pyro sims?

[Juraj]

I see there are many ways to cache simulation - in SOPs, DOPs and ROPs as well .

Yes I am using Smoke Object and I haven't noticed that option there neither, thanks

[danw]

There's only really two ways, they can just be created in a bunch of different places.  You can either cache the simulation state (.sim files), or you can import the geometry you're interested in from the sim into SOPs, and just cache the geometry data instead (.bgeo files)

 

The simulation state is useful for being able to re-launch a crashed sim without starting from the first frame again, but you still need to import the geometry you're interested in into SOPs at some later point, and they tend to take up more disk space, as they're caching everything about the sim, not just the result you're interested in.  (With huge FLIP sims that can add a fair bit of time to the equation, as moving an extra 1GB of file data per frame around a network takes time, plus you eat through the terabytes rather quicker!)

 

I presume that saving the .sim data and leaving the subsequent processing into SOPs until later would be the optimal approach if you're aiming to keep memory usage to a minimum during the simulation itself.

[johner]

I think FLIP sims require DOPs caching to function correctly, so no, don't turn it off from there.

Go to the FLIP Object node inside DOPs, under the Creation tab, and untick "Allow Caching" there instead.  That will cause the FLIP sim to unload any unrequired data as it goes, without messing up what it actually needs for the sim to function.

 

As Dan points out, this is indeed the recommended approach.  It's described a bit in the documentation, although that's not the greatest explanation.  DOPs needs caching enabled for generating collision geometry during substeps.  When solving for frame 10.5, for example, it still needs frame 10 in cache memory so it can generate a subframe version of the collision geometry as a blend between frame 10 and frame 11.  The FLIP Fluid object does not need to be cached; it's mostly important just for collisions.

 

Caching doesn't seem to have an effect for Pyro since it defaults to 1 substep, while FLIP defaults to 2.  If you increase the number of substeps for Pyro you'll need caching for the collisions there as well.

[Juraj]

Thanks for explanation, now it is giving much more sense.

 

One more question how do you restart your simulation?

File node can load .sim files, but how can I attach them to flipsolver or anywhere to continue in simulating?

[johnLIC]

I would also like to know how to restart a simulation from the last good .sim file.  ???