Pyro sim eats all my RAM even when cache in dop is off

[cgartashish]

Hi,

Whenever I run a pyro cache, that is, I import the pyro fields and attach a fileCache node and write out the cache. The simulation eats up all my RAM as soon as the caching starts and then almost freezes my system. I always check off the cache sim option in DOP node that disables saving cache in RAM, but even then my RAM quickly gets filled up. I have created a basic pyro explosion not very large scale and I have 32 GB RAM. 

Is there any way to control or limit how much RAM Houdini uses for simulation? Like say, use 28 GB RAM and no more or flush previous cache from RAM automatically.

Thanks

[3dome]

how much RAM a simulation uses does not depend on the container size alone! It's the relation between container size and voxel size!
so without both values or a scene file it's probably impossible to help you.

I don't know of a way to control the RAM usage and dont think it's possible

[cgartashish]

On 5/18/2018 at 2:48 PM, 3dome said:

how much RAM a simulation uses does not depend on the container size alone! It's the relation between container size and voxel size!
so without both values or a scene file it's probably impossible to help you.

I don't know of a way to control the RAM usage and dont think it's possible

Hi,

I have attached my file, please can you look into the file and help me with this ram issue??

explosion_v01.hip

[3dome]

well you allow the sim to have almost 50 million voxels (when the explosion is fully expanded)

i'm currently only on a laptop with 8 gigs otherwise with 64gigs I could tell you how much RAM it will use,
because I think 50 mio voxels is not gonna fit into 32 gigs. if you dont have access to more RAM, you just have to lower the resolution until 32GB works and then maybe run a upres pyro

[cgartashish]

13 hours ago, 3dome said:

well you allow the sim to have almost 50 million voxels (when the explosion is fully expanded)

i'm currently only on a laptop with 8 gigs otherwise with 64gigs I could tell you how much RAM it will use,
because I think 50 mio voxels is not gonna fit into 32 gigs. if you dont have access to more RAM, you just have to lower the resolution until 32GB works and then maybe run a upres pyro

Thank you so much, One more thing, sorry to ask silly question but is there a way to estimate how much ram will be used at what resolution??

[3dome]

not that I know of.

I mean you could probably go and say ok I got 8 scalar fields and 2 vector fields that would be 8 bytes/voxel for each scalar field and 24 bytes/voxel for vectors = 112 byte for each voxel in sim
mutliply by 50mio = 5.6GB\frame

just to store the data!

now comes all the math, things get probably stored in variables and so on and all that is without any overhead/housekeeping, any custom fields, SOP calculations (like fluid source), maybe internal solver substeps double/triple... this value

and I could be completely wrong

Edited May 22, 2018 by 3dome

[Atom]

Here are two expressions you can use to estimate how many voxels your current settings generate. One for pyro and one for flip. This can help in dialing in RAM usage.

Pyro Voxel Estimate: (Apply To SmokeObject)
ch("sizex")/ch("divsize")*ch("sizey")/ch("divsize")*ch("sizez")/ch("divsize")

Fluid Voxel Estimate: ( Apply To FlipFluidObject)
ch("../flipsolver1/limit_sizex")/ch("particlesep")*ch("../flipsolver1/limit_sizey")/ch("particlesep")*ch("../flipsolver1/limit_sizez")/ch("particlesep")

The expression evaluates the division size and the box/domain area specified to contain the volume.

How To Install:

Select the FluidObject and click the gear icon to add a new parameter to the node. Add a float value and place it at the top. then paste the expression into the field. Click on the label to lock it in and you should see a scientific number presented to you.

At the end of the scientific notation is typically a +06, +07,+08. +06 means x 1 million, +7 means x 10 million and +08 means x100 million voxels. So this value is read as 1.9 million voxels (1.95313e+06).

The expression expects nodes to be named the same as the shelf tool generates. So if you use the shelf tools the expression should work. If you have custom node names in your network adjust the expression as needed.

Edited May 22, 2018 by Atom

[Atom]

As far as what to fields to cache out to disk, consider what fields you want to use for rendering. For Redshift rendering I only save Density and Temperature. For Mantra you may want to include Heat as well. The vel fields actually take up a lot of room because you have a scalar field for each axis (XYZ). I typically discard them when caching unless I know I need to leverage motion blur when rendering or I know I may have to re-time the cache after exporting.

Try setting up a test render and determine which fields you actually need to save. This can help optimize disk usage for cache's in general.

Edited May 22, 2018 by Atom

[cgartashish]

3 hours ago, Atom said:

As far as what to fields to cache out to disk, consider what fields you want to use for rendering. For Redshift rendering I only save Density and Temperature. For Mantra you may want to include Heat as well. The vel fields actually take up a lot of room because you have a scalar field for each axis (XYZ). I typically discard them when caching unless I know I need to leverage motion blur when rendering or I know I may have to re-time the cache after exporting.

Try setting up a test render and determine which fields you actually needs to save. This can help optimize disk usage for cache's in general.

Wow, I am very thankful to you. Thank you for helping me. I am sorry to ask one more thing, can you please explain me how to calculate the RAM usage based on the total voxels??

[Atom]

I don't know of a way to limit RAM usage other than managing your voxel amount. When I am trying to "fit" a simulation into my physical ram, on Windows, I bring up the Task Manager and observe the RAM usage. If it flat lines along the top then the simulation is too complex for the computer and you need to reduce the voxel count, or purchase more RAM. In a practical sense all Houdini simulations must fit within the physical RAM of the computer. This is where the estimator comes in handy. Lower or raise your Division Size in small amounts and observe the estimate value. What is also nice is that the Voxel Estimate expression works even if you set Houdini into Manual update mode (drop down in lower right corner of app). With large voxel counts even just changing divsize can result in long lags while Houdini updates. By setting this to manual and dialing in the Voxel Estimate value you can arrive at a new voxel count much more quickly. Remember to return it back to AutoUpdate once you have determined your divsize.

I often work at low voxel counts to get the overall simulation motion working, then lower the divsize to the final value for the overnight caching.

Edited May 22, 2018 by Atom

[Atom]

Here is your posted scene with the Voxel Estimator installed into the smokeobject. One thing you want to leave off when working on large voxel count simulation is OpenCL. If the simulation won't fit in your system ram it certainly won't fit in your video ram. The last thing you want is to have to re-simulate after an OpenCL exception occurs because Houdini ran out of VRAM.

Also, notice how the explosion looks fine when small. So you need less voxel count the futher the explosion is from the camera. This is where shot planning can really help out in speeding up your simulations. ILM uses 65million voxels for preview and 165 million voxels for final render. If you are generating more voxels than that, it would be considered a "Hero" shot and you may want to consider using a "Hero" computer to compute the solution.

ap_explosion_v01.hiplc

Edited May 22, 2018 by Atom

[vectorhuter]

On 22/05/2018 at 8:07 PM, Atom said:

Here is your posted scene with the Voxel Estimator installed into the smokeobject. One thing you want to leave off when working on large voxel count simulation is OpenCL. If the simulation won't fit in your system ram it certainly won't fit in your video ram. The last thing you want is to have to re-simulate after an OpenCL exception occurs because Houdini ran out of VRAM.

Also, notice how the explosion looks fine when small. So you need less voxel count the futher the explosion is from the camera. This is where shot planning can really help out in speeding up your simulations. ILM uses 65million voxels for preview and 165 million voxels for final render. If you are generating more voxels than that, it would be considered a "Hero" shot and you may want to consider using a "Hero" computer to compute the solution.

ap_explosion_v01.hiplc

Hi there! I have the same issue now but when you are previewing sim with low resolution, it looks nice and once you change the resolution, the sim becomes completely different. How can you do with that? Upres?