Jump to content
cudarsjanis

Houdini RAM usage, CPU? limit

Recommended Posts

Is there any way how I can limit Houdini RAM and CPU usage while writing out simulations with ROP outputdriver?

My system crashed few times already while writing out whitewater particles. and I had to start all over again.

In Adobe software there's simple setting in preferences - how much RAM to leave for other software to use.

In houdini what I found is that I can bump down cache size on DOP, and that is basically it.

I'm new to houdini.

Also I'm using save to disk in background. so it should use less ram

Now simulation is 85% done. so gonna render everything I hope.

What I did is I unchecked write cache al all on DOP net and whitewater DOP, don't know if this is the most efficient way to render.

 

Any advice would be appreciated.

Thank you

 

My system is not so bad, so I should be able to write out large simulations.

 

Intel® Core™i7 Quad Core Processor i7-4790k (4.0GHz) 8MB Cache

32GB KINGSTON HYPER-X FURY DUAL-DDR3 1600MHz (4 x 8GB)

4GB NVIDIA GEFORCE GTX 970

 

Share this post


Link to post
Share on other sites

Reduce the resolution of your simulation. It pretty much needs as much RAM as it needs.

Share this post


Link to post
Share on other sites

You can also try the resume cache feature that way you can pick up where the crash happened. I just learned about this. It is located on the Cache TAB of the AutoDop Network. Try activating Save Checkpoints and read the help card.

Share this post


Link to post
Share on other sites

what operating system?  windows really sucks with memory management in houdini.  linux is boatloads better.

 

save in background actually INCREASES ram, so if you're hitting a problem with ram, turn it off.  save in bg will spawn off copies of your sim data to a write cache after each frame is simmed so it can continue the sim while a separate thread does the writing from that cache.  if you have a really massive data set or a slow drive, you might end up with two or three or more whole sims in ram at the same time.

Share this post


Link to post
Share on other sites

32 GB is not so much for simulations - more than enough for most programs and rendering etc, but simulations require a lot. Think 64GB as the median.

Share this post


Link to post
Share on other sites

what operating system?  windows really sucks with memory management in houdini.  linux is boatloads better.

 

save in background actually INCREASES ram, so if you're hitting a problem with ram, turn it off.  save in bg will spawn off copies of your sim data to a write cache after each frame is simmed so it can continue the sim while a separate thread does the writing from that cache.  if you have a really massive data set or a slow drive, you might end up with two or three or more whole sims in ram at the same time.

 

I'm on Win 7

drive isn't slow as I'm running my system on SSD

Writing data on SATA hdd, though.

I rendered out the with lower particle count. around 3 Gb per frame on peak now.

not so much detail in foam now as the most of the particles are bubbles.

 

So save in background really uses more RAM at the end? ok, so not gonna use that for large sims. 

Edited by cudarsjanis

Share this post


Link to post
Share on other sites

yeah, i figured it was windows....

 

writing directly to sata.... that should be reasonably fast.  turning off save in bg means your sim will park while writing to disk, but you don't have the ram to spare.

are you doing much post processing on your geo prior to writing?  you might need to unload any sops you're adding into the mix to help ease the ram load.

Share this post


Link to post
Share on other sites

what operating system?  windows really sucks with memory management in houdini.  linux is boatloads better.

 

There really needs to be a fix for this. There are those of us who are  _relatively_  technical minded, but not to a Linux degree - and I'd much rather spend my time trying to master Houdini then trying to figure out how to simply install a driver or a program. I know for many of you that's laughable, but I do have priorities in my life - and I'd rather focus my efforts elsewhere. 

Share this post


Link to post
Share on other sites

Is there any way how I can limit Houdini RAM and CPU usage while writing out simulations with ROP outputdriver?

My system crashed few times already while writing out whitewater particles. and I had to start all over again.

In Adobe software there's simple setting in preferences - how much RAM to leave for other software to use.

In houdini what I found is that I can bump down cache size on DOP, and that is basically it.

I'm new to houdini.

Also I'm using save to disk in background. so it should use less ram

Now simulation is 85% done. so gonna render everything I hope.

What I did is I unchecked write cache al all on DOP net and whitewater DOP, don't know if this is the most efficient way to render.

 

Any advice would be appreciated.

Thank you

 

My system is not so bad, so I should be able to write out large simulations.

 

Intel® Core™i7 Quad Core Processor i7-4790k (4.0GHz) 8MB Cache

32GB KINGSTON HYPER-X FURY DUAL-DDR3 1600MHz (4 x 8GB)

4GB NVIDIA GEFORCE GTX 970

 

What you want to do is make sure you disable caching on the Autodop node (which you already had), and inside on the object. Disable pre-conditionar maybe?  Clean any residual caches in the cache manager. When you write out your sim files add dot sc e.g. bgeo.sc (use the BLOSC compression) which should be the default if you're using compressed fluids workflow. Now you want to get rid of attributes you don't need before the ROP with an attribute delete node. You might want to delete primitive groups you don't need as well since it was packed data (depending on what type of sim it is). I just started to learn Houdini, and these things helped to reduce file size, and speed things up reading them back in. Not sure if this is what you're trying to do.  Your system is enough to do a lot of sims. I have a similar system, with 6700K 4.6ghz and Titan X SC with 32 Gigs of DDR4 memory.  I don't see any improvement using OPENCL for fluid sims. I understand you have to make the whole workflow opencl friendly, but I haven't don't that yet, and I haven't got to learning PYRO either, except for making little torches for lighting.

Edited by Syrena

Share this post


Link to post
Share on other sites

There really needs to be a fix for this. There are those of us who are  _relatively_  technical minded, but not to a Linux degree - and I'd much rather spend my time trying to master Houdini then trying to figure out how to simply install a driver or a program. I know for many of you that's laughable, but I do have priorities in my life - and I'd rather focus my efforts elsewhere. 

 

This sounds fine in theory but memory issues may not be fixed for a while, so the realpolitik is you have to use Linux or OsX as it has superior mallocs.

 

Think of learning Linux, which has tones of help online with Ubuntu or Mint, as a long term investment in the best architecture for the future.

Share this post


Link to post
Share on other sites

This sounds fine in theory but memory issues may not be fixed for a while, so the realpolitik is you have to use Linux or OsX as it has superior mallocs.

 

Think of learning Linux, which has tones of help online with Ubuntu or Mint, as a long term investment in the best architecture for the future.

 

Yes. You're probably right. :D.

 

There's just so much to learn in our industry and so little time! 

Edited by LukeLetellier

Share this post


Link to post
Share on other sites

Man, I was worried at first about diving into Linux. These days, Ubuntu or Linux Mint have so many drivers ready to go out of the box, it's very, very straightforward to get going, and as intimated earlier, there's lots of help online. And if you can't completely shed Windows, either Dual Boot, or get a second machine + display and use http://synergy-project.org/ to share your keyboard/mouse. I find Mint is a wonderful blend of stability (based on Ubuntu Long Term Support), recent features/fixes, and easy to install drivers.

 

But back to the OP: yeah 32GB will get eaten up pretty fast, especially with SIMs volumes. I've worked on 96GB and 128GB workstations, and still find myself hitting the limit. Another win for having gobs of RAM, is it gives you more time to catch a mistake that's eating all of your memory. Accidentally hitting 0.0001 on Voxel Size, or scattering billions of points, can be annoying (working in manual cook mode is a good workaround).

Share this post


Link to post
Share on other sites

Ya - a good tip for fiddling with Linux is to have a second computer on the Google, then you just search your issues when the main one drops to grub!

  • Like 1

Share this post


Link to post
Share on other sites

What you want to do is make sure you disable caching on the Autodop node (which you already had), and inside on the object. Disable pre-conditionar maybe?  Clean any residual caches in the cache manager. When you write out your sim files add dot sc e.g. bgeo.sc (use the BLOSC compression) which should be the default if you're using compressed fluids workflow. Now you want to get rid of attributes you don't need before the ROP with an attribute delete node. You might want to delete primitive groups you don't need as well since it was packed data (depending on what type of sim it is). I just started to learn Houdini, and these things helped to reduce file size, and speed things up reading them back in. Not sure if this is what you're trying to do.  Your system is enough to do a lot of sims. I have a similar system, with 6700K 4.6ghz and Titan X SC with 32 Gigs of DDR4 memory.  I don't see any improvement using OPENCL for fluid sims. I understand you have to make the whole workflow opencl friendly, but I haven't don't that yet, and I haven't got to learning PYRO either, except for making little torches for lighting.

Getting rip of attributes that I don`t need is a good idea. Thanx for the tip!

 

Not sure If I want to set up Linux and try to learn that as well. Already struggling with Houdini :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×