Creating Sims Beyond Memory Limit?

[Atom]

Hi All,

I am working on a low memory footprint machine. I would still like to generate a high-quality pyro sim. Is this possible? I have found that if I leave OpenCL turned on I get OpenCL warnings when I set the particle division to 0.02. So I have turned off OpenCL on my 2GB card. So I have lost that acceleration on this very slow machine. When I walk away and I come back, the machine is completely locked up and unresponsive. This forces me to reboot the machine. So I disabled any caching at all, I am just trying to write out a .bgeo.sc series to my disk which has hundreds of gigs free.

 

Is it possible to create sims beyond the physical memory limit?

Or is this just an exercise in futility...

 

Ok, I guess my computer answers this one with a Fatal error: Segmentation fault.

I get the segmentation fault even before physical memory is exceeded. This is a frankenstein parts machine so I guess memory could be bad. But I still wonder is it possible to just write simulation data to disk even with a low memory footprint?

 

What do I look for in the crash.log to resolve this issue? I am just assuming it is my hardware but maybe it is a Houdini bug?

A website states...

Quote

The Sig11 FAQ

 

QUESTION

Signal 11, what does that mean?

 

ANSWER

Signal 11, or officially know as "segmentation fault", means that the program accessed a memory location that was not assigned. That's usually a bug in the program. So if you're writing your own program, that's the most likely cause. However, this FAQ will concentrate on the possibilities besides that.

 

Edited December 7, 2016 by Atom

[Skybar]

Well, simple answer is no. The simulation needs to keep X amount of voxels/particles in the memory - and at some point of resolution the amount of information needed will exceed your physical amount of RAM, where it simply cannot compute anymore and usually crashes. 

There are steps to reduce the overall footprint of your sim, as you said. But even then, if it can't fit it can't fit. It's like shoving apples in a box, at some point it will be full. You can cut the apples into squares (optimize) to fit a few more, but the box is still the limit. 

[Atom]

Thanks for the response, I guess I'll give up that sim approach on this junk machine.

[Guest tar]

using hbatch saves some memory

 

[Yon]

Don't use .02, always .025.

It's like a magical threshold number.

I had a pyro sim at .02 and it was like 19 hours at 60/120. I moved it to .025 and it ran the whole range in like 6 hours. I've seen others mention it to.

[Guest tar]

You can also sim with less depth to the domain and billboard it. One of these is 130 mil voxels, the other is 615 mil voxels.  Save memory and sim time, and, keep details!

Fire.hipnc

[nigelgardiner]

Hi Marty

Sorry can you explain what you mean by "sim with less depth to the domain and billboard it"?

thanks

Nigel.

 

[Guest tar]

It's in the above .hip. The idea is that you want the outline of the fire, the silhouette, to have the detail. So you shrink the box in the direction of viewing.  Billboarding is when you populate a scene with flat renders/ilveaction/sprites all facing the camera. It's very effective, though it always breaks down for realism when pushed too far.

Lots of work is done this way. faking it. exactly the same idea of using movie sets instead of building something real

 

[lukeiamyourfather]

On 12/7/2016 at 0:04 PM, Atom said:

Is it possible to create sims beyond the physical memory limit?

The short answer is yes but it depends on a few things. When the system runs out of physical memory it can use swap or virtual memory. As long as there's enough swap or virtual memory it should work. If you run into this situation much I'd avoid Windows. Everything is going to shit the bed to some extent when you run it out of memory but Windows is going to be much worse about it than other platforms (specifically Linux and somewhat OS X). I've run jobs that end up using hundreds of gigabytes of memory with 128GB of memory in the machine (and 256GB of swap).

On 12/7/2016 at 0:04 PM, Atom said:

Or is this just an exercise in futility...

It reduces the performance significantly. We're talking many orders of magnitude. Tasks that would take minutes or hours with enough memory quickly become overnight tasks (or next week/month/year tasks). Upgrading your hardware is the much more practical solution. Memory is cheap these days and consumer hardware can handle 64GB and 128GB of memory.

[Atom]

Thanks Luke, this AMD machine is not worth investing in. I am just playing around with Apprentice on this HP Pavilion while my main machine is locked up rendering.

[shawn_kearney]

Here is a random thought... what if you took a low resolution sim for the general shape, extracted the outer-most voxels and use that as a source for a higher resolution sim, kind of like how Mantaflow works? Because you're not simulating the interior, there would be fewer voxels to calculate. You'd have to set it up the pyro parameters specifically for this purpose, but there could be some interesting options possibilities here, also.

Edited December 11, 2016 by shawn_kearney