Fluid Sim, Higher Resolution?

[Atom]

Hi All,

I am working on a fluid sim and I have reached the limits of my 32GB physical ram. I can surface my simulation at a particle separation of 0.4, but this resolution does not offer the quality I am seeking. When I lower the particle separation to 0.375 I get an acceptable quality but surfacing maxes out my physical ram and output slows to a crawl, slower than a crawl, think slug movement. When I went to sleep last night my computer was on frame #137, this morning it is still on frame #137. The CPU does seem to be working, but mostly single core. My memory graph is a flat line at the top, so all memory is in use.

My motherboard claims to support 64GB, but it is desktop non-ecc memory, not server ecc. A memory upgrade would take the form of a 16GB non-ecc unbuffered ddr3 ram. This memory chip does not seem to exist anywhere in the world. I can't find a manufacturer that makes such a chip.

Is there anyway to partition flip simulations on a local machine so I can dedicate all 32GB to only a portion of fluid sim? I am aware of the Distribution tab for the solver but I don't know how to set it up. Will distribution help in this situation or will a distributed sim still require all my memory?

Thanks for any tips!

Here is my fluid sim up to frame #137 @ 0.0375 particle separation, maxing out 32GB of physical ram.

Edited January 25, 2017 by Atom

[Federico]

I had a similiar problem in my last project and i'm curios about the distributed sim too. The only things that i can suggest it's to set a .sim file every 50 frames for example, and when you see that the ram it's satured you can restart the pc and restart the simulation with the .sim checkpoint. Otherwise you can try with linux since has the ability to flush the ram, something that windows seems incapable to do.

[merlino]

Hi Atom,

The distibuted sim works only if you have 2+ pc and halp exactly in this situation. So you have a rough total of ram that is the sum of your pcs ram. In this case you can't use distributed sim as you have just one pc.

I think that depending on the flip sim it can be possible to split it. If the fluid is fast moving it will be very hard to set up, but for slow moving FLIPs I think it will be possible. Anyway I should try to be sure.

Can you share the HIP to see if is there any optimization that can help? I'm thinking for example that the convert vdb node has and automatic partition in it, maybe you can set the partitions to higer values. Also the "VDB from particle fluid" has an option "Tile size" you can try setting the size to smaller values as that way should use less ram.

I hope this can help.

Cheers!

Marco

[Atom]

Thanks for the tips, I may look into the .sim checkpoint option. I may have to look into building a Centos7 machine to see if Linux is really any better than windows at this VFX stuff. Trouble with Linux is you always need to know secret commands to get things finalized. Time being a hardware tech is time away from working on my Houdini stuff.

I am attaching my current scene. I have locked the terrain geometry and a single frame of the animated character. Thanks for taking a look at my file.

ap_cmf_storm_drain_fluid_sim_share.hipnc

Edited January 25, 2017 by Atom

[dimovfx]

Are you creating cache of the flip points before trying to surface them or you go directly with caching the surface as simulation goes in background?

also I think that the memory modules you are looking for are just a standart DDR3 modules.

 

[merlino]

Can you upload the frame 137 cache? maybe the compressed_cache one 7zipped... so I can play with that directly

[lukeiamyourfather]

1 hour ago, Federico said:

Otherwise you can try with linux since has the ability to flush the ram, something that windows seems incapable to do.

If it takes more memory than the system has it doesn't matter what operating system is used. They're all going to crap out and use swap or virtual memory.

57 minutes ago, Atom said:

I may have to look into building a Centos7 machine to see if Linux is really any better than windows at this VFX stuff.

Linux is better for VFX production for sure but if you're using a single machine and have no render farm most of the benefits will be a moot point. If you want to work in a medium to large VFX facility then it wouldn't hurt to get familiar with Linux.

The memory is out there. For example this from Newegg but it's so expensive you could buy 64GB of DDR4, a new motherboard, and a processor for the same price. Or at the least put the difference towards newer hardware that's better. Putting $800 worth of memory in an outdated machine is not worthwhile.

https://www.newegg.com/Product/Product.aspx?Item=9SIA7S65619005

Edited January 25, 2017 by lukeiamyourfather

[Atom]

Thanks for your advice, Luke, I was hoping to find a single 16gb stick I had not searched for a kit. You're right about the price, however.

@Merlino & @Sasho: I have not kicked out any cache, I was hoping to skip that step and just surface from the sim directly.

 

[merlino]

It's really better if you cache out the sim using the "compressed_cache" node. It will use only the ram needed for the sim, and then only the ram needed for the surfacing. Maybe just with that it will works.

1 - "Save to disk in background" on "compressed_cache" it will start hbatch so you can close your current Houdini session

2 - Save to disk in background on "particles fluid surface" ... same as point 1

this way is more efficient.

[Atom]

Thanks, i'll give that a try.

[Yon]

You can use a Wedge ROP and the seed parameter. Though I don't think it is a work around for meshing, you can cache particles efficiently with this technique

 

[Atom]

Currently I have 48 frames exported from the CompressedCache node. This has taken 12 hours with a particle separation of 0.025. I need 1,518 frames to achieve the coverage for my piece. If I stay the course, I will have complete coverage in 15.5 days assuming my computer can run that long and there are no electricity outages during that time.

I have never used a wedge, will it reduce my time? How?

[Federico]

27 minutes ago, Atom said:

Currently I have 48 frames exported from the CompressedCache node. This has taken 12 hours with a particle separation of 0.025. I need 1,518 frames to achieve the coverage for my piece. If I stay the course, I will have complete coverage in 15.5 days assuming my computer can run that long and there are no electricity outages during that time.

I have never used a wedge, will it reduce my time? How?

It seems that waiting it's part of the FX artist, i saw that gridmarkets now allows also the cloud simulation along with the rendering. Maybe you can try to look into it?

[Atom]

Perhaps, but if I am spending money, it might as well go to a new motherboard/ram combination as Luke mentioned above.

In your previous post you mention .sim files. I tried kicking out a few .sim files but their file size was very large. The compressed cache does make smaller files, but they are .bgeo.sc.

What is the difference between .sim and .bgeo.sc?

I exported my .sim files by adding a File node in write mode to the output of the AutoDopNetwork. Is this the correct place to export .sim files?

Edited January 26, 2017 by Atom

[Federico]

1 hour ago, Atom said:

Perhaps, but if I am spending money, it might as well go to a new motherboard/ram combination as Luke mentioned above.

In your previous post you mention .sim files. I tried kicking out a few .sim files but their file size was very large. The compressed cache does make smaller files, but they are .bgeo.sc.

What is the difference between .sim and .bgeo.sc?

I exported my .sim files by adding a File node in write mode to the output of the AutoDopNetwork. Is this the correct place to export .sim files?

If you wish to improve your hardware a couple of xeon will surely improve your sim times (i know they are expensive for this reason you can look into the ES processor, they shouldn't be on the market but well they are super cheap).

For the .sim part: A file with the .sim extension include every data from the dops to restart the simulation (since they are quite heavy, and so they may be increase your sim times just for writing it, i suggest to write it one only 50 or 100 frames) while the bgeo.sc just include what you want (for example just the density and the temperature if we are talking about the pyro).

To write the sim files i don't believe that you have to add a file node. You can just check the "explicit cache" option on the dopnet and after you have to check the "checkpoints option" .

[Atom]

Thanks for the info.

 

Do you happen to know what attributes are required to keep for surfacing?

What should I delete prior to CompresseCache node to insure I have enough data for surfacing to work yet present the smallest file size on the disk?

[Federico]

41 minutes ago, Atom said:

Thanks for the info.

 

Do you happen to know what attributes are required to keep for surfacing?

What should I delete prior to CompresseCache node to insure I have enough data for surfacing to work yet present the smallest file size on the disk?

Well, i'm not an expert so take my suggestions with caution. To generate the whitewater and render the water once meshed, you should need only the velocity and the vorticity of the flip fluid simulation i believe.

[Atom]

I just conducted a low-res test and it looks like you are right. You only need to keep @P and @v on the points. I also kept @name on the primitives so the volumes would retain their original naming as well. This does produce some file size savings, but the compressed files are about 5X bigger than the final surface files.

I'm still thinking that in my case it may be best to skip writing compressed cache file and just go for final surfacing.

[Federico]

9 minutes ago, Atom said:

I just conducted a low-res test and it looks like you are right. You only need to keep @P and @v on the points. I also kept @name on the primitives so the volumes would retain their original naming as well. This does produce some file size savings, but the compressed files are about 5X bigger than the final surface files.

I'm still thinking that in my case it may be best to skip writing compressed cache file and just go for final surfacing.

 

The problem is that if go immediately to cache out the meshed water, and something happens in the simulation/meshing phase (like a power shortage) you will have to restart from the beginning. Also if later on you wish to modify something of the meshed water you will have to resimulate everything. The compressed cache isn't really heavy in terms of disk space, i saw simulations done with the old tools going from 400 gb to 20 gb with the compressed cache.

Edited January 26, 2017 by Federico

[rbowden]

For doing a simulation like this with a workstation like yours, you need to be a bit smarter when doing flip simulations. Just by looking at your file, I can see many areas that can be improved on to help with simulation times and your meshing process. Here are some of my findings so far:

- At a particle separation of  0.025, how many particles is that at frame 48? I am going to say that it is alot. You need to me more conscious of how many particles you are dealing with in your flip simulations. The particle separation is going to be different for every flip simulation you ever do because it depends on the scale of your simulation. I took your file and ran 10 frames of your simulation at a particle separation 0.025. At frame 1 you had 4 million particles and at frame 10 you have 22 million. You are roughly injecting a little over 2 million particles per frame at that particle separation and you keep doing so every frame until the end of the simulation. The only time particles are getting killed are when the waterfall hits the bottom of your container. If you are going to emit particles like this every frame, you are best off maybe having some sinks around your main body of water to help regulate the number of particles. For a body of water like this, you can easily get away with regulating the number of particles to 25-30 million and have that be enough resolution for you. If you don't want to emit particles like this every frame in the large body of water, don't make it an emitter and turn on reseed particles instead. Reseeding will help regulate the amount of particles that you start out with and try to maintain that if particles are getting killed off. Read up more about it in the help and do some testing with it.

- The tube emitters. I have no idea why you are even adding these into the same simulation as your body of water. Why not separate out the process and do each tube emitter separately? Sure, in the real world you would have that water fill your body of water but you have to stop thinking real world and start thinking about how to fake it. Do your emitters first, have them emit down and go a little bit into where your body of water is going to be. As soon as they are past that threshold, kill those particles. You aren't going to see them churning around inside the water in your shot. The most you are going to see is maybe some splash kicking up and some ripples (all of which you can go outside your flip simulation after it is done). If you do it like this, it gives you faster turnaround if you need to make changes to anything in your simulation. If you were to get notes on a change to the body of water and not the tube water coming out, you can just change the body of water because now you have it separated out and your tube water will look exactly the same. 

- As for meshing, you are trying to mesh many particles and of course you are running out of ram. Do you really need the particle fluid surface to iterate through all those particles that are under the water? Not really. You have some pretty deep areas of your flip simulation and unless something is going to be interacting with those deep areas, you can probably just delete those particles that under the surface. I believe you have some settings on the particle fluid surface to do this now but if you want more of a manual approach, take the SDF that gets written out when you simulate your flip simulation (surface), do a gradient on that sdf and use that to delete any particles based on the threshold of that gradient. That should speed up things exponentially with your meshing. If you want an alternative, give this thread a look and try out what bunker suggests: http://forums.odforce.net/topic/26814-meshing-huge-amount-of-flip-particle

And I agree with @Federico on caching out your simulation to a compressed simulation first and then meshing. I don't understand why you wouldn't want to do this. If you don't cache out and you surface your simulation live, you are denying yourself the ability to even tweak your mesh. You are at the mercy of whatever you have your particle fluid surface set to. It is only beneficial to you to cache out your simulation first.