Fluid sim - how to avoid that...

[hall]

Quick question. Flat tank ocean, one boat using RBD with object transform.

 

The simulation looks quite nice as expected for calm ocean, but after a long time it starts to accumulate water and create a big wave :/

 

My idea is to keep the typical wake effect like a V style as long as possible, you can notice that the V starts to open over time until extrapolation and accumulation occurs, that's the problem here and I don't know how to fix that.

 

Check out the link // Private video - password: odforce

 

https://vimeo.com/101677591

 

 

 

 

 

[hall]

Serious!? Nobody has experienced that? C'mon guys, this should be a simple question. Am I missing anything here? 

 

I don't think post my HIP will make difference, but anyway I can post by request. thx!

[portdusk]

You could try masking off the velocity field towards the sides of the sim.

 

From the video, it looks like you could also trim off a lot of area to the sides  and probably get away with 1/2 of the height of the fliptank.

 

Don't you want to have the water breaking on the bow of the ship though? What scale of boat are you simulating? The way the wave breaks looks pretty good for a large ship, if maybe a bit wide. Fairly comperable to this footage:

 

http://www.shutterstock.com/video/clip-667075-stock-footage-the-bow-of-a-boat-cutting-through-a-calm-sea-surface-red-sea-egypt.html

 

You should also probably add some intial velocity noise into the ocean surface to shape your waves and break that uniformity.

[danw]

I wonder if that's the thing... perhaps the shape and speed would naturally tend to find an equilibrium that had the breaking bow-wave... and it just takes a while for the sim to move towards that state - water being water, it can take a long run-up to hit a stable state on something of this size.

 

On the other hand, the effect might be being exagerrated not by the depth, but by the shallowness.  Deep water means there's a lot of volume below to absorb compression waves... a shallower pool means all the energy can only dissipate sideways... stronger compression waves near the surface, bigger surface waves, and bigger surface waves are more prone to break.

Of course that brings up the other big problem with fluid sims - deep water simulation is almost impossible if you want any kind of usable resolution!

 

As ever, I guess you'll just have to keep testing and refining until you find the right balance for your purposes.

Edited July 29, 2014 by danw

[hall]

Hey guys, thx for notes, I guess the solution was simply turning off reseed, I think its clear why its extrapolated, overloaded after a long time simulating, because theres always more particles on this tank, so reseed doesn't make sense for my scenario I can live without new seeds, moving the grid is always bringing "new seeds", right? 

 

Check out the results:

Pass: odforce for both

 

Reseed on:

http://vimeo.com/102049631

 

Reseed off:

http://vimeo.com/102049586

[ryew]

I agree with Dan, the shallowness of the tank in comparison to your collision wedge might be contributing quite a lot to the mis-behavior you're seeing.  You might want to try a deeper tank or allowing a gap between your collision wedge and the bottom of the tank in order to allow the simulation to travel underneath instead of re-cycling and building up velocities at your collision surface.  Giving the sim a bit of wiggle room to let those velocities escape and even out might settle the sim down

[hall]

Indeed Ryew!  but as Dan mentioned: deep water simulation is almost impossible if you want any kind of usable resolution! 

And he is right, the second problem will be solve all this data. Thats the reason I cutout the tank size a lot. This must be a scene using a big ship 80meters length. So if I can get rid of this issues disabling reseed, its a cheap solution, right?

[danw]

Ah, yes, it didn't occur to me that it might be down to that, but yep, *always* turn off reseed!  It will basically turn all sims into a gradually-growing-blob-monster... the reseed microsolver will constantly re-populate surface cells that it considers under-resolved, which is basically any where the fluid doesn't occupy an entire cell (ie, almost any smoothly sloping surface cell).  Once it repopulates a cell, the radius of those new particles extend the surface on the next substep, at which point it considers new cells underresolved, and the whole thing continues.

 

About the only thing it can be useful for is culling excess particles when too many get bunched together, in which case leave it on, but turn the Birth Threshold to 0, so it never meets the conditions to create new ones.

 

I really don't understand why reseed is enabled by default in the FLIP solver... it's a great idea in theory, but the current implementation is essentially broken.

 

 

Excellent comparison videos by the way!  This is pretty much the perfect demonstration of exactly what's wrong with the current reseed implementation.  Because of the nature of the problem, the more you churn up the fluid, the more surface-area it has to birth new particles in, so you end up with that effect of a rapidly ballooning bow-wave that completely overtakes the boat, and the busier it gets, the busier it gets.

Compare that to the one with it turned off, and everything hits a lovely stable equilibrium.

Edited July 31, 2014 by danw

[johner]

In this case I agree there's probably no need for Reseeding, and in fact it should maybe be off for default for moving tanks.  Indeed there are plenty of new particles being introduced on each timestep.

 

Ah, yes, it didn't occur to me that it might be down to that, but yep, *always* turn off reseed!  It will basically turn all sims into a gradually-growing-blob-monster... the reseed microsolver will constantly re-populate surface cells that it considers under-resolved, which is basically any where the fluid doesn't occupy an entire cell (ie, almost any smoothly sloping surface cell).  Once it repopulates a cell, the radius of those new particles extend the surface on the next substep, at which point it considers new cells underresolved, and the whole thing continues.

 

 

There's been several changes to the Reseeding algorithm in the last year or so.  Now Reseeding does try to properly place particles in partially filled voxels via rejection sampling.  So when placing new particles if the proposed location doesn't fit within the surface at a depth of the particle radius, it will never create a particle there.  For the most part this avoids perturbing the surface and the resulting volume gain, but indeed it can happen for splashy simulations like this.

 

About the only thing it can be useful for is culling excess particles when too many get bunched together, in which case leave it on, but turn the Birth Threshold to 0, so it never meets the conditions to create new ones.

 

I generally recommend people turn Birth / Death thresholds down to 0.25 / 1.25 if they're seeing too much volume change from Reseeding, but you can definitely just turn off birthing altogether.  I also always recommend turning off Reseeding for highly viscous sims where the individual particle additions are really obvious in the slow moving fluid.

 

 

I really don't understand why reseed is enabled by default in the FLIP solver... it's a great idea in theory, but the current implementation is essentially broken.

 

I'd encourage / beg you to submit an easily reproducible example of blob-monsteriness to support so we can try to kill it (or I guess send it back to the watery depths, more appropriately).  I consider the Flat Tank a bit of a special case, so even more useful if it's just your garden variety static tank / river / etc.

Edited July 31, 2014 by johner

[danw]

I did some fairly extensive tests a few months back, and found it was a major issue.  I suppose I was trying to push it to its limits (having 32-64 surface layer particles, to 2-3 interior particles per voxel) in order to squeeze the most detail I could out of deep-water sims for a give memory footprint.

 

The solution I eventually came up with was to switch off the standard reseed, unlock the solver, rebuild a copy of the surface field so that the 0-level was a voxel or two inside the fluid, and then plug a new reseed microsolver in that used that field instead of surface.  The new particles wouldn't contribute straight away, but wherever the fluid churned around, high-densities of particles would naturally get pushed to the surface over time, so after a couple of seconds preroll, the top few voxels would be consisitently detailed, even though all the reseeding was happening beneath the surface rather than at the surface.  It also completely avoids the problem of new particles causing popping in the meshing that way.

I presume due to the fact that the surface volume is always going to be a slightly simplified, inaccurate representation of the volume represented by the particles themselves, it needed to be pushed at least that full voxel distance inside before the growth stopped happening entirely.

 

I'm away for a month or so working on a project, but I do mean to get my test scenes packaged up and submitted when I get back.

They were fairly basic dump-tank-inside-a-closed-domain sims.  The 32x surface oversampling was a pretty reliable way to lure the blob-monster out of his cave if I recall... the slightest tendency to grow the surface with those kind of numbers would often cause the entire domain to fill up to the brim in a couple of seconds.

 

It's definitely something that would be great to help solve properly, as the hack-solution I put together gave me some amazing results.  Having massive surface oversampling counts ultimately allowed me to push the voxel resolution of the sims up into crazy territory (which houdini's pressure solve seems to eat for breakfast), while still keeping particle counts managable.  Concentrating that many particles at the surface, combined with grid resolution not being a bottleneck, meant I didn't have to worry so much about avoiding those wasteful large bodies of fluid.

Edited August 1, 2014 by danw

[wateryfield]

I did some fairly extensive tests a few months back, and found it was a major issue.  I suppose I was trying to push it to its limits (having 32-64 surface layer particles, to 2-3 interior particles per voxel) in order to squeeze the most detail I could out of deep-water sims for a give memory footprint.

 

The solution I eventually came up with was to switch off the standard reseed, unlock the solver, rebuild a copy of the surface field so that the 0-level was a voxel or two inside the fluid, and then plug a new reseed microsolver in that used that field instead of surface.  The new particles wouldn't contribute straight away, but wherever the fluid churned around, high-densities of particles would naturally get pushed to the surface over time, so after a couple of seconds preroll, the top few voxels would be consisitently detailed, even though all the reseeding was happening beneath the surface rather than at the surface.  It also completely avoids the problem of new particles causing popping in the meshing that way.

I presume due to the fact that the surface volume is always going to be a slightly simplified, inaccurate representation of the volume represented by the particles themselves, it needed to be pushed at least that full voxel distance inside before the growth stopped happening entirely.

 

I'm away for a month or so working on a project, but I do mean to get my test scenes packaged up and submitted when I get back.

They were fairly basic dump-tank-inside-a-closed-domain sims.  The 32x surface oversampling was a pretty reliable way to lure the blob-monster out of his cave if I recall... the slightest tendency to grow the surface with those kind of numbers would often cause the entire domain to fill up to the brim in a couple of seconds.

 

It's definitely something that would be great to help solve properly, as the hack-solution I put together gave me some amazing results.  Having massive surface oversampling counts ultimately allowed me to push the voxel resolution of the sims up into crazy territory (which houdini's pressure solve seems to eat for breakfast), while still keeping particle counts managable.  Concentrating that many particles at the surface, combined with grid resolution not being a bottleneck, meant I didn't have to worry so much about avoiding those wasteful large bodies of fluid.

It looks interesting.

The standard reseed system not working well at all sometimes.

 

The houdini help  

Setting the various Reseeding values too high can lead to fluid gaining volume over time in very splashy fluid simulations. 

 

Daniel.

Is it possible to have a look with your  reseed system.

Thank you.

[danw]

Daniel.

Is it possible to have a look with your  reseed system.

Thank you.

 

As I mentioned, I'm away contracting for the next month, so I don't have access to my old scene files just at the moment.

 

It involved a little bit too much black-magic hackery for me to remember exactly what I did and recreate it either.  That mostly involved very particular tricks to shrink the simulation surface SDF while still maintaining its integrity, so that the new field wouldn't cause the reseed microsolver to start birthing particles all around the domain boundary, or in random glitch cells that should have had background values.

 

It also required a fair bit of trial-and-error tweaking to find particle separation values that would maintain the internal volume of the fluid even when being so sparse on the interior.  Unfortunately, but understandably the FLIP solver currently assumes all particles to have the same pscale throughout the sim, which means you need to define them as uniformly very fat particles otherwise a sparse interior will end up behaving like spray rather than a solid, and proceed to simply collapse in on itself.

 

Play around with it - the basic principle is just to shrink the fluid surface before reseeding.  I'll dig up my scenes once I finish this job.

Edited August 1, 2014 by danw

[johner]

The 32x surface oversampling was a pretty reliable way to lure the blob-monster out of his cave if I recall... the slightest tendency to grow the surface with those kind of numbers would often cause the entire domain to fill up to the brim in a couple of seconds.

 

Hi Dan,

 

Thank you for the detailed description, very interesting.  I look forward to seeing your tests whenever you get the chance to share them.

 

Just to be clear, when you say 32x oversampling, you mean around 32 particles per voxel right?  So in the default setup you'd set Surface Oversampling to 4 (i.e. not 32, right?)  It's possible we just haven't done enough testing with really high surface oversampling values.  I tend to set Oversampling in the 2-3 range myself, so we might just need to do more testing in the 4-8 range.  I generally felt we got diminishing returns from oversampling that high, but obviously you found different, yes?

 

Thanks again

[danw]

It's a little difficult to remember exactly, as it ended up a fairly complicated setup with some fairly particular values from trial-and-error to get it working right.

 

I think I was trying to maintain around 3 particles per cell for the interior, culling it if it got too high, but not birthing any.  I think when I attempted to use 1-per-cell, it just tended to collapse the fluid, it needed at least 2 to maintain volume.

 

Now I think about it, I was actually using two Gas Seed Markers nodes instead of one... one handles the birthing, and the other the culling.  Not sure exactly what settings I was using, but I was aiming to keep the surface cells at 32+ particles per cell.  I was avoiding birthing any in the interior, so I think for the birthing node, I may have been using 1-per-cell with 32 oversampling, and different multiples on the culling node.

 

The ultimate aim was to make it stable with as few interior and as many surface particles as possible... which I did manage to get it to do eventually.

 

 

I didn't find it gave diminishing returns as such... more that there was plenty of room to go before hitting them. Seeing as the VDB meshing is purely dependant on the particle density, and has no notion of grid resolution, having a massively oversampled surface layer meant that the meshing could be bumped up to a massive resolution, and give a very crisp, very smooth result, with a relatively managable sim.  Sims using identical grid resolution and geared to have an identical surface particle density, but using the usual ~8-per-cell, with 2x oversampling approach might reach 200-300 million particles, while this technique would give an almost identical sim behaviour, with smoother meshing, for nearer 50-million.

Past about 32 particles per cell, and it did give diminishing returns - at that point it really needed more grid resolution to gain any more useful detail.

 

Ultimately, the most important aspect was knocking the interior down to 2-3 rather than the default 8, while managing to bump the surface count at the same time.  It was the low interior count that would allow managable deep water while keeping a usable level of detail at the surface.

 

 

Hope that gives some useful insight for the moment... I feel like I'm mostly just confusing myself trying to think back to it, and I really need to root through my scene files and kick out a couple of demonstration playblasts to actually make some decent sense of it :-)

Edited August 2, 2014 by danw

[johner]

Thanks Dan, very interesting stuff.

 

I think I was trying to maintain around 3 particles per cell for the interior, culling it if it got too high, but not birthing any.  I think when I attempted to use 1-per-cell, it just tended to collapse the fluid, it needed at least 2 to maintain volume.

 

 

This is with the default Grid Scale of 2, and a Particle Radius Scale of 1.2 or somewhere near there?  It seems some tests are in order with a Particles Per Voxel setting of 3 or 4, and a Surface Oversampling of 10 or 8.  That could be an interesting deep water optimization.

 

By the way, I don't know when you did this test, but there have been several changes to Reseeding over the past year.  Just to summarize in one place:

 

Quote:
Thursday, September 5, 2013 
Houdini 13.0.166: Improved the uniformity of the particles generated by Reseeding in the FLIP Solver, as well as removing grid artifacts that could be caused when Reseeding deleted particles from the jittered grid of points created by the PointsFromVolume or ParticleFluidTank SOPS. 

Here we were trying to be too clever and moving particles around inside the surface to fill the voxels, rather than just rejection sampling as we should have been. 
 

Quote:
Monday, September 16, 2013 
Houdini 13.0.177: The PointsFromVolume, ParticleFluidTank, and FluidSource SOPs now have Scattering Density and Oversampling parameters, which allow creating additional points at and below the surface of geometry or volumes. 
These additional points help create a flat, stable surface for the initial state or emitters in a FLIP simulation. The shelf tools that create these nodes for a FLIP simulation will now connect their Oversampling parameters to FLIP's reseeding Oversampling parameter. 

The problem here is we do ParticleToSDF, PressureSolve, then Reseed. Unfortunately that means on the first timestep, that initial solve is done on the particles as created by ParticleFluidTank, before any Reseeding. So that first timestep would not be oversampled and would introduce a little noise, which would then take many frame of pre-roll to subside. 

So we added Oversampling and Scattering on the surface into the sources that create the initial particles in the first place.
 

Quote:
Thursday, February 6, 2014 
Houdini 13.0.316: The FLIP Solver's Reseeding operation now uses a time-varying random function to generate new point positions, modulated by the new Random Seed parameter. The time-varying function fixes a bug that could cause streaking in the particle distribution with Reseeding enabled as slow moving liquid leaves a voxel. Varying the Random Seed can also be used to generate multiple simulations with similar bulk motion but different splash behavior from otherwise identical simulation parameters. Note this change will give different simulation results for any FLIP simulation that uses Reseeding. 

This fixed a nasty problem where we were using the voxel index as a seed for generating new particles, so if the domain was not resizing, the first Reseeded particle in a particular voxel would always be generated at the same position, causing these little streaks of particles as they slowly moved out of a voxel. 

 

So, with all these changes hopefully Reseeding and Surface Oversampling are in much better shape, even in H13.

 

 

[eetu]

Just random birthing ideas I got while reading this, wouldn't work here, or generally, but maybe in some situations.

 

2-pass ideas:

 

- Keep all birthed particles (and .sim cache) until end of sim, then advect backwards to start position. Now you know where to have oversampling at the start, and might not need to birth during sim (no popping). If the backwards advection is numerically unstable, kill particles outside initial surface. If it's great then you might just as well use those particles for your surfacing

 

- When particles are birthed, increase an "oversample" attribute on nearby original points. At end of sim take the original particle ids and accumulated oversample values, and create more points according to that info on the first frame and resim.

 

Generally, and more vaguely, how to "fade in" birthed particles to reduce popping? Scaling them up would look crappy, but could there be a "transparency" for them that would work with building an sdf? Need some think-time

[danw]

This is with the default Grid Scale of 2, and a Particle Radius Scale of 1.2 or somewhere near there?  It seems some tests are in order with a Particles Per Voxel setting of 3 or 4, and a Surface Oversampling of 10 or 8.  That could be an interesting deep water optimization.

 

By the way, I don't know when you did this test, but there have been several changes to Reseeding over the past year.  Just to summarize in one place:

 

I actually started this testing on the 13.0.316 build, so I guess everything I've come across has been since all of those changes.

 

For the sim to hold up, I found I needed to define the sim in terms of the interior - ie, the particle separation and resulting pscale needed to be set to the 3-particles-per-voxel scale... and then just fill the surface layers with more.  I would have assume doing so would have caused the 32+ surface particles to tend to just define a very fat blobby surface relative to their local separation, and get pulled around by each other in a very diffused way, but they actually gave a lot of unique motion detail regardless (It possibly helped that I tend to set my Volume Motion> Smoothing down to 0.05-0.1 instead of 0.3, so particles maintain a lot of individuality)

What would be really ideal is if the sim actually allowed surface particles to have an explicitly different pscale and separation to interior particles.  I assume that would involve a pretty horrendous re-write of the current solver though, as I know certain parts depend on a uniform particle separation/scale for speed.

 

 - incidentally - when it comes to meshing, I use the sim surface field to group off the surface layer particles, and apply a more sensible pscale for the denser particle separation before putting it all through the VDB mesher at that surface particle separation value, that's where all the crisp surface detail comes through.  The interior particle get left with their larger pscale.

 

I tend to set my scenes up with a few intermediate controls so I can specify my sim in voxel-size and particles per voxel, rather than using particle seperation and grid scale directly (I came from Naiad, so I still feel like it's more natural to define everything in voxel resolultion rather than particle spacing :-)

 

So, I tend to define a voxel size as my base measure of resolution, and calculate the rest from that - So a 0.1 voxel size with 3 particles per cell would give a particle separation of 0.1/∛3... and so the grid scale would work out at ∛3 (as in 0.1/(0.1/∛3))

The theory of all that being, that my initial emitter particle separation, and the general density I expected from a sim would match up with the reseeding, which is also specified in particles-per-voxel... so it just seemed a cleaner way to set it up.

 

...and then yep, oversampling at ~10, and default particle radius scale I think.

 

 

I think I'll try and get someone to send my files over to me if I get the chance... seems like it'd be worth trying to get it done sooner rather than wait until I get back, seeing as we're already having the discussion :-)  Might take me a couple of days to pick them apart though.

Edited August 5, 2014 by danw