FLIP fields accuracy questions

[mrWolf]

Hi all

I am assembling my own whitewater generator and in order to check if a particle is inside, outside or on top of the water surface I am using the "surface" field provided by the FLIP simulation.

Everything works well.

 

Later I played a bit with the various VDB tools to convert the FLIP particles into mesh.

I tried:

1 - flip particles -> old "particlefieldsurface"

2 - flip particles -> vdb from particles -> vdb resample sdf (dilate) -> vdb smooth sdf -> vdb resample sdf (erode) -> vdb convert (polygons)

3 - flip particles -> vdb from particle fluid ->  vdb convert (polygons)

 

I found (3) to be the best option cause it's faster than 2 even if the result is very similar.

 

Then I remembered about the "surface" field provided by the FLIP and I thought that should be the best option cause it's what flip uses internally.

I noticed that the resolution of this field is very low, plus I tried to convert it to VDB SDF and then apply a VDB reshape SDF and got a "... skipped because it's not a level set".

 

So my first naive question is :

- what is the difference between VDB SDF and Houdini Volume SDF and how do "level sets" come into play in this difference ?

 

Now, considered that the detail of the sdf generated by (3) can be set to whatever resolution , versus the FLIP "surface" field which is fixed, I've the following questions:

- once the FLIP data is cached to disk,  in general what is the FLIP "surface" field for ?

- does it make more sense to use the FLIP particles to generate a more accurate sdf for any possible calculation ? (meshing / whitewater boundary checks / generic distance from water surface calculations etc) ?

- same question goes for the other flip fields, like the velocity. Does it make more sense to rebuild a new, more detailed, velocity vector field from the FLIP particles, instead of relying on the one coming from FLIP ?

Edited August 2, 2014 by mrWolf

[Solitude]

VDB's have a limited bandwidth of where they keep data (spare grids). Regular houdini sdf's are more accurate through the volume, but also slower as a result. You can force a vdb from polygons to fill the interior, and it will also give you the same result as a regular volume. Use a volume slice to see the difference on a sphere put through vdb from polys and isooffset sdf.  Flip uses regular volumes for all of it's internal calculations. The reason you have a lower resolution grid is so the sim is relatively fast, so I wouldn't bother changing that too much. The default grid scale is 2, I've brought it down to 1.5 and 1.25 to help with some details, but it's not usually a huge necessity.  

 

As you mentioned, using the surface field typically won't match the mesh you end up generating on your own. So, depending on the needs (ie: accuracy), making a new surface field for your whitewater sim to match the meshed sim can be a good idea, other times it's not necessary because you won't see the difference if the foam particles are directly on the surface or not. 

[johner]

Ian had some good advice, so I'll only add a couple of things.

 

 

Then I remembered about the "surface" field provided by the FLIP and I thought that should be the best option cause it's what flip uses internally.

I noticed that the resolution of this field is very low, plus I tried to convert it to VDB SDF and then apply a VDB reshape SDF and got a "... skipped because it's not a level set".

 

 

Right, the surface field is used by the FLIP solver to track the liquid surface.  It won't match the particles in splashes due to lack of resolution, but should be pretty close for broader expanses of water (i.e. where foam would float). The default WhitewaterSolver uses it.

 

 

I noticed that the resolution of this field is very low, plus I tried to convert it to VDB SDF and then apply a VDB reshape SDF and got a "... skipped because it's not a level set".

 

 

You probably need to use a Primitive SOP to (re-)set the border of the surface volume to SDF first, as unfortunately that information gets lots on the way out of DOPs.  Then it should be converted as a level set into VDB.

 

Now, considered that the detail of the sdf generated by (3) can be set to whatever resolution , versus the FLIP "surface" field which is fixed, I've the following questions:

- once the FLIP data is cached to disk,  in general what is the FLIP "surface" field for ?

- does it make more sense to use the FLIP particles to generate a more accurate sdf for any possible calculation ? (meshing / whitewater boundary checks / generic distance from water surface calculations etc) ?

- same question goes for the other flip fields, like the velocity. Does it make more sense to rebuild a new, more detailed, velocity vector field from the FLIP particles, instead of relying on the one coming from FLIP ?

 

You can certainly try all of this, but since that velocity field is used internally by FLIP, you're likely not going to get more detail than what comes out of the sim, i.e.the FLIP particles themselves are advected through that velocity field. If you want higher resolution grids out of FLIP you can lower the Grid Scale to 1.5 or so as Ian suggests.  Also depending on your sim you might want to reduce the Smoothing parameter on the FLIP Solver.

 

One last thing to be aware of, VDBFromParticles and ParticleFluidSurface are designed for meshing the resulting SDF, so they only ensure that the zero isosurface is at the right point.  However they don't build a true Signed Distance field with accurate depths below the surface.  

 

For example, imagine a 5 meter deep tank, with a  grid point 1 meter below the surface and a particle at that grid point with a radius of 0.1.  At that grid point the depth value in a true Signed Distance Function should be -1, but on output from these SOPs that value will actually be -0.1, which is the distance the grid point is within the particle radius, *not* its distance from the overall surface.

 

To get accurate depths you need to "rebuild" the SDF using either VolumeSDF for a Houdini volume or VDB Resample for a VDB.  The surface field from FLIP had already been rebuilt as a true SDF.  

 

Also note, this may or may not matter for your secondary effects.  If you only want to know if a particle is "underwater", checking for negative depths might be enough.  But if you need to know how deep underwater, you'll likely need to rebuild the SDF.

Edited August 3, 2014 by johner

[edward]

I think the VDB From Particle Fluid SOP has a "Rebuild SDF" parameter these days.

[johner]

I think the VDB From Particle Fluid SOP has a "Rebuild SDF" parameter these days.

 

Ah, I totally forgot, and it's fast too!  Thanks Edward.

[mrWolf]

VDB's have a limited bandwidth of where they keep data (spare grids). Regular houdini sdf's are more accurate through the volume, but also slower as a result. You can force a vdb from polygons to fill the interior, and it will also give you the same result as a regular volume. Use a volume slice to see the difference on a sphere put through vdb from polys and isooffset sdf.  Flip uses regular volumes for all of it's internal calculations. The reason you have a lower resolution grid is so the sim is relatively fast, so I wouldn't bother changing that too much. The default grid scale is 2, I've brought it down to 1.5 and 1.25 to help with some details, but it's not usually a huge necessity.  

 

As you mentioned, using the surface field typically won't match the mesh you end up generating on your own. So, depending on the needs (ie: accuracy), making a new surface field for your whitewater sim to match the meshed sim can be a good idea, other times it's not necessary because you won't see the difference if the foam particles are directly on the surface or not. 

 

 

Thank you for the explanation Ian !

In general I've used the FLIP surface field for pretty much every setup post sim.

I was aware that VDB SDF are hollow (if not otherwise specified in vdb from poly node). It was only when I converted a Volume SDF to VDB SDF and then tried to reshape it , that I realized some piece of data was missing. Hence my curiosity about the whole SDF thing.

 

I am probably using a FLIP sim a bit too coarse, so there is a big difference between the FLIP surface field (meshed) and the mesh generated from the FLIP particles. This difference introduces a huge discrepancy between the data I use to generate my white water and the location of the water surface mesh. I'll probably generate a higher res FLIP sim and this should reduce this discrepancy.

[mrWolf]

John thank you a ton for taking the time to clarify all this. Finally I have a clear understanding about the differences between the various kind of SDFs.

 

 

You probably need to use a Primitive SOP to (re-)set the border of the surface volume to SDF first, as unfortunately that information gets lots on the way out of DOPs.  Then it should be converted as a level set into VDB.

 

 

That's an immensely handy piece of information.

I added a Primitive SOP with Volumes-->VDB--->class set to "Level Set".

This fixed the "level set" issue but introduced a "VDB has a narrow band width less than 3 voxel size".

If I drop a VDB Reshape SDF immediately after, it won't work, but if i add a VDB Resaple in between, it finally works, even if the warning about the narrow band still remains on the Reshape node.

Is there a more proper way to increase a VDB SDF narrow band width ? 

 

 

You can certainly try all of this, but since that velocity field is used internally by FLIP, you're likely not going to get more detail than what comes out of the sim, i.e.the FLIP particles themselves are advected through that velocity field. If you want higher resolution grids out of FLIP you can lower the Grid Scale to 1.5 or so as Ian suggests.  Also depending on your sim you might want to reduce the Smoothing parameter on the FLIP Solver.

 

 

I thought about this.

Please correct me if I am wrong here:

I remember from the amazing "FLIP solver from scratch" masterclass, that at each time step the particles velocity is updated with the difference between the vel in the previous time step and the one after the non divergent pressure solution, which still allows a certain individuality in the particle's velocity after the advection (as far as I remember, this is the advantage of FLIP over PIC).

I am definitely not an expert in this so I might have misunderstood

But, if this is the case, it means that for each voxel (one velocity) there might be more than one particle, each one with it's own velocity (in general slightly different from the one present in the voxel they occupy) ?

 

 

One last thing to be aware of, VDBFromParticles and ParticleFluidSurface are designed for meshing the resulting SDF, so they only ensure that the zero isosurface is at the right point.  However they don't build a true Signed Distance field with accurate depths below the surface.  

 

For example, imagine a 5 meter deep tank, with a  grid point 1 meter below the surface and a particle at that grid point with a radius of 0.1.  At that grid point the depth value in a true Signed Distance Function should be -1, but on output from these SOPs that value will actually be -0.1, which is the distance the grid point is within the particle radius, *not* its distance from the overall surface.

 

To get accurate depths you need to "rebuild" the SDF using either VolumeSDF for a Houdini volume or VDB Resample for a VDB.  The surface field from FLIP had already been rebuilt as a true SDF.  

 

Also note, this may or may not matter for your secondary effects.  If you only want to know if a particle is "underwater", checking for negative depths might be enough.  But if you need to know how deep underwater, you'll likely need to rebuild the SDF.

 

 

This info is massively useful, thank you John !
I had no idea the SDF generated by VDB From Particles and VDB from Particle Fluid would return a non Signed Distance Field. I thought an volume is either an SDF (meaning, that each voxel contains the distance from the surface, positive outside, negative inside) or it's not an SDF.

 

In my case I definitely need the depth info cause I like to smooth the bubbles speed when they approach the surface.

[mrWolf]

UPDATE :

 

After generating my White water source I realized there are strong peaks of vorticity deep inside the tank.

This didn't make any sense to me, cause before processing the FLIP particles I made sure to work only on the particles very close to the surface (I deleted the particles deeper than a certain value).

After analizing the content of the "surface" field, I figured out the problem:

 

 

Apparently there are "holes" in the surface field. These holes appear suddenly and disappear in a matter of 1 frame, which messes up the vorticity in the surrounding area.

 

In order to eliminate the issue I had to convert to VDB sdf ... dilate and erode ... and convert back to volume sdf.

 

 

Which fixes the issue (kinda):

 

 

The problem is that this process is quite long , plus it softens the details on the surface of the water.

 

Is there any way to avoid those bubbles in the surface field during the FLIP simulation ?

Edited August 4, 2014 by mrWolf

[Skybar]

Do you have reseeding turned on? Looks like it could be related to that. If not I guess you could try to set the "Update Surface" (under Volume Motion tab on the flipsolver) to Rebuild rather than Advect, which is more accurate (but slower).

Edited August 5, 2014 by Skybar

[mrWolf]

Do you have reseeding turned on? Looks like it could be related to that. If not I guess you could try to set the "Update Surface" (under Volume Motion tab on the flipsolver) to Rebuild rather than Advect, which is more accurate (but slower).

 

Hey David,thank you for your reply,

no the reseeding is off (I always keep it off).

I'll definitely try the "rebuild" option and see if things change.

[danw]

I'm not certain it's applicable here, but when I've worked with the surface field for some reseed hacking, I found those holes tend to appear when the particle radius multiplier isn't quite high enough, and when it recreates the surface at the start of the substep, it can end up with small cavities.  The rebuild possibly won't do much different, as from what I can tell, it gets rebuilt at the beginning of the next timestep regardless of what you pick, rebuild will just do it at the end of the previous frame rather than waiting, so the field data you cache out to disk is synchronised with the particle data at each frame, rather than being offset.  (I could be very wrong with that deduction though)

[mrWolf]

I'm not certain it's applicable here, but when I've worked with the surface field for some reseed hacking, I found those holes tend to appear when the particle radius multiplier isn't quite high enough, and when it recreates the surface at the start of the substep, it can end up with small cavities.  The rebuild possibly won't do much different, as from what I can tell, it gets rebuilt at the beginning of the next timestep regardless of what you pick, rebuild will just do it at the end of the previous frame rather than waiting, so the field data you cache out to disk is synchronised with the particle data at each frame, rather than being offset.  (I could be very wrong with that deduction though)

 

Hi Daniel,

You're right about the rebuild option. I re-simmed with that option on , and as far as I can tell, the cavities are slightly smaller, but they are still there.

You're saying that it's caused by the fact that the particle radius multiplier is not high enough.

The problem is that if I increase the particle radius I'll have less details on the surface water.

So is it fair to say that the only way to fix the surface cavities is to post-process the surface field ?

Is anyone aware of a better / quicker method than the one I posted in the initial message ?

[danw]

Hi Daniel,

You're right about the rebuild option. I re-simmed with that option on , and as far as I can tell, the cavities are slightly smaller, but they are still there.

You're saying that it's caused by the fact that the particle radius multiplier is not high enough.

The problem is that if I increase the particle radius I'll have less details on the surface water.

So is it fair to say that the only way to fix the surface cavities is to post-process the surface field ?

Is anyone aware of a better / quicker method than the one I posted in the initial message ?

 

You wouldn't lose much detail from the sim particles themselves by increasing the radius... you don't have to increase the particle separation, just have as many particles, on the same resolution grid, but they splat a fatter surface for surface tracking.  That's the problem though, the one thing it will reduce detail on is the surface field.  That's not usually too much of a bother, because you end up VDB-meshing the particles only, and you can always just reduce the pscale a bit to sharpen the detail up.

 

If you're looking to pass a more detailed and truer surface field to a whitewater sim, I've actually found it worked better for me to throw away the sim's surface field entirely, and pass in the VDB surface I generate for the mesh instead... It seemed to work just as well for whitewater when I tried it, as Houdini's default setup was only concerned with flagging particles as above/below or near the surface, not exact distances.  The sim velocity field can still be used which will give all the motion-detail you need.

 

I'm not sure how any of that might fit with your custom solver plans, but it ended up the optimal approach for me... it means foam rides precisely on top of the surface mesh you end up rendering, rather than along a slightly low-detailed, blurry approximation of it.

 

Edit: Just realised that's precisely what Ian recommended doing in the first reply of the thread :-P  So, err, yeah... I second that idea :-)

Edited August 6, 2014 by danw

[mrWolf]

You wouldn't lose much detail from the sim particles themselves by increasing the radius... you don't have to increase the particle separation, just have as many particles, on the same resolution grid, but they splat a fatter surface for surface tracking.  That's the problem though, the one thing it will reduce detail on is the surface field.  That's not usually too much of a bother, because you end up VDB-meshing the particles only, and you can always just reduce the pscale a bit to sharpen the detail up.

 

If you're looking to pass a more detailed and truer surface field to a whitewater sim, I've actually found it worked better for me to throw away the sim's surface field entirely, and pass in the VDB surface I generate for the mesh instead... It seemed to work just as well for whitewater when I tried it, as Houdini's default setup was only concerned with flagging particles as above/below or near the surface, not exact distances.  The sim velocity field can still be used which will give all the motion-detail you need.

 

I'm not sure how any of that might fit with your custom solver plans, but it ended up the optimal approach for me... it means foam rides precisely on top of the surface mesh you end up rendering, rather than along a slightly low-detailed, blurry approximation of it.

 

Edit: Just realised that's precisely what Ian recommended doing in the first reply of the thread :-P  So, err, yeah... I second that idea :-)

 

Ok you guys convinced me.

I was hoping there was a way to reuse the FLIP surface field cause it's for free ...

I guess it doesn't deserve to be healed, no mercy for broken surface fields !

I'll rebuild it from scratch starting from the particles as Ian and Daniel suggests.

 

Thank you all for the huge help