Real Flow, Fluids, Dops, And Stuff

[Guest xionmark]

Greetings,

After having been working with Real Flow for some 3 years now, I do believe the time has come to let it go ... the damn thing (RF) is so unstable and severely limited for real production use it's just not worth the headaches (and grey hair) to fight with it any longer.

The other very real frustration is the company itself, Next Limit, which is horrible with support for studios, doesn't offer much training and well to be perfectly blunt, really screwed up my trip to Madrid for training (which was paid for by DNA Productions).

So while I'll continue to support the Houdini Real Flow plugins (yes, there's still a few more features to implement and subtle bugs to fix) for the near future, I'll be releasing the code eventually to the Houdini Sourceforge project so everyone can have access and maintane and extend it if desired.

As far as the Maxwell plugins go (for Next Limit's new renderer, Maxwell), I would rather bag the project now because of the reasons mentioned above as well as the fact that Next Limit is focusing so much of it's resources on it's development that I can't see any advances to Real Flow in the future much less fixing the outstanding bugs and required features to make Real Flow a viable fluid sim tool for production.

This pains me a bit because all in all the people at Next Limit are very nice but the bottom line is that they've dropped the ball continously with supporting studios (at least the studios that I've worked with who are using Houdini and Real Flow).

In regards to the Maxwell plugins, I know Erich Turner has been working on a solution and may be willing to release his work to the Houdini community but I'm not sure (Erich?).

Anyway ...

What I'm looking for now is an open source fluid sim project that we (I and any other masochistic programmers) can implement into DOPs like the ODE project. Of course SESI will soon (I hope) have their own fluid solvers in Houdini and I'm sure they will be incredible, and most important, production ready. So this idea is really to provide something to Houdini users in the interim and to help promote more HDK development in general.

Does any know of any "open source" fluid sim projects? Dr. Mark Carlson (at DNA) pointed me to the BFAST project (I think that's what it's called) but it's not ready for prime time yet.

Cheers!

--Mark

(Whew! OK, I got that all off my brain for once and for all!!!)

[sidandmj]

i understand just what you mean about the support side of NL

terrible at best . . .

Mark you are correct i do have a working, minus texutres version of a Maxwell Exporter not sure if its a bug on maxwell side or my side, things worked prior to beta 1.1.2a

there are some minor things that are unsupported, and need to doc it a little bit for use

further i need to implement apprentice code so as not to screw sideFx by releasing a plugin that allows apprentice to be used commercially for renders

will be talking with nl about getting them the plugin once i have the textures and apprentice code working

but i am beginning to wonder if i should proceed with nl or find another next gen type renderer like maxwell . . . . .but im not writing one!

[rjpieke]

Does any know of any "open source" fluid sim projects?  Dr. Mark Carlson (at DNA) pointed me to the BFAST project (I think that's what it's called) but it's not ready for prime time yet.

19761[/snapback]

Short answer: no.

But, if you want to write one from scratch (which actually isn't tremendously difficult), I'd highly suggest using the following three papers as your guides (these are the ones I found most useful):

* "Practical Animation of Liquids" (for most of the infrastructure)

* Carlson's PhD thesis (for the boundary cases and formation of the A matrix - I couldn't get Fedkiw's to work ... I was probably misreading something though)

* Magnus Wrenninge's thesis (http://www.ep.liu.se/exjobb/itn/2003/mt/018/exjobb.pdf)

Then, from there, you can look at Fedkiw's later papers for cool things like octree simulation.

Oh, and you might want to start with smoke before trying water. The infrastructure is pretty much the same, but you don't have to worry about the air-water interface (which I personally found to be, by far, the hardest challenge). In this case, "Visual Simulation of Smoke" is your friend.

Cheers!

[Jason]

further i need to implement apprentice code so as not to screw sideFx by releasing a plugin that allows apprentice to be used commercially for renders

Do you have to do anything on purpose for Apprentice-ifying? I was under the impression Apprentice *will not* load custom ROPs... but I'm probably out of date on that one since you can get Escape Plus (Escape + RendermanROP)- which may also allow any custom ROPs to be attached.

Good luck with the code:)

[Slapdaddy]

Hey Mark,

Adam B. who is head of the BFAST project said there should be fluid code up on the sourceforge site before siggraph... but as that is only a week away I still don't know for sure. But if you want to start with fluid I would suggest using Stam's smoke code. He actually has some source code that you could turn to 3d and he has instructions on how to do that. The best one with code and a layman's explanation is "Real-Time Fluid Dynamics for Games". at

http://www.dgp.toronto.edu/people/stam/rea...search/pub.html

If I ever get motivated I will get my fluid code published, and as you know it already works in houdini. But I would not be able to publish the houdini part because that is DNA IP

Anyway, It would be great to get this project rolling.

Mark Carlson

PS If anyone does get interested in writing their own fluid solver, and they actually read my thesis, then please feel free to ask me questions... if I am ever to improve it then I need all the feedback I can get

[Guest xionmark]

Hey Mark,

Adam B. who is head of the BFAST project said there should be fluid code up on the sourceforge site before siggraph... but as that is only a week away I still don't know for sure.  But if you want to start with fluid I would suggest using Stam's smoke code.  He actually has some source code that you could turn to 3d and he has instructions on how to do that.  The best one with code and a layman's explanation is "Real-Time Fluid Dynamics for Games". at

http://www.dgp.toronto.edu/people/stam/rea...search/pub.html

If I ever get motivated I will get my fluid code published, and as you know it already works in houdini.  But I would not be able to publish the houdini part because that is DNA IP

Anyway, It would be great to get this project rolling.

Mark Carlson

PS  If anyone does get interested in writing their own fluid solver, and they actually read my thesis, then please feel free to ask me questions... if I am ever to improve it then I need all the feedback I can get

19909[/snapback]

Mark!!!! So good to hear from you!!!! Can't wait to see you and the crazy DNA guys at SIGGIE!

I'll check the references mentioned above and it would be cool to implement your stuff into Houdini, after all, I was sittin' right next to you there at DNA and was constantly in awe of what you were writing. We can remedy the DNA IP issue by me doing the heavy lifting into DOPs, eh?

I'll be reading your thesis, though I doubt if I'll understand much of it, my math skills are *so* rusty ... but I know I could get a handle on things enough to deploy it in Houdini, just need a little more time getting up to speed on the DOPs API in the HDK.

Gotta run right now, things are cookin' at Charlex. Many beers to consume at the SESI party, eh?

Take care Mark!

--Mark

[rjpieke]

PS  If anyone does get interested in writing their own fluid solver, and they actually read my thesis, then please feel free to ask me questions

19909[/snapback]

Hey Mark ... I read your thesis after being unable to get a fluid solver (for water) working, reading Fedkiw & Foster's "Practical Animation of Liquids". I was wondering about the A matrix that goes into the CG solver and, in particular, about the surface cells.

NOTE: the following is all "as best as I understand it..."

* In Fedkiw's paper, he comments that surface cells should be set to atmospheric pressure, and have their air-facing velocities explicitly set to enforce incompressibilty. Then, since the surface cells have the same pressure as their neighbouing air cells, terms drop out of the A matrix.

* In your paper, you also enforce (mostly, at least) incompressibility in surface cells, but leave the pressure unknown and solve for it by keeping the A matrix terms.

Now, when I implemented Fedkiw's construction, the CG solver blew up. When I implemented your construction (literally a 3 line change in my code), everything worked near-perfectly.

I was wondering if you had any insights on these two different approaches, and why you chose to describe your system differently than Fedkiw did his.

Cheers!

-Rob

[Jason]

* Magnus Wrenninge's thesis (http://www.ep.liu.se/exjobb/itn/2003/mt/018/exjobb.pdf)

I just spent 10 days in a shoebox in the middle of the road in Tokyo with Magnus. I'll point him to this thread and if anyone has any questions for him, fire away.

[Guest xionmark]

i understand just what you mean about the support side of NL

terrible at best . . .

Sad isn't it .. so much potential in the company yet they squander away their relationships with the most important people ... oh well.

Mark you are correct i do have a working, minus texutres version of a Maxwell Exporter not sure if its a bug on maxwell side or my side, things worked prior to beta 1.1.2a

there are some minor things that are unsupported, and need to doc it a little bit for use

further i need to implement apprentice code so as not to screw sideFx by releasing a plugin that allows apprentice to be used commercially for renders

will be talking with nl about getting them the plugin once i have the textures  and apprentice code working

Cool, I've played with the pluginsa bit, it's fun but to be perfectly honest don't know if I'll be using Maxwell much at all for a while, I'm busy getting up to speed with Mental Ray.

but i am beginning to wonder if i should proceed with nl or find another next gen type renderer like maxwell . . . . .but im not writing one!

19863[/snapback]

Yea, I know what you mean ...

See you at SIGGRAPH!

--Mark

[Slapdaddy]

When I got the e-mail from rob about this I didn't realize it was an od[force] post... we exchanged a few e-mail about it. I will most my e-mails to him below for any trolls out there. It's up to rob to post his if he wants

Hey Mark ... I read your thesis after being unable to get a fluid solver (for water) working, reading Fedkiw & Foster's "Practical Animation of Liquids". I was wondering about the A matrix that goes into the CG solver and, in particular, about the surface cells.

NOTE: the following is all "as best as I understand it..."

* In Fedkiw's paper, he comments that surface cells should be set to atmospheric pressure, and have their air-facing velocities explicitly set to enforce incompressibilty. Then, since the surface cells have the same pressure as their neighbouing air cells, terms drop out of the A matrix.

* In your paper, you also enforce (mostly, at least) incompressibility in surface cells, but leave the pressure unknown and solve for it by keeping the A matrix terms.

Now, when I implemented Fedkiw's construction, the CG solver blew up. When I implemented your construction (literally a 3 line change in my code), everything worked near-perfectly.

I was wondering if you had any insights on these two different approaches, and why you chose to describe your system differently than Fedkiw did his.

Cheers!

-Rob

19936[/snapback]

------ e-mail 1 -----

Hello Rob,

The answer is that Fedkiw and I both do the same thing (as far as I understand it); it's just hard to tell because in a siggraph paper there is often too much to say in the page limit. Any cell that has fluid (level set <=0 if you are using level sets) has a place in the A matrix. The ones that drop out are the empty air cells near the surface. At least that is the way I understood the Foster Fedkiw, and the Enright et al siggraph papers.

You can actually drop out the surface cells with fluid in them if you want, but you will probably get a noisy surface... the key to solving the A matrix is that there must be at least one known (Dirichlet) pressure boundary condition (like the air pressure) or the matrix is singular and can't be inverted with normal CG. 0 is an easy air pressure to use because it means you don't have to move anything over to the B vector.

So, to sum up (or make my rambling explanation more muddy for you), Fedkiw said the terms drop out of the matrix... Not the whole cell drops out of the matrix--just the terms for the empty air faces. Think of a 1d cell with water on both sides. You would have the matrix row template -1 2 -1 for a "full" cell. If you had air on the left face then you would have 2 -1 because the term for the left face would drop out of the matrix.

The big difference between my thesis and the Enright et al. paper is that I explicitly set my cell face velocities by copying face velocities from nearby faces. Enright uses an extension velocity.

Hmm, I hope I answered your question... if I made no sense at all feel free to ask more questions any time.

Good luck!

Mark

---- e-mail 2 ----

Hey Rob,

1 -1 is for a cell that is near a solid boundary with a 0 Neumann boundary condition. 2 -1 would be a 0 Dirichlet boundary condition.

The only thing I can figure is that in Foster Fedkiw section 7.1.2 the "Liquid Surface" is the cells that have phi > 0 but still contain part of the iso-contour. These cells have no place in A because A only contains phi <= 0 cells. Surface cells (phi > 0) are not liquid cells, they are air cells that contain a part of the iso surface. Liquid cells (phi <= 0) can be fully submerged if there are no air or boundary cells around them. But Liquid cells can have faces that are next to air cells (empty air or surface cells) and in that case they would still be in the matrix. (The definition surface cell and other words don't translate from that paper to my thesis. I tried to define each type of cell; that Foster Fedkiw paper did not)

If you wanted to you could set up the pressure matrix this way. For each Liquid cell (cell with phi <= 0) make a diagonal entry -6. Go through each liquid cell and look at the 6 faces. For each face that is by a solid boundary add 1 to that diagonal. For each face that if by another liquid cell, add a 1 to the off diagonal column that coresopnds to the row that neighbor cell is in. For each face that is by a cell with phi > 0 do nothing.

I would love to have a good explanation of this from the authors but I have never gotten it. If you get it let me know

Also, I never could figure out how they handle solid objects with tangent velocities and things like that, so if you ever find that out let me know There are no details.

Mark

[rjpieke]

When I got the e-mail from rob about this I didn't realize it was an od[force] post... we exchanged a few e-mail about it.  I will most my e-mails to him below for any trolls out there.  It's up to rob to post his if he wants

Not much to post, but here goes.

------ e-mail 1 -----

Hello Rob,

The answer is that Fedkiw and I both do the same thing (as far as I understand it); it's just hard to tell because in a siggraph paper there is often too much to say in the page limit.  Any cell that has fluid (level set <=0 if you are using level sets) has a place in the A matrix.  The ones that drop out are the empty air cells near the surface.  At least that is the way I understood the Foster Fedkiw, and the Enright et al siggraph papers.

You can actually drop out the surface cells with fluid in them if you want, but you will probably get a noisy surface... the key to solving the A matrix is that there must be at least one known (Dirichlet) pressure boundary condition (like the air pressure) or the matrix is singular and can't be inverted with normal CG.  0 is an easy air pressure to use because it means you don't have to move anything over to the B vector.

So, to sum up (or make my rambling explanation more muddy for you), Fedkiw said the terms drop out of the matrix... Not the whole cell drops out of the matrix--just the terms for the empty air faces.  Think of a 1d cell with water on both sides.  You would have the matrix row template -1 2 -1 for a "full" cell.  If  you had air on the left face then you would have 2 -1 because the term for the left face would drop out of the matrix.

The big difference between my thesis and the Enright et al. paper is that I explicitly set my cell face velocities by copying face velocities from nearby faces.  Enright uses an extension velocity.

Hmm, I hope I answered your question... if I made no sense at all feel free to ask more questions any time.

Good luck!

Mark

I agree this is what your paper explains (and it makes a ton of sense, and it works in my solver). But my impression from Fedkiw's paper was that, because the surface cells got set to atmospheric pressure, Pn - Pn-1 = 0, so the row would not be 0 2 -1, but 0 1 -1. This was further enforced by his statement that the diagonal was constructed by the number of adjacent LIQUID cells. But this doesn't seem to work (at least, not for me).

---- e-mail 2 ----

Hey Rob,

1 -1 is for a cell that is near a solid boundary with a 0 Neumann boundary condition.  2 -1 would be a 0 Dirichlet boundary condition.

The only thing I can figure is that in Foster Fedkiw section 7.1.2 the "Liquid Surface" is the cells that have phi > 0 but still contain part of the iso-contour.  These cells have no place in A because A only contains phi <= 0 cells.  Surface cells (phi > 0) are not liquid cells, they are air cells that contain a part of the iso surface.  Liquid cells (phi <= 0) can be fully submerged if there are no air or boundary cells around them.  But Liquid cells can have faces that are next to air cells (empty air or surface cells) and in that case they would still be in the matrix.  (The definition surface cell and other words don't translate from that paper to my thesis.  I tried to define each type of cell; that Foster Fedkiw paper did not)

If you wanted to you could set up the pressure matrix this way.  For each Liquid cell (cell with phi <= 0) make a diagonal entry -6.  Go through each liquid cell and look at the 6 faces.  For each face that is by a solid boundary add 1 to that diagonal.  For each face that if by another liquid cell, add a 1 to the off diagonal column that coresopnds to the row that neighbor cell is in.  For each face that is by a cell with phi > 0 do nothing.

I would love to have a good explanation of this from the authors but I have never gotten it. If you get it let me know

Also, I never could figure out how they handle solid objects with tangent velocities and things like that, so if you ever find that out let me know There are no details.

Mark

19994[/snapback]

Ok, thanks. Well, as mentioned, when I set up the A matrix the way you described, everything works, so I'll just grumble inwardly at the lack of detail in Fedkiw's work.

[MADjestic]

I am just speechless. Most of the stuff you, guys, are discussing is just on or beyond the rim of my comprehension.

I saw the lack of feedback from the rest of odforce ppl and decided to say that you are doing an awesome project! Still just can't believe you can beat RealFlow.

Anyway - Good luck!

[Guest xionmark]

Still just can't believe you can beat RealFlow.

20009[/snapback]

Real Flow will be a distant and fading memory soon!

--Mark

[Marc]

Seriously though, where is Magnus. We gotta get him in here.

Actually I think we have more than one person here at Sony who has written a fluid solver for different companies (DD, R&H, Dreamworks). I'll try and sucker.. er, I mean encourage them to join this discussion.

[FrankFirsching]

Does any know of any "open source" fluid sim projects?  Dr. Mark Carlson (at DNA) pointed me to the BFAST project (I think that's what it's called) but it's not ready for prime time yet.

19761[/snapback]

I've seen this thread quite late, but I hope you all have your notebooks with you at siggraph...

A friend of mine (Nils Thuerey) is developing a quite sophisticated fluid simulator during his phd. Currently he is integrating it with blender as a "Summer of Code" project. (http://wiki.blender.org/bin/view.pl/Blende...CFluidAnimation). You should also take a look at his fluids project page on his private website (http://www.ntoken.de/p_fluid.html). I tried to convince him to do an Houdini integration, but he prefered an open source animation software. He is also at siggraph, presenting a poster about his fluid solver. So go to the poster sessions and try to convince him to participate ;-)

[Guest xionmark]

Hi Frank!

Thanks for the info, I will certainly be visiting the posters area and will try to meet with Nils and talk to him about his project. It'd be great if we could get a couple of fluid solvers implemented ASAP, this is really encouraging.

This is great, I love the energy this is generating! Thanks everyone!

--Mark

[Charlemagne]

i'm here now.. but jeez.. it was two years since i coded any fluids stuff... though in case there are any questions about anything in my thesis though, fire away. though to be quite honest, i had no idea anyone had even read it

-Magnus

[rjpieke]

i had no idea anyone had even read it 

20046[/snapback]

Hehe ... I tripped across it a while back and thought it was really good!

On a completely unrelated note, I really enjoyed the SIG sketch on terrain generation in Stealth, yesterday ... good job!

[Slapdaddy]

I've seen this thread quite late, but I hope you all have your notebooks with you at siggraph...

A friend of mine (Nils Thuerey) is developing a quite sophisticated fluid simulator during his phd. Currently he is integrating it with blender as a "Summer of Code" project. (http://wiki.blender.org/bin/view.pl/Blende...CFluidAnimation).  You should also take a look at his fluids project page on his private website (http://www.ntoken.de/p_fluid.html). I tried to convince him to do an Houdini integration, but he prefered an open source animation software. He is also at siggraph, presenting a poster about his fluid solver. So go to the poster sessions and try to convince him to participate ;-)

20031[/snapback]

I saw Nils' LBM fluid stuff at siggraph, and downloaded his papers as soon as I got back. His stuff looked very good.

[Slapdaddy]

i'm here now.. but jeez.. it was two years since i coded any fluids stuff... though in case there are any questions about anything in my thesis though, fire away. though to be quite honest, i had no idea anyone had even read it 

-Magnus

20046[/snapback]

Heh, you are too modest. I referenced your thesis in my Rigid Fluid paper and in my own thesis. It's very well laid out, and I would pick it up to read whenever I got confused and too bogged down in my own design to understand fluids. You made fluids sound easy. Seriously, Thanks.

Anyway, I don't have any current questions on your thesis, but I would be interested in your interpretation of the Moving Objects section of the Foster-Fedkiw practical animation of liquids paper. Section 9 leaves a lot to be desired. If you know of any better explanation of that anywhere that would be great.

Mark Carlson