The Plan

[Mario Marengo]

Hey Jason...

OK. I'm trying to get a clear idea of the goal here, and after reading through all the posts so far, the most concise thing I saw was "an ODE DOP Solver". So.... is the goal to write a DOP (solver) as a front-end to the ODE RBD solver(s)? ... yes?

If this is the case, then I would imagine the first thing to do would be to bridge all the base data structures so the two libs can exchange the basic stuff like geo-detail (ODE trades in triangles I believe, right?), attributes, and even maybe some fundamental composite types like vectors and such (i.e: does ODE store state as a "phase space", or as assorted attributes -- I would expect the second method on the DOPs side, but I haven't looked yet).

Anyway... all of this would just be the glue needed to bind the two libs together, and I suspect it will be needed regardless of what shape the front end takes in the end (a specialty DOP?, or a series of SOPs a-la SyFlex? -- which makes for a very intuitive flow BTW).

I'll spend some time this weekend looking at both the HDK and ODE, but I would like to know a little more about the "plan" if possible. And also if some work has already been done in any of those areas I just mentioned.

As an aside, I did a lot of research on cloth dynamics about two or three years ago for an in-house thing, and compiled a *lot* of very good reference material... about 8 megs (~500 pages) of .pdf documentation (including the Baraff papers but quite a lot of other stuff as well). There is a definite bias toward cloth solvers in that collection, but a lot of it is generic to all solvers -- basic math, collision concepts, ODEs (ord. diff. eqn.) etc... If anyone's interested let me know.

Cheers!

P.S: Just popped a couple of Advil... now I'm ready to look inside the HDK.... I'm going *in* people!... wish me luck!... see you on the other side....

[Jason]

You're totally right... I've neglected this project for the last 3 weeks since Aeon Flux went into crunch time and it's totally stalled save for an odd mail here and there to keep me idling.

So to start off I went blindly flailing into the fray and I was surprised how easy it was to get started with the absolute basics.

So here is the current state:

With a basic skeleton as a hint provided by SESI, I started throwing in ODE calls to create the world, define the objects as bounding boxes, define some of their basic properties, define the uniform forces and the gravity force.

Currently it defines and destroys the entire world for every time step.

For each timestep

- Define world

- Create ODE object (bbox) for every SIM object

- Define forces and Gravity

- Step the ODE solver

- Pull info from ODE and stuff back into DOPs sim objects.

- Destroy the world.

So as a test, it was pretty successful, but it really needs to do more - so this what I think it outstanding:

- Decent coliision routine that'll also populate Impacts subdata

- DOPs Constraints -> ODE Joints translation (tricky)

- Options for geometry representation - BBox, OBB, Trimesh

- - Houdini GDP->ODE Trimesh translation. (tricky too)

As for optimization - currently defining the ODE objects is slow so the whole thing solves slowly. After querying the ODE forum as to what the slowest portion is - (world creation? hash space creation? object definition? ) - it turns out that defining the objects is the slowest. So we'd best extract the object defining portion to be persistent, probably by defining a map to the ODE objects maintaining their DOPs unique ID and merely update their positions throughout the simulation. I suppose in the future we'd have to be careful that any geometry changes during the simulation are respected. SESI, are there easy ways to tell if the geometry changed from the last timestep?

We'd obviously have to extract the WorldCreate() and HashSpaceCreate()'s out too. I'm not entirely sure if the Partition DOP can be mapped to ODE Hash spaces... or are partitions all purely managed by Houdini?

I have a couple of hints about how to attach custom SIM_Data for the ODE information but I'm not sure I quite *get it* yet. I feel it'd be great to expose as much data into the SIM_Data as possible but I have no real clue on what or how quite yet.

I'll upload the zip file and post in again. Beware, its messy - not the code -but there is no Makefile and I don't know the "correct" way of packaging up libs and such. I was hoping a seasoned programmer like Mark Story would be able to prep it professionally for an efficient distribution - Mark?

Thanks for taking an interest, Mario!

Jason

[Mario Marengo]

Well.... as it turns out, I only managed to squeeze in about two hours of looking into ODE and HDK-DOPs this weekend

Starting to get some gut-level feel for both structures, but will need some more exploring to be able to start doing some preliminary damage

I'll continue to explore, but if you could post those files, no matter how "raw" they are, it would really help things along.

I'll keep you posted as I get more comfy with things... BTW... I'm starting to get all kinds of nasty ideas just looking at the DOP api.... and the ODE api seems pretty straight forward.... everything is an array, lol! (object smobject!...take *that* Mr. C++!!, hehehe).

Cheers!

[Jason]

The Solver So Far

As I said, it could use a makefile and it could also be made to link against USER32.dll on windows (as explained how in the MessageBox() thread) in order to stop needing to custsomize the .lib on w32.

It's very raw, but thats just me

[Daniel]

Hey Guys, I'm intrested in contibuting as well. At Sony we've already built ode into a sop.. so it sort of runs like the old particle sop where it's a sim inside a sop. Dave Davies at Sony built it for Polar Express, he had a pretty decent version working in just a few days (less then a week).. then he added constrains and other features over time. I'll talk to him and see if he'd be into participating. It does seem benificial to get a comunity effort going rather then each fx house rolling thier own.

The beauty of ODE is it mainly deals in very simple primitive types.. box, sphere, capped cylinder... this is so it's fast for it to do inside/outside tests. It can deal in triangulated poly data, though it's not nearly as good at that. Personally I like the simplicity of it.. you can combine these primitaves into a composite object to make arbitrary shapes that act as a single body.. in this way you can build a tailored rbd object but the collide tests are still extreamly fast. I woudln't want DOP's intergration to slow the whole thing down too much

I'll take a look at the code sample, thanks Jason!

daniel

[Jason]

Hi Daniel,

Fantastic to have you involved!

It seems like this is perfect time now to start playing with this code in ernest ,considering SESI have extracted the Collider and gotten the constraints structure in DOPs 99% of the way there. Perhaps someone from SESI could give us a bunch of recommendations for tackling this - basics on colliders, how to carry data (like the trimesh structures) and such in SIM_Data. Any way to make it play the best with the rest of DOPs.

This test implementation gets basic collisions with bounding boxes going, but the overhead in this test is kinda expensive. SESI, could you give us some advice on how you'd structure it?

I also think there is a place for simple fast primitive solver alongside the one SESI has in there. Perhaps these days its meant to be more a matter of swapping a Volume/Volume Collider with a simple BoundingPrim/BoundingPrim Collider kinda thing?

Thanks,

Jason

(I suppose I should clean it up and commit this to the odforce sourceforge page soon.)

[Daniel]

(I suppose I should clean it up and commit this to the odforce sourceforge page soon.)

18698[/snapback]

Right, I tried compiling the code and did have to make some modifications to get it to work with the latest HDK headers, some of the class interfaces have changed since you last worked on this.

d

[Jason]

Right, I tried compiling the code and did have to make some modifications to get it to work with the latest HDK headers, some of the class interfaces have changed since you last worked on this.

d

18699[/snapback]

Fantastic- thanks! From the comments on the SESI H8 forum, it seems to me like a major part of the interface have been through enough revisions to hope it's fairly stable nowadays - now this is out there perhaps SESI can drop hints when libraries change to make keeping up with 8-alpha a simple process.

[Mario Marengo]

Hey Jason,

Apologies for not doing much with this yet. Still interested, but won't have a chance to explore until my next "play time" arrives...

BTW, I think the sourceforge thing is a great idea! Thanks for setting that up.

Cheers!

[Jason]

Apologies for not doing much with this yet. Still interested, but won't have a chance to explore until my next "play time" arrives...

BTW, I think the sourceforge thing is a great idea! Thanks for setting that up.

That's cool; take your time..

Does anyone here have much experience with setting up distribution thats acceptable for sharing on sourceforge?

[BrianK]

Just came across this sub-forum.

I've been working on an ODE SOP for a couple weeks - had no idea you guys were too (working with ODE)... I'm up for helping out.

[mtucker]

My first ever odforce post...

Yes indeedy, the DOPs code has settled down quite a bit, so it should be safe to start working on something like this without worrying about the code breaking with each new release. I will try to remember to post here if anything important changes in the SIM libraries. I expect there will be a couple of small things, but certainly nothing major coming up to the release.

For those interested in developing DOPs plugins, the best place to look is the HDK header files under the SIM directory. Because we wanted to encourage 3rd party development with DOPs in particular we've tried to keep the important header files well documented. The key ones to start with are SIM_Engine, SIM_Data, and SIM_Solver. There are also a couple of sample solvers in the toolkit/samples/SIM directory. Certainly nothing exciting, but I find it always helps to have a starting point.

As for hints on where to go next, I'll start with optimization. I assume the biggest performance penalty right now comes from the fact that the entire ODE world is rebuilt from scratch on each timestep. Ideally, when advancing to a new timestep, you would simply call the ODE function to advance the world to the next timestep, and then copy the new information out of the ODE world onto the DOP objects. But this can only be done if nothing has changed on the DOP objects since the end of the last timestep (such as a DOP node directly manipulating the velocity or position of an object). If such a change has occurred, then you may have to rebuild the ODE world from scratch (or simply update the information of that one object - I don't know if ODE allows that kind of brute force manipulation of objects). So the key is detecting when the DOP objects have changed since the last ODE solve. This is very easily done by looking at the unique id of the pertinent data on each object. If the unique id on the position and geometry data of the object have not changed since the end of the previous timestep, then they have not been modified. This just means storing the unique ids of relevent data along with the ODE world.

Speaking of the ODE world, how do you keep using the same ODE world from one frame to the next (assuming none of the objects gets modified by the DOP Network)? I would keep the ODE World and the unique data ids (or maybe a hash combining all the unique ids of all relevent data to save space) in a SIM_Data that would be subdata of the ODE_Solver. By putting them in subdata of the ODE_Solver (instead of member data of the ODE Solver) you ensure that there is one and only one copy of the ODE_World data. At each timestep, a new copy of the ODE_Solver data is created, but all copies of this data will point to the same ODE_World subdata (along with the associated unique id hash). At each timestep, the ODE_Solve would look for this ODE_World subdata. If it isn't there, it gets built by adding all the DOP objects to the ODE World, and calculating the unique id hash. If it is there, the current DOP objects are compared to the current unique id hash. If they don't match, the ODE_World data is removed and a new ODE_World is built. If they do match, the solver just tells the ODE_World to have ODE do a timestep, copies the object position data out of the ODE_World into the DOP Objects, and updates the unique data id hash.

Note that it is the ODE_World subdata that owns the ODE World, and is responsible for creating and deleting the world. Also note that the unique data id hash would have to be mutable data on the ODE_World. If it was not mutable, then the value couldn't be updated on each timestep without making a new copy of the ODE_World, which would mean there would no longer be a one to one correspondence of ODE_World SIM_Data and ODE worlds, so they couldn't be deallocated properly. Finally, the ODE_World data will not be able to write to or read from disk. If a simulation with an ODE solver is written to disk and read back in, and then asked to simulate, the ODE_World will have to be created from scratch. This is unavoidable because ODE doesn't support saving and loading worlds to disk (as far as I know).

Now on to constraints. Just as position data will get copied into the ODE world from te DOP objects, constraints will also have to be copied in. This is something you may want to do on each timestep, since it s quite common for constraints to be animating, and since I imagine adding and removing constraints from an ODE world would be a pretty fast operation. I think ODE can represent all constraint types currently supported by DOPs. Plus I believe ODE supports some constraint types not supported by DOPs. DOPs versions of these ODE constraints can certainly be created (though they would not be supported by othe DOPs solvers). But I would save that for later...

As for collisions, it is not necessary to use a SIM_Collider to do your collision detection. You can let ODE do the collision detection internally. But to make ODE fit well into DOPs, you'd want to at least extract any collision information out of ODE and into SIM_Impacts data on the objects. This is how I would do things initially. If you decided later that you wanted more contro lover collision detection, you could change this so that collision detection is done with an existing SIM_Collider, or with a new SIM_Collider that just treats objects as bounding primitives or whatever. I believe ODE lets you customize the collision detection functionality, but I'm not sure how... The advantage to using a SIM_Collider is that you would e able to change collision detection routines very easily (to use our SDF collider, for example). It would also mean that whatever collider you wrote could be used by other DOPs solvers (such as our RBD solver).

One final thing to consider in all this is interactions between objects with different solvers. When two DOP objects have different solvers (such as a POP Solver on some particles, and an RBD Solver on a cube), they are still solved sequentially. First the RBD Solver is run, then the POP Solver (or perhaps the reverse). When the RBD Solver is run, it is not told about the POP Object at all, so it behaves as if it isn't there. When the POP Objet solves, it is told that the RBD Object is there, and that it is an object to which "feedback" impacts should be applied. So the POP Solver collides particles with the RBD Object, and attache impact data to bothe the particles and the RBD Object. It doesn't actually move or change the RBD Object in any way, it just attaches the impact data. Then the POP Solver returns control to the engine. The engine notices that "feedback" impacts have been added to the RBD Object, so it rolls back the simulation to the previous timstep, except it keeps all the impacts between the POP and RBD Objects. Then it resimulates the timestep. Again the RBD Object is ignorant of the POP Object, but this time it has these feedback impacts which are applied and alter the end position of the object. Then the POP Solver is run again. It also has the feedback impacts, and applies them. Plus it again does collision detection with the RBD Object applying more feedback impacts if it still hits the RBD Object. This iteration continue until the "Feedback Iterations" limit is reached, or the solvers run without any new feedback impacts being added.

So what does all this mean for the ODE Solver? Since ODE does not support rolling back of the simulation, to support this sort of feedback I believe the ODE World will need to be recreated from scratch at each feedback iteration. So there will be a definite performance penalty when interacting with objects using other solvers. The up side is that I believe the mechanism I describe above will give the correct behavior of rebuilding the ODE world already. The other up side is that the ODE solver will be able to inteact with DOP objects using other solvers just like any other DOP solver.

That's probably a lot of information to digest. I hope it is helpful. Please feel free to ask more questions. I'll be checking here regularly now (or on SourceForge if that gets set up - I don't know much about how SourceForge works or what it supports in terms of discussions or forums or whatever).

I'd also like to point out that SourceForge seems to have a couple of fluid simulation projects that would also be great to see in DOPs

Mark

[Mario Marengo]

Wow Mark!!!

That's a tonne of useful information. And especially useful because it's the kind of "design" stuff that is so hard to glean from just looking at the headers.

Thank you!

And a big welcome to od[force]!

[Jason]

A very excellent post that has me scratching my head already:)

[Marc]

Yeah, that's awesome. Thanks Mark.

And welcome to od[force] .

M

[stevenong]

Great stuff Mark!!! Thank you!

I feel empowered now with all this information but not that I know how to use it.

Maybe in the near future.

Cheers!

steven