mantra "clusterThis" procedural geometry DSO

[Guest xionmark]

Greetings,

I'm pleased to release the first version of the Houdini/mantra "clusterThis" geometry/volume instancer. The DSO will create instances of primitives for each point of the incoming geometry and be "displaced" around the point based on a number of user parameters and methods.

The project is an open source project and will also be on the source forge odforce site for other developers to contribute if they would like. And the project is also going to be a teaching project for students attending workshops at Digital Cinema Arts and soon will end up as a robust, production worthy tool (one of several tools we plan to build to teach with *and* to use in production). Ever since I was on "Surf's Up" I've been wanting to write a few render time plugins, especially seeing how much the prman "cluster" DSO added to the detail of some those shots, and it seemed to be an easy application of a simple idea. Fun stuff mon!

The current state of the code is just the basic structure to get the project started, very simple algorithms for now ... and there's definitely bugs! Use at our own risk! The noise generator is right out of the HDK toolkit, nothing else used yet. Soon I will have the CVEX functionality then you'll be able to use standard VEX code for processing the instanced objects. Truthfully, I've been having so much fun with it I've been a bit tardy in getting the code released, so I know once a few more features are implemented there will be good fun for all.

Keep in mind this is a geometry instancer, it merely off loads the creation of geometry until render time, so keep an eye on memory usage. Future projects will include some shaders that will do similar kinds of things but of course limited to what can be done with just shaders ... the two tools together will give an artist a lot of capability for managing complex detail.

The current version is for mantra only but my plan is to generalize the project appropriately (through proper design) to provide multi renderer support, mental ray, 3Delight and others are on the future list for deployment.

We have a small team of programmers to work on the project, any contributions, suggestions from the Houdini community are always welcome. DCA will "drive" the project as it's intended first and foremost as a "teaching project", but of course we want it to be part of the ever growing Houdini community tool set.

Here's the web page where you can download the code, DSO/DLL (and yes, there's a Mac OSX version!), and see some example pics/movies as we test and develop the project:

http://www.digitalcinemaarts.com/dev/clusterThis/index.html

(please excuse the web site's condition ... it's in the process of a badly needed face lift ...)

I also have a little utility SOP to allow the setting of a number of point attribute in one SOP (I got tired of chains of AttributeCreate SOPs) but the local variable support isn't done yet, it will be in the next release. Here's the web page in case you find it useful as it is (I do).

Point attribute SOP:

http://www.digitalcinemaarts.com/dev/SOP_P...bute/index.html

And, here's a more or less complete list of current and planned features for this project. The docs, web page, examples, and of course the code, will be updated quite regularly, check back often.

Features:

(*) features coming soon

# Instance a variety of geometry primitives (sphere, cube, grid, circle, tube, curve (*), and BGEO file (*))

# Copy point attributes to the instanced geometry

# "Recursive" point generation to displace the instanced geometry "about" the position of the incoming point's position.

# Noise generator for additional displacement data

# Attribute driven behavior (pscale, speed stretching, etc.) and blending (via CVEX (*))

# CVEX processing (*)

# Application of volumetric modulation data. (*)

# Multi render pass optimizations (almost there, will be in next release) (*)

# Motion blur (almost there, will be in next release) (*)

# Volumes instancing

# CVEX processing (*)

# More sophisticated noise/filter functions (fBm, gaussian, bezel, impulse, butterworth, sinc) (*)

# Lacunarity and true fractalization (*)

# "RiPoints" instancing (*)

# RIB output (already there, just need interface change) (*)

# Proper primitive/point attribute blending/processing (CVEX *)

# Read volumetric data files and use for modulation of instanced geometry and attribute processing (DT grids). (*)

# Optimizations of all kinds (especially when generating deep shadow passes) (*)

# Velocity/Force stretching the prims and blending/interpolating the attributes (*)

# Primitive selection from point attribute (prim_type) (*)

Please send bug reports, questions, feature requests to me at mstory@xion.org.

Thanks,

Mark

[aracid]

Brilliant

thanks for sharing this

[eetu]

Whee!

This and Ole's lightrig both in the same day, it must be christmas!

eetu.

[symek]

Whee!

This and Ole's lightrig both in the same day, it must be christmas!

eetu.

Indeed, sounds amazing! I missed such functionality a lot.

Thank you Mark!. Could you tell few words about educational aspect of that work? Are you going to start some HDK course?

Thanks!

Simon.

[Mario Marengo]

Sounds good, Mark.

Thanks for sharing!

[Alanw]

This is excellent. Thanks a lot for sharing it with the community.

[Guest xionmark]

Brilliant

thanks for sharing this

My pleasure!

[Guest xionmark]

Whee!

This and Ole's lightrig both in the same day, it must be christmas!

eetu.

Seems the holiday spirit is spreading! Excellent!

I was wondering if anyone had a robust camera rig that they could contribute to the exchange. Something like the one I've attached but with a frustrum, ner/far planes, etc. Maybe it's on out there and I missed it but wanted to check before making one.

Thanks,

Mark

[Guest xionmark]

Indeed, sounds amazing! I missed such functionality a lot.

Thank you Mark!. Could you tell few words about educational aspect of that work? Are you going to start some HDK course?

Thanks!

Simon.

Sure! There's nothing like using good tools in production right, and even better, when you're trying learn how to use a new method or tool. Most of my experience with schools and tech training "providers" (sorry, can't find the right word) has been disappointing in how minimal the resources are for students trying to learn CG/VFX. So our intent is to provide training services/workshops that use real production tools and a professional pipeline.

Image Engine (in Vancouver) has a nice project started to share a set of their production tools, here's the web site: http://code.google.com/p/cortex-vfx/

I'll be writing some of the Houdini related components for their libraries.

But, for instance, it would so nice not to have to write another "setshot" ... etc. ... And I know there's a number of ways to design a pipeline but "setshot" is one of those basic tools that's not only surprisingly absent in some studios, but pretty much non existent in education facilities. So if a student comes out of a course or program not even exposed to the notion and practice of working in a proper production environment, it's difficult to think they have a well developed understanding of a real world production pipeline.

We're hoping to start providing workshops within a month or two, certainly by the new year. Stay tuned! :-)

Mark

[Guest xionmark]

Sounds good, Mark.

Thanks for sharing!

You're welcome! Ummmm .... I've got a few math geniuses within arm's reach to help me out with the mathematics for this project, but you're input/feedback/recommendations would be great too! Once this starts rolling along, I want to do some SSS kind of stuff with the CVEX code, point cloud mania as it were.

I'll be posting more details on more of this but I'm so excited about using CVEX! I have plans (soon to start) to apply CVEX to CSound audio synthesis generation. CSound is continuing to develop in really interesting ways, so much potential for audio/visual expression.

See ya!

Mark

[Guest xionmark]

NOTE: There's a new version of the first DSO with the copy/recursion loop fixed. I'll get the motion blur and a few other things working whilst figuring out the problem with my addProcedural() implementation.

[eetu]

I've been playing a bit with this (thanks!) - and some questions have risen..

Mpoly vs raytrace engine. When using mpoly engine with raytrace shadows, it seems to create the instances twice, taking about twice the memory when compared to RT engine. Is this just the way mantra works with this combination? It always creates a duplicate of the scene for raytracing the shadows?

I've always envisioned that an instancer would only create its geometry one bucket at a time, and only the geometry intersecting that bucket. Is that possible with mantra, can one hook that "late" into the pipeline? The way it works now feels like it "expands" the ifd while loading, by inserting the entities created by the procedural. (?)

Also, I had troubles getting it to work with just geo points instead of particles. I get a "VRAY_clusterThis_Exception::what() - VRAY_clusterThis exception: VRAY_clusterThis::render() Incoming points must have CD, v, N, pscale and id attributes! " - even though I do have all the same attributes defined on the points as in the particles.

Is there something simple I'm missing?

Anyway, thanks a lot for sharing this stuff!

eetu.

[Guest xionmark]

I've been playing a bit with this (thanks!) - and some questions have risen..

Mpoly vs raytrace engine. When using mpoly engine with raytrace shadows, it seems to create the instances twice, taking about twice the memory when compared to RT engine. Is this just the way mantra works with this combination? It always creates a duplicate of the scene for raytracing the shadows?

I've always envisioned that an instancer would only create its geometry one bucket at a time, and only the geometry intersecting that bucket. Is that possible with mantra, can one hook that "late" into the pipeline? The way it works now feels like it "expands" the ifd while loading, by inserting the entities created by the procedural. (?)

Also, I had troubles getting it to work with just geo points instead of particles. I get a "VRAY_clusterThis_Exception::what() - VRAY_clusterThis exception: VRAY_clusterThis::render() Incoming points must have CD, v, N, pscale and id attributes! " - even though I do have all the same attributes defined on the points as in the particles.

Is there something simple I'm missing?

Anyway, thanks a lot for sharing this stuff!

eetu.

Hello,

Sorry for delayed replies, really swamped. I've got a new version ready for you to check out and will try to answer your questions.

Here's the link:

http://digitalcinemaarts.com/dev/clusterThis/index.html

Try this version and let me know.

As for your questions, as long as *any* point coming into the shader has those 5 required attributes, it's should work fine (double check your spreadsheet), It doesn't have to be points generated from a particles system.

Yes, it seems when mantra needs to raytrace shadows or generate deep shadow maps, it calls the render() method for *each* light and then once again for what seems to be the final call before the rendering starts. There's another thread going on the SESI forum that is related to this subject:

http://www.sidefx.com/index.php?option=com...sc&start=25

As far as the idea of the geometry being created for each bucket, this is not that type of procedural, but, can of course be achieved (to some extent) as just a surface/disp shader, but there's limitations as well.

The benefits for this type of procedural are (at least):

* off loading the generation of geometry until render time and thus keeping the IFD light and interactive session less encumbered by not having all the geo in memory.

* By using the addProcedural() method (which is in the new version) it has the "built in" benefit (apparently) of not having it's render() method called unless the point's in camera view.

* You can "do things" at render time that isn't achievable without mantra running (obvious).

** Some of those things might be: processing the instanced geometry with external data, running additional processing functions at render time based on data retrievable from the renderer, surfacing functions optimized based on rendering conditions, etc.

Check out the thread I mentioned above on SESI's forum, I'm going into detail about what I'm finding with issues regarding depth maps, motion blur, etc., should be a good learning project as well as create lots of examples to work from.

Mark

[Marc]

Any chance of a 64bit windows version of this? I could compile it myself, of course, but I'm lazy .

[Guest xionmark]

Any chance of a 64bit windows version of this? I could compile it myself, of course, but I'm lazy .

Hi Marc,

I would but I have no access to a 64 bit Win machine.

Do you have a compiler/dev tools installed?

Mark

[Marc]

sadly not... It's ok though, I've jumped over into 64bit linux for now. So I should be fine.

Thanks

Marc

[symek]

Hi Marc,

I would but I have no access to a 64 bit Win machine.

Do you have a compiler/dev tools installed?

Mark

Hello Mark,

are there any chances for implementing metaballs instancing, or some other smart way for rendertime surfacing millions of particles ?

[Guest xionmark]

Hello Mark,

are there any chances for implementing metaballs instancing, or some other smart way for rendertime surfacing millions of particles ?

Absolutely. I've been distracted the last week and a half but am back to working on it. I need to get a few basic things solved first, motion blur, and point attribute shader strings being passed to the instanced geo, and to hopefully figure out why the (what seems to be) the preferred way of instancing (using the addProcedural() method instead of generating all the geometry at once) is behaving so weird.

I also want to get the volume instancing happening ASAP.

I will likely have another minor release tomorrow, just looking at it right now.

Also should have another programmer (or two) on the project soon to help move it forward, I *really* want to see the CVEX feature in there soon!

Mark

[Guest xionmark]

Hello Mark,

are there any chances for implementing metaballs instancing, or some other smart way for rendertime surfacing millions of particles ?

Hi there,

I have metaball instancing working, will be in the next release.

Check out the blobbies in the attached image.

[symek]

Hi there,

I have metaball instancing working, will be in the next release.

Check out the blobbies in the attached image.

Wow! Are you compete with SESI upon the fastest development feedback? I wonder, wouldn't this technique had more sense if we were be able to shape mesh with points attributes (CVEX)? In general such dso should have at least as much control over mesh shape as particle surface SOP. Is it possible at all in case of general instancer dso? Just thinking. Thanks for another great tool, Mark!

Simon.

Edited October 29, 2008 by SYmek