Render Insane amount of geo in mantra ?

[sebkaine]

Hi Guys,

 

After watching 2 film render with clarisse, my desire to replicate this kind of shot with so much geo is now pretty high.

It would be nice to get as much advise from Mantra Expert on how to put Mantra on his knees in the most elegant way.

 

http://www.isotropix.com/index.php?to=news&view=viewid&id=58

http://www.isotropix.com/index.php?to=news&view=viewid&id=77

 

This brings 2 questions :

 

-> How to instance geo in the most efficient way inside H ?

 

As i use Houdini Indie i'm not sure i will be possible to generate ifd and call them and shade them at Rendertime,

Like you would do with a tradional RIB Box workflow.

 

So this leave us with instancing directly inside H. To my knowledge you only have 3 ways 

1- the copy SOP 

2- the new instance geometry node

3- the old instance node that use the copy internally as it is basically the same thing as 1 i will exlude it.

 

From my test the instance geometry is the fatest and most efficient way , but i would be curious to get some external feedback about this ?

Maybe i've misse other technics ?

 

-> How to limit as much as you can the load time of insane amount of geo ?

 

Instancing ifd will be i guess the fastest way cause you send geo in Mantra native format , but as we don't have this in indie

I would say that Pack Primitive Poly will be our best alternative ?

 

I would be glad to hear as many advise on the subject ...

That would be fun to build those kind of huge scene and see how Mantra can compet in those scenario.

 

Thanks for your time !

 

Cheers 

 

E

Edited January 25, 2016 by sebkaine

[fathom]

i've usually used the "instancefile" point attributes which pull in geo off disk based on the filename you pass to each point.  copying packed prims may be just as efficient, so i may switch over to that now.

 

in terms of instancing, you have to avoid dicing or you'll break the instance since the dicing will be different for each copy of your geo.  keep in mind that an empty displacement shader will still dice (or it used to anyway).  so only apply surface shaders, not "materials" which include empty displacements.

 

it'd definitely be an interesting test.

[sebkaine]

Thanks for your helpful answer Miles !

 

I have never use the instancefile tricks you mention.

this i guess the node you mention in Shop > Point Instance Procedral 

http://www.sidefx.com/docs/houdini15.0/nodes/vop/ptinstance

 

Instance Object : The object to instantiate on each point on the point object.

- This can be overridden, on a per-point basis, by the @instance attribute on the point object.
- In addition, it’s possible to do a delayed load of geometries, using the @instancefile attributes. 

 

This looks to work in a similar fashion to the new instance node, but you don't have to pay the viewport display price, interesting !

- @instance should work with pack polygon written as .bgeo.sc ?  what would be the clever way to write "RIB" in mantra in the most efficient way (outside of IFD)

- @instancefile look to be reserved for delayed load geo so i guess this means IFD only ? and not indie viable ?

 

You lost me on the dicing step ! what do you mean exactly ? in renderman it was the process of splitting the geo into micropolygon but in Mantra what does it mean exactly ?

Edited January 25, 2016 by sebkaine

[fathom]

the instancefile stuff is all normal geo files, not ifd.  i don't think you ever really instance ifd's anywhere except for material archives.  mantra is different from renderman in that it has geo formats where renderman geo is only handled via rib calls.  in an ifd, geometry is included "inline" as opposed to in a file, but it's still the same format and i would wager goes thru nearly the exact same code path.

 

as for dicing, i guess mantra technically calls it "refining"... it's for nurbs, subd, displacement... it has to subdivide until it gets something it can shade.

 

here's a detailed write-up of the process:

 

http://www.sidefx.com/docs/houdini15.0/render/understanding

 

 

mantra does dice into micro-polys for all objects in micropoly rendering, but those get tossed after the bucket, so they don't break instancing.  polys can be traced directly in mantra, so they don't need to be refined/diced unless they're displaced.

[Mandrake0]

i don't know if you have seen it but these tips can help.

 

Rendering tips:

http://www.sidefx.com/docs/houdini15.0/render/renderingmanypolys

 

Instancing tips:

http://www.sidefx.com/docs/houdini15.0/copy/packed_and_soups

[sebkaine]

Thanks again for your very helpful answer guys, like always with H i feel that when you start to dig the problem you are rarely disapointed,

I'm quite sure those kind of rendering can be achieve ... 

 

 

the instancefile stuff is all normal geo files, not ifd.  i don't think you ever really instance ifd's anywhere except for material archives.  mantra is different from renderman in that it has geo formats where renderman geo is only handled via rib calls.  in an ifd, geometry is included "inline" as opposed to in a file, but it's still the same format and i would wager goes thru nearly the exact same code path.

 

To be honest i haven't understand at 100% when to use IFD, from what you said we just need them when we want to take advantage of the infinite Mantra Token.

So you generate IFD with a valid license and then you can send those IFD to render on your farm with as much machine as you have.

 

Do you see other scenario where IFD might be useful Miles ? you mention "material archives." But you can directly export vex code ? 

 

From what i understand from "inline" basically mantra doesn't write any geo / UV / info in those IFD, it only contain a path to a geo , but it doesn't contain the geo itself.

Exactly what RIB do when you include your shaders in the RIB. It just write a path that point to the .sl written on disk.

 

Now what is the most convenient format to store geo on disc bgeo.sc of pack polygons ? Does the sc compression give you any benefit in the memory ? or does mantra has to uncompress the geo ?

 

I have some reading to do for the moment !

When i have some time i will prepare a test hip to test all this in practice !

 

Cheers 

 

E

Edited January 26, 2016 by sebkaine

[Stalkerx777]

 keep in mind that an empty displacement shader will still dice (or it used to anyway).  so only apply surface shaders, not "materials" which include empty displacements.

This was true a couple of H versions ago, i think it's already fixed, but i may wrong.

Using displacement is not an option with instancing, however vm_sharedisplace may help, although i've never used it.

 

I would be glad to hear as many advise on the subject ...

From my experience - instancefile attribute (+point instance procedural) was the fastest and most efficient way of instancing. The downside of it is that there is no easy way to randomize values in shader. You either have to prepare N variations of you geometry (unique attrib in each bgeo file) or use shader tricks such as using point clouds to read nearest point this particular instance refer to, and get random() from this point. Or you can use clusterThis  procedural which allows you to modify instanced geometry with CVEX shader.

 

Another approach is Packed Primitives. In this case this primitives will be written to IFD (or streamed directly to mantra).If you have small amount of unique primitives, and not going to modify their attributes via copy - stamping, they will be shared in memory, even if you clone them 10e12 times. But if you want to randomize something (for example random ID on each packed prim) you lose all benefits of this type of shared access, since all primitives will become unique.

Randomizing shading on packed primitives can be achieved with renderstate function, BUT, since this is only one primitive, you can't have multiple materials on it. But you can use Style Sheets to randomize shader values. 

So for example if you have 20 heavy unique trees, each of them is packed primitive, Houdini will export them to Mantra each time you hit render, and this can be very long process. 

 

There is also Packed Disk Primitives, which is a "shortcut" to delayed load procedural (+ sharing in memory). When mantra see this type of primitive, it will read directly from disk. Same as Alembic Delayed Load.

Another example:

1. Load any geometry, prepare it, assign attributes.
2. Convert to PolySoup, export to bgeo.
3. Load as Packed Disk Primitives using file sop.
4. Use Copy Sop to scatter.

In IFD, you will see a reference to packed disk file on each scatter point. Same fo Alembic Delayed Load.

[
    {
        "expandfilename":false,
        "expandframe":1,
        "filename":"my_geometry.bgeo",
        "pointinstancetransform":false
    },80
],

 

 

From what i understand from "inline" basically mantra doesn't write any geo / UV / info in those IFD, it only contain a path to a geo , but it doesn't contain the geo itself.

 

 

Houdini writes only regular geometry(including volumes) to IFD stream. If you use Packed, Alembic, Delayed Load, IFD archives, only reference to this geometry will be written to IFD in this case. Nowadays Houdini streams data directly to Mantra process bypassing intermediate IFD, but it uses the same format).

 

 

Now what is the most convenient format to store geo on disc bgeo.sc of pack polygons ? Does the sc compression give you any benefit in the memory ? or does mantra has to uncompress the geo ?

bgeo.sc is a new one, don't have an experience wiht that. Any compressed data should be decompressed. So yes, Mantra will decompress it. This is a trade-off. You can save memory OR you can save processing time.

Edited January 26, 2016 by Stalkerx777

[Mzigaib]

This thread is awesome!

 

I am going through a process like this myself where I need also to have a insane amount of geometry in my scenario, it is a big area like Kilometers wide where the camera will travel through so no matte matte paints allowed for grass, trees and stuff like that.

 

I did a simple test with one instance tree in Houdini using and Clarisse each one with an area of a Kilometer wide with a lot of trees 500.000 copies to be exact each copy with a little more that 1 Mil polys and I got similar render times the only difference that I've felt is that Houdini took more time to build geometry before start render, I am not sure if this is the right term is what it says when it starts, but besides that I think you can do the same thing in both in terms of actually achieving rendering lots of geometry, I didn't test it with more geometry variations but now with those tips I can expand my options, the "instancefile" attribute is the next on my list since my test in Houdini was the standard instance node with the geometry loading with the hip file.

 

I've never used Poly soups but it also seem a good option, like Alexey was explaining saving as a poly soup and loading as packed primitives is also something I've never tried, I would lke to understand better the difference between poly soups and packed primitives and where is the best time to use one or another.

 

Can anyone tell me if it is possible to load as packed primitives somehow using the "instancefile" attribute? 

 

Anyway thanks for the tips, cheers!

[fathom]

 

Thanks again for your very helpful answer guys, like always with H i feel that when you start to dig the problem you are rarely disapointed,

I'm quite sure those kind of rendering can be achieve ... 

 

 

 

To be honest i haven't understand at 100% when to use IFD, from what you said we just need them when we want to take advantage of the infinite Mantra Token.

So you generate IFD with a valid license and then you can send those IFD to render on your farm with as much machine as you have.

 

Do you see other scenario where IFD might be useful Miles ? you mention "material archives." But you can directly export vex code ? 

 

From what i understand from "inline" basically mantra doesn't write any geo / UV / info in those IFD, it only contain a path to a geo , but it doesn't contain the geo itself.

Exactly what RIB do when you include your shaders in the RIB. It just write a path that point to the .sl written on disk.

 

Now what is the most convenient format to store geo on disc bgeo.sc of pack polygons ? Does the sc compression give you any benefit in the memory ? or does mantra has to uncompress the geo ?

 

I have some reading to do for the moment !

When i have some time i will prepare a test hip to test all this in practice !

 

Cheers 

 

E

 

 

ifd's are beneficial when rendering to a farm so you don't tie up houdini tokens waiting for mantra to process and so you can get more renders going (which usually take longer than the ifd generation process).

 

material archives are what you get when you run a mantra archive rop.  the geo is dumped to a normal geometry file and the materials are dumped to an ifd which can be used later in a delayload procedural so you can have your lookdev easily separated from your working lighting file.  not sure how much it gets used in practice tho, since there are other ways to handle that kind of thing (importing the shaders into your houdini file, for example).  so in general, you don't typically use ifd archives in the same way a rib archive gets used -- mostly because mantra can read geo of off disk very easily.

 

i still use bgeo for my files.  bgeo.sc might be nice.  i dunno.  the issue is that you have to decompress them.  i've used bgeo.gz in the past when i have super huge sims.

[fathom]

This thread is awesome!

 

I am going through a process like this myself where I need also to have a insane amount of geometry in my scenario, it is a big area like Kilometers wide where the camera will travel through so no matte matte paints allowed for grass, trees and stuff like that.

 

I did a simple test with one instance tree in Houdini using and Clarisse each one with an area of a Kilometer wide with a lot of trees 500.000 copies to be exact each copy with a little more that 1 Mil polys and I got similar render times the only difference that I've felt is that Houdini took more time to build geometry before start render, I am not sure if this is the right term is what it says when it starts, but besides that I think you can do the same thing in both in terms of actually achieving rendering lots of geometry, I didn't test it with more geometry variations but now with those tips I can expand my options, the "instancefile" attribute is the next on my list since my test in Houdini was the standard instance node with the geometry loading with the hip file.

 

I've never used Poly soups but it also seem a good option, like Alexey was explaining saving as a poly soup and loading as packed primitives is also something I've never tried, I would lke to understand better the difference between poly soups and packed primitives and where is the best time to use one or another.

 

Can anyone tell me if it is possible to load as packed primitives somehow using the "instancefile" attribute? 

 

Anyway thanks for the tips, cheers!

 

so another complication with houdini and instancing relates to how houdini generates the ifd.  there are times where it spends a long time processing the geo thru the soho interface (the thing that takes your scene and makes the ifd or rib or whatever-other-format).  i've seen cases in the past where instancing can slow WAY down because soho is iterating over each instance to create unique data.  that might be what you were experiencing when you say mantra took longer to start rendering -- it might have been houdini prepping data.  that said, mantra does often spend a bit of time prepping things.  the refining step is still single-threaded in mantra, i believe.

 

polysoups don't differ much from polys.  the big difference is if you have primitive attributes.  if you have 1million polys that have prim materials assigned, that's 1 million copies of that prim material specification.  if you convert those polys into polysoups of your different material types, you can drop that prim attribute count down to only as many as you need for your different materials.  beyond that, i don't think there's much difference between soups and just regular polys.

 

packed prims are a whole different animal.  you can pack any other prim into a packed prim, but that geo is effectively frozen -- can't be changed at all unless you unpack it first.  well, aside from moving it around the scene or copying it.

 

in addition, there's a "disk packed prim" which is a link to a file on disk.  alembic also has alembic prims which are links to the alembic file on disk.

 

 

i would imagine you can use packed prims in instancefiles just fine, but there'd be no real advantage i can think of.

[Mzigaib]

So, no way do reduce the time generating those "preps"? 

 

Like I said I am going to try to use the "instacefile" attribute to see if there is any practical advantage.

[sebkaine]

@Miles + @Alexey Thanks a million for all those very helpful answer guys !

 

This really help to understand the mantra logic in depth !

 

I owe you a beer ... (or a vodka) 

 

Cheers 

 

E

 

ps : one mystery remain intact , bgeo.sc or bgeo ?

as you have to uncompress them at rendertime and as our goal is to get as close as clarisse with best interactivity , i'd say bgeo ?

Edited January 26, 2016 by sebkaine

[goldleaf]

Packed Prims + Material Stylesheets are awesome. If you set a File SOP's Load parm to Packed Disk Primitive, the IFDs are super small/fast to generate, since it just points at the PackedPrim on-disk. Alembic can work almost exactly the same.

 

That will show a Bounding Box by default (you can change how it's displayed), and now you can copy them to points, particles, resize by pscale, whatever, and the IFDs are still lightweight; and if you use a Material Stylesheet, you can modulate the shader with attributes.

[fathom]

so i think i must take back something i said earlier.  it appears that subd surfaces do NOT break instancing, which is nice.  not sure how they manage that -- maybe all get gets the same subdivision based on the closest instance?

[sekow]

chimming in on compressed bgeo or raw data.

if interactivity is key, than I would reckon having that data local on a ssd, and without any compression.

but remember that mantra does not need to load that geo for every iteration in ipr.

Ive never used clarisse but I am very sure that it need some time to load that alembic data on frame changes too.

 

I am usually preparing my scene before going to lighting and shading. I try to delayed load everything at this stage, which will result in longer prep times, and 

some pre caching work when I have to change geometry, but thats something hou is very forgiving as we all know and with the benefit of very short export times.

(and small ifds too)