mantra memory explosion with packed primitives

[Activate]

I have a large ground (grid 120x120) on which I grew a meadow, e.g. 6 packed primitives copied to 8 million points. To make this more? efficient, I fractured the ground into 20 fragments and read each fragment as a packed disk primitive - to avoid loading when the camera does not see it.

When I render an individual fragment, mantra takes about 1.4GB of memory. When I render all of them together, it's around 30GB. According to SideFX, each instance of a packed primitive should take only 400 bytes. That would translate to 150M or 3GB respectively.

Can somebody here help me understand what's going on?

If you are interested, here is the scene file: https://drive.google.com/file/d/0B3SRWSd2MwvPVWNqNlZxazlPN0U/view?usp=sharing (Warning, that's about 600MB)

Ps: The grass code comes from: http://blog.schdbr.de/houdini-grass/

[Guest tar]

 I quickly tested packed disk prims on test geo and it worked, and testing it on your supplied geo didn't. i.e. copySoping it. Just work backwards to see where memory doesn't jump and send it into SideFx as a bug.

[Activate]

Ignoring the packed disked fragments and going much simpler, here is another test:

Box -> Points from Volume - Copy SOP packed primitive

~10k points -> 0.35GB Mantra usage

~100k points -> 0.6GB Mantra usage (2900 bytes per packed primitive - by delta)

~1M points -> 2.45GB Mantra usage (2100 bytes per packed primitive - by delta)

~10M points -> ~21GB Mantra usage (2060 bytes per packed primitive - by delta)

That seems to be 5 times larger than what the SideFX documentation claims: 400 bytes per packed primitive (http://www.sidefx.com/docs/houdini/model/packed)

[Skybar]

Did the same test as you and I'm noticing the same thing. Not sure how it is supposed to work though. However if you middle-click on that Copy SOP the memory reported there is a lot closer to what SideFX says in the docs, but when rendering it's about 5x as much as you say.

[Guest tar]

Just make sure to send in your results to SideFx - the devs working on this part are very very good. My theory is the acceleration structure is using up the memory but there isn't any statistics to verify afaik.

My simple tests showed unpacked at 800Mb vs 280Mb, unpacked vs packed. 

[lisux]

Hey Niels.

This issue with Mantra instances has been our major issue in the last two years in  every show using Mantra as the main renderer.

Almost two years ago we provided detailed tests to SESI showing the issue, one of them similar to your cube test.

I managed to improve instancing performance doing my own procedural, but there is definitely something not working well in Mantra instances.

We conclude that an object representation inside Mantra, all the data include in any rendered object/instance, apart from the geometry itself, is too heavy. It takes a lot to create multiple instances of the same object, and it takes too much memory. This obviously explodes in real production cases. Some work has been done, i got some emails from SESI about some bugs fixed, but the main issue is still there.

In their last visit to London we showed them again the issue, when compared with clarisse, which apart from many other problems, is pretty good at instancing, the difference is clear.

I hope this time they can decide to tackle this issue, which for me to be honest is the biggest issue in Mantra at the moment.

i haven't tested 16 for rendering yet, so let's see is Santa bring us something amazing this next months

[Activate]

Thank you all for confirming this. I will stop banging my head against the wall.

For my particular experiments, there are likely memory optimizations I might be able to do, e.g. for the forest cull leaves that are facing away from the camera, etc, but the lack of scalability of packed primitives is certainly disappointing.