Rendering a very large set of contiguous geometry

[Adrian]

I'm trying to render a very large collection of points connected by lines (twin-vertex open polygons) which in Mantra unfortunately exceeds the amount of memory I have available on my system. This is an attempt at data visualization of node connections, so correct representation of the data matters to me, even if the viewer couldn't possibly notice. It's similar to the image below, just orders of magnitude larger.

 

 

I made sure to get rid of any geometry attributes not needed for rendering.

 

Since many points are shared by different lines (i.e. the whole thing is interconnected), I don't see how I could e.g. partition the network into cubes of space to help memory management. To my understanding, none of the nifty tools like packed primitives can help me when dealing with one contiguous piece of geometry.

 

I played around a bit with mantra's KD-Tree settings but it's not enough to save me. If it just isn't doable, I'll have to scale back my ambitions, but right now I'm still hoping to find a way.

 

Any ideas/suggestions welcome! Thanks.

Edited January 10, 2016 by Adrian

[mestela]

Typing out loud, you could use the convert-line sop to make it one prim per line, might give mantra a fighting chance to cull geo. Don't need pbr? Try micropoly render and simple shaders. OpenGL rop? Cull by camera frustum? Render in chunks and recombine in comp? Depth/deep map shadows rather than ray traced? You're rendering curves right, not doing polywires?

Edited January 10, 2016 by mestela

[lukeiamyourfather]

I'd try rendering it on a Linux machine with an SSD and a shit ton of swap. It might take a week but it should get it done. Out of curiosity how much memory do you have now that the render is exceeding?

[Adrian]

Typing out loud, you could use the convert-line sop to make it one prim per line, might give mantra a fighting chance to cull geo. Don't need pbr? Try micropoly render and simple shaders. OpenGL rop? Cull by camera frustum? Render in chunks and recombine in comp? Depth/deep map shadows rather than ray traced? You're rendering curves right, not doing polywires?

 

The convert-line SOP looks interesting, but it appears that it does nothing to a structure such as the one in my screenshot. I currently create the "network" by first placing points and then running a Python SOP to create polygon primitives connecting various points (using external data). In my thinking, the shared points are what makes the structure indivisible. I was already using the micropoly renderer and a flat color shader, but I haven't tried rendering the screen in multiple segments, that might actually work! Thanks!

 

I'd try rendering it on a Linux machine with an SSD and a shit ton of swap. It might take a week but it should get it done. Out of curiosity how much memory do you have now that the render is exceeding?

 

I'm always so worried about drive longevity, a bad habit I guess. I'll consider it as a last resort, if only to see how the end result would look. I have 64GB RAM, which is the maximum supported by my CPU/Mobo, and the geometry data itself fits into that - it's the renderer's acceleration structures which make this balloon beyond capacity I think.

[dennis.albus]

How many points / primitives do you actually have? It seems a little bit weird that Houdini can handle all that geometry in the viewport but mantra crashes. Maybe it is because you already filled up your ram with the SOP cache of the geo and there's only little space left for mantra. Try to create an ifd and render that directly with mantra. If that's not possible because you are using a non commercial version please save your geo as a bgeo and use a delayed load procedural so you won't fill up your ram as soon as the geometry is generated in SOPs.

 

Did you actually try to render it as a more efficient data type like polysoup or packed primitives?

 

First I would try to split the points so you can have each segment as a separate primitive. Then you can do your own space partitioning based on position. Create a unique attribute for each partition which you can use to create separate polysoups / packed primitives. Try if that already helps rendering. If that does not help you can save those pieces as alembics / bgeos and load them with an alembic procedural / delayed load procedural so mantra can load them on demand.

 

Just some thoughts,

Dennis

Edited January 12, 2016 by dennis.albus

[Adrian]

Thanks for the suggestions Dennis! I only display the geometry as a point cloud in the viewport - the connecting primitives outnumber them about 60:1. I tried the PolySoup SOP but like prior attempts, mantra seems to find no way around having to see all geometry at once (at some point). I'll try to find a way to create a new point each time my PythonSOP tries to make a new connection. If I retain the original point cloud and hide the newly created ones, I imagine I can still work in the viewport; the locations stay the same after all.

[dennis.albus]

Did you check your memory consumption when looking at the geometry? Do you have redundant connections that could be removed (a connection from point A to B and a connection from B to A at the same time)?

 

If you don't mind sharing an example file we can find a solution for sure