How to render very high resolution volumes without crashing?

[magneto]

Hi,

 

If I have a volume with 0.01 voxel size, and I want to render this volume using 0.001 voxel size where my modifications (Volume VOP) will also use 0.001 voxel size at render time, how can I achieve this without crashing Houdini?

 

I copy pasted my network and set the render flag to a version where the volume uses 0.001 voxel size, which didn't trigger cooking inside Houdini as expected. But when I fired Mantra, I saw my memory filling and crashing Houdini at the Volume VOP cook after a while. I thought at render time somehow there is a way to handle finer resolutions without crashing?

 

Or do I have to turn my Volume VOP into a shader? I never did this but somehow I feel this way Mantra can efficiently modify my volume without using max memory? Because I start with an empty volume that doesn't take much time or memory. But once I start modifying it using a Volume VOP, it gets out of control.

 

 

Thanks

[Guest tar]

Can't answer the Volume Vop into a shader but memory requiments calculation should be fairly simply - at a guess 1 voxel holds 3 32bit or 64bit floating point values, then just scale that up to the number of fields and number of voxels.

 

Edit: some testing I couldn't nail a consistent calculation but I did hit a 2 billion tile limit

Edited March 8, 2014 by tar

[ikarus]

If you cant cook a volume that hi res interactively why would you be able to when IFD is generated?

 

You can convert the volume vops contents into cvex shader and use vex volume procedural to run the cvex shader on your volume data

Edited March 9, 2014 by ikarus

[magneto]

I thought Mantra could because certain things can be handled at render time more efficiently without allocating them full into the memory, like rendering instances, no? I thought you might not be able to have more than your memory can handle in the viewport, but at render time, Mantra can efficiently do this without running out of memory?

 

Also that 2B limit is because of Int32 limit I reckon.

[ikarus]

In interactive sessions you use more memory by having every sop cached automatically (you can walk back up your chain without recooking). Renders have no need to cache sops they only care about the output (render flagged node) afaik.

[edward]

Edit: some testing I couldn't nail a consistent calculation but I did hit a 2 billion tile limit

 

To put some perspective on this, Houdini's native volumes use 16^3=2^12=4096 voxels per tiles. So 2^31*2^12=2^43 is over a trillion voxels. Of course in practice this will be less if all your tiles are constant valued but it will still be an enormous amount that probably exceeds the amount of physical memory on an average person's computer.

[magneto]

In interactive sessions you use more memory by having every sop cached automatically (you can walk back up your chain without recooking). Renders have no need to cache sops they only care about the output (render flagged node) afaik.

 

So should I fire the render without launching Houdini? Because when I fired Mantra, even though my render flagged network never cooked in Houdini, Mantra started cooking them, and using an empty volume, it takes very little time, but the memory requirements became too much when my Volume VOP started changing the values uniquely.

 

To put some perspective on this, Houdini's native volumes use 16^3=4096 voxels per tiles. So 2^31*4096 is over a trillion voxels. Of course in practice this will be less if all your tiles are constant valued but it will still be an enormous amount that probably exceeds the amount of physical memory on an average person's computer.

 

How much memory would that require in GBs? I assume the reason you used Int32 for the capacity of a volume is because with Int64, none of the standard C++ operations are defined properly, i.e. for loops increments, array index operators, etc that it would be a pain to use?

Edited March 9, 2014 by magneto

[edward]

I thought Mantra could because certain things can be handled at render time more efficiently without allocating them full into the memory, like rendering instances, no? I thought you might not be able to have more than your memory can handle in the viewport, but at render time, Mantra can efficiently do this without running out of memory?

 

This is true only if:

- You could cook an entire chain of SOPs for only a subset of the geometry, and

- Mantra could run this magical SOP network at render time

 

When you generate an IFD (always required for Mantra), it fully cooks the geometry and embeds it into the IFD for Mantra to render. What you really want here is the VEX Volume Procedural as Ikarus mentioned. This is similar to the idea of doing render time displacements on a per sample level, or doing render time adaptive subdivision.

 

 

I assume the reason you used Int32 for the capacity of a volume is because with Int64, none of the standard C++ operations are defined properly, i.e. for loops increments, array index operators, etc that it would be a pain to use?

 

C++ handles int64 fine. std::vector::size() returns a uint64 on 64-bit platforms, and so does UT_Array::entries() recently as well. There are a few reasons why int32 is still being used:

- Backwards compatibility for file formats

- Changing file formats is a pain

- Lots and lots of code, and some of it possibly going back 20 years ago

- Not all limits need to be 2^63. For example, 2^43 voxels at 4 bytes per single precision float is 2^43*2^2=2^45, or 32 terabytes.

- int32 is half the storage of int64

- Less memory might mean faster performance because you can do more work by fitting more pieces into the CPU cache

[magneto]

Thanks alot Edward, I owe you one I understand it better now.

[edward]

PS. To be clear, the voxel count for Houdini's volumes is *not* limited to a 32-bit int. The 32-bit int is the limit on the number of *tiles* a volume can currently have, where a tile represents 4096 voxels.

[magneto]

PS. To be clear, the voxel count for Houdini's volumes is *not* limited to a 32-bit int. The 32-bit int is the limit on the number of *tiles* a volume can currently have, where a tile represents 4096 voxels.

 

So the number of tiles depend on the physical size of the volume? If you have a very fine resolution volume that's 0.1x0.1x0.1 in physical size, you would run into this issue?

[edward]

PPS. For those that like cool marketing terms, that's a teravoxel core

[edward]

So the number of tiles depend on the physical size of the volume? If you have a very fine resolution volume that's 0.1x0.1x0.1 in physical size, you would run into this issue?

 

This only depends on your volume resolution. Please read my post above as I've edited after you quoted me. We have a 2^31 limit on tiles, so take a rough cube root of that, which is like 1290 x 1290 x 1290 or tiles in each dimension. So 1290 * 16 = 20,640 voxels. So your theoretical limit for a cubic volume is 20640 x 20640 x 20640 volume.

 

Again, this is independent of "physical size". You can put 1 voxel in 0.1 x 0.1 x 0.1 physical size, or a trillion voxels in that same space.

[magneto]

Thanks alot Edward, this makes more sense. I can now see that's an extremely high resolution volume.

[Guest tar]

Very cool stuff! 

 

The VolumeSop node has 'Use 16bit Float' under Properties, the help says computation is done at 32 bits but memory consumption is cut in half - is that only for the disk space or in ram too?

 

Edit: some testing and using the VolumeCompressSop seems to do both ram and disk based savings. How can we use 16bit processing to save ram too?

 

.hip attached

 

Ram usage per volume node

130KB 16 bit

495KB 32 bit

 

Disk usage bgeo output

502188 32BitVolumes.bgeo

128543 16BitVolumes.bgeo

 

 

 

 

 

Volume16_32Tests.hip

Edited March 9, 2014 by tar

[magneto]

PPS. For those that like cool marketing terms, that's a teravoxel core

 

Next up terapolygon core