Prman Pc + Brickmap Vs Mantra Pc + I3d

[pbowmar]

Hi,

So, I'm out of touch with PRman, having used Mantra pretty much exclusively for the last few years. I have some naive Pointcloud/Brickmap/I3D questions.

PC: In Mantra, writing them is easy with pcwrite(). In PRman you use what, bake3d()?

PC: In Mantra, the PC is unstructured (i.e. no tiling/mipmapping) so when reading one Mantra has to load the whole thing into memory which results in potentially severe limitations. In PRman, what's the situation?

Brickmap: In PRman can you write a Brickmap directly or do you still have to write the PC then convert after with the brickmake (or whatever it's called) utility?

Brickmap: Does I3D in Houdini/Mantra has the same capabilities as a PRman brickmap? If not, what's different other than they are different file formats?

Brickmap: Is there any advantage other than the tiling, mipmapping and lack of resolution restrictions to a brickmap/I3D over a PC?

I3D: Is there a PC to I3D utility available like brickmake or will I have to write one in the HDK if I need one?

Brickmap: How fast is creating a PRman Brickmap? I3D writing is notoriously slow, does brickmap creation suffer this same problem?

Cheers,

Peter B

[kodiak]

Hi Peter,

I'm pretty much in the same boat, just haven't been that long to be using prman

PC: In Mantra, writing them is easy with pcwrite(). In PRman you use what, bake3d()?

Yes.

PC: In Mantra, the PC is unstructured (i.e. no tiling/mipmapping) so when reading one Mantra has to load the whole thing into memory which results in potentially severe limitations. In PRman, what's the situation?

It's pretty much a one to one match between pc-pc and i3d-brickmap. Prman's PC has the same limitation, reading the whole thing to memory at the same time, while the brickmaps are sampled from disk as needed.

Brickmap: In PRman can you write a Brickmap directly or do you still have to write the PC then convert after with the brickmake (or whatever it's called) utility?

You still need to convert, as there is no brickmap API as of now (Prman 13.5).

Brickmap: Does I3D in Houdini/Mantra has the same capabilities as a PRman brickmap? If not, what's different other than they are different file formats?

I would say yes. I'm sure there are minor differences, but the basic idea and feature set seems to be the same.

Brickmap: Is there any advantage other than the tiling, mipmapping and lack of resolution restrictions to a brickmap/I3D over a PC?

Not really.

I3D: Is there a PC to I3D utility available like brickmake or will I have to write one in the HDK if I need one?

Doesn't the I3Dconvert support that? I'm not quite sure, but seem to remember trying something.

Brickmap: How fast is creating a PRman Brickmap? I3D writing is notoriously slow, does brickmap creation suffer this same problem?

Converting from PC is relatively fast.

I would also add one more question: what's the difference between an I3D and a TBF in terms of feature set and performance (memory mapping..etc)? And what's the reason for the two to coexist (TBF seems to be some kind of leftover)?

[crunch]

I would also add one more question: what's the difference between an I3D and a TBF in terms of feature set and performance (memory mapping..etc)? And what's the reason for the two to coexist (TBF seems to be some kind of leftover)?

TBF is a generic tiled block format used for many purposes in Houdini.

2D Textures (.rat files) use TBF underneath to store tiles. This allows the data to be paged from disk as required.

3D Textures (.i3d files) use TBF underneath to store tiles.

The irradiance caches in mantra use TBF underneath, as do photon maps.

The underlying TBF library shares the in core memory cache between all of these different file formats.

You can have image files in different formats (i.e. JPEG, TIFF, etc.), and reference them as textures in Houdini. However, these go outside the TBF caching process and work differently.

Just like you can reference TIFF files, you can either reference point clouds as TBF files or as .bgeos. If you reference point clouds as TBF files, some of the data can be shared in the same memory pool. Though, since the data is stored un-ordered for point clouds, some channels have to be loaded in their entirety for queries. However, non-query channels may possibly be tiled and loaded on demand.

To convert data to TBF:

	# Convert 2D raster to TBF
	% icp foo.pic foo.rat

	#Convert point cloud
	% i3dconvert -c foo.bgeo foo.pc

Hope this helps,

[kodiak]

Makes sense, thank you.

[AdamJ]

This is all dependent on your hardware but on a current day dual core.. a fairly high res ptc file (12mil+) takes about 10min to convert

with brickmake. Depending on a shot I found that the ptc version would go out of memory but the brick version would go through with

<1GB memory usage. Also the speed with ptc is much slower. Same shot using ptc files would take 2h+ where as the brick version would

do it in 30min.

@kodiak: So if I save out my ptc file using pcwrite().. will i3dconvert the ptc files into a tbf format? ..or do I have to go ptc->i3d->tbf with some other utility?

thanks

Adam

Hi,

So, I'm out of touch with PRman, having used Mantra pretty much exclusively for the last few years. I have some naive Pointcloud/Brickmap/I3D questions.

PC: In Mantra, writing them is easy with pcwrite(). In PRman you use what, bake3d()?

PC: In Mantra, the PC is unstructured (i.e. no tiling/mipmapping) so when reading one Mantra has to load the whole thing into memory which results in potentially severe limitations. In PRman, what's the situation?

Brickmap: In PRman can you write a Brickmap directly or do you still have to write the PC then convert after with the brickmake (or whatever it's called) utility?

Brickmap: Does I3D in Houdini/Mantra has the same capabilities as a PRman brickmap? If not, what's different other than they are different file formats?

Brickmap: Is there any advantage other than the tiling, mipmapping and lack of resolution restrictions to a brickmap/I3D over a PC?

I3D: Is there a PC to I3D utility available like brickmake or will I have to write one in the HDK if I need one?

Brickmap: How fast is creating a PRman Brickmap? I3D writing is notoriously slow, does brickmap creation suffer this same problem?

Cheers,

Peter B

[kodiak]

It looks like (although I confused things with .bgeo->.pc conversion. That's what I tried before, and it's done using crunch's example code.

However, based on the i3dconvert help, it seems to be possible to go .bgeo->.i3d straight away

(This is not to say RTFM, just in case someone has missed the docs on the commandline util).

Usage: i3dconvert [-v] [-p|-t|-i] [-d x y z] file1 file2
Converts 3D texture file formats to geometry (and vice versa).
The -c, -i, -p, or -t or option can be used to generate a specific type of
3D texture image from the geometry file:
        -c      Point Cloud Texture map will be generated
        -p      Photon map will be generated
        -i      An irradiance map will be generated
        -t      .i3d texture file will be generated

When converting from geomtry to texture file and none of
the above options are specified, then the attribute set
on the geometry will be used to determine the output format.

When working with 3D texture files (.i3d)
  The geometry file generated will be a set of points with attributes
  defined for each channel of the i3d image.  When converting from
  a geometry file to a 3d texture, the points must be evenly
  spaced in a regular axis-aligned lattice for the program to work.
  Currently, only ASCII geometry files can be created, though
  it is possible to read any format of geometry files.

  The -d option allows you to specify the resolution of the target
  texture file when converting from a .geo file to a .i3d file
  If the -d option is not specified, the resolution of the image will be
  determined by analyzing the geometry coordinates.
  The -v option will cause verbose output

  Example:  i3dconvert cloud.i3d stdout.geo | gconvert stdin > cloud.bgeo
          i3dconvert cloud.bgeo cloud1.i3d

When working with photon maps
  When converting a photon map to geometry, the .geo file
  will contain a point for each photon in the photon map.
  Each point will have attributes for the associated photons.
  Converting .geo files to photon maps requires the correct set of photon
  attributes for the photon map to work properly.

When working with irradiance maps
  When converting an irradiance map to geometry, the .geo file
  will contain a point for each photon in the photon map.
  Each point will have attributes for the associated photons.
  Converting .geo files to photon maps requires the correct set of photon
  attributes for the irradiance map to work properly.

When making point cloud texture maps
  Point cloud textures are stored in the Tiled Block Format (TBF)
  Each channel in the point cloud has a prefix associated with it.  This
  prefix is used to differentiate between point and primitive
  attributes of the same name.  The prefix for each geometric element
  is specified by:
     -P point_prefix       default [pcloud:]
     -V vertex_prefix      default [vertex:]
     -R primitive_previx   default [primitive:]
     -D detail_prefix      default [detail:]
  At the current time, only point attributes can be stored in the TBF files

[AdamJ]

It looks like (although I confused things with .bgeo->.pc conversion. That's what I tried before, and it's done using crunch's example code.

However, based on the i3dconvert help, it seems to be possible to go .bgeo->.i3d straight away

(This is not to say RTFM, just in case someone has missed the docs on the commandline util).

cool thanks

[symek]

ok, I'm lost. Is that mean that I can save (resave) my point cloud texture as a TBF file to allow Mantra use tiled architecture or not? If I convert *.pc to *.i3d would I be able to use it as a point cloud?

thanks,

sy.

[crunch]

ok, I'm lost. Is that mean that I can save (resave) my point cloud texture as a TBF file to allow Mantra use tiled architecture or not? If I convert *.pc to *.i3d would I be able to use it as a point cloud?

thanks,

sy.

If I understand your question, no.

The underlying format for an I3D (3D Texture Map -- voxel array etc.) is the same as the underlying format for a Point Cloud (i.e. un-ordered bunch of points). However, they represent different data. Just like a 2D texture map represents different data than a .bgeo file.

i3dconvert can work on various different formats and convert between them. I'm pretty sure that to convert from a .bgeo directly to an .i3d file, the .bgeo has to be in the "right" format. That is, a lattice of points representing the voxels.

Converting an arbitrary un-ordered set of points to a 3d texture map is a different process.

I hope that doesn't confuse things more.

[old school]

That is why the Box SOP has a "Points" option: to generate a lattice cube of points suitable to generate an i3d file.

Generate the cube of points then add your attributes then generate a .bgeo then to i3d.

[symek]

If I understand your question, no.

The underlying format for an I3D (3D Texture Map -- voxel array etc.) is the same as the underlying format for a Point Cloud (i.e. un-ordered bunch of points). However, they represent different data. Just like a 2D texture map represents different data than a .bgeo file.

i3dconvert can work on various different formats and convert between them. I'm pretty sure that to convert from a .bgeo directly to an .i3d file, the .bgeo has to be in the "right" format. That is, a lattice of points representing the voxels.

Converting an arbitrary un-ordered set of points to a 3d texture map is a different process.

I hope that doesn't confuse things more.

If so then I don't understand this part:

Just like you can reference TIFF files, you can either reference point clouds as TBF files or as .bgeos. If you reference point clouds as TBF files, some of the data can be shared in the same memory pool. Though, since the data is stored un-ordered for point clouds, some channels have to be loaded in their entirety for queries. However, non-query channels may possibly be tiled and loaded on demand.

I understand the part about limitations but what does mean "to refer point cloud as TBF"? That we can think about it oc this way? If so, I got the point...

hmmm, I wonder.... It seems that photons routines in VEX handles photons in pretty similar way as pcopen() and company handles point clouds.

If one would convert pc to photons he could use it as TBF point clouds!

Edited March 5, 2008 by SYmek

[edward]

Really, I just wish the whole term "TBF" could just be removed from all discussion, anywhere. From a user's point of view, it doesn't matter. All this really means is that point clouds, rats, i3d's etc are tiled and can be loaded on demand from disk. "TBF" is an internal term to mean a generic file format that treats its data as tiled.

Is that mean that I can save (resave) my point cloud texture as a TBF file to allow Mantra use tiled architecture or not?

You can't convert to or from TBF. You can convert to/from .rat though, and as it was mentioned, .rat's are tiled, point cloud's are tiled.

[pbowmar]

Small correction here. Unless something has changed, a point cloud (unlike a RAT or I3D) must be fully loaded into memory the first time it is accessed which completely removes any tiling advantages. Hence my original question at about making an .i3d which does not ever have to be fully loaded into memory...

I still don't have an answer about converting .pc to .i3d I'm assuming then it's not natively possible. i3dconvert won't do it because i3dconvert requires a uniform box (as Oldschool points out) which a .pc most definitely is not.

Thanks to all for the interesting tidbits though.

Oh, technical question to SESI peeps: If you have 10 .pc files, each 1 gig in RAM usage when fully loaded, will Mantra (say) load the first one, build the (I forget, KD tree or something) info it needs, then flush it out before loading and "indexing" (or whatever) the subsequent ones? If so, you could have multiple .pc files that if in one file would run out of RAM space, right? Assuming you turn off the Preload option

Cheers,

Peter B

All this really means is that point clouds, rats, i3d's etc are tiled and can be loaded on demand from disk.

[David Gary]

Hey.

Mantra is getting close to be the best and most complete renderer in the industry ( with such a speed, volumetrics, PBR, and properties overrides, not to mention the procedurals) , so i think the more we know about Mantra specifics the more we can optimize our scenes and produce fantastic images. And tiling is a critical thing for large scenes.

But, reading this interesting post, i'm still a little confused...

The fact is, aplogies for throwing some fuel on the fire, it actually seems that point clouds ARE tiled :

the wiki says so (here) , the Houdini help as well ( if you go under the Geometry File Format ), besides this looks completely consistent with the way the pc handle operates ( who should i trust, or am i missing something)

So, my current opinion up to that point is that point clouds actually win on all fronts:

-size: lighter files, sparse distributions

-way to work in the shaders formulas to more or less replace raytracing (SSS or AOC classic approximations)

-way to store/bake shaders - irradiance caching

-more or less unified with photons paradigm

Up to the point where i actually wonder if they shouldn't be also the way to go for volumetrics ( i mean to replace i3d, not for procedural textures, but to replace brickmaps). i mean i don't know the i3d specifcs, but tiling is a good thing, multi-scale ( kd-tree like) is another even better, and a sparse point distribution even a third. Does i3d have all that?

If this is the case, then Point Clouds become the basis of almost all advanced rendering techniques, and this kind of unification is very satisfactory ( from photons to SSS) both theoretically and implementation-wise.

anyway, Mantra rocks!

[symek]

The fact is, aplogies for throwing some fuel on the fire, it actually seems that point clouds ARE tiled :

Perhaps TBF architecture doesn't prejudge if a file has or hasn't to be loaded fully into memory - actually this is what I understood from that thread.

Anyway, following F. Bacon principles I made a simply test - to experience something everyone seems to be aware of. I created a huge point cloud. First one which was 1.7GB killed (32bit Linux) both mantra and i3dconvert, so I tuned it down to ~300MB. Mantra loads +500MB for simple render with it.

Besides that Mantra most of the time does something with that texture without rendering (building accel. structure?). So I thought I could help converting PC explicitly into TBF.

That conversion gave me 600MB file, which loads 1.2GB during rendering on single thread in Mantra.

Not sure if this concludes anything though.

cheers,

sy.

Edited March 28, 2008 by SYmek