Creating a rawvox format importer

[run]

Hi,

Jason Iverson over at the SideFX Houdini forums suggested I cross-post this here.

Since he thought this might be achieved through a PythonSOP I've posted it in this section.

If it would be more appropriate in a different section please let me know.

There is a voxel format called .rawvox that can exported from 3D-Coat.

Here's a description of the format from Andrew Shpagin the creator of 3D-Coat

___________________________________________________________

raw format is very simple -

just values in 3D-grid that are set as bytes, words or float

byte 0..255, 128 means surface

word 0..65535, 32768 means surface

float 0..1, 0.5 - surface

rawvox format

int Magic 'RVOX'

int SizeX

int SizeY

int SizeZ

int bits_per_voxel - 8,16,32, 32 means float values

...raw voxel data...

___________________________________________________________

I would like to create a Houdini loader for this format can anyone point

me in the right direction on what I'd need to learn to do this.

This isn't an ASCII file, so I assume it can't just be "read" via simple scripting

(Update: But, after what Jason said maybe you can.)

My knowledge of, and ability with, Houdini is modest at best, but I have

done a fair amount of (self-taught) scripting in other apps so I'm not

opposed to getting my hands dirty and digging a little deeper into the

scripting/programming side of things.

Thanks,

-Jeff

[Jason]

Thanks for coming over:)

Do you have a (small) example of a .rawvox file we can test with?

FYI, I can see using HOM's hou.Volume in a PythonSOP to do this.

Hi,

Jason Iverson over at the SideFX Houdini forums suggested I cross-post this here.

Since he thought this might be achieved through a PythonSOP I've posted it in this section.

If it would be more appropriate in a different section please let me know.

There is a voxel format called .rawvox that can exported from 3D-Coat.

Here's a description of the format from Andrew Shpagin the creator of 3D-Coat

___________________________________________________________

raw format is very simple -

just values in 3D-grid that are set as bytes, words or float

byte 0..255, 128 means surface

word 0..65535, 32768 means surface

float 0..1, 0.5 - surface

rawvox format

int Magic 'RVOX'

int SizeX

int SizeY

int SizeZ

int bits_per_voxel - 8,16,32, 32 means float values

...raw voxel data...

___________________________________________________________

I would like to create a Houdini loader for this format can anyone point

me in the right direction on what I'd need to learn to do this.

This isn't an ASCII file, so I assume it can't just be "read" via simple scripting

(Update: But, after what Jason said maybe you can.)

My knowledge of, and ability with, Houdini is modest at best, but I have

done a fair amount of (self-taught) scripting in other apps so I'm not

opposed to getting my hands dirty and digging a little deeper into the

scripting/programming side of things.

Thanks,

-Jeff

[run]

Well, it's nice to be here:)

I've attached a rar with 2 samples of a small sphere

The first is encoded as 16-bit, the second as 32-bit float.

I've also included a jpeg screen cap of the export options,

but it pretty much just repeats the previous info.

Thanks,

-Jeff

Thanks for coming over:)

Do you have a (small) example of a .rawvox file we can test with?

FYI, I can see using HOM's hou.Volume in a PythonSOP to do this.

3dc_rawvox.rar

[eloop]

The python solution is probably the easiest to get going but it's not going to be very fast.

This simple SOP I posted a while ago may be of use - SideFX forums

Also see $HT/sample/standalone/geo2voxel.C

-Drew

Well, it's nice to be here:)

I've attached a rar with 2 samples of a small sphere

The first is encoded as 16-bit, the second as 32-bit float.

I've also included a jpeg screen cap of the export options,

but it pretty much just repeats the previous info.

Thanks,

-Jeff

[eloop]

Actually I hadn't noticed the hou.Volume module before, how long has that been around ? The setAllVoxels(self, values) method is probably going to be reasonably efficient, definitely worth trying before diving into the HDK.

-Drew

The python solution is probably the easiest to get going but it's not going to be very fast.

This simple SOP I posted a while ago may be of use - SideFX forums

Also see $HT/sample/standalone/geo2voxel.C

-Drew

[Wolfwood]

One question

First how are the voxels stored in the rawvox file?

x-axis then y-axis then z-axis?

That said....

Here is a crappy Python implemenation to verify that the format is correct. (Yea Little Endian!) If this rawvox format is something that you'll be using a lot at much higher resolutions it should be implemented as an HDK op.

Also.... I haven't had any coffee yet so the code is icky and brute force.

Oh I assumed a Fog Volume was what you wanted. You can easily convert your volume to other types via the Primitive SOP and/or IsoOffset.

Edited September 26, 2009 by Wolfwood

[Wolfwood]

Sorry for the double post... I don't seem to be able to edit attachements once a post is completed.

test.hip

[run]

Wow, Thanks! Never underestimate curious minded people

It doesn't work in Houdini Apprentice HD 9.5.134.

I figured I'd have to upgrade when Jason mentioned using

hou.Volume which wasn't listed in the installed docs.

But after looking at the .hip in a text editor I had

hopes since hou.Volume wasn't used.

It throws this error on loading

Skipping unrecognized parameter "name"

[/obj/geo1/isooffset1]

I'll hopefully have time to get a more current ApprenticeHD

license for the new computer up and running within the next

couple of days. Didn't want to upgrade on the old machine

and licenses for HAHD can't be transferred.

I've attached a rawvox cone_box object which if loaded

should demonstrate if the assumptions about order are

correct. The cone points up on Y and the box runs along Z

I'll e-mail the programmer and ask him to confirm things

too.

One of the things 3D-Coat is, is a very nice Volumetric

sculpting app, which is the reason I've been thinking about

getting it's output straight into Houdini as a volume and

letting Houdini handle the visualization as,... well, pretty

much anything.

Thanks again,

Jeff

cone_box.rar

[Wolfwood]

Oh yea. You'll need Houdini 10 for this. That's when they added the Volume support for HOM.

Good news, cone points up and the box goes along the Z-axis. So all the voxel data can be dumped directly into setVoxels() without any processing.

[run]

I've got Houdini10 running now. Digging into the Houdini and Python docs I've gotten a fairly clear understanding of what's happening with the loader. The Master Class videos for Python and tool building at sidefx.com have been a good source of info too.

The Python "struct" module makes getting the data out of the rawvox file a lot less ugly than I thought it would be.

I do have some things that I haven't been able to track down yet

Is there a way to edit an embedded OTL in Houdini?

UPDATE:Found it. Windows>Operator Type Manager>Locate the embedded OTL in list> right-click>Choose "Type Properties" And I feel smart and stupid at the same time

Is changing the bounding box volume with the optional bounding box feature of "createVolume" preferred over adjusting the scale of the primitive node you pipe the loader into?

Good news, cone points up and the box goes along the Z-axis. So all the voxel data can be dumped directly into setVoxels() without any processing.

Unless I'm missing something, all the steps you created seem necessary. You meant further processing, right?

Thanks,

-Jeff

Edited September 29, 2009 by run

[Wolfwood]

Is changing the bounding box volume with the optional bounding box feature of "createVolume" preferred over adjusting the scale of the primitive node you pipe the loader into?

Either method achieves the same thing. You could even use a Transform SOP. So in that sense, I don't think there is really a preferred method.

Unless I'm missing something, all the steps you created seem necessary. You meant further processing, right?

Correct. I just meant if the voxels were aligned differently additional sorting would have been needed. But since that isn't the case the voxel data can be dumped directly into the Volume.

[run]

Here's a version of that will correctly size the aspect ratio

of the rawvox's bounding box and scale it to fit conveniently

within Houdini's default view

It's a .hipnc so it will only work with a non-commercial Houdini.

But the little bit of code I added/changed can be copied and used to

update JIm's original otl embedded in the .hip he uploaded.

bbx = ((voxSizeX*0.01)/2)
bby = ((voxSizeY*0.01)/2)
bbz = ((voxSizeZ*0.01)/2)

bb = hou.BoundingBox(bbx*-1,bby*-1,bbz*-1,bbx,bby,bbz)

vol = geo.createVolume(voxSizeX, voxSizeY, voxSizeZ,bb)

-Jeff

rawvox_test_05.hipnc

[pbowmar]

Very cool! And as it turns out, immediately useful for the modeller who is evaluating 3DC for production at our studio.

Do you guys mind if I make this a commercial OTL and put it on the Exchange? Full credit to you guys of course. Do you want your forum or real names used in the source?

Cheers,

Peter B

[pbowmar]

So, I'm getting Python Memory Error trying to use a non-trivial volume. Is Python only 32bit even on my 64bit machine? I can't remember if I've run across this before.

I might have to use the HDK, not that I remember how

Cheers,

Peter B

[eloop]

Under 64 bit Linux -

$ file `which hython-bin`
/opt/hfs10.0.407/bin/hython-bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped

-Drew

So, I'm getting Python Memory Error trying to use a non-trivial volume. Is Python only 32bit even on my 64bit machine? I can't remember if I've run across this before.

I might have to use the HDK, not that I remember how

Cheers,

Peter B

[pbowmar]

Ah I see it's the struct.py. It needs to use double instead of float, but my python/coding isn't strong enough.

Any ideas?

Cheers,

Peter B

[eloop]

You can change the "f"s in format string (ie unpack(fmt,...)) to "d"'s to unpack doubles.

-Drew

Ah I see it's the struct.py. It needs to use double instead of float, but my python/coding isn't strong enough.

Any ideas?

Cheers,

Peter B

[pbowmar]

Hi Drew,

Yeah, I tried that (header of struct.py is very helpful) but no love on the massive (640x640x640) volume I'm trying to load. Houdini doesn't run out of RAM (though as it turns out, it would on my tiny machine) but struct fails.

I think I'm going to have a stab at HDK-ifying this, see if I can speed it up too.

Cheers,

Peter B

[eloop]

That's strange, I'll have to try that out when I'm back at my work machine.

You should be able to adapt the simple HDK SOP that I posted earlier in this thread to read doubles. I've since built a more complete one as well that currently reads byte, short int and floats, it could easily be adapted to read doubles as well, ping me if you need it.

-Drew

Hi Drew,

Yeah, I tried that (header of struct.py is very helpful) but no love on the massive (640x640x640) volume I'm trying to load. Houdini doesn't run out of RAM (though as it turns out, it would on my tiny machine) but struct fails.

I think I'm going to have a stab at HDK-ifying this, see if I can speed it up too.

Cheers,

Peter B

[dulo]

I have the same problem.

Get a out of memory exception when loading bigger volumes. ( got 64bit houdini and 50 gig of ram. sould be no problem)

Has anybody found a solution for the problem ?

thx dulo