inline cpp pc2 exporter

[glassman3d]

Hi there

 

I am building a pc2 writer sop to export point cache data from Houdini

 

I have a working python solution but it can be a bit slow (code has been hacked from the various examples on odforce ect) see below:

import sys, struct

def doCache():

    sf = hou.evalParm("startframe")
    ef = hou.evalParm("endframe")
    sr = 1
    ns = (ef - sf) + 1
    geo = hou.pwd().geometry()
    points = geo.points()
    np = len(points)
   
    pc2File = open(hou.evalParm("file"), "wb")
    geo = hou.pwd().geometry()

    # Write header
    headerFormat='<12siiffi'
    headerStr = struct.pack(headerFormat, "POINTCACHE2\0", 1, np, sf, sr, ns)
    pc2File.write(headerStr)
   
    # iterate points
    def writePP(p,f):
        hou.setFrame(f)
        curp = p.position()
        curps = pc2File.write( struct.pack('<fff', float(curp[0]), float(-curp[2]), float(curp[1]) ) )

    a = [ writePP(p,f) for f in xrange(sf, sf+ns, sr) for p in geo.points() ]

    # close file
    pc2File.flush()
    pc2File.close()

Due to slow speeds I am attempting to implement this using inlinecpp to speed up the execution time.

 

My attempt can be found below, (non-functional at the mo). I am a HDK newb so there are probably lots of 

very simple mistakes in there. Would anyone with HDK experience be able to give me a few pointers in where I am going wrong?

import sys
import struct
import inlinecpp


writePts = inlinecpp.createLibrary(
        name="cpp_string_library",
        includes="#include <GU/GU_Detail.h>, #include <FS/FS_Writer.h>, #include <UT/UT_Vector3.h>, #include <HOM/HOM_Module.h>",
        function_sources=[


        """void writePC2(GU_Detail *gdp,
                         const char *filename, 
                         const char *pc2,
                         int *numPoints,
                         float *start,
                         float *samplerate,
                         int *numSamples )
{ 
        // open the file for writing
        FS_Writer fs(filename);
        ostream *file  = fs.getStream();
        
        // write header
        file << pc2;
        file << int(1);
        file << numPoints;
        file << start;
        file << samplerate;
        file << numSamples;


        // iterate through frames through points
        
        for ( float i=start;  i<end;  ++i)
        {
            HOM_Module::setFrame( double(i) )
            
            //iterate through the points


            GA_Offset ptoff;
            GA_FOR_ALL_PTOFF(gdp, ptoff)
            {  
               UT_Vector3 pos = gdp->getPos3(ptoff);
               file << pos.x << pos.y << pos.z;
            }
    
        }


        // close the file
        file.close()
}
"""])


def doCache():
    sf = hou.evalParm("startframe")
    ef = hou.evalParm("endframe")
    sr = 1
    ns = (ef - sf) + 1
    geo = hou.pwd().geometry()
    filename = hou.evalParm("file")
    pc2 =   "POINTCACHE2\0"  


    writePts.writePC2(geo,
                      filename,
                      pc2,
                      len(geo.points),
                      sf,
                      sr,
                      ns)

Much obliged!!!

Sam Swift-Glasman

[symek]

I know this is not what you're asking for, but I can hand you some code to shorten your pre-alembic pain. Written carelessly as a last resort for XSI->Houdini pipeline, was in use permanently few years. But at least I'm sure it works. 

 

 

btw it compiles on H13, but lots' of warnings remain as code is outdated.

SOP_PointCache.zip

Edited February 19, 2014 by symek

[glassman3d]

Hi Symek

 

thanks for that code!

it'll come in very handy when I get to the reader, and  I will go through it to see if there is anything I can use on the way out of Houdini as well!

 

we were using alembic but the exocortex implementation in softimage is quite buggy as a point cache

 

best

Sam

Edited February 19, 2014 by glassman3d

[symek]

This is what I've heard about this plugin and wasn't even considering buying it. Alembic pipe from and to Maya is such a pleasure to work with after pc2...

 

 As to your code, I can't be sure since I rarely use inlinecpp, but meanwhile someone more educated watch this thread, I would say taht I hardly believe you can cook PythonSOP in different frame than current one. I'm actually not even sure you can cook any SOP like this.

 

If you don't want to iterate over frames in Python, AFAIK you need to set context to different time, then force cook node, copy points and such. Something like (untested):

other_frame_context = context;

for ( float i=start;  i<end;  ++i)
{
           float time = OPgetDirector()->getChannelManger()->getTime(i);
           other_frame_context.setTime(time);
           if(lockInput(0, other_frame_contex) >= UT_ERROR_ABORD)
            {
               unlockInput(0);
               return error();
            }

           duplicatePointSource(other_frame_context);
            
        /// and the rest the same... ////

            //iterate through the points
            GA_Offset ptoff;
            GA_FOR_ALL_PTOFF(gdp, ptoff)
            {  
               UT_Vector3 pos = gdp->getPos3(ptoff);
               file << pos.x << pos.y << pos.z;
            }
            unlockInput(0);
    }

This is more or less how this could be done in ordinary SOP, but not sure about PythonSOP...

Edited February 19, 2014 by symek

[glassman3d]

Hi Symek

 

Every package besides Softimage seems to have implemented Alembic in a robust and sensible fashion to my immense frustration, even renderers like Arnold support it natively! 

 

thanks again for the code, one more piece to the puzzle!

 

I should have specified, this is only a sop in the sense that it lives in sops - more of a 'soprop' like the filecache node for example. 

My code lives in a callback on an export button, 

 

My intention was to to iterate through the timeline, cook each frame, and append the point positions to the binary file

in python for example:

# iterate points
def writePP(p,f):
hou.setFrame(f)
curp = p.position()
    curps = pc2File.write( struct.pack('<fff', float(curp[0]), float(-curp[2]), float(curp[1]) ) )

a = [ writePP(p,f) for f in xrange(sf, sf+ns, sr) for p in geo.points() ]

could I ask is it necessary to unlock/lock the Input as below if I am in a callback context?

if(lockInput(0, other_frame_contex) >= UT_ERROR_ABORD)
{
unlockInput(0);
return error();
}

duplicatePointSource(other_frame_context);

Could I avoid this by iterating the timeline in python and calling my hdk function only to iterate the points?

 

thanks for all your help, I have really jumped in the deep end here!

Edited February 20, 2014 by glassman3d

[symek]

Afaik locking is necessary while cooking nodes in HDK. Iterating over frames in python seems to be the easiest approach, though little slower. Are you sure costs are worth efforts?

 

You can get any attribute as binary string and save it quickly to file like (untested):

header = struck.pack(....) 
file = open(..., "wb")
file.write(header)

for frame in frames:
    hou.setFrame(frame)
    geo = some_node.geometry()
    pos = geo.pointFloatAttribValuesAsString("P")
    file.write(pos) // for all points at once

file.close()

[glassman3d]

oh I wasn't aware of pointFloatAttribValuesAsString() - that could work very well

 

I will give that a go now

 

thanks!

 

 

 

UPDATE:

 

It worked like a charm - I can cache out 150 frames of point positions for 90,000 points  in about 30 secs

perfectly fine for my current needs - here is my  code for perusal

 

thanks symek!

 

import sys
import struct


def doCache():
    sf = hou.evalParm("startframe")
    ef = hou.evalParm("endframe")
    sr = 1
    ns = (ef - sf) + 1
    geo = hou.pwd().geometry()
    points = geo.points()
    np = len(points)
    
    pc2File = open(hou.evalParm("file"), "wb")

    with hou.InterruptableOperation("PC2 Cache",open_interrupt_dialog=True) as operation:


        with hou.InterruptableOperation("Exporting %s" % hou.pwd().name(),open_interrupt_dialog=True) as export:
    
            # Write header
            headerFormat='<12siiffi'
            #headerStr = struct.pack(headerFormat, 'P','O','I','N','T','C','A','C','H','E','2','\0', 1, np, sf, sr, ns)
            headerStr = struct.pack(headerFormat, "POINTCACHE2\0", 1, np, sf, sr, ns)
            pc2File.write(headerStr)
        
            
            for f in range(sf, sf+ns, sr):
                hou.setFrame(f)
                pos = geo.pointFloatAttribValuesAsString("P")
                pc2File.write(pos)
                export.updateProgress( f/ns )
                operation.updateLongProgress(0.5 * (f/ns) )
            
        with hou.InterruptableOperation("Finishing",open_interrupt_dialog=True) as finish: 


            pc2File.flush()
            pc2File.close()
            finish.updateProgress(1)
            operation.updateLongProgress(1)
            

Edited February 26, 2014 by glassman3d

[glassman3d]

for completeness thought I would post the feedback I have received from SESI

I will probably keep going with the hdk implementation as a learning exercise:

 

You need to include <GA/GA_Types.h> to get GA_Offset. And you need to include <GA/GA_GBMacros.h> to get GA_FOR_ALL_PTOFF.

 

You can implement hou.setFrame() in the HDK like so:

     #include <CH/CH_Manager.h>

     #include <OP/OP_Director.h>

     OPgetDirector()->setTime(

         OPgetDirector()->getChannelManager()->getTime(the_frame_to_be_set));

 

You can use the convenience function, UTwrite() to write out binary data to the file. For example:

     #include <FS/FS_Writer.h>

     #include <UT/UT_NTStreamUtil.h>

     int some_number = 3;

     FS_Writer writer("/path/to/file");

     UTwrite(*writer.getStream(), some_number);

Edited February 26, 2014 by glassman3d

[glassman3d]

also on the reading side I managed to hack the pc2read otl to get faster performance using numpy 

import numpy


# This code is called when instances of this SOP cook.
geo = hou.pwd().geometry()


# Read from Point Cache


if hou.parm("read").eval():
    
    pc2File = open(hou.evalParm("file"), "rb")


    # Header
    headerFormat='<12siiffi'
    pc2Header = pc2File.read(struct.calcsize(headerFormat))
    fields = struct.unpack(headerFormat, pc2Header)
    signature, fileVer, numPoints, startFrame, sampleRate, numSamples = fields
    hou.parm("signature").set(signature)
    hou.parm("fileversion").set(str(fileVer))
    hou.parm("points").set(str(numPoints))
    hou.parm("samples").set(str(numSamples))
    hou.parm("startframe").set(str(startFrame))
    hou.parm("samplerate").set(str(sampleRate))
    hou.parm("headerbytes").set(str(struct.calcsize(headerFormat)))
    
    points = geo.points()
    frame_int = ((int(hou.evalParm("frame")) - startFrame) + 1)
    xyz_size = struct.calcsize('<fff')
    hou.parm("posbytes").set(str(xyz_size))
    
    seek_offset = max([0, frame_int-1]) * len(points) * xyz_size
    hou.parm("seekoffset").set(str(seek_offset))

    if ((numPoints==len(points)) and (frame_int<=numSamples)):
        # Seek relative to the current file pos (os.SEEK_CUR==1)
        pc2File.seek(seek_offset, 1)
        buffer = pc2File.read(xyz_size * len(points))
        positions = numpy.frombuffer(buffer, dtype="f4,f4,f4").copy()
        positions.dtype.names = ("x", "y", "z")
        geo.setPointFloatAttribValuesFromString("P", positions)
    
    
    pc2File.close()

[Owl]

Hi, dont know if mine is faster, but I used to use this:

pcROP.rar

 

maybe it'll help

[glassman3d]

thanks owl, I will give that a go as well!

[louiszeifer]

Hi Owl,

 

your pcROP is not working.

The pc2write otl is running a pre and post render script in owl_rop_utils.py module which is not part of your rar file.

If I disable these render scripts, the pc2write nodes is tying to run a soho_programm called execute.py which

is not part of your rar file, either.

 

Can you please help me with that?

 

Thank you very much,

Karsten

[Owl]

Hi, Karsten,

 

execute.py is inside of the otl and it should remain there. (extra files tab)

 

pre script is for creation of non existing paths

post script is to open export path in system

so they aren't esential

 

all you need to do is copy pointcache.py into $HOME/houdiniX.Y/scripts/python/

 

could you post exact error message you are getitng?