Persistent Storage for VEX Shadeops

[Mario Marengo]

Someone PM'd me a while back about this topic and it was suggested I post a response to the public forums in case someone else found it useful -- or just to exchange ideas on different solutions. It's been a while since that request (sorry nanocell), but better late than never.

The goal is to come up with a way to enable a shader to store arbitrary data that is persistent across multiple shader calls. It's not often you'll need something like this, but it comes in handy to have something worked out for the somewhat rare cases when you do need it.

So, first a little utility code: a header with a few macros to simplify API function definitions and make RSL and VEX sources somewhat more uniform:

// file: ShadeOp.h
#include <UT/UT_DSOVersion.h>
#include <VEX/VEX_VexOp.h>

// RSL-like prototypes for the function definitions
#define SHADEOP(name)         void  name (int argc, void *argv[], void *data)
#define SHADEOP_INIT(name)    void *name ()
#define SHADEOP_CLEANUP(name) void  name (void *data)

// Parameter access
#define SHADEOP_PARMREF(ndx,type) *reinterpret_cast<type *>(argv[ndx])
#define SHADEOP_PARMVAL(ndx,type) SHADEOP_PARMREF(ndx,type)
#define SHADEOP_PARMPTR(ndx,type) reinterpret_cast<type *>(argv[ndx])

OK. Let's find out what the lifespan of a dso global variable is. In theory, this should be the same as that of the process which called dlopen(), although 'global' symbols could be treated differently depending on the flags passed to dlopen(). So let's see how VEX behaves.

// file: vex_store.C

#include <ShadeOp.h>

#include <iostream.h>

using std::cerr;

using std::endl;

class VexStore {

public:

VexStore(float val=0) : data(val) {

cerr<<"Constructor: "<<data<<endl;

}

~VexStore() {

cerr<<"Destructor: "<<data<<endl;

}

void set(float val) { data = val; }

private:

float data;

};

static VexStore store(-1);

SHADEOP(testdso) {

float &r = SHADEOP_PARM(0,float);

r = SHADEOP_PARM(1,float);

store.set

VexStore.tar.gz

[nanocell]

o_o

o_O

O_O

+_+

Wow...amazing. Hardcore. After reading that I feel a bit overwhelmed and...insignificant.

Well done Mario, and thanks for your time! It's definitely a place for me to start.

CheerZ,

Van Aarde.

[Wolfwood]

Someone PM'd me a while back about this topic and it was suggested I post a response to the public forums in case someone else found it useful -- or just to exchange ideas on different solutions. It's been a while since that request (sorry nanocell), but better late than never.

The goal is to come up with a way to enable a shader to store arbitrary data that is persistent across multiple shader calls. It's not often you'll need something like this, but it comes in handy to have something worked out for the somewhat rare cases when you do need it.

Awesome indeed!

One thing to note is if you just need to stash a "simple" data type, (int/float/vector/matrix/matrix3/vector4), then you can use point clouds. Exporting data to point clouds is persistent across multiple shader calls. As long as which ever shader refers to the same point cloud file you can fetch the data. What this means is you can create a point cloud with a single point in it, then with the pcopen() set a huge search radius which will encompass the scene to find that single point. Then stash what ever data you want into that point and reference it from any shader.

surface
test( string pc = "/mnt/users/jimp/pc.pc"
{
    vector Po = ptransform("space:object",P);
    int handle = 0;
    int success = 0;

    handle = pcopen(pc,"P",Po,1e6,1);

    while ( pcunshaded(handle,"stash") ) {
        float rand = nrandom("twister");
        success = pcexport(handle,"stash",rand);
    }

    float getStash = 0;

    while ( pciterate(handle) ) {
        pcimport(handle, "stash", getStash);
    }
    pcclose(handle);

    string name = "";
    renderstate("object:name",name);

    printf("name: %s\tstash: %g\n",name, getStash);

    Cf = diffuse(normalize(N));

}

So if you just need to stash basic type, (no arrays or strings or structs etc), then this should work too.

Cheers.

[Mario Marengo]

Hey Jim,

One thing to note is if you just need to stash a "simple" data type, (int/float/vector/matrix/matrix3/vector4), then you can use point clouds.

Absolutely. And point clouds should be the first thing to try.

In fact, between point clouds and arrays, there are now even fewer cases when one may have need of something like this. But of course, Murphy's Law applies: It is when you *don't* have something like this that you will absolutely need it -- so I guess having it protects you from ever needing it

In this case, nanocell was already using point clouds, but was wondering if there was an HDK-based approach that could provide more flexibility.

Personally, I think a far more common application is not so much generic storage, as some expensive (but frame-constant) external processing which gets queried by the shader (where you can't resort to the built-in point cloud mechanism). If nothing else, the above shows you how to open the back door to something like that.

...and trying to come up with a non-template yet type-generic attribute container is a fun programming challenge

[Wolfwood]

Ah! Sorry didn't know the back story. I didn't want a random passerby to think this was the only way.

Don't get me wrong. Its a gem. I've bookmarked and filed it away under "Abuse later".

Heck just finding out the scope the DSO lives through is valuable. Darn fine investigative fun.

[rdg]

Ah - I like this post!

QUOTE(Wolfwood @ Jan 8 2009, 09:50 AM) *

One thing to note is if you just need to stash a "simple" data type, (int/float/vector/matrix/matrix3/vector4), then you can use point clouds.

If am able to encode my data structure into a cubic form I don't even need to dive into this vex p0rn.

And this is the first step into relational structures, where I could reference pointclouds from pointclouds from point clouds.

OMG - what a nerdy morning.

[Mario Marengo]

Ah - I like this post!

If am able to encode my data structure into a cubic form I don't even need to dive into this vex p0rn.

And this is the first step into relational structures, where I could reference pointclouds from pointclouds from point clouds.

Hey Georg,

What do you mean by "encode my data structure into a cubic form"?

[rdg]

Hey Georg,

What do you mean by "encode my data structure into a cubic form"?

The simpler version would be a row column structure where an [x][y] pair would point to a identifiable point in the point cloud.

It also could work in three dimensions of course. [x][y][z]

If you want you can randomize the point positions ... but this together with an array would allow something like an associative array/hash.

I once read or heard - I think Garman told me that Alex told him - that as there were no arrays in Houdini you could use a subdivided line to store attrinbutes in a array fashion.

[Mario Marengo]

The simpler version would be a row column structure where an [x][y] pair would point to a identifiable point in the point cloud.

It also could work in three dimensions of course. [x][y][z]

I see. Thanks for the explanation.

Yeah, I agree that there are all sorts of ways to abuse pointclouds to hold frame-constant information, and I've used them for that purpose too. It has its own set of limitations though, so as usual, it all depends on the situation.

To reiterate, I don't think a generic storage like the one I described above is all that useful in and of itself -- it makes for an interesting learning exercise, yes, but a more realistic DSO would be far more efficient if tailor made for a specific situation. The above can be used as a reference, which is all it was intended to be.

And in the meantime, there are always point clouds to abuse

Cheers.

[rdg]

I see. Thanks for the explanation.

Yeah, I agree that there are all sorts of ways to abuse pointclouds to hold frame-constant information, and I've used them for that purpose too. It has its own set of limitations though, so as usual, it all depends on the situation.

In my situation it's unfortunately comepletely useless as the rows|columns aren't normalised - means I don't know how many columns a row will have.

Has anybody written a persistent python dictionary reader for vex ?

[Mario Marengo]

In my situation it's unfortunately comepletely useless as the rows|columns aren't normalised - means I don't know how many columns a row will have.

You can probably work your way around that, but that's one of the limitations of PC's -- It's a static structure (by the time it arrives at the shader), so its size and spacial distribution has to be determined a-priori (assuming that's possible in a particular situation).

So... the main drawbacks are 1) it can't grow dynamically at shading time, 2) Any initialization/allocation has to happen before shading begins, 3) Multiple shader instances would need their own separate point cloud, 4) data points can only be visited in order of proximity, and so on... None of which makes point clouds useless, mind you, just somewhat limited for some of the more complex cases.

Regarding a way to determine the size of some arbitrary PC data lattice from inside the shader, I imagine you could circumvent the problem by:

1) Adopting the convention that each data point ([row][col] coordinate) can only exist on positive integer locations in space, so the elements of their spacial coordinates belong strictly to the natural numbers [0,+inf].

2) Given #1, a shader can derive the extents of the data by opening the cloud at {0,0,0} with a huge radius and then calling pcfarthest() to infer the shape, num of entries, etc.

3) ... some scaffolding will be needed inside the shader to map named attributes to predefined [row][col] coords of course...

4) Last but not least, the fact that you're forced to visit the data by proximity means it's not a random access structure (you can't just "index into a point" to read/write its value) -- this bit alone could break all your efforts at a "structured" layout... again, this may or may not be a deal breaker, depending on the situation.

Just thinking out loud.

[rdg]

Just thinking out loud.

thanks for sharing!