How to access geometry data in C++ for VEX enhancement?

[Eddy]

Hello guys!

I have a simple C++ code which should be used by VEX in Houdini.

I need to access to geometry data in specific node from current frame till frame 1 in backward direction until certain condition is met.

The problem is that this access cause a huge impact on runtime when using wetmap function in a point wrangle node!

Here is the sample:

#include <iostream>
#include <VEX/VEX_VexOp.h>
#include <OP/OP_Director.h>
#include <GU/GU_Detail.h>
#include <SOP/SOP_Node.h>
#include <GEO/GEO_VolumeSampler.h>
#include <GU/GU_PrimVolume.h>
#include <GU/GU_SopResolver.h>
#include <VM/VM_Math.h>
#include <OP/OP_AutoLockInputs.h>
template <VEX_Precision PREC>
static void
wetmap(int argc, void *argv[], void *)
{
    VEXvec3<PREC> *resultCd = static_cast<VEXvec3<PREC>*>(argv[0]);
    const char *surfaceAddress = static_cast<const char *>(argv[1]);
    VEXvec3<PREC> *P = static_cast<VEXvec3<PREC>*>(argv[2]);
    VEXvec3<PREC> *Cd = static_cast<VEXvec3<PREC>*>(argv[3]);
    *resultCd = VEXvec3<PREC>{0, 1, 0};

    SOP_Node *surfaceNode = OPgetDirector()->findSOPNode(surfaceAddress);
    exint currentFrame = CHgetFrameFromTime(CHgetEvalTime());
    OP_Context context;

    VEXvec3<PREC> color{0, 0, 1};

    if(surfaceNode != nullptr)
    {
        for (exint i = currentFrame; i > 0; --i)
        {
            context.setTime(CHgetTimeFromFrame(i));
            GU_DetailHandle gd_handle = surfaceNode->getCookedGeoHandle(context);
        }
    }
}
void newVEXOp(void *)
{
    using UT::Literal::operator""_sh;
    new VEX_VexOp("wetness@&VSVV"_sh,      // Signature
                  wetmap<VEX_32>,         // Evaluator 32
                  wetmap<VEX_64>         // Evaluator 64
                  );
}

I also try to lock the referenced node input just like SOP node examples in HDK but it makes no difference!

This is an image of use case:

After couple of frames pointwrangle1 become slow to cook and I don't know why!

Can anyone help me?

Thanks in advance!

[symek]

I'm not sure what you're trying to achieve, but whatever it is, you seem to have chosen wrong path. VEX is high performance streaming instructions language. All data it operates on, should be static, monotone arrays of numbers it can slice up and process concretely. Your code is equivalent to embedding web browser in GLSL shader (my shader could access texture from http server! lets do it! LOL!), you can image such thing doesn't make any sense. Additionally you ask VEX to access Houdini's nodes concurrency, which requires locking, so you end  up with high contention of threads (they mostly wait), and entire function has exponential complexity over time. I'm guessing it's slower than Python.

I have a sneaking suspicion that thing you're trying to do can be accomplished without extending VEX (if accumulating attribute's values over time is what you're after), but if you insist on using C++, this part

context.setTime(CHgetTimeFromFrame(i));
GU_DetailHandle gd_handle = surfaceNode->getCookedGeoHandle(context);

should be inside  VEX_VexOpInit  callback, which should do all preliminary job only once before actual computation takes action. Note, that ifaik VEX by itself doesn't guarantee single execution of that callback. It should be guarded by proper atomic primitives by you.

Again, I highly doubt you need this extension, but whatever are your feelings about it, it definitely shouldn't access external nodes concurrently and recursively from within VEX extension.

[Eddy]

Thank you so much@symek!

My goal is to create a function called wetmap in VEX (which should be used in shader context) so that it can create wetmap looking in rendering instead of using traditional way which is scattering lots of points on model and then use Solver SOP to create an attribute then use point clouds to access this attribute in shading context...

So I decided to create a function called wetmap and pass it these arguments:

vector wetmap(string address, vector position, vector color);

address is node contains surface field of simulated fluid.

position is the position of current ray which is hit to the surface of the geometry which is being shading.

color is the current color of the surface which is being shading.

When I pass node address as the first argument I need to access to geometry in different frames to check whether current position is in fluid field or not.

So in this case I can't only use VEX_VexOpInit as I need to access to geometry in different frames!

I update the code based on your suggestions:

#include <iostream>
#include <VEX/VEX_VexOp.h>
#include <OP/OP_Director.h>
#include <OP/OP_AutoLockInputs.h>
#include <SOP/SOP_Node.h>
#include <GEO/GEO_VolumeSampler.h>
#include <GEO/GEO_PrimVolume.h>
#include <GU/GU_PrimVolume.h>
#include <GU/GU_SopResolver.h>
#include <VM/VM_Math.h>
#include <UT/UT_Map.h>
#include <UT/UT_Lock.h>
#include <UT/UT_UniquePtr.h>
#include <UT/UT_LockUtil.h>

using volumeCache = UT_SortedMap<exint, UT_UniquePtr<const GU_Detail>, std::greater<exint>>;


template <VEX_Precision PREC>
static void* construction()
{
    CH_Manager *mgr = OPgetDirector()->getChannelManager();
    std::cout << "Start frame is: " << CHgetFrameFromTime(mgr->getGlobalStart()) << '\n';
    std::cout << "End frame is: " <<CHgetFrameFromTime(mgr->getGlobalEnd()) << '\n';
    void* map =  new volumeCache{};
    return map;
}

template <VEX_Precision PREC>
static void destruction(void* data)
{
    delete static_cast<volumeCache*> (data);
}

template <VEX_Precision PREC>
static void
wetmap(int argc, void *argv[], void *data)
{
    VEXvec3<PREC> *resultCd = static_cast<VEXvec3<PREC>*>(argv[0]);
    const char *surfaceAddress = static_cast<const char *>(argv[1]);
    VEXvec3<PREC> *P = static_cast<VEXvec3<PREC>*>(argv[2]);
    VEXvec3<PREC> *Cd = static_cast<VEXvec3<PREC>*>(argv[3]);
    volumeCache* map = static_cast<volumeCache*> (data);

    *resultCd = VEXvec3<PREC>{0, 1, 0};

    SOP_Node *surfaceNode = OPgetDirector()->findSOPNode(surfaceAddress);
    exint currentFrame = CHgetFrameFromTime(CHgetEvalTime());

    OP_Context context{CHgetEvalTime()};

    VEXvec3<PREC> color{0, 0, 1};

    for (exint i = currentFrame; i > 0; --i)
    {
        if(map->contains(i) != true)
        {
            context.setTime(CHgetTimeFromFrame(i));
            OP_AutoLockInputs inputs{surfaceNode};
            inputs.lock(context);
            const GU_Detail *gdp = surfaceNode->getCookedGeo(context);
            if (gdp != nullptr)
            {
                UT_UniquePtr<GU_Detail> temp {new GU_Detail};
                temp->duplicate(*gdp);
                (*map)[i] = std::move(temp);
            }
        }
        const GEO_PrimVolume* volume = static_cast<const GEO_PrimVolume *>(map->at(i).get()->getGEOPrimitive(0));
        GEO_VolumeSampler volumeSampler{volume};
        fpreal sample = volumeSampler.getValueF(*P);
        if (sample < 0)
        {
            *resultCd = color * (currentFrame - i) * 0.01;
            break;
        }
    }
}

void
newVEXOp(void *)
{
    using UT::Literal::operator""_sh;
    new VEX_VexOp("wetness@&VSVV"_sh,
                  wetmap<VEX_32>,
                  wetmap<VEX_64>,
                  VEX_ALL_CONTEXT,
                  construction<VEX_32>,
                  construction<VEX_64>,
                  destruction<VEX_32>,
                  destruction<VEX_64>
                  );

}

I tried to access to fluild field geometry at different frames on the fly but I find out that for each request Houdini cooks the geometry for that frame and it become costly since for each position on surface this process should be repeated!

So I create a map of items which first (key) is frame number and second (value) is pointer to GU_Detail which holds a deep copy of cooked flip fluid geometry!

Now speed increased but still there is a huge difference between C++ and pure VEX implementation version!

Again thank you so much dear @symek for your great answer.

I highly yearn for the further help!

Edited January 31, 2021 by Eddy

[symek]

On 1/31/2021 at 9:57 AM, Eddy said:

My goal is to create a function called wetmap in VEX (which should be used in shader context)

Ok, but you realize that in shader context you won't have access to node's geometry (entire OP_Director business doesn't exists neither in Mantra nor Karma)? 

[Eddy]

Thank you so much dear @symek!

Are you sure about this? because I used Volume Sample VOP and pass it address of node which contains SDF volume and everything works as expected!

Also here: https://www.tokeru.com/cgwiki/index.php?title=Houdini_Lighting_Shading#Trail_from_sampling_an_SDF_and_a_material_wrangle

Shows that volumesample function can access to node's geometry succesfully!

So how volumesample can access to node's geometry?

Thanks again dude!

Edited February 4, 2021 by Eddy

[symek]

11 hours ago, Eddy said:

So how volumesample can access to node's geometry?

Note this:

Quote

you just have to make sure the sdf is available to the renderer by exporting it as its own object, but then also making sure 'renderable' is turned off so it doesn't render.

Basically, geometry is accessible in Mantra as long as it was exported to IFD file as an renderable object. In some cases Houdini can do it even for you (as in case of textures from COPS). So, I didn't say you can't use volumesample, I said you can't use OP_Director to bake geometry from nodes present in Houdini - neither in current frames nor previous one. Mantra doesn't have access to Houdini nodes. It sees static objects (GU_Detail) as exported to IFD file which are named by their Houdini paths.

To access arbitrary geometry, you would have to save it to disk,  or... use HEngine rendertime procedural to load loads of geometry at rendertime (inside your HDA) and do your trick there.

[Eddy]

Dear @symek Thank you so much!

Dear @symek now two questions are coming:

1- Does volume sample function access to current geometry via IFD file in shader context?

2- Why backing and rereading geometry afterward are so slow when we are using in Point Wrangle SOP in SOP context? Is this normal? No matter if I pointing to a light or a very dense SDF it is always slow! If I comment out rereading geometry section from the code, speed increases noticeably!

When I rewind and play, in couple of first frames it is fast but afterward Houdini starts lagging and getting slow quickly!

Thank you so much again for the great responses!

Edited February 4, 2021 by Eddy

[symek]

On 2/4/2021 at 6:56 PM, Eddy said:

1- Does volume sample function access to current geometry via IFD file in shader context?

Sorry, I haven't noticed that before. You probably solved it already but for a sake of completeness: it isn't typically necessary, because current geometry is available in a shader as geometry attribute (via parameters binding) , but generally, yes, volumesample() will work in shader context with op:/path/to/geometry as long as geometry is present in IFD file (which is the case, when /obj/object has the display flag active). You can export additional volume to a IFD and set its renderable parm empty to make it invisible, but people usually dump volumes on disk to keep IFD files small.

[Eddy]

Thank you @symek so much!

SESI said me they use other methods and complex caching mechanism which unfortunately is not published as part of public API!

Using op:/path/to/geometry style is not causing to access geometry data in vex either for user defined C++ VEX functions...

BTW the case is still open and I couldn't find any solution for this problem so far.