Hashes and Caching

[Jason]

Hi all and SESI,

A portion of a current project of ours has an large dependance on CopyStamping custom objects into the scene. Each copy is quite a lengthy cook - around 1-3 seconds - but these are being stamped easily 1000-3000 times into the average scene. This adds up. One thing, though is that there are really only around 50 variations - a set number. So if we could cache the 50 variations we'd be away!

So entered the idea of using the CacheSOP before the CopyStamp - using an index which is a hash of the varying parameters. The HDK has a UT_Hash class which we exploit for this. We wrote a "getnodehash(node,parm_pattern)" expression and tried to use that in CacheSOP. These hashes can get REALLY big, and the index has to fall in the Start/End range of the CacheSOP, so we might have to tell the CacheSOP that there are 1e7 frames in the sequence. This almost works , but Houdini doesnt really like that much.

So basically, we have a couple of issues:

1) It would be fantasic to extend the CacheSOP (or make a new one) that handles the following:

1.a ) An arbitrary index - like an index into a indexed-array. In this case the index would be the "getnodehash" value.

1.b ) Support a RAM limit, or a #entries limit or both

1.c) If the cache limit is blown, to cook the input SOP and pass through the result - i.e you will always get a valid result (if the cache is up to date, of course).

2 ) In our implementation of the getnodehash() function, the UT_Hash_String class could not be effectively subclassed because myData is not private data and there is no member function to load it. So we subclassed UT_Hash instead and duplicated the members from UT_String. I wonder if Side Effects are aware of this? Is there a better way to do it?

Are we going in the right direction with any of this? Does this sound useful? I would love to hear if the CacheSOP enhancements sound reasonable and useful?

Here is the new class for hashing a node's parameters along with its parameter name - in order to make it REALLY unique.

#include "UT/UT_String.h"
#include "PRM/PRM_Parm.h"
#include "PRM/PRM_Type.h"
#include "PRM/PRM_ParmList.h"
#include "OP/OP_Node.h"
#include "OP/OP_Director.h"

class UT_API UT_Hash_Node : public UT_Hash {
 public:
	UT_Hash_Node( const char *s )
	{
     myData = strdup( s );
	}

    UT_Hash_Node( OP_Node *op, const char* pattern, float now )
      {
          //Init the parameter value string as empty
          UT_String parmstr = "";
          
          // ***** Loop through the OP_Node parms and concat onto "parmstr" with tostring(parm values)
          
          //get at the parm list
          PRM_ParmList *opparms = op->getParmList();
          int parmcount = opparms->getEntries();
          
          //loop through the parms
          for (int i=0; i<parmcount; i++)
          {
              PRM_Parm *curparm = opparms->getParmPtr(i);
              
              UT_String temp;
              
              curparm->getToken(temp);
              
              if (temp.match(pattern) &&
                  ((curparm->getType()&PRM_TYPE_STRING) ||
                   (curparm->getType()&PRM_TYPE_ANIMATED) ||
                   (curparm->getType()&PRM_TYPE_ORDINAL))
                  )
              {
                  parmstr += temp;
                  UT_String curvalue;

                  int numvalues = curparm->getVectorSize();
                  for (int j=0; j<numvalues; j++)
                  {
                      curparm->getValue(now, curvalue, j, 1); //1=evaluate?
                      parmstr += curvalue;
                  }
                  parmstr += " ";
              }
          }        
          //set my data
          myData = strdup((char*)parmstr);
      }


    private:
        char  *myData;
    public:

	~UT_Hash_Node()
	{
     free(myData);
	}

	int compare( const UT_Hash & a) const
	{
     return strcmp(myData, ((const UT_Hash_Node&)a).myData);
	}

	unsigned	hash() const
	{
     return UT_String::hash(myData);
	}

	UT_Hash*	copy() const
	{
     return new UT_Hash_Node( myData );
	}

	long getMemUsage() const
	{
     return (sizeof(*this) + (::strlen(myData) + 1) * sizeof(char));
	}
    
};

[malexander]

You can use a UT_TokenString & OP_Node::getHashCode() to get a very unique description of parameters set on a node & its descendents (or, OP_Node::getParmHashCode() for just that node).

They can get rather large depending on the network size, however they're heavily optimized for both space and comparison time. COPs uses them for its smart caching.

Since a UT_TokenString has an 'int getHashCode()' method, as well as operator==() and operator=(), it would be pretty easy to make a UT_HashTokenString class.

Might solve half your problem

[Jason]

You can use a UT_TokenString & OP_Node::getHashCode() to get a very unique description of parameters set on a node & its descendents (or, OP_Node::getParmHashCode() for just that node).

They can get rather large depending on the network size, however they're heavily optimized for both space and comparison time. COPs uses them for its smart caching.

Since a UT_TokenString has an 'int getHashCode()' method, as well as operator==() and operator=(), it would be pretty easy to make a UT_HashTokenString class.

Might solve half your problem

15415[/snapback]

Heh heh.. I should have known... sigh.. Guess I'll switch to that then - thanks Mark.

Now, onto the Cache SOP thing. Can you think of an efficient way to do what I'm thinking of doing using the current incarnation of the CacheSOP? It seems like it could be extended to be a more general-purpose cache then it is at the moment. I'd like to RFE the above suggestions if it doesn't completely break the model of the CacheSOP. The pass-through-if-cannot-cache thing would be great in the current SOP, and it'd be great to add info about cache-hit statistics too.

What do you think?