Pops / Vex / Multithreading

[symek]

This is a bunch of teoretical questions:

1. Are POPS multi-threaded ( I suppose they are not)?

2. Can I do in VEX *all stuff I can achieve with POPs operators? If not what are the examples of "can't be done"?

(Vex can't create any particles, so any birth/split cases are impossible, right?)

3. If 1. is "NO" and 2. is YES/partially YES: is that means that in case of heavy particles simulation it's better to work in Vex code as it can be multi-threaded?

In H9 I can work easily with millions of points what encourages me to try things that were hard to do previously. This would mean collisions detection in Vex for example.

thanks for any commends,

sy.

[jhiggins]

you might want to post this on the houdini mailing list!!

my guess would be that some parts of POPs are multithreaded others not.

VEX pops are definitely multithreaded

and you cannot do everything VEX that you can do POP networks

and finally yes, using VEX for what you can use it for i would expect would be faster if you had more than 1 prcessor at your disposal (assuming the equivalent pop wasn't multithreaded).

like i said you may want to get an official sesi answer as i'm sure it would be good to know whats multi-threaded and whats not!

please post back with what you find out !!

hope this helps!

[3dbeing]

Pops is a sim which requires access to all parts being calculated at all times, classically this negates the abillity to multi-threading. Vex is faster and will multi-thread, but if it is a vex pop, then it is bound by the limitations of Pops, which means it cannot multi-thread.

You bring up an interesting topic, worth some inspection. As processors have become more powerfull and carry more local cache ram, I wonder if it is not possible to multi-thread on a dual-core proc which shares the same cache. Seems like we should be getting close to a multi-proc sim solution.

[jhiggins]

i'm going to have to disagree with you 3dbeing.... create vex pop and you'll see that it has options for multithreading when you drop it into you pop network.

also, i think its a generalization that simulation isn't multithreaded, it really depends on the algorithm, but often when you have lot of objects that your applying the same rules to, thats very easily multithreaded.

[Jason]

i'm going to have to disagree with you 3dbeing.... create vex pop and you'll see that it has options for multithreading when you drop it into you pop network.

also, i think its a generalization that simulation isn't multithreaded, it really depends on the algorithm, but often when you have lot of objects that your applying the same rules to, thats very easily multithreaded.

Jeffrey is right here - the native C++ nodes throughout Houdini are not multi-threaded, at least not in a general way. C++ built-in nodes (POPs included) need to be threaded on a case-by-case basis explicitly by the developers. All VEX nodes are all multi-threaded, however; it's a benefit of the underlying architecture (called SIMD) that allows, nay, encourages them to threaded and pipelined.

Aside: You might wonder why there isn't a kind of general multi-threading going on in Houdini amongst all nodes and as far as I am aware the major obstacle is that Hscript and Expressions are not multi-threaded which, by extension, rules out parameter evaluation - which is pretty important, they tell me Perhaps one day they'll be able to detect if parts of a scene graph are all python-based and multi-thread those? With all these multi-core systems the market pressure must be very high to do so.

[3dbeing]

I wont disagree, in fact I was struggling to understand why they weren't being multi-threaded, but the way it was explained to me is that every piece of information depends on every other piece, so how can you be solving one piece on one side and another piece on another side and have them communicating simultaniously, I was told you can't, and their argument seemed rational. And when running sims in multi-thread mode, I didn't see more than one proc going, I can't recall if it was in pops or dops or both, but I did indeed verify that sims were using a single virtual proc in multi-thread mode, so maybe I'm missing something.

[Jason]

I wont disagree, in fact I was struggling to understand why they weren't being multi-threaded, but the way it was explained to me is that every piece of information depends on every other piece, so how can you be solving one piece on one side and another piece on another side and have them communicating simultaniously, I was told you can't, and their argument seemed rational. And when running sims in multi-thread mode, I didn't see more than one proc going, I can't recall if it was in pops or dops or both, but I did indeed verify that sims were using a single virtual proc in multi-thread mode, so maybe I'm missing something.

Certainly cooking down a single SOP chain can't be multi-threaded since each node requires the previous node have completed its task in order to hand the geometry to the following node, but in the case of branching or multiple independant trees (multiple objects, for instance) there is no theoretical reason why these can't be multi-threaded.

DOPs are threaded on a node-by-node basis too. You'll see the particle fluids lend themselves to be solved in parallel very well but only very few stages of the PLS/Volume fluids are efficiently threaded.

[el_diablo]

I dont see why a single chain of SOPs couldnt be multithreaded...

For example you could seperate primitives in N pieces subdivided seperatly by multiple threads of Subdivide SOP, bounds should be probably expanded by 1 tho. Actually some of the 'live' subdivision editing options in other packages seem to do exactly this.

Then there is of course branch threading... I could think of some other ways to use a multiprocessor machine.

I think Houdini will have to get there eventually...It really needs to 'get up to speed' on some tasks. Using GPU is an option also. Its all nice that it can do almost anything, but as M. Perez already said on one of his wip threads, most other app plugins smoke (pun intended) Houdini speedwise.

[Jason]

I dont see why a single chain of SOPs couldnt be multithreaded...

For example you could seperate primitives in N pieces subdivided seperatly by multiple threads of Subdivide SOP, bounds should be probably expanded by 1 tho. Actually some of the 'live' subdivision editing options in other packages seem to do exactly this.

Then there is of course branch threading... I could think of some other ways to use a multiprocessor machine.

I think Houdini will have to get there eventually...It really needs to 'get up to speed' on some tasks. Using GPU is an option also. Its all nice that it can do almost anything, but as M. Perez already said on one of his wip threads, most other app plugins smoke (pun intended) Houdini speedwise.

Well, each SOP would have to be multi-threaded separately - considering there are no rules governing what a SOP does to the geometry. Some iterate over prims, some over points, some regenerate the geometry, destroy it, create new geometry, etc. VEX has clear rules - you can ONLY modify a point's attributes - and add to the Point Group dictionary.

As for sub-d previews - it should be the viewport's responsibility to generate sub-d preview meshes and I'm sure that when SESI get around to doing this that it might be easy to do. I can imagine that the hard part is updating the viewport asynchronously and also allow the sub-d thread to be interrupted so that users are not forced to wait for it to complete.

CPUs with multiple cores are looking like they are going to out-pace GPU development soon. Intel is rumoured to be working on an 80-core CPU within 3 release cycles. The writing is on the wall... and SESI is reading it - remember the big development for Mantra (and Houdini) to support the Cell processor before SIGGRAPH? I hope we get to see the benefits of that development in Houdini proper in the future.

[el_diablo]

Well, each SOP would have to be multi-threaded separately - considering there are no rules governing what a SOP does to the geometry. Some iterate over prims, some over points, some regenerate the geometry, destroy it, create new geometry, etc. VEX has clear rules - you can ONLY modify a point's attributes - and add to the Point Group dictionary.

As for sub-d previews - it should be the viewport's responsibility to generate sub-d preview meshes and I'm sure that when SESI get around to doing this that it might be easy to do. I can imagine that the hard part is updating the viewport asynchronously and also allow the sub-d thread to be interrupted so that users are not forced to wait for it to complete.

CPUs with multiple cores are looking like they are going to out-pace GPU development soon. Intel is rumoured to be working on an 80-core CPU within 3 release cycles. The writing is on the wall... and SESI is reading it - remember the big development for Mantra (and Houdini) to support the Cell processor before SIGGRAPH? I hope we get to see the benefits of that development in Houdini proper in the future.

Yes, I know what you meant with dependence. Its important to flag OPs with its area of influence (attrs, topology etc) then you can make dependency graphs and out of order executions and also concat some of the ops. I'm sure Houdini has some form of this functionality for recooking. Anyway, I really hope for Houdini to get faster, maya fluids may not be integrated with rest of the dynamics like Houdini fluids are, but they are many times faster to evaluate.