Houdini Distributed Simulations Explanation

[MrBrN197]

Hello Everyone, I am wondering can some one explain the details of a clustered simulation. I was wondering when I saw the clustering simulation why the individual clusters can't be saved as a scene and have it run on multiple computers since it would work on a renderfarm. So I'm wondering how the actual process works does the simulation have to work simultaneously where each cluster is still somehow referring to each other for calculations. 

[Solitude]

Clustered sims (a type of distributed simulation) are for when you do want the sims to run independently on a renderfarm and not look at each other. You need a script to iterate an env variable that changes something in your scene. This would switch the cluster the computer runs -- you could save out a hip file for each one of these, or the scripts should tell the farm to set a certain env variable. With pyro is usually means switching a bounding box, source, and cache file based on that env variable. 

I realize I am over-explaining in this next part, but figure it may be helpful for somebody else.

Distritubed sims are a bit different in that they do rely on each other for information, effectively simming one big sim over multiple machines. This requires slices to be set up where there is a tiny bit of overlap a so the grids and particles can transfer data to each other. Same thing goes here you need to set env variable on the farm machine so they know what slice they should be running. A sim tracker job has to be run as well and keeps tabs on the status of each of the sim jobs as they run so they know when to fetch data from the neighboring grids. Each grid has to wait for the other grids before it can start it's next frame though.

Keep in mind it may be faster to sim on one big machine than over 4 slower ones... it depends on the network overhead. It is going to be as slow as the slowest machine you sim on too. Speed is not linear as you increase machines either. I recently tested, and got some big sims (70million, and 135 million) done on 4 machines. They simmed about twice as fast as the same sim on a single machine. Ram use on each slice was about 1/2 vs the single sim too, so if you have a big sim that can't fit on your machines, distributing will help with that.

There is a masterclass on this that's pretty recent... he even shows how to set up an arbitrary particle sim for distribution at the end.

 

Edited October 21, 2016 by Solitude

[Yon]

For clustering, you can make a Take for each cluster, changing the cluster filter in clusterpoints node. Then use Render With Take in ROP_geo. They don't depend on each other.

[FurlongFX]

Thank you Yon!! just came across this and I finally got distributed clusters simming on the farm independently!!!! Thanks