Resuming distributed flip simulation

[wes]

I am testing distributed flip sim, everything goes fine.  Now I can think of a situation where my simulation failed due to whatever reason, how should I restart it from where it left off, so it won't redo the simulation all over again?

 

I tried using ${SLICE} in DOP file in my DOP network  e.g  $HIP/geo/xxx_${SLICE}_$F.sim , and set to "Automatic", but it seems that once Dynamic rop driver is in use to generate .sim files, the dop file node gives no effect.  

 

Then, I tried using ${SLICE} in DOP network initial state,  but jobs won't start in hqueue farm after submission, it seems that machines are waiting for each other.

 

I also tried explicit cache and hope that it will keep a full version of my sim for a few frames so I can use it as my initial state when I resume the simulation.

 

Merging .sim sequence files from different machines in a new DOP also failed, only slice can be seen.

 

So far I can't find a solution,  is there a standard way to do this?

 

 

[Luis Silva]

I'm also curious about this.

I've also been testing distributed FLIP sims on a small HQueue farm, trying to max out RAM memory resources. Everynow and then there's a machine failure, and I can't find a way to resume sims from there.

I've also tried similar mechanisms as Wesley did, to no avail.

Can anyone shed some light on this, please?

Thank you in advance.

Edited May 15, 2018 by Luis Silva

[bunker]

That's not setup by default as far as I'm aware.
You can write a simple "pre-render" python script on your Geometry ROP
to check the latest bgeo file written and set it as the FLIP initial data File.
You don't need .sim files for that. You could use the commands library for that:

# returns the file before last
import commands
path  = '/Users/myuser/mydata/'
c='ls -t '+path+'*.bgeo.sc | tail -2 | head -1'
print commands.getstatusoutput(c)[1]
 

Edited May 19, 2018 by bunker