restarting sim from cache

[sschoellhammer]

Hello there,

I've searched the forum but I couldn't find the answer to just this cache question so here goes:
I'm doing a big simulation and I'm caching it to .sim files via "Save to Disk" in the DOP network. 
Sadly I had to interrupt the simulation half way and now I want to restart from where I left off,
which seems to be possible looking at the docs. So I have the dop in playback mode (everything looks good so far) and I change the 
start frame, then press "Save to disk" again. Now the sim is just halting on the first frame and goes on for hours and it seems 
to me it's just restarting from the beginning...:/  is there anything I'm doing wrong? Do I have to use the checkpoints instead?

Thanks a lot!

Seb

[DennisSchmidt]

Yep, you have to use checkpoints. Be sure to set a trail length. Those files will be picked up and used for continuing the sim after it failed or was stopped manually. When setting a different output directory for the checkpoint files make sure you create those intermediate directories otherwise the files won't be saved.
Check the attached file if you get stuck.

Dennis

DOP_continue_cache_v01.hipnc

[sschoellhammer]

Hi Dennis, 

thanks for the quick answer and example!!
After I switched on the checkpoints option I'm now getting a ton of openCL errors :/.  (this is using the sand solver)
And when I do Save to Disk in background (I thought that might help) , it's finishing really fast but without any meaningful results.. arg, time for bed.

seb

Edited June 21, 2016 by sschoellhammer

[sschoellhammer]

OK, the problem with the background render is the same, it's the OpenCL thing, this is the sort of error I'm getting:
 

OpenCL Exception: clSetKernelArg (-51)
OpenCL Context error: CL_INVALID_KERNEL_ARGS error executing CL_COMMAND_NDRANGE_KERNEL on GeForce GTX 980 (Device 0).

OpenCL Exception: clEnqueueNDRangeKernel (-52)
OpenCL Exception: clSetKernelArg (-51)
OpenCL Context error: CL_INVALID_KERNEL_ARGS error executing CL_COMMAND_NDRANGE_KERNEL on GeForce GTX 980 (Device 0).


I just go the latest drivers for the card but no change sadly.

Weirdly this only happens if I switch on checkpoints. Otherwise it's just working fine.
I'm rendering without opencl now but it's really a good deal slower..

 

[sschoellhammer]

Mmm, now that I ran it with openCL off, I see that it took longer but while I got no errors I still don't get a usable result.

There is one unusually big sim file and then lots of way too small ones. When I render without checkpoints each file is about 600 mb and all is fine.

So I suppose my only option is to just render the whole thing in one go and hope there's no power cut in those 2 days

I'd still love to know if others had similar issues...

 

[DennisSchmidt]

Strange. I only used checkpoints with pyro but I think it should work for other solvers as well. I will give it a try later. Oh and Open CL with pyro works fine.
Are those files your caches or the checkpoints? Have you tried recreating a simple setup? If it's still happening maybe you could post the hipfile.

[DennisSchmidt]

Just made a simple sand/grain setup and it works fine for me. And Open CL even works with the checkpoints. I think its something specific to your scenario.
Are your files that small after you try to continue the sim? Be sure to have a trail of at least 2 or 3 since the last checkpoint is not complete when the sim crashes(to be sure just delete it, normally the frame is smaller than the others). Another important part is continuing from the correct frame.  Note that the checkpoints frame count starts at 1. So if your sim starts on let's say frame 1000 and crashes on frame 1020 the last checkpoint would be 21. Now you would continue caching from frame 10019 since the last checkpoint(21) should not be used. I think the workflow is clear and it's something specific to your setup. But just in case i attached the sample file.

Is it possible to upload your scene?

sand_simtest_v01.hipnc