managing .sim checkpoint files

[ikoon]

Please, how do you manage your .sim checkpoint files? So far, I just delete them manually. I am thinking about writing a simple python scripts:

- delete all the .sim files
- delete all, but keep the last one
- keep the first .sim file in the preview range (for the case, when I am satisfied with previous frames and I animate parameters in the preview range)

Or is there any existing tool or script, that could help me? I would use os.remove()

If not, I will post my solution here, later.

[LaidlawFX]

Generally a follow up farm process that deletes the check point file # numbers back. You don't want to delete the immediately finished one encase the sim crashed on it. Where you add the python script to cull the files depends on how you are siming your files whether locally or on a farm. Pretty much the script is os.remove() with some refinements. I am not aware of any public script functions like that at this time. I would not be surprised if deadline has something like it already, but certainly they python functions in most major farms will have it.

[ikoon]

Ben, thank you very much for your answer and valuable insight! I will do some simple python and post it here.

[pezetko]

For *.sim files I'm using built-in rolling cache, usually keeping latest 3-5 frames is enough to continue in case of a crash.

Cleaning old files with Windows 10 built-in feature and cleaning temp folders on each restart with a simple batch script.

 

Edited August 28, 2018 by pezetko
enter submitted it to early

[midicon]

Why are you using those bloated files? if you are managing your scene and your sim correctly and efficiently Houdini can solve correctly without crashing. 

[LaidlawFX]

 

@midiconYes you are right in a well thought out simple network you should never need the .sim data and just cache after your network to .bgeo.sc or the equivalent.

However, when building out a pipeline you need to factor in for all the edge cases. There are many factors that can cause you to need that data. 

Example 1. In a large production it's good to have the option to save the .sim for when you have many artist of varying skill levels. For instance, most staff in production hire a log to linear scale of artist with more junior to mid-level than senior artist. So you can not guarantee everyone will make a sim as cleanly. So a junior artist may not properly balance their sim.

Example 2. Especially with the nature of large productions and commercials you are always challenged to work on something new. Nobody wants to see the same commercial or feature film again. Which means invariable changes that you can account for. You won't always be working on the same sim for every project, especially in commercials this can change quite frequently.

Example 3. Even for Senior people working on the most advanced simulations you may be on the cutting edge of the technology and running into memory leaks. The deadline might be tight so you can't wait for SideFX to fix the issue.

Example 4. Another edge case is your are splitting the sim data on multiple machines. You can get quite tricky with this trying to balance intensely large simulations. Your sim may start on one machine but grow to use many computers.

Example 5. Often is the case some one let's a rogue sim go on the farm and they did not define the memory footprint correctly ahead of time, say they used the default of 32 GB when it really needs 64 GB. It competes with another sim or render on the same node causing that node to crash. Generally not knowing which caused the crash these are spun up again crashing more machines. If the render wrangler is not aware this can spiral out of control. Instead of restarting the sim and renders from scratch you can restart the sim on a new node. 

Mantra renders have the same concept with .checkpoint files these are the partially rendered images where you can start up a render from again. This really gets important as each generation of hardware demands higher resolution, so you need to produce images at the edge of the hardware capability. Also hardware in production has been purchased over several years, so not all of it will be up to the latest standards.

This is often a common conversation of when you should delete .ifd to in production? Delete directly after render, delete when render is done, wait a week then cull the data encase the ifd need to be re-rendered again with a few style sheet changes. 

 

[ikoon]

Ben, thank you very much!!!

I also plan to use .sim checkpoints for additional animating of forces, or amounts of dissipation, sourcing... Lets say I have simulated 300 frames. Until frame 150 the sim looks fine. Then the particles should stick more to the moving geometry ... but they fly away, so I will animate the attraction force, and slowly increase drag, from frames 160 to 200. Maybe it is not the right attitude, to patch the simulations like this. I am DOP beginner, I do quite simple motion design and learn, so far. Technically it should be fine, as long as I only animate the channels?

[LaidlawFX]

You can certainly do that. Whether or not it's the best method ¯\_(ツ)_/¯. Patching sims/renders is quite common to do art direction on. Just think of all the compositing that happens on renders as a correlated reference. My suspicions is you may not have enough processing hardware to waist cycles to re-run that sim again from scratch each time producing the same results? In that case hell yes! No need to redo the part that works already especially if it took a bunch of time to process. I guess the only thing I would say is when you make your patch try as be "non-destructive" as possible and make the fixes so that they work with the original sim controls. Like you said animating the key frames. That way in the event you need to re-run the sim again from scratch you can. I guess some ancillary advice is each time your run the sim, as part of the python process you are working on make sure you saved a version of that .hip along with the sim data so you can roll back to that source .hip and recompute that data. If you are in the final sprint to your deadline be destructive and go all photoshop and tweak the sim, but you don't want to loose all the benefits of being procedural too soon. 

[midicon]

On 8/29/2018 at 6:41 AM, LaidlawFX said:

 

@midiconYes you are right in a well thought out simple network you should never need the .sim data and just cache after your network to .bgeo.sc or the equivalent.

However, when building out a pipeline you need to factor in for all the edge cases. There are many factors that can cause you to need that data. 

Example 1. In a large production it's good to have the option to save the .sim for when you have many artist of varying skill levels. For instance, most staff in production hire a log to linear scale of artist with more junior to mid-level than senior artist. So you can not guarantee everyone will make a sim as cleanly. So a junior artist may not properly balance their sim.

Example 2. Especially with the nature of large productions and commercials you are always challenged to work on something new. Nobody wants to see the same commercial or feature film again. Which means invariable changes that you can account for. You won't always be working on the same sim for every project, especially in commercials this can change quite frequently.

Example 3. Even for Senior people working on the most advanced simulations you may be on the cutting edge of the technology and running into memory leaks. The deadline might be tight so you can't wait for SideFX to fix the issue.

Example 4. Another edge case is your are splitting the sim data on multiple machines. You can get quite tricky with this trying to balance intensely large simulations. Your sim may start on one machine but grow to use many computers.

Example 5. Often is the case some one let's a rogue sim go on the farm and they did not define the memory footprint correctly ahead of time, say they used the default of 32 GB when it really needs 64 GB. It competes with another sim or render on the same node causing that node to crash. Generally not knowing which caused the crash these are spun up again crashing more machines. If the render wrangler is not aware this can spiral out of control. Instead of restarting the sim and renders from scratch you can restart the sim on a new node. 

Mantra renders have the same concept with .checkpoint files these are the partially rendered images where you can start up a render from again. This really gets important as each generation of hardware demands higher resolution, so you need to produce images at the edge of the hardware capability. Also hardware in production has been purchased over several years, so not all of it will be up to the latest standards.

This is often a common conversation of when you should delete .ifd to in production? Delete directly after render, delete when render is done, wait a week then cull the data encase the ifd need to be re-rendered again with a few style sheet changes. 

 

I guess so you have a couple of situation where they might come in handy. I have worked on both large scale and small and have never run into a time that I needed to start the sim over midway. If I did it was because the sim was broken but I also am an FX TD, not a pipeline TD, so I can't disagree with you (although anyone else with a different answer, I would still argue the point that the sim is prolly messed up and they need to fix whatever is wrong and re-sim). Has these above-mentioned issues come into play at your studio? or are these preventive measure?   

Edited August 30, 2018 by midicon

[LaidlawFX]

Yeah they were in effect at R&H and I've maintained them going forward for the most part mostly for the failed farm nodes deal. I am not too heavy into simulations any more. I'm doing more world building now.

[ikoon]

So, I made few little scripts to work with my .sim checkpoint files. So ... it is useful for long non-heavy sims, with keyframed behaviour. Now I work on a background animation for a dancer event performance which is ~ 7200 frames. I plan to make the scripts even better ... to reset multiple DOPs with one hotkey press.

http://lex.ikoon.cz/sim-files-toolset

The script contains os.remove(file_path), so check it before use.

[fencer]

I'm too lazy)), just run *.bat in cache forlder
 

:loop
for /f "skip=3 eol=: delims=" %%F in ('dir /b /o-d /a-d *.sc') do @del "%%F"
timeout /t 10
goto loop

 

[ikoon]

My intention with this "toolset" was to build long sims (I do dancer performance synchronized visuals, something like "ENRA Visual Dance"). From my client, I have recording of the dance.

I prepare lots of forces, sources, pops, pyro, pop drag etc, animated "invisible" objects etc ... and then I keyframe those impacts.

So I use the .sim files as some "checkpoints". I edit the keyframes only in the preview range ... when it looks good, I move one or more ".sim step" further. I am not sure if it is the best workflow, I will work further on it.

Edited November 1, 2018 by ikoon

[LaidlawFX]

Cool Toolset