PDG/TOPS noob question for optimizing heavy scenes

[WhyGee]

Hi,

I have very basic experience with PDG/TOPS and would appreciate some advice.

I have a fairly computationally expensive setup we have been developing at work. I've tried to optimize it as much as I can but have seemed to hit a dead end. Some scenes could take up to 30 mins to compute and it doesn't seem to be maxing out my CPU most of the time. 

I was thinking if certain tasks especially for loops could be split up easily to run in parallel (compile nodes don't support some of the nodes we used which currently have no alternatives) that could drastically improve compute times. I've tried doing some simple tests with TOPS and conceptually it works but with significant overhead of loading and calculating the entire scene up to the point where I'm splitting tasks, and also I'm not sure if it's something with my setup, but even writing out small files from each tasks are quite slow. Anyone else experience this?

So after explaining briefly my intentions I am hoping to get some suggestions from people with more PDG experience. 

- Is it possible to not have PDG process the entire scene up to the "start" point in the graph where the tasks are split? what would be the best approach to minimize load times, possibly have it calculate once, then split the tasks?
- Is it mandatory for PDG task outputs to be written to disk? Is there a way to have sops merge the results without going through file caches?

Thanks alot!
/YG