distributed simulation with H15

[borisb2]

Hi.

I'm setting up distributed simulations on a small farm with tractor. It all seems to work ok, simtracker gets started, the slices are started, synced and doing their job. I also can measure expected time-benefits when simulating on 2 or 4 machines ..

BUT ..

I can see barely any benefits in memory usage, whether I am doing flip (with pressure solve) or pyro .. It all takes up the same amount of RAM, no matter if I simulate single, on 2 machines or more. Maybe 1-2GB benefit on a 30GB flip sim. Is that expected? I can see that the flip points are half the amount per slice (when distributed on 2 machines), the volumes are half the voxels etc. .. but it still takes the same amount of RAM - which makes the whole point of distributing a bit meaningless.

What is wrong here? Did I forget something? The only thing I added in dops (besides the slicing plane) is the gasnetsliceexchange as well as setting the address etc. in distribution tab of course .. as mentioned it seems to work fine (according to simtracker.py).

does anybody noticed similar behaviors?

Edited December 14, 2016 by borisb2

[Solitude]

I had recently sliced up a sim into 4 parts -- on a single machine it was 24 gigs or so, on the 4 machines it was about 12 gigs each.  I only had a single collision volume and a single source. Do you have anything extra complex going on?

[borisb2]

Hmm, its a customized wavelayer-setup (120m x 50m) with one moving collision object (which is also used to drive the moving boundaries.. so nothing too special.

Particle separation is at .085, roughly (only) 14mio particles, 16mio voxels. Single simulated it takes up about 30gb, sliced up on 2 machines it takes up about 25gb each - but at least half the sim-time (which is good)

I also noticed sometimes the sim freezes after 30-50 frames for no reason (simtracker doesnt show pending)..