Data Storage....

[Macha]

Don't know if this is the right forum but we're having huge trouble with storing and transferring our data. We move about gigabytes of ifd files and geometry and images, movies and renderfarm and whatnot and it's just not working anymore. It's slow, it's expensive. Our fuses blow...

We have a small NAS server, dabbled with an iSCSI, shared network drives... We're not network experts as you can see. What kind of infrastructure would you suggest? Where to start?

We have about 30 computers.

Edited November 19, 2010 by Macha

[pclaes]

I don't know much about the network, but when we started to run into issues here it was because effects artists tend to generate lots of caches and lots of IO. So they provided us with these nice 2Tb raid 5 towers as "personal caching space". They are just for caching and they are a lot faster than doing that over the network. Our scene files are of course stored on the network and backed up.

In regards to IFD files.. almost everything is read archives/delayed loads, that saves a lot on disk space when generating iterations. So the final cache goes onto the network and gets then picked up by the renders, all other caches are on the "local drive".

Also old stuff (caches/rib-ifd files/not approved renders) gets deleted after 2 weeks to make space. The raid drives are managed by the artists.

Sorry I can't be more specific, I just don't know.

[Macha]

Peter, with regard to delayed loads: I noticed that if I use them with large geometries the ifd still stays small. I assumed that is because the geometry is not included in the ifd. In that case you would still have to get them on the network for the render slaves to see, right? (I have only tried it locally so far). Can we perhaps do something like ifd.gz? And also, delayed load shaders load all everything into memory at once, or am I mistaken there? Our render slaves have different specs from the artists machines so I'm not sure how they handle this.

But yes, personal caching space is what I thought of too. It's good to hear that it's a working solution.

[pbowmar]

We have a Isilon NAS, everything is on it, with 10gigabit core switches, gigabit to the desktops. Not horribly expensive, unless it's for home use, but well, well worth it. Super stable, fast and reliable.

Naturally, when things are going full-bore, you can detect slowdowns sometimes, but nothing that grinds the studio to a halt.

All Linux with some OSX machines, the OSX machines do have some speed issues with the NAS, which we're trying to figure out but Apple aren't the most helpful

Cheers,

Peter B