slow saving of hip files over network

[BrianK]

It takes a really long time (5-10 seconds) to save hip files over our NFS share. Is this typical or is it an NFS issue?

While we're at it, any suggestions on how to speed up NFS? I have full gigabit everywhere, but usually only see about 10MB/sec (which is 100 Mb). I run async, have experimented with all sorts of different block sizes, have subtree checking turned off. Ideas, suggestions? I think it may be a ram issue - my ram fills up and stays full... up to the point at which it's filled, NFS goes really quick; after that point, it slows to 100Mb. hmmmm....

[Jason]

It takes a really long time (5-10 seconds) to save hip files over our NFS share.  Is this typical or is it an NFS issue?

We're on NFS and its pretty quick. It only takes time if you have 1000's of nodes and locked geometry, etc - file sizes over 20mb. Most of my scenes are about 1mb and save in the blink of an eye.

[edward]

Does that sound like your network card is configured at a slower speed?

If all else fails, there's always the environment variable HOUDINI_BUFFEREDSAVE

[BrianK]

Does that sound like your network card is configured at a slower speed?

If all else fails, there's always the environment variable HOUDINI_BUFFEREDSAVE

13956[/snapback]

I've tried HOUDINI_BUFFEREDSAVE to no avail.

The NIC's are setup properly - I can ftp or scp much faster than NFS... still not nearly 1000Mb, but 300Mb or so.

[sibarrick]

We have the same problem on a windows network. Photoshop saves a 100Mb file no problem in about 10 seconds or so. Houdini however can't save anything over about 5Mb without completely grinding to a halt, to save a 10Mb file can take about 15 minutes. This has been true since the first release of Houdini, we actually now have the working practise of saving to a local disk and then copying it to the server as it is quicker. Our netwrok is all gigabit but it doesn't help Houdini....

[edward]

This is true even with HOUDINI_BUFFEREDSAVE?

[sibarrick]

This is true even with HOUDINI_BUFFEREDSAVE?

14061[/snapback]

Don't know, I've never heard of that setting before, I'll try it out.

It works, after all these years, why isn't this the default?

[dmaas]

Some tips:

1. tell NFS to use the TCP protocol instead of UDP. "proto=tcp" mount option. This nearly doubled NFS performance on my network.

2. make sure you are using NFS version 3 ("vers=3" mount option). This is required for >2GB files anyway.

3. set rsize and wsize ("rsize=" and "wsize=" mount options). I found the fastest value for my network was 8192.

4. Unless both your client and server have 64-bit PCI busses, don't expect more than about 60MB/sec from NFS.

5. With my 64-bit PCI server and PCI Express client, I can read files from the server at 105MB/sec .

[BrianK]

Some tips:

1. tell NFS to use the TCP protocol instead of UDP. "proto=tcp" mount option. This nearly doubled NFS performance on my network.

this sounds interesting... in doing my homework (after I read this), it appears it may solve a good bit of problems over here. That said, I can't find any documentaion *anywhere* on how to tell my server to serve over TCP...

# rpcinfo -p | grep nfs

100003 2 udp 2049 nfs

100003 3 udp 2049 nfs

# cat /etc/services | grep nfs

nfs 2049/tcp nfsd

nfs 2049/udp nfsd

??? How do I get nfs to run over TCP?

3. set rsize and wsize ("rsize=" and "wsize=" mount options). I found the fastest value for my network was 8192.

I've tried several block sizes all the way up to 16K (maybe higher? I don't remember). While I did notice some slow down when the block size was very small, the block sizes above about 2K were all very similar.

4. Unless both your client and server have 64-bit PCI busses, don't expect more than about 60MB/sec from NFS.

5. With my 64-bit PCI server and PCI Express client, I can read files from the server at 105MB/sec .

15291[/snapback]

... interesting ... so I should only expect 60 MB/sec to get from the NIC to memory (or HD) on a typical 32 bit PC? Forgive my lack of comp. engineering knowledge, but which part is the slow down, the NIC, the path the RAM, or the path to HD?

In other words, how can I fix this part? Should I buy a new mobo with PCI Express? If so, should I get a PCI express NIC and hard drive (RAID) controller? Are people making those yet?

105 MB/sec?!?! that's fert near 100% efficiency! WOW! Even on my best days, I rarely see more than about 30%

Thanks for th response! Best answer I've had yet (and I've been working on this problem for quite some time).

[dmaas]

How do I get nfs to run over TCP?

On the server, it's a kernel configuration option. "Provide NFS over TCP support". It's marked "experimental," so you might need to set the "code quality option" to see it. (this assumes you are compiling your own kernels. If not, you'll have to see if your vendor's kernel supports it).

On the client side, you just need the "proto=tcp" mount option. Typically this will go in /etc/fstab. Here is the fstab entry for one of my NFS mounts:

server2:/storage/3ware /storage/3ware nfs vers=3,proto=tcp,hard,intr,rsize=8192,wsize=8192 0 0

which part is the slow down, the NIC, the path the RAM, or the path to HD?

The PCI bus is the bottleneck. Your NIC and disk controllers share the PCI bus, and it is the only pathway to get data from those into RAM. Standard 32-bit PCI is supposed to have a peak bandwidth of 133MB/sec, but that is rarely seen in practice.

On the client you just have a problem with getting data from the NIC to RAM over the PCI bus. On the server you have a more serious problem because the disk controller must also share the PCI bandwidth. So the most you would ever see pulling data from disk to the network on 32-bit PCI is half of 133MB/sec.

Both 64-bit PCI and PCI Express are fast enough (>400MB/sec) that you won't encounter any problems at gigabit rates.

Note that quality of your NIC hardware and its driver are also important. There are some crappy NICs out there that won't achieve bus/wire speed under any circumstances. Stick with the "reputable" vendors (Intel, 3Com, etc). I use Intel PRO/1000 ethernet interfaces.

The setup that got me 105MB/sec was:

server: Intel motherboard, 64-bit PCI, 3Ware RAID controller (8 disk SATA RAID-5), Intel PRO/1000 NIC (on motherboard), Linux 2.4.28

client: Dell Precision 670, PCI Express, Intel PRO/1000 NIC (on motherboard), Linux 2.6.9.

Writing speed is somewhat lower than reading speed, but I haven't measured it precisely.

I think an on-board ethernet interface is the same thing as having a NIC on the fastest bus the motherboard supports. So if you get a PCI Express motherboard with integrated ethernet, it's like having a PCI Express NIC. (not 100% sure of that)

Regarding the behavior where file saving goes quickly at first but then slows down - this is because of the Linux page cache. At first the data gets buffered in RAM without going to the network. Once enough has accumulated, the kernel starts sending the data to the server, which is when things slow down. Linux tends to buffer things in RAM a lot more than Windows, so this can be a surprise to Windows users.

[BrianK]

Wow, thanks for the great info. It appears that my kernel does not support NFS over TCP:

# grep NFSD_TCP /boot/config-2.4.20-8

# CONFIG_NFSD_TCP is not set

Unfortunately, I don't know if I can rebuild due to the fact that my (very very crummy) Promise SATA RAID controller requires a very specific kernel build.

While on the subject of crummy RAID controllers, might I reccommend against anyone purchasing a Promise RAID controller. They are little more than low quality SATA controllers with proprietary software RAID management.

On bootup one faitful day, my Promise Fasttrack S150 SX4 did not bring up one of the hard drives (for no apparant reason), but continues to boot and run normally. I did not discover the problem until my daily log watch e-mailed me - the RAID message was buried in with my other typical warnings ("root has logged in"... and such). Between bootup & my catching the problem, the RAID 5 array was in a "critical state" (as defined by Promise) in which ALL writes to the drive corrupted their containing folder and all subfolders - i.e., one write to the root of the array corrupts the entire array. Promise tech support said, "well, you'll need to buy third party recovery software to get your data back."

I've since rebuilt the reiser filesystem (thanks to reiserfsck), but will be replacing the Promise card with a 3Ware card as soon as time allows.

Thanks again for the fantastic advice on my NFS woes. I now, at least, have a direction to go to get up to speed.

[dmaas]

The 3Ware cards are great. As far as I know they are the only IDE/SATA cards that do actual hardware RAID. Make sure you get one of the 9000 series cards, not the older 8000 series. The 9000s perform quite a bit better.

I'm not sure if the 9000 driver is in the main Linux kernel yet, if not, you can get the sources from 3Ware's website and drop them into the kernel tree.

Here are some benchmarks I did with my 9000 with 8 250GB SATA disks

(raw I/O rates to/from RAM on the server itself):

RAID 0 - 190MB/sec read, 130MB/sec write

RAID 5 - 180MB/sec read, 90MB/sec write

RAID 50 - 125MB/sec read, 78MB/sec write

I also tried making two hardware RAID-5s and striping them with Linux software RAID. That was slower (87/42).

[edward]

Don't know, I've never heard of that setting before, I'll try it out.

It works, after all these years, why isn't this the default? 

14081[/snapback]

Because it was uses an excessive amount of memory. I blame MS. There's no reason for it to be slow when using their iostream library. Instead, HOUDINI_BUFFEREDSAVE generates an entire copy in RAM first and then sends one single write call to the OS instead of doing it in multiple chunks.