Strange NFS client behavior on 2.6.6 and higher {Scanned}

Strange NFS client behavior on 2.6.6 and higher {Scanned}

[Rett D. Walters]

I have been working towards moving my systems to kernel
2.6 and I have noticed that 2.6.6 and higher NFS client
code exhibits some strange behavior when writing files
using cat /dev/video0 > <some file on NFS mount>.

Using a 2.4 client, the file slowly counts up as data is
written when pushing to a 2.4 server.  Using 2.6.6 against
a 2.6 server the file is written in large multi-megabyte
clumps instead.  However using a 2.6.6 client against a
2.4 server acts just like it used to, with a "trickle"
write.  A tcpdump trace of the 2.6.6 client against a 2.6
server show no traffic being transmitted and then suddenly
a burst of 20,000 packets sent then nothing, until the
next burst.

It appears to me that the 2.6.6 client against a 2.6
server scenario that the 2.6.6 client is caching the data
then writing it in these large clumps.  As a data comm
engineer by profession, this seems a little strange. 
Sending 20000 packets in large, very fast bursts will
increase the likelyhood of causing congestion at the
receiver which could lead to packet loss. I am using NFS
v3 over UDP, and have tried using async and sync, and
setting the rsize/wsize to 8k etc to no avail.

The /dev/video0 device is an MPEG encoder card.  Encoding
at  8Mbit/sec.

Another concern I have here is that in kernel versions
higher than 2.6.6 the cat seems to hang and never write
data to the nfs mount after about 70 or so MB, no matter
if its going to a 2.6 server or a 2.4 server.

Can anyone out there in kernel land shed some light on
this behavior?

Thanks in advance

Rett Walters

Re: Strange NFS client behavior on 2.6.6 and higher {Scanned}

[Trond Myklebust]

På fr , 20/08/2004 klokka 16:04, skreiv Rett D. Walters:

> Using a 2.4 client, the file slowly counts up as data is
> written when pushing to a 2.4 server.  Using 2.6.6 against
> a 2.6 server the file is written in large multi-megabyte
> clumps instead.

Probably. The VM decides when to push out data in 2.6.x. In 2.4.x we had
artificial hard limits on the number of dirty pages that were allowed to
exist, now all that is controlled by the VM.

>   However using a 2.6.6 client against a
> 2.4 server acts just like it used to, with a "trickle"
> write.

Huh?

>   A tcpdump trace of the 2.6.6 client against a 2.6
> server show no traffic being transmitted and then suddenly
> a burst of 20,000 packets sent then nothing, until the
> next burst.

See above. Note that this is pretty much the way block devices work
too...

> It appears to me that the 2.6.6 client against a 2.6
> server scenario that the 2.6.6 client is caching the data
> then writing it in these large clumps.  As a data comm
> engineer by profession, this seems a little strange. 
> Sending 20000 packets in large, very fast bursts will
> increase the likelyhood of causing congestion at the
> receiver which could lead to packet loss.

That's why we have added TCP support as well as congestion control
algorithms for UDP. I've seen no signs of new congestion issues with
2.6.x in tests on a clean network. Rather I've seen vastly improved
write performance against faster servers precisely because the new
VM-driven stuff allows us to cache more data both on the client and the
server side.

> I am using NFS v3 over UDP, and have tried using async and sync, and
> setting the rsize/wsize to 8k etc to no avail.

It sounds to me as if you really want to be using O_SYNC or (better
still) O_DIRECT writes. You basically want to stream the data to the
server immediately without any caching right?

The "sync" mount option should also give you similar behaviour.

Cheers,
  Trond