Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust <trond.myklebust@fys.uio.no>]

On Fri, 2006-03-31 at 15:21 +0200, Jakob Oestergaard wrote:
> On Fri, Mar 31, 2006 at 08:06:32AM -0500, Trond Myklebust wrote:
> > On Fri, 2006-03-31 at 14:45 +0200, Jakob Oestergaard wrote:
> > > On Fri, Mar 31, 2006 at 07:22:50AM -0500, Trond Myklebust wrote:
> > > ...
> > > > 
> > > > Some nfsstat output comparing the good and bad cases would help.
> > > 
> > > Clean boot on 2.6.15 and 2.6.14.7, one run of nfsbench with
> > > LEADING_EMPTY_SPACE=1.  I've skipped the NFS v2 stats because they're
> > > all 0.
> > 
> > Why all the GETATTR calls?
> 
> That's the $1000 question I guess :)
> 
> > Are you running with 'noac' set?
> 
> I didn't set it, that's for sure :)  I got the option lines from
> /proc/mounts, if there is any way I can get more/other information
> please advise.
> 
> 2.6.14.7:
>  rw,v3,rsize=32768,wsize=32768,hard,intr,udp,lock,addr=...
> 
> 2.6.15:
>  rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=...
> 
> Except for some formatting (proto=udp insted of udp) I fail to see any
> difference.
> 
> The way I see it, it seems like caching on the client side fails
> completely if the read requests are not aligned (to some
> buffer/block/page/whatever).
> 
> > I don't have a 2.6.15 kernel to run with, but on a recent git pull, I
> > get a total of 6 GETATTR calls when I run your nfsbench program.
> 
> The performance regression is present on 2.6.16.1 too.
> 
> Do you have a released kernel you can test with (2.6.1[56].*), or can I
> somehow get the kernel you're testing with, just so that we test on the
> same kernel?

Just apply
http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif

to a clean 2.6.16.

> > The number of READ calls is 1, and the number of WRITE calls is 161 (I'm
> > running with 64k wsize).
> 
> I can't set 64k wsize:
> 
> puffin:~# mount -o rw,wsize=65536,udp sparrow:/exported/joe /mnt
> puffin:~# cat /proc/mounts 
> ...
> sparrow:/exported/joe /mnt nfs rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=sparrow 0 0
> puffin:~#
> 
> Server is running a patched 2.6.11.11 kernel - could that be what's
> preventing me from 64k wsize?

Linux servers do not yet support anything larger than a 32k r/wsize (and
in any case, you would have to switch towards using TCP). My home
directory is on a filer.

> What happens if you run with 32k rsize/wsize?

Number of writes balloons to 321.

If I switch to UDP, I get no change.

Cheers,
  Trond