linux-2.4.20-pre8-ac3: NFS performance regression

linux-2.4.20-pre8-ac3: NFS performance regression

[Andreas Pfaller]

While trying to get the onboard HPT372 working on a new motherboard
(Epox 8k5a2+ - KT333/VT8235) i tried successfully 2.4.20-pre8-ac3.
2.4.19 could read the fs on the HPT372 without any problem at full
media speed but could not write to it at all. I can provide details
if necessary. I did not use any of the raid stuff.

However I noticed a significant NFS performance drop with
2.4.20-pre8-ac3. Other network throughput is not affected.

Server: P-II 300, cheap RTL8139C, 2.4.19 (nearly vanilla).
Client: Athlon 2000+, onboard via-rhine.

### Client with linux-2.4.19 ###

[diavolo inferno-l2#> bonnie -s 1500 -S 100 -y -u
Bonnie 1.3: File './Bonnie.976', size: 1572864000, volumes: 1
Writing with putc_unlocked()...done:  10328 kB/s  10.5 %CPU
Rewriting...                   done:   4280 kB/s   3.2 %CPU
Writing intelligently...       done:  10319 kB/s   3.8 %CPU
Reading with getc_unlocked()...done:  10436 kB/s  14.3 %CPU
Reading intelligently...       done:  10712 kB/s   2.8 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              ---Sequential Output (sync)----- ---Sequential Input-- --Rnd Seek-
              -CharUnlk- --Block--- -Rewrite-- -CharUnlk- --Block--- --00k (03)-
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU
diavol 1*1500 10328 10.5 10319  3.8  4280  3.2 10436 14.3 10712  2.8    9.3  3.0

# ftp transfer
1072458941 bytes received in 01:37 (10.53 MB/s)


### Client with linux-2.4.20-pre8-ac3 ###

[diavolo inferno-l2#> bonnie -s 1500 -S 100 -y -u
Bonnie 1.3: File './Bonnie.987', size: 1572864000, volumes: 1
Writing with putc_unlocked()...done:   7356 kB/s   7.6 %CPU
Rewriting...                   done:   3168 kB/s   1.3 %CPU
Writing intelligently...       done:   6939 kB/s   1.7 %CPU
Reading with getc_unlocked()...done:   7389 kB/s   7.0 %CPU
Reading intelligently...       done:   7722 kB/s   2.1 %CPU
Seeker 1...Seeker 3...Seeker 2...start 'em...done...done...done...
              ---Sequential Output (sync)----- ---Sequential Input-- --Rnd Seek-
              -CharUnlk- --Block--- -Rewrite-- -CharUnlk- --Block--- --00k (03)-
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU
diavol 1*1500  7356  7.6  6939  1.7  3168  1.3  7389  7.0  7722  2.1    9.9  3.8

[diavolo inferno-l2#> cat /proc/mounts | grep l2
inferno:/mnt/l2 /mnt/inferno-l2 nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=inferno 0 0

# ftp transfer
1072458941 bytes received in 01:35 (10.69 MB/s)

Cheers,
Andreas

PS: Does a patch for only the HPT372 problem exist for 2.4.19? I get
somewhat nervous running an pre-ac kernel on a machine intended for
production use.

Re: linux-2.4.20-pre8-ac3: NFS performance regression

[Alan Cox]

On Thu, 2002-10-03 at 19:32, Andreas Pfaller wrote:
> However I noticed a significant NFS performance drop with
> 2.4.20-pre8-ac3. Other network throughput is not affected.

I see this with all recent 2.4.20pre and 2.4.20pre-ac kernels. I've not
had time to retest with Trond's fixes to recheck it all

Re: linux-2.4.20-pre8-ac3: NFS performance regression

[Trond Myklebust]

>>>>> " " == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

     > On Thu, 2002-10-03 at 19:32, Andreas Pfaller wrote:
    >> However I noticed a significant NFS performance drop with
    >> 2.4.20-pre8-ac3. Other network throughput is not affected.

     > I see this with all recent 2.4.20pre and 2.4.20pre-ac
     > kernels. I've not had time to retest with Trond's fixes to
     > recheck it all

FYI, here is the 'fix' Alan is talking about. It could be worth
trying...

Cheers,
  Trond

--- linux/net/sunrpc/xprt.c.orig	Fri Aug 30 20:16:17 2002
+++ linux/net/sunrpc/xprt.c	Tue Sep 24 00:08:59 2002
@@ -171,10 +171,10 @@
 
 	if (xprt->snd_task)
 		return;
-	if (!xprt->nocong && RPCXPRT_CONGESTED(xprt))
-		return;
 	task = rpc_wake_up_next(&xprt->resend);
 	if (!task) {
+		if (!xprt->nocong && RPCXPRT_CONGESTED(xprt))
+			return;
 		task = rpc_wake_up_next(&xprt->sending);
 		if (!task)
 			return;
@@ -1013,7 +1013,6 @@
 		}
 		rpc_inc_timeo(&task->tk_client->cl_rtt);
 		xprt_adjust_cwnd(req->rq_xprt, -ETIMEDOUT);
-		__xprt_put_cong(xprt, req);
 	}
 	req->rq_nresend++;
 
@@ -1150,10 +1149,7 @@
 		req->rq_bytes_sent = 0;
 	}
  out_release:
-	spin_lock_bh(&xprt->sock_lock);
-	__xprt_release_write(xprt, task);
-	__xprt_put_cong(xprt, req);
-	spin_unlock_bh(&xprt->sock_lock);
+	xprt_release_write(xprt, task);
 	return;
  out_receive:
 	dprintk("RPC: %4d xmit complete\n", task->tk_pid);

Re: linux-2.4.20-pre8-ac3: NFS performance regression

[Andreas Pfaller]

On Friday 04 October 2002 00:50, Trond Myklebust wrote:
> >>>>> " " == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
>      > On Thu, 2002-10-03 at 19:32, Andreas Pfaller wrote:
>      >
>     >> However I noticed a significant NFS performance drop with
>     >> 2.4.20-pre8-ac3. Other network throughput is not affected.
>     >>
>      > I see this with all recent 2.4.20pre and 2.4.20pre-ac
>      > kernels. I've not had time to retest with Trond's fixes to
>      > recheck it all
>
> FYI, here is the 'fix' Alan is talking about. It could be worth
> trying...

That 'fix' is already included in 2.4.20-pre8-ac3.

Cheers,
Andreas

Re: linux-2.4.20-pre8-ac3: NFS performance regression

[Bryan O'Sullivan]

On Thu, 2002-10-03 at 15:50, Trond Myklebust wrote:

> FYI, here is the 'fix' Alan is talking about. It could be worth
> trying...

Trond -

I'd seen a similar problem under 2.4.19 with your NFS-ALL patch
applied.  The client would lock up for periods of up to several minutes
at a time without generating any traffic, then would wake up and
retransmit.  This would continue for a few seconds (rarely more than 30
seconds of uninterrupted IO) before another huge pause-and-retransmit
cycle began.

Your patch restores client performance to sane levels.  It applies
cleanly against 2.4.19 with NFS-ALL.  I suggest you republish NFS-ALL
with this patch applied, in case other people who are stuck with 2.4.19
run into this problem.

Thanks,

	<b