I/O issues, iowait problems, 2.4 v 2.6

[Paul Venezia <pvenezia@jpj.net>]

I've been running some benchmarks on 2.4 v 2.6 recently, and came across
another oddity. 

Running smbtorture's NBENCH test against the 2.6 kernel shows a
significant performance disparity vs Redhat 2.4.20 or 2.4.22. The target
system is running RH AS 3.0, and is an IBM x335 dual P4 XEON with 1.5GB
RAM, Broadcom gigabit NIC linked at 1000/full and an MPT RAID
controller.

Running a 12-client NBENCH test against this server running 2.4.22
consistently produces a result of ~33MB/s. Running 2.6.0-test9 through
bk-11 however, produces a much lower result, usually ~14MB/s. The test
will start at ~80MB/s, sustained for 10-15 seconds, then throughput
drops precipitously, and the file transfers slow to a crawl. The target
system shows that it's 100% I/O bound, but I can't seem to locate the
constraint. iostat and sar don't show anything out of the ordinary, but
top shows the CPUs at 99% iowait. Eventually, the bottleneck disappears,
and the performance increases, but never substantially. 

I can reproduce this at will on this system, and a dual Itanium2 system
with Samba 2.2.8a and 3.0. Removing Samba from the equation, copying a
550MB file via NFS takes 240s under 2.6.0-test9-bk11 (~2.5MB/s average),
exhibiting the same iowait problem, while under 2.4.22 the same transfer
takes ~13s (~44MB/s average) without any iowait issues. A raw IP
throughput test shows ~900Mb throughput between the two boxes.

This could be a driver issue, but I don't have any other test boxes at
the moment. I can provide any debug info requested.

-Paul