kswapd, kupdated, and bdflush at 99% under intense IO

[Jeff Lessem <Jeff.Lessem@Colorado.EDU>]

My machine is an 8 processor Dell P-III 700Mhz with 8GB of memory.
The disk system I am using is a 12 drawer JBOD with 5 disks in a raid
5 arrangement attached to an AMI Megaraid 438/466/467/471/493
controller with a total of 145GB of space.  The machine has been in
use for about 6 months doing primarily cpu and memory intensive
scientific computing tasks.  It has been very stable in this role and
everybody involved has been pleased with its performance.  Recently a
decision was made to conglomerate people's home directories from
around the network and put them all on this machine (hence the JBOD
and RAID).

These tests are all being done with Linux 2.4.3 + the bigpatch fix for
knfsd and quotas.  The rest of the OS is Debian unstable.

Before moving the storage into production I am performing tests on it
to gauge its stability.  The first test I performed was a single
bonnie++ -s 16096 instance, and the timing results are inline with
what I would expect from fast SCSI disks.

However, multiple instance of bonnie++ completely kill the machine.
Once two or three bonnies are running kswapd, kupdated, and bdflush
each jump to using 99% of a cpu and the machine becomes incredibly
unresponsive.  Even using a root shell at nice -20 it can take several
minutes for "killall bonnie++" to appear after being typed and then
run.  After the bonnies are killed and kswapd, kupdated, and bdflush
are given a minute or two to finish whatever they are doing, the
machine becomes responsive again.

I don't think the machine should be behaving like this.  I certainly
expect some slowdowns with that much IO, but the computer should still
be resonably responsive, particularly because no system or user files
that need to be accessed are on that channel of the SCSI controller.

Any advice on approaching this problem would be appreciated.  I will
try my best to provide any debugging information that would be useful,
but the machine is on another continent from myself, so without a
serial console I have a hard time getting any information that doesn't
make it into a logfile.

--
Thanks,
Jeff Lessem.