Disk IO Slowly Grinds to a (near) Halt

["David Rees" <drees76@gmail.com>]

I've got a server running Fedora 9 server running kernel
2.6.27.9-73.fc9 where inevitably, any sort of IO activity slows down
significantly.

The machine is a basic Athlon 64 X2 5000+, 3GB RAM and 4 disks
(2x250GB IDE and 2x1TB SATA) running in two software RAID1 arrays.

One IDE drive is connected to the onboard (Nvidia MCP51 pata_amd), the
other connected to a Promise (PDC20268 pata_pdc2027x) controller.

Both SATA drivers are connected to the onboard Nvidia MCP51 sata_nv controller.

I suspect that the culprit has something to do with the fact that this
machine acts as a BackupPC server, and as such, has a filesystem with
over 7million inodes on it (about 175GB in use on that partition), but
that's just a hunch.

A simple test to confirm the issue is to drop the caches, and then do
a directory listing on an empty directory.  Doesn't matter which raid1
array it's on.  On the normally performing systems I tested this
typically took 0.15-0.45 seconds.  On this slow system, it takes over
a second for the same test to run when it's acting up.  After a fresh
reboot, it takes less than 0.2 seconds.

Any ideas? The only thing that seems to help is rebooting the server.
Let me know if there is any more information I can provide that would
be helpful.

-Dave