Problem with 2.6 kernel and lots of I/O

[Roy Keene <rkeene@psislidell.com>]

Hello,

 	I have a (well, at least one) show-stopping problems with the 2.6 
kernel while doing heavy I/O.  I have a (software) RAID1 of network block 
devices (nbd0 and nbd1) set up on two identical machines in an 
active-passive HA cluster configuration.  When the "primary" node goes 
down and comes back up it recovers the RAID as follows:

 	Start RAID in degraded mode with remote device (nbd1)
 	Hot-add local device (nbd0)

This all works.  Hot-adding the local device causes a resync and that is 
where the problems begin.  Once the resync begins the system becomes 
unusable.  Anything that wants to write something to the syslog socket 
("/dev/log") syncronously hangs until the resync completes.  The system 
load goes up to 18 or so.  Writing stuff to the local disk ("/etc" for 
example, which is not part of the RAID) sometimes hangs.  When the resync 
is complete everything is happy again.  Resyncing takes about 25 minutes 
(20GB over a dedicated network interface to the client at 1000Mbps) and 
makes the recovery time unacceptable.  Also, during this recovery the OOM 
killer will occasionally be invoked and kill something randomly even 
though there is typically plenty of unused swap lying around before 
(though perhaps "java" just decides to eat all of that VERY quickly and I 
don't notice this, since that is what the OOM killer choses to kill.)

Does anyone have any ideas ?


Information about the systems:

Info: Linux cog1 2.6.9-5.0.5.ELsmp #1 SMP Fri Apr 8 14:29:47 EDT 2005 i686 i686 i386 GNU/Linux
Dist: RedHat Enterprise Linux 4
Spec:
     2 x 3.2GHz Xeon (each system, with hyperthreading so 4 logical processors)
     4GB of physical RAM
     2GB of configured swap (partition, contigious)
     2 x 1000Mbps (Intel 82546GB) network cards (HA cluster link is
               provided by a cross over cable between the two nodes)