2.6.5-rc2-mm1 - swapoff dies with OOM, why?

["John Stoffel" <stoffel@lucent.com>]

Hi all,

I've run into a strange situation here.  I was having *terrible*
performance while doing a complile of the 2.6.5-rc2-mm2 kernel on my
system (Debian completely bleeding edge, plus udev and hotplug) along
with dealing with a USB problem where if I removed my Cuzer USB
device, it would never get de-allocated properly and the system load
would start to hang.

I've got 768mb of RAM, and 1gb of SWAP on the system, and I was using
about 110mb of swap, and around 360mb of cache, with around 20-30mb of
cache.  So using vmstat 1, I could see that the system was swapping in
just all the time, and the compile was just dog slow.  So I figured I
could get rid of some useless cache if I could just turn off swap.

No go.  Every time I tried, it would drop the swap down a bit (say
5-20mb), but then go OOM on me and die.  I thought this was completely
bogus.  If I've got the physical RAM available, and more cache and/or
buffer than is currently swapped to disk, why can't I kill swap?

John