2.6.5-rc2-mm1 - swapoff dies with OOM, why?

2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[John Stoffel]

Hi all,

I've run into a strange situation here.  I was having *terrible*
performance while doing a complile of the 2.6.5-rc2-mm2 kernel on my
system (Debian completely bleeding edge, plus udev and hotplug) along
with dealing with a USB problem where if I removed my Cuzer USB
device, it would never get de-allocated properly and the system load
would start to hang.

I've got 768mb of RAM, and 1gb of SWAP on the system, and I was using
about 110mb of swap, and around 360mb of cache, with around 20-30mb of
cache.  So using vmstat 1, I could see that the system was swapping in
just all the time, and the compile was just dog slow.  So I figured I
could get rid of some useless cache if I could just turn off swap.

No go.  Every time I tried, it would drop the swap down a bit (say
5-20mb), but then go OOM on me and die.  I thought this was completely
bogus.  If I've got the physical RAM available, and more cache and/or
buffer than is currently swapped to disk, why can't I kill swap?

John

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[Andrew Morton]

"John Stoffel" <stoffel@lucent.com> wrote:
>
>  I've run into a strange situation here.  I was having *terrible*
>  performance while doing a complile of the 2.6.5-rc2-mm2 kernel

Were you using rc2-mm1 at the time?  It had a memory leak in ext3.  Lots of
memroy was leaked, so swapoff cannot get sufficient memory to do its thing.

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[John Stoffel]

Andrew> Were you using rc2-mm1 at the time?  It had a memory leak in
Andrew> ext3.  Lots of memroy was leaked, so swapoff cannot get
Andrew> sufficient memory to do its thing.

Yup, it looks like that might be the problem here.  I've now gotten
control of the system again (since I also had a disk die, thankfully
the data is all mirrored!) at the same time, and when badblocks runs
away on your, it's a nasty thing.

So yes, I was running 2.6.5-rc2-mm1, and I've now just rebooted the
system into 2.6.5-rc2-mm4 (with -mm5 released as I type this... :-).
Things are looking better now, time to load test this system and see
how it handles again.

I'm still wondering why swapoff dies though.  Shouldn't it complete,
or at least have some way *to* complete if needed?  I realize, with a
memory leak in the filesystem, it's a hard thing to deal with.  

Anyway, thanks for the quick response, using -mm4 to compile -mm5 is
looking much better.  Much lower cache size, decent buffer, and still
some free memory.  Not to try a badblocks pass on my dead drive...

Here's some vmstat output (dual proc 550mhz Xeon, 768mb of RAM, -j3
compile)

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 7  0      0 543328  17176 111344    0    0    12     0 1086   749 80 16  1  4
 2  0      0 533408  17180 111408    0    0    16     0 1084   260 87 12  0  0
 5  0      0 531552  17188 111468    0    0     4     0 1084   509 90 10  0  0
 2  0      0 532448  17208 111652    0    0    48    56 1093   356 83 15  1  1
 3  0      0 538912  17248 113924    0    0    80  1624 1152   329 72 17  7  3
 3  0      0 522464  17256 113984    0    0     8   332 1146   330 84 10  6  0
 2  0      0 540832  17276 114236    0    0    20     0 1087   524 81 17  0  2
 2  0      0 539936  17292 114356    0    0    16     0 1087   388 85 15  0  0

Thanks Andrew,
John

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[Hugh Dickins]

On Mon, 29 Mar 2004, John Stoffel wrote:
> 
> I'm still wondering why swapoff dies though.  Shouldn't it complete,
> or at least have some way *to* complete if needed?  I realize, with a
> memory leak in the filesystem, it's a hard thing to deal with.  

If there's not enough freeable memory for what's out on swap, swapoff
cannot very well complete.  Either it can hang while other processes
get killed by the OOM killer once swapoff has filled memory (as 2.4),
until there's enough memory free to take in what's still needed from
swap; or the OOM killer can kill it off (as 2.6).  I much prefer the
2.6 behaviour - unlike many processes, swapoff can safely be restarted.
So the admin can then choose what else to kill, or add replacement
swap, then try the original swapoff again.

Hugh

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[John Stoffel]

>>>>> "Hugh" == Hugh Dickins <hugh@veritas.com> writes:

Hugh> On Mon, 29 Mar 2004, John Stoffel wrote:
>> 
>> I'm still wondering why swapoff dies though.  Shouldn't it complete,
>> or at least have some way *to* complete if needed?  I realize, with a
>> memory leak in the filesystem, it's a hard thing to deal with.  

Hugh> If there's not enough freeable memory for what's out on swap,
Hugh> swapoff cannot very well complete.  Either it can hang while
Hugh> other processes get killed by the OOM killer once swapoff has
Hugh> filled memory (as 2.4), until there's enough memory free to take
Hugh> in what's still needed from swap; or the OOM killer can kill it
Hugh> off (as 2.6).  I much prefer the 2.6 behaviour - unlike many
Hugh> processes, swapoff can safely be restarted.  So the admin can
Hugh> then choose what else to kill, or add replacement swap, then try
Hugh> the original swapoff again.

But in this case, there was no way to force the turning off of swap,
since the ext3 bug in 2.6.5-rc2-mm1 had filled the cache, and wasn't
going away.  Is this right?  

I wonder if there's a way to tell swap to 'go away when you can, don't
allow more swap (kill new requests), and generally work on pushing
stuff back to memory, or other swap partitions.'

This doesn't have to be the default, but it would be nice to have a
big hammer to beat on the system at times.

Thanks for the feedback Hugh, I do appreciate it.

John

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[Hugh Dickins]

On Mon, 29 Mar 2004, John Stoffel wrote:
> 
> But in this case, there was no way to force the turning off of swap,
> since the ext3 bug in 2.6.5-rc2-mm1 had filled the cache, and wasn't
> going away.  Is this right?  

Well, it was a memory leak, filling more and more resident memory,
leaving less and less for other uses.  I didn't investigate, but
I'm guessing not cache.

> I wonder if there's a way to tell swap to 'go away when you can, don't
> allow more swap (kill new requests), and generally work on pushing
> stuff back to memory, or other swap partitions.'

User space thing really: you could add an option to swapoff(8) to loop
retrying (well, I think it would need to fork off a child to do the
sys_swapoff, fork another when that one gets OOM-killed; a bit messy,
I agree).

Hugh

Re: 2.6.5-rc2-mm1 - swapoff dies with OOM, why?

[Felipe Alfaro Solana]

On Mon, 2004-03-29 at 04:38, John Stoffel wrote:
> Hi all,
> 
> I've run into a strange situation here.  I was having *terrible*
> performance while doing a complile of the 2.6.5-rc2-mm2 kernel on my
> system (Debian completely bleeding edge, plus udev and hotplug) along
> with dealing with a USB problem where if I removed my Cuzer USB
> device, it would never get de-allocated properly and the system load
> would start to hang.

Are you using ext3? 2.6.5-rc2-mm1 has a memory leak that affects ext3
code. Thus, after some uptime of disk intensive work, nearly all memory
is wasted up. Please, upgrade to latest -mm tree (which ATM is
2.6.5-rc2-mm4).