Overcommit problems with 2.6.12-rc4 (on AMD64)

Overcommit problems with 2.6.12-rc4 (on AMD64)

[Steinar H. Gunderson]

(Please Cc me on answers, I don't follow LKML.)

Hi,

Suddenly one of our servers, a Dual Opteron with 2GB memory (running 32-bit
userland, but 64-bit kernel) started to behave oddly:

  imapd[31528]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffd394 error 4
  imapd[31527]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffcbe4 error 4
  sh[31530]: segfault at 00000000ffff7ff4 rip 000000005555e556 rsp 00000000ffff7ff8 error 6
  sh[31531]: segfault at 00000000ffff7e5c rip 00000000555dc575 rsp 00000000ffff7e60 error 6
  Unable to load interpreter /lib/ld-linux.so.2
  Unable to load interpreter /lib/ld-linux.so.2
  (ad infinitum)

It turned out it had some sort of memory problem:

  Jun  2 11:56:02 cassarossa smbd[7171]: oplock_break: malloc fail for input buffer. 
  Jun  2 11:56:02 cassarossa smbd[7171]: open_mode_check: FAILED when breaking oplock (3) on file login.bat, dev = 900, inode = 110665 

This wasn't a RAM problem, as the machine has ECC RAM and we received no
warnings from it. Also, we definitely had enough swap:

  cassarossa:~# free
               total       used       free     shared    buffers     cached
  Mem:       2058300    2041136      17164          0      39576    1601468
  -/+ buffers/cache:     400092    1658208
  Swap:      3903712          0    3903712

It looks like somehow, the kernel couldn't really distinguish between memory
used as cache and just "used". It couldn't even swapoff:

  cassarossa:~# swapoff -a
  swapoff: /dev/sda5: Cannot allocate memory
  swapoff: /dev/sdf5: Cannot allocate memory

However, we run with vm.overcommit_memory=2, so we figured out it was worth a
shot:

  cassarossa:~# echo 0 > /proc/sys/vm/overcommit_memory 
  cassarossa:~# swapoff -a
  cassarossa:~# swapon -a 
  cassarossa:~# free -m
               total       used       free     shared    buffers     cached
  Mem:          2010       1993         16          0         39       1595
  -/+ buffers/cache:        358       1651
  Swap:         3812          0       3812

Suddenly everything seems to be back to normal (ie. we could swapoff, and the
programs stopped running out of memory; no changes in the cache used,
though), and after a quick restart of services, everything is back to normal.
So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64.
Any ideas why this would happen?

for the record:

  cassarossa:~# uname -a
  Linux cassarossa 2.6.12-rc4 #1 SMP Fri May 13 18:49:40 CEST 2005 x86_64 unknown

No kernel patches except for a microscopic forward-port of the ELF fix from
2.6.11.9.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Overcommit problems with 2.6.12-rc4 (on AMD64)

[Alan Cox]

On Iau, 2005-06-02 at 11:18, Steinar H. Gunderson wrote:
>   imapd[31528]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffd394 error 4
>   imapd[31527]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffcbe4 error 4
>   sh[31530]: segfault at 00000000ffff7ff4 rip 000000005555e556 rsp 00000000ffff7ff8 error 6
>   sh[31531]: segfault at 00000000ffff7e5c rip 00000000555dc575 rsp 00000000ffff7e60 error 6
>   Unable to load interpreter /lib/ld-linux.so.2
>   Unable to load interpreter /lib/ld-linux.so.2
>   (ad infinitum)

You ran out of address space

> Suddenly everything seems to be back to normal (ie. we could swapoff, and the
> programs stopped running out of memory; no changes in the cache used,
> though), and after a quick restart of services, everything is back to normal.
> So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64.
> Any ideas why this would happen?

vm.overcommit_memory=2 prevents the possibility of overcommitting - ie
of address space being allocated to someone which is not used. Your swap
allocation data is showing pages allocated not pages that could be
allocated due to page faults. You need to look as the AS figures in
/proc/meminfo to see the address space committed.

Basically it went back to "sane" because you said "ok I might get OOM
but take the gamble", and since many programs allocate lots of space
they never touch it worked.