Hi,


vanilla linux 2.6.29.3, AMD64, tested on openSUSE 11.1 and Kubuntu 9.04.


I observe the following behaviour:

When any user application (non-kernel, non-root) consumes all the
available system memory, the system freezes completely instead of any
application being killed by the oom-killer.

The mouse pointer stalls, and there is a seemingly endless loop of
hard-disc access, even when no swap space on any harddisc is activated.

IMO, this is severe, because any application can practically crash the
system (e.g. in case of a memory-leak), causing data loss in case of
unsaved data.


To reproduce this, I've attached a small C++ utility
(compiles with g++ memory_overcommit.cc -o memory_overcommit.bin) which
allocates chunks of memory of user defined size.

On the Ubuntu system, the system freeze can be observed with swap
enabled on a cryptographic swap partition (dm-crypt; /etc/crypttab).
With openSUSE the lock-up also occurs with deactivated swap.
(swapoff -a).

Tested from a KDE terminal window as regular urser, the
steps to reproduce are:

* use cryptographic swap partition or disable swap
  (maybe also reproducible with normal swap; but apparently not on Suse)

* compile and invoke the attached code:
  ./memory_overcommit.bin

* enter a number (in MiB) of memory that is slightly smaller than the
available memory and press "enter" key once.

* enter a smaller number (minimum 1 MiB) and confirm again, do the same
again,..., successively approaching the limit of available memory with
smaller chunks.

* finally, when most of the memory/buffers/cache are used-up, the system
becomes unresponsive and constant, heavy harddisk-access commences.

Sometimes, killing the X-server or shutting down via hotkeys works after
several minutes of waiting, but this is not consistent.


I've reported this issue on the Ubuntu bug tracker before:
https://bugs.launchpad.net/ubuntu/+bug/283420

but as stated above, I also had the problem on another system.


I'd be glad if I could help if you need further information.