coping with swap-exhaustion in 2.4.33-4

coping with swap-exhaustion in 2.4.33-4

[Yakov Lerner]

    Where can I read anything about how kernel is supposed to
react to the 'swap-full' condition ? We have troubles on the
production machine which routinely arrives to the swap-full state
no matter how I increase the swap, because user proceses multi-fork
and then want to allocate a lot of virtual memory.

   I made a small test (below) and I run it as root and as non-root, having
two ssh sessions to the target machine (I run it as
as while true; do memstress; done ).

   Is the right solution in the kernel, or in the ulimiting the processes ?
I'm not sure what to do with forking, they fork many children ...
Is looking for kernel patches the right direction ? I want kernel not
to kill root-owned processes when non-root process is memory hog, but
that's not what I see.

Running my memstress, I see two things:
  1) if memory-hog process is non-root, then it is never killed
but hangs, and other pprocesses (which are root processes) are
killed instead of the mem-hog process.
  2) if memory-hog process is run as root, it is promptly
killed when swap is exhausted. Sometimes other processes
are killed, too (sshd server happen to be killed).

The (1) does not sound right for me. Is there a patch for it ?
Is there a redhat patch for this maybe ? Where can I find such
patches ?

Thanks
Yakov

Re: coping with swap-exhaustion in 2.4.33-4

[Yakov Lerner]

[-- Attachment #1: Type: text/plain, Size: 214 bytes --]

On 11/23/06, Yakov Lerner <iler.ml@gmail.com> wrote:
>    I made a small test (below) and I run it as root and as non-root, having

Here is memstress.c, tiny memory hog that I referred to in previous email.

Yakov

[-- Attachment #2: memstress.c --]
[-- Type: application/octet-stream, Size: 0 bytes --]

Re: coping with swap-exhaustion in 2.4.33-4

[Xavier Bestel]

On Thu, 2006-11-23 at 14:30 +0200, Yakov Lerner wrote:
>     Where can I read anything about how kernel is supposed to
> react to the 'swap-full' condition ? We have troubles on the
> production machine which routinely arrives to the swap-full state
> no matter how I increase the swap, because user proceses multi-fork
> and then want to allocate a lot of virtual memory.

Did you disable memory overcommit ?

	Xav

Re: coping with swap-exhaustion in 2.4.33-4

[Yakov Lerner]

On 11/23/06, Xavier Bestel <xavier.bestel@free.fr> wrote:
> On Thu, 2006-11-23 at 14:30 +0200, Yakov Lerner wrote:
> >     Where can I read anything about how kernel is supposed to
> > react to the 'swap-full' condition ? We have troubles on the
> > production machine which routinely arrives to the swap-full state
> > no matter how I increase the swap, because user proceses multi-fork
> > and then want to allocate a lot of virtual memory.
>
> Did you disable memory overcommit ?

The /proc/sys/vm/overcommit_memory is 0 (the default is 0, no ?)

This is test program, memstress.c. We make sure we touch all allocated chunks.

#include <stdio.h>
#include <stdlib.h>
#include <poll.h>
#include <string.h>


int main(int argc, char **argv) {
    char *p;
    int size = 10*1024*1024;
    int k;
    int bypass_prompt = 0;
    int mb = 0;
    int bytes = 0;

    printf("Warning this allocation test may kill your computer by
exhausting the swap+memory\n");
    if( argc > 1 && 0 == strcmp("-f", argv[1])) {
        bypass_prompt = 1;
    }
    if(!bypass_prompt) {
        printf("Press Enter to proceed, or Ctrl-C to cancel; or use -f
option to bypass the prompt\n");
        getchar();
    }


    for(k=0; ; k++)  {
        p = malloc(size);
        if(p) {
            memset(p, 0xff, size );
            bytes += size;
            printf("%d mb\n", bytes / (1024*1024) );
        }
        poll(0,0, 1);
    }
}

Re: coping with swap-exhaustion in 2.4.33-4

[Yakov Lerner]

On 11/23/06, Xavier Bestel <xavier.bestel@free.fr> wrote:
> On Thu, 2006-11-23 at 14:30 +0200, Yakov Lerner wrote:
> >     Where can I read anything about how kernel is supposed to
> > react to the 'swap-full' condition ? We have troubles on the
> > production machine which routinely arrives to the swap-full state
> > no matter how I increase the swap, because user proceses multi-fork
> > and then want to allocate a lot of virtual memory.
>
> Did you disable memory overcommit ?

When I set overcommit to 1 (2.4 seems not to have 1 vs 2
distinction, am I right), then still root processes are killed
when non-root process is a memory hog. Is there an option
to have malloc return NULL and never kill a process when
malloc fails ?

Thanks
Yakov