Re: Memory issues with Opteron 6220

[Jesper Krogh <jesper@krogh.cc>]

On 2012-02-09 09:33, Ingo Molnar wrote:
> * Anders Ossowicki<aowi@novozymes.com>  wrote:
>> Hey,
>>
>> We're seeing unexpected slowdowns and other memory issues with a new system.
>> Enough to render it unusable. For example:
>>
>> Error: open3: fork failed: Cannot allocate memory
>>
>> at times where there's no real memory pressure:
>>                     total       used       free     shared    buffers     cached
>>        Mem:     132270720  131942388     328332          0     299768  103334420
>>        -/+ buffers/cache:   28308200  103962520
>>        Swap:      7811068      13760    7797308
>>
>> [...]
Anders' co-worker here..  below C-code (Summary: for -t processes that 
repeatedly
allocates and dallocates 2GB of memory) can excersize the bug
pretty frequently using -t 32 on this machine. On the other 128GB
machine it can run without issues.

It actually ended up toasting the machine:
jk@nysvin:~$ ./foo -t 32
-bash: fork: Cannot allocate memory
jk@nysvin:~$ w
-bash: fork: Cannot allocate memory
jk@nysvin:~$ top
-bash: fork: Cannot allocate memory
jk@nysvin:~$ ls
-bash: fork: Cannot allocate memory

I dont know what to conclude.

jk@nysvin:~$ ./foo -t 32
Upper bound: 1953 MB
malloc(1953) MB failed. iterations: 6
malloc(1953) MB failed. iterations: 2
malloc(1953) MB failed. iterations: 8

foo.c
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/wait.h>

void worker(void)
{
     long long i;
     char *p;
     int action;
     int mult = 500000;
     int size = mult * 4096;
     fprintf(stderr,"Upper bound: %lu MB\n",(long int) (size/1024/1024));
     for (i=0; ; i++) {
         action = i%2;
         switch(action) {
         case 0:
             p = malloc(size);
             if (!p){
                 fprintf(stderr,"malloc(%lu) MB failed. iterations: 
%lli\n", (long int)size/1024/1024,i);
                 exit(1);
             }
             break;
         case 1:
             free(p);
             break;
         }
     }
}

void usage(const char *cmd)
{
     fprintf(stderr,"Usage: %s [-t numthreads]\n", cmd);
     exit(1);
}

int main(int argc, char **argv)
{
     int c, i;
     int nproc = sysconf(_SC_NPROCESSORS_ONLN);


     while ((c = getopt(argc, argv, "t:")) != EOF) {
         switch (c) {
         case 't':
             nproc = strtol(optarg, 0, 0);
             break;
         default:
             usage(argv[0]);
         }
     }

     //printf("forking %d children\n", nproc);
     for (i=0; i < nproc; i++) {
         switch(fork()) {
         case -1:
             fprintf(stderr,"fork: %s\n", strerror(errno));
             exit(1);
         case 0: /* child */
             worker();
             exit(0);
         default: /* parent */
             /* nothing */
             break;
         }
     }

     for (i=0; i < nproc; i++) {
         int x, p;
         p = wait(&x);
     }

     return 0;
}

Can also be found here: http://shrek.krogh.cc/~jesper/foo.c

-- 
Jesper Krogh