memory allocation problems

memory allocation problems

[majer]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 923 bytes --]



  Hi. Im hoping someone on here can help me out. I posted something
  similar to this back in June 2000 when I was on the 2.2.X line and
  was waiting to see if the 2.4 kernel would provide a fix.

  Essentially, the problem can be summarized to be that on a machine
  with ample ram (2G, 4G, etc), I am unable to malloc a gig if I ask 
  for the memory in small ( <= 128k) chunks. I've enclosed some results 
  and a little program which was put together to demonstrate the problems
  we're having.  All of the failures seem to occur around 930MB.

  I'm more than happy to try any tunings, patches, etc and my 
  time is at your disposal.

  Thanks,

  Karl

----
        Karl Majer                          majer@endeca.com                 
        Sr Systems Architect                617 577 7999 xt 251                

 "Think for yourselves and let others enjoy the privilege to do so, too."
						     --Voltaire

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1433 bytes --]

  The fields are: 
  Iteration, Addr of ptr, Addr stored by ptr, alloc'd mem

  size: 4096
  #228852 (401F07D8) (3FFFC948) allocated 937381888 bytes total.
  #228853 (401F07DC) (3FFFD950) allocated 937385984 bytes total.
  malloc error Cannot allocate memory
  
  size: 8192
  #114536 (40180DA8) (3FFF94E8) allocated 938287104 bytes total.
  #114537 (40180DAC) (3FFFB4F0) allocated 938295296 bytes total.
  malloc error Cannot allocate memory
   
  size: 16384
  #57294 (40148F40) (3FFF1818) allocated 938721280 bytes total.
  #57295 (40148F44) (3FFF5820) allocated 938737664 bytes total.
  malloc error Cannot allocate memory
  
  size: 32768
  #28653 (4012CFBC) (3FFE9910) allocated 938934272 bytes total.
  #28654 (4012CFC0) (3FFF1918) allocated 938967040 bytes total.
  malloc error Cannot allocate memory
  
  size: 65536
  #14325 (805797C) (3FFC5958) allocated 938868736 bytes total.
  #14326 (8057980) (3FFD5960) allocated 938934272 bytes total.
  #14327 (8057984) (3FFE5968) allocated 938999808 bytes total.
  malloc error Cannot allocate memory
  
  size: 131072
  #8186 (8051990) (3FF9F980) allocated 1073086464 bytes total.
  #8187 (8051994) (3FFBF988) allocated 1073217536 bytes total.
  malloc error Cannot allocate memory
  
  size: 262144
  #4094 (804D9A0) (37FD39A0) allocated 1073479680 bytes total.
  #4095 (804D9A4) (380139A8) allocated 1073741824 bytes total.
  success
   

[-- Attachment #3: Type: TEXT/PLAIN, Size: 836 bytes --]

#include<stdio.h>
#include<unistd.h>
#include<stdlib.h>
#include<errno.h>

unsigned int i,num,bites,total,delay=0;
char **memhog;

int main(int argc,char *argv[]){

  if(argc!=4){
    fprintf(stderr,"usage: %s [# of chunks] [bytes per chunk] [delay in sec]\n",argv[0]);
    return(1);
  } 

  num=atoi(argv[1]);
  bites=atoi(argv[2]);
  delay=atoi(argv[3]);

  memhog=malloc(sizeof(char*) * num); 
  if(!memhog){
    fprintf(stderr,"malloc error %s\n",strerror(errno));  
    exit(1);
  }

  for(i=0;i<num;i++){
    *memhog=malloc(bites);
    if(!*memhog){
      fprintf(stderr,"malloc error %s\n",strerror(errno));  
      exit(1);
    }
    total+=bites;
    printf("#%u (%X) (%X) allocated %u bytes total.\n",i,memhog,*memhog,total);
    memhog++;
    sleep(delay);
  }  

 return(0);
} 
  

Re: memory allocation problems

[Andi Kleen]

On Fri, Apr 06, 2001 at 10:06:47AM -0400, majer@endeca.com wrote:
>   Essentially, the problem can be summarized to be that on a machine
>   with ample ram (2G, 4G, etc), I am unable to malloc a gig if I ask 
>   for the memory in small ( <= 128k) chunks. I've enclosed some results 
>   and a little program which was put together to demonstrate the problems
>   we're having.  All of the failures seem to occur around 930MB.

It's bumping against some mapping (just do system("cat /proc/self/maps")
on allocation failure to see which). Usual suspects are shared libraries.
One possible solution is to upgrade to a newer glibc, the 2.2 glibc malloc 
should handle this case better.  
A way to get mappings like shared libraries out of the way is to 
increase the value of TASK_UNMAPPED_BASE in include/asm-i386/processor.h.
For that the kernel needs to be recompiled and it should be smaller
TASK_SIZE-enough space for your shared libraries. With that even the older
malloc will probably work.


-Andi

Re: memory allocation problems

[Wayne Whitney]

In mailing-lists.linux-kernel, you wrote:

>  Essentially, the problem can be summarized to be that on a machine
>  with ample ram (2G, 4G, etc), I am unable to malloc a gig if I ask 
>  for the memory in small ( <= 128k) chunks. 

Take a look at this message by Szabolcs Szakacsits:

http://marc.theaimsgroup.com/?l=linux-kernel&m=97898653909227&w=2

There are other messages that may be of interest to you in that
thread, although they are spread out in a large thread.

Briefly, malloc in glibc will use brk() for "small" chunks and mmap()
for "large" chunks.  On a usual i386 linux kernel, the 4GB address
space is set up so that brk() can get at most 870MB or so and mmap()
can get at most 2GB.  Newer glibc's allow you to tune the definition
of "small" via an environment variable.

Cheers,
Wayne

Re: memory allocation problems

[Mark Hahn]

> can get at most 2GB.  Newer glibc's allow you to tune the definition
> of "small" via an environment variable.

eventually, perhaps libc will be smart enough to create 
more arenas in mmaped space once sbrk fails.  note, though,
that you *CAN* actually malloc a lot more than 1G: you just
have to avoid causing mmaps that chop your VM at TASK_UNMAPPED_BASE:

#include <unistd.h>
#include <stdlib.h>
#include <fcntl.h>

void printnumber(unsigned n) {
    char number[20];
    int i;
    for (i=sizeof(number)-1; i>=0 && n; i--) {
        number[i] = '0' + (n % 10);
        n /= 10;
    }
    i++;
    write(1,number+i, sizeof(number)-i);
}
int main() {
    unsigned total = 0;
    const unsigned size = 32*1024;

    while (malloc(size)) {
        total += size;
        printnumber(total>>20);
        write(1,"\n",1);
    }
    return 0;
}

compile -static, of course; printnumber is to avoid stdio, which seems
to use mmap for a small scratch buffer.  I allocated 2942 MB on my 128M 
machine(had to add a swapfile temporarily, since so many tiny mallocs 
do touch nontrivial numbers of pages for arena bookkeeping.)

Re: memory allocation problems

[Wayne Whitney]

On Fri, 6 Apr 2001, Mark Hahn wrote:

> note, though, that you *CAN* actually malloc a lot more than 1G: you
> just have to avoid causing mmaps that chop your VM at
> TASK_UNMAPPED_BASE:

Neat trick.  I didn't realize that you could avoid allocating the mmap()
buffers for stdin and stdout.

As was pointed out to me in January, another solution for i386 would be to
fix a maximum stack size and have the mmap() allocations grow downward
from the "top" of the stack (3GB - max stack size).  I'm not sure why that
is not currently done.

I once wrote a tiny patch to do this, and ran it successfully for a couple
days, but knowing so little about the kernel I probably did it in a
completely wrong, inefficient way.  For example, some of the vma
structures are sorted in increasing address order, and so perhaps to do
this properly one should change them to decreasing address order.

Cheers,
Wayne

Re: memory allocation problems

[Mark Hahn]

> > note, though, that you *CAN* actually malloc a lot more than 1G: you
> > just have to avoid causing mmaps that chop your VM at
> > TASK_UNMAPPED_BASE:
> 
> Neat trick.  I didn't realize that you could avoid allocating the mmap()
> buffers for stdin and stdout.

noone ever said you had to use stdio.  or even use libc, for that matter!

> As was pointed out to me in January, another solution for i386 would be to
> fix a maximum stack size and have the mmap() allocations grow downward
> from the "top" of the stack (3GB - max stack size).  I'm not sure why that
> is not currently done.

problems get fixed when there's some pain involved: people bumping 
into a limit, or painfully bad code, etc.  not enough people are 
feeling any pain about the current design.

this (and the "move TASK_UNMAPPED_BASE" workaround) have been known
for years; I think someone even coded up a "grow vmareas down" patch
the last time we all discussed this.

> I once wrote a tiny patch to do this, and ran it successfully for a couple
> days, but knowing so little about the kernel I probably did it in a
> completely wrong, inefficient way.  For example, some of the vma
> structures are sorted in increasing address order, and so perhaps to do
> this properly one should change them to decreasing address order.

oh, I guess you did the patch ;)
seriously, resubmit it when 2.5 opens up.  the fact is that we currently
have two things that grow up, and one that grows down.  so obviously,
one up-grower must have an arbitrary limit.  switching vma's to down-growing
is a good solution, since it's actually *good* to limit stack growth.  
I wonder whether fortraners still put all their data on the stack;
they wouldn't be happy ;)

a simple workaround would be to turn TASK_UNMAPPED_AREA into a variable,
either system-wide or thread-specific (like ia64 already has!).  that's 
compatible with the improved vmas-down approach, too.

regards, mark hahn.

Re: memory allocation problems

[Hugh Dickins]

On Fri, 6 Apr 2001, Wayne Whitney wrote:
> 
> As was pointed out to me in January, another solution for i386 would be to
> fix a maximum stack size and have the mmap() allocations grow downward
> from the "top" of the stack (3GB - max stack size).  I'm not sure why that
> is not currently done.

I'd be interested in the answer to that too.  Typically, the memory
layout has ELF text at the lowest address, starting at 0x08048000 -
which is a curious place to put it, until you realize that if you
place the stack below it, you can use (in a typical small program)
just one page table for stack + text + data (then another for mmaps
and shared libs from 3GB down): two page tables instead of present three.

Hugh