Why is get_user_pages so slow?

Why is get_user_pages so slow?

[Micha Nelissen]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

Hi all,

Why is get_user_pages much slower than taking the faults? (I would 
expect it to be faster).

Attached example program first mallocs a piece of memory (64MB in this 
case) then reads it "to take the faults". Afterwards, it uses mmap with 
MAP_POPULATE to "speed up" and not to have to take the faults, but have 
everything mapped in one go. I think mmap is using get_user_pages in 
this case.

$ ./memspeed
malloc took 0 msecs
read took 14 msecs
write took 0 msecs
free took 1 msecs
mmap took 45 msecs
munmap took 5 msecs

Using MAP_POPULATE is 3 times as slow as the 'stupid' implementation! 
I'm running a Core 2 duo e6300 system with linux 2.6.28.4.

Am I doing something wrong? MAP_POPULATE seems a bit of a joke to me.

Thanks,

Micha

[-- Attachment #2: memspeed.c --]
[-- Type: text/x-csrc, Size: 1584 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/time.h>

#define SIZE (64 << 20)

unsigned tv_msecs(struct timeval *tvs, struct timeval *tve)
{
        return (tve->tv_sec - tvs->tv_sec) * 1000 +
                (tve->tv_usec - tvs->tv_usec) / 1000;
}

int main(void)
{
        struct timeval tvs, tve;
        void *buf;
        int *p, *e, i;

        gettimeofday(&tvs, NULL);
        buf = malloc(SIZE);
        gettimeofday(&tve, NULL);
        printf("malloc took %d msecs\n", tv_msecs(&tvs, &tve));

        gettimeofday(&tvs, NULL);
        for (p = buf, e = buf + SIZE; p < e; p += 0x1000)
                i = *(volatile int*)p;
        gettimeofday(&tve, NULL);
        printf("read took %d msecs\n", tv_msecs(&tvs, &tve));

        gettimeofday(&tvs, NULL);
        for (p = buf, e = buf + SIZE; p < e; p += 0x1000)
                *(volatile int*)p = 0xaa55aa55;
        gettimeofday(&tve, NULL);
        printf("write took %d msecs\n", tv_msecs(&tvs, &tve));

        gettimeofday(&tvs, NULL);
        free(buf);
        gettimeofday(&tve, NULL);
        printf("free took %d msecs\n", tv_msecs(&tvs, &tve));

        gettimeofday(&tvs, NULL);
        buf = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE |
                MAP_ANONYMOUS | MAP_POPULATE, -1, 0);
        gettimeofday(&tve, NULL);
        printf("mmap took %d msecs\n", tv_msecs(&tvs, &tve));

        gettimeofday(&tvs, NULL);
        munmap(buf, SIZE);
        gettimeofday(&tve, NULL);
        printf("munmap took %d msecs\n", tv_msecs(&tvs, &tve));

        return 0;
}

Re: Why is get_user_pages so slow?

[Kevin Easton]

Quoting Micha Nelissen <micha@neli.hopto.org>:

> Hi all,
>
> Why is get_user_pages much slower than taking the faults? (I would  
> expect it to be faster).
>
> Attached example program first mallocs a piece of memory (64MB in  
> this case) then reads it "to take the faults". Afterwards, it uses  
> mmap with MAP_POPULATE to "speed up" and not to have to take the  
> faults, but have everything mapped in one go. I think mmap is using  
> get_user_pages in this case.
>
> $ ./memspeed
> malloc took 0 msecs
> read took 14 msecs
> write took 0 msecs
> free took 1 msecs
> mmap took 45 msecs
> munmap took 5 msecs
>
> Using MAP_POPULATE is 3 times as slow as the 'stupid'  
> implementation! I'm running a Core 2 duo e6300 system with linux  
> 2.6.28.4.
>
> Am I doing something wrong? MAP_POPULATE seems a bit of a joke to me.

Hi Micha,

Yep, you are.  Because your pointer 'p' is a pointer to int, when you  
increment it by 0x1000 in your loops you are actually incrementing it  
by 0x1000 * sizeof(int) - so you're only actually touching one page in  
four.

If you change the types of 'buf', 'p' and 'e' to 'char *' then it  
touches every page - and (and least on my test box) the MAP_POPULATE  
case pulls ahead.

     - Kevin



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: Why is get_user_pages so slow?

[Micha Nelissen]

Hi Kevin,

Kevin Easton wrote:
> Yep, you are.  Because your pointer 'p' is a pointer to int, when you 
> increment it by 0x1000 in your loops you are actually incrementing it by 
> 0x1000 * sizeof(int) - so you're only actually touching one page in four.

Oops sorry, thanks for catching my mistake.

I also discovered the following: if I read from all pages, then call 
get_user_pages, it is still quite slow (did I get a read-only page?). 
However, if I touch all pages by writing to them, then get_user_pages 
becomes a factor 40 times faster or so.

All is clear now, I think. Thanks.

Micha