Re: Huge performance issue with cciss driver on HP DL385 servers (2.6.13 -> 2.6.17)

[Erik Mouw <erik@harddisk-recovery.com>]

On Tue, Jul 11, 2006 at 03:02:48PM +0200, Xavier Roche wrote:
> Hi folks,

[snip]

> Test program:
> -------------
> 
> #define _XOPEN_SOURCE 500
> #define _GNU_SOURCE
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <stdio.h>
> #include <errno.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netinet/in.h>
> #include <arpa/inet.h>
> #include <fcntl.h>
> 
> /* a running ttytst source IP */
> #define CHARGEN_IP "192.168.115.175"
> 
> int main(int argc, char **argv) {
>   if (argc != 3) {
>     fprintf(stderr, "usage: %s FILE BUFSIZE\n", argv[0]);
>     return -1;
>   }
> 
>   char* fname = argv[1];
>   size_t bufflen = strtol(argv[2], NULL, 10);
>   if (bufflen % 512 != 0 || bufflen == 0) {
>     fprintf(stderr, "illegal buffer size: %s\n", argv[2]);
>     return -1;
>   }
> 
>   /* Allocate a 512-bytes aligned buffer for O_DIRECT transfers. */
>   void* buffer = malloc(bufflen + 512);
>   size_t delta = 512 - ((size_t) buffer) % 512;
>   void *abuffer = buffer + delta;

AFAIK buffers for direct IO need to be *page* aligned. Use something
like:

    abuffer = memalign(getpagesize(), bufflen);

Or:

    abuffer = valloc(bufflen);

I guess you got away with it cause your 512 byte alignment happened to
align on a page, but you shouldn't count on that. However...

[...]

>     /* The following pwrite call is pathologically slow when the following
>      * conditions are met:
>      *
>      *  - "fd" is opened with the O_DIRECT flag
>      *  - "bufflen" is greater than 1024K
>      *  - the file is located on an ext3 filesystem
>      *  - the program must be run just after a reboot, with an idle machine
>      */
>     pwrite(fd, abuffer, bufflen, offset);

You should check the return value of pwrite(). It could very well be
that you get errors due to unaligned writes. IIRC the glibc memory
allocator can decide from what memory pool it allocates memory
depending on the request size. It could very well be that the alignment
changes with the size. Anyway, try to recreate the problem with a page
aligned buffer.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands