Re: Bottleneck on /dev/null

["Richard B. Johnson" <root@chaos.analogic.com>]

On Fri, 21 Mar 2003, Srihari Vijayaraghavan wrote:

> Linux-2.4.latest
> PACKET_MMAP
> PCAP_FRAMES=max for tcpdump-3.8/libpcap-0.8 (from http://public.lanl.gov/cpw/)
> e1000 driver
>
> 2 * Xeon 2800 MHz, 512 KB L2
> 1 GB RAM
> 70 GB HW RAID-0 on SmartArray 5i
> 2 * 2 port Intel GigE cards (only using 1 per card for the testing purposes)
>
> Capturing all packets and writting to /dev/null causes more packet drops than
> writting to hard drives (approx 40,000 packets/sec of 70 bytes for couple of
> minutes). I will have a comparision between those figures in a day or two,
> but /dev/null was well over SCSI hard drives. I thought writting (even
> multiple of them simultaneously) to /dev/null should be faster than fastest
> SCSI drives out there :) Interesting.
>
> (And yes I see plenty of "errors", "dropped", and "overruns" in ifconfig stats
> on those interfaces. %system is over 80%, and tcpdump goes to "D" state many
> times. Simon Kirby suggested to use irq-smp_affinity to see if that helps for
> reducing %system time. A well optimised e1000 would definitely help as tg3
> does it very well.)
>
> I mean to test this /dev/null behavior on 2 tg3 driver configuration perhaps
> in couple of days time. (But the 2 tg3 cards with out-of-the-box NAPI support
> on 2.4.latest is able to not to loose a single packet even while writting to
> hard drives, then I didn't care to test it on /dev/null)
>
> BTW I found 2.5.51 backport of e1000 NAPI support at
> http://havoc.gtf.org/lunz/linux/net/
> Anyone knows of a recent backport or improved one for 2.4.latest (including
> 2.4.21-pre5 or -pre6). Patches for testing or URL is welcome.
>
> Thanks
> --
> Hari
> harisri@bigpond.com

You are correct and here's a little program to show the problem
and demonstrate when it gets corrected.




#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <signal.h>
#include <time.h>
#define BUF_LEN 0x10000
unsigned long amount = 0L;
void timer(int unused) {
    fprintf(stdout, "Kilobytes / sec = %lu\n", amount >> 10);
    fflush(stdout);
    amount = 0L;
    alarm(1);
}
main() {
    int fd, len;
    char *buf;
    if((fd = open("/dev/null", O_RDWR)) < 0)
        exit(EXIT_FAILURE);
    if((buf = malloc(BUF_LEN)) == NULL)
        exit(EXIT_FAILURE);
    (void)signal(SIGALRM, timer);
    alarm(1);
    while((len = write(fd, buf, BUF_LEN)) > 0)
        amount += (unsigned long) len;
    free(buf);
    return 0;
}

On my SMP system, using kernel version 2.4.20, I get:

Kilobytes / sec = 3987136
Kilobytes / sec = 4101760
Kilobytes / sec = 1984
Kilobytes / sec = 4138304
Kilobytes / sec = 33664
Kilobytes / sec = 4189888
Kilobytes / sec = 432
Kilobytes / sec = 4122880

There is an awful big variation and I'm the only one on
this system!! If I disconnect the network line so I
truly get the entire attention of both CPUs, I get:

Kilobytes / sec = 3717402
Kilobytes / sec = 320250
Kilobytes / sec = 239501
Kilobytes / sec = 1893527
Kilobytes / sec = 23
Kilobytes / sec = 6783920
Kilobytes / sec = 1296789
Kilobytes / sec = 5109001


How, what the :F: makes the thing stumble down to 23 kilobytes
per second?a `taint right.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.