Kernel drops ethernet packets during disk writes

Kernel drops ethernet packets during disk writes

[Tiemen Schut]

Summary: The linux kernel appears to drop raw ethernet packets if
another process is writing to disk.

Test environment: Used a p4 1.7 GHz with gigabit interface
point-to-point connection to another p4 (windows pc). This windows PC
generates raw ethernet frames holding a counter and sends 'm on to the
linux PC, at a transfer rate of 350 Mbit/s. When not writing to disk,
everything goes quite fine. I can check the counters at the linux
side, and will notice a minimal packet loss (<  0.001 % or so).

However, for my application I want to write each and every frame to
disk. So, I created a second app, and through a fifo the receiver app
and the disk writer app communicate. Now I'm losing like 80% of my
packets :o Reducing the throughput on the network doesn't really help
(though it does help a little).

Note: I tried everything in the same app first, but my guess was that
the write operation delayed the app, so I decided to put everything in
two apps to give the scheduler some work ;) Didn't work though.

My guess would be that the kernel drops the packets because the write
operation takes to long (how long can it take, it's just a stupid 1512
bytes frame). Anyway, I tried to enlarge
.sys.net.core.netdev_max_backlog, but that didn't do the trick.

This problem occures on both 2.6.13 and 2.4.idontremember.

It kinda sucks, what's the use of receiving traffic if you can't write
it to disk?

I'm sending the packets using the winpcap library, and I'm receiving
the packets using the pcap library.

Any help would be _greatly_ appreciated, and if neccessary, please ask
for additional information/used software/test results/etc.

Tiemen Schut


PS: Personal CC's are okay with me, but if it's to much trouble never mind :)

Re: Kernel drops ethernet packets during disk writes

[Alan Cox]

Ar Maw, 2006-09-05 am 18:06 +0200, ysgrifennodd Tiemen Schut:
> Summary: The linux kernel appears to drop raw ethernet packets if
> another process is writing to disk.

No suprise. 

1Gbit is a bit over 100Mbyte/second

Thats from the card over PCI to memory
Then over PCI from memory to the disk controller

We are up to 200Mbytes/second

Best case performance for a 32bit PCI bus is 133Mbytes/second and you
won't get close to that. Even if both devices are PCI 66Mhz you are
right on the bus limit.

And if its an earlyish PIV what is your main memory bandwidth ?

Re: Kernel drops ethernet packets during disk writes

[Tiemen Schut]

Now this is something constructive!

I can indeed maintain 350Mbit/s to disk (about 55MB/s to disk
actually), and I can receive all the packets, as long as I don't try
to do both things simultaniously.

I'll try to increase the buffers tomorrow, will let you know how it
worked out. I doubt though that the tcp buffers have any impact, we're
talking raw ethernet frames here.

Thanks,

Tiemen

On 9/5/06, Stephen Degler <stephen@degler.net > wrote:
>  Tiemen Schut wrote:
> > Summary: The linux kernel appears to drop raw ethernet packets if
> > another process is writing to disk.
> >
> > Test environment: Used a p4 1.7 GHz with gigabit interface
> > point-to-point connection to another p4 (windows pc). This windows PC
> > generates raw ethernet frames holding a counter and sends 'm on to the
> > linux PC, at a transfer rate of 350 Mbit/s. When not writing to disk,
> > everything goes quite fine. I can check the counters at the linux
> > side, and will notice a minimal packet loss (<  0.001 % or so).
> >
> > However, for my application I want to write each and every frame to
> > disk. So, I created a second app, and through a fifo the receiver app
> > and the disk writer app communicate. Now I'm losing like 80% of my
> > packets :o Reducing the throughput on the network doesn't really help
> > (though it does help a little).
> >
> > Note: I tried everything in the same app first, but my guess was that
> > the write operation delayed the app, so I decided to put everything in
> > two apps to give the scheduler some work ;) Didn't work though.
> >
> > My guess would be that the kernel drops the packets because the write
> > operation takes to long (how long can it take, it's just a stupid 1512
> > bytes frame). Anyway, I tried to enlarge
> > .sys.net.core.netdev_max_backlog, but that didn't do the trick.
> >
> > This problem occures on both  2.6.13 and 2.4.idontremember.
> >
> > It kinda sucks, what's the use of receiving traffic if you can't write
> > it to disk?
> >
> > I'm sending the packets using the winpcap library, and I'm receiving
> > the packets using the pcap library.
> >
> > Any help would be _greatly_ appreciated, and if neccessary, please ask
> > for additional information/used software/test results/etc.
> >
> > Tiemen Schut
> Hi your packets are being dropped perhaps because there is no room to
> store them while the disk is busy, not that you don't receive them.
>
> Can you sustain 35M/second to disk?  If you can't do that with dd
>  if=/dev/zero of="foo" bs=1024k, then you can not expect to do it with
> the network plus your writer.  Writing
> sequentially to a large modern disk you should be able to do this (large
> capacity SATA or SCSI disks, or hardware raid controllers).
>
> You can improve matters greatly by increasing the kernel network buffers
> available.  This will give you more buffer room while the disk subsystem
> is busy.  Look at the following tuneables: you should be able to greatly
> reduce packet loss by increasing them from the default values.
>
> net.core.rmem_max = 2500000
> net.core.wmem_max = 2500000
> net.ipv4.tcp_rmem = 4096 5000000 5000000
> net.ipv4.tcp_wmem = 4096 65536 5000000
>
> I can assure you that with the proper system configuration, high network
> to disk throughput is indeed possible.
>
> Hope this helps,
> skd
>
>

Re: Kernel drops ethernet packets during disk writes

[Tiemen Schut]

Okay, I think I've found some sort of workaround/solution thingie. Not
really a solution though, it's more like a different approach of the
problem.

Instead of using pcap, I'm now using a socket interface on the
ethernet interface. Receiving packets this way and saving them to disk
(at a speed of 350Mbit/s), goes way better than using pcap!

With pcap, I lost like > 80% of the packets, using a raw socket I lose
like < 1% of the packets, which is a huge increase (well, actually a
decrease ;))!

So, to anybody trying to use pcap to capture packets and write 'm to
disk at the "same" time, I recommend using sockets instead of pcap.

TIemen


On 9/5/06, Tiemen Schut <tschut@gmail.com> wrote:
> Now this is something constructive!
>
> I can indeed maintain 350Mbit/s to disk (about 55MB/s to disk
> actually), and I can receive all the packets, as long as I don't try
> to do both things simultaniously.
>
> I'll try to increase the buffers tomorrow, will let you know how it
> worked out. I doubt though that the tcp buffers have any impact, we're
> talking raw ethernet frames here.
>
> Thanks,
>
> Tiemen
>
> On 9/5/06, Stephen Degler <stephen@degler.net > wrote:
> >  Tiemen Schut wrote:
> > > Summary: The linux kernel appears to drop raw ethernet packets if
> > > another process is writing to disk.
> > >
> > > Test environment: Used a p4 1.7 GHz with gigabit interface
> > > point-to-point connection to another p4 (windows pc). This windows PC
> > > generates raw ethernet frames holding a counter and sends 'm on to the
> > > linux PC, at a transfer rate of 350 Mbit/s. When not writing to disk,
> > > everything goes quite fine. I can check the counters at the linux
> > > side, and will notice a minimal packet loss (<  0.001 % or so).
> > >
> > > However, for my application I want to write each and every frame to
> > > disk. So, I created a second app, and through a fifo the receiver app
> > > and the disk writer app communicate. Now I'm losing like 80% of my
> > > packets :o Reducing the throughput on the network doesn't really help
> > > (though it does help a little).
> > >
> > > Note: I tried everything in the same app first, but my guess was that
> > > the write operation delayed the app, so I decided to put everything in
> > > two apps to give the scheduler some work ;) Didn't work though.
> > >
> > > My guess would be that the kernel drops the packets because the write
> > > operation takes to long (how long can it take, it's just a stupid 1512
> > > bytes frame). Anyway, I tried to enlarge
> > > .sys.net.core.netdev_max_backlog, but that didn't do the trick.
> > >
> > > This problem occures on both  2.6.13 and 2.4.idontremember.
> > >
> > > It kinda sucks, what's the use of receiving traffic if you can't write
> > > it to disk?
> > >
> > > I'm sending the packets using the winpcap library, and I'm receiving
> > > the packets using the pcap library.
> > >
> > > Any help would be _greatly_ appreciated, and if neccessary, please ask
> > > for additional information/used software/test results/etc.
> > >
> > > Tiemen Schut
> > Hi your packets are being dropped perhaps because there is no room to
> > store them while the disk is busy, not that you don't receive them.
> >
> > Can you sustain 35M/second to disk?  If you can't do that with dd
> >  if=/dev/zero of="foo" bs=1024k, then you can not expect to do it with
> > the network plus your writer.  Writing
> > sequentially to a large modern disk you should be able to do this (large
> > capacity SATA or SCSI disks, or hardware raid controllers).
> >
> > You can improve matters greatly by increasing the kernel network buffers
> > available.  This will give you more buffer room while the disk subsystem
> > is busy.  Look at the following tuneables: you should be able to greatly
> > reduce packet loss by increasing them from the default values.
> >
> > net.core.rmem_max = 2500000
> > net.core.wmem_max = 2500000
> > net.ipv4.tcp_rmem = 4096 5000000 5000000
> > net.ipv4.tcp_wmem = 4096 65536 5000000
> >
> > I can assure you that with the proper system configuration, high network
> > to disk throughput is indeed possible.
> >
> > Hope this helps,
> > skd
> >
> >
>