Re: zero-copy networking & a performance drop

Re: zero-copy networking & a performance drop

[Nivedita Singhvi]

> > On sending data, we have all the information trusted, because we
> > checked that already, as the user sets it. With sendfile, we have
> > even trusted and mapped data (because we just paged it in before).
> > 
> > If we take this into account, rx MUST be always slower, or tx
> > isn't really optimized yet.
> 
> Indeed, the above does make sense. But, what about when the receive side
> is trustable, too. This is obviously not the normal case; but with
> quadrics I think that it is. Since quadrics is a shared memory
> architecture, with all of the processing taken care of on the cards, we
> should be able to reliably trust the receive side as much as the send
> side. Unless I'm misunderstanding your use of trust?

Perhaps a little bit - the work the rx processing has to do
involves determining the validity of the packet as well as
demultiplexing at each protocol stage..You will always
have to at least test for a valid packet, parse options, find
the recipients, etc. If you're saying you have offloaded
the tcp/ip protocol stack on the cards, thats a whole different 
issue. 

 
> If I am right on this, how much of an overhaul would it be to implement, 
> for instance, a NETIF_RX_TRUSTED flag in the net_device struct to force a 
> receive side fast path? I don't expect this to bring the receive side even 
> with the transmit side, but right now we (and I have heard the same from 
> others) are running at ~70% of the transmit side on the receive side, 
> which seems to leave a good margin for improvement. 

I'm no maintainer, but I would at least say whoa! Not sure if we
have a problem, not sure where the problems are, and not sure
if there are problems in determining whether there is a problem :)..

fer'nstance:
	- if the rx code path is say 1000 lines and the tx
	  stack length is 700 lines of code, then I'd expect
	  the tx time to be 70% of rx time. A difference
	  in the stack length is not a problem by itself,
	  given that they have different functions to perform?

	- in reality, there is no such clean concept,
	  right? rx processing sends acks, interrupts happen,
	  and a whole bunch of profiling and lockmeter
	  data can end up pretty misleading.. Do you
	  have pure unidirectional data transfer? What 
	  applications are you running? How are you
	  measuring things, where are you measuring things,
	  and what are the skewing factors? 

	- and I'm not even a benchmarking or performance person,
	  wait till you get done talking to them :)	 	
	  
	 
> This seems like it could be useful not just for the special case of 
> quadrics (and other shared mamory architectures)- if, for instance, cards 
> begin to do more TCP processing in hardware (or we modify the firmware to 
> do this for us, ala AceNIC), it would be nice to bypass it in the kernel. 
> This is, of course, a quick idea that's just popped into my head, and 
> probably is inherently impossible or foolish :)

Thats a question of is the stack modified to take advantage
of what the hardware can do, and, as I said above, thats a rather bigger 
and different issue. 

> --Gus

thanks,
Nivedita

Is ff:00:00:00:00:00 a broadcast frame?

[Dax Kelson]

In the "earlier" 2.4 kernels (those that shipped/errata with RHL 7.1
~2.4.6), an ethernet frame destined to ff:00:00:00:00:00 were not
processed and passed up the stack.

Now, with "current" 2.4 kernels (RHL 7.2 errata, and RHL 7.3
2.4.9-2.4.18), the same frame IS processed and passed up the stack.

The hardware is identical, the NICs are 3c905C.

Is this an intentional optimization/bug?

Here is an ARP request encapsulated in a "bogus" ethernet frame (look at
the layer 2 destination). A RHL 7.3 (2.4.18) system will respond (see
Frame 2 below) to this frame, an older RHL 7.1 (2.4.6) system will not
respond.

Frame 1 (42 on wire, 42 captured)
    Arrival Time: Jul  1, 2002 17:16:35.565996000
    Time delta from previous packet: 1.009672000 seconds
    Time relative to first packet: 1.009833000 seconds
    Frame Number: 3
    Packet Length: 42 bytes
    Capture Length: 42 bytes
Ethernet II
    Destination: ff:00:00:00:00:00 (ff:00:00:00:00:00)
    Source: ff:00:00:00:00:00 (ff:00:00:00:00:00)
    Type: ARP (0x0806)
Address Resolution Protocol (request)
    Hardware type: Ethernet (0x0001)
    Protocol type: IP (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (0x0001)
    Sender MAC address: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
    Sender IP address: 10.100.0.8 (10.100.0.8)
    Target MAC address: 00:00:00:00:00:00 (00:00:00:00:00:00)
    Target IP address: 10.100.0.10 (10.100.0.10)

Frame 2 (60 on wire, 60 captured)
    Arrival Time: Jul  1, 2002 17:16:35.566146000
    Time delta from previous packet: 0.000150000 seconds
    Time relative to first packet: 1.009983000 seconds
    Frame Number: 4
    Packet Length: 60 bytes
    Capture Length: 60 bytes
Ethernet II
    Destination: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
    Source: 00:01:03:de:57:37 (00:01:03:de:57:37)
    Type: ARP (0x0806)
    Trailer: 00000000000000000000000000000000...
Address Resolution Protocol (reply)
    Hardware type: Ethernet (0x0001)
    Protocol type: IP (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (0x0002)
    Sender MAC address: 00:01:03:de:57:37 (00:01:03:de:57:37)
    Sender IP address: 10.100.0.10 (10.100.0.10)
    Target MAC address: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
    Target IP address: 10.100.0.8 (10.100.0.8)

Re: Is ff:00:00:00:00:00 a broadcast frame?

[Matti Aarnio]

On Mon, Jul 01, 2002 at 05:27:11PM -0600, Dax Kelson wrote:
> In the "earlier" 2.4 kernels (those that shipped/errata with RHL 7.1
> ~2.4.6), an ethernet frame destined to ff:00:00:00:00:00 were not
> processed and passed up the stack.
> 
> Now, with "current" 2.4 kernels (RHL 7.2 errata, and RHL 7.3
> 2.4.9-2.4.18), the same frame IS processed and passed up the stack.

  That IEEE 802 MAC address is NOT a broadcast address.
  Broadcast would be one with all bits set:  ff:ff:ff:ff:ff:ff

  The above mentioned MAC address is a multicast frame, because
  bit  0x01 in the first octet is set.  (All broadcast-frames are
  also multicast.)

> The hardware is identical, the NICs are 3c905C.
> 
> Is this an intentional optimization/bug?
> 
> Here is an ARP request encapsulated in a "bogus" ethernet frame (look at
> the layer 2 destination). A RHL 7.3 (2.4.18) system will respond (see
> Frame 2 below) to this frame, an older RHL 7.1 (2.4.6) system will not
> respond.

  Neither should respond.  IPv4 ARP is defined using only broadcast
  frames.

> Frame 1 (42 on wire, 42 captured)
>     Arrival Time: Jul  1, 2002 17:16:35.565996000
>     Time delta from previous packet: 1.009672000 seconds
>     Time relative to first packet: 1.009833000 seconds
>     Frame Number: 3
>     Packet Length: 42 bytes
>     Capture Length: 42 bytes
> Ethernet II
>     Destination: ff:00:00:00:00:00 (ff:00:00:00:00:00)
>     Source: ff:00:00:00:00:00 (ff:00:00:00:00:00)
>     Type: ARP (0x0806)
> Address Resolution Protocol (request)
>     Hardware type: Ethernet (0x0001)
>     Protocol type: IP (0x0800)
>     Hardware size: 6
>     Protocol size: 4
>     Opcode: request (0x0001)
>     Sender MAC address: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
>     Sender IP address: 10.100.0.8 (10.100.0.8)
>     Target MAC address: 00:00:00:00:00:00 (00:00:00:00:00:00)
>     Target IP address: 10.100.0.10 (10.100.0.10)
> 
> Frame 2 (60 on wire, 60 captured)
>     Arrival Time: Jul  1, 2002 17:16:35.566146000
>     Time delta from previous packet: 0.000150000 seconds
>     Time relative to first packet: 1.009983000 seconds
>     Frame Number: 4
>     Packet Length: 60 bytes
>     Capture Length: 60 bytes
> Ethernet II
>     Destination: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
>     Source: 00:01:03:de:57:37 (00:01:03:de:57:37)
>     Type: ARP (0x0806)
>     Trailer: 00000000000000000000000000000000...
> Address Resolution Protocol (reply)
>     Hardware type: Ethernet (0x0001)
>     Protocol type: IP (0x0800)
>     Hardware size: 6
>     Protocol size: 4
>     Opcode: reply (0x0002)
>     Sender MAC address: 00:01:03:de:57:37 (00:01:03:de:57:37)
>     Sender IP address: 10.100.0.10 (10.100.0.10)
>     Target MAC address: 00:01:03:de:56:a4 (00:01:03:de:56:a4)
>     Target IP address: 10.100.0.8 (10.100.0.8)