Re: [RFC v2: Patch 1/3] net: hand off skb list to other cpu to submit to upper layer

[Tom Herbert <therbert@google.com>]

On Thu, Mar 12, 2009 at 11:43 PM, Zhang, Yanmin
<yanmin_zhang@linux.intel.com> wrote:
>
> On Thu, 2009-03-12 at 14:08 +0000, Ben Hutchings wrote:
> > On Thu, 2009-03-12 at 16:16 +0800, Zhang, Yanmin wrote:
> > > On Wed, 2009-03-11 at 12:13 +0100, Andi Kleen wrote:
> > [...]
> > > >  and just use the hash function on the
> > > > NIC.
> > > Sorry. I can't understand what the hash function of NIC is. Perhaps NIC hardware has something
> > > like hash function to decide the RX queue number based on SRC/DST?
> >
> > Yes, that's exactly what they do.  This feature is sometimes called
> > Receive-Side Scaling (RSS) which is Microsoft's name for it.  Microsoft
> > requires Windows drivers performing RSS to provide the hash value to the
> > networking stack, so Linux drivers for the same hardware should be able
> > to do so too.
> Oh, I didn't know the background. I need study more about network.
> Thanks for explain it.
>

You'll definitely want to look at the hardware provided hash.  We've
been using a 10G NIC which provides a Toeplitz hash (the one defined
by Microsoft) and a software RSS-like capability to move packets from
an interrupting CPU to another for processing.  The hash could be used
to index to a set of CPUs, but we also use the hash as a connection
identifier to key into a lookup table to steer packets to the CPU
where the application is running based on the running CPU of the last
recvmsg.  Using the device provided hash in this manner is a HUGE win,
as opposed to taking cache misses to get 4-tuple from packet itself to
compute a hash.  I posted some patches a while back on our work if
you're interested.

We also using multiple RX queues of the 10G device in concert with
pretty good results.  We have noticed that the interrupt overheads
substantially mitigate the benefits.  In fact, I would say the
software packet steering has provided the greater benefit (and it's
very useful on our many 1G NICS that don't have multiq!).

Tom