mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Leonid Grossman" <leonid.grossman@s2io.com>
To: <hadi@cyberus.ca>
Cc: "'Jeff Garzik'" <jgarzik@pobox.com>,
	"'David S. Miller'" <davem@davemloft.net>,
	<alan@lxorguk.ukuu.org.uk>, <paul@clubi.ie>, <netdev@oss.sgi.com>,
	<linux-kernel@vger.kernel.org>
Subject: RE: The ultimate TOE design
Date: Thu, 16 Sep 2004 07:57:48 -0700	[thread overview]
Message-ID: <200409161458.i8GEvtqH022604@guinness.s2io.com> (raw)
In-Reply-To: <1095328673.1063.130.camel@jzny.localdomain>

 

> -----Original Message-----
> From: jamal [mailto:hadi@cyberus.ca] 
> Sent: Thursday, September 16, 2004 2:58 AM
> To: Leonid Grossman
> Cc: 'Jeff Garzik'; 'David S. Miller'; 
> alan@lxorguk.ukuu.org.uk; paul@clubi.ie; netdev@oss.sgi.com; 
> linux-kernel@vger.kernel.org
> Subject: RE: The ultimate TOE design
> 
> On Thu, 2004-09-16 at 01:25, Leonid Grossman wrote:
> >  
> > > -----Original Message-----
> > > From: jamal [mailto:hadi@cyberus.ca]
> 
> > > On a serious note, I think that PCI-express (if it lives upto its
> > > expectation) will demolish dreams of a lot of these TOE 
> investments.
> > > Our problem is NOT the CPU right now (80% idle processing 450Kpps 
> > > forwarding). Bus and memory distance/latency are.
> > 
> > In servers, both bottlenecks are there - if you look at the cost of 
> > TCP and filesystem processing at 10GbE, CPU is a huge problem (and 
> > will be for foreseeable future), even for fastest 64-bit systems.
> 
> True, but with the bus contention being a non-issue you got 
> more of that xeon being available for use (lets say i can use 
> 50% more of its capacity then i can do more). IOW, it becomes 
> a compute capacity problem mostly - one that you should in 
> theory be able to throw more CPU at. SMT (the way power5 and 
> some of the network processors do it[1]) should go a long way 
> to address both additional compute and hardware threading to 
> work around memory latencies. With PCI-express, compute power 
> in mini-clustering in the form of AS 
> (http://www.asi-sig.org/home) is being plotted as we speak.
> To sumarize: The problem to solve in 24 months maybe 100Gige.
> 
> > I agree though that bus and memory are bigger issues, this 
> is exactly 
> > the reason for all these RDMA over Ethernet investments :-)
> 
> And AS does a damn good job at specing all those RDMA 
> requirements; my view is that intel is going to build them 
> chips - so it can be done on a
> $5 board off the pacific rim. This takes most of the small 
> players out of the market.
> 
> > Anyways, did not mean to start an argument - with all the 
> new CPU, bus 
> > and HBA technologies coming to the market it will be another 18-24 
> > months before we know what works and what doesn't...
> 
> Agreed. Would you like to invest on something that will obsoleted in
> 18-24 months though? OR even not obsoleted, but holds that 
> uncertainty?
> I think thats the risk facing you when you are in the offload 
> bussiness.

Well.. Any business has risks, this one doesn't seem to be higher than
others :-)
I view 18-26 mo timeframe as a start of the offload mass-adoption, not the
end of it.

In our tests, the bus contention and the %cpu are mostly orthogonal
problems; PCI-X DDR and PCI-Express will help but only to a point.
(BTW this is all related to the higher end systems - 2-4 way and above,
running 10GbE NICs. Client is a different story, cpu is mostly "free"
there).
My sense is that (unlike on previous cycles) the "slow host, fast network"
scenario is here to stay for a long while, and will have to be addressed one
way or another - whether it is a full TOE+RDMA offload in a longer run, or
an improvement to "static" offloads. 
In server space, applications will never be happy with less than 80% cpu.

Leonid

> 
> Here are results for Hifn 7956 ref board on 2.6GHz P4 (HT) 
> system, kernel  2.6.6 SMP as compared to a s/ware only setup 
> on same machine.
> [Name of tester withheld to protect privacy].
> 
> first column - algo, second - packet size, third - time in us 
> spend by hw crypto, forth - time in us spent by sw crypto:
> 
> des   64:       28      3
> des  128:       29      6
> des  192:       33      9
> des  256:       33      12
> des  320:       37      15
> des  384:       38      18
> des  448:       41      21
> des  512:       42      23
> des  576:       45      26
> des  640:       46      29
> des  704:       49      33
> des  768:       50      35
> des  832:       53      38
> des  896:       54      41
> des  960:       57      44
> des 1024:       58      47
> des 1088:       61      50
> des 1152:       62      53
> des 1216:       66      56
> des 1280:       66      59
> des 1344:       70      62
> des 1408:       71      65
> des 1472:       74      68
> des3_ede   64:  28      6
> des3_ede  128:  30      13
> des3_ede  192:  34      20
> des3_ede  256:  43      26
> des3_ede  320:  38      33
> des3_ede  384:  48      40
> des3_ede  448:  44      45
> des3_ede  512:  54      53
> des3_ede  576:  50      60
> des3_ede  640:  59      67
> des3_ede  704:  55      74
> des3_ede  768:  66      78
> des3_ede  832:  61      85
> des3_ede  896:  72      94
> des3_ede  960:  67      100
> des3_ede 1024:  77      107
> des3_ede 1088:  73      114
> des3_ede 1152:  82      121
> des3_ede 1216:  79      127
> des3_ede 1280:  88      128
> des3_ede 1344:  84      135
> des3_ede 1408:  94      147
> des3_ede 1472:  90      153
> aes   64:       28      2
> aes  192:       33      6
> aes  320:       37      10
> aes  448:       46      15
> aes  576:       53      19
> aes  704:       53      23
> aes  832:       65      28
> aes  960:       66      32
> aes 1088:       71      37
> aes 1216:       80      41
> aes 1344:       83      45
> aes 1472:       92      50
> 
> Moral of the data above: The 2.6Ghz is already showing signs 
> of obsoleting the hifn crypto offloader[2]. I think it took 
> less than a year for it to happen.
> 
> cheers,
> jamal
> 
> [1] I also like the MIPS.com approach to SMT
> 
> [2] There are actually issues with some of the crypto 
> offloading in Linux; however this does serve as a good example.
> 


  reply	other threads:[~2004-09-16 15:07 UTC|newest]

Thread overview: 58+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2004-09-15 19:33 Jeff Garzik
2004-09-15 20:04 ` Paul Jakma
2004-09-15 19:14   ` Alan Cox
2004-09-15 20:41     ` Jeff Garzik
2004-09-15 21:01       ` David S. Miller
2004-09-15 21:08         ` Jeff Garzik
2004-09-15 21:13           ` David S. Miller
2004-09-15 21:23             ` Jeff Garzik
2004-09-15 21:29               ` David S. Miller
2004-09-15 22:26                 ` Jeff Garzik
2004-09-15 23:29                 ` Leonid Grossman
2004-09-24 13:07                   ` Lennert Buytenhek
2004-09-24 13:21                     ` Leonid Grossman
2004-09-24 18:09                       ` Lennert Buytenhek
2004-09-24 19:39                         ` Joel Jaeggli
2004-09-16  0:57               ` jamal
2004-09-16  5:25                 ` Leonid Grossman
2004-09-16  9:57                   ` jamal
2004-09-16 14:57                     ` Leonid Grossman [this message]
2004-09-16  9:29               ` Lincoln Dale
2004-09-16 12:19                 ` Alan Cox
2004-09-16 13:33                   ` Andi Kleen
2004-09-16 12:57                     ` Alan Cox
2004-09-16 22:37                       ` Lincoln Dale
2004-09-17 13:38                         ` Jörn Engel
2004-09-15 22:31             ` Jeff Garzik
2004-09-15 21:15         ` Michael Richardson
2004-09-15 20:53     ` David S. Miller
2004-09-16  1:05       ` Andrea Arcangeli
2004-09-15 21:10     ` David Lang
2004-09-15 23:05     ` Paul Jakma
2004-09-15 20:26   ` Neil Horman
2004-09-15 21:03     ` Wes Felter
2004-09-15 21:15       ` Jeff Garzik
     [not found]         ` <ciacj6$bfj$1@sea.gmane.org>
2004-09-15 21:42           ` Jeff Garzik
2004-09-15 21:25       ` Imran Badr
2004-09-16 11:37       ` Neil Horman
2004-09-16  5:51     ` Matt Porter
2004-09-15 21:36   ` Deepak Saxena
2004-09-15 23:03     ` Paul Jakma
2004-09-24 13:11     ` Lennert Buytenhek
2004-09-15 21:59   ` Tony Lee
2004-09-15 20:11 ` David Stevens
2004-09-15 20:16   ` David Schwartz
2004-09-15 20:25   ` Jeff Garzik
2004-09-15 20:54     ` Neil Horman
2004-09-15 21:41   ` Joel Jaeggli
2004-09-16  6:33   ` Valdis.Kletnieks
2004-09-17  6:46   ` Eric Mudama
2004-09-17 14:15     ` Alan Cox
2004-09-17 20:27     ` Valdis.Kletnieks
2004-09-17 20:36       ` David Lang
2004-09-17 23:20         ` Tony Lee
2004-09-17 23:36           ` Leonid Grossman
2004-09-22 23:25       ` Eric Mudama
2004-09-16  9:03 ` Lars Marowsky-Bree
     [not found] <200409162034.i8GKYq39023185@guinness.s2io.com>
2004-09-22 20:18 ` Nivedita Singhvi
2004-09-23  4:46   ` Leonid Grossman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=200409161458.i8GEvtqH022604@guinness.s2io.com \
    --to=leonid.grossman@s2io.com \
    --cc=alan@lxorguk.ukuu.org.uk \
    --cc=davem@davemloft.net \
    --cc=hadi@cyberus.ca \
    --cc=jgarzik@pobox.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@oss.sgi.com \
    --cc=paul@clubi.ie \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®