From: "Leonid Grossman" <leonid.grossman@s2io.com>
To: <hadi@cyberus.ca>
Cc: "'Jeff Garzik'" <jgarzik@pobox.com>,
"'David S. Miller'" <davem@davemloft.net>,
<alan@lxorguk.ukuu.org.uk>, <paul@clubi.ie>, <netdev@oss.sgi.com>,
<linux-kernel@vger.kernel.org>
Subject: RE: The ultimate TOE design
Date: Thu, 16 Sep 2004 07:57:48 -0700 [thread overview]
Message-ID: <200409161458.i8GEvtqH022604@guinness.s2io.com> (raw)
In-Reply-To: <1095328673.1063.130.camel@jzny.localdomain>
> -----Original Message-----
> From: jamal [mailto:hadi@cyberus.ca]
> Sent: Thursday, September 16, 2004 2:58 AM
> To: Leonid Grossman
> Cc: 'Jeff Garzik'; 'David S. Miller';
> alan@lxorguk.ukuu.org.uk; paul@clubi.ie; netdev@oss.sgi.com;
> linux-kernel@vger.kernel.org
> Subject: RE: The ultimate TOE design
>
> On Thu, 2004-09-16 at 01:25, Leonid Grossman wrote:
> >
> > > -----Original Message-----
> > > From: jamal [mailto:hadi@cyberus.ca]
>
> > > On a serious note, I think that PCI-express (if it lives upto its
> > > expectation) will demolish dreams of a lot of these TOE
> investments.
> > > Our problem is NOT the CPU right now (80% idle processing 450Kpps
> > > forwarding). Bus and memory distance/latency are.
> >
> > In servers, both bottlenecks are there - if you look at the cost of
> > TCP and filesystem processing at 10GbE, CPU is a huge problem (and
> > will be for foreseeable future), even for fastest 64-bit systems.
>
> True, but with the bus contention being a non-issue you got
> more of that xeon being available for use (lets say i can use
> 50% more of its capacity then i can do more). IOW, it becomes
> a compute capacity problem mostly - one that you should in
> theory be able to throw more CPU at. SMT (the way power5 and
> some of the network processors do it[1]) should go a long way
> to address both additional compute and hardware threading to
> work around memory latencies. With PCI-express, compute power
> in mini-clustering in the form of AS
> (http://www.asi-sig.org/home) is being plotted as we speak.
> To sumarize: The problem to solve in 24 months maybe 100Gige.
>
> > I agree though that bus and memory are bigger issues, this
> is exactly
> > the reason for all these RDMA over Ethernet investments :-)
>
> And AS does a damn good job at specing all those RDMA
> requirements; my view is that intel is going to build them
> chips - so it can be done on a
> $5 board off the pacific rim. This takes most of the small
> players out of the market.
>
> > Anyways, did not mean to start an argument - with all the
> new CPU, bus
> > and HBA technologies coming to the market it will be another 18-24
> > months before we know what works and what doesn't...
>
> Agreed. Would you like to invest on something that will obsoleted in
> 18-24 months though? OR even not obsoleted, but holds that
> uncertainty?
> I think thats the risk facing you when you are in the offload
> bussiness.
Well.. Any business has risks, this one doesn't seem to be higher than
others :-)
I view 18-26 mo timeframe as a start of the offload mass-adoption, not the
end of it.
In our tests, the bus contention and the %cpu are mostly orthogonal
problems; PCI-X DDR and PCI-Express will help but only to a point.
(BTW this is all related to the higher end systems - 2-4 way and above,
running 10GbE NICs. Client is a different story, cpu is mostly "free"
there).
My sense is that (unlike on previous cycles) the "slow host, fast network"
scenario is here to stay for a long while, and will have to be addressed one
way or another - whether it is a full TOE+RDMA offload in a longer run, or
an improvement to "static" offloads.
In server space, applications will never be happy with less than 80% cpu.
Leonid
>
> Here are results for Hifn 7956 ref board on 2.6GHz P4 (HT)
> system, kernel 2.6.6 SMP as compared to a s/ware only setup
> on same machine.
> [Name of tester withheld to protect privacy].
>
> first column - algo, second - packet size, third - time in us
> spend by hw crypto, forth - time in us spent by sw crypto:
>
> des 64: 28 3
> des 128: 29 6
> des 192: 33 9
> des 256: 33 12
> des 320: 37 15
> des 384: 38 18
> des 448: 41 21
> des 512: 42 23
> des 576: 45 26
> des 640: 46 29
> des 704: 49 33
> des 768: 50 35
> des 832: 53 38
> des 896: 54 41
> des 960: 57 44
> des 1024: 58 47
> des 1088: 61 50
> des 1152: 62 53
> des 1216: 66 56
> des 1280: 66 59
> des 1344: 70 62
> des 1408: 71 65
> des 1472: 74 68
> des3_ede 64: 28 6
> des3_ede 128: 30 13
> des3_ede 192: 34 20
> des3_ede 256: 43 26
> des3_ede 320: 38 33
> des3_ede 384: 48 40
> des3_ede 448: 44 45
> des3_ede 512: 54 53
> des3_ede 576: 50 60
> des3_ede 640: 59 67
> des3_ede 704: 55 74
> des3_ede 768: 66 78
> des3_ede 832: 61 85
> des3_ede 896: 72 94
> des3_ede 960: 67 100
> des3_ede 1024: 77 107
> des3_ede 1088: 73 114
> des3_ede 1152: 82 121
> des3_ede 1216: 79 127
> des3_ede 1280: 88 128
> des3_ede 1344: 84 135
> des3_ede 1408: 94 147
> des3_ede 1472: 90 153
> aes 64: 28 2
> aes 192: 33 6
> aes 320: 37 10
> aes 448: 46 15
> aes 576: 53 19
> aes 704: 53 23
> aes 832: 65 28
> aes 960: 66 32
> aes 1088: 71 37
> aes 1216: 80 41
> aes 1344: 83 45
> aes 1472: 92 50
>
> Moral of the data above: The 2.6Ghz is already showing signs
> of obsoleting the hifn crypto offloader[2]. I think it took
> less than a year for it to happen.
>
> cheers,
> jamal
>
> [1] I also like the MIPS.com approach to SMT
>
> [2] There are actually issues with some of the crypto
> offloading in Linux; however this does serve as a good example.
>
next prev parent reply other threads:[~2004-09-16 15:07 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2004-09-15 19:33 Jeff Garzik
2004-09-15 20:04 ` Paul Jakma
2004-09-15 19:14 ` Alan Cox
2004-09-15 20:41 ` Jeff Garzik
2004-09-15 21:01 ` David S. Miller
2004-09-15 21:08 ` Jeff Garzik
2004-09-15 21:13 ` David S. Miller
2004-09-15 21:23 ` Jeff Garzik
2004-09-15 21:29 ` David S. Miller
2004-09-15 22:26 ` Jeff Garzik
2004-09-15 23:29 ` Leonid Grossman
2004-09-24 13:07 ` Lennert Buytenhek
2004-09-24 13:21 ` Leonid Grossman
2004-09-24 18:09 ` Lennert Buytenhek
2004-09-24 19:39 ` Joel Jaeggli
2004-09-16 0:57 ` jamal
2004-09-16 5:25 ` Leonid Grossman
2004-09-16 9:57 ` jamal
2004-09-16 14:57 ` Leonid Grossman [this message]
2004-09-16 9:29 ` Lincoln Dale
2004-09-16 12:19 ` Alan Cox
2004-09-16 13:33 ` Andi Kleen
2004-09-16 12:57 ` Alan Cox
2004-09-16 22:37 ` Lincoln Dale
2004-09-17 13:38 ` Jörn Engel
2004-09-15 22:31 ` Jeff Garzik
2004-09-15 21:15 ` Michael Richardson
2004-09-15 20:53 ` David S. Miller
2004-09-16 1:05 ` Andrea Arcangeli
2004-09-15 21:10 ` David Lang
2004-09-15 23:05 ` Paul Jakma
2004-09-15 20:26 ` Neil Horman
2004-09-15 21:03 ` Wes Felter
2004-09-15 21:15 ` Jeff Garzik
[not found] ` <ciacj6$bfj$1@sea.gmane.org>
2004-09-15 21:42 ` Jeff Garzik
2004-09-15 21:25 ` Imran Badr
2004-09-16 11:37 ` Neil Horman
2004-09-16 5:51 ` Matt Porter
2004-09-15 21:36 ` Deepak Saxena
2004-09-15 23:03 ` Paul Jakma
2004-09-24 13:11 ` Lennert Buytenhek
2004-09-15 21:59 ` Tony Lee
2004-09-15 20:11 ` David Stevens
2004-09-15 20:16 ` David Schwartz
2004-09-15 20:25 ` Jeff Garzik
2004-09-15 20:54 ` Neil Horman
2004-09-15 21:41 ` Joel Jaeggli
2004-09-16 6:33 ` Valdis.Kletnieks
2004-09-17 6:46 ` Eric Mudama
2004-09-17 14:15 ` Alan Cox
2004-09-17 20:27 ` Valdis.Kletnieks
2004-09-17 20:36 ` David Lang
2004-09-17 23:20 ` Tony Lee
2004-09-17 23:36 ` Leonid Grossman
2004-09-22 23:25 ` Eric Mudama
2004-09-16 9:03 ` Lars Marowsky-Bree
[not found] <200409162034.i8GKYq39023185@guinness.s2io.com>
2004-09-22 20:18 ` Nivedita Singhvi
2004-09-23 4:46 ` Leonid Grossman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=200409161458.i8GEvtqH022604@guinness.s2io.com \
--to=leonid.grossman@s2io.com \
--cc=alan@lxorguk.ukuu.org.uk \
--cc=davem@davemloft.net \
--cc=hadi@cyberus.ca \
--cc=jgarzik@pobox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@oss.sgi.com \
--cc=paul@clubi.ie \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®