From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: linux-kernel@vger.kernel.org
Cc: linux-scsi@vger.kernel.org, linux-net@vger.kernel.org,
Alan Piszcz <ap@solarrain.com>
Subject: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?
Date: Sat, 5 Feb 2011 14:35:52 -0500 (EST) [thread overview]
Message-ID: <alpine.DEB.2.00.1102051427360.8518@p34.internal.lan> (raw)
Hi,
I have two identical X8DTH-6's, each with 5520s XEONS..
=== RAID
Two 3ware RAID-0's (testing only) with 24 drives (2TB WD).
Read: 1.2Gbyte/sec
Write: 1.2Gbyte/sec
=== NETWORK
When I run iperf between the two 10GbE interfaces, I get 1.1-1.2Gbyte/sec.
MTU=9000 on both hosts.
== COPY OVER NETWORK
When I copy using cp/NFS, tar/nc, etc.. I only see an average of about
250MiB/s, I also have two desktop boards (8GB ram/ea) from which I get
500-550MiB/s from (the raid is slower on the desktop boards, it gets what
the RAID can read it), how come these server boards cannot push > 1Gbyte/sec
sustained?
I know on this board:
CPU1 => SLOT 1,2,3
CPU2 => SLOT 4,5,6,7
The RAID card is plugged into 2 (CPU1)
The 10GbE card is plugged into 6 (CPU2)
Should both cards be plugged into the same set of slots controlled by the
same CPU?
Has anyone experienced anything like this before, e.g., > 1Gbyte/sec read
& write for network and 3ware RAID but when using them in combination, it
is slow.
OS = CentOS 5.5 x86_64
Filesystem = XFS
NIC = 10GbE AT2 Server Adapter
No single component is < 1Gbyte/sec but when data is transferred from
hostA to hostB, then it degrades to 250MiB/s.
Justin.
next reply other threads:[~2011-02-05 19:35 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-05 19:35 Justin Piszcz [this message]
2011-02-05 19:42 ` Jean Gobin
2011-02-05 19:54 ` Justin Piszcz
2011-02-05 20:18 ` Justin Piszcz
2011-02-05 20:39 ` Stan Hoeppner
2011-02-05 20:45 ` Emmanuel Florac
2011-02-05 21:05 ` Justin Piszcz
2011-02-05 22:06 ` Dr. David Alan Gilbert
2011-02-05 22:30 ` Justin Piszcz
2011-02-05 22:56 ` Justin Piszcz
2011-02-05 23:54 ` Stan Hoeppner
2011-02-06 1:08 ` Justin Piszcz
2011-02-06 3:16 ` Stan Hoeppner
2011-02-06 10:16 ` Justin Piszcz
2011-02-06 13:46 ` Justin Piszcz
2011-02-06 16:44 ` Emmanuel Florac
2011-02-06 16:55 ` Zdenek Kaspar
2011-02-06 17:52 ` Justin Piszcz
2011-02-06 22:01 ` Stan Hoeppner
2011-02-07 3:59 ` Julian Calaby
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.00.1102051427360.8518@p34.internal.lan \
--to=jpiszcz@lucidpixels.com \
--cc=ap@solarrain.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®