From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: Stan Hoeppner <stan@hardwarefreak.com>
Cc: "Dr. David Alan Gilbert" <linux@treblig.org>,
Emmanuel Florac <eflorac@intellique.com>,
linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-net@vger.kernel.org, Alan Piszcz <ap@solarrain.com>
Subject: Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?
Date: Sun, 6 Feb 2011 08:46:08 -0500 (EST) [thread overview]
Message-ID: <alpine.DEB.2.00.1102060817590.28109@p34.internal.lan> (raw)
In-Reply-To: <alpine.DEB.2.00.1102060515230.28109@p34.internal.lan>
On Sun, 6 Feb 2011, Justin Piszcz wrote:
>
>
> On Sat, 5 Feb 2011, Stan Hoeppner wrote:
>
>> Justin Piszcz put forth on 2/5/2011 7:08 PM:
>>
Hi,
1. Defaults below:
sysctl -w net.core.wmem_max=131071
sysctl -w net.core.rmem_max=131071
sysctl -w net.core.wmem_default=118784
sysctl -w net.core.rmem_default=118784
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="379104 505472 758208"
sysctl -w net.ipv4.tcp_wmem="4096 16384 4194304"
sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
sysctl -w net.ipv4.udp_mem="379104 505472 758208"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1024
2. Optimized settings, for > 800MiB/:
# for 3ware raid, use 16384 readahead, > 16384 readahead, no improvement
blockdev --setra 16384 /dev/sda
# not sure if this helps much
ethtool -K eth0 lro on
# seems to get performance > 600-700MiB/s faster
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_wmem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_rmem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1048576
# the main option that makes all of the difference, the golden option
# is the rszie and wsize of 1megabyte below:
10.0.1.4:/r1 /nfs/box2/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=1048576,wsize=1048576 0 0
CPU utilization:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2069 root 20 0 18640 1304 688 R 91 0.0 0:15.50 cp
703 root 20 0 0 0 0 S 25 0.0 2:46.95 kswapd0
With a single copy I get roughly 700-800MiB/s:
Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### #### Curr: 808.71 MByte/s
###################### #################### #### Avg: 706.11 MByte/s
###################### #################### #### Min: 0.00 MByte/s
###################### #################### #### Max: 860.17 MByte/s
###################### #################### #### Ttl: 344.70 GByte
With two copies I get up to 830-850MiB/s:
Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ #### Curr: 846.61 MByte/s
############################################ #### Avg: 683.14 MByte/s
############################################ #### Min: 0.00 MByte/s
############################################ #### Max: 860.17 MByte/s
############################################ #### Ttl: 305.71 GByte
Using a 4MiB r/w size with NFS improves performance to sustain > 750MiB/s
a little better I think:
10.0.1.4:/r1 /nfs/box2/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=4194304,wsize=4194304 0
Anyhow, roughly 750-850MiB/s it would be nice to get 1Gbyte/sec but I guess
the kerrnel (or my HW, CPU not fast enough) is not there yet.
Also found a good doc from RedHat:
http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf
Justin.
next prev parent reply other threads:[~2011-02-06 13:46 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-05 19:35 Justin Piszcz
2011-02-05 19:42 ` Jean Gobin
2011-02-05 19:54 ` Justin Piszcz
2011-02-05 20:18 ` Justin Piszcz
2011-02-05 20:39 ` Stan Hoeppner
2011-02-05 20:45 ` Emmanuel Florac
2011-02-05 21:05 ` Justin Piszcz
2011-02-05 22:06 ` Dr. David Alan Gilbert
2011-02-05 22:30 ` Justin Piszcz
2011-02-05 22:56 ` Justin Piszcz
2011-02-05 23:54 ` Stan Hoeppner
2011-02-06 1:08 ` Justin Piszcz
2011-02-06 3:16 ` Stan Hoeppner
2011-02-06 10:16 ` Justin Piszcz
2011-02-06 13:46 ` Justin Piszcz [this message]
2011-02-06 16:44 ` Emmanuel Florac
2011-02-06 16:55 ` Zdenek Kaspar
2011-02-06 17:52 ` Justin Piszcz
2011-02-06 22:01 ` Stan Hoeppner
2011-02-07 3:59 ` Julian Calaby
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.00.1102060817590.28109@p34.internal.lan \
--to=jpiszcz@lucidpixels.com \
--cc=ap@solarrain.com \
--cc=eflorac@intellique.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=linux@treblig.org \
--cc=stan@hardwarefreak.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®