* Only ~85MiB/s (e1000e/write) to ~310MiB/s RAID-0 on ATOM board?
@ 2011-01-07 20:18 Justin Piszcz
2011-01-16 18:49 ` Ian E. Morgan
0 siblings, 1 reply; 3+ messages in thread
From: Justin Piszcz @ 2011-01-07 20:18 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-net, Alan Piszcz
Hi,
Motherboard: Supermicro X7SPA-HF (MINI-ITX)
I tried changing various settings with ethtool, ifconfig, and e1000e
module parameters.
=> I get ~310MiB/s write with 5 Raptor 150s in RAID-0
atom166:/r1# dd if=/dev/zero of=bigfile bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 34.7194 s, 309 MB/s
atom166:/r1#
=> I get 239MiB/s read with 5 Raptor 150s in RAID-0
# dd if=bigfile of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 44.9199 s, 239 MB/s
When I use samba to push files to this RAID I only get between 80-90MiB/s
instead of the usual 100-120MiB/s I see between my main (ATX) boards.. It
seems to hover around 85MiB/s average..
I also tried an external PCI-e 1.0 Intel NIC, made no difference. From
the NM10 chipset block diagram, there should be enough bandwidth to fully
saturate a gigabit link, is the CPU (Dual core Atom D510) too slow to
achieve the maximum speed?
FTP is slightly faster:
`20101030-t500.tib' at 400541736 (3%) 88.72M/s eta:2m [Receiving data]
`20101030-t500.tib' at 565772280 (4%) 91.51M/s eta:2m [Receiving data]
`20101030-t500.tib' at 982224624 (7%) 94.75M/s eta:2m [Receiving data]
`20101030-t500.tib' at 1967568384 (15%) 96.39M/s eta:2m [Receiving data]
`20101030-t500.tib' at 2299966640 (17%) 96.90M/s eta:2m [Receiving data]
Getting the same file to an ATX motherboard:
`20101030-t500.tib' at 1020865896 (7%) 106.13M/s eta:2m [Receiving data]
`20101030-t500.tib' at 1394552704 (10%) 107.53M/s eta:2m [Receiving data]
`20101030-t500.tib' at 1861689088 (14%) 108.49M/s eta:97s [Receiving data]
`20101030-t500.tib' at 2702497824 (21%) 109.56M/s eta:88s [Receiving data]
`20101030-t500.tib' at 3636710872 (28%) 110.27M/s eta:80s [Receiving data]
Why cannot I achieve the last 10-20MiB/s in performance on the Atom board?
Also, the other direction (READ) I get 105-110MiB/s:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 37260 328 3888560 0 0 3187 6746 2295 309 1 13 84 1
0 0 0 36632 328 3890492 0 0 105184 0 23332 4924 3 21 75 0
0 0 0 35516 328 3893688 0 0 105616 0 23567 5143 3 22 75 0
1 0 0 35888 328 3894428 0 0 106404 0 23247 5167 3 21 76 0
0 0 0 35392 328 3897068 0 0 106472 0 23351 5221 3 21 76 0
What is causing the bottleneck for write?
Justin.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Only ~85MiB/s (e1000e/write) to ~310MiB/s RAID-0 on ATOM board?
2011-01-07 20:18 Only ~85MiB/s (e1000e/write) to ~310MiB/s RAID-0 on ATOM board? Justin Piszcz
@ 2011-01-16 18:49 ` Ian E. Morgan
2011-01-16 19:08 ` Justin Piszcz
0 siblings, 1 reply; 3+ messages in thread
From: Ian E. Morgan @ 2011-01-16 18:49 UTC (permalink / raw)
To: Justin Piszcz; +Cc: linux-kernel, linux-net, Alan Piszcz
Justin,
I have two Supermicro 5015a-EHF-D525's. They use the same Intel 82574L
NICs as your slightly older board.
I asked myself the same questions as you just recently, and my
research into multi-queue NIC support led to:
Receive Packet Steering
Take a look at the output of NET_TX and NET_RX in /proc/softirq.
The interrupts were not balanced across the cpus until I did:
echo f > /sys/devices/pci0000:00/0000:00:1c.1/0000:02:00.0/net/eth0/queues/rx-0/rps_cpus
("f" being a bitmask of cpus)
Then the interrupts started balancing across all cpus.
I haven't had a chance to try this in a high throughput transfer
situation yet, as I just happened to look into this last night.
Hope it leads you down the right path, and please let me know if it
does or doesn't help, or any other solutions you've found or come up
with.
--
Ian Morgan
On Fri, Jan 7, 2011 at 15:18, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> Hi,
>
> Motherboard: Supermicro X7SPA-HF (MINI-ITX)
>
> I tried changing various settings with ethtool, ifconfig, and e1000e module
> parameters.
>
> => I get ~310MiB/s write with 5 Raptor 150s in RAID-0
> atom166:/r1# dd if=/dev/zero of=bigfile bs=1M count=10240
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB) copied, 34.7194 s, 309 MB/s
> atom166:/r1#
>
>
> => I get 239MiB/s read with 5 Raptor 150s in RAID-0
> # dd if=bigfile of=/dev/null bs=1M count=10240
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB) copied, 44.9199 s, 239 MB/s
>
> When I use samba to push files to this RAID I only get between 80-90MiB/s
> instead of the usual 100-120MiB/s I see between my main (ATX) boards.. It
> seems to hover around 85MiB/s average..
>
> I also tried an external PCI-e 1.0 Intel NIC, made no difference. From the
> NM10 chipset block diagram, there should be enough bandwidth to fully
> saturate a gigabit link, is the CPU (Dual core Atom D510) too slow to
> achieve the maximum speed?
>
> FTP is slightly faster:
> `20101030-t500.tib' at 400541736 (3%) 88.72M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 565772280 (4%) 91.51M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 982224624 (7%) 94.75M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 1967568384 (15%) 96.39M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 2299966640 (17%) 96.90M/s eta:2m [Receiving data]
>
> Getting the same file to an ATX motherboard:
> `20101030-t500.tib' at 1020865896 (7%) 106.13M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 1394552704 (10%) 107.53M/s eta:2m [Receiving data]
> `20101030-t500.tib' at 1861689088 (14%) 108.49M/s eta:97s [Receiving data]
> `20101030-t500.tib' at 2702497824 (21%) 109.56M/s eta:88s [Receiving data]
> `20101030-t500.tib' at 3636710872 (28%) 110.27M/s eta:80s [Receiving data]
>
> Why cannot I achieve the last 10-20MiB/s in performance on the Atom board?
>
> Also, the other direction (READ) I get 105-110MiB/s:
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 0 0 37260 328 3888560 0 0 3187 6746 2295 309 1 13 84
> 1
> 0 0 0 36632 328 3890492 0 0 105184 0 23332 4924 3 21
> 75 0
> 0 0 0 35516 328 3893688 0 0 105616 0 23567 5143 3 22
> 75 0
> 1 0 0 35888 328 3894428 0 0 106404 0 23247 5167 3 21
> 76 0
> 0 0 0 35392 328 3897068 0 0 106472 0 23351 5221 3 21
> 76 0
>
> What is causing the bottleneck for write?
>
> Justin.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: Only ~85MiB/s (e1000e/write) to ~310MiB/s RAID-0 on ATOM board?
2011-01-16 18:49 ` Ian E. Morgan
@ 2011-01-16 19:08 ` Justin Piszcz
0 siblings, 0 replies; 3+ messages in thread
From: Justin Piszcz @ 2011-01-16 19:08 UTC (permalink / raw)
To: Ian E. Morgan; +Cc: linux-kernel, linux-net, Alan Piszcz
On Sun, 16 Jan 2011, Ian E. Morgan wrote:
> Justin,
>
> I have two Supermicro 5015a-EHF-D525's. They use the same Intel 82574L
> NICs as your slightly older board.
> I asked myself the same questions as you just recently, and my
> research into multi-queue NIC support led to:
>
> Receive Packet Steering
>
> Take a look at the output of NET_TX and NET_RX in /proc/softirq.
> The interrupts were not balanced across the cpus until I did:
>
> echo f > /sys/devices/pci0000:00/0000:00:1c.1/0000:02:00.0/net/eth0/queues/rx-0/rps_cpus
> ("f" being a bitmask of cpus)
>
> Then the interrupts started balancing across all cpus.
> I haven't had a chance to try this in a high throughput transfer
> situation yet, as I just happened to look into this last night.
>
> Hope it leads you down the right path, and please let me know if it
> does or doesn't help, or any other solutions you've found or come up
> with.
>
> --
> Ian Morgan
Hi Ian,
Thanks for the response & suggestion!
Tried it on kernel 2.6.37:
Made sure it was enabled:
# grep RPS .config
CONFIG_RPS=y
Only achieved 79.98M/s on > 4 GB of data via FTP (downloading), does not
appear to change much.
# cat /sys/class/net/eth1/queues/rx-0/rps_cpus
0
# echo f > /sys/class/net/eth1/queues/rx-0/rps_cpus
# cat /sys/class/net/eth1/queues/rx-0/rps_cpus
f
#
# cat /proc/softirqs
CPU0 CPU1 CPU2 CPU3
HI: 0 0 0 0
TIMER: 508490 507869 508325 508253
NET_TX: 22933 24274 23800 22124
NET_RX: 94412 218903 93273 95787
BLOCK: 63945 22646 21324 8084
BLOCK_IOPOLL: 0 0 0 0
TASKLET: 8 4 2 5
SCHED: 244572 218617 262071 242491
HRTIMER: 0 0 0 0
RCU: 52859 53519 53715 55199
What speed(s) do you achieve with the D525s?
If anyone finds out why these boards cannot achieve > 90-100MiB/s to disk,
I'd be curious to know why, thanks.
Justin.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-01-16 19:08 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-07 20:18 Only ~85MiB/s (e1000e/write) to ~310MiB/s RAID-0 on ATOM board? Justin Piszcz
2011-01-16 18:49 ` Ian E. Morgan
2011-01-16 19:08 ` Justin Piszcz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®