Re: Limits of the 965 chipset & 3 PCI-e cards/southbridge? ~774MiB/s peak for read, ~650MiB/s peak for write?

[Justin Piszcz <jpiszcz@lucidpixels.com>]

On Sun, 1 Jun 2008, Daniel J Blueman wrote:

> On 1 Jun, 10:50, Justin Piszcz <jpis...@lucidpixels.com> wrote:
>> I have 12 enterprise-class seagate 1TiB disks on a 965 desktop board and
>> it appears I have hit the limit, if I were able to get the maximum speed
>> of all drives, ~70MiB/avg * 12 = 840MiB/s but it seems to stop aound 774
>> MiB/s (currently running badblocks on all drives)..
>
> Nice test. The Seagate 7200.11 drives deliver 120MB/s (outer zone,
> raw) each, and there is an issue with CFQ dispatching requests; see:
>
> http://groups.google.co.uk/group/linux.kernel/browse_thread/thread/b88264b084a2dfe0/a1bc0f67837bad00
>
> A quick workaround tweak is:
>
> # echo 0 >/sys/block/sda/queue/iosched/slice_idle
>
> Does this help any? This gives the difference of ~68MB/s vs ~120MB/s
> on my 7200.11 ;-) .
>
> That said, the i965 chipset is fairly contemporary, but if that 2GB/s
> DMI connection is the bidirectional bandwidth (likely), then maybe
> you're hitting that limit: Intel's DMI bus is based on PCIe, thus will
> use 128 byte PCI-e Max Payload packets (as in the rest of the
> chipset), which IIRC theoretically maxes you out near 800MB/s.
>
> The X48 chipset may allow you to crank the Max Payload to 256 (setpci
> and the Intel chipset docs), if it doesn't default to 256, like in
> 5400 server chipsets. This chipset is where the fun really starts eg
> hdparm -T giving >10GB/s, like in Itanium2s ;-) .
>
> Thanks,
>  Daniel
> -- 
> Daniel J Blueman
>

slice 0 does help a little with my 10-disk raid5 for sequential 
read/writes:

http://home.comcast.net/~jpiszcz/raid/20080601/raid5.html