Re: Drive performance bottleneck

[Andrew Morton <akpm@osdl.org>]

Ian Godin <Ian.Godin@lowrydigital.com> wrote:
>
> 
>    I am trying to get very fast disk drive performance and I am seeing 
> some interesting bottlenecks.  We are trying to get 800 MB/sec or more 
> (yes, that is megabytes per second).  We are currently using 
> PCI-Express with a 16 drive raid card (SATA drives).  We have achieved 
> that speed, but only through the SG (SCSI generic) driver.  This is 
> running the stock 2.6.10 kernel.  And the device is not mounted as a 
> file system.  I also set the read ahead size on the device to 16KB 
> (which speeds things up a lot):
> ...
> samples  %        symbol name
> 848185    8.3510  __copy_to_user_ll
> 772172    7.6026  do_anonymous_page
> 701579    6.9076  _spin_lock_irq
> 579024    5.7009  __copy_user_intel
> 361634    3.5606  _spin_lock
> 343018    3.3773  _spin_lock_irqsave
> 307462    3.0272  kmap_atomic
> 193327    1.9035  page_fault

Something funny is happening here - it looks like there's plenty of CPU
capacity left over.

It's odd that you're getting a lot of pagefaults in this test but not with
the sg_dd test, too.  I wonder why dd is getting so many pagefaults?  (I
recall that sg_dd did something cheaty, but I forget what it was).

Could you monitor the CPU load during the various tests?  If the `dd'
workload isn't pegging the CPU then it could be that there's something
wrong with the I/O submission patterns.