On Monday, October 25, 2004 6:14 pm, Jesse Barnes wrote:
> I've been doing some simple disk I/O benchmarking with an eye towards
> improving large, striped volume bandwidth.  I ran some tests on individual
> disks and filesystems to establish a baseline and found that things
> generally scale quite well:
>
> o one thread/disk using O_DIRECT on the block device
>   read avg: 2784.81 MB/s
>   write avg: 2585.60 MB/s
>
> o one thread/disk using O_DIRECT + filesystem
>   read avg: 2635.98 MB/s
>   write avg: 2573.39 MB/s
>
> o one thread/disk using buffered I/O + filesystem
>   read w/default (128) block/*/queue/read_ahead_kb avg: 2626.25 MB/s
>   read w/max (4096) block/*/queue/read_ahead_kb avg: 2652.62 MB/s
>   write avg: 1394.99 MB/s
>
> Configuration:
>   o 8p sn2 ia64 box
>   o 8GB memory
>   o 58 disks across 16 controllers
>     (4 disks for 10 of them and 3 for the other 6)
>   o aggregate I/O bw available is about 2.8GB/s
>
> Test:
>   o one I/O thread per disk, round robined across the 8 CPUs
>   o each thread did ~450MB of I/O depending on the test (ran for 10s)
>     Note: the total was > 8GB so in the buffered read case not everything
>     could be cached

More results here.  I've run some tests on a large dm striped volume formatted 
with XFS.  It had 64 disks with a 64k stripe unit (XFS was made aware of this 
at format time), and I explicitly set the readahead using blockdev to 524288 
blocks.  The results aren't as bad as my previous runs, but are still much 
slower than they ought to be I think given the direct I/O results above.  
This is after a fresh mount, so the pagecache was empty when I started the 
tests.

o one thread on one large volume using buffered I/O + filesystem
  read (1 thread, one volume, 131072 blocks/request) avg: ~931 MB/s
  write (1 thread, one volume, 131072 blocks/request) avg: ~908 MB/s

I'm intentionally issuing very large reads and writes here to take advantage 
of the striping, but it looks like both the readahead and regular buffered 
I/O code will split the I/O into page sized chunks?  The call chain is pretty 
long, but it looks to me like do_generic_mapping_read() will split the reads 
up by page and issue them independently to the lower levels.  In the direct 
I/O case, up to 64 pages are issued at a time, which seems like it would help 
throughput quite a bit.  The profile seems to confirm this.  Unfortunately I 
didn't save the vmstat output for this run (and now the fc switch is 
misbehaving so I have to fix that before I run again), but iirc the system 
time was pretty high given that only one thread was issuing I/O.

So maybe a few things need to be done:
  o set readahead to larger values by default for dm volumes at setup time
    (the default was very small)
  o maybe bypass readahead for very large requests?
    if the process is doing a huge request, chances are that readahead won't
    benefit it as much as a process doing small requests
  o not sure about writes yet, I haven't looked at that call chain much yet

Does any of this sound reasonable at all?  What else could be done to make the 
buffered I/O layer friendlier to large requests?

Thanks,
Jesse