Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)

Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)

[Miquel van Smoorenburg]

Hello,

	I'm running Linux 2.6.0 with a 3ware 8506 controller in
hardware RAID5 mode. The RAID5 array is built of 7+1 200 MB SATA
disks.

Now it appears that more "mappings" on the array have a bad
influence on speed. /dev/sda is the fastest, /dev/sda1 is quite
a bit slower, LVM on /dev/sda is slower yet and LVM on /dev/sda1
is the slowest.

Results of a dd write directly on the RAID5 whole device:

time -p sh -c "dd if=/dev/zero of=/dev/sda bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 5.120484 seconds (79992437 bytes/sec)
real 5.23
user 0.03
sys 0.98

Results on the first partition:

time -p sh -c "dd if=/dev/zero of=/dev/sda1 bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 6.141125 seconds (66697878 bytes/sec)
real 6.24
user 0.04
sys 2.06

LVM is created on /dev/sda or /dev/sda1 using the following commands:
pvcreate $DEVICE
vgcreate vg0 $DEVICE
lvcreate -L 100G -n test vg0

Results for a logical volume on /dev/sda:

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.624306 seconds (47493676 bytes/sec)
real 8.65
user 0.05
sys 1.03

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
Results for a logical volume on /dev/sda1:
100000+0 records in
100000+0 records out
409600000 bytes transferred in 9.223930 seconds (44406234 bytes/sec)
real 9.27
user 0.07
sys 0.97

This is reproducable. I also tested this on a software RAID5
array, but with SW RAID5 it doesn't matter if I use the whole
disk, a partition, or a logical volume; performance is always
the same (slightly faster than HW RAID5; 100 MB/sec).

Any idea how to get the performance back ? How can I go about
debugging this ?

Mike.

Re: Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)

[Miquel van Smoorenburg]

In article <20031229125412.GA28262@cistron.nl>,
Miquel van Smoorenburg  <miquels@cistron.nl> wrote:
>Hello,
>
>	I'm running Linux 2.6.0 with a 3ware 8506 controller in
>hardware RAID5 mode. The RAID5 array is built of 7+1 200 MB SATA
>disks.
>
>Now it appears that more "mappings" on the array have a bad
>influence on speed. /dev/sda is the fastest, /dev/sda1 is quite
>a bit slower, LVM on /dev/sda is slower yet and LVM on /dev/sda1
>is the slowest.

For reference, I just took 2.4.24-pre2, applied the device-mapper
patches to it, and re-ran the tests.

Base performance on /dev/sda is a bit slower (70 MB/sec vs 80 MB/sec
raw writespeed), but speeds on /dev/sda1 and /dev/vg0/test are
exactly the same.

So it doesn't look like a 3ware driver or LVM issue as those are
basically the same between the 2.4 and 2.6 systems.

Mike.
-- 
When life hands you lemons, grab the salt and pass the tequila.

Performance issue with non-4K blocksize on 3ware [was: Re: Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)]

[Miquel van Smoorenburg]

In article <cistron.20031229125412.GA28262@cistron.nl>,
Miquel van Smoorenburg  <miquels@cistron.nl> wrote:
>Hello,
>
>	I'm running Linux 2.6.0 with a 3ware 8506 controller in
>hardware RAID5 mode. The RAID5 array is built of 7+1 200 GB SATA
>disks.
>
>Now it appears that more "mappings" on the array have a bad
>influence on speed. /dev/sda is the fastest, /dev/sda1 is quite
>a bit slower, LVM on /dev/sda is slower yet and LVM on /dev/sda1
>is the slowest.

Okay, I found a clue. It appears that by default, the blocksize
of /dev/sda is 4096 and the blocksize of /dev/sda1 is 1024 on my system:

# blockdev --getbsz /dev/sda
4096
# blockdev --getbsz /dev/sda1
1024

On 2.4, setting the blocksize is sticky, on 2.6 it isn't. So I have
to work around that to test with other blocksizes, for example:

# /usr/bin/time -p sh -c "blockdev --setbsz 4096 /dev/stdout; dd if=/dev/zero bs=4k count=100000; sync" > /dev/sda1
100000+0 records in
100000+0 records out
real 5.00
user 0.03
sys 0.89

409600000 bytes in 5 seconds =~ 80 MB/sec, which is quite good and
the same throughput as through /dev/sda. So, it appears that the
3ware driver really really wants a 4K blocksize, otherwise performance
drops at least 30-40%.

Now with LVM on top of it, performance drops again. Say I create a
logical volume on /dev/sda1:

# pvcreate /dev/sda1
# vgcreate vg0 /dev/sda1
# lvcreate -L 100G -n test vg0

# blockdev --getbsz /dev/sda1
1024
# blockdev --getbsz /dev/vg0/test
4096

Performance:
# /usr/bin/time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.512663 seconds (48116553 bytes/sec)
real 8.54
user 0.04
sys 1.02

So, I tried a small patch in dm-table.c to set the blocksize of the
underlying device to 4096:

# diff -u dm-table.c.b4 dm-table.c
--- dm-table.c.b4       2003-12-29 15:18:18.000000000 +0100
+++ dm-table.c  2003-12-30 13:08:06.000000000 +0100
@@ -359,8 +359,10 @@
        r = bd_claim(bdev, _claim_ptr);
        if (r)
                blkdev_put(bdev, BDEV_RAW);
-       else
+       else {
                d->bdev = bdev;
+               set_blocksize(bdev, 4096);
+       }
        return r;
 }
  
While it appears to work, it doesn't seem to make any difference:

# blockdev --getbsz /dev/sda1
4096
# blockdev --getbsz /dev/vg0/test
4096

Performance:
# /usr/bin/time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.528826 seconds (48025367 bytes/sec)
real 8.54
user 0.05
sys 0.98

Not so weird, since in dm-table.c the device is opened in RAW mode
and raw devices do I/O at hardsect_size, right? So that would be
512 bytes, explaining the bad performance.

I'm hoping someone has read this far and can offer me a clue as
how to fix this. Are somehow requests not merged right, and are
small 512 byte requests submitted to the 3ware driver? Is the
3ware driver responsible for merging requests ? I really don't
know anything about the block layer or bio's so I'm kind of
lost here.

Mike.