Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)

[Miquel van Smoorenburg <miquels@cistron.nl>]

Hello,

	I'm running Linux 2.6.0 with a 3ware 8506 controller in
hardware RAID5 mode. The RAID5 array is built of 7+1 200 MB SATA
disks.

Now it appears that more "mappings" on the array have a bad
influence on speed. /dev/sda is the fastest, /dev/sda1 is quite
a bit slower, LVM on /dev/sda is slower yet and LVM on /dev/sda1
is the slowest.

Results of a dd write directly on the RAID5 whole device:

time -p sh -c "dd if=/dev/zero of=/dev/sda bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 5.120484 seconds (79992437 bytes/sec)
real 5.23
user 0.03
sys 0.98

Results on the first partition:

time -p sh -c "dd if=/dev/zero of=/dev/sda1 bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 6.141125 seconds (66697878 bytes/sec)
real 6.24
user 0.04
sys 2.06

LVM is created on /dev/sda or /dev/sda1 using the following commands:
pvcreate $DEVICE
vgcreate vg0 $DEVICE
lvcreate -L 100G -n test vg0

Results for a logical volume on /dev/sda:

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.624306 seconds (47493676 bytes/sec)
real 8.65
user 0.05
sys 1.03

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
Results for a logical volume on /dev/sda1:
100000+0 records in
100000+0 records out
409600000 bytes transferred in 9.223930 seconds (44406234 bytes/sec)
real 9.27
user 0.07
sys 0.97

This is reproducable. I also tested this on a software RAID5
array, but with SW RAID5 it doesn't matter if I use the whole
disk, a partition, or a logical volume; performance is always
the same (slightly faster than HW RAID5; 100 MB/sec).

Any idea how to get the performance back ? How can I go about
debugging this ?

Mike.