Re: 3Ware 9500S-12 RAID controller -- poor performance

["Miquel van Smoorenburg" <miquels@cistron.nl>]

In article <20050930065058.84446.qmail@web30315.mail.mud.yahoo.com>,
subbie subbie  <subbie_subbie@yahoo.com> wrote:
>I'm using a 3Ware 9500S-12 card and am able to produce
>up to 400MB/s sustained read from my 12-disk 4.1TB
>RAID5 SATA array, 128MB cache onboard, ext3 formatted.
>  All is well when performing a single read -- it
>works nice and fast.
>
>The system is a web server, serving mid-size files
>(50MB, each, on average).  All hell breaks loose when
>doing many concurrent reads, anywhere between 200 to
>400 concurrent streams things simply grind to a halt
>and the system transfers a maximum of 12-14MB/s.

There are a couple of things you should do:

1. Use the CFQ I/O scheduler, and increase nr-requests:
   echo cfq > /sys/block/hda/queue/scheduler
   echo 1024 > /sys/block/hda/queue/nr_requests

2. Make sure that your filesystem knows about the stripe size
   and number of disks in the array. E.g. for a raid5 array
   with a stripe size of 64K and 6 disks (effectively 5,
   because in every stripe-set there is on disk doing parity):

   # ext3 fs, 5 disks, 64K stripe, units in 4K blocks
   mkfs -text3 -E stride=$((64/4))

   # xfs, 5 disks, 64K stripe, units in 512 bytes
   mkfs -txfs -d sunit=$((64*2)) -d swidth=$((5*64*2))

3. Don't use partitions. Partions do not start on a multiple of
   the (stripe_size * nr_disks), so your I/O will be misaligned
   and the settings in (2) will have no or an adverse effect.
   If you must use partitions, either build them manually
   with sfdisk so that partitions do start on that multiple,
   or use LVM.

4. Reconsider your stripe size for streaming large files.
   If you have say 4 disks, and a 64K
   stripe size, then a read of a block of 256K will busy all 4 disks.
   Many simultaneous threads reading blocks of 256K will result in
   trashings disks as they all want to read from all 4 disks .. so
   in that case, using a stripesize of 256K will make things better.
   One read of 256K (in the ideal, aligned case) will just keep one
   disk busy. 4 reads can happen in parallel without trashing. Esp.
   in this case, you need the alignment I talked about in (3).

5. Defragment the files.
   If the files are written sequentially, they will not be fragmented.
   But if they were stored by writing to thousands of them appending
   a few K at a time in round-robin fashion, you need to defragment..
   in the case of XFS, run xfs_fsr every so often.

Good luck,

Mike.