Re: Slow disks.

[Jeff Moyer <jmoyer@redhat.com>]

Rogier Wolff <R.E.Wolff@BitWizard.nl> writes:

> Hi,
>
> A friend of mine has a server in a datacenter somewhere. His machine
> is not working properly: most of his disks take 10-100 times longer
> to process each IO request than normal. 
>
> iostat -kx 10 output: 
> Device: rrqm/s wrqm/s r/s  w/s  rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sdd     0.30   0.00   0.40 1.20 2.80  1.10  4.88     0.43  271.50 271.44  43.43

You are doing ~4KB I/Os and driving a queue depth of <1.  If you are
seeking all over the disk and mixing reads and writes, you may very well
trigger bad behaviour like this for SATA disks.

To further diagnose the issue, I'd recommend running blktrace on one of
the devices.  If you report results here, could you also include more
information about the hardware, the storage layout (including any dm/md
drivers and file systems involved) and the workload?  Also, please let
us know which I/O scheduler you're using.

> These syptoms started when the system was running 2.6.33, but are
> still present now the system has been upgraded to 2.6.36.

Are you sure it was a kernel change that caused the issue?  In other
words, can you run with the 2.6.32 and confirm that the issue is not
present?

Thanks,
Jeff