3ware 9650SE-4PML RAID-1 Question: One reader process starves I/O to the rest of the system?

3ware 9650SE-4PML RAID-1 Question: One reader process starves I/O to the rest of the system?

[Justin Piszcz]

Hello,

I tried CFQ/deadline schedulers, when I run the following:

# /usr/bin/time dd if=/dev/sdb2 of=/raid6/sdb2.img

Behind which are 2 x Raptor 150GB disks, the system is pretty much unusable
when I control-c the dd, it comes back to life.

The RAID-6 is a 15-disk RAID-6 which does 600-700MB/s writes.

All cards are on PCI-e and have adequate bandwidth on a P55 board (DP55KG).

Are there any recommended tuning options for a RAID-1 configuration, such that
a heavy I/O read or write operation does not pause the system?

During the dd..

Drive Performance Monitor Configuration for /c1 ...
Performance Monitor: ON
Version: 1
Max commands for averaging: 100
Max latency commands to save: 10
Requested data: Instantaneous Drive Statistics

                                Queue           Xfer         Resp
Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)
------------------------------------------------------------------------
p0     OK               u0     25      762     26.219       19 
p1     OK               u0     23      780     25.953       24 
p2     OK               u1     1       0       0.000        98 
p3     NOT-PRESENT      -      -       -       -            -

Justin.

Re: 3ware 9650SE-4PML RAID-1 Question: One reader process starves I/O to the rest of the system?

[adam radford]

Justin,

Can you try turning down the queue depth of the disk that is getting
written to to see if that resolves your responsiveness issue?

I.E.:     echo "16" > /sys/block/sdX/device/queue_depth

(replace X with disk letter of device from /raid6 mountpoint).

FYI:   It's probably better to send questions like this to linux-scsi rather
         than linux-kernel.

-Adam

On Fri, Oct 16, 2009 at 2:46 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> Hello,
>
> I tried CFQ/deadline schedulers, when I run the following:
>
> # /usr/bin/time dd if=/dev/sdb2 of=/raid6/sdb2.img
>
> Behind which are 2 x Raptor 150GB disks, the system is pretty much unusable
> when I control-c the dd, it comes back to life.
>
> The RAID-6 is a 15-disk RAID-6 which does 600-700MB/s writes.
>
> All cards are on PCI-e and have adequate bandwidth on a P55 board (DP55KG).
>
> Are there any recommended tuning options for a RAID-1 configuration, such
> that
> a heavy I/O read or write operation does not pause the system?
>
> During the dd..
>
> Drive Performance Monitor Configuration for /c1 ...
> Performance Monitor: ON
> Version: 1
> Max commands for averaging: 100
> Max latency commands to save: 10
> Requested data: Instantaneous Drive Statistics
>
>                               Queue           Xfer         Resp
> Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)
> ------------------------------------------------------------------------
> p0     OK               u0     25      762     26.219       19 p1     OK
>           u0     23      780     25.953       24 p2     OK               u1
>     1       0       0.000        98 p3     NOT-PRESENT      -      -       -
>       -            -
>
> Justin.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 3ware 9650SE-4PML RAID-1 Question: One reader process starves I/O to the rest of the system?

[Justin Piszcz]

On Fri, 16 Oct 2009, adam radford wrote:

> Justin,
>
> Can you try turning down the queue depth of the disk that is getting
> written to to see if that resolves your responsiveness issue?
>
> I.E.:     echo "16" > /sys/block/sdX/device/queue_depth
>
> (replace X with disk letter of device from /raid6 mountpoint).
>
> FYI:   It's probably better to send questions like this to linux-scsi rather
>         than linux-kernel.
>
> -Adam
>

Hi,

Will do in the future for related queries, I set it to 16, its almost usable,
I set it to 8-- both are a bit slow during a dd read but at least it does not 
starve processes anymore to the point of failure, e.g.:

Oct 16 18:17:18 p34 amavis[12685]: (12685-10) (!)SA TIMED OUT, backtrace: at /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 1855\n\teval {...} called at /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 1855\n\tMail::SpamAssassin::BayesStore::DBM::tok_unpack('Mail::SpamAssassin::BayesStore::DBM=HASH(0x4bf2400)', '\\x{0}\\x{97}\\x{0}\\x{0}\\x{0}\\x{e2}\\x{3}\\x{0}\\x{0}k\\x{af}\\x{d8}J') called at /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 865\n\tMail::SpamAssassin::BayesStore::DBM::tok_get_all('Mail::SpamAssassin::BayesStore::DBM=HASH(0x4bf2400)', '\\x{e5}a\\x{96}\\x{ce} ', '\\x{b3}]\\x{5}\\x{13}\\x{dd}', 'd\\x{b7}\\x{88}\\x{f2}%', '\\x{da}\\x{1}\\x{d4}/T', '\\x{fe}\\x{18}\\x{c6}\\x{d0}_', 'J\\x{9}\\x{9b}\\x{9a} ', 'l\\x{ce}p\\x{92}Y', '\\x{b8}\\x{8e}\\x{cc}z\\x{b3}', ...) called at /usr/share/perl5/Mail/SpamAssassin/Bayes.pm line 1185\n\tMail::SpamAssassin::Bayes::scan('Mail::SpamAssassin::Bayes=HASH(0x214b4e0)', 'Mail::SpamAssassin::PerMsgS
 t[...]

Thank you!

Justin.