Re: SATA Harddisk speed drop of 100 MB/s

Re: SATA Harddisk speed drop of 100 MB/s

[Al Boldi]

Carlo Wood wrote:
> Just one kernel version? The problem here is in every
> kernel revision after 551c012d7eea3dc5ec063c7ff9c718d39e77634f
>
> 2.6.20-rc2,rc3,rc4,rc5,rc6,rc7 ... 2.6.20 ... 2.6.21 ... 2.6.22-rc5
>
> noop:
>  Timing buffered disk reads:  254 MB in  3.00 seconds =  84.66 MB/sec
>
> anticipatory:
>  Timing buffered disk reads:  252 MB in  3.02 seconds =  83.43 MB/sec
>
> deadline:
>  Timing buffered disk reads:  258 MB in  3.02 seconds =  85.41 MB/sec
>
> cfq:
>  Timing buffered disk reads:  194 MB in  3.03 seconds =  64.06 MB/sec
>
> The normal value is cfq. So, all other schedulars are somewhat faster,
> but still far from the correct 165 MB/s.

You may want to play with /sys/block/*/queue/max_sectors_kb and 
read_ahead_kb.  On my system, setting them to 192 gives best performance, 
but cfq is always slower.


Thanks!

--
Al

SATA Harddisk speed drop of 100 MB/s

[Carlo Wood]

hdparm -tT gives over 160 MB/s for my tripple 10k rpm Raptor RAID5
on 2.6.18, as expected.

In 2.6.22-rc5 this dropped to only 60 MB/s !

Is this a known issue?

I also measured it with bonnie; I get 108 MB/s reading speed on
2.6.18 and only 68 MB/s on 2.6.22-rc5.

-- 
Carlo Wood <carlo@alinoe.com>

Re: SATA Harddisk speed drop of 100 MB/s

[Jeff Garzik]

Carlo Wood wrote:
> hdparm -tT gives over 160 MB/s for my tripple 10k rpm Raptor RAID5
> on 2.6.18, as expected.
> 
> In 2.6.22-rc5 this dropped to only 60 MB/s !
> 
> Is this a known issue?
> 
> I also measured it with bonnie; I get 108 MB/s reading speed on
> 2.6.18 and only 68 MB/s on 2.6.22-rc5.

More info needed...  kernel config, before-and-after dmesg output, lspci 
output.

	Jeff

Re: SATA Harddisk speed drop of 100 MB/s

[Arjan van de Ven]

On Wed, 2007-06-20 at 19:06 -0400, Jeff Garzik wrote:
> Carlo Wood wrote:
> > hdparm -tT gives over 160 MB/s for my tripple 10k rpm Raptor RAID5
> > on 2.6.18, as expected.
> > 
> > In 2.6.22-rc5 this dropped to only 60 MB/s !
> > 
> > Is this a known issue?
> > 
> > I also measured it with bonnie; I get 108 MB/s reading speed on
> > 2.6.18 and only 68 MB/s on 2.6.22-rc5.
> 
> More info needed...  kernel config, before-and-after dmesg output, lspci 
> output.




also do a 

echo 60 > /proc/sys/vm/dirty_ratio

first to see if that helps

Re: SATA Harddisk speed drop of 100 MB/s

[Carlo Wood]

On Wed, Jun 20, 2007 at 08:36:52PM -0700, Arjan van de Ven wrote:
> also do a 
> 
> echo 60 > /proc/sys/vm/dirty_ratio
> 
> first to see if that helps

That makes no difference.

It took a while before I replied - because I first want to do a bisect
to find a version closer to the problem.

So far I found out that it's RAID only.

I have four disks:

Three times a western digital Raptor 10k rpm:

ATA device, with non-removable media
        Model Number:       WDC WD740ADFD-00NLR1
        Firmware Revision:  20.07P20

And one Seagate Barracuda 7200 rpm:

ATA device, with non-removable media
        Model Number:       ST3320620AS
        Firmware Revision:  3.AAE

The Raptors are in RAID5.

hdparm -tT gives me always > 4 GB/s cached reads, for example:

 Timing cached reads:   8166 MB in  2.00 seconds = 4086.82 MB/sec

and the buffered disk reads do not significantly vary when
reading the devices directly (/dev/sda, /dev/sdb, /dev/sdc for
the Raptors and /dev/sdd for the Barracuda):

/dev/sda etc: Around 67 MB/sec
/dev/sdd    : Around 75 MB/sec   (so much for 10k rpm ...)

These numbers are weird in itself (the 67 should have been 75,
and the 75 should have been 54) - but at least they are constant.

What DOES *signficantly* change as function of kernel revision
is the buffered disk read with hdparm -tT /dev/md2

The normal speed I get is > 160, often 165 MB/sec. But with
kernel revisions over a certain version that drops to around
67 MB/sec - and at one point I even observed a drop in speed
every time I repeated the measurement until the speed was
around 30 MB/s !

At the moment I am (still) doing the bisect; I had some problems
compiling a pristine 2.6.18 and some problems with the config
(that differs significantly between 2.6.18 and 2.6.22) as well
as the problem that the kernel simply didn't compile for a large
range of revisions needed for the bisect (I needed to export
current_is_keventd with this patch:

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index c525731..012e11b 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -654,6 +654,7 @@ int current_is_keventd(void)
        return ret;

 }
+EXPORT_SYMBOL_GPL(current_is_keventd);

 /* Take the work from this (downed) CPU. */
 static void take_over_work(struct workqueue_struct *wq, unsigned int cpu)

)

I'll report back when I'm done with the bisect :)

-- 
Carlo Wood <carlo@alinoe.com>

Re: SATA Harddisk speed drop of 100 MB/s

[Henrique de Moraes Holschuh]

On Fri, 22 Jun 2007, Carlo Wood wrote:
> So far I found out that it's RAID only.

If you change the IO schedulers, does it help?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA Harddisk speed drop of 100 MB/s

[Carlo Wood]

On Fri, Jun 22, 2007 at 06:17:46PM -0300, Henrique de Moraes Holschuh wrote:
> On Fri, 22 Jun 2007, Carlo Wood wrote:
> > So far I found out that it's RAID only.
> 
> If you change the IO schedulers, does it help?

How do I do that?

-- 
Carlo Wood <carlo@alinoe.com>

Re: SATA Harddisk speed drop of 100 MB/s

[Henrique de Moraes Holschuh]

Hi Carlo!

On Fri, 22 Jun 2007, Carlo Wood wrote:

> On Fri, Jun 22, 2007 at 06:17:46PM -0300, Henrique de Moraes Holschuh wrote:
> > On Fri, 22 Jun 2007, Carlo Wood wrote:
> > > So far I found out that it's RAID only.
> > 
> > If you change the IO schedulers, does it help?
> 
> How do I do that?

for i in /sys/block/sd*/queue/scheduler ; do echo -n "scheduler for $i was:" ; cat $i ; echo anticipatory > $i ; done

Should show you the scheduler for all libata/scsi discs, and switch to
anticipatory.  It probably works for hd* as well, but I don't have any
around to test.

I had ridiculous md raid performance (on resync) using cfq at least once.
Unfortunately I never bothered trying to track it down, and I didn't check
the throughput after the resync was done either. Also, I don't recall the
kernel version, but it was either 2.6.18 or 2.6.20.

Since it is a very simple test...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA Harddisk speed drop of 100 MB/s

[Carlo Wood]

On Fri, Jun 22, 2007 at 10:31:32PM -0300, Henrique de Moraes Holschuh wrote:
> for i in /sys/block/sd*/queue/scheduler ; do echo -n "scheduler for $i was:" ; cat $i ; echo anticipatory > $i ; done
> 
> Should show you the scheduler for all libata/scsi discs, and switch to
> anticipatory.  It probably works for hd* as well, but I don't have any
> around to test.
> 
> I had ridiculous md raid performance (on resync) using cfq at least once.
> Unfortunately I never bothered trying to track it down, and I didn't check
> the throughput after the resync was done either. Also, I don't recall the
> kernel version, but it was either 2.6.18 or 2.6.20.

Just one kernel version? The problem here is in every
kernel revision after 551c012d7eea3dc5ec063c7ff9c718d39e77634f

2.6.20-rc2,rc3,rc4,rc5,rc6,rc7 ... 2.6.20 ... 2.6.21 ... 2.6.22-rc5

noop:
 Timing buffered disk reads:  254 MB in  3.00 seconds =  84.66 MB/sec

anticipatory:
 Timing buffered disk reads:  252 MB in  3.02 seconds =  83.43 MB/sec

deadline:
 Timing buffered disk reads:  258 MB in  3.02 seconds =  85.41 MB/sec

cfq:
 Timing buffered disk reads:  194 MB in  3.03 seconds =  64.06 MB/sec

The normal value is cfq. So, all other schedulars are somewhat faster,
but still far from the correct 165 MB/s.

-- 
Carlo Wood <carlo@alinoe.com>

Re: SATA Harddisk speed drop of 100 MB/s

[Andrew Morton]

On Sat, 23 Jun 2007 04:59:06 +0200 Carlo Wood <carlo@alinoe.com> wrote:

> On Fri, Jun 22, 2007 at 10:31:32PM -0300, Henrique de Moraes Holschuh wrote:
> > for i in /sys/block/sd*/queue/scheduler ; do echo -n "scheduler for $i was:" ; cat $i ; echo anticipatory > $i ; done
> > 
> > Should show you the scheduler for all libata/scsi discs, and switch to
> > anticipatory.  It probably works for hd* as well, but I don't have any
> > around to test.
> > 
> > I had ridiculous md raid performance (on resync) using cfq at least once.
> > Unfortunately I never bothered trying to track it down, and I didn't check
> > the throughput after the resync was done either. Also, I don't recall the
> > kernel version, but it was either 2.6.18 or 2.6.20.
> 
> Just one kernel version? The problem here is in every
> kernel revision after 551c012d7eea3dc5ec063c7ff9c718d39e77634f
> 
> 2.6.20-rc2,rc3,rc4,rc5,rc6,rc7 ... 2.6.20 ... 2.6.21 ... 2.6.22-rc5
> 
> noop:
>  Timing buffered disk reads:  254 MB in  3.00 seconds =  84.66 MB/sec
> 
> anticipatory:
>  Timing buffered disk reads:  252 MB in  3.02 seconds =  83.43 MB/sec
> 
> deadline:
>  Timing buffered disk reads:  258 MB in  3.02 seconds =  85.41 MB/sec
> 
> cfq:
>  Timing buffered disk reads:  194 MB in  3.03 seconds =  64.06 MB/sec
> 
> The normal value is cfq. So, all other schedulars are somewhat faster,
> but still far from the correct 165 MB/s.
> 

Please see http://bugzilla.kernel.org/show_bug.cgi?id=8636

In that report a pretty large slowdown with CFQ has been identified.
Jens has plunked a patch in there (Comment #50) and then ran away on
vacation.

If someone can test that patch and demonstrate its goodness, I'll
queue it up, thanks.

Re: SATA Harddisk speed drop of 100 MB/s

[Jeff Garzik]

Andrew Morton wrote:
> Please see http://bugzilla.kernel.org/show_bug.cgi?id=8636
> 
> In that report a pretty large slowdown with CFQ has been identified.
> Jens has plunked a patch in there (Comment #50) and then ran away on
> vacation.
> 
> If someone can test that patch and demonstrate its goodness, I'll
> queue it up, thanks.

This has unfortunately split up into two threads.  I identified the root 
cause of the slowdown as NCQ in the thread "Re: SATA RAID5 speed drop of 
100 MB/s" on LKML.

	Jeff

Re: SATA Harddisk speed drop of 100 MB/s

[Lennart Sorensen]

On Sat, Jun 23, 2007 at 04:59:06AM +0200, Carlo Wood wrote:
> Just one kernel version? The problem here is in every
> kernel revision after 551c012d7eea3dc5ec063c7ff9c718d39e77634f
> 
> 2.6.20-rc2,rc3,rc4,rc5,rc6,rc7 ... 2.6.20 ... 2.6.21 ... 2.6.22-rc5
> 
> noop:
>  Timing buffered disk reads:  254 MB in  3.00 seconds =  84.66 MB/sec
> 
> anticipatory:
>  Timing buffered disk reads:  252 MB in  3.02 seconds =  83.43 MB/sec
> 
> deadline:
>  Timing buffered disk reads:  258 MB in  3.02 seconds =  85.41 MB/sec
> 
> cfq:
>  Timing buffered disk reads:  194 MB in  3.03 seconds =  64.06 MB/sec
> 
> The normal value is cfq. So, all other schedulars are somewhat faster,
> but still far from the correct 165 MB/s.

How do you know 165MB/s is correct?

As for 10k rpm being faster, well remember rotation speed _and_ areal
density is what gives transfer rate.  a 750GB 7200rpm is likely able to
have a higher throughput than a 73GB 10k rpm drive since more bits pass
by the head in any given unit of time on the larger drive.  The 7200 rpm
has a slower average access time though.

--
Len Sorensen

Re: SATA Harddisk speed drop of 100 MB/s

[Carlo Wood]

On Mon, Jun 25, 2007 at 11:18:41AM -0400, Lennart Sorensen wrote:
> How do you know 165MB/s is correct?

It is the maximum value I measure. It's also the value that I
measure consistently with NCQ off, like with 2.6.18. Finally,
even before I bought this PC, I did some research and the RAID
with Raptors was recommended by someone with a lot of experience
with them. He told me that I could count on (N - 1) * 75 MB/s
(on average) for an array with N disks. I had expected 150 MB/s,
which is about what I get. The 165 MB/s is on the outside
of the disk (low partition).

-- 
Carlo Wood <carlo@alinoe.com>