md mirror / ext3 / dual core performance strange phenomenon?

md mirror / ext3 / dual core performance strange phenomenon?

[Erik Slagter]

Hi,

This may be a very stupid question or equally misplaced but I really
don't know where to ask it elsewhere.

I have this configuration (imho significant parts): dual core P915, 2x
sata harddisk configured as (root) md mirror on intel ICH7R (ahci mode),
1G ram.

The harddisks can sustain both around 55 Mb/s, according to hdparm -tT
(okay, I now that's only straight reads). Also they can deliver this
amount of data independent of each other, if both hdparms are run in
parallel, the same performance is reported. I also did similar tests
using ddrescue which shows the same.

Now, what puzzles me, is that compiling the kernel (2.6.17.7) using
either "make -j1 ..." or "make -j2 ..." or "make -j3 ..." makes the
building take about 6.5 minutes, which is really dissatisfying for this
cpu/harddisks combination. Also, top shows that most of the time both
core are between 10-40% idle.

BUT... starting from -j4 (and upwards) the compile time suddenly goes to
3.5 minutes!

I am really blown away here. It looks like disk access is the bottleneck
here, but I can't imagine my disks being so slow (at seeking, I guess)
it should matter that much.

So, I tried changing the readahead value of all relevant devices and...
there is no difference at all!

I changed journalling parameters to data writeback and commit time =
30s. Still no difference.

I kicked one of the two disks out of the mirror. Still no joy.

Then, I tried all i/o schedulers on all relevant devices (except md0 of
course) and... still no difference at all!

Is there anybody who can shed some light on this or give me a clue or
pointer?

Re: md mirror / ext3 / dual core performance strange phenomenon?

[Keith Owens]

Erik Slagter (on Mon, 14 Aug 2006 15:12:14 +0200) wrote:
>Now, what puzzles me, is that compiling the kernel (2.6.17.7) using
>either "make -j1 ..." or "make -j2 ..." or "make -j3 ..." makes the
>building take about 6.5 minutes, which is really dissatisfying for this
>cpu/harddisks combination. Also, top shows that most of the time both
>core are between 10-40% idle.
>
>BUT... starting from -j4 (and upwards) the compile time suddenly goes to
>3.5 minutes!

Nothing to do with the disks, it is a design flaw in the kernel build
system.  If you want a useful parallel make using -j<n>, set <n> to 3,
4 or 5 higher than the real number of parallel jobs that you want.  The
exact value to add depends on which kernel tree you are building.  See
http://marc.theaimsgroup.com/?l=linux-kernel&m=115553906404695&w=2

Re: md mirror / ext3 / dual core performance strange phenomenon?

[Erik Slagter]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On ma, 2006-08-14 at 23:22 +1000, Keith Owens wrote:

> >BUT... starting from -j4 (and upwards) the compile time suddenly goes to
> >3.5 minutes!
> 
> Nothing to do with the disks, it is a design flaw in the kernel build
> system.  If you want a useful parallel make using -j<n>, set <n> to 3,
> 4 or 5 higher than the real number of parallel jobs that you want.  The
> exact value to add depends on which kernel tree you are building.  See
> http://marc.theaimsgroup.com/?l=linux-kernel&m=115553906404695&w=2

Okay, so it basically means I shouldn't worry here.

Sorry for bothering, I couldn't come up with a proper search term...

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2771 bytes --]

Re: md mirror / ext3 / dual core performance strange phenomenon?

[Alistair John Strachan]

On Monday 14 August 2006 14:26, Erik Slagter wrote:
> On ma, 2006-08-14 at 23:22 +1000, Keith Owens wrote:
> > >BUT... starting from -j4 (and upwards) the compile time suddenly goes to
> > >3.5 minutes!
> >
> > Nothing to do with the disks, it is a design flaw in the kernel build
> > system.  If you want a useful parallel make using -j<n>, set <n> to 3,
> > 4 or 5 higher than the real number of parallel jobs that you want.  The
> > exact value to add depends on which kernel tree you are building.  See
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=115553906404695&w=2
>
> Okay, so it basically means I shouldn't worry here.
>
> Sorry for bothering, I couldn't come up with a proper search term...

I find that for mainline, using an odd number of jobs works better. I can get 
my X2 3800+ to full saturation with -j5.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: md mirror / ext3 / dual core performance strange phenomenon?

[Alan Cox]

Ar Llu, 2006-08-14 am 15:12 +0200, ysgrifennodd Erik Slagter:
> I am really blown away here. It looks like disk access is the bottleneck
> here, but I can't imagine my disks being so slow (at seeking, I guess)
> it should matter that much.

Keith has answered the main question but your assumptions about seeking
are also not neccessarily correct. Disk seek times as opposed to data
rates and density have not materially improved for many years now, nor
for IDE has the rotation speed and thus rotational latency to access
data.

RAID-1 mirroring generally helps performance as it speeds up the rate of
read command completion at the cost of slower writes. Writes are usually
buffered so tend not to impact performance unless you have a lot of I/O.

Alan

Re: md mirror / ext3 / dual core performance strange phenomenon?

[Erik Slagter]

On ma, 2006-08-14 at 15:01 +0100, Alan Cox wrote:
> Ar Llu, 2006-08-14 am 15:12 +0200, ysgrifennodd Erik Slagter:
> > I am really blown away here. It looks like disk access is the bottleneck
> > here, but I can't imagine my disks being so slow (at seeking, I guess)
> > it should matter that much.
> 
> Keith has answered the main question but your assumptions about seeking
> are also not neccessarily correct. Disk seek times as opposed to data
> rates and density have not materially improved for many years now, nor
> for IDE has the rotation speed and thus rotational latency to access
> data.

I was suspecting disk seeking because increasing the number of jobs
(beyond the number of cores) would help. But I really didn't take the
actual kernel building system into consideration. And I do know that ATA
disk seek performance is ehrrmm... suboptimal and absolutely not related
to interface speed or even straight read-from-disk speed. I was hoping
fiddling with the i/o scheduler and readahead knobs would make _some_
difference here.

Interesting fact btw, that if I kick one of the disks out of the mirror,
the kernel building time remains more or less the same. So it looks like
the md mirror theory doesn't have that great impact on practise.

Anyway thank for the clues.