Re: 2.5.2-pre performance degradation on an old 486

Re: 2.5.2-pre performance degradation on an old 486

[Mikael Pettersson]

On Mon, 7 Jan 2002 08:43:04 -0800 (PST), Linus Torvalds wrote:
>Hey, that would do it. It looks like the idle task ends up being a
>_normal_ process (just nice'd down), so it will get real CPU time instead
>of only getting scheduled when nothing else is runnable.
>
>Davide, I think the bounce-buffer is a red herring, it's simply that we're
>wasting time in idle..

This does seem to be the case. As a quick hack I added

	if (p == &init_task) return -50;

at the start of kernel/sched.c:goodness() [to approximate the old
scheduler's behaviour], and this immediately restored performance
on my 486 to the old scheduler's levels.

/Mikael

Re: 2.5.2-pre performance degradation on an old 486

[Davide Libenzi]

On Mon, 7 Jan 2002, Mikael Pettersson wrote:

> On Mon, 7 Jan 2002 08:43:04 -0800 (PST), Linus Torvalds wrote:
> >Hey, that would do it. It looks like the idle task ends up being a
> >_normal_ process (just nice'd down), so it will get real CPU time instead
> >of only getting scheduled when nothing else is runnable.
> >
> >Davide, I think the bounce-buffer is a red herring, it's simply that we're
> >wasting time in idle..
>
> This does seem to be the case. As a quick hack I added
>
> 	if (p == &init_task) return -50;
>
> at the start of kernel/sched.c:goodness() [to approximate the old
> scheduler's behaviour], and this immediately restored performance
> on my 486 to the old scheduler's levels.

I'll post a patch to Linus in 20 minutes otherwise Linus simply

sched.c::init_idle()

current->dyn_prio = -100;




- Davide

Re: 2.5.2-pre performance degradation on an old 486

[Davide Libenzi]

On Mon, 7 Jan 2002, Matthias Hanisch wrote:

> On Mon, 7 Jan 2002, Davide Libenzi wrote:
>
> > In sched.c::init_idle() :
> >
> > current->dyn_prio = -100;
> >
> > Let me know.
>
> Aehm. I already added the same line at the beginning of cpu_idle() in
> arch/i386/process.c, which brought back the old performance. Your patch
> should be analogous, but cleaner.
>
> So: Bingo!!!!
>
> I just wonder, why only two people with slow machines saw this behavior...
>
> Now 2.5.2 can come :)

The problem is that slow machines shows different dyn_prio distribution.
What happened was that if a process with dyn_prio == was wake up while the
idle was running, preemption_goodness() failed to kick out the idle ( with
dyn_prio == 0 ) because of the strict > 0




- Davide

Re: 2.5.2-pre performance degradation on an old 486

[Matthias Hanisch]

On Mon, 7 Jan 2002, Davide Libenzi wrote:

> In sched.c::init_idle() :
> 
> current->dyn_prio = -100;
> 
> Let me know.

Aehm. I already added the same line at the beginning of cpu_idle() in
arch/i386/process.c, which brought back the old performance. Your patch
should be analogous, but cleaner.

So: Bingo!!!!

I just wonder, why only two people with slow machines saw this behavior...

Now 2.5.2 can come :)

Regards,
	Matze

Re: 2.5.2-pre performance degradation on an old 486

[Davide Libenzi]

On Mon, 7 Jan 2002, Matthias Hanisch wrote:

> On Sat, 5 Jan 2002, Davide Libenzi wrote:
>
> > There should be some part of the kernel that assume a certain scheduler
> > behavior. There was a guy that reported a bad  hdparm  performance and i
> > tried it. By running  hdparm -t  my system has a context switch of 20-30
> > and an irq load of about 100-110.
>
> This guy was me, IMHO (just with my office email address :).
>
>
> > The scheduler itself, even if you code it in visual basic, cannot make
> > this with such loads.
> > Did you try to profile the kernel ?
>
> To answer your question, I wanted to profile 2.5.2-pre8 against
> 2.5.2-pre8-old-scheduler. _Fortunately_ I made some mistake and forgot to
> back out the following chunk of memory.
>
> --- v2.5.1/linux/arch/i386/kernel/process.c     Thu Oct  4 18:42:54 2001
> +++ linux/arch/i386/kernel/process.c    Thu Dec 27 08:21:28 2001
> @@ -125,7 +125,6 @@
>         /* endless idle loop with no priority at all */
>         init_idle();
>         current->nice = 20;
> -       current->counter = -100;

In sched.c::init_idle() :

current->dyn_prio = -100;

Let me know.




- Davide

Re: 2.5.2-pre performance degradation on an old 486

[Linus Torvalds]

On Mon, 7 Jan 2002, Matthias Hanisch wrote:
>
> To answer your question, I wanted to profile 2.5.2-pre8 against
> 2.5.2-pre8-old-scheduler. _Fortunately_ I made some mistake and forgot to
> back out the following chunk of memory.
>
> --- v2.5.1/linux/arch/i386/kernel/process.c     Thu Oct  4 18:42:54 2001
> +++ linux/arch/i386/kernel/process.c    Thu Dec 27 08:21:28 2001
> @@ -125,7 +125,6 @@
>         /* endless idle loop with no priority at all */
>         init_idle();
>         current->nice = 20;
> -       current->counter = -100;
>
>         while (1) {
>                 void (*idle)(void) = pm_idle;

Hey, that would do it. It looks like the idle task ends up being a
_normal_ process (just nice'd down), so it will get real CPU time instead
of only getting scheduled when nothing else is runnable.

Davide, I think the bounce-buffer is a red herring, it's simply that we're
wasting time in idle..

		Linus

Re: 2.5.2-pre performance degradation on an old 486

[Matthias Hanisch]

On Sat, 5 Jan 2002, Davide Libenzi wrote:

> There should be some part of the kernel that assume a certain scheduler
> behavior. There was a guy that reported a bad  hdparm  performance and i
> tried it. By running  hdparm -t  my system has a context switch of 20-30
> and an irq load of about 100-110.

This guy was me, IMHO (just with my office email address :).


> The scheduler itself, even if you code it in visual basic, cannot make
> this with such loads.
> Did you try to profile the kernel ?

To answer your question, I wanted to profile 2.5.2-pre8 against
2.5.2-pre8-old-scheduler. _Fortunately_ I made some mistake and forgot to
back out the following chunk of memory.

--- v2.5.1/linux/arch/i386/kernel/process.c     Thu Oct  4 18:42:54 2001
+++ linux/arch/i386/kernel/process.c    Thu Dec 27 08:21:28 2001
@@ -125,7 +125,6 @@
        /* endless idle loop with no priority at all */
        init_idle();
        current->nice = 20;
-       current->counter = -100;
 
        while (1) {
                void (*idle)(void) = pm_idle;

So it seems, that removing this line from kernel sources with the old
scheduler causes this unresponsive behavior. This chunk looks also a
little bit strange. In most (all?) the other chunks "counter" gots
replaced with "dyn_prio", not completely removed.

I'll verify this tonight (have to earn some money at first :). I'll do
also some profiling.

Mikael, if you have time, maybe you can try to apply only this chunk of
patch (or only remove the line) to a clean 2.4.18-pre1 and report the
behavior.


Davide, regarding your question in the other mail:

> Can you try some changes that i'll tell you ?

Please forward to me also. Sometimes it takes a little bit longer, because
there is also life without LKML, but I want to get this understood and
fixed, so I'll try to help you as much as I can.


Regards,
	Matze

Re: 2.5.2-pre performance degradation on an old 486

[Andre Hedrick]

On Sun, 6 Jan 2002, Jens Axboe wrote:

> On Sat, Jan 05 2002, Davide Libenzi wrote:
> > > > (*) 100MHz 486DX4, 28MB ram, no L2 cache, two old and slow IDE disks,
> > > > small custom no-nonsense RedHat 7.2, kernels compiled with gcc 2.95.3.
> > >
> > > Is this ISA (maybe it has something to do with ISA bouncing)? Mine is:
> > >
> > > 486 DX/2 ISA, Adaptec 1542, two slow scsi disks and a self-made
> > > slackware-based system.
> > >
> > > Can you also backout the scheduler changes to verify this? I have a
> > > backout patch for 2.5.2-pre6, if you don't want to do this for yourself.
> > 
> > There should be some part of the kernel that assume a certain scheduler
> > behavior. There was a guy that reported a bad  hdparm  performance and i
> > tried it. By running  hdparm -t  my system has a context switch of 20-30
> > and an irq load of about 100-110.
> > The scheduler itself, even if you code it in visual basic, cannot make
> > this with such loads.
> > Did you try to profile the kernel ?
> 
> Davide,
> 
> If this is caused by ISA bounce problems, then you should be able to
> reproduce by doing something ala
> 
> [ drivers/ide/ide-dma.c ]
> 
> ide_toggle_bounce()
> {
> 	...
> 
> +	addr = BLK_BOUNCE_ISA;
> 	blk_queue_bounce_limit(&drive->queue, addr);
> }

Jens, how about getting a hardware list because I have prime2/3 ISA DMA
cards.  Just not ready to test in 2.5.

Regards,


Andre Hedrick
CEO/President, LAD Storage Consulting Group
Linux ATA Development
Linux Disk Certification Project

Re: 2.5.2-pre performance degradation on an old 486

[Jens Axboe]

On Sat, Jan 05 2002, Davide Libenzi wrote:
> > > (*) 100MHz 486DX4, 28MB ram, no L2 cache, two old and slow IDE disks,
> > > small custom no-nonsense RedHat 7.2, kernels compiled with gcc 2.95.3.
> >
> > Is this ISA (maybe it has something to do with ISA bouncing)? Mine is:
> >
> > 486 DX/2 ISA, Adaptec 1542, two slow scsi disks and a self-made
> > slackware-based system.
> >
> > Can you also backout the scheduler changes to verify this? I have a
> > backout patch for 2.5.2-pre6, if you don't want to do this for yourself.
> 
> There should be some part of the kernel that assume a certain scheduler
> behavior. There was a guy that reported a bad  hdparm  performance and i
> tried it. By running  hdparm -t  my system has a context switch of 20-30
> and an irq load of about 100-110.
> The scheduler itself, even if you code it in visual basic, cannot make
> this with such loads.
> Did you try to profile the kernel ?

Davide,

If this is caused by ISA bounce problems, then you should be able to
reproduce by doing something ala

[ drivers/ide/ide-dma.c ]

ide_toggle_bounce()
{
	...

+	addr = BLK_BOUNCE_ISA;
	blk_queue_bounce_limit(&drive->queue, addr);
}

pseudo-diff, just add the addr = line. Now compare performance with and
without your scheduler changes.

-- 
Jens Axboe

Re: 2.5.2-pre performance degradation on an old 486

[Davide Libenzi]

On Sat, 5 Jan 2002, Matthias Hanisch wrote:

> On Sat, 5 Jan 2002, Mikael Pettersson wrote:
>
> > When running 2.5.2-pre7 on my old for-testing-only 486(*),
> > file-system accesses seem to come in distinct bursts preceded
> > by lengthy pauses. Overall performance is down quite significantly
> > compared to 2.4.18pre1 and 2.2.20pre2. To measure it I ran two
> > simple tests:
> >
> > Test 1: time to boot the kernel, from hitting enter at the LILO
> > prompt to getting a login prompt
> > Test 2: time to "rm -rf" a clean linux-2.4.17 source tree, using
> > the newly booted kernel (no other access to the tree before that,
> > so it wasn't cached in any way, and the machine was otherwise idle)
> >
> > 		Test 1		Test 2
> > 2.2.21pre2:	71 sec		 75 sec
> > 2.4.18pre1:	64 sec		 72 sec
> > 2.5.2-pre7:	97 sec		251 sec
> >
> > I haven't noticed any slowdowns on my other boxes, so I didn't
> > do any measurements on them. On the 486 it's very very obvious.
>
> This is exactly, what I see with my old 486 box. It started with
> 2.5.2-pre3, which contained two major items:
>
> - bio changes from Jens
> - scheduler changes from Davide
>
> Surprisingly, backing out the bio changes didn't help. Backing out the
> scheduler changes from Davide did!!
>
> Maybe the problem lies somewhere in between, because it is often I/O
> related, e.g. first call of ldconfig is horrible slow, as is e2fsck.
>
> But I also see system hiccups from time to time, where console switching
> does not work for 1 second on an idle box.
>
>
> > (*) 100MHz 486DX4, 28MB ram, no L2 cache, two old and slow IDE disks,
> > small custom no-nonsense RedHat 7.2, kernels compiled with gcc 2.95.3.
>
> Is this ISA (maybe it has something to do with ISA bouncing)? Mine is:
>
> 486 DX/2 ISA, Adaptec 1542, two slow scsi disks and a self-made
> slackware-based system.
>
> Can you also backout the scheduler changes to verify this? I have a
> backout patch for 2.5.2-pre6, if you don't want to do this for yourself.

There should be some part of the kernel that assume a certain scheduler
behavior. There was a guy that reported a bad  hdparm  performance and i
tried it. By running  hdparm -t  my system has a context switch of 20-30
and an irq load of about 100-110.
The scheduler itself, even if you code it in visual basic, cannot make
this with such loads.
Did you try to profile the kernel ?




- Davide

Re: 2.5.2-pre performance degradation on an old 486

[Matthias Hanisch]

On Sat, 5 Jan 2002, Mikael Pettersson wrote:

> When running 2.5.2-pre7 on my old for-testing-only 486(*),
> file-system accesses seem to come in distinct bursts preceded
> by lengthy pauses. Overall performance is down quite significantly
> compared to 2.4.18pre1 and 2.2.20pre2. To measure it I ran two
> simple tests:
> 
> Test 1: time to boot the kernel, from hitting enter at the LILO
> prompt to getting a login prompt
> Test 2: time to "rm -rf" a clean linux-2.4.17 source tree, using
> the newly booted kernel (no other access to the tree before that,
> so it wasn't cached in any way, and the machine was otherwise idle)
> 
> 		Test 1		Test 2
> 2.2.21pre2:	71 sec		 75 sec
> 2.4.18pre1:	64 sec		 72 sec
> 2.5.2-pre7:	97 sec		251 sec
> 
> I haven't noticed any slowdowns on my other boxes, so I didn't
> do any measurements on them. On the 486 it's very very obvious.

This is exactly, what I see with my old 486 box. It started with
2.5.2-pre3, which contained two major items:

- bio changes from Jens
- scheduler changes from Davide

Surprisingly, backing out the bio changes didn't help. Backing out the
scheduler changes from Davide did!!

Maybe the problem lies somewhere in between, because it is often I/O
related, e.g. first call of ldconfig is horrible slow, as is e2fsck.

But I also see system hiccups from time to time, where console switching
does not work for 1 second on an idle box.


> (*) 100MHz 486DX4, 28MB ram, no L2 cache, two old and slow IDE disks,
> small custom no-nonsense RedHat 7.2, kernels compiled with gcc 2.95.3.

Is this ISA (maybe it has something to do with ISA bouncing)? Mine is:

486 DX/2 ISA, Adaptec 1542, two slow scsi disks and a self-made
slackware-based system.

Can you also backout the scheduler changes to verify this? I have a
backout patch for 2.5.2-pre6, if you don't want to do this for yourself.

Regards,
	Matze (trying 2.5.2-pre8 now)

2.5.2-pre performance degradation on an old 486

[Mikael Pettersson]

When running 2.5.2-pre7 on my old for-testing-only 486(*),
file-system accesses seem to come in distinct bursts preceded
by lengthy pauses. Overall performance is down quite significantly
compared to 2.4.18pre1 and 2.2.20pre2. To measure it I ran two
simple tests:

Test 1: time to boot the kernel, from hitting enter at the LILO
prompt to getting a login prompt
Test 2: time to "rm -rf" a clean linux-2.4.17 source tree, using
the newly booted kernel (no other access to the tree before that,
so it wasn't cached in any way, and the machine was otherwise idle)

		Test 1		Test 2
2.2.21pre2:	71 sec		 75 sec
2.4.18pre1:	64 sec		 72 sec
2.5.2-pre7:	97 sec		251 sec

I haven't noticed any slowdowns on my other boxes, so I didn't
do any measurements on them. On the 486 it's very very obvious.

/Mikael

(*) 100MHz 486DX4, 28MB ram, no L2 cache, two old and slow IDE disks,
small custom no-nonsense RedHat 7.2, kernels compiled with gcc 2.95.3.