Re: 2.6.13-rc6-rt9

Re: 2.6.13-rc6-rt9

[Karsten Wiese]

Chuck wrote:
> I'm still getting the same oops when rebooting. the same process (reboot)
> similar call trace (some addresses are slightly different but the functions
> are the same:
> disable_IO_APIC+0x5a/0x90 (8)
> machine_restart+0x5/0x9 (28)
> sys_reboot+0x147/0x156 (4)
> netdev_run_todo+0xa4/0x209 (4)
> etc.

Does this patch help?

------
diff -up arch/i386/kernel/io_apic.c.rt9 arch/i386/kernel/io_apic.c
--- arch/i386/kernel/io_apic.c.rt9      2005-08-19 12:28:42.000000000 +0200
+++ arch/i386/kernel/io_apic.c  2005-08-19 12:29:30.000000000 +0200
@@ -1758,8 +1758,8 @@ void disable_IO_APIC(void)
                 * Add it to the IO-APIC irq-routing table:
                 */
                spin_lock_irqsave(&ioapic_lock, flags);
-               io_apic_write(0, 0x11+2*pin, *(((int *)&entry)+1));
-               io_apic_write(0, 0x10+2*pin, *(((int *)&entry)+0));
+               io_apic_write(ioapic_data[0], 0x11+2*pin, *(((int *)&entry)+1));
+               io_apic_write(ioapic_data[0], 0x10+2*pin, *(((int *)&entry)+0));
                spin_unlock_irqrestore(&ioapic_lock, flags);
        }
        disconnect_bsp_APIC(pin != -1);
------

   Karsten

	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: 2.6.13-rc6-rt9

[Chuck Harding]

Sure did. At least on a normal reboot. I will try
SysRq+B and see what happens. Thanks.

On Fri, 19 Aug 2005, Karsten Wiese wrote:

> Chuck wrote:
>> I'm still getting the same oops when rebooting. the same process (reboot)
>> similar call trace (some addresses are slightly different but the functions
>> are the same:
>> disable_IO_APIC+0x5a/0x90 (8)
>> machine_restart+0x5/0x9 (28)
>> sys_reboot+0x147/0x156 (4)
>> netdev_run_todo+0xa4/0x209 (4)
>> etc.
>
> Does this patch help?
>
> ------
> diff -up arch/i386/kernel/io_apic.c.rt9 arch/i386/kernel/io_apic.c
> --- arch/i386/kernel/io_apic.c.rt9      2005-08-19 12:28:42.000000000 +0200
> +++ arch/i386/kernel/io_apic.c  2005-08-19 12:29:30.000000000 +0200
> @@ -1758,8 +1758,8 @@ void disable_IO_APIC(void)
>                 * Add it to the IO-APIC irq-routing table:
>                 */
>                spin_lock_irqsave(&ioapic_lock, flags);
> -               io_apic_write(0, 0x11+2*pin, *(((int *)&entry)+1));
> -               io_apic_write(0, 0x10+2*pin, *(((int *)&entry)+0));
> +               io_apic_write(ioapic_data[0], 0x11+2*pin, *(((int *)&entry)+1));
> +               io_apic_write(ioapic_data[0], 0x10+2*pin, *(((int *)&entry)+0));
>                spin_unlock_irqrestore(&ioapic_lock, flags);
>        }
>        disconnect_bsp_APIC(pin != -1);
> ------
>
>   Karsten
>
>
>
>
>
> ___________________________________________________________
> Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601
------------------ http://tinyurl.com/5w5ey -----------------------
-- The Lab called... Your brain is ready! --

Re: 2.6.13-rc6-rt9

[Chuck Harding]

On Fri, 19 Aug 2005, Chuck Harding wrote:

> Sure did. At least on a normal reboot. I will try
> SysRq+B and see what happens. Thanks.
>

And no oops with SysRq+B either. Thanks.

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601
------------------ http://tinyurl.com/5w5ey -----------------------
-- The world is coming to an end... save your buffers! --

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> diff -up arch/i386/kernel/io_apic.c.rt9 arch/i386/kernel/io_apic.c
> --- arch/i386/kernel/io_apic.c.rt9      2005-08-19 12:28:42.000000000 +0200
> +++ arch/i386/kernel/io_apic.c  2005-08-19 12:29:30.000000000 +0200
> @@ -1758,8 +1758,8 @@ void disable_IO_APIC(void)
>                  * Add it to the IO-APIC irq-routing table:
>                  */
>                 spin_lock_irqsave(&ioapic_lock, flags);
> -               io_apic_write(0, 0x11+2*pin, *(((int *)&entry)+1));
> -               io_apic_write(0, 0x10+2*pin, *(((int *)&entry)+0));
> +               io_apic_write(ioapic_data[0], 0x11+2*pin, *(((int *)&entry)+1));
> +               io_apic_write(ioapic_data[0], 0x10+2*pin, *(((int *)&entry)+0));
>                 spin_unlock_irqrestore(&ioapic_lock, flags);

i've hand-applied it - note that your mailer has converted all tabs to 
spaces.

	Ingo

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Thu, 2005-09-29 at 18:46 -0700, Paul E. McKenney wrote:
> > you are not the culprit :)
> 
> Woo-hoo!!!  Exonerated!!!  This time, anyway...  ;-)

My pleasure :)


> > It can not be run from hardirq context, as it takes a lot of locks
> > (especially our favorites: tasklist_lock and sighand->siglock). :(
> > 
> > Maybe another playground for rcu, but it might also be solved by some
> > other mechanism for accounting and delayed execution in the PREEMPT_RT
> > case.
> 
> Certainly check_thread_timers() and check_process_timers() are playing
> with a number of task_struct fields, so it is not immediately clear
> to me how to safely replace tasklist_lock with RCU, at least not with
> a simple and small patch.
> 
> What did you have in mind for delayed execution?

Do only the time check in hard irq context and defer the lock protected
operations to a softirq context. Have to look deeper at the details
though.

tglx

Re: 2.6.13-rc6-rt9

[Paul E. McKenney]

On Fri, Sep 30, 2005 at 03:07:29AM +0200, Thomas Gleixner wrote:
> On Thu, 2005-09-29 at 18:00 -0700, Paul E. McKenney wrote:
> > > Even with a current -rt (2.6.14-rc2-rt5) UML does not run. The issue is
> > > indeed (as jeff pointed out) that VTALRM is never send. The small test
> > > programm below illustrates this.
> > > 
> > > On a non-rt kernel it completed in 1 second.
> > > On a -rt kernel it waits at infinitum.
> > 
> > Will play with it and see what I broke...
> 
> Paul, 
> 
> you are not the culprit :)

Woo-hoo!!!  Exonerated!!!  This time, anyway...  ;-)

> The run_posix_cpu_timers(p) call is #ifdef'd out with PREEMPT_RT. 
> 
> Thats a hard to fix issue. 
> 
> It can not be run from hardirq context, as it takes a lot of locks
> (especially our favorites: tasklist_lock and sighand->siglock). :(
> 
> Maybe another playground for rcu, but it might also be solved by some
> other mechanism for accounting and delayed execution in the PREEMPT_RT
> case.

Certainly check_thread_timers() and check_process_timers() are playing
with a number of task_struct fields, so it is not immediately clear
to me how to safely replace tasklist_lock with RCU, at least not with
a simple and small patch.

What did you have in mind for delayed execution?

							Thanx, Paul

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Thu, 2005-09-29 at 18:00 -0700, Paul E. McKenney wrote:
> > Even with a current -rt (2.6.14-rc2-rt5) UML does not run. The issue is
> > indeed (as jeff pointed out) that VTALRM is never send. The small test
> > programm below illustrates this.
> > 
> > On a non-rt kernel it completed in 1 second.
> > On a -rt kernel it waits at infinitum.
> 
> Will play with it and see what I broke...

Paul, 

you are not the culprit :)

The run_posix_cpu_timers(p) call is #ifdef'd out with PREEMPT_RT. 

Thats a hard to fix issue. 

It can not be run from hardirq context, as it takes a lot of locks
(especially our favorites: tasklist_lock and sighand->siglock). :(

Maybe another playground for rcu, but it might also be solved by some
other mechanism for accounting and delayed execution in the PREEMPT_RT
case.

tglx

Re: 2.6.13-rc6-rt9

[Paul E. McKenney]

On Thu, Sep 29, 2005 at 09:54:23AM +0200, Peter Zijlstra wrote:
> On Sat, 2005-08-20 at 17:24 -0400, Jeff Dike wrote:
> > On Sat, Aug 20, 2005 at 09:27:25PM +0200, Peter Zijlstra wrote:
> > > Jeff, could you help us out here?
> > > What exactly does uml need to get out of the calibrate delay loop?
> > 
> > Interrupts, it's not too demanding :-)
> > 
> > If it's not seeing VTALRM, then it will never leave the calibration loop.
> > 
> > Try stracing it and see what it's getting.
> 
> Sorry for the late reply.
> 
> Yes, that does seem to be the problem.
> 
> Even with a current -rt (2.6.14-rc2-rt5) UML does not run. The issue is
> indeed (as jeff pointed out) that VTALRM is never send. The small test
> programm below illustrates this.
> 
> On a non-rt kernel it completed in 1 second.
> On a -rt kernel it waits at infinitum.

Will play with it and see what I broke...

							Thanx, Paul

> Kind regards,
> 
> Peter Zijlstra
> 
> ---------------------------
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <sys/time.h>
> #include <signal.h>
> 
> volatile int quit = 0;
> 
> void sig_vtalrm(int signr, siginfo_t * si, void * arg)
> {
>         if (signr == SIGVTALRM) quit = 1;
> }
> 
> int main()
> {
>         struct itimerval ival = {{0,0}, {1, 0}};
> 
>         struct sigaction sa;
>         sa.sa_sigaction = sig_vtalrm;
>         sigemptyset(&sa.sa_mask);
>         sa.sa_flags = 0;
>         sigaction(SIGVTALRM, &sa, NULL);
> 
>         setitimer(ITIMER_VIRTUAL, &ival, NULL);
> 
>         printf("wait\n");
>         while (!quit) ;
>         printf("done\n");
> }
> 
> 
> -- 
> Peter Zijlstra <a.p.zijlstra@chello.nl>
> 
> 

Re: 2.6.13-rc6-rt9

[Peter Zijlstra]

On Sat, 2005-08-20 at 17:24 -0400, Jeff Dike wrote:
> On Sat, Aug 20, 2005 at 09:27:25PM +0200, Peter Zijlstra wrote:
> > Jeff, could you help us out here?
> > What exactly does uml need to get out of the calibrate delay loop?
> 
> Interrupts, it's not too demanding :-)
> 
> If it's not seeing VTALRM, then it will never leave the calibration loop.
> 
> Try stracing it and see what it's getting.

Sorry for the late reply.

Yes, that does seem to be the problem.

Even with a current -rt (2.6.14-rc2-rt5) UML does not run. The issue is
indeed (as jeff pointed out) that VTALRM is never send. The small test
programm below illustrates this.

On a non-rt kernel it completed in 1 second.
On a -rt kernel it waits at infinitum.

Kind regards,

Peter Zijlstra

---------------------------

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/time.h>
#include <signal.h>

volatile int quit = 0;

void sig_vtalrm(int signr, siginfo_t * si, void * arg)
{
        if (signr == SIGVTALRM) quit = 1;
}

int main()
{
        struct itimerval ival = {{0,0}, {1, 0}};

        struct sigaction sa;
        sa.sa_sigaction = sig_vtalrm;
        sigemptyset(&sa.sa_mask);
        sa.sa_flags = 0;
        sigaction(SIGVTALRM, &sa, NULL);

        setitimer(ITIMER_VIRTUAL, &ival, NULL);

        printf("wait\n");
        while (!quit) ;
        printf("done\n");
}


-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Darren Hart <dvhltc@us.ibm.com> wrote:

> Ingo Molnar wrote:
> >i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from 
> >the usual place:
> >
> >  http://redhat.com/~mingo/realtime-preempt/
> >
> 
> I'm looking into getting HRT and RT booting on a SUMMIT NUMA machine 
> (cyclone timer), but after s/error/warning/ in 
> arch/i386/timers/timer.c for the HRT cyclone ifdef, I still get the 
> following link error:

should be fixed in 2.6.13-rc6-rt12 and later patches.

	Ingo

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Tue, 2005-08-23 at 14:36 +0200, Ingo Molnar wrote:
> > * Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > Ingo, can't you get rt.c to be more confusing. I mean it is too 
> > > simple. We need to add a few more underscores here and there :-) 
> > > Seriously, that rt.c is mind boggling. It was nice before, now it is 
> > > just screaming for a cleanup (come now, do we really need the four 
> > > underscores?). Same with latency.c.
> > 
> > i agree that it's ugly, but some of that ugliness is to achieve the 
> > 7-instructions fail-through codepath for the common acquire (and 
> > release) codepath:
> > 
> >  c03a5320 <__down_mutex>:
> >  c03a5320:       89 c1                   mov    %eax,%ecx
> >  c03a5322:       8b 15 08 76 3a c0       mov    0xc03a7608,%edx
> >  c03a5328:       31 c0                   xor    %eax,%eax
> >  c03a532a:       0f b1 51 14             cmpxchg %edx,0x14(%ecx)
> >  c03a532e:       85 c0                   test   %eax,%eax
> >  c03a5330:       75 01                   jne    c03a5333 <__down_mutex+0x13>
> >  c03a5332:       c3                      ret
> > 
> 
> Impressive!
> 
> > that's how much it takes to acquire an RT lock, and i worked hard to get 
> > there. As long as the fastpath is kept this tight, feel free to do 
> > cleanups. But i really want to avoid having to write mutex_down/up in 
> > assembly for 24 architectures ...
> 
> Warning! I'm hacking hard to get rid of the global pi_lock, and I'm not
> worrying now about efficiency.  I figure that if I can get it to work,
> then we can speed it up afterwards.  Since it's complex enough keeping
> all the locks straight, I just want it to work without deadlocking. 
> 
> Once I get it to work, I'll let you figure out how get it back down to 
> 7-instructions :-)

yeah. It can always be done after the fact - the basics wont change.  
(Note that the above disassembly is for UP, on SMP the fastpath is 
longer and around 10-15 instructions.)

	Ingo

Re: 2.6.13-rc6-rt9

[Steven Rostedt]

On Tue, 2005-08-23 at 14:36 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > Ingo, can't you get rt.c to be more confusing. I mean it is too 
> > simple. We need to add a few more underscores here and there :-) 
> > Seriously, that rt.c is mind boggling. It was nice before, now it is 
> > just screaming for a cleanup (come now, do we really need the four 
> > underscores?). Same with latency.c.
> 
> i agree that it's ugly, but some of that ugliness is to achieve the 
> 7-instructions fail-through codepath for the common acquire (and 
> release) codepath:
> 
>  c03a5320 <__down_mutex>:
>  c03a5320:       89 c1                   mov    %eax,%ecx
>  c03a5322:       8b 15 08 76 3a c0       mov    0xc03a7608,%edx
>  c03a5328:       31 c0                   xor    %eax,%eax
>  c03a532a:       0f b1 51 14             cmpxchg %edx,0x14(%ecx)
>  c03a532e:       85 c0                   test   %eax,%eax
>  c03a5330:       75 01                   jne    c03a5333 <__down_mutex+0x13>
>  c03a5332:       c3                      ret
> 

Impressive!

> that's how much it takes to acquire an RT lock, and i worked hard to get 
> there. As long as the fastpath is kept this tight, feel free to do 
> cleanups. But i really want to avoid having to write mutex_down/up in 
> assembly for 24 architectures ...

Warning! I'm hacking hard to get rid of the global pi_lock, and I'm not
worrying now about efficiency.  I figure that if I can get it to work,
then we can speed it up afterwards.  Since it's complex enough keeping
all the locks straight, I just want it to work without deadlocking. 

Once I get it to work, I'll let you figure out how get it back down to
7-instructions :-)

-- Steve

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Ingo, can't you get rt.c to be more confusing. I mean it is too 
> simple. We need to add a few more underscores here and there :-) 
> Seriously, that rt.c is mind boggling. It was nice before, now it is 
> just screaming for a cleanup (come now, do we really need the four 
> underscores?). Same with latency.c.

i agree that it's ugly, but some of that ugliness is to achieve the 
7-instructions fail-through codepath for the common acquire (and 
release) codepath:

 c03a5320 <__down_mutex>:
 c03a5320:       89 c1                   mov    %eax,%ecx
 c03a5322:       8b 15 08 76 3a c0       mov    0xc03a7608,%edx
 c03a5328:       31 c0                   xor    %eax,%eax
 c03a532a:       0f b1 51 14             cmpxchg %edx,0x14(%ecx)
 c03a532e:       85 c0                   test   %eax,%eax
 c03a5330:       75 01                   jne    c03a5333 <__down_mutex+0x13>
 c03a5332:       c3                      ret

that's how much it takes to acquire an RT lock, and i worked hard to get 
there. As long as the fastpath is kept this tight, feel free to do 
cleanups. But i really want to avoid having to write mutex_down/up in 
assembly for 24 architectures ...

	Ingo

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 2005-08-19 at 02:39 -0400, Steven Rostedt wrote:
> 
> > Ingo, can't you get rt.c to be more confusing. I mean it is too simple.
> > We need to add a few more underscores here and there :-)  Seriously,
> > that rt.c is mind boggling. It was nice before, now it is just screaming
> > for a cleanup (come now, do we really need the four underscores?).  Same
> > with latency.c. 
> 
> Ingo,
> 
> Here's one example of cleaning up rt.c.  I like an extra parameter 
> instead of having two functions that are exactly the same except for 
> one line.  I'll probably submit more.

it was done like that deliberately, so that the fastpath doesnt include 
conditional code. (when all debugging options are disabled)

	Ingo

Re: 2.6.13-rc6-rt9

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 2005-08-19 at 09:00 -0400, Steven Rostedt wrote:
> > On Fri, 2005-08-19 at 02:39 -0400, Steven Rostedt wrote:
> 
> > I haven't thought of a good way yet to solve the race condition with
> > dependent sleeper. (Except by turning off CONFIG_WAKEUP_TIMING :-)
> > 
> 
> OK, I found one simple solution. The problem stems from max_mutex 
> being grabbed.  Since this uses the RT locks, and since tracing 
> shouldn't really care about PI and all that, I switched this to a 
> compat_semaphore, but only if CONFIG_WAKEUP_TIMING is set. This seems 
> to get rid of this race condition that I have.

ok, i have applied your patch and have done a small tweak: i made it a 
compat semaphore unconditionally. There's no point in #ifdefing it on 
WAKEUP_TIMING.

> I found more bugs, but for now this message is about this specific 
> race.

ok.

	Ingo

Re: 2.6.13-rc6-rt9

[Jeff Dike]

On Sat, Aug 20, 2005 at 09:27:25PM +0200, Peter Zijlstra wrote:
> Jeff, could you help us out here?
> What exactly does uml need to get out of the calibrate delay loop?

Interrupts, it's not too demanding :-)

If it's not seeing VTALRM, then it will never leave the calibration loop.

Try stracing it and see what it's getting.

				Jeff

Re: 2.6.13-rc6-rt9

[Peter Zijlstra]

On Fri, 2005-08-19 at 11:43 -0700, Paul E. McKenney wrote:
> On Fri, Aug 19, 2005 at 08:30:05PM +0200, Peter Zijlstra wrote:
> > On Fri, 2005-08-19 at 18:56 +0200, Peter Zijlstra wrote:
> > > Hi Ingo, Paul, others,
> > > 
> > > I'm trying to run a user-mode-linux guest under the RT kernel however
> > > the uml process never gets out of the calibrate delay loop. It seems as
> > > if the signal never gets through.
> > > 
> > one clarification: the guest kernel is a non -rt kernel, a modified
> > 2.6.13-rc6 in my case.
> > 
> > > A non -rt host kernel does work (with a similar .config).
> > > 
> > > Could this be related to pauls task list changes?
> 
> Possibly.  What signal?  This is a signal to a single process, right?
> 
Jeff, could you help us out here?
What exactly does uml need to get out of the calibrate delay loop?

Kind regards,

Peter Zijlstra

Re: 2.6.13-rc6-rt9

[Darren Hart]

Thomas Gleixner wrote:
> On Fri, 2005-08-19 at 15:13 -0700, Darren Hart wrote:
> 
>>I was trying to use another HRT clock source and couldn't get menuconfig 
>>to let me select acpi-pm-timer, turns out it has been disabled in 
>>arch/i386/Kconfig, but the description is still in the help...
>>
>>
>># config HIGH_RES_TIMER_ACPI_PM
>>#       bool "ACPI-pm-timer"
>>
>>Is the pm timer incompatible with the RT portion of this patch?
> 
> 
> The only timesource I came around to fixup is TSC in combination with
> PIT or preferred Local APIC. Add "lapic" to your kernel command line for
> UP boxen. Therefor it is disabled for now.
> 
> 
>>I'm looking into getting HRT and RT booting on a SUMMIT NUMA machine 
>>(cyclone timer), but after s/error/warning/ in arch/i386/timers/timer.c 
>>for the HRT cyclone ifdef, I still get the following link error:
> 
> 
> It should be simple to fix this. Just not right now. I have no such box
> and restricted time resources. Can you test a patch when I find a slot?

Absolutely.

> But of course you are heartely invited to fix it yourself :)

As always :-)

> 
> tglx


Thanks for the response, I wanted to hear your take on this rather than making 
any assumptions as to the state of the patch.

> 
> 
> 


-- 
Darren Hart
IBM Linux Technology Center
Linux Kernel Team
Phone: 503 578 3185
   T/L: 775 3185

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Fri, 2005-08-19 at 15:13 -0700, Darren Hart wrote:
> I was trying to use another HRT clock source and couldn't get menuconfig 
> to let me select acpi-pm-timer, turns out it has been disabled in 
> arch/i386/Kconfig, but the description is still in the help...
> 
> 
> # config HIGH_RES_TIMER_ACPI_PM
> #       bool "ACPI-pm-timer"
> 
> Is the pm timer incompatible with the RT portion of this patch?

The only timesource I came around to fixup is TSC in combination with
PIT or preferred Local APIC. Add "lapic" to your kernel command line for
UP boxen. Therefor it is disabled for now.

> I'm looking into getting HRT and RT booting on a SUMMIT NUMA machine 
> (cyclone timer), but after s/error/warning/ in arch/i386/timers/timer.c 
> for the HRT cyclone ifdef, I still get the following link error:

It should be simple to fix this. Just not right now. I have no such box
and restricted time resources. Can you test a patch when I find a slot?
But of course you are heartely invited to fix it yourself :)

tglx

Re: 2.6.13-rc6-rt9

[Darren Hart]

Ingo Molnar wrote:
> i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from 
> the usual place:
> 
>   http://redhat.com/~mingo/realtime-preempt/
> 

I was trying to use another HRT clock source and couldn't get menuconfig 
to let me select acpi-pm-timer, turns out it has been disabled in 
arch/i386/Kconfig, but the description is still in the help...


# config HIGH_RES_TIMER_ACPI_PM
#       bool "ACPI-pm-timer"

Is the pm timer incompatible with the RT portion of this patch?

Thanks,

--Darren

> it's a fixes-only release. Changes since 2.6.13-rc6-rt3:
> 
>  - USB irq flags use cleanups (Alan Stern)
> 
>  - RCU tasklist-lock fixes (Paul McKenney, Thomas Gleixner)
> 
>  - HR-timers waitqueue splitup, better HRT latencies (Thomas Gleixner)
> 
>  - latency tracer fixes, irq flags tracing cleanups (Steven Rostedt, me)
> 
>  - NFSd BKL unlock fix (Steven Rostedt)
> 
>  - stackfootprint-max-printer fix (Steven Rostedt)
> 
>  - stop_machine fix (Steven Rostedt)
> 
>  - lpptest fix (me)
> 
>  - turned off IOAPIC_POSTFLUSH when CONFIG_X86_IOAPIC_FAST. Now with
>    Karsten's VIA fixes my testbox does not show PCI-POST weirnesses
>    anymore. In case of IRQ problems please turn off IOAPIC_FAST. (me)
> 
> to build a 2.6.13-rc6-rt9 tree, the following patches should be applied:
> 
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
>    http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
>    http://redhat.com/~mingo/realtime-preempt/patch-2.6.13-rc6-rt9
> 
> 	Ingo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 


-- 
Darren Hart
IBM Linux Technology Center
Linux Kernel Team
Phone: 503 578 3185
   T/L: 775 3185

Re: 2.6.13-rc6-rt9

[Darren Hart]

Ingo Molnar wrote:
> i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from 
> the usual place:
> 
>   http://redhat.com/~mingo/realtime-preempt/
> 

I'm looking into getting HRT and RT booting on a SUMMIT NUMA machine 
(cyclone timer), but after s/error/warning/ in arch/i386/timers/timer.c 
for the HRT cyclone ifdef, I still get the following link error:


   LD      .tmp_vmlinux1
net/built-in.o(.init.text+0xdae): In function `sock_ioctl':
net/socket.c:868: undefined reference to `__you_cannot_kmalloc_that_much'
make: *** [.tmp_vmlinux1] Error 1


I was expecting to be able to build the kernel, and have it crash on 
boot (due to unsynched TSCs I'm guessing) and then start debugging/devel 
from there.  But the the rt9 patch won't build (HRT standalone for 
2.6.10 does build).  Anyone else seeing this?

> it's a fixes-only release. Changes since 2.6.13-rc6-rt3:
> 
>  - USB irq flags use cleanups (Alan Stern)
> 
>  - RCU tasklist-lock fixes (Paul McKenney, Thomas Gleixner)
> 
>  - HR-timers waitqueue splitup, better HRT latencies (Thomas Gleixner)
> 
>  - latency tracer fixes, irq flags tracing cleanups (Steven Rostedt, me)
> 
>  - NFSd BKL unlock fix (Steven Rostedt)
> 
>  - stackfootprint-max-printer fix (Steven Rostedt)
> 
>  - stop_machine fix (Steven Rostedt)
> 
>  - lpptest fix (me)
> 
>  - turned off IOAPIC_POSTFLUSH when CONFIG_X86_IOAPIC_FAST. Now with
>    Karsten's VIA fixes my testbox does not show PCI-POST weirnesses
>    anymore. In case of IRQ problems please turn off IOAPIC_FAST. (me)
> 
> to build a 2.6.13-rc6-rt9 tree, the following patches should be applied:
> 
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
>    http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
>    http://redhat.com/~mingo/realtime-preempt/patch-2.6.13-rc6-rt9
> 
> 	Ingo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 


-- 
Darren Hart
IBM Linux Technology Center
Linux Kernel Team
Phone: 503 578 3185
   T/L: 775 3185

Re: 2.6.13-rc6-rt9

[Paul E. McKenney]

On Fri, Aug 19, 2005 at 08:30:05PM +0200, Peter Zijlstra wrote:
> On Fri, 2005-08-19 at 18:56 +0200, Peter Zijlstra wrote:
> > Hi Ingo, Paul, others,
> > 
> > I'm trying to run a user-mode-linux guest under the RT kernel however
> > the uml process never gets out of the calibrate delay loop. It seems as
> > if the signal never gets through.
> > 
> one clarification: the guest kernel is a non -rt kernel, a modified
> 2.6.13-rc6 in my case.
> 
> > A non -rt host kernel does work (with a similar .config).
> > 
> > Could this be related to pauls task list changes?

Possibly.  What signal?  This is a signal to a single process, right?

						Thanx, Paul

Re: 2.6.13-rc6-rt9

[Peter Zijlstra]

On Fri, 2005-08-19 at 18:56 +0200, Peter Zijlstra wrote:
> Hi Ingo, Paul, others,
> 
> I'm trying to run a user-mode-linux guest under the RT kernel however
> the uml process never gets out of the calibrate delay loop. It seems as
> if the signal never gets through.
> 
one clarification: the guest kernel is a non -rt kernel, a modified
2.6.13-rc6 in my case.

> A non -rt host kernel does work (with a similar .config).
> 
> Could this be related to pauls task list changes?
> 
> Kind regards,
> 
-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: 2.6.13-rc6-rt9

[Peter Zijlstra]

Hi Ingo, Paul, others,

I'm trying to run a user-mode-linux guest under the RT kernel however
the uml process never gets out of the calibrate delay loop. It seems as
if the signal never gets through.

A non -rt host kernel does work (with a similar .config).

Could this be related to pauls task list changes?

Kind regards,

-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: 2.6.13-rc6-rt9

[Steven Rostedt]

On Fri, 2005-08-19 at 09:00 -0400, Steven Rostedt wrote:
> On Fri, 2005-08-19 at 02:39 -0400, Steven Rostedt wrote:

> I haven't thought of a good way yet to solve the race condition with
> dependent sleeper. (Except by turning off CONFIG_WAKEUP_TIMING :-)
> 

OK, I found one simple solution. The problem stems from max_mutex being
grabbed.  Since this uses the RT locks, and since tracing shouldn't
really care about PI and all that, I switched this to a
compat_semaphore, but only if CONFIG_WAKEUP_TIMING is set. This seems to
get rid of this race condition that I have.

I found more bugs, but for now this message is about this specific race.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_ernie/kernel/latency.c
===================================================================
--- linux_realtime_ernie/kernel/latency.c	(revision 297)
+++ linux_realtime_ernie/kernel/latency.c	(working copy)
@@ -102,7 +102,19 @@
 /*
  * Track maximum latencies and save the trace:
  */
+#ifdef CONFIG_WAKEUP_TIMING
+/*
+ * The WAKEUP_TIMING has a race condition, since 
+ * trace_stop_sched_switched might be called with run queue locks held
+ * which eventually calls down_trylock.
+ * But the RT version of down_trylock grabs a bunch of locks that
+ * are used by rt_up.  rt_up can call wake_up_process which 
+ * eventually grabs a run queue lock.
+ */
+static __cacheline_aligned_in_smp COMPAT_DECLARE_MUTEX(max_mutex);
+#else
 static __cacheline_aligned_in_smp DECLARE_MUTEX(max_mutex);
+#endif
 /*
  * Sequence count - we record it when starting a measurement and
  * skip the latency if the sequence has changed - some other section

Re: 2.6.13-rc6-rt9

[Steven Rostedt]

On Fri, 2005-08-19 at 02:39 -0400, Steven Rostedt wrote:

> Ingo, can't you get rt.c to be more confusing. I mean it is too simple.
> We need to add a few more underscores here and there :-)  Seriously,
> that rt.c is mind boggling. It was nice before, now it is just screaming
> for a cleanup (come now, do we really need the four underscores?).  Same
> with latency.c. 

Ingo,

Here's one example of cleaning up rt.c.  I like an extra parameter
instead of having two functions that are exactly the same except for one
line.  I'll probably submit more.

I haven't thought of a good way yet to solve the race condition with
dependent sleeper. (Except by turning off CONFIG_WAKEUP_TIMING :-)

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux_realtime_ernie/kernel/rt.c
===================================================================
--- linux_realtime_ernie/kernel/rt.c	(revision 296)
+++ linux_realtime_ernie/kernel/rt.c	(working copy)
@@ -1331,8 +1331,7 @@
 	FREE_WAITER(&waiter);
 }
 
-static void __up_mutex_waiter_savestate(struct rt_mutex *lock __EIP_DECL__);
-static void __up_mutex_waiter_nosavestate(struct rt_mutex *lock __EIP_DECL__);
+static void __up_mutex_waiter(struct rt_mutex *lock, int save_state __EIP_DECL__);
 
 /*
  * release the lock:
@@ -1361,12 +1360,9 @@
 	if (plist_empty(&lock->wait_list))
 		check_pi_list_empty(lock, lock_owner(lock));
 #endif
-	if (unlikely(!plist_empty(&lock->wait_list))) {
-		if (save_state)
-			__up_mutex_waiter_savestate(lock __EIP__);
-		else
-			__up_mutex_waiter_nosavestate(lock __EIP__);
-	} else
+	if (unlikely(!plist_empty(&lock->wait_list)))
+			__up_mutex_waiter(lock, save_state __EIP__);
+	else
 		lock->owner = NULL;
 	__raw_spin_unlock(&pi_lock);
 	__raw_spin_unlock(&lock->wait_lock);
@@ -1759,7 +1755,7 @@
 	return __down_trylock(&rwsem->lock __CALLER0__);
 }
 
-static void __up_mutex_waiter_nosavestate(struct rt_mutex *lock __EIP_DECL__)
+static void __up_mutex_waiter(struct rt_mutex *lock, int save_state __EIP_DECL__)
 {
 	struct thread_info *old_owner_ti, *new_owner_ti;
 	struct task_struct *old_owner, *new_owner;
@@ -1790,43 +1786,12 @@
 		new_owner->pending_owner = lock;
 	}
 #endif
-	wake_up_process(new_owner);
+	if (save_state)
+		wake_up_process_mutex(new_owner);
+	else
+		wake_up_process(new_owner);
 }
 
-static void __up_mutex_waiter_savestate(struct rt_mutex *lock __EIP_DECL__)
-{
-	struct thread_info *old_owner_ti, *new_owner_ti;
-	struct task_struct *old_owner, *new_owner;
-	struct rt_mutex_waiter *w;
-	int prio;
-
-	old_owner_ti = lock_owner(lock);
-	old_owner = old_owner_ti->task;
-	new_owner_ti = pick_new_owner(lock, old_owner_ti, 1 __EIP__);
-	new_owner = new_owner_ti->task;
-
-	/*
-	 * If the owner got priority-boosted then restore it
-	 * to the previous priority (or to the next highest prio
-	 * waiter's priority):
-	 */
-	prio = old_owner->normal_prio;
-	if (unlikely(!plist_empty(&old_owner->pi_waiters))) {
-		w = plist_first_entry(&old_owner->pi_waiters, struct rt_mutex_waiter, pi_list);
-		if (w->ti->task->prio < prio)
-			prio = w->ti->task->prio;
-	}
-	if (unlikely(prio != old_owner->prio))
-		pi_setprio(lock, old_owner, prio);
-#ifdef CAPTURE_LOCK
-	if (lock != &kernel_sem.lock) {
-		new_owner->rt_flags |= RT_PENDOWNER;
-		new_owner->pending_owner = lock;
-	}
-#endif
-	wake_up_process_mutex(new_owner);
-}
-
 /*
  * Do owner check too:
  */

Re: 2.6.13-rc6-rt9

[Steven Rostedt]

Deadlock finally found!!! 

I've been debugging this all week. And at 2:30 in the morning I finally
found where it is. It really sucks when you need to debug on something
that doesn't have a serial, and netconsole doesn't work that reliably.

This also explains why this only happened on my laptop. It is caused by
the dependent_sleeper, which is used quite a bit when you have SCHED_HT
turned on.  Although I wouldn't be surprised if this deadlock exists
elsewhere.

The dependent_sleeper (used by SCHED_HT) grabs all the run queue locks
for the physical CPU.  Then it calls trace_stop_sched_switched. So this
is the calling chain.

dependent_sleeper
    +==> grabs CPU0 rq and CPU1 rq lock (saying CPU 0 and 1 are on the
same physical CPU)
   -> trace_stop_sched_switched
      -> check_wakeup_timing
          -> down_trylock(max_mutex)
                -> rt_down_trylock
                    -> __down_trylock
                       +==> grabs trace_lock

Now lets look at something at the same time that is unlocking.

rt_up
  -> __up_mutex_nosavestate_inline
    ->  ___up_mutex_nosavestate
       -> ____up_mutex
          +==> grabs trace_lock
         -> __up_mutex_waiter_nosavestate
            -> wake_up_process
               -> try_to_wake_up
                 +==> grabs rq lock

Here we can see that there's a reverse order here and we have a
deadlock.  I actually witness this using my logger to show the traces.
All I needed was the last few lines, so the console was fine here.

Ingo, can't you get rt.c to be more confusing. I mean it is too simple.
We need to add a few more underscores here and there :-)  Seriously,
that rt.c is mind boggling. It was nice before, now it is just screaming
for a cleanup (come now, do we really need the four underscores?).  Same
with latency.c. 

Well, there's the deadlock, I'm too tired to figure out all the paths,
and what the heck is going on. So I'll give you the honour of writing
the patch. ;-)

Thanks,

-- Steve

Re: 2.6.13-rc6-rt9

[Chuck Harding]

On Thu, 18 Aug 2005, Ingo Molnar wrote:

>
> i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from
> the usual place:
>
>  http://redhat.com/~mingo/realtime-preempt/
>
> it's a fixes-only release. Changes since 2.6.13-rc6-rt3:
>
> - USB irq flags use cleanups (Alan Stern)
>
> - RCU tasklist-lock fixes (Paul McKenney, Thomas Gleixner)
>
> - HR-timers waitqueue splitup, better HRT latencies (Thomas Gleixner)
>
> - latency tracer fixes, irq flags tracing cleanups (Steven Rostedt, me)
>
> - NFSd BKL unlock fix (Steven Rostedt)
>
> - stackfootprint-max-printer fix (Steven Rostedt)
>
> - stop_machine fix (Steven Rostedt)
>
> - lpptest fix (me)
>
> - turned off IOAPIC_POSTFLUSH when CONFIG_X86_IOAPIC_FAST. Now with
>   Karsten's VIA fixes my testbox does not show PCI-POST weirnesses
>   anymore. In case of IRQ problems please turn off IOAPIC_FAST. (me)

I'm still getting the same oops when rebooting. the same process (reboot)
similar call trace (some addresses are slightly different but the functions
are the same:
disable_IO_APIC+0x5a/0x90 (8)
machine_restart+0x5/0x9 (28)
sys_reboot+0x147/0x156 (4)
netdev_run_todo+0xa4/0x209 (4)
etc.

Another interesting data point is that I did a SysRq+B right after the
machine came up and got a different oops.

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Thu, 2005-08-18 at 17:24 +0200, Thomas Gleixner wrote:
> finally found the deadlock. It was caused by IRQ flood, which was
> introduced by the end_irq() changes.

Found another one to back out. It creaped in with the same patch. 

It's slow and just conceals bad configured PICs and crappy demux
handlers.


tglx


--- linux-2.6.13-rc6-rt9/arch/ppc/syslib/open_pic.c	2005-08-18 17:37:39.000000000 +0200
+++ linux-2.6.13-rc6-rt9.work/arch/ppc/syslib/open_pic.c	2005-08-18 23:02:12.000000000 +0200
@@ -816,10 +816,6 @@ static void openpic_set_sense(u_int irq,
 }
 #endif /* notused */
 
-#ifdef CONFIG_PREEMPT_RT
-#define __SLOW_VERSION__
-#endif
-
 /* No spinlocks, should not be necessary with the OpenPIC
  * (1 register = 1 interrupt and we have the desc lock).
  */

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Thu, 2005-08-18 at 17:24 +0200, Thomas Gleixner wrote:

Oops, mailer madness.

tglx


diff -uprN --exclude-from=/usr/local/bin/diffit.exclude linux-2.6.13-rc6-rt8/kernel/irq/handle.c linux-2.6.13-rc6-rt-debug/kernel/irq/handle.c
--- linux-2.6.13-rc6-rt8/kernel/irq/handle.c	2005-08-17 17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/handle.c	2005-08-18 16:32:54.000000000 +0200
@@ -171,7 +171,7 @@ fastcall notrace unsigned int __do_IRQ(u
 		 */
 		desc->handler->ack(irq);
 		action_ret = handle_IRQ_event(irq, regs, desc->action);
-		end_irq(desc, irq);
+		desc->handler->end(irq);
 		return 1;
 	}
 
@@ -241,7 +241,7 @@ out:
 	 * The end-handler has to deal with interrupts which got
 	 * disabled while the handler was running:
 	 */
-	end_irq(desc, irq);
+	desc->handler->end(irq);
 out_no_end:
 	spin_unlock(&desc->lock);
 
diff -uprN --exclude-from=/usr/local/bin/diffit.exclude linux-2.6.13-rc6-rt8/kernel/irq/internals.h linux-2.6.13-rc6-rt-debug/kernel/irq/internals.h
--- linux-2.6.13-rc6-rt8/kernel/irq/internals.h	2005-08-17 17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/internals.h	2005-08-18 16:39:56.000000000 +0200
@@ -6,21 +6,6 @@ extern int noirqdebug;
 
 void recalculate_desc_flags(struct irq_desc *desc);
 
-/*
- * On PREEMPT_HARDIRQS, the ->ack handler masks interrupts, so that
- * they can be redirected to an IRQ thread, if needed. So here we
- * have to unmask the interrupt line, if needed:
- */
-static inline void end_irq(irq_desc_t *desc, unsigned int irq)
-{
-#ifdef CONFIG_PREEMPT_HARDIRQS
-	if (!(desc->status & IRQ_DISABLED))
-		desc->handler->enable(irq);
-#else
-	desc->handler->end(irq);
-#endif
-}
-
 #ifdef CONFIG_PROC_FS
 extern void register_irq_proc(unsigned int irq);
 extern void register_handler_proc(unsigned int irq, struct irqaction *action);
diff -uprN --exclude-from=/usr/local/bin/diffit.exclude linux-2.6.13-rc6-rt8/kernel/irq/manage.c linux-2.6.13-rc6-rt-debug/kernel/irq/manage.c
--- linux-2.6.13-rc6-rt8/kernel/irq/manage.c	2005-08-17 17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/manage.c	2005-08-18 16:31:46.000000000 +0200
@@ -442,7 +442,7 @@ static void do_hardirq(struct irq_desc *
 		 * The end-handler has to deal with interrupts which got
 		 * disabled while the handler was running:
 		 */
-		end_irq(desc, irq);
+		desc->handler->end(irq);
 	}
 	spin_unlock_irq(&desc->lock);
 

Re: 2.6.13-rc6-rt9

[Thomas Gleixner]

On Thu, 2005-08-18 at 08:01 +0200, Ingo Molnar wrote:
> i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from 
> the usual place:

Hi Ingo,

finally found the deadlock. It was caused by IRQ flood, which was
introduced by the end_irq() changes.

They change the semantics in two ways - both wrong. 

1. The condition is different
2. ->end() can contain different code than ->enable()

That's the code in end_8259A_irq():

	if (!(irq_desc[irq].status & (IRQ_DISABLED|IRQ_INPROGRESS)) &&
							irq_desc[irq].action)
		enable_8259A_irq(irq);

The code in end_irq():

	if (!(desc->status & IRQ_DISABLED))
		desc->handler->enable(irq);


What was the reason for those changes ?

tglx


diff -uprN --exclude-from=/usr/local/bin/diffit.exclude
linux-2.6.13-rc6-rt8/kernel/irq/handle.c
linux-2.6.13-rc6-rt-debug/kernel/irq/handle.c
--- linux-2.6.13-rc6-rt8/kernel/irq/handle.c 2005-08-17
17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/handle.c 2005-08-18
16:32:54.000000000 +0200
@@ -171,7 +171,7 @@ fastcall notrace unsigned int __do_IRQ(u
*/
desc->handler->ack(irq);
action_ret = handle_IRQ_event(irq, regs, desc->action);
- end_irq(desc, irq);
+ desc->handler->end(irq);
return 1;
}

@@ -241,7 +241,7 @@ out:
* The end-handler has to deal with interrupts which got
* disabled while the handler was running:
*/
- end_irq(desc, irq);
+ desc->handler->end(irq);
out_no_end:
spin_unlock(&desc->lock);
diff -uprN --exclude-from=/usr/local/bin/diffit.exclude linux-2.6.13-rc6-rt8/kernel/irq/internals.h linux-2.6.13-rc6-rt-debug/kernel/irq/internals.h
--- linux-2.6.13-rc6-rt8/kernel/irq/internals.h	2005-08-17 17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/internals.h	2005-08-18 16:39:56.000000000 +0200
@@ -6,21 +6,6 @@ extern int noirqdebug;
 
 void recalculate_desc_flags(struct irq_desc *desc);
 
-/*
- * On PREEMPT_HARDIRQS, the ->ack handler masks interrupts, so that
- * they can be redirected to an IRQ thread, if needed. So here we
- * have to unmask the interrupt line, if needed:
- */
-static inline void end_irq(irq_desc_t *desc, unsigned int irq)
-{
-#ifdef CONFIG_PREEMPT_HARDIRQS
-	if (!(desc->status & IRQ_DISABLED))
-		desc->handler->enable(irq);
-#else
-	desc->handler->end(irq);
-#endif
-}
-
 #ifdef CONFIG_PROC_FS
 extern void register_irq_proc(unsigned int irq);
 extern void register_handler_proc(unsigned int irq, struct irqaction *action);
diff -uprN --exclude-from=/usr/local/bin/diffit.exclude linux-2.6.13-rc6-rt8/kernel/irq/manage.c linux-2.6.13-rc6-rt-debug/kernel/irq/manage.c
--- linux-2.6.13-rc6-rt8/kernel/irq/manage.c	2005-08-17 17:53:13.000000000 +0200
+++ linux-2.6.13-rc6-rt-debug/kernel/irq/manage.c	2005-08-18 16:31:46.000000000 +0200
@@ -442,7 +442,7 @@ static void do_hardirq(struct irq_desc *
 		 * The end-handler has to deal with interrupts which got
 		 * disabled while the handler was running:
 		 */
-		end_irq(desc, irq);
+		desc->handler->end(irq);
 	}
 	spin_unlock_irq(&desc->lock);
 

2.6.13-rc6-rt9

[Ingo Molnar]

i have released the 2.6.13-rc6-rt9 tree, which can be downloaded from 
the usual place:

  http://redhat.com/~mingo/realtime-preempt/

it's a fixes-only release. Changes since 2.6.13-rc6-rt3:

 - USB irq flags use cleanups (Alan Stern)

 - RCU tasklist-lock fixes (Paul McKenney, Thomas Gleixner)

 - HR-timers waitqueue splitup, better HRT latencies (Thomas Gleixner)

 - latency tracer fixes, irq flags tracing cleanups (Steven Rostedt, me)

 - NFSd BKL unlock fix (Steven Rostedt)

 - stackfootprint-max-printer fix (Steven Rostedt)

 - stop_machine fix (Steven Rostedt)

 - lpptest fix (me)

 - turned off IOAPIC_POSTFLUSH when CONFIG_X86_IOAPIC_FAST. Now with
   Karsten's VIA fixes my testbox does not show PCI-POST weirnesses
   anymore. In case of IRQ problems please turn off IOAPIC_FAST. (me)

to build a 2.6.13-rc6-rt9 tree, the following patches should be applied:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
   http://redhat.com/~mingo/realtime-preempt/patch-2.6.13-rc6-rt9

	Ingo