Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[Mark_H_Johnson]

>i've uploaded a fixed kernel (-V0.6.8) to:
>
>  http://redhat.com/~mingo/realtime-preempt/

This build appears to run OK and then in the middle of the real time
tests stops doing useful work (during network test).

 - can move mouse & switch displays
 - script checking latency hangs (no errors, apparently stuck in sleep)
 - ps is OK, but top does not work (no display, Ctrl-C gives prompt again)
 - cannot ping the test system from another system (don't get the "no route
message, just no response)
 - Alt-SysRq keys still work
 - eventually the mouse would not switch displays

Rebooted with Alt-SysRq-B.

Sending serial console results separately.
  --Mark

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> This build appears to run OK and then in the middle of the real time
> tests stops doing useful work (during network test).

weird, the deadlock detector did not trigger, although it is a clear
circular deadlock:

R     ksoftirqd/0/    3 [dffe8020, 105] blocked on: [dfcafc20] {r:0,a:-1,&((sk)->sk_lock.slock)}
.. held by:               rcp/ 4791 [d84bd910, 118]
... acquired at:  ip_send_reply+0x12b/0x250

D             rcp/ 4791 [d84bd910, 118] blocked on: [c03cb900] {r:0,a:-1,ptype_lock}
.. held by:       ksoftirqd/0/    3 [dffe8020, 105]
... acquired at:  netif_receive_skb+0xae/0x330

i.e. ksoftirqd blocked on rcp and rcp blocked on ksoftirqd, cleanly
deadlocking each other.

anyway, does the patch below, ontop of -V0.6.8, fix this type of hang?

	Ingo

--- linux/net/core/dev.c.orig
+++ linux/net/core/dev.c
@@ -1916,7 +1916,9 @@ static void net_rx_action(struct softirq
 		dev = list_entry(queue->poll_list.next,
 				 struct net_device, poll_list);
 
+		rcu_read_lock_read(&ptype_lock);
 		if (dev->quota <= 0 || dev->poll(dev, &budget)) {
+			rcu_read_unlock_read(&ptype_lock);
 			local_irq_disable();
 			list_del(&dev->poll_list);
 			list_add_tail(&dev->poll_list, &queue->poll_list);
@@ -1926,6 +1928,7 @@ static void net_rx_action(struct softirq
 				dev->quota = dev->weight;
 		} else {
 			dev_put(dev);
+			rcu_read_unlock_read(&ptype_lock);
 			local_irq_disable();
 		}
 

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > This build appears to run OK and then in the middle of the real time
> > tests stops doing useful work (during network test).
> 
> weird, the deadlock detector did not trigger, although it is a clear
> circular deadlock:

ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
-V0.6.9 that fixes this.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[K.R. Foley]

Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> 
>>>This build appears to run OK and then in the middle of the real time
>>>tests stops doing useful work (during network test).
>>
>>weird, the deadlock detector did not trigger, although it is a clear
>>circular deadlock:
> 
> 
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.
> 
> 	Ingo
> 
I probably shouldn't say this, but I have yet to have -V0.6.9 die on me. 
And at least right now, the peppiness is back in the X interface, the 
menus, etc.

kr

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.9

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.
>

OK. Already tested RT-V0.6.9 and things are really good, so far. Got it
under the jackd-R + 9*fluidsynth workload test, and the comparison to
2.6.9 (vanilla) is getting kind of humiliating :)

                                   2.6.9   RT-V0.6.9
                                 --------- ---------
 XRUN Rate   . . . . . . . . . :    415         0    /hour
 Delay Rate (>spare time)  . . :    493         0    /hour
 Delay Rate (>1000 usecs)  . . :    913         1    /hour
 Delay Maximum . . . . . . . . :   6877       864    usecs
 Cycle Maximum . . . . . . . . :   1440      1552    usecs
 Average DSP Load. . . . . . . :     38.9      40.5  %
 Average Interrupt Rate  . . . :   1337      1338    /sec
 Average Context-Switch Rate . :   7488      9048    /sec

As before, these stats were taken by running jackd -v -dalsa -dhw:0
-r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on a
P4@2.533Ghz laptop, against the onboard sound device (snd-ali5451).
The results were averaged for 12 consecutive runs of 5 minutes each.

On the RT kernel, the IRQ 5 handler thread, that serves the ali5451 sound
device, has been chrt'ed to SCHED_FIFO and to priority=60 (chrt -p -f 60
`pidof "IRQ 5"`). This time thought, I haven't touched the ksoftirqd/0
scheduling policy nor priority.

I am also rehearsing these same tests on my P4/SMT desktop. I'll post
those a bit later today.

As a personal comment, I've never, never seen so good figures in any other
kernel I've came across in the last couple of years. I hope this can only
go to the better ;)

See ya.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.9

[Bill Huey]

On Tue, Nov 02, 2004 at 11:21:43PM -0000, Rui Nuno Capela wrote:
> As a personal comment, I've never, never seen so good figures in any other
> kernel I've came across in the last couple of years. I hope this can only
> go to the better ;)

Hard RT guarantees is the goal of this project afterall. It's absolutely
going to happen. You can bet on it.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.9

[Rui Nuno Capela]

Rui Nuno Capela wrote:
>
> I am also rehearsing these same tests on my P4/SMT desktop.
> I'll post those a bit later today.
>

So here they are. This machine is a P4 2.80C@3.366GHz, HT enabled on a
Asus P4P800 mobo, Intel 82801EB onboard sound device (snd-intel8x0).

                                   2.6.9smp   RT-V0.6.9smp
                                 ------------ ------------
 XRUN Rate   . . . . . . . . . :       0            0      /hour
 Delay Rate (>spare time)  . . :       0            0      /hour
 Delay Rate (>1000 usecs)  . . :       0            0      /hour
 Delay Maximum . . . . . . . . :     346          166      usecs
 Cycle Maximum . . . . . . . . :     986         1028      usecs
 Average DSP Load. . . . . . . :      25.0         25.7    %
 Average Interrupt Rate  . . . :    1717         1718      /sec
 Average Context-Switch Rate . :   10082        15793      /sec

As you can see, the results here aren't so disparate as on my UP laptop.

I guess that I can stress this out and lower the period size on jackd -R,
to -p64. And even increase the workload with more client instances (from 9
to 18 ?) to let the RT show it's potential, if that makes sense at all.

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.9

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Rui Nuno Capela wrote:
> >
> > I am also rehearsing these same tests on my P4/SMT desktop.
> > I'll post those a bit later today.
> >
> 
> So here they are. This machine is a P4 2.80C@3.366GHz, HT enabled on a
> Asus P4P800 mobo, Intel 82801EB onboard sound device (snd-intel8x0).
> 
>                                    2.6.9smp   RT-V0.6.9smp
>                                  ------------ ------------
>  XRUN Rate   . . . . . . . . . :       0            0      /hour
>  Delay Rate (>spare time)  . . :       0            0      /hour
>  Delay Rate (>1000 usecs)  . . :       0            0      /hour
>  Delay Maximum . . . . . . . . :     346          166      usecs
>  Cycle Maximum . . . . . . . . :     986         1028      usecs
>  Average DSP Load. . . . . . . :      25.0         25.7    %
>  Average Interrupt Rate  . . . :    1717         1718      /sec
>  Average Context-Switch Rate . :   10082        15793      /sec
> 
> As you can see, the results here aren't so disparate as on my UP
> laptop.

i suspect it has to do with the different load - the UP laptop had 40%
utilization, this box is 25% utilized.

> I guess that I can stress this out and lower the period size on jackd
> -R, to -p64. And even increase the workload with more client instances
> (from 9 to 18 ?) to let the RT show it's potential, if that makes
> sense at all.

yeah, it would really be nice to see how much it can take and to see the
point where the -RT kernel starts to suffer from xruns too.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.9

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. Already tested RT-V0.6.9 and things are really good, so far. Got
> it under the jackd-R + 9*fluidsynth workload test, and the comparison
> to 2.6.9 (vanilla) is getting kind of humiliating :)
> 
>                                    2.6.9   RT-V0.6.9
>                                  --------- ---------
>  XRUN Rate   . . . . . . . . . :    415         0    /hour
>  Delay Rate (>spare time)  . . :    493         0    /hour
>  Delay Rate (>1000 usecs)  . . :    913         1    /hour
>  Delay Maximum . . . . . . . . :   6877       864    usecs
>  Cycle Maximum . . . . . . . . :   1440      1552    usecs
>  Average DSP Load. . . . . . . :     38.9      40.5  %
>  Average Interrupt Rate  . . . :   1337      1338    /sec
>  Average Context-Switch Rate . :   7488      9048    /sec

cool, good numbers. May i have a stats suggestion for future runs? Could
you monitor the system (kernel) CPU overhead and userspace overhead as
well?

> As before, these stats were taken by running jackd -v -dalsa -dhw:0
> -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on
> a P4@2.533Ghz laptop, against the onboard sound device (snd-ali5451).
> The results were averaged for 12 consecutive runs of 5 minutes each.
> 
> On the RT kernel, the IRQ 5 handler thread, that serves the ali5451
> sound device, has been chrt'ed to SCHED_FIFO and to priority=60 (chrt
> -p -f 60 `pidof "IRQ 5"`). This time thought, I haven't touched the
> ksoftirqd/0 scheduling policy nor priority.
> 
> I am also rehearsing these same tests on my P4/SMT desktop. I'll post
> those a bit later today.
> 
> As a personal comment, I've never, never seen so good figures in any
> other kernel I've came across in the last couple of years. I hope this
> can only go to the better ;)

i hope so too :) There's no reason why the performance of the current
-RT kernel couldnt be maintained, there are no corners cut.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[Fernando Pablo Lopez-Lezcano]

On Tue, 2004-11-02 at 11:27, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> > > This build appears to run OK and then in the middle of the real time
> > > tests stops doing useful work (during network test).
> > 
> > weird, the deadlock detector did not trigger, although it is a clear
> > circular deadlock:
> 
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.

This happens with V0.6.9 running on an Athlon64:

r8169 Gigabit Ethernet driver 1.6LK loaded
ACPI: PCI interrupt 0000:00:0b.0[A] -> GSI 16 (level, low) -> IRQ 16
divert: allocating divert_blk for eth0
eth0: Identified chip type is 'RTL8169s/8110s'.
eth0: RTL8169 at 0xf88a0f00, 00:0c:76:b3:c2:43, IRQ 16
BUG: atomic counter underflow at:
 [<c0108153>] dump_stack+0x23/0x30 (20)
 [<c02f5e91>] qdisc_destroy+0xe1/0xf0 (28)
 [<c02f60ad>] dev_shutdown+0x3d/0xa0 (28)
 [<c02e6ffb>] unregister_netdevice+0x12b/0x2a0 (36)
 [<c0273ffe>] unregister_netdev+0x1e/0x30 (16)
 [<f8b0b8b5>] rtl8169_remove_one+0x25/0x50 [r8169] (32)
 [<c01f55a6>] pci_device_remove+0x76/0x80 (24)
 [<c025f15d>] device_release_driver+0x6d/0x70 (24)
 [<c025f18b>] driver_detach+0x2b/0x40 (20)
 [<c025f5ff>] bus_remove_driver+0x3f/0x70 (20)
 [<c025fb0c>] driver_unregister+0x1c/0x30 (16)
 [<c01f586c>] pci_unregister_driver+0x1c/0x30 (16)
 [<f8b0d357>] rtl8169_cleanup_module+0x17/0x1b [r8169] (12)
 [<c013fdb1>] sys_delete_module+0x121/0x150 (96)
 [<c010729d>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013dc3d>] .... print_traces+0x1d/0x60
.....[<c0108153>] ..   ( <= dump_stack+0x23/0x30)
                                                                                
divert: freeing divert_blk for eth0
ip_tables: (C) 2000-2002 Netfilter core team

-- Fernando

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.6.8

[K.R. Foley]

Fernando Pablo Lopez-Lezcano wrote:
> On Tue, 2004-11-02 at 11:27, Ingo Molnar wrote:
> 
>>* Ingo Molnar <mingo@elte.hu> wrote:
>>
>>>>This build appears to run OK and then in the middle of the real time
>>>>tests stops doing useful work (during network test).
>>>
>>>weird, the deadlock detector did not trigger, although it is a clear
>>>circular deadlock:
>>
>>ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
>>detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
>>-V0.6.9 that fixes this.
> 
> 
> This happens with V0.6.9 running on an Athlon64:
> 
> r8169 Gigabit Ethernet driver 1.6LK loaded
> ACPI: PCI interrupt 0000:00:0b.0[A] -> GSI 16 (level, low) -> IRQ 16
> divert: allocating divert_blk for eth0
> eth0: Identified chip type is 'RTL8169s/8110s'.
> eth0: RTL8169 at 0xf88a0f00, 00:0c:76:b3:c2:43, IRQ 16
> BUG: atomic counter underflow at:
>  [<c0108153>] dump_stack+0x23/0x30 (20)
>  [<c02f5e91>] qdisc_destroy+0xe1/0xf0 (28)
>  [<c02f60ad>] dev_shutdown+0x3d/0xa0 (28)
>  [<c02e6ffb>] unregister_netdevice+0x12b/0x2a0 (36)
>  [<c0273ffe>] unregister_netdev+0x1e/0x30 (16)
>  [<f8b0b8b5>] rtl8169_remove_one+0x25/0x50 [r8169] (32)
>  [<c01f55a6>] pci_device_remove+0x76/0x80 (24)
>  [<c025f15d>] device_release_driver+0x6d/0x70 (24)
>  [<c025f18b>] driver_detach+0x2b/0x40 (20)
>  [<c025f5ff>] bus_remove_driver+0x3f/0x70 (20)
>  [<c025fb0c>] driver_unregister+0x1c/0x30 (16)
>  [<c01f586c>] pci_unregister_driver+0x1c/0x30 (16)
>  [<f8b0d357>] rtl8169_cleanup_module+0x17/0x1b [r8169] (12)
>  [<c013fdb1>] sys_delete_module+0x121/0x150 (96)
>  [<c010729d>] sysenter_past_esp+0x52/0x71 (-8124)
> ---------------------------
> | preempt count: 00000001 ]
> | 1-level deep critical section nesting:
> ----------------------------------------
> .. [<c013dc3d>] .... print_traces+0x1d/0x60
> .....[<c0108153>] ..   ( <= dump_stack+0x23/0x30)
>                                                                                 
> divert: freeing divert_blk for eth0
> ip_tables: (C) 2000-2002 Netfilter core team
> 
> -- Fernando
> 
> 
> 
Several of us get this. I believe this bug exists in upstream kernels 
also, it just doesn't get caught without the detection Ingo has added.

kr