Ingo,

On an IBM Intellistation A-Pro I am seeing the bug regularly as seen below.
We also see this bug on an LS-20 blade.  The way this bug is triggered is
to add 'irqpoll' to the kernel command line.  If we remove this from the
command line, we don't see this bug.  I know the easy answer is that if you
don't want to see the bug, don't use irqpoll.  So we don't.  But we thought
that just in case this bug does expose something real that it should be
reported.

I booted with a fresh vanilla linux-2.6.16 + patch-2.6.16-rt18 kernel.  I
attached the config file I used.

--Vernon


BUG at kernel/rtmutex.c:639!
------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:639!
invalid opcode: 0000 [#1]
PREEMPT SMP
Modules linked in: sata_sil libata sd_mod
CPU:    0
EIP:    0060:[<c0312d09>]    Not tainted VLI
EFLAGS: 00010082   (2.6.16-rt18 #1)
EIP is at rt_lock_slowlock+0x9b/0x195
eax: 00000020   ebx: f7c9e000   ecx: f7c9fc8c   edx: c0328a2e
esi: c0409680   edi: c0400600   ebp: 00000001   esp: f7c9fc88
ds: 007b   es: 007b   ss: 0068   preempt: 00010003
Process kseriod (pid: 294, threadinfo=f7c9e000 task=cd552030 stack_left=7252 worst_left=-1)
Stack: <0>c0328a2e c03309d5 0000027f f7c9fc90 c022b72f 0000008c f7c9fca0 f7c9fca0
       f7c9fca8 f7c9fca8 00000000 0000008c f7c9fcb8 f7c9fcb8 f7c9fcc0 f7c9fcc0
       00000000 00000000 11111111 11111111 11111111 11111111 c0313843 c0313466
Call Trace:
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (20)
 [<c0313843>] _raw_spin_unlock+0xd/0x21 (72)
 [<c0313466>] rt_unlock+0x5/0x6 (4)
 [<c012530e>] __mod_timer+0xa3/0xab (4)
 [<c0313460>] __lock_text_start+0x8/0x9 (20)
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (4)
 [<c013350e>] try_to_take_rt_mutex+0x165/0x172 (4)
 [<c014067d>] misrouted_irq+0x85/0x118 (32)
 [<c01407d4>] note_interrupt+0x43/0x9f (36)
 [<c013fbfe>] __do_IRQ+0xbd/0xf7 (16)
 [<c0104736>] do_IRQ+0x3e/0x4b (32)
 [<c010337e>] common_interrupt+0x1a/0x20 (12)
 [<c01c8dab>] delay_tsc+0x7/0x13 (44)
 [<c01c8de0>] __delay+0x6/0x7 (16)
 [<c022b309>] i8042_wait_write+0x19/0x3b (4)
 [<c022b43f>] i8042_command+0x8d/0x152 (8)
 [<c022b582>] i8042_aux_write+0x27/0x39 (20)
 [<c022c631>] ps2_sendbyte+0x49/0x114 (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c013ee94>] add_preempt_count+0x10/0x9b (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c022c960>] ps2_command+0xf5/0x2c9 (40)
 [<c0134e57>] rt_up+0x2c/0x42 (24)
 [<c022cb29>] ps2_command+0x2be/0x2c9 (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (24)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c02a1fbb>] intellimouse_detect+0x20/0xb2 (20)
 [<c02a205f>] im_explorer_detect+0x12/0xcd (20)
 [<c02a233c>] psmouse_extensions+0x11b/0x15b (20)
 [<c02a2945>] psmouse_switch_protocol+0xb1/0x14e (20)
 [<c02a2b30>] psmouse_connect+0x14e/0x1d2 (16)
 [<c0233ca9>] __driver_attach+0x0/0x5b (24)
 [<c022a287>] serio_connect_driver+0x1b/0x2a (4)
 [<c0233c06>] driver_probe_device+0x44/0x95 (16)
 [<c0233ce3>] __driver_attach+0x3a/0x5b (16)
 [<c023349a>] bus_for_each_dev+0x38/0x5c (16)
 [<c0233d15>] driver_attach+0x11/0x13 (36)
 [<c0233ca9>] __driver_attach+0x0/0x5b (4)
 [<c023386d>] bus_add_driver+0x58/0x8d (4)
 [<c022a708>] serio_thread+0x0/0xee (16)
 [<c022a64c>] serio_handle_event+0x5b/0x74 (4)
 [<c022a711>] serio_thread+0x9/0xee (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (32)
 [<c0312148>] schedule+0xcd/0xed (12)
 [<c012de21>] kthread+0x73/0x9b (16)
 [<c012ddae>] kthread+0x0/0x9b (16)
 [<c0101005>] kernel_thread_helper+0x5/0xb (16)
---------------------------
| preempt count: 00010003 ]
| 3-level deep critical section nesting:
----------------------------------------
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c03136e1>] .... _raw_spin_lock_irqsave+0x11/0x2e
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)

------------------------------
| showing all locks held by: |  (kseriod/294 [cd552030, 110]):
------------------------------

#001:             [c03a6224] {serio_sem.lock}
... acquired at:               rt_down+0xe/0x26

#002:             [f7d0a5e0] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#003:             [f7d0a2cc] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#004:             [f7d0a0b4] {(struct semaphore *)(&serio->drv_sem)}
... acquired at:               rt_down+0xe/0x26

#005:             [c03ba184] {psmouse_sem.lock}
... acquired at:               rt_down+0xe/0x26

#006:             [f7d88a10] {(struct semaphore *)(&ps2dev->cmd_sem)}
... acquired at:               rt_down+0xe/0x26

#007:             [c0409680] {drivers/input/serio/i8042.c:89}
... acquired at:               i8042_command+0x3c/0x152

Code: 00 e9 17 01 00 00 8b 46 1c bb 00 e0 ff ff 21 e3 83 e0 fc 3b 03 75 1f 68 7f 02 00 00 68 d5 09 33 c0 68 2e 8a 32 c0 e8 a3 b0 e0 ff <0f> 0b 7f 02 d5 09 33 c0 83 c4 0c bd 04 00 00 00 8b 03 87 28 bf
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 [<c011d684>] panic+0x41/0xe3 (8)
 [<c0103b98>] die+0x166/0x170 (20)
 [<c0103cf0>] do_invalid_op+0x0/0x8b (48)
 [<c0103d6f>] do_invalid_op+0x7f/0x8b (8)
 [<c013ee94>] add_preempt_count+0x10/0x9b (40)
 [<c0312d09>] rt_lock_slowlock+0x9b/0x195 (24)
 [<c0134e57>] rt_up+0x2c/0x42 (36)
 [<c011e0e0>] release_console_sem+0x6d/0xbf (8)
 [<c011dfd1>] vprintk+0x214/0x238 (16)
 [<c013ee94>] add_preempt_count+0x10/0x9b (4)
 [<c0134720>] debug_rt_mutex_lock+0xc0/0xc8 (28)
 [<c013350e>] try_to_take_rt_mutex+0x165/0x172 (20)
 [<c01034b3>] error_code+0x4f/0x54 (12)
 [<c0312d09>] rt_lock_slowlock+0x9b/0x195 (44)
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (28)
 [<c0313843>] _raw_spin_unlock+0xd/0x21 (72)
 [<c0313466>] rt_unlock+0x5/0x6 (4)
 [<c012530e>] __mod_timer+0xa3/0xab (4)
 [<c0313460>] __lock_text_start+0x8/0x9 (20)
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (4)
 [<c013350e>] try_to_take_rt_mutex+0x165/0x172 (4)
 [<c014067d>] misrouted_irq+0x85/0x118 (32)
 [<c01407d4>] note_interrupt+0x43/0x9f (36)
 [<c013fbfe>] __do_IRQ+0xbd/0xf7 (16)
 [<c0104736>] do_IRQ+0x3e/0x4b (32)
 [<c010337e>] common_interrupt+0x1a/0x20 (12)
 [<c01c8dab>] delay_tsc+0x7/0x13 (44)
 [<c01c8de0>] __delay+0x6/0x7 (16)
 [<c022b309>] i8042_wait_write+0x19/0x3b (4)
 [<c022b43f>] i8042_command+0x8d/0x152 (8)
 [<c022b582>] i8042_aux_write+0x27/0x39 (20)
 [<c022c631>] ps2_sendbyte+0x49/0x114 (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c013ee94>] add_preempt_count+0x10/0x9b (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c022c960>] ps2_command+0xf5/0x2c9 (40)
 [<c0134e57>] rt_up+0x2c/0x42 (24)
 [<c022cb29>] ps2_command+0x2be/0x2c9 (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (24)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c02a1fbb>] intellimouse_detect+0x20/0xb2 (20)
 [<c02a205f>] im_explorer_detect+0x12/0xcd (20)
 [<c02a233c>] psmouse_extensions+0x11b/0x15b (20)
 [<c02a2945>] psmouse_switch_protocol+0xb1/0x14e (20)
 [<c02a2b30>] psmouse_connect+0x14e/0x1d2 (16)
 [<c0233ca9>] __driver_attach+0x0/0x5b (24)
 [<c022a287>] serio_connect_driver+0x1b/0x2a (4)
 [<c0233c06>] driver_probe_device+0x44/0x95 (16)
 [<c0233ce3>] __driver_attach+0x3a/0x5b (16)
 [<c023349a>] bus_for_each_dev+0x38/0x5c (16)
 [<c0233d15>] driver_attach+0x11/0x13 (36)
 [<c0233ca9>] __driver_attach+0x0/0x5b (4)
 [<c023386d>] bus_add_driver+0x58/0x8d (4)
 [<c022a708>] serio_thread+0x0/0xee (16)
 [<c022a64c>] serio_handle_event+0x5b/0x74 (4)
 [<c022a711>] serio_thread+0x9/0xee (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (32)
 [<c0312148>] schedule+0xcd/0xed (12)
 [<c012de21>] kthread+0x73/0x9b (16)
 [<c012ddae>] kthread+0x0/0x9b (16)
 [<c0101005>] kernel_thread_helper+0x5/0xb (16)
---------------------------
| preempt count: 00010003 ]
| 3-level deep critical section nesting:
----------------------------------------
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c011d64e>] .... panic+0xb/0xe3
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)

------------------------------
| showing all locks held by: |  (kseriod/294 [cd552030, 110]):
------------------------------

#001:             [c03a6224] {serio_sem.lock}
... acquired at:               rt_down+0xe/0x26

#002:             [f7d0a5e0] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#003:             [f7d0a2cc] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#004:             [f7d0a0b4] {(struct semaphore *)(&serio->drv_sem)}
... acquired at:               rt_down+0xe/0x26

#005:             [c03ba184] {psmouse_sem.lock}
... acquired at:               rt_down+0xe/0x26

#006:             [f7d88a10] {(struct semaphore *)(&ps2dev->cmd_sem)}
... acquired at:               rt_down+0xe/0x26

#007:             [c0409680] {drivers/input/serio/i8042.c:89}
... acquired at:               i8042_command+0x3c/0x152

 kseriod/294[CPU#0]: BUG in smp_call_function at arch/i386/kernel/smp.c:551
 [<c011e515>] __WARN_ON+0x41/0x54 (8)
 [<c010f4cd>] stop_this_cpu+0x0/0x29 (36)
 [<c010f423>] smp_call_function+0x4c/0xf6 (12)
 [<c011ddba>] printk+0xe/0x11 (12)
 [<c010f509>] smp_send_stop+0x13/0x1c (44)
 [<c011d697>] panic+0x54/0xe3 (8)
 [<c0103b98>] die+0x166/0x170 (20)
 [<c0103cf0>] do_invalid_op+0x0/0x8b (48)
 [<c0103d6f>] do_invalid_op+0x7f/0x8b (8)
 [<c013ee94>] add_preempt_count+0x10/0x9b (40)
 [<c0312d09>] rt_lock_slowlock+0x9b/0x195 (24)
 [<c0134e57>] rt_up+0x2c/0x42 (36)
 [<c011e0e0>] release_console_sem+0x6d/0xbf (8)
 [<c011dfd1>] vprintk+0x214/0x238 (16)
 [<c013ee94>] add_preempt_count+0x10/0x9b (4)
 [<c0134720>] debug_rt_mutex_lock+0xc0/0xc8 (28)
 [<c013350e>] try_to_take_rt_mutex+0x165/0x172 (20)
 [<c01034b3>] error_code+0x4f/0x54 (12)
 [<c0312d09>] rt_lock_slowlock+0x9b/0x195 (44)
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (28)
 [<c0313843>] _raw_spin_unlock+0xd/0x21 (72)
 [<c0313466>] rt_unlock+0x5/0x6 (4)
 [<c012530e>] __mod_timer+0xa3/0xab (4)
 [<c0313460>] __lock_text_start+0x8/0x9 (20)
 [<c022b72f>] i8042_interrupt+0x2c/0x1f0 (4)
 [<c013350e>] try_to_take_rt_mutex+0x165/0x172 (4)
 [<c014067d>] misrouted_irq+0x85/0x118 (32)
 [<c01407d4>] note_interrupt+0x43/0x9f (36)
 [<c013fbfe>] __do_IRQ+0xbd/0xf7 (16)
 [<c0104736>] do_IRQ+0x3e/0x4b (32)
 [<c010337e>] common_interrupt+0x1a/0x20 (12)
 [<c01c8dab>] delay_tsc+0x7/0x13 (44)
 [<c01c8de0>] __delay+0x6/0x7 (16)
 [<c022b309>] i8042_wait_write+0x19/0x3b (4)
 [<c022b43f>] i8042_command+0x8d/0x152 (8)
 [<c022b582>] i8042_aux_write+0x27/0x39 (20)
 [<c022c631>] ps2_sendbyte+0x49/0x114 (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c013ee94>] add_preempt_count+0x10/0x9b (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c022c960>] ps2_command+0xf5/0x2c9 (40)
 [<c0134e57>] rt_up+0x2c/0x42 (24)
 [<c022cb29>] ps2_command+0x2be/0x2c9 (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (24)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (20)
 [<c02a1fbb>] intellimouse_detect+0x20/0xb2 (20)
 [<c02a205f>] im_explorer_detect+0x12/0xcd (20)
 [<c02a233c>] psmouse_extensions+0x11b/0x15b (20)
 [<c02a2945>] psmouse_switch_protocol+0xb1/0x14e (20)
 [<c02a2b30>] psmouse_connect+0x14e/0x1d2 (16)
 [<c0233ca9>] __driver_attach+0x0/0x5b (24)
 [<c022a287>] serio_connect_driver+0x1b/0x2a (4)
 [<c0233c06>] driver_probe_device+0x44/0x95 (16)
 [<c0233ce3>] __driver_attach+0x3a/0x5b (16)
 [<c023349a>] bus_for_each_dev+0x38/0x5c (16)
 [<c0233d15>] driver_attach+0x11/0x13 (36)
 [<c0233ca9>] __driver_attach+0x0/0x5b (4)
 [<c023386d>] bus_add_driver+0x58/0x8d (4)
 [<c022a708>] serio_thread+0x0/0xee (16)
 [<c022a64c>] serio_handle_event+0x5b/0x74 (4)
 [<c022a711>] serio_thread+0x9/0xee (8)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (12)
 [<c012e1ed>] autoremove_wake_function+0x0/0x2d (32)
 [<c0312148>] schedule+0xcd/0xed (12)
 [<c012de21>] kthread+0x73/0x9b (16)
 [<c012ddae>] kthread+0x0/0x9b (16)
 [<c0101005>] kernel_thread_helper+0x5/0xb (16)
---------------------------
| preempt count: 00010005 ]
| 5-level deep critical section nesting:
----------------------------------------
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c011d64e>] .... panic+0xb/0xe3
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c0313807>] .... _raw_spin_lock+0xd/0x1d
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)
.. [<c03136e1>] .... _raw_spin_lock_irqsave+0x11/0x2e
.....[<00000000>] ..   ( <= rest_init+0x3feffd64/0x3e)

------------------------------
| showing all locks held by: |  (kseriod/294 [cd552030, 110]):
------------------------------

#001:             [c03a6224] {serio_sem.lock}
... acquired at:               rt_down+0xe/0x26

#002:             [f7d0a5e0] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#003:             [f7d0a2cc] {(struct semaphore *)(&dev->sem)}
... acquired at:               rt_down+0xe/0x26

#004:             [f7d0a0b4] {(struct semaphore *)(&serio->drv_sem)}
... acquired at:               rt_down+0xe/0x26

#005:             [c03ba184] {psmouse_sem.lock}
... acquired at:               rt_down+0xe/0x26

#006:             [f7d88a10] {(struct semaphore *)(&ps2dev->cmd_sem)}
... acquired at:               rt_down+0xe/0x26

#007:             [c0409680] {drivers/input/serio/i8042.c:89}
... acquired at:               i8042_command+0x3c/0x152