Re: PATCH/RFC: [kdump] fix APIC shutdown sequence

[Vivek Goyal <vgoyal@in.ibm.com>]

On Mon, Aug 06, 2007 at 05:08:05PM +0200, Martin Wilck wrote:
> PATCH/RFC: [kdump] fix APIC shutdown sequence
> 
> This patch fixes a problem that we have encountered
> with kdump under high I/O load on some machines.
> The machines showing the errors have an Intel ICH7
> chip set with a 6702PXH  PCI Express-to-PCI Bridge
> (8086:032c) containing an IO-APIC.
> 
> The bug symptom is that certain controllers connected
> to the 6702PXH bridge wouldn't receive any IRQs in the
> kdump kernel. In the error case (which is about 20% of
> all cases) the IRR bit of the IO-APIC pin for that
> controller is always set after the start of the kdump
> kernel, indicating an IRQ in progress. We haven't found
> a way to recover from this situation when it has once
> occured, except for a system reset.
> 
> The error is caused by IRQs arriving while the APIC
> subsystem is deactivated in machine_crash_shutdown().
> 
> Apparently, the IO-APIC gets stuck if it sends an IRQ
> message to a Local APIC and never receives an EOI for that
> message. This can have several possible reasons:
> 

Got this oops while testing your patch when I did
"echo c > /proc/sysrq-trigger"

[root@llm37 ~]# SysRq : Trigger a crashdump
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
 [<0000000000000000>]
PGD 22229b067 PUD 0
Oops: 0000 [1] SMP
CPU 0
Modules linked in:
Pid: 2947, comm: klogd Not tainted 2.6.23-rc1-apic-issue #2
RIP: 0010:[<0000000000000000>]  [<0000000000000000>]
RSP: 0018:ffffffff807daf50  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffffffff8073f480 RCX: ffffffff807ddea0
RDX: ffffffff807717c0 RSI: ffffffff8073f480 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff807ddd28 R11: 0000000000000150 R12: 0000000000000000
R13: ffffffff8073f4d0 R14: 000000000000000c R15: 0000000000000000
FS:  00002b204ea686f0(0000) GS:ffffffff8073c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000223e9e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process klogd (pid: 2947, threadinfo ffff81021b7b8000, task ffff8102251a27b0)
Stack:  ffffffff8025afd0 0000000000000046 ffffffff807dddf8 0000000000000000
 0000000000000030 0000000000000000 ffffffff8020e1ae ffffffff807d15c8
 0000000000000000 ffffffff807dde20 0000000000000000 ffffffff807ddee8
Call Trace:
 <IRQ>  [<ffffffff8025afd0>] handle_edge_irq+0x5c/0x127
 [<ffffffff8020e1ae>] do_IRQ+0xf1/0x15f
 [<ffffffff8020c191>] ret_from_intr+0x0/0xa
 <EOI>  <NMI>  [<ffffffff80351ae3>] __delay+0x6/0x10
 [<ffffffff8021e328>] crash_nmi_callback+0x4b/0x77
 [<ffffffff80557075>] notifier_call_chain+0x29/0x4c
 [<ffffffff80249a11>] notify_die+0x2d/0x34
 [<ffffffff805555a3>] default_do_nmi+0x55/0x197
 [<ffffffff80555f0e>] do_nmi+0x2e/0x44
 [<ffffffff805553af>] nmi+0x7f/0x90
 [<ffffffff8022e00a>] find_busiest_group+0x20e/0x6b3
 <<EOE>>  [<ffffffff80552d13>] __sched_text_start+0x1cb/0x5ba
 [<ffffffff80282294>] do_sync_write+0xc9/0x10c
 [<ffffffff80235c32>] do_syslog+0x11d/0x3a9
 [<ffffffff80246a03>] autoremove_wake_function+0x0/0x2e
 [<ffffffff802731c6>] free_pages_and_swap_cache+0x73/0x8f
 [<ffffffff802bbeac>] kmsg_read+0x3a/0x44
 [<ffffffff802b5245>] proc_reg_read+0x7e/0x99
 [<ffffffff80282b08>] vfs_read+0xaa/0x132
 [<ffffffff80282ea4>] sys_read+0x45/0x6e
 [<ffffffff8020bc7e>] system_call+0x7e/0x83


Code:  Bad RIP value.
RIP  [<0000000000000000>]
 RSP <ffffffff807daf50>
CR2: 0000000000000000

Thanks
Vivek