From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932617AbZCZWEX (ORCPT ); Thu, 26 Mar 2009 18:04:23 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1758479AbZCZWEO (ORCPT ); Thu, 26 Mar 2009 18:04:14 -0400 Received: from mga09.intel.com ([134.134.136.24]:42724 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757424AbZCZWEN (ORCPT ); Thu, 26 Mar 2009 18:04:13 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.38,428,1233561600"; d="scan'208";a="501154274" Date: Thu, 26 Mar 2009 15:04:11 -0700 From: Chris Leech To: linux-kernel@vger.kernel.org Subject: 2.6.29 bug with SPARSE_IRQ + NUMA_MIGRATE_IRQ_DESC Message-ID: <20090326220411.GA16286@cleech-lnx.jf.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I traced down panics in 2.6.29 to these configuration settings. It looks to me like an irq_desc is being freed while handle_edge_irq is running? This is on x86_64, 2.6.29 running on a Fedora 10 system, irqbalance is running. It's pretty easy for me to reproduce with slab and spinlock debugging turned on. It catches an irq_desc that has been cleared to POISON_FREE. All I have to do is load ixgbe on a 2 port NIC, which allocates 34 MSI-X vectors, and wait. With sparse irqs disabled I don't see any problems. - Chris BUG: spinlock bad magic on CPU#2, swapper/0 general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.1/irq CPU 2 Modules linked in: ixgbe dca netconsole configfs sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput i2c_i801 pcspkr i2c_core e1000 iTCO_wdt iTCO_vendor_support [last unloaded: microcode] Pid: 0, comm: swapper Not tainted 2.6.29-cdl-debug #13 RIP: 0010:[] [] spin_bug+0x77/0xab RSP: 0018:ffff88003e59fee8 EFLAGS: 00010002 RAX: 00000000ffffffff RBX: 6b6b6b6b6b6b6b6b RCX: ffffffff814e7779 RDX: 0000000029c429c3 RSI: 0000000000000001 RDI: 0000000000000046 RBP: ffff88003e59ff08 R08: 0000000000000002 R09: 000000006b6b6b6b R10: ffffffff814d82ee R11: 000000000000000a R12: ffff8800355282b8 R13: ffffffff814d82c0 R14: ffff8800355282b8 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88003fe72708(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fc2dfe20000 CR3: 000000007c057000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88007d196000, task ffff88003e5a0000) Stack: ffff88003e59ff28 ffff8800355282b8 0000000000000050 ffff88007d1b5b58 ffff88003e59ff28 ffffffff811a4c89 ffff8800355282b8 ffff8800355282b8 ffff88003e59ff48 ffffffff81394529 ffff880035528248 ffff880035528248 Call Trace: <0> [] _raw_spin_unlock+0x21/0x94 [] _spin_unlock+0x2b/0x2f [] handle_edge_irq+0xd1/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e <0> [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 Code: 00 48 8d 88 e8 04 00 00 31 c0 65 8b 14 25 24 00 00 00 e8 f9 ca 1e 00 83 c8 ff 48 85 db 45 8b 4c 24 08 48 c7 c1 79 77 4e 81 74 0d <8b> 83 98 02 00 00 48 8d 8b e8 04 00 00 41 8b 54 24 04 41 89 c0 RIP [] spin_bug+0x77/0xab RSP ---[ end trace 95ba74f6171957f6 ]--- Kernel panic - not syncing: Fatal exception in interrupt ------------[ cut here ]------------ WARNING: at /home/cleech/linux-2.6/kernel/smp.c:329 smp_call_function_many+0x46/0x259() Modules linked in: ixgbe dca netconsole configfs sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput i2c_i801 pcspkr i2c_core e1000 iTCO_wdt iTCO_vendor_support [last unloaded: microcode] Pid: 0, comm: swapper Tainted: G D 2.6.29-cdl-debug #13 Call Trace: [] warn_slowpath+0xb6/0xf2 [] ? delay_tsc+0x2b/0x5d [] ? _raw_spin_unlock+0x8e/0x94 [] ? down_trylock+0x14/0x39 [] ? crash_kexec+0x20/0xf4 [] ? __mutex_unlock_slowpath+0x128/0x143 [] ? trace_hardirqs_off_caller+0x1f/0xc0 [] ? trace_hardirqs_off+0xd/0xf [] ? __mutex_unlock_slowpath+0x128/0x143 [] smp_call_function_many+0x46/0x259 [] ? stop_this_cpu+0x0/0x36 [] ? trace_hardirqs_off_caller+0x1f/0xc0 [] ? trace_hardirqs_off+0xd/0xf [] ? _spin_unlock_irqrestore+0x45/0x5c [] smp_call_function+0x25/0x29 [] native_smp_send_stop+0x27/0x6f [] panic+0x89/0x138 [] ? down_trylock+0x32/0x39 [] oops_end+0xb9/0xc9 [] die+0x5a/0x63 [] do_general_protection+0x11e/0x127 [] general_protection+0x25/0x30 [] ? spin_bug+0x77/0xab [] ? spin_bug+0x63/0xab [] _raw_spin_unlock+0x21/0x94 [] _spin_unlock+0x2b/0x2f [] handle_edge_irq+0xd1/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 ---[ end trace 95ba74f6171957f7 ]--- [] handle_edge_irq+0xf2/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 handlers: general protection fault: 0000 [#2] SMP DEBUG_PAGEALLOC last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.1/irq CPU 6 Modules linked in: ixgbe dca netconsole configfs sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput i2c_i801 pcspkr i2c_core e1000 iTCO_wdt iTCO_vendor_support [last unloaded: microcode] Pid: 0, comm: swapper Tainted: G D W 2.6.29-cdl-debug #13 RIP: 0010:[] [] __report_bad_irq+0x51/0x8c RSP: 0018:ffff88007d22bef8 EFLAGS: 00010002 RAX: 000000000000000d RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000001 RDX: 0000000030af30ae RSI: ffffffff813944e7 RDI: 0000000000000046 RBP: ffff88007d22bf08 R08: 0000000000000002 R09: 0000000000000000 R10: 0000000000000096 R11: 0000000000000000 R12: ffff8800355286d8 R13: ffff88007b478190 R14: 0000000000000001 R15: 0000000000000052 FS: 0000000000000000(0000) GS:ffff88003fe73388(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fc2e366b000 CR3: 000000007c057000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88003e67c000, task ffff88003e682340) Stack: 0000000000000000 ffff8800355286d8 ffff88007d22bf48 ffffffff81099ba3 ffff88007d22bf48 ffff8800355286d8 0000000000000052 ffff88007b478190 ffff880035528748 0000000000000000 ffff88007d22bf78 ffffffff8109a16c Call Trace: <0> [] note_interrupt+0x11d/0x186 [] handle_edge_irq+0xf2/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e <0> [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 Code: eb 10 89 fe 31 c0 48 c7 c7 18 2d 4c 81 e8 e7 7c 2f 00 e8 2a 7b 2f 00 48 c7 c7 59 2d 4c 81 31 c0 e8 d4 7c 2f 00 48 8b 5b 48 eb 32 <48> 8b 33 48 c7 c7 67 2d 4c 81 31 c0 e8 bd 7c 2f 00 48 8b 33 48 RIP [] __report_bad_irq+0x51/0x8c RSP ---[ end trace 95ba74f6171957f8 ]--- Kernel panic - not syncing: Fatal exception in interrupt ------------[ cut here ]------------ WARNING: at /home/cleech/linux-2.6/kernel/smp.c:329 smp_call_function_many+0x46/0x259() Modules linked in: ixgbe dca netconsole configfs sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput i2c_i801 pcspkr i2c_core e1000 iTCO_wdt iTCO_vendor_support [last unloaded: microcode] Pid: 0, comm: swapper Tainted: G D W 2.6.29-cdl-debug #13 Call Trace: [] warn_slowpath+0xb6/0xf2 [] ? print_lock_contention_bug+0x1e/0x110 [] ? crash_kexec+0x20/0xf4 [] ? __mutex_unlock_slowpath+0x128/0x143 [] ? trace_hardirqs_off_caller+0x1f/0xc0 [] ? trace_hardirqs_off+0xd/0xf [] ? __mutex_unlock_slowpath+0x128/0x143 [] smp_call_function_many+0x46/0x259 [] ? trace_hardirqs_off_caller+0x1f/0xc0 [] ? stop_this_cpu+0x0/0x36 [] ? trace_hardirqs_off+0xd/0xf [] ? _spin_unlock_irqrestore+0x45/0x5c [] ? release_console_sem+0x1ca/0x1ff [] smp_call_function+0x25/0x29 [] native_smp_send_stop+0x27/0x6f [] panic+0x89/0x138 [] ? _spin_unlock_irqrestore+0x45/0x5c [] oops_end+0xb9/0xc9 [] die+0x5a/0x63 [] do_general_protection+0x11e/0x127 [] general_protection+0x25/0x30 [] ? _spin_unlock_irqrestore+0x45/0x5c [] ? __report_bad_irq+0x51/0x8c [] ? __report_bad_irq+0x4b/0x8c [] note_interrupt+0x11d/0x186 [] handle_edge_irq+0xf2/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 ---[ end trace 95ba74f6171957f9 ]--- ------------[ cut here ]------------ WARNING: at /home/cleech/linux-2.6/kernel/smp.c:226 smp_call_function_single+0x4c/0x143() Modules linked in: ixgbe dca netconsole configfs sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput i2c_i801 pcspkr i2c_core e1000 iTCO_wdt iTCO_vendor_support [last unloaded: microcode] Pid: 0, comm: swapper Tainted: G D W 2.6.29-cdl-debug #13 Call Trace: [] warn_slowpath+0xb6/0xf2 [] ? dump_stack+0x77/0x80 [] ? warn_slowpath+0xd0/0xf2 [] ? print_lock_contention_bug+0x1e/0x110 [] smp_call_function_single+0x4c/0x143 [] ? trace_hardirqs_off+0xd/0xf [] ? __mutex_unlock_slowpath+0x128/0x143 [] smp_call_function_many+0xc9/0x259 [] ? trace_hardirqs_off_caller+0x1f/0xc0 [] ? stop_this_cpu+0x0/0x36 [] ? trace_hardirqs_off+0xd/0xf [] ? _spin_unlock_irqrestore+0x45/0x5c [] ? release_console_sem+0x1ca/0x1ff [] smp_call_function+0x25/0x29 [] native_smp_send_stop+0x27/0x6f [] panic+0x89/0x138 [] ? _spin_unlock_irqrestore+0x45/0x5c [] oops_end+0xb9/0xc9 [] die+0x5a/0x63 [] do_general_protection+0x11e/0x127 [] general_protection+0x25/0x30 [] ? _spin_unlock_irqrestore+0x45/0x5c [] ? __report_bad_irq+0x51/0x8c [] ? __report_bad_irq+0x4b/0x8c [] note_interrupt+0x11d/0x186 [] handle_edge_irq+0xf2/0x123 [] do_IRQ+0xe1/0x15a [] ret_from_intr+0x0/0x2e [] ? mwait_idle+0x9e/0xc7 [] ? mwait_idle+0x95/0xc7 [] ? atomic_notifier_call_chain+0xf/0x11 [] ? enter_idle+0x27/0x29 [] ? cpu_idle+0x9b/0xe8 [] ? start_secondary+0x1b0/0x1b5 ---[ end trace 95ba74f6171957fa ]---