From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757184AbYHTT4a (ORCPT ); Wed, 20 Aug 2008 15:56:30 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756531AbYHTT4I (ORCPT ); Wed, 20 Aug 2008 15:56:08 -0400 Received: from mail-gx0-f16.google.com ([209.85.217.16]:48474 "EHLO mail-gx0-f16.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756091AbYHTT4F (ORCPT ); Wed, 20 Aug 2008 15:56:05 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=WSPUu2t2m3qDCP3/2CFVjdskmVR3WsNfvavBHBSVNFR3tgMibojccKndNMb+6/jd1Y pxZ8qb0IdEY9QSgdEPBZpLGHNlD10kN83ZJ5xezciaNhpuKpEg7925IcpamSnjkO0H3P 9coeGtL+XNerFJbCaRuJk1/YgK0cbUng2g2ck= Message-ID: <19f34abd0808201256v563c81f6x79518d6e32b188d6@mail.gmail.com> Date: Wed, 20 Aug 2008 21:56:03 +0200 From: "Vegard Nossum" To: "Pekka J Enberg" Subject: Re: [RFC][PATCH] netconsole: avoid deadlock on printk from driver code Cc: "David Miller" , vegardno@ifi.uio.no, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, jgarzik@pobox.com, adobriyan@gmail.com In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080813.033749.165504341.davem@davemloft.net> <84144f020808140628m31d6d099t1fecdb1701c0bbdc@mail.gmail.com> <20080814.151502.251134900.davem@davemloft.net> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 20, 2008 at 7:54 PM, Pekka J Enberg wrote: > Subject: [PATCH] 8139too: avoid deadlock with netconsole > From: Pekka Enberg > > As explained by Vegard Nossum, the 8139too driver can deadlock with netconsole: > > I encountered a hard-to-debug deadlock when I pulled out the plug of my > RealTek 8139 which was also running netconsole: The driver wants to print a > "link down" message. However, this triggers netconsole, which wants to print > the message using the same device. Here is a backtrace: > > [] _spin_lock_irqsave+0x76/0x90 > [] rtl8139_start_xmit+0x65/0x130 <-- spin_lock(&tp->lock) > [] netpoll_send_skb+0x158/0x1a0 > [] netpoll_send_udp+0x1db/0x1f0 > [] write_msg+0x8c/0xc0 > [] __call_console_drivers+0x53/0x60 > [] _call_console_drivers+0x4b/0x90 > [] release_console_sem+0xc5/0x1f0 > [] vprintk+0x1ab/0x3e0 > [] printk+0x1b/0x20 > [] mii_check_media+0x196/0x1e0 > [] rtl_check_media+0x24/0x30 > [] rtl8139_interrupt+0x42a/0x4a0 <-- spin_lock(&tp->lock) > [] handle_IRQ_event+0x28/0x70 > [] handle_fasteoi_irq+0x6b/0xe0 > [] do_IRQ+0x48/0xa0 > > To avoid the deadlock, use netif_tx_lock() for the TX paths and make sure we > never call printk() while holding that lock as suggested by David Miller. > > Cc: Alexey Dobriyan > Cc: David Miller > Cc: Jeff Garzik > Reported-by: Vegard Nossum > Signed-off-by: Pekka Enberg > --- I'm sorry, but it doesn't help :-) console [netcon0] enabled ============================================= [ INFO: possible recursive locking detected ] 2.6.27-rc3-00468-g86f91c5 #19 --------------------------------------------- swapper/1 is trying to acquire lock: (_xmit_ETHER#2){....}, at: [] rtl8139_start_xmit+0xc0/0x230 but task is already holding lock: (_xmit_ETHER#2){....}, at: [] netpoll_send_skb+0x13b/0x1a0 other info that might help us debug this: 3 locks held by swapper/1: #0: (target_list_lock){....}, at: [] write_msg+0x2d/0xc0 #1: (_xmit_ETHER#2){....}, at: [] netpoll_send_skb+0x13b/0x1a0 #2: (&dev->tx_global_lock){....}, at: [] rtl8139_start_xmit+0x91/0x2 30 stack backtrace: Pid: 1, comm: swapper Not tainted 2.6.27-rc3-00468-g86f91c5 #19 [] validate_chain+0xbbb/0xe90 [] ? save_stack_trace+0x2b/0x50 [] __lock_acquire+0x259/0xa00 [] lock_acquire+0x89/0xc0 [] ? rtl8139_start_xmit+0xc0/0x230 [] _spin_lock+0x40/0x70 [] ? rtl8139_start_xmit+0xc0/0x230 [] rtl8139_start_xmit+0xc0/0x230 [] ? netpoll_send_skb+0x13b/0x1a0 [] netpoll_send_skb+0x158/0x1a0 [] netpoll_send_udp+0x1db/0x1f0 [] write_msg+0x8c/0xc0 [] __call_console_drivers+0x53/0x60 [] _call_console_drivers+0x4b/0x90 [] release_console_sem+0xc5/0x1f0 [] register_console+0x138/0x2c0 [] init_netconsole+0x1a0/0x1c0 [] ? ktime_get+0x19/0x40 [] do_one_initcall+0x30/0x170 [] ? init_netconsole+0x0/0x1c0 [] ? register_irq_proc+0xb3/0xd0 [] kernel_init+0x11a/0x180 [] ? kernel_init+0x0/0x180 [] kernel_thread_helper+0x7/0x14 ======================= After that, I also got lots of kgdbts errors, don't know exactly how it relates to your patch... <3>kgdbts: BP mismatch c03763a0 expected c0139b80 <4>------------[ cut here ]------------ <4>WARNING: at /uio/arkimedes/s29/vegardno/git-working/linux-2.6/drivers/misc/kg dbts.c:302 check_and_rewind_pc+0xb2/0xe0() Pid: 1, comm: swapper Not tainted 2.6.27-rc3-00468-g86f91c5 #19 [] warn_on_slowpath+0x4f/0x80 [] ? sprint_symbol+0x92/0xc0 [] ? vprintk+0x6d/0x350 [] ? vprintk+0x6d/0x350 [] ? __change_page_attr_set_clr+0x693/0xc30 [] ? __copy_to_user_ll+0x57/0x60 [] ? probe_kernel_write+0x4a/0x90 [] ? do_fork+0x0/0x2b0 [] ? printk+0x1b/0x20 [] ? do_fork+0x0/0x2b0 [] check_and_rewind_pc+0xb2/0xe0 [] ? delay_tsc+0x20/0xb8 [] ? do_fork+0x0/0x2b0 [] validate_simple_test+0x22/0xa0 [] run_simple_test+0x107/0x260 [] ? probe_kernel_read+0x4a/0x90 [] kgdbts_put_char+0x14/0x20 [] put_packet+0x86/0xe0 [] kgdb_handle_exception+0x446/0xe40 [] ? vprintk+0x6d/0x350 [] kgdb_notify+0xa1/0x1d0 [] notifier_call_chain+0x37/0x70 [] __atomic_notifier_call_chain+0x35/0x50 [] atomic_notifier_call_chain+0x1a/0x20 [] notify_die+0x2d/0x30 [] die_nmi+0x3a/0x100 [] nmi_watchdog_tick+0x1d5/0x1e0 [] do_nmi+0x97/0x2d0 [] nmi_stack_correct+0x26/0x2b [] ? delay_tsc+0x20/0xb8 [] __delay+0x9/0x10 [] _raw_spin_lock+0xbc/0x150 [] ? get_lock_stats+0x1e/0x50 [] ? put_lock_stats+0xd/0x30 [] _spin_lock+0x5c/0x70 [] ? rtl8139_start_xmit+0xc0/0x230 [] rtl8139_start_xmit+0xc0/0x230 [] ? netpoll_send_skb+0x13b/0x1a0 [] netpoll_send_skb+0x158/0x1a0 [] netpoll_send_udp+0x1db/0x1f0 [] write_msg+0x8c/0xc0 [] __call_console_drivers+0x53/0x60 [] _call_console_drivers+0x4b/0x90 [] release_console_sem+0xc5/0x1f0 [] register_console+0x138/0x2c0 [] init_netconsole+0x1a0/0x1c0 [] ? ktime_get+0x19/0x40 [] do_one_initcall+0x30/0x170 [] ? init_netconsole+0x0/0x1c0 [] ? register_irq_proc+0xb3/0xd0 [] kernel_init+0x11a/0x180 [] ? kernel_init+0x0/0x180 [] kernel_thread_helper+0x7/0x14 ======================= <4>---[ end trace 3a158aadebbe6259 ]--- (This repeats every few seconds.) When pulling the cable, I get this: <0>BUG: spinlock lockup on CPU#1, swapper/1, f58599ac Pid: 1, comm: swapper Tainted: G W 2.6.27-rc3-00468-g86f91c5 #19 [] _raw_spin_lock+0x110/0x150 [] _spin_lock+0x5c/0x70 [] ? rtl8139_start_xmit+0xc0/0x230 [] rtl8139_start_xmit+0xc0/0x230 [] ? netpoll_send_skb+0x13b/0x1a0 [] netpoll_send_skb+0x158/0x1a0 [] netpoll_send_udp+0x1db/0x1f0 [] write_msg+0x8c/0xc0 [] __call_console_drivers+0x53/0x60 [] _call_console_drivers+0x4b/0x90 [] release_console_sem+0xc5/0x1f0 [] register_console+0x138/0x2c0 [] init_netconsole+0x1a0/0x1c0 [] ? ktime_get+0x19/0x40 [] do_one_initcall+0x30/0x170 [] ? init_netconsole+0x0/0x1c0 [] ? register_irq_proc+0xb3/0xd0 [] kernel_init+0x11a/0x180 [] ? kernel_init+0x0/0x180 [] kernel_thread_helper+0x7/0x14 ======================= ...and machine is unusable. Thanks for trying! :-D Vegard -- "The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation." -- E. W. Dijkstra, EWD1036