From: Valdis.Kletnieks@vt.edu
To: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org
Subject: 2.6.22-rc3-mm1 - pppd hanging in netdev_run_todo while holding mutex
Date: Mon, 04 Jun 2007 14:00:56 -0400 [thread overview]
Message-ID: <11184.1180980056@turing-police.cc.vt.edu> (raw)
In-Reply-To: Your message of "Wed, 30 May 2007 23:58:23 PDT." <20070530235823.793f00d9.akpm@linux-foundation.org>
[-- Attachment #1: Type: text/plain, Size: 4943 bytes --]
On Wed, 30 May 2007 23:58:23 PDT, Andrew Morton said:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.22-rc3/2.6.22-rc3-mm1/
Under 22-rc2-mm1, if my VPN connection got reset, ppp0 just quietly went away.
Under 22-rc3-mm1, it seems to end up wedged and waiting for references to
go away:
Jun 4 09:23:01 turing-police kernel: [90089.270707] unregister_netdevice: waiting for ppp0 to become free. Usage count = 8
Jun 4 09:23:11 turing-police kernel: [90099.396121] unregister_netdevice: waiting for ppp0 to become free. Usage count = 8
Jun 4 09:23:21 turing-police kernel: [90109.520574] unregister_netdevice: waiting for ppp0 to become free. Usage count = 8
Jun 4 09:23:32 turing-police kernel: [90119.653129] unregister_netdevice: waiting for ppp0 to become free. Usage count = 8
'echo t > /proc/sysrq_trigger' shows pppd hung up here:
Jun 4 10:52:57 turing-police kernel: [95478.047892] pppd D 0000000105ad3830 4968 3815 1 (NOTLB)
Jun 4 10:52:57 turing-police kernel: [95478.047902] ffff810008d5fd78 0000000000000086 0000000000000000 ffff810003490000
Jun 4 10:52:57 turing-police kernel: [95478.047911] ffff810008d5fd28 ffff810008d4a040 ffff810003461820 ffff810008d4a2b0
Jun 4 10:52:57 turing-police kernel: [95478.047920] 0000000105ad3733 0000000000000202 00000000000000ff ffffffff80239795
Jun 4 10:52:57 turing-police kernel: [95478.047928] Call Trace:
Jun 4 10:52:57 turing-police kernel: [95478.047936] [<ffffffff805207a2>] schedule_timeout+0x8d/0xb4
Jun 4 10:52:57 turing-police kernel: [95478.047945] [<ffffffff805207e2>] schedule_timeout_uninterruptible+0x19/0x1b
Jun 4 10:52:57 turing-police kernel: [95478.047954] [<ffffffff802397bb>] msleep+0x14/0x1e
Jun 4 10:52:57 turing-police kernel: [95478.047963] [<ffffffff8048aa4e>] netdev_run_todo+0x12f/0x234
Jun 4 10:52:57 turing-police kernel: [95478.047972] [<ffffffff8049166f>] rtnl_unlock+0x35/0x37
Jun 4 10:52:57 turing-police kernel: [95478.047981] [<ffffffff804894a9>] unregister_netdev+0x1e/0x23
Jun 4 10:52:57 turing-police kernel: [95478.047994] [<ffffffff88a5f2c2>] :ppp_generic:ppp_shutdown_interface+0x67/0xbb
Jun 4 10:52:57 turing-police kernel: [95478.048018] [<ffffffff88a5f5b8>] :ppp_generic:ppp_release+0x33/0x65
Jun 4 10:52:57 turing-police kernel: [95478.048028] [<ffffffff8028d54a>] __fput+0xac/0x176
Jun 4 10:52:57 turing-police kernel: [95478.048036] [<ffffffff8028d628>] fput+0x14/0x16
Jun 4 10:52:57 turing-police kernel: [95478.048045] [<ffffffff8028a9c6>] filp_close+0x66/0x71
Jun 4 10:52:57 turing-police kernel: [95478.048054] [<ffffffff8028bd54>] sys_close+0x98/0xd7
Jun 4 10:52:57 turing-police kernel: [95478.048062] [<ffffffff8020a03c>] tracesys+0xdc/0xe1
Jun 4 10:52:57 turing-police kernel: [95478.048073] [<00002b45cd2429a0>]
Which in itself wouldn't be so bad, except that it's holding a mutex and
lots of other stuff gets wedged up waiting for it (here's 1 of 6 processes
that was wedged this morning):
Jun 4 10:52:58 turing-police kernel: [95478.051129] ifconfig D ffff810005e19820 5800 9787 20510 (NOTLB)
Jun 4 10:52:58 turing-police kernel: [95478.051141] ffff81000868fd08 0000000000000082 ffff81000868fec8 0000000000000246
Jun 4 10:52:58 turing-police kernel: [95478.051150] 0000000100000101 ffff810005e19820 ffff810003fe0820 ffff810005e19a90
Jun 4 10:52:58 turing-police kernel: [95478.051159] 000000000a3f26c0 0000000000000006 ffff81000868ff28 ffffffff8028aacc
Jun 4 10:52:58 turing-police kernel: [95478.051167] Call Trace:
Jun 4 10:52:58 turing-police kernel: [95478.051176] [<ffffffff80520bc4>] __mutex_lock_slowpath+0x74/0xb6
Jun 4 10:52:58 turing-police kernel: [95478.051185] [<ffffffff805209f3>] mutex_lock+0xe/0x10
Jun 4 10:52:58 turing-police kernel: [95478.051193] [<ffffffff8048a938>] netdev_run_todo+0x19/0x234
Jun 4 10:52:58 turing-police kernel: [95478.051202] [<ffffffff8049166f>] rtnl_unlock+0x35/0x37
Jun 4 10:52:58 turing-police kernel: [95478.051210] [<ffffffff8048a3f2>] dev_ioctl+0x3e3/0x483
Jun 4 10:52:58 turing-police kernel: [95478.051218] [<ffffffff8047df30>] sock_ioctl+0x1ef/0x1fc
Jun 4 10:52:58 turing-police kernel: [95478.051227] [<ffffffff802989be>] do_ioctl+0x2a/0x77
Jun 4 10:52:58 turing-police kernel: [95478.051235] [<ffffffff80298c52>] vfs_ioctl+0x247/0x264
Jun 4 10:52:58 turing-police kernel: [95478.051243] [<ffffffff80298cce>] sys_ioctl+0x5f/0x85
Jun 4 10:52:58 turing-police kernel: [95478.051252] [<ffffffff8020a03c>] tracesys+0xdc/0xe1
(And of course, you can't shutdown cleanly, because /etc/init.d/network tries
to down other interfaces on the way out, and....)
I'd bisect this, except I don't have a better way to replicate it than "wait for
our VPN box to reset the connection after 24 hours of connect" - basically means
I get 2 tries per weekend..)
An hour or so of digging through the -rc3-mm1 broken-out/ didn't find any
obvious-to-me culprits. Any ideas/suggestions?
[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]
next prev parent reply other threads:[~2007-06-04 18:01 UTC|newest]
Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-05-31 6:58 2.6.22-rc3-mm1 Andrew Morton
2007-05-31 12:09 ` 2.6.22-rc3-mm1 Cornelia Huck
2007-05-31 12:15 ` 2.6.22-rc3-mm1 Matthew Wilcox
2007-05-31 12:20 ` 2.6.22-rc3-mm1 Cornelia Huck
2007-05-31 12:35 ` 2.6.22-rc3-mm1 Jeff Garzik
2007-05-31 15:11 ` 2.6.22-rc3-mm1 Cornelia Huck
2007-05-31 15:13 ` 2.6.22-rc3-mm1 Christoph Hellwig
2007-05-31 22:10 ` 2.6.22-rc3-mm1 Andrew Morton
2007-06-01 7:09 ` 2.6.22-rc3-mm1 Cornelia Huck
2007-05-31 15:29 ` 2.6.22-rc3-mm1 Michal Piotrowski
2007-05-31 19:58 ` 2.6.22-rc3-mm1 Rafael J. Wysocki
2007-05-31 21:30 ` [PATCH -mm] Freezer: migration_thread should not be freezable (was: Re: [linux-pm] Re: 2.6.22-rc3-mm1) Rafael J. Wysocki
[not found] ` <6bffcb0e0705311425q2e00cc69j5006d95c3f0df158@mail.gmail.com>
2007-05-31 21:34 ` 2.6.22-rc3-mm1 Rafael J. Wysocki
2007-05-31 17:53 ` 2.6.22-rc3-mm1 Michal Piotrowski
2007-05-31 18:08 ` 2.6.22-rc3-mm1 Michal Piotrowski
2007-05-31 18:31 ` 2.6.22-rc3-mm1 Andrew Morton
2007-05-31 18:41 ` 2.6.22-rc3-mm1 Christoph Lameter
2007-05-31 18:53 ` 2.6.22-rc3-mm1 Andrew Morton
2007-05-31 18:57 ` 2.6.22-rc3-mm1 Christoph Lameter
2007-05-31 20:43 ` 2.6.22-rc3-mm1 Mariusz Kozlowski
2007-05-31 21:19 ` 2.6.22-rc3-mm1 Andrew Morton
2007-06-01 20:50 ` 2.6.22-rc3-mm1 Mariusz Kozlowski
2007-06-01 21:02 ` 2.6.22-rc3-mm1 Andrew Morton
2007-06-01 21:21 ` 2.6.22-rc3-mm1 Mariusz Kozlowski
2007-06-01 23:30 ` 2.6.22-rc3-mm1 Benjamin Herrenschmidt
2007-06-02 8:40 ` 2.6.22-rc3-mm1 Segher Boessenkool
2007-05-31 22:05 ` 2.6.22-rc3-mm1 - works-for-me and a meta-question Valdis.Kletnieks
2007-05-31 22:16 ` Andrew Morton
[not found] ` <6bffcb0e0705311527ge774f48s8014f5108556f651@mail.gmail.com>
2007-05-31 23:21 ` andrea
2007-05-31 23:13 ` 2.6.22-rc3-mm1 - page_mkwrite() breakage Mark Fasheh
2007-06-01 1:01 ` Nick Piggin
2007-06-01 1:24 ` Mark Fasheh
2007-06-01 1:34 ` Nick Piggin
2007-06-01 1:45 ` Mark Fasheh
2007-06-01 1:53 ` Nick Piggin
2007-06-01 5:20 ` Mark Fasheh
2007-06-01 22:01 ` Mark Fasheh
2007-06-01 22:25 ` Andrew Morton
2007-06-01 22:33 ` Mark Fasheh
2007-06-01 22:47 ` Andrew Morton
2007-06-01 22:53 ` Mark Fasheh
2007-06-01 2:01 ` 2.6.22-rc3-mm1 Arnaldo Carvalho de Melo
2007-06-01 2:12 ` 2.6.22-rc3-mm1 Andrew Morton
2007-06-01 2:24 ` 2.6.22-rc3-mm1 Arnaldo Carvalho de Melo
2007-06-01 3:52 ` 2.6.22-rc3-mm1 Michael Ellerman
2007-06-01 5:55 ` 2.6.22-rc3-mm1 Eric W. Biederman
2007-06-01 16:42 ` 2.6.22-rc3-mm1: IA64 make allnoconfig broken Mel Gorman
2007-06-01 17:00 ` Andrew Morton
2007-06-01 18:50 ` Mel Gorman
2007-06-01 20:55 ` [PATCH] Fix problem with IA64 make allnoconfig Mel Gorman
2007-06-02 13:57 ` 2.6.22-rc3-mm1: Xen compile error with X86_CMPXCHG=n Adrian Bunk
2007-06-28 23:36 ` Adrian Bunk
2007-06-29 3:21 ` Jeremy Fitzhardinge
2007-06-02 17:06 ` [-mm patch] fix XEN_BLKDEV_FRONTEND Makefile entry Adrian Bunk
2007-06-02 17:14 ` 2.6.22-rc3-mm1: __attribute__((weak)) considered harmful Adrian Bunk
2007-06-04 21:22 ` Andrew Morton
2007-06-04 23:52 ` Martin Peschke
2007-06-05 3:59 ` Russell King
2007-06-02 19:09 ` [-mm patch] the ASYNC_* options shouldn't be user visible Adrian Bunk
2007-06-04 16:19 ` Williams, Dan J
2007-06-03 20:54 ` [-mm patch] make drivers/firmware/dmi-id.c:dmi_id_init() static Adrian Bunk
2007-06-08 4:38 ` Greg KH
2007-06-03 20:54 ` [-mm patch] drivers/i2c/chips/ds1682.c: make code static Adrian Bunk
2007-06-04 8:15 ` Jean Delvare
2007-06-03 20:54 ` [-mm patch] make xpad_play_effect() static Adrian Bunk
2007-06-03 20:54 ` [-mm patch] fix the tea5761 tuner support Adrian Bunk
2007-06-04 18:00 ` Valdis.Kletnieks [this message]
2007-06-06 6:14 ` 2.6.22-rc3-mm1 - pppd hanging in netdev_run_todo while holding mutex Andrew Morton
2007-06-04 22:12 ` [-mm patch] e1000: #if 0 two functions Adrian Bunk
2007-06-04 22:13 ` [-mm patch] drivers/mmc/core/core.{h,c}: cleanups Adrian Bunk
2007-06-06 18:36 ` Pierre Ossman
2007-06-05 9:11 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:18 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:24 ` 2.6.22-rc3-mm1 Andrew Morton
2007-06-05 9:33 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:39 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:42 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:45 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:50 ` 2.6.22-rc3-mm1 Ingo Molnar
2007-06-05 9:56 ` 2.6.22-rc3 nmi watchdog hang Ingo Molnar
2007-06-10 18:10 ` Björn Steinbrink
2007-06-18 12:11 ` Ingo Molnar
2007-06-18 14:31 ` Björn Steinbrink
2007-06-25 6:18 ` Ingo Molnar
2007-06-25 6:59 ` Ingo Molnar
2007-06-25 8:05 ` [patch, 2.6.22-rc6] fix nmi_watchdog=2 bootup hang Ingo Molnar
2007-06-25 8:26 ` Ingo Molnar
2007-06-25 12:45 ` Björn Steinbrink
2007-06-25 12:49 ` Jeremy Fitzhardinge
2007-06-25 13:06 ` Björn Steinbrink
2007-06-25 18:50 ` [patch, 2.6.22-rc6] fix nmi_watchdog=2 bootup hang, take #2 Ingo Molnar
2007-06-25 12:40 ` [patch, 2.6.22-rc6] fix nmi_watchdog=2 bootup hang Jeremy Fitzhardinge
2007-06-25 13:13 ` Björn Steinbrink
2007-06-05 15:16 ` 2.6.22-rc3-mm1 Rusty Russell
2007-06-05 21:50 ` [-mm patch] kernel/power/disk.c: make code static Adrian Bunk
2007-06-05 22:10 ` Rafael J. Wysocki
2007-06-05 21:50 ` [-mm patch] kernel/sched{,_fair}.c: " Adrian Bunk
2007-06-06 6:54 ` Andrew Morton
2007-06-06 7:30 ` Ingo Molnar
2007-06-06 12:31 ` Adrian Bunk
2007-06-06 7:02 ` Ingo Molnar
2007-06-05 21:51 ` [-mm patch] kernel/lockdep_proc.c: make 2 functions static Adrian Bunk
2007-06-05 22:34 ` Peter Zijlstra
2007-06-05 21:51 ` [-mm patch] fs/proc/base.c: make a struct static Adrian Bunk
2007-06-06 7:32 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=11184.1180980056@turing-police.cc.vt.edu \
--to=valdis.kletnieks@vt.edu \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®