From: Simon Kirby <sim@hostway.ca>
To: Linus Torvalds <torvalds@linux-foundation.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: Linux 3.1-rc9
Date: Fri, 7 Oct 2011 10:48:48 -0700 [thread overview]
Message-ID: <20111007174848.GA11011@hostway.ca> (raw)
In-Reply-To: <20111007070842.GA27555@hostway.ca>
On Fri, Oct 07, 2011 at 12:08:42AM -0700, Simon Kirby wrote:
> On Tue, Oct 04, 2011 at 06:40:14PM -0700, Linus Torvalds wrote:
>
> > Peter Zijlstra (1):
> > posix-cpu-timers: Cure SMP wobbles
>
> Hello!
>
> I upgraded a few boxes from 3.1-rc6+fixes to 3.1-rc9 (actually 538d2882),
> and now they're hard locking every 15 minutes. Below is a serial console
> capture of the lockup. I suspect this is from d670ec13. I'll confirm that
> they stop crashing with that commit reverted...
Yes, they stopped locking up with d670ec13 reverted.
Simon-
> [ 1717.560007] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
> [ 1717.560007] Pid: 18034, comm: php Not tainted 3.1.0-rc9-hw+ #45
> [ 1717.560007] Call Trace:
> [ 1717.560007] <NMI> [<ffffffff816b3544>] panic+0xba/0x1fb
> [ 1717.560007] [<ffffffff81018d70>] ? native_sched_clock+0x20/0x80
> [ 1717.560007] [<ffffffff81018dd9>] ? sched_clock+0x9/0x10
> [ 1717.560007] [<ffffffff810a4751>] watchdog_overflow_callback+0xb1/0xc0
> [ 1717.560007] [<ffffffff810d0a12>] __perf_event_overflow+0xa2/0x1f0
> [ 1717.560007] [<ffffffff810c9f11>] ? perf_event_update_userpage+0x11/0xc0
> [ 1717.560007] [<ffffffff810d0f64>] perf_event_overflow+0x14/0x20
> [ 1717.560007] [<ffffffff81025a11>] intel_pmu_handle_irq+0x351/0x5f0
> [ 1717.560007] [<ffffffff816b7ff6>] perf_event_nmi_handler+0x36/0xb0
> [ 1717.560007] [<ffffffff816ba21f>] notifier_call_chain+0x3f/0x80
> [ 1717.560007] [<ffffffff816ba285>] atomic_notifier_call_chain+0x15/0x20
> [ 1717.560007] [<ffffffff816ba2be>] notify_die+0x2e/0x30
> [ 1717.560007] [<ffffffff816b76e2>] do_nmi+0xa2/0x250
> [ 1717.560007] [<ffffffff816b7080>] nmi+0x20/0x30
> [ 1717.560007] [<ffffffff8137e20d>] ? __write_lock_failed+0xd/0x20
> [ 1717.560007] <<EOE>> [<ffffffff816b6819>] _raw_write_lock_irq+0x19/0x20
> [ 1717.560007] [<ffffffff810587c3>] copy_process+0xb23/0x1270
> [ 1717.560007] [<ffffffff81058fc2>] do_fork+0xb2/0x2f0
> [ 1717.560007] [<ffffffff8101a7e3>] sys_clone+0x23/0x30
> [ 1717.560007] [<ffffffff816be533>] stub_clone+0x13/0x20
> [ 1717.560007] [<ffffffff816be292>] ? system_call_fastpath+0x16/0x1b
> [ 1717.560005] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
> [ 1717.560005] Pid: 18038, comm: httpd Not tainted 3.1.0-rc9-hw+ #45
> [ 1717.560005] Call Trace:
> [ 1717.560005] <NMI> [<ffffffff816b3544>] panic+0xba/0x1fb
> [ 1717.560005] [<ffffffff81018d70>] ? native_sched_clock+0x20/0x80
> [ 1717.560005] [<ffffffff81018dd9>] ? sched_clock+0x9/0x10
> [ 1717.560005] [<ffffffff810a4751>] watchdog_overflow_callback+0xb1/0xc0
> [ 1717.560005] [<ffffffff810d0a12>] __perf_event_overflow+0xa2/0x1f0
> [ 1717.560005] [<ffffffff810c9f11>] ? perf_event_update_userpage+0x11/0xc0
> [ 1717.560005] [<ffffffff810d0f64>] perf_event_overflow+0x14/0x20
> [ 1717.560005] [<ffffffff81025a11>] intel_pmu_handle_irq+0x351/0x5f0
> [ 1717.560005] [<ffffffff816b7ff6>] perf_event_nmi_handler+0x36/0xb0
> [ 1717.560005] [<ffffffff816ba21f>] notifier_call_chain+0x3f/0x80
> [ 1717.560005] [<ffffffff816ba285>] atomic_notifier_call_chain+0x15/0x20
> [ 1717.560005] [<ffffffff816ba2be>] notify_die+0x2e/0x30
> [ 1717.560005] [<ffffffff816b76e2>] do_nmi+0xa2/0x250
> [ 1717.560005] [<ffffffff816b7080>] nmi+0x20/0x30
> [ 1717.560005] [<ffffffff816b6644>] ? _raw_spin_lock+0x14/0x20
> [ 1717.560005] <<EOE>> [<ffffffff8104b4e5>] task_rq_lock+0x55/0xa0
> [ 1717.560005] [<ffffffff8104b8d4>] task_sched_runtime+0x24/0x90
> [ 1717.560005] [<ffffffff8107c924>] thread_group_cputime+0x74/0xb0
> [ 1717.560005] [<ffffffff8107d126>] thread_group_cputimer+0xa6/0xf0
> [ 1717.560005] [<ffffffff8107d198>] cpu_timer_sample_group+0x28/0x90
> [ 1717.560005] [<ffffffff8107d3c3>] set_process_cpu_timer+0x33/0x110
> [ 1717.560005] [<ffffffff8107d4da>] update_rlimit_cpu+0x3a/0x60
> [ 1717.560005] [<ffffffff8106fe9e>] do_prlimit+0xfe/0x1f0
> [ 1717.560005] [<ffffffff8106ffd6>] sys_setrlimit+0x46/0x60
> [ 1717.560005] [<ffffffff816be292>] system_call_fastpath+0x16/0x1b
> [ 1717.564005] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 1
> [ 1717.564005] Pid: 8, comm: migration/1 Not tainted 3.1.0-rc9-hw+ #45
> [ 1717.564005] Call Trace:
> [ 1717.564005] <NMI> [<ffffffff816b3544>] panic+0xba/0x1fb
> [ 1717.564005] [<ffffffff81018d70>] ? native_sched_clock+0x20/0x80
> [ 1717.564005] [<ffffffff81018dd9>] ? sched_clock+0x9/0x10
> [ 1717.564005] [<ffffffff810a4751>] watchdog_overflow_callback+0xb1/0xc0
> [ 1717.564005] [<ffffffff810d0a12>] __perf_event_overflow+0xa2/0x1f0
> [ 1717.564005] [<ffffffff810c9f11>] ? perf_event_update_userpage+0x11/0xc0
> [ 1717.564005] [<ffffffff810d0f64>] perf_event_overflow+0x14/0x20
> [ 1717.564005] [<ffffffff81025a11>] intel_pmu_handle_irq+0x351/0x5f0
> [ 1717.564005] [<ffffffff816b7ff6>] perf_event_nmi_handler+0x36/0xb0
> [ 1717.564005] [<ffffffff816ba21f>] notifier_call_chain+0x3f/0x80
> [ 1717.564005] [<ffffffff816ba285>] atomic_notifier_call_chain+0x15/0x20
> [ 1717.564005] [<ffffffff816ba2be>] notify_die+0x2e/0x30
> [ 1717.564005] [<ffffffff816b76e2>] do_nmi+0xa2/0x250
> [ 1717.564005] [<ffffffff816b7080>] nmi+0x20/0x30
> [ 1717.564005] [<ffffffff816b6640>] ? _raw_spin_lock+0x10/0x20
> [ 1717.564005] <<EOE>> [<ffffffff81048cfd>] double_rq_lock+0x4d/0x60
> [ 1717.564005] [<ffffffff8104fee8>] __migrate_task+0x78/0x120
> [ 1717.564005] [<ffffffff8104ff90>] ? __migrate_task+0x120/0x120
> [ 1717.564005] [<ffffffff8104ffae>] migration_cpu_stop+0x1e/0x30
> [ 1717.564005] [<ffffffff810a370c>] cpu_stopper_thread+0xcc/0x190
> [ 1717.564005] [<ffffffff8105049d>] ? default_wake_function+0xd/0x10
> [ 1717.564005] [<ffffffff81043e0a>] ? __wake_up_common+0x5a/0x90
> [ 1717.564005] [<ffffffff810a3640>] ? cgroup_release_agent+0x1d0/0x1d0
> [ 1717.564005] [<ffffffff810a3640>] ? cgroup_release_agent+0x1d0/0x1d0
> [ 1717.564005] [<ffffffff8107adb6>] kthread+0x96/0xb0
> [ 1717.564005] [<ffffffff816c0374>] kernel_thread_helper+0x4/0x10
> [ 1717.564005] [<ffffffff8107ad20>] ? kthread_worker_fn+0x190/0x190
> [ 1717.564005] [<ffffffff816c0370>] ? gs_change+0x13/0x13
> [ 1717.560007] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
> [ 1717.560007] Pid: 15190, comm: httpd Not tainted 3.1.0-rc9-hw+ #45
> [ 1717.560007] Call Trace:
> [ 1717.560007] <NMI> [<ffffffff816b3544>] panic+0xba/0x1fb
> [ 1717.560007] [<ffffffff81018d70>] ? native_sched_clock+0x20/0x80
> [ 1717.560007] [<ffffffff81018dd9>] ? sched_clock+0x9/0x10
> [ 1717.560007] [<ffffffff810a4751>] watchdog_overflow_callback+0xb1/0xc0
> [ 1717.560007] [<ffffffff810d0a12>] __perf_event_overflow+0xa2/0x1f0
> [ 1717.560007] [<ffffffff810c9f11>] ? perf_event_update_userpage+0x11/0xc0
> [ 1717.560007] [<ffffffff810d0f64>] perf_event_overflow+0x14/0x20
> [ 1717.560007] [<ffffffff81025a11>] intel_pmu_handle_irq+0x351/0x5f0
> [ 1717.560007] [<ffffffff816b7ff6>] perf_event_nmi_handler+0x36/0xb0
> [ 1717.560007] [<ffffffff816ba21f>] notifier_call_chain+0x3f/0x80
> [ 1717.560007] [<ffffffff816ba285>] atomic_notifier_call_chain+0x15/0x20
> [ 1717.560007] [<ffffffff816ba2be>] notify_die+0x2e/0x30
> [ 1717.560007] [<ffffffff816b76e2>] do_nmi+0xa2/0x250
> [ 1717.560007] [<ffffffff816b7080>] nmi+0x20/0x30
> [ 1717.560007] [<ffffffff816b6644>] ? _raw_spin_lock+0x14/0x20
> [ 1717.560007] <<EOE>> [<ffffffff81048064>] update_curr+0x174/0x1a0
> [ 1717.560007] [<ffffffff8104c75c>] enqueue_task_fair+0x5c/0x520
> [ 1717.560007] [<ffffffff81048ea1>] enqueue_task+0x61/0x70
> [ 1717.560007] [<ffffffff81048ed9>] activate_task+0x29/0x40
> [ 1717.560007] [<ffffffff81050589>] wake_up_new_task+0xb9/0x160
> [ 1717.560007] [<ffffffff81059056>] do_fork+0x146/0x2f0
> [ 1717.560007] [<ffffffff81114d80>] ? fd_install+0x30/0x60
> [ 1717.560007] [<ffffffff8101a7e3>] sys_clone+0x23/0x30
> [ 1717.560007] [<ffffffff816be533>] stub_clone+0x13/0x20
> [ 1717.560007] [<ffffffff816be292>] ? system_call_fastpath+0x16/0x1b
>
> Config: http://0x.ca/sim/ref/3.1-rc9/config
next prev parent reply other threads:[~2011-10-07 17:48 UTC|newest]
Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-05 1:40 Linus Torvalds
2011-10-07 7:08 ` Simon Kirby
2011-10-07 17:48 ` Simon Kirby [this message]
2011-10-07 18:01 ` Peter Zijlstra
2011-10-08 0:33 ` Simon Kirby
2011-10-08 0:50 ` Simon Kirby
2011-10-08 7:55 ` Peter Zijlstra
2011-10-12 21:35 ` Simon Kirby
2011-10-13 23:25 ` Simon Kirby
2011-10-17 1:39 ` Linus Torvalds
2011-10-17 4:58 ` Ingo Molnar
2011-10-17 9:03 ` Thomas Gleixner
2011-10-17 10:40 ` Peter Zijlstra
2011-10-17 11:40 ` Alan Cox
2011-10-17 18:49 ` Ingo Molnar
2011-10-17 20:35 ` H. Peter Anvin
2011-10-17 21:19 ` Ingo Molnar
2011-10-17 21:22 ` H. Peter Anvin
2011-10-17 21:39 ` Ingo Molnar
2011-10-17 22:03 ` Ingo Molnar
2011-10-17 22:04 ` Ingo Molnar
2011-10-17 22:08 ` H. Peter Anvin
2011-10-18 6:01 ` Ingo Molnar
2011-10-18 7:12 ` Geert Uytterhoeven
2011-10-18 18:50 ` H. Peter Anvin
2011-10-17 21:31 ` Ingo Molnar
2011-10-17 7:55 ` Martin Schwidefsky
2011-10-17 9:12 ` Peter Zijlstra
2011-10-17 9:18 ` Martin Schwidefsky
2011-10-17 20:48 ` H. Peter Anvin
2011-10-18 7:20 ` Martin Schwidefsky
2011-10-17 10:34 ` Peter Zijlstra
2011-10-17 14:07 ` Martin Schwidefsky
2011-10-17 14:57 ` Linus Torvalds
2011-10-17 17:54 ` Peter Zijlstra
2011-10-17 18:31 ` Linus Torvalds
2011-10-17 19:23 ` Peter Zijlstra
2011-10-17 21:00 ` Thomas Gleixner
2011-10-18 8:39 ` Thomas Gleixner
2011-10-18 9:05 ` Peter Zijlstra
2011-10-18 14:59 ` Linus Torvalds
2011-10-18 15:26 ` Thomas Gleixner
2011-10-18 18:07 ` Ingo Molnar
2011-10-18 18:14 ` [GIT PULL] timer fix Ingo Molnar
2011-10-18 16:13 ` Linux 3.1-rc9 Dave Jones
2011-10-18 18:20 ` Simon Kirby
2011-10-18 19:48 ` Thomas Gleixner
2011-10-18 20:12 ` Linus Torvalds
2011-10-25 15:26 ` Simon Kirby
2011-10-26 1:47 ` Yong Zhang
2011-10-24 19:02 ` Simon Kirby
2011-10-25 7:13 ` Linus Torvalds
2011-10-25 9:01 ` David Miller
2011-10-25 12:30 ` Thomas Gleixner
2011-10-25 23:18 ` David Miller
2011-10-25 20:20 ` Simon Kirby
2011-10-31 17:32 ` Simon Kirby
2011-11-02 16:40 ` Thomas Gleixner
2011-11-02 17:27 ` Eric Dumazet
2011-11-02 17:46 ` Linus Torvalds
2011-11-02 17:53 ` Eric Dumazet
2011-11-02 18:00 ` Linus Torvalds
2011-11-02 18:05 ` Eric Dumazet
2011-11-02 18:10 ` Linus Torvalds
2011-11-02 17:49 ` Eric Dumazet
2011-11-02 17:58 ` Eric Dumazet
2011-11-02 19:16 ` Simon Kirby
2011-11-02 22:42 ` Eric Dumazet
2011-11-03 0:24 ` Thomas Gleixner
2011-11-03 0:52 ` Simon Kirby
2011-11-03 22:07 ` David Miller
2011-11-03 6:06 ` Jörg-Volker Peetz
2011-11-03 6:26 ` Eric Dumazet
2011-11-03 6:43 ` David Miller
2011-11-02 17:54 ` Thomas Gleixner
2011-11-02 18:04 ` Eric Dumazet
2011-11-02 18:28 ` Simon Kirby
2011-11-02 18:30 ` Thomas Gleixner
2011-11-02 22:10 ` Steven Rostedt
2011-11-02 23:00 ` Steven Rostedt
2011-11-03 0:09 ` Simon Kirby
2011-11-03 0:15 ` Steven Rostedt
2011-11-03 0:17 ` Simon Kirby
2011-11-18 23:11 ` [tip:perf/core] lockdep: Show subclass in pretty print of lockdep output tip-bot for Steven Rostedt
2011-10-20 14:36 ` Linux 3.1-rc9 Martin Schwidefsky
2011-10-23 11:34 ` Ingo Molnar
2011-10-24 7:48 ` Martin Schwidefsky
2011-10-24 7:51 ` Linus Torvalds
2011-10-24 8:08 ` Martin Schwidefsky
2011-10-18 5:40 ` Simon Kirby
2011-10-09 20:51 ` Arkadiusz Miśkiewicz
2011-10-10 2:29 ` [tpmdd-devel] " Stefan Berger
2011-10-10 16:23 ` Rajiv Andrade
2011-10-10 17:05 ` Arkadiusz Miśkiewicz
2011-10-10 17:22 ` Stefan Berger
2011-10-10 17:57 ` Arkadiusz Miśkiewicz
2011-10-10 21:08 ` Arkadiusz Miśkiewicz
2011-10-11 7:09 ` [tpmdd-devel] " Peter.Huewe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111007174848.GA11011@hostway.ca \
--to=sim@hostway.ca \
--cc=a.p.zijlstra@chello.nl \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome