From: Petr Mladek <pmladek@suse.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@kernel.org>
Subject: Re: NMI watchdog dump does not print on hard lockup
Date: Fri, 13 Oct 2017 13:14:44 +0200 [thread overview]
Message-ID: <20171013111444.GB2795@pathway.suse.cz> (raw)
In-Reply-To: <20171012121658.187c5af6@gandalf.local.home>
On Thu 2017-10-12 12:16:58, Steven Rostedt wrote:
> static void lock_up_cpu(void *data)
> {
> unsigned long flags;
> raw_spin_lock_irqsave(&global_trace.start_lock, flags);
> raw_spin_lock(&global_trace.start_lock);
> raw_spin_unlock(&global_trace.start_lock);
> raw_spin_unlock_irqrestore(&global_trace.start_lock, flags);
> }
>
> [..]
>
> on_each_cpu(lock_up_cpu, NULL, 1);
>
> This too triggered the warning. But I noticed that the calling function
> didn't hard lockup. (Not all CPUs were hard locked).
>
> Finally I did:
>
> on_each_cpu(lock_up_cpu, NULL, 0);
> lock_up_cpu(tr);
>
> And boom! It locked up (lockdep was enabled, so I could see it showing
> the deadlock), but then it stopped there. No output. The NMI watchdog
> will only detect hard lockups if there is at least one CPU that is
> still active. This could be an issue on non SMP boxes.
>
> We need a way to have NMI flush to consoles when a lockup is detected,
> and not depend on an irq_work to do so.
I thought that enabling CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE
could help. panic() flushes the printk_save buffers, see
printk_safe_flush_on_panic(). But it somehow does not help.
I need to dig more into it.
In general, we could either improve detection of situations when
the entire system is locked. It would be a reason to risk calling
consoles even in NMI.
Or we could accept that the "default" printk is not good for all
situations and allow more special "debugging" modes:
+ Peter's force_early_printk stuff
+ Allow to disable printk_safe and printk_safe_nmi.
There will be a risk of a deadlock caused by printk.
But there also will be a chance to see the messages.
Best Regards,
Petr
next prev parent reply other threads:[~2017-10-13 11:14 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-12 16:16 Steven Rostedt
2017-10-12 19:26 ` Peter Zijlstra
2017-10-13 11:14 ` Petr Mladek [this message]
2017-10-13 13:18 ` Steven Rostedt
2017-10-13 19:12 ` Linus Torvalds
2017-10-16 11:12 ` Petr Mladek
2017-10-16 13:13 ` Sergey Senozhatsky
2017-10-16 14:15 ` Steven Rostedt
2017-10-17 7:50 ` Sergey Senozhatsky
2018-10-23 6:49 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171013111444.GB2795@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®