mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Petr Mladek <pmladek@suse.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>
Subject: Re: NMI watchdog dump does not print on hard lockup
Date: Fri, 13 Oct 2017 13:14:44 +0200	[thread overview]
Message-ID: <20171013111444.GB2795@pathway.suse.cz> (raw)
In-Reply-To: <20171012121658.187c5af6@gandalf.local.home>

On Thu 2017-10-12 12:16:58, Steven Rostedt wrote:
> static void lock_up_cpu(void *data)
> {
> 	unsigned long flags;
> 	raw_spin_lock_irqsave(&global_trace.start_lock, flags);
> 	raw_spin_lock(&global_trace.start_lock);
> 	raw_spin_unlock(&global_trace.start_lock);
> 	raw_spin_unlock_irqrestore(&global_trace.start_lock, flags);
> }
> 
> [..]
> 
> 	on_each_cpu(lock_up_cpu, NULL, 1);
> 
> This too triggered the warning. But I noticed that the calling function
> didn't hard lockup. (Not all CPUs were hard locked).
> 
> Finally I did:
> 
> 	on_each_cpu(lock_up_cpu, NULL, 0);
> 	lock_up_cpu(tr);
> 
> And boom! It locked up (lockdep was enabled, so I could see it showing
> the deadlock), but then it stopped there. No output. The NMI watchdog
> will only detect hard lockups if there is at least one CPU that is
> still active. This could be an issue on non SMP boxes.
> 
> We need a way to have NMI flush to consoles when a lockup is detected,
> and not depend on an irq_work to do so.


I thought that enabling CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE
could help. panic() flushes the printk_save buffers, see
printk_safe_flush_on_panic(). But it somehow does not help.
I need to dig more into it.

In general, we could either improve detection of situations when
the entire system is locked. It would be a reason to risk calling
consoles even in NMI.

Or we could accept that the "default" printk is not good for all
situations and allow more special "debugging" modes:

	   + Peter's force_early_printk stuff

	   + Allow to disable printk_safe and printk_safe_nmi.
	     There will be a risk of a deadlock caused by printk.
	     But there also will be a chance to see the messages.


Best Regards,
Petr

  parent reply	other threads:[~2017-10-13 11:14 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-12 16:16 Steven Rostedt
2017-10-12 19:26 ` Peter Zijlstra
2017-10-13 11:14 ` Petr Mladek [this message]
2017-10-13 13:18   ` Steven Rostedt
2017-10-13 19:12     ` Linus Torvalds
2017-10-16 11:12       ` Petr Mladek
2017-10-16 13:13         ` Sergey Senozhatsky
2017-10-16 14:15           ` Steven Rostedt
2017-10-17  7:50             ` Sergey Senozhatsky
2018-10-23  6:49             ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20171013111444.GB2795@pathway.suse.cz \
    --to=pmladek@suse.com \
    --cc=akpm@linux-foundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=tglx@linutronix.de \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®