[PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Petr Mladek]

NBCON consoles emit messages in a dedicated kthreads when the system
is working properly. printk() tries to flush them synchronously in
explicitly marked emergency context and in panic().

Another situation where printk() could not rely on kthreads are the various
reboot and halt code paths. They can be detected by the `system_state`
variable.

Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
printk() will automatically try flushing the consoles synchronously.
Also do not rely on printk() and explicitly flush the consoles
after these states are set.

Note that suspend code paths have already been handled, see
console_suspend_all() and console_suspend().

Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/all/202608061008.48a1e76e-lkp@intel.com/
Suggested-by: John Ogness <john.ogness@linutronix.de>
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
Hi,

ah, this somehow fallen through cracks.
I am sending it as a proper patch finally.

I am sorry I have been somehow too busy before
the Plumbers conference.

Best Regards,
Petr
---
 kernel/printk/nbcon.c | 4 ++++
 kernel/reboot.c       | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index fb1b37741952..d8f8ec836eea 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1446,6 +1446,10 @@ enum nbcon_prio nbcon_get_default_prio(void)
 	if (panic_on_this_cpu())
 		return NBCON_PRIO_PANIC;
 
+	/* Do not rely on kthreads when the system is going down. */
+	if (system_state > SYSTEM_RUNNING)
+		return NBCON_PRIO_EMERGENCY;
+
 	cpu_emergency_nesting = nbcon_get_cpu_emergency_nesting();
 	if (*cpu_emergency_nesting)
 		return NBCON_PRIO_EMERGENCY;
diff --git a/kernel/reboot.c b/kernel/reboot.c
index d177d89fcc33..776784a82499 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -8,6 +8,7 @@
 #define pr_fmt(fmt)	"reboot: " fmt
 
 #include <linux/atomic.h>
+#include <linux/console.h>
 #include <linux/ctype.h>
 #include <linux/export.h>
 #include <linux/kexec.h>
@@ -94,6 +95,7 @@ void emergency_restart(void)
 {
 	kmsg_dump(KMSG_DUMP_EMERG);
 	system_state = SYSTEM_RESTART;
+	printk_trigger_flush();
 	machine_emergency_restart();
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
@@ -102,6 +104,7 @@ void kernel_restart_prepare(char *cmd)
 {
 	blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd);
 	system_state = SYSTEM_RESTART;
+	printk_trigger_flush();
 	usermodehelper_disable();
 	device_shutdown();
 }
@@ -305,6 +308,7 @@ static void kernel_shutdown_prepare(enum system_states state)
 	blocking_notifier_call_chain(&reboot_notifier_list,
 		(state == SYSTEM_HALT) ? SYS_HALT : SYS_POWER_OFF, NULL);
 	system_state = state;
+	printk_trigger_flush();
 	usermodehelper_disable();
 	device_shutdown();
 }
-- 
2.55.0

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Bradley Morgan]

On 8 October 2026 16:08:52 BST, Petr Mladek <pmladek@suse.com> wrote:
>NBCON consoles emit messages in a dedicated kthreads when the system
>is working properly. printk() tries to flush them synchronously in
>explicitly marked emergency context and in panic().
>
>Another situation where printk() could not rely on kthreads are the
>various
>reboot and halt code paths. They can be detected by the `system_state`
>variable.
>
>Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
>printk() will automatically try flushing the consoles synchronously.
>Also do not rely on printk() and explicitly flush the consoles
>after these states are set.
>
>Note that suspend code paths have already been handled, see
>console_suspend_all() and console_suspend().
>
>Reported-by: kernel test robot <lkp@intel.com>
>Closes: https://lore.kernel.org/all/202608061008.48a1e76e-lkp@intel.com/
>Suggested-by: John Ogness <john.ogness@linutronix.de>

Hi, this looks good to me reboot.c ended! :)

Reviewed-by: Bradley Morgan <brads@mainlining.org>

>Signed-off-by: Petr Mladek <pmladek@suse.com>
>---
>Hi,
>
>ah, this somehow fallen through cracks.
>I am sending it as a proper patch finally.
>
>I am sorry I have been somehow too busy before
>the Plumbers conference.

That's ok :) I'm too unlucky as to not go to any conferences :c

>
>Best Regards,
>Petr
>---
> kernel/printk/nbcon.c | 4 ++++
> kernel/reboot.c       | 4 ++++
> 2 files changed, 8 insertions(+)
>
>diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
>index fb1b37741952..d8f8ec836eea 100644
>--- a/kernel/printk/nbcon.c
>+++ b/kernel/printk/nbcon.c
>@@ -1446,6 +1446,10 @@ enum nbcon_prio nbcon_get_default_prio(void)
> 	if (panic_on_this_cpu())
> 		return NBCON_PRIO_PANIC;
> 
>+	/* Do not rely on kthreads when the system is going down. */
>+	if (system_state > SYSTEM_RUNNING)
>+		return NBCON_PRIO_EMERGENCY;
>+
> 	cpu_emergency_nesting = nbcon_get_cpu_emergency_nesting();
> 	if (*cpu_emergency_nesting)
> 		return NBCON_PRIO_EMERGENCY;
>diff --git a/kernel/reboot.c b/kernel/reboot.c
>index d177d89fcc33..776784a82499 100644
>--- a/kernel/reboot.c
>+++ b/kernel/reboot.c
>@@ -8,6 +8,7 @@
> #define pr_fmt(fmt)	"reboot: " fmt
> 
> #include <linux/atomic.h>
>+#include <linux/console.h>
> #include <linux/ctype.h>
> #include <linux/export.h>
> #include <linux/kexec.h>
>@@ -94,6 +95,7 @@ void emergency_restart(void)
> {
> 	kmsg_dump(KMSG_DUMP_EMERG);
> 	system_state = SYSTEM_RESTART;
>+	printk_trigger_flush();
> 	machine_emergency_restart();
> }
> EXPORT_SYMBOL_GPL(emergency_restart);
>@@ -102,6 +104,7 @@ void kernel_restart_prepare(char *cmd)
> {
> 	blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd);
> 	system_state = SYSTEM_RESTART;
>+	printk_trigger_flush();
> 	usermodehelper_disable();
> 	device_shutdown();
> }
>@@ -305,6 +308,7 @@ static void kernel_shutdown_prepare(enum system_states state)
> 	blocking_notifier_call_chain(&reboot_notifier_list,
> 		(state == SYSTEM_HALT) ? SYS_HALT : SYS_POWER_OFF, NULL);
> 	system_state = state;
>+	printk_trigger_flush();
> 	usermodehelper_disable();
> 	device_shutdown();
> }
>

--- Thanks!
"I'm not a very positive person" - Linus torvalds

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Petr Mladek]

On Thu 2026-10-08 16:19:03, Bradley Morgan wrote:
> On 8 October 2026 16:08:52 BST, Petr Mladek <pmladek@suse.com> wrote:
> >NBCON consoles emit messages in a dedicated kthreads when the system
> >is working properly. printk() tries to flush them synchronously in
> >explicitly marked emergency context and in panic().
> >
> >Another situation where printk() could not rely on kthreads are the
> >various
> >reboot and halt code paths. They can be detected by the `system_state`
> >variable.
> >
> >Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
> >printk() will automatically try flushing the consoles synchronously.
> >Also do not rely on printk() and explicitly flush the consoles
> >after these states are set.
> >
> >Note that suspend code paths have already been handled, see
> >console_suspend_all() and console_suspend().
> >
> >Reported-by: kernel test robot <lkp@intel.com>
> >Closes: https://lore.kernel.org/all/202608061008.48a1e76e-lkp@intel.com/
> >Suggested-by: John Ogness <john.ogness@linutronix.de>
> 
> Hi, this looks good to me reboot.c ended! :)
> 
> Reviewed-by: Bradley Morgan <brads@mainlining.org>

Honestly, I am more or less ignoring Reviewed-by: Bradley Morgan.
I can't recall any problem reported by this person. And the excited,
AI-generated, acks do not add much value.

> >Signed-off-by: Petr Mladek <pmladek@suse.com>
> >---
> >Hi,
> >
> >ah, this somehow fallen through cracks.
> >I am sending it as a proper patch finally.
> >
> >I am sorry I have been somehow too busy before
> >the Plumbers conference.
> 
> That's ok :) I'm too unlucky as to not go to any conferences :c

I am not sure if there will be any conferences when robots will
be ready to attend them.

Best Regards,
Petr

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Sebastian Andrzej Siewior]

On 2026-10-09 11:16:51 [+0200], Petr Mladek wrote:
> > Hi, this looks good to me reboot.c ended! :)
> > 
> > Reviewed-by: Bradley Morgan <brads@mainlining.org>
> 
> Honestly, I am more or less ignoring Reviewed-by: Bradley Morgan.
> I can't recall any problem reported by this person. And the excited,
> AI-generated, acks do not add much value.

to be fair, there is one:
| $ git log --grep "Reported-by: Bradley Morgan" --oneline | wc -l
| 1

and there are even fixes:

| $ git log --author "Bradley Morgan" --grep "Fixes:" --oneline | wc -l
| 12

> > That's ok :) I'm too unlucky as to not go to any conferences :c
> 
> I am not sure if there will be any conferences when robots will
> be ready to attend them.

Would you or would not exclude Lieutenant Commander Data from a
conference? We don't have positronic brains yet but maybe in a few years
:)

> Best Regards,
> Petr

Sebastian

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Sebastian Andrzej Siewior]

On 2026-10-08 17:08:52 [+0200], Petr Mladek wrote:
> NBCON consoles emit messages in a dedicated kthreads when the system
> is working properly. printk() tries to flush them synchronously in
> explicitly marked emergency context and in panic().
> 
> Another situation where printk() could not rely on kthreads are the various
> reboot and halt code paths. They can be detected by the `system_state`
> variable.
> 
> Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
> printk() will automatically try flushing the consoles synchronously.
> Also do not rely on printk() and explicitly flush the consoles
> after these states are set.

while this seems okay, didn't we have pr_flush() to flush the output on
shutdown/ reboot?

> Note that suspend code paths have already been handled, see
> console_suspend_all() and console_suspend().
> 
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/all/202608061008.48a1e76e-lkp@intel.com/
> Suggested-by: John Ogness <john.ogness@linutronix.de>
> Signed-off-by: Petr Mladek <pmladek@suse.com>

Sebastian

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Petr Mladek]

On Fri 2026-10-09 08:51:52, Sebastian Andrzej Siewior wrote:
> On 2026-10-08 17:08:52 [+0200], Petr Mladek wrote:
> > NBCON consoles emit messages in a dedicated kthreads when the system
> > is working properly. printk() tries to flush them synchronously in
> > explicitly marked emergency context and in panic().
> > 
> > Another situation where printk() could not rely on kthreads are the various
> > reboot and halt code paths. They can be detected by the `system_state`
> > variable.
> > 
> > Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
> > printk() will automatically try flushing the consoles synchronously.
> > Also do not rely on printk() and explicitly flush the consoles
> > after these states are set.
> 
> while this seems okay, didn't we have pr_flush() to flush the output on
> shutdown/ reboot?

Good point! We should clean this up.

Anyway, IMHO, we should switch to NBCON_PRIO_EMERGENCY for the post-running
states and flush the pending messages immediately. It is more
reliable. It is easy to find the locations when system_state() is set.
And all follow-up printk() calls will try the direct flush so that
they won't rely on an explicit flush.

An ideal solution would be to add an wrapper, e.g.

	void set_syste_state(enum system_states state)
	{
		system_state = state;

		if (system_state > SYSTEM_RUNNING)
			pr_flush(0, true);
	}


Another thing is that printk_trigger_flush() is an overkill.
It tries to wake kthreads even via irq_work but we are
interested only in the direct flush.

I am not sure why neither me nor John used pr_flush().
It might be because it originally did not flush atomic consoles
directly. At least I had an outdated mental map.

Also the timeout should not be needed because all consoles should
be flushed directly. But it can be solved by using zero timeout.

In fact, we should block the kthreads to prevent seeing more
incomplete/interrupted messages, see the commit c41c0ebfa1e0eb
("printk/nbcon: Block printk kthreads when any CPU is in an emergency
context"). Something like:

diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index d8f8ec836eea..1c21c1e65b00 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1187,13 +1187,14 @@ static bool nbcon_kthread_should_wakeup(struct console *con, struct nbcon_contex
 		return true;
 
 	/*
-	 * Block the kthread when the system is in an emergency or panic mode.
-	 * It increases the chance that these contexts would be able to show
-	 * the messages directly. And it reduces the risk of interrupted writes
-	 * where the context with a higher priority takes over the nbcon console
-	 * ownership in the middle of a message.
+	 * Block the kthread when the system is in an emergency, going down,
+	 * or panic mode. It increases the chance that these contexts would
+	 * be able to show the messages directly. And it reduces the risk of
+	 * interrupted writes where the context with a higher priority takes
+	 * over the nbcon console ownership in the middle of a message.
 	 */
 	if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
+	    unlikely(system_state > SYSTEM_RUNNING) ||
 	    unlikely(panic_in_progress()))
 		return false;
 
@@ -1249,10 +1250,12 @@ static int nbcon_kthread_func(void *__console)
 			return 0;
 
 		/*
-		 * Block the kthread when the system is in an emergency or panic
-		 * mode. See nbcon_kthread_should_wakeup() for more details.
+		 * Block the kthread when the system is in an emergency, going
+		 * down, or panic mode. See nbcon_kthread_should_wakeup() for
+		 * more details.
 		 */
 		if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
+		    unlikely(system_state > SYSTEM_RUNNING) ||
 		    unlikely(panic_in_progress()))
 			goto wait_for_event;
 

But wait, this might cause regression on netconsole which sets
CON_NBCON_ATOMIC_UNSAFE and is not able to flush the messages
directly a safe way.

A possibility would be to use con->write_thread() when pr_flush()
is called in task context. It should be safe in most shutdown
code paths except for the emergency_restart().

And we need an explicit pr_flush() even later in the halt
and maybe even some unsafe flush later in emergency_restart()
code path.

Sigh, this is getting complicated.

Best Regards,
Petr

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Sebastian Andrzej Siewior]

On 2026-10-09 11:08:02 [+0200], Petr Mladek wrote:
> On Fri 2026-10-09 08:51:52, Sebastian Andrzej Siewior wrote:
> > On 2026-10-08 17:08:52 [+0200], Petr Mladek wrote:
> > > NBCON consoles emit messages in a dedicated kthreads when the system
> > > is working properly. printk() tries to flush them synchronously in
> > > explicitly marked emergency context and in panic().
> > > 
> > > Another situation where printk() could not rely on kthreads are the various
> > > reboot and halt code paths. They can be detected by the `system_state`
> > > variable.
> > > 
> > > Let's default to NBCON_PRIO_EMERGENCY for the post-running states.
> > > printk() will automatically try flushing the consoles synchronously.
> > > Also do not rely on printk() and explicitly flush the consoles
> > > after these states are set.
> > 
> > while this seems okay, didn't we have pr_flush() to flush the output on
> > shutdown/ reboot?
> 
> Good point! We should clean this up.
> 
> Anyway, IMHO, we should switch to NBCON_PRIO_EMERGENCY for the post-running
> states and flush the pending messages immediately. It is more
> reliable. It is easy to find the locations when system_state() is set.
> And all follow-up printk() calls will try the direct flush so that
> they won't rely on an explicit flush.
> 
> An ideal solution would be to add an wrapper, e.g.
> 
> 	void set_syste_state(enum system_states state)
> 	{
> 		system_state = state;
> 
> 		if (system_state > SYSTEM_RUNNING)
> 			pr_flush(0, true);
> 	}
> 
> 
> Another thing is that printk_trigger_flush() is an overkill.
> It tries to wake kthreads even via irq_work but we are
> interested only in the direct flush.
> 
> I am not sure why neither me nor John used pr_flush().
> It might be because it originally did not flush atomic consoles
> directly. At least I had an outdated mental map.

The nbcon should be awake because there is outstanding printing to be
done as far as I understand. It is just not running or behind. So you
would need just wait until the thread is done before you continue.

This pops up now because the 8250 does it via the thread now. If you
don't have an atomic console implementation but use the thread anyway
(say the PREEMPT_RT case) then you would end up with the same problem,
right?

Recently I did play with CPU shutdown and noticed that PREEMPT_RT misses
an irq_work sync which explained missing printk output. You probably
recall the printk part of this :)

Now that I look more into it, the above with pr_flush() does not work
because by the time SYSTEM_SUSPEND is assigned it expects interrupts to
be disabled (or disables them a few lines earlier).

emergency_restart() looks like a bad candidate for a flush.
kernel_restart_prepare() on the other hand would be good. And there is a
flush in kernel_power_off().

In general I would expect sysrq-b to reboot immediately without the
flush but an ordinary reboot should flush.

> Also the timeout should not be needed because all consoles should
> be flushed directly. But it can be solved by using zero timeout.
> 
> In fact, we should block the kthreads to prevent seeing more
> incomplete/interrupted messages, see the commit c41c0ebfa1e0eb
> ("printk/nbcon: Block printk kthreads when any CPU is in an emergency
> context"). Something like:

But if we have more than one console and more than one CPU we could let
them flush in parallel and just wait for them do be done?

> diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
> index d8f8ec836eea..1c21c1e65b00 100644
> --- a/kernel/printk/nbcon.c
> +++ b/kernel/printk/nbcon.c
> @@ -1187,13 +1187,14 @@ static bool nbcon_kthread_should_wakeup(struct console *con, struct nbcon_contex
>  		return true;
>  
>  	/*
> -	 * Block the kthread when the system is in an emergency or panic mode.
> -	 * It increases the chance that these contexts would be able to show
> -	 * the messages directly. And it reduces the risk of interrupted writes
> -	 * where the context with a higher priority takes over the nbcon console
> -	 * ownership in the middle of a message.
> +	 * Block the kthread when the system is in an emergency, going down,
> +	 * or panic mode. It increases the chance that these contexts would
> +	 * be able to show the messages directly. And it reduces the risk of
> +	 * interrupted writes where the context with a higher priority takes
> +	 * over the nbcon console ownership in the middle of a message.
>  	 */
>  	if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
> +	    unlikely(system_state > SYSTEM_RUNNING) ||
>  	    unlikely(panic_in_progress()))
>  		return false;
>  
> @@ -1249,10 +1250,12 @@ static int nbcon_kthread_func(void *__console)
>  			return 0;
>  
>  		/*
> -		 * Block the kthread when the system is in an emergency or panic
> -		 * mode. See nbcon_kthread_should_wakeup() for more details.
> +		 * Block the kthread when the system is in an emergency, going
> +		 * down, or panic mode. See nbcon_kthread_should_wakeup() for
> +		 * more details.
>  		 */
>  		if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
> +		    unlikely(system_state > SYSTEM_RUNNING) ||
>  		    unlikely(panic_in_progress()))
>  			goto wait_for_event;
>  
> 
> But wait, this might cause regression on netconsole which sets
> CON_NBCON_ATOMIC_UNSAFE and is not able to flush the messages
> directly a safe way.

same as PREEMPT_RT.

> A possibility would be to use con->write_thread() when pr_flush()
> is called in task context. It should be safe in most shutdown
> code paths except for the emergency_restart().
> 
> And we need an explicit pr_flush() even later in the halt
> and maybe even some unsafe flush later in emergency_restart()
> code path.
> 
> Sigh, this is getting complicated.

So maybe just identify the few spots in the reboot case.

> Best Regards,
> Petr

Sebastian

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[John Ogness]

On 2026-10-09, Petr Mladek <pmladek@suse.com> wrote:
> Anyway, IMHO, we should switch to NBCON_PRIO_EMERGENCY for the post-running
> states and flush the pending messages immediately. It is more
> reliable. It is easy to find the locations when system_state() is set.
> And all follow-up printk() calls will try the direct flush so that
> they won't rely on an explicit flush.

We need to be careful about abusing CPU state in order to get certain
functionality. We already have things in place to gracefully switch to
atomic after shutting down the printing kthreads.

I will not have a chance to seriously look at this until I get back next
week. But at a quick glance, it looks like a roll of duct tape.

John

Re: [PATCH] nbcon/reboot: Flush nbcon consoles synchronously on reboot

[Petr Mladek]

On Fri 2026-10-09 15:34:30, John Ogness wrote:
> On 2026-10-09, Petr Mladek <pmladek@suse.com> wrote:
> > Anyway, IMHO, we should switch to NBCON_PRIO_EMERGENCY for the post-running
> > states and flush the pending messages immediately. It is more
> > reliable. It is easy to find the locations when system_state() is set.
> > And all follow-up printk() calls will try the direct flush so that
> > they won't rely on an explicit flush.
> 
> We need to be careful about abusing CPU state in order to get certain
> functionality. We already have things in place to gracefully switch to
> atomic after shutting down the printing kthreads.

This explains why I see the messages even without this patch.
I wonder what is actually going on in the test.

The log contains:

</paste>
LKP: ttyS0: 256: LKP: tbox cant kexec and rebooting forcely
</paste>

I see this message in lkp-tests/bin/lkp-setup-rootfs:

<cat lkp-tests/bin/lkp-setup-rootfs>
lkp_reboot()
{
	local tbox_state="rebooting"
	[ -n "$1" ] && tbox_state="rebooting_$1"

	set_tbox_wtmp $tbox_state

	# Avoid get stuck in yocto minimal rootfs.
	# LKP: rebooting
	# [    8.124081] LKP: rebooting
	# # (shell prompt)
	is_virt && [ -e '/proc/sysrq-trigger' ] && {
		sync
		echo s >/proc/sysrq-trigger
		echo b >/proc/sysrq-trigger
	}

	start_reboot_watchdog
	reboot_tbox 2>/dev/null

	exit
}
[...]
	if tbox_cant_kexec; then
		echo_to_tty "LKP: tbox cant kexec and rebooting forcely"
		lkp_reboot "cant_kexec"
	fi
</cat lkp-tests/bin/lkp-setup-rootfs>

so, it ends with reboot_tbox which is:

<cat lkp-tests/lib/reboot.sh>
reboot_tbox()
{
	reboot
}
</cat lkp-tests/lib/reboot.sh

which is classic reboot. I am not sure where it expects the warning.
I actually can't find the code which prints:

<paste>
BUG: kernel reboot-without-warning in test stage
</paste>

I might need to try reproducing it with the lkp-test framework.
But it has to wait for the next week.

Best Regards,
Petr