[RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Ryan Roberts]

Netconsole registers with CON_NBCON_ATOMIC_UNSAFE, so its write_atomic()
callback is only usable during nbcon_atomic_flush_unsafe(). Previously
this was only called by vpanic() if panic_timeout=0 - i.e. if the system
was configured not to reboot on panic. So if the system was configured
to reboot on panic, netconsole would never receive the panic logs.

Move vpanic()'s emergency_restart() call to after the call to
nbcon_atomic_flush_unsafe() to solve this problem. The downside is that
potentially unsafe operations are now performed prior to
emergency_restart() which could theoretically reduce the chances of the
restart succeeding. But we are already in a panic situation so it could
be argued that everything is best effort already.

In older kernels (v6.19 and earlier) netconsole is able to print these
panic logs while the system is configured to reboot on panic. So from a
user perspective this is a regression caused by commit 7eab73b18630
("netconsole: convert to NBCON console infrastructure").

We have automated test systems without BMC access, which rely on these
two features working together.

Fixes: 7eab73b18630 ("netconsole: convert to NBCON console infrastructure")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
---
 kernel/panic.c | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/kernel/panic.c b/kernel/panic.c
index 213725b612aa..473dc7d13b23 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -732,32 +732,22 @@ void vpanic(const char *fmt, va_list args)
 			mdelay(PANIC_TIMER_STEP);
 		}
 	}
-	if (panic_timeout != 0) {
-		/*
-		 * This will not be a clean reboot, with everything
-		 * shutting down.  But if there is a chance of
-		 * rebooting the system it will be rebooted.
-		 */
-		if (panic_reboot_mode != REBOOT_UNDEFINED)
-			reboot_mode = panic_reboot_mode;
-		emergency_restart();
-	}
+	if (panic_timeout == 0) {
 #ifdef __sparc__
-	{
 		extern int stop_a_enabled;
 		/* Make sure the user can actually press Stop-A (L1-A) */
 		stop_a_enabled = 1;
 		pr_emerg("Press Stop-A (L1-A) from sun keyboard or send break\n"
-			 "twice on console to return to the boot prom\n");
-	}
+			"twice on console to return to the boot prom\n");
 #endif
 #if defined(CONFIG_S390)
-	disabled_wait();
+		disabled_wait();
 #endif
-	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
+		pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);

-	/* Do not scroll important messages printed above */
-	suppress_printk = 1;
+		/* Do not scroll important messages printed above */
+		suppress_printk = 1;
+	}

 	/*
 	 * The final messages may not have been printed if in a context that
@@ -767,6 +757,17 @@ void vpanic(const char *fmt, va_list args)
 	console_flush_on_panic(CONSOLE_FLUSH_PENDING);
 	nbcon_atomic_flush_unsafe();

+	if (panic_timeout != 0) {
+		/*
+		 * This will not be a clean reboot, with everything
+		 * shutting down.  But if there is a chance of
+		 * rebooting the system it will be rebooted.
+		 */
+		if (panic_reboot_mode != REBOOT_UNDEFINED)
+			reboot_mode = panic_reboot_mode;
+		emergency_restart();
+	}
+
 	local_irq_enable();
 	for (i = 0; ; i += PANIC_TIMER_STEP) {
 		touch_softlockup_watchdog();
--
2.43.0

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Ryan Roberts]

On 15/09/2026 11:01, Ryan Roberts wrote:
> Netconsole registers with CON_NBCON_ATOMIC_UNSAFE, so its write_atomic()
> callback is only usable during nbcon_atomic_flush_unsafe(). Previously
> this was only called by vpanic() if panic_timeout=0 - i.e. if the system
> was configured not to reboot on panic. So if the system was configured
> to reboot on panic, netconsole would never receive the panic logs.
> 
> Move vpanic()'s emergency_restart() call to after the call to
> nbcon_atomic_flush_unsafe() to solve this problem. The downside is that
> potentially unsafe operations are now performed prior to
> emergency_restart() which could theoretically reduce the chances of the
> restart succeeding. But we are already in a panic situation so it could
> be argued that everything is best effort already.
> 
> In older kernels (v6.19 and earlier) netconsole is able to print these
> panic logs while the system is configured to reboot on panic. So from a
> user perspective this is a regression caused by commit 7eab73b18630
> ("netconsole: convert to NBCON console infrastructure").
> 
> We have automated test systems without BMC access, which rely on these
> two features working together.
> 
> Fixes: 7eab73b18630 ("netconsole: convert to NBCON console infrastructure")
> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
> ---

Oops forgot to add commentary to the bottom:

I've set this as RFC because I'm not sure if the current behaviour is a
deliberate design decision? I guess if we flush the unsafe console, there is a
possibility it deadlocks, preventing the subsequent reboot. If people are
concerned about this, perhaps this needs a cmdline flag?

Thanks,
Ryan



>  kernel/panic.c | 35 ++++++++++++++++++-----------------
>  1 file changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 213725b612aa..473dc7d13b23 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -732,32 +732,22 @@ void vpanic(const char *fmt, va_list args)
>  			mdelay(PANIC_TIMER_STEP);
>  		}
>  	}
> -	if (panic_timeout != 0) {
> -		/*
> -		 * This will not be a clean reboot, with everything
> -		 * shutting down.  But if there is a chance of
> -		 * rebooting the system it will be rebooted.
> -		 */
> -		if (panic_reboot_mode != REBOOT_UNDEFINED)
> -			reboot_mode = panic_reboot_mode;
> -		emergency_restart();
> -	}
> +	if (panic_timeout == 0) {
>  #ifdef __sparc__
> -	{
>  		extern int stop_a_enabled;
>  		/* Make sure the user can actually press Stop-A (L1-A) */
>  		stop_a_enabled = 1;
>  		pr_emerg("Press Stop-A (L1-A) from sun keyboard or send break\n"
> -			 "twice on console to return to the boot prom\n");
> -	}
> +			"twice on console to return to the boot prom\n");
>  #endif
>  #if defined(CONFIG_S390)
> -	disabled_wait();
> +		disabled_wait();
>  #endif
> -	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> +		pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> 
> -	/* Do not scroll important messages printed above */
> -	suppress_printk = 1;
> +		/* Do not scroll important messages printed above */
> +		suppress_printk = 1;
> +	}
> 
>  	/*
>  	 * The final messages may not have been printed if in a context that
> @@ -767,6 +757,17 @@ void vpanic(const char *fmt, va_list args)
>  	console_flush_on_panic(CONSOLE_FLUSH_PENDING);
>  	nbcon_atomic_flush_unsafe();
> 
> +	if (panic_timeout != 0) {
> +		/*
> +		 * This will not be a clean reboot, with everything
> +		 * shutting down.  But if there is a chance of
> +		 * rebooting the system it will be rebooted.
> +		 */
> +		if (panic_reboot_mode != REBOOT_UNDEFINED)
> +			reboot_mode = panic_reboot_mode;
> +		emergency_restart();
> +	}
> +
>  	local_irq_enable();
>  	for (i = 0; ; i += PANIC_TIMER_STEP) {
>  		touch_softlockup_watchdog();
> --
> 2.43.0
> 

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Bradley Morgan]

Hey, I think thisll conflict with the arch_do_panic() patch (b45d6e4b3580)

Once you rebase. I'd reckon the
change shrinks to just moving the panic_timeout != 0 block below
nbcon_atomic_flush_unsafe().

On your RFC question, whether the current behavior is deliberate: I
don't think it is. nbcon_atomic_flush_unsafe() was placed at the end of
vpanic() by 5dde3b735413 as a final attempt to flush pending messages, and
at
that point no console was actually CON_NBCON_ATOMIC_UNSAFE, netconsole
was still legacy. There was nothing to lose by putting it last. The
ordering only became a bug once the first unsafe console showed up.
John and Petr can confirm, but I found no sign anyone chose this.

 
--- Thanks!
https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Ryan Roberts]

On 15/09/2026 19:49, Bradley Morgan wrote:
> Hey, I think thisll conflict with the arch_do_panic() patch (b45d6e4b3580)

Ahh I see it here:
https://lore.kernel.org/all/20260831-arch_do_panic-v6-0-a1e170a9e7fd@linux.ibm.com/

And looks like it's in Andrew's mm-nonmm-unstable branch. I guess I should
repost based on that branch?

> 
> Once you rebase. I'd reckon the
> change shrinks to just moving the panic_timeout != 0 block below
> nbcon_atomic_flush_unsafe().

I think I'll also want to make the call to arch_do_panic() conditional on
(panic_timeout == 0) so it's equivalent to the patch I sent?

> 
> On your RFC question, whether the current behavior is deliberate: I
> don't think it is. nbcon_atomic_flush_unsafe() was placed at the end of
> vpanic() by 5dde3b735413 as a final attempt to flush pending messages, and
> at
> that point no console was actually CON_NBCON_ATOMIC_UNSAFE, netconsole
> was still legacy. There was nothing to lose by putting it last. The
> ordering only became a bug once the first unsafe console showed up.
> John and Petr can confirm, but I found no sign anyone chose this.

OK great; I'll wait a few days for others to chime in, otherwise I'll plan to
repost based on mm-nonmm-unstable, dropping the RFC tag.

Thanks,
Ryan


> 
>  
> --- Thanks!
> https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Petr Mladek]

On Tue 2026-09-15 11:01:38, Ryan Roberts wrote:
> Netconsole registers with CON_NBCON_ATOMIC_UNSAFE, so its write_atomic()
> callback is only usable during nbcon_atomic_flush_unsafe(). Previously
> this was only called by vpanic() if panic_timeout=0 - i.e. if the system
> was configured not to reboot on panic. So if the system was configured
> to reboot on panic, netconsole would never receive the panic logs.
> 
> Move vpanic()'s emergency_restart() call to after the call to
> nbcon_atomic_flush_unsafe() to solve this problem. The downside is that
> potentially unsafe operations are now performed prior to
> emergency_restart() which could theoretically reduce the chances of the
> restart succeeding. But we are already in a panic situation so it could
> be argued that everything is best effort already.

The panic() code is like walking on bumpy roads. It tries to be as
safe as possible. IMHO, it has few important tasks:

  + show debug messages
  + create crash dump (optional)
  + safely reboot (optional)

The debug messages are important but some users might prioritize
reboot over the debug messages. It helps when the panic() is rare
(low likely race) and the system is quickly able to handle the load
after the reboot.

This is why we put nbcon_atomic_flush_unsafe() only into the path
which ends with an infinite loop.

> In older kernels (v6.19 and earlier) netconsole is able to print these
> panic logs while the system is configured to reboot on panic. So from a
> user perspective this is a regression caused by commit 7eab73b18630
> ("netconsole: convert to NBCON console infrastructure").
> 
> We have automated test systems without BMC access, which rely on these
> two features working together.

I see. To be precise, I would say that the primary culprit is the
commit 187de7c212e5fa87 ("printk: nbcon: Allow unsafe write_atomic()
for panic"). It adds support for consoles which will be flushed
in panic() only by nbcon_atomic_flush_unsafe().

It is a big difference because printk() tries to flush both
legacy and "safe" NBCON consoles directly in panic(). In addition,
they are explicitely flushed by console_flush_on_panic().

console_flush_on_panic() is called more times in panic().
And it tries to flush legacy consoles the unsafe way even
before emergency_restart().

OK, we should close this gap. But I suggest another solution.

We already have the following functions for flushing NBCON
consoles using write_atomic()


  + nbcon_atomic_flush_pending() - flush CON_NBCON consoles
	using con->write_atomic() but only when safe

  + nbcon_atomic_flush_unsafe() - flush CON_NBCON consoles
	using con->write_atomic() and allow unsafe takeover.

I suggest to add 3rd variant, for example:

  + nbcon_atomic_unsafe_flush_unsafe() - flush
	CON_NBCON_ATOMIC_UNSAFE consoles with con->write_atomic()
	and allow unsafe takeover.

And I would call this 3rd variant in console_flush_on_panic(),
something like:

void console_flush_on_panic(enum con_flush_mode mode)
{
[...]
	printk_get_console_flush_type(&ft);
	if (ft.nbcon_atomic) {
		nbcon_atomic_flush_pending();
		nbcon_atomic_unsafe_flush_unsafe();
	}

	/* Flush legacy consoles once allowed, even when dangerous. */
	if (legacy_allow_panic_sync)
		console_flush_all(false, &next_seq, &handover);
}

But maybe, this is over engineered and we should just call
nbcon_atomic_flush_unsafe() in console_flush_on_panic(), like:

void console_flush_on_panic(enum con_flush_mode mode)
{
[...]
	printk_get_console_flush_type(&ft);
	if (ft.nbcon_atomic) {
		nbcon_atomic_flush_pending();
		nbcon_atomic_flush_unsafe();
	}

	/* Flush legacy consoles once allowed, even when dangerous. */
	if (legacy_allow_panic_sync)
		console_flush_all(false, &next_seq, &handover);
}

If there there are people really concerned about the reboot
reliability then we might need to make it configurable, e.g.
by adding reliable_panic command line option which would skip
the unsafe flush before reboot.

See below.

> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -732,32 +732,22 @@ void vpanic(const char *fmt, va_list args)
>  			mdelay(PANIC_TIMER_STEP);
>  		}
>  	}
> -	if (panic_timeout != 0) {
> -		/*
> -		 * This will not be a clean reboot, with everything
> -		 * shutting down.  But if there is a chance of
> -		 * rebooting the system it will be rebooted.
> -		 */
> -		if (panic_reboot_mode != REBOOT_UNDEFINED)
> -			reboot_mode = panic_reboot_mode;
> -		emergency_restart();
> -	}
> +	if (panic_timeout == 0) {
>  #ifdef __sparc__
> -	{
>  		extern int stop_a_enabled;
>  		/* Make sure the user can actually press Stop-A (L1-A) */
>  		stop_a_enabled = 1;
>  		pr_emerg("Press Stop-A (L1-A) from sun keyboard or send break\n"
> -			 "twice on console to return to the boot prom\n");
> -	}
> +			"twice on console to return to the boot prom\n");
>  #endif
>  #if defined(CONFIG_S390)
> -	disabled_wait();
> +		disabled_wait();
>  #endif
> -	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> +		pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> 
> -	/* Do not scroll important messages printed above */
> -	suppress_printk = 1;
> +		/* Do not scroll important messages printed above */
> +		suppress_printk = 1;

We should not supress printk when emergency_restart() is going to
be called. It might print some error messages which might explain
why it failed, ...

> +	}
> 
>  	/*
>  	 * The final messages may not have been printed if in a context that
> @@ -767,6 +757,17 @@ void vpanic(const char *fmt, va_list args)
>  	console_flush_on_panic(CONSOLE_FLUSH_PENDING);
>  	nbcon_atomic_flush_unsafe();
> 
> +	if (panic_timeout != 0) {
> +		/*
> +		 * This will not be a clean reboot, with everything
> +		 * shutting down.  But if there is a chance of
> +		 * rebooting the system it will be rebooted.
> +		 */
> +		if (panic_reboot_mode != REBOOT_UNDEFINED)
> +			reboot_mode = panic_reboot_mode;
> +		emergency_restart();
> +	}
> +
>  	local_irq_enable();
>  	for (i = 0; ; i += PANIC_TIMER_STEP) {
>  		touch_softlockup_watchdog();

Best Regards,
Petr

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Bradley Morgan]

On 16 September 2026 08:44:53 BST, Ryan Roberts <ryan.roberts@arm.com>
wrote:
>On 15/09/2026 19:49, Bradley Morgan wrote:
>> Hey, I think thisll conflict with the arch_do_panic() patch
>(b45d6e4b3580)
>
>Ahh I see it here:
>https://lore.kernel.org/all/20260831-arch_do_panic-v6-0-a1e170a9e7fd@linux.ibm.com/
>
>And looks like it's in Andrew's mm-nonmm-unstable branch. I guess I should
>repost based on that branch?

Yes.

(That branch also goes to linux-next)

>
>> 
>> Once you rebase. I'd reckon the
>> change shrinks to just moving the panic_timeout != 0 block below
>> nbcon_atomic_flush_unsafe().
>
>I think I'll also want to make the call to arch_do_panic() conditional on
>(panic_timeout == 0) so it's equivalent to the patch I sent?
>
>> 
>> On your RFC question, whether the current behavior is deliberate: I
>> don't think it is. nbcon_atomic_flush_unsafe() was placed at the end of
>> vpanic() by 5dde3b735413 as a final attempt to flush pending messages,
>and
>> at
>> that point no console was actually CON_NBCON_ATOMIC_UNSAFE, netconsole
>> was still legacy. There was nothing to lose by putting it last. The
>> ordering only became a bug once the first unsafe console showed up.
>> John and Petr can confirm, but I found no sign anyone chose this.
>
>OK great; I'll wait a few days for others to chime in, otherwise I'll plan
>to
>repost based on mm-nonmm-unstable, dropping the RFC tag.
>
>Thanks,
>Ryan
>
>
>> 
>>  
>> --- Thanks!
>>
>https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/
>

--- Thanks!
https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Gustavo Luiz Duarte]

On 2026-09-16 15:08:44+02:00, Petr Mladek wrote:
> On Tue 2026-09-15 11:01:38, Ryan Roberts wrote:
> 
> > Netconsole registers with CON_NBCON_ATOMIC_UNSAFE, so its write_atomic()
> > callback is only usable during nbcon_atomic_flush_unsafe(). Previously
> > this was only called by vpanic() if panic_timeout=0 - i.e. if the system
> > was configured not to reboot on panic. So if the system was configured
> > to reboot on panic, netconsole would never receive the panic logs.
> > 
> > Move vpanic()'s emergency_restart() call to after the call to
> > nbcon_atomic_flush_unsafe() to solve this problem. The downside is that
> > potentially unsafe operations are now performed prior to
> > emergency_restart() which could theoretically reduce the chances of the
> > restart succeeding. But we are already in a panic situation so it could
> > be argued that everything is best effort already.
> 
> The panic() code is like walking on bumpy roads. It tries to be as
> safe as possible. IMHO, it has few important tasks:
> 
>   + show debug messages
>   + create crash dump (optional)
>   + safely reboot (optional)
> 
> The debug messages are important but some users might prioritize
> reboot over the debug messages. It helps when the panic() is rare
> (low likely race) and the system is quickly able to handle the load
> after the reboot.
> 
> This is why we put nbcon_atomic_flush_unsafe() only into the path
> which ends with an infinite loop.
> 
> > In older kernels (v6.19 and earlier) netconsole is able to print these
> > panic logs while the system is configured to reboot on panic. So from a
> > user perspective this is a regression caused by commit 7eab73b18630
> > ("netconsole: convert to NBCON console infrastructure").
> > 
> > We have automated test systems without BMC access, which rely on these
> > two features working together.
> 
> I see. To be precise, I would say that the primary culprit is the
> commit 187de7c212e5fa87 ("printk: nbcon: Allow unsafe write_atomic()
> for panic"). It adds support for consoles which will be flushed
> in panic() only by nbcon_atomic_flush_unsafe().
> 
> It is a big difference because printk() tries to flush both
> legacy and "safe" NBCON consoles directly in panic(). In addition,
> they are explicitely flushed by console_flush_on_panic().
> 
> console_flush_on_panic() is called more times in panic().
> And it tries to flush legacy consoles the unsafe way even
> before emergency_restart().
> 
> OK, we should close this gap. But I suggest another solution.
> 
> We already have the following functions for flushing NBCON
> consoles using write_atomic()
> 
> 
>   + nbcon_atomic_flush_pending() - flush CON_NBCON consoles
> 	using con->write_atomic() but only when safe
> 
>   + nbcon_atomic_flush_unsafe() - flush CON_NBCON consoles
> 	using con->write_atomic() and allow unsafe takeover.
> 
> I suggest to add 3rd variant, for example:
> 
>   + nbcon_atomic_unsafe_flush_unsafe() - flush
> 	CON_NBCON_ATOMIC_UNSAFE consoles with con->write_atomic()
> 	and allow unsafe takeover.
> 
> And I would call this 3rd variant in console_flush_on_panic(),
> something like:
> 
> void console_flush_on_panic(enum con_flush_mode mode)
> {
> [...]
> 	printk_get_console_flush_type(&ft);
> 	if (ft.nbcon_atomic) {
> 		nbcon_atomic_flush_pending();
> 		nbcon_atomic_unsafe_flush_unsafe();
> 	}
> 
> 	/* Flush legacy consoles once allowed, even when dangerous. */
> 	if (legacy_allow_panic_sync)
> 		console_flush_all(false, &next_seq, &handover);
> }
> 
> But maybe, this is over engineered and we should just call
> nbcon_atomic_flush_unsafe() in console_flush_on_panic(), like:
> 
> void console_flush_on_panic(enum con_flush_mode mode)
> {
> [...]
> 	printk_get_console_flush_type(&ft);
> 	if (ft.nbcon_atomic) {
> 		nbcon_atomic_flush_pending();
> 		nbcon_atomic_flush_unsafe();
> 	}
> 
> 	/* Flush legacy consoles once allowed, even when dangerous. */
> 	if (legacy_allow_panic_sync)
> 		console_flush_all(false, &next_seq, &handover);
> }

Hi Ryan, are you still planning to send a non-RFC patch? If not, I'm
happy to take over and send a patch with Petr's suggestion.
We are seeing cases in production of panic logs not reaching netconsole
because of this.

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[John Ogness]

Sorry for the late response... this fell through the cracks. Response
below...

On 2026-09-16, Petr Mladek <pmladek@suse.com> wrote:
>> In older kernels (v6.19 and earlier) netconsole is able to print these
>> panic logs while the system is configured to reboot on panic. So from a
>> user perspective this is a regression caused by commit 7eab73b18630
>> ("netconsole: convert to NBCON console infrastructure").
>> 
>> We have automated test systems without BMC access, which rely on these
>> two features working together.
>
> I see. To be precise, I would say that the primary culprit is the
> commit 187de7c212e5fa87 ("printk: nbcon: Allow unsafe write_atomic()
> for panic"). It adds support for consoles which will be flushed
> in panic() only by nbcon_atomic_flush_unsafe().
>
> It is a big difference because printk() tries to flush both
> legacy and "safe" NBCON consoles directly in panic(). In addition,
> they are explicitely flushed by console_flush_on_panic().
>
> console_flush_on_panic() is called more times in panic().
> And it tries to flush legacy consoles the unsafe way even
> before emergency_restart().
>
> OK, we should close this gap. But I suggest another solution.
>
> We already have the following functions for flushing NBCON
> consoles using write_atomic()
>
>
>   + nbcon_atomic_flush_pending() - flush CON_NBCON consoles
> 	using con->write_atomic() but only when safe
>
>   + nbcon_atomic_flush_unsafe() - flush CON_NBCON consoles
> 	using con->write_atomic() and allow unsafe takeover.
>
> I suggest to add 3rd variant, for example:
>
>   + nbcon_atomic_unsafe_flush_unsafe() - flush
> 	CON_NBCON_ATOMIC_UNSAFE consoles with con->write_atomic()
> 	and allow unsafe takeover.

nbcon_atomic_flush_unsafe() already includes CON_NBCON_ATOMIC_UNSAFE.

> And I would call this 3rd variant in console_flush_on_panic(),
> something like:
>
> void console_flush_on_panic(enum con_flush_mode mode)
> {
> [...]
> 	printk_get_console_flush_type(&ft);
> 	if (ft.nbcon_atomic) {
> 		nbcon_atomic_flush_pending();
> 		nbcon_atomic_unsafe_flush_unsafe();

s/nbcon_atomic_unsafe_flush_unsafe/nbcon_atomic_flush_unsafe/

console_flush_on_panic() is used in multiple places and depending on the
user, it may be too risky. Yes, if there are legacy consoles the user is
already taking this risk. But with nbcon we would like to excercise some
risk management.

> If there there are people really concerned about the reboot
> reliability then we might need to make it configurable, e.g.
> by adding reliable_panic command line option which would skip
> the unsafe flush before reboot.

Yes. When originally developing the nbcon, tglx and I talked about many
more details related to this topic. For example, the ability to order
the unsafe flushing. That is one of the reasons that nbcon flushes the
full backlog for each console, rather than each console only flushing a
single record.

If we are going to expose an interface to configure if unsafe flushing
should occur, then we should also consider ordering. Even if we do not
want to build in the ordering support right now, the interface should be
such that it can be easily extended to support this.

*** WARNING: Now we get into my brainstorming, which could be rough. ***

Rather than a big hammer option, I see this more as a per-console
configuration (similar to how we want to go with the console loglevels).

Netconsole is a bit tricky, because it has its own format separate from
console. This would need to be addressed for the per-console loglevels
as well. Maybe it should be extended to something like:

netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][,options]

Which would allow things like:

netconsole=@/,@10.10.0.2/,loglevel:3,unsafe_panic:1

This would also allow to specify that consoles should _not_
perform unsafe flushing:

console=ttyS0,115200,unsafe_panic:0

And it could be used for ordering, flusing from highest value to lowest:

netconsole=@/,@10.10.0.2/,unsafe_panic:2 console=ttyS0,115200,unsafe_panic:12

(First ttyS0 is unsafe-flushed, then the netconsole is unsafe-flushed.)

For consoles with CON_NBCON_ATOMIC_UNSAFE, the default would be
unsafe_panic:0. Otherwise the default would be unsafe_panic:1.

Unfortunately the above idea does not consider the reboot vs. busy-loop
paths. I could imagine there are those that only want a different
ordering/disabling for paths where the kernel will continue to be useful
(for example, a reboot). I think that makes sense. But I am not really
sure how that distinction should be made. We don't want to configure
ourselves to death. Maybe that distinction is not relevant.

John

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Petr Mladek]

On Fri 2026-10-23 10:03:47, John Ogness wrote:
> Sorry for the late response... this fell through the cracks. Response
> below...
> 
> On 2026-09-16, Petr Mladek <pmladek@suse.com> wrote:
> >> In older kernels (v6.19 and earlier) netconsole is able to print these
> >> panic logs while the system is configured to reboot on panic. So from a
> >> user perspective this is a regression caused by commit 7eab73b18630
> >> ("netconsole: convert to NBCON console infrastructure").
> >> 
> >> We have automated test systems without BMC access, which rely on these
> >> two features working together.
> >
> > I see. To be precise, I would say that the primary culprit is the
> > commit 187de7c212e5fa87 ("printk: nbcon: Allow unsafe write_atomic()
> > for panic"). It adds support for consoles which will be flushed
> > in panic() only by nbcon_atomic_flush_unsafe().
> >
> > It is a big difference because printk() tries to flush both
> > legacy and "safe" NBCON consoles directly in panic(). In addition,
> > they are explicitely flushed by console_flush_on_panic().
> >
> > console_flush_on_panic() is called more times in panic().
> > And it tries to flush legacy consoles the unsafe way even
> > before emergency_restart().
> >
> > OK, we should close this gap. But I suggest another solution.
> >
> > We already have the following functions for flushing NBCON
> > consoles using write_atomic()
> >
> >
> >   + nbcon_atomic_flush_pending() - flush CON_NBCON consoles
> > 	using con->write_atomic() but only when safe
> >
> >   + nbcon_atomic_flush_unsafe() - flush CON_NBCON consoles
> > 	using con->write_atomic() and allow unsafe takeover.
> >
> > I suggest to add 3rd variant, for example:
> >
> >   + nbcon_atomic_unsafe_flush_unsafe() - flush
> > 	CON_NBCON_ATOMIC_UNSAFE consoles with con->write_atomic()
> > 	and allow unsafe takeover.
> 
> nbcon_atomic_flush_unsafe() already includes CON_NBCON_ATOMIC_UNSAFE.

I was not clear enoungh. The purpose of the 3rd variant was
to allow the unsafe takeover only for consoles with
CON_NBCON_ATOMIC_UNSAFE. And rely only on the safe atomic flush
for consoles which support it.

But it is likely too complicated and probably not worth it.


> > And I would call this 3rd variant in console_flush_on_panic(),
> > something like:
> >
> > void console_flush_on_panic(enum con_flush_mode mode)
> > {
> > [...]
> > 	printk_get_console_flush_type(&ft);
> > 	if (ft.nbcon_atomic) {
> > 		nbcon_atomic_flush_pending();
> > 		nbcon_atomic_unsafe_flush_unsafe();
> 
> s/nbcon_atomic_unsafe_flush_unsafe/nbcon_atomic_flush_unsafe/

Yes, I am fine with this.

> console_flush_on_panic() is used in multiple places and depending on the
> user, it may be too risky. Yes, if there are legacy consoles the user is
> already taking this risk. But with nbcon we would like to excercise some
> risk management.
> 
> > If there there are people really concerned about the reboot
> > reliability then we might need to make it configurable, e.g.
> > by adding reliable_panic command line option which would skip
> > the unsafe flush before reboot.
> 
> Yes. When originally developing the nbcon, tglx and I talked about many
> more details related to this topic. For example, the ability to order
> the unsafe flushing. That is one of the reasons that nbcon flushes the
> full backlog for each console, rather than each console only flushing a
> single record.
> 
> If we are going to expose an interface to configure if unsafe flushing
> should occur, then we should also consider ordering. Even if we do not
> want to build in the ordering support right now, the interface should be
> such that it can be easily extended to support this.
> 
> *** WARNING: Now we get into my brainstorming, which could be rough. ***
> 
> Rather than a big hammer option, I see this more as a per-console
> configuration (similar to how we want to go with the console loglevels).
> 
> Netconsole is a bit tricky, because it has its own format separate from
> console. This would need to be addressed for the per-console loglevels
> as well. Maybe it should be extended to something like:
> 
> netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][,options]
> 
> Which would allow things like:
> 
> netconsole=@/,@10.10.0.2/,loglevel:3,unsafe_panic:1
> 
> This would also allow to specify that consoles should _not_
> perform unsafe flushing:
> 
> console=ttyS0,115200,unsafe_panic:0
> 
> And it could be used for ordering, flusing from highest value to lowest:
> 
> netconsole=@/,@10.10.0.2/,unsafe_panic:2 console=ttyS0,115200,unsafe_panic:12
> 
> (First ttyS0 is unsafe-flushed, then the netconsole is unsafe-flushed.)
> 
> For consoles with CON_NBCON_ATOMIC_UNSAFE, the default would be
> unsafe_panic:0. Otherwise the default would be unsafe_panic:1.
> 
> Unfortunately the above idea does not consider the reboot vs. busy-loop
> paths. I could imagine there are those that only want a different
> ordering/disabling for paths where the kernel will continue to be useful
> (for example, a reboot). I think that makes sense. But I am not really
> sure how that distinction should be made. We don't want to configure
> ourselves to death. Maybe that distinction is not relevant.

This sounds reasonable. And it is good to have a plan.  Well, I would
add this complexity only when there is a real demand for it :-)

Best Regards,
Petr

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Breno Leitao]

On Fri, Oct 23, 2026 at 12:36:15PM +0200, Petr Mladek wrote:
> On Fri 2026-10-23 10:03:47, John Ogness wrote:
> > Sorry for the late response... this fell through the cracks. Response
> > below...
> > 
> > On 2026-09-16, Petr Mladek <pmladek@suse.com> wrote:
> > >> In older kernels (v6.19 and earlier) netconsole is able to print these
> > >> panic logs while the system is configured to reboot on panic. So from a
> > >> user perspective this is a regression caused by commit 7eab73b18630
> > >> ("netconsole: convert to NBCON console infrastructure").
> > >> 
> > >> We have automated test systems without BMC access, which rely on these
> > >> two features working together.
> > >
> > > I see. To be precise, I would say that the primary culprit is the
> > > commit 187de7c212e5fa87 ("printk: nbcon: Allow unsafe write_atomic()
> > > for panic"). It adds support for consoles which will be flushed
> > > in panic() only by nbcon_atomic_flush_unsafe().
> > >
> > > It is a big difference because printk() tries to flush both
> > > legacy and "safe" NBCON consoles directly in panic(). In addition,
> > > they are explicitely flushed by console_flush_on_panic().
> > >
> > > console_flush_on_panic() is called more times in panic().
> > > And it tries to flush legacy consoles the unsafe way even
> > > before emergency_restart().
> > >
> > > OK, we should close this gap. But I suggest another solution.
> > >
> > > We already have the following functions for flushing NBCON
> > > consoles using write_atomic()
> > >
> > >
> > >   + nbcon_atomic_flush_pending() - flush CON_NBCON consoles
> > > 	using con->write_atomic() but only when safe
> > >
> > >   + nbcon_atomic_flush_unsafe() - flush CON_NBCON consoles
> > > 	using con->write_atomic() and allow unsafe takeover.
> > >
> > > I suggest to add 3rd variant, for example:
> > >
> > >   + nbcon_atomic_unsafe_flush_unsafe() - flush
> > > 	CON_NBCON_ATOMIC_UNSAFE consoles with con->write_atomic()
> > > 	and allow unsafe takeover.
> > 
> > nbcon_atomic_flush_unsafe() already includes CON_NBCON_ATOMIC_UNSAFE.
> 
> I was not clear enoungh. The purpose of the 3rd variant was
> to allow the unsafe takeover only for consoles with
> CON_NBCON_ATOMIC_UNSAFE. And rely only on the safe atomic flush
> for consoles which support it.
> 
> But it is likely too complicated and probably not worth it.
> 
> 
> > > And I would call this 3rd variant in console_flush_on_panic(),
> > > something like:
> > >
> > > void console_flush_on_panic(enum con_flush_mode mode)
> > > {
> > > [...]
> > > 	printk_get_console_flush_type(&ft);
> > > 	if (ft.nbcon_atomic) {
> > > 		nbcon_atomic_flush_pending();
> > > 		nbcon_atomic_unsafe_flush_unsafe();
> > 
> > s/nbcon_atomic_unsafe_flush_unsafe/nbcon_atomic_flush_unsafe/
> 
> Yes, I am fine with this.

This fixes the reboot case, since console_flush_on_panic() runs before
emergency_restart(). 

But I got the feeling that it does not help when a crash kernel is
loaded, which is a common production setup, and how our fleet runs.

With kdump, the kernel kexecs before any of the console_flush_on_panic()
calls in panic():

 1) panic() calls __crash_kexec() before panic_other_cpus_shutdown(),
    or right after kmsg_dump() with crash_kexec_post_notifiers.

 2) On an oops, oops_end() (x86) and die() (arm64) call crash_kexec()
    directly when kexec_should_crash() says so (panic_on_oops, oops in
    interrupt, ...), so panic() never runs. With
    crash_kexec_post_notifiers they go through panic() and hit 1).

So netconsole, or any CON_NBCON_ATOMIC_UNSAFE console, prints nothing
on kdump hosts.

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Breno Leitao]

On Tue, Oct 27, 2026 at 09:12:09AM -0700, Breno Leitao wrote:
> So netconsole, or any CON_NBCON_ATOMIC_UNSAFE console, prints nothing
> on kdump hosts.

For flushing unsafe on nbcon on hosts with kdump environment, this seems
to do the work, but, maybe this is too unsafe !?

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index d0bd2d0cf8990b..9f1a89e301ff01 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -140,6 +140,8 @@ void __noclone __crash_kexec(struct pt_regs *regs)

                        crash_setup_regs(&fixed_regs, regs);
                        crash_save_vmcoreinfo();
+                       /* Flush consoles with an unsafe write_atomic(), e.g. netconsole. */
+                       nbcon_atomic_flush_unsafe();
                        machine_crash_shutdown(&fixed_regs);
                        crash_cma_clear_pending_dma();
                        machine_kexec(kexec_crash_image);

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[John Ogness]

On 2026-10-27, Breno Leitao <leitao@debian.org> wrote:
> On Tue, Oct 27, 2026 at 09:12:09AM -0700, Breno Leitao wrote:
>> So netconsole, or any CON_NBCON_ATOMIC_UNSAFE console, prints nothing
>> on kdump hosts.
>
> For flushing unsafe on nbcon on hosts with kdump environment, this seems
> to do the work, but, maybe this is too unsafe !?

For the general case, it is certainly too unsafe. As I suggested [0] we
need a way to allows users to make this decision. nbcon was designed to
allow for printing policies, but we have not yet provided the interfaces
for the user specify such policies nor the mechanism to enforce the
policies.

Perhaps we need to begin by identifying all the sites where the current
kernel is ending (whether reboot, shutdown, panic, kexec, hang).

> diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> index d0bd2d0cf8990b..9f1a89e301ff01 100644
> --- a/kernel/crash_core.c
> +++ b/kernel/crash_core.c
> @@ -140,6 +140,8 @@ void __noclone __crash_kexec(struct pt_regs *regs)
>
>                         crash_setup_regs(&fixed_regs, regs);
>                         crash_save_vmcoreinfo();
> +                       /* Flush consoles with an unsafe write_atomic(), e.g. netconsole. */
> +                       nbcon_atomic_flush_unsafe();

I am guessing that the CPU is _always_ in emergency state here?
Otherwise the flushing problem exists for all atomic consoles.

>                         machine_crash_shutdown(&fixed_regs);
>                         crash_cma_clear_pending_dma();
>                         machine_kexec(kexec_crash_image);
>

John

[0] https://lore.kernel.org/lkml/87bj8kkfwk.fsf@jogness.linutronix.de/

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Breno Leitao]

On Fri, Oct 23, 2026 at 10:03:47AM +0206, John Ogness wrote:
> *** WARNING: Now we get into my brainstorming, which could be rough. ***
> 
> Rather than a big hammer option, I see this more as a per-console
> configuration (similar to how we want to go with the console loglevels).
> 
> Netconsole is a bit tricky, because it has its own format separate from
> console. This would need to be addressed for the per-console loglevels
> as well. Maybe it should be extended to something like:
> 
> netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][,options]
> 
> Which would allow things like:
> 
> netconsole=@/,@10.10.0.2/,loglevel:3,unsafe_panic:1

This cannot be per target, given that netconsole has multiple targets
(as remote hosts/IPs), but a single registered console.

As of right now, all targets share the same remote, and would have the
same unsafe_panic priority.

> This would also allow to specify that consoles should _not_
> perform unsafe flushing:
> 
> console=ttyS0,115200,unsafe_panic:0
> 
> And it could be used for ordering, flusing from highest value to lowest:
> 
> netconsole=@/,@10.10.0.2/,unsafe_panic:2 console=ttyS0,115200,unsafe_panic:12
> 
> (First ttyS0 is unsafe-flushed, then the netconsole is unsafe-flushed.)
> 
> For consoles with CON_NBCON_ATOMIC_UNSAFE, the default would be
> unsafe_panic:0. Otherwise the default would be unsafe_panic:1.

How do you plan to use 'unsafe_panic' order exactly?

Thanks,
--breno

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Breno Leitao]

On Tue, Oct 27, 2026 at 07:07:30PM +0106, John Ogness wrote:
> On 2026-10-27, Breno Leitao <leitao@debian.org> wrote:
> > On Tue, Oct 27, 2026 at 09:12:09AM -0700, Breno Leitao wrote:

> Perhaps we need to begin by identifying all the sites where the current
> kernel is ending (whether reboot, shutdown, panic, kexec, hang).

Ack. This is what I have so far on the panic side:

 1) panic() ending in the final loop (panic_timeout=0): flushed at the
    end, including the unsafe consoles (d988d9a9b9d180, 5dde3b735413).

 2) panic() ending in emergency_restart(): no unsafe flush (Ryan's RFC).

 3) panic() ending in __crash_kexec(): the console_flush_on_panic() and
    nbcon_atomic_flush_unsafe() calls all come later, with or without
    crash_kexec_post_notifiers.

 4) An oops ending in crash_kexec() straight from oops_end()/die(),
    when kexec_should_crash() is true. panic() never runs.

 5) Other crash_kexec() callers, such as s390 and hypervisors.

> > +                       /* Flush consoles with an unsafe write_atomic(), e.g. netconsole. */
> > +                       nbcon_atomic_flush_unsafe();
>
> I am guessing that the CPU is _always_ in emergency state here?
> Otherwise the flushing problem exists for all atomic consoles.

True. But for the unsafe consoles, the end of panic() already handles
this.

d988d9a9b9d180 ("panic: Flush kernel log buffer at the end") added
a final console_flush_on_panic(), and 5dde3b735413 ("printk: nbcon: Add
unsafe flushing on panic") added nbcon_atomic_flush_unsafe() after it,
to make sure the last messages get out.

That _only_ runs when panic() ends in the final loop, though. kdump never
gets there, and neither does reboot-on-panic or, possibly, some of the
other "kernel ending" sites you mentioned above.

The kdump case is the one I care about most. It is a regression for us
(Meta) in production, where kdump is always armed: netconsole has lost
these messages since it moved to nbcon.

So far we have identified two places that need the same unsafe flush
that 5dde3b735413 added to the final loop:

 a) panic() ending in emergency_restart(). Petr's suggestion to flush
    from console_flush_on_panic() covers this one.

 b) __crash_kexec(), reached from panic() or straight from
    oops_end()/die().

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[John Ogness]

On 2026-10-28, Breno Leitao <leitao@debian.org> wrote:
>> *** WARNING: Now we get into my brainstorming, which could be rough. ***
>> 
>> Rather than a big hammer option, I see this more as a per-console
>> configuration (similar to how we want to go with the console loglevels).
>> 
>> Netconsole is a bit tricky, because it has its own format separate from
>> console. This would need to be addressed for the per-console loglevels
>> as well. Maybe it should be extended to something like:
>> 
>> netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][,options]
>> 
>> Which would allow things like:
>> 
>> netconsole=@/,@10.10.0.2/,loglevel:3,unsafe_panic:1
>
> This cannot be per target, given that netconsole has multiple targets
> (as remote hosts/IPs), but a single registered console.

Well, actually there are two. I can setup a target using extended=0 and
another using extended=1 and then I see both "netconsole hubs": netcon0
and netcon_ext0.

> As of right now, all targets share the same remote, and would have the
> same unsafe_panic priority.

s/all targets/all targets of the same mode/

As you mentioned in the replay thread [0], maybe we should consider
registering a console for each target. I would expect that may even
simplify the "netconsole hub" dance that the netconsole code is
currently doing.

This change should not require any changes to existing netconsole
usage. AFAICT the differences would only be visible in
/proc/consoles. The advantage would be that each target could be
additionally individually configured using the generic
/sys/class/consoles interface for general features like:

loglevel, sync, replay, unsafe_panic

I am not suggesting removing the /sys/kernel/config/netconsole
interface. That works well for setting up the netconsoles. (Although
"extended" should then also migrate (or symlink) to the generic
per-console interface.) The /sys/class/consoles interface is
supplemental for the general console features (and maybe both could
provide symlinks to the other).

>> This would also allow to specify that consoles should _not_
>> perform unsafe flushing:
>> 
>> console=ttyS0,115200,unsafe_panic:0
>> 
>> And it could be used for ordering, flusing from highest value to lowest:
>> 
>> netconsole=@/,@10.10.0.2/,unsafe_panic:2 console=ttyS0,115200,unsafe_panic:12
>> 
>> (First ttyS0 is unsafe-flushed, then the netconsole is unsafe-flushed.)
>> 
>> For consoles with CON_NBCON_ATOMIC_UNSAFE, the default would be
>> unsafe_panic:0. Otherwise the default would be unsafe_panic:1.
>
> How do you plan to use 'unsafe_panic' order exactly?

Currently at the end of panic() an unsafe flush occurs by simply
iterating through the list of consoles. That list ordering does have a
fixed algorithm behind it, but it is quite hard to follow.

By explicitly providing an unsafe ordering, the user could specify the
order of the consoles where the backlog is flushed unsafely. That
ordering may be based on probable reliability (maybe the chances of an
unsafe netconsole flush hanging are less than the chances of an unsafe
serial console flush hanging) or priority (if I only get 1 flush, then I
want it with the netconsole, so try it first).

And the feature could also be used to disable unsafe flushing so that a
particular console _never_ unsafe flushes (maybe the unsafe flushing
with a particular console driver is guaranteed fireworks, so it should
be avoided).

John

[0] https://lore.kernel.org/lkml/auDETo3ot3enCEv_@gmail.com/

Re: [RFC PATCH v1] panic: Flush unsafe consoles before panic reboot

[Breno Leitao]

On Wed, Oct 28, 2026 at 03:43:42PM +0106, John Ogness wrote:
> On 2026-10-28, Breno Leitao <leitao@debian.org> wrote:
> >> For consoles with CON_NBCON_ATOMIC_UNSAFE, the default would be
> >> unsafe_panic:0. Otherwise the default would be unsafe_panic:1.
> >
> > How do you plan to use 'unsafe_panic' order exactly?
> 
> Currently at the end of panic() an unsafe flush occurs by simply
> iterating through the list of consoles. That list ordering does have a
> fixed algorithm behind it, but it is quite hard to follow.

But is that flush invoked at all on the kdump/kexec path?

As far as I can see, nbcon_atomic_flush_unsafe() has a single caller,
at the very end of vpanic(). So the unsafe consoles are only flushed in
one specific "kernel ending" case: panic() with panic_timeout=0 and no
crash kernel loaded.

I think we are mixing two different topics here:

   1) How to set the order of the unsafe flush (per console, from the
      cmdline and/or sysfs).

   2) A hook at the "kernel ending" sites (the panic reboot path,
      __crash_kexec(), ...), so we do not forget to flush the unsafe
      consoles there.

Should we split them? 

The missing hook in 2) is what drops the netconsole panic output today,
while 1) is policy on top of it.

Thanks for the suggestions,
--breno