Re: [RFC PATCH] panic, printk, sys_info: Introduce crash_kexec_in_memory_sys_info

[Petr Mladek <pmladek@suse.com>]

Adding Guilherme and John into Cc.

On Wed 2026-09-16 16:15:46, Aaron Tomlin wrote:
> When investigating kernel panics, capturing post-mortem diagnostic
> telemetry (e.g. memory zone metrics, lock states, active timers, and
> blocked tasks) is vital for root-cause analysis.
>
> While crash_kexec_post_notifiers allows executing panic notifiers and
> sys_info() before jumping to the kdump kernel, it is frequently avoided
> in production environments due to the risk of watchdog timeouts induced
> by synchronous hardware console emission.

There seems to be various motivations to set/clear
crash_kexec_post_notifiers.

Guilherme wanted to add some filtering because some notifiers
were failing, see
see https://lore.kernel.org/all/20220108153451.195121-1-gpiccoli@igalia.com/

I believe that they are called after kdump by default because
the information provided by them is included in the dump.
This idea is supported by the commit f06e5153f4ae2e2f3
("kernel/panic.c: add "crash_kexec_post_notifiers" option
for kdump after panic_notifers").

On the other hand, crash_kexec_post_notifiers is explicitely when
the kernel is running on some hypervisors because the hypervisors
need to get notified about the panic() before crash dump.

All I want to say is that the situation around
crash_kexec_post_notifiers is much more complicated. And I hear
about the watchdog timeouts in this context for the first time.

> To resolve this dilemma, introduce the crash_kexec_in_memory_sys_info
> boot parameter. When enabled, it captures diagnostic telemetry directly
> into the printk ring buffer entirely in RAM before jumping to
> __crash_kexec(), completing in milliseconds rather than tens of seconds.
> 
> To make this safe, fast, and reliable without risking buffer overflow:
>     1.  Scoped console flush suppression
> 
>         Provide printk_suppress_console_flush(bool) to clear the console
>         flush mask in printk_get_console_flush_type(). Messages written
>         via vprintk_store() remain in the printk ring buffer in memory
>         and avoid synchronous hardware console emission and waking
>         kthreads.

This might help when the claim about watchdog reports is true.
I am not sure about it. Anyway, there are other ways how to
prevent watchdogs stepping in (touching them, disabling them, ...)

The console output is important when the crashdump fails.

>      2. Scoped ring buffer tail freezing
> 
>         Introduce printk_freeze_tail(bool) in printk_ringbuffer. When
>         active, desc_push_tail() and data_push_tail() refuse to advance
>         the tail. If diagnostic logging exhausts available ring buffer
>         headroom, new records are safely dropped, guaranteeing that the
>         initial panic Oops, faulting registers, and primary stack trace
>         are never overwritten.

This is another questionable feature. The ring buffer would need to be
super big to hold all messages since the boot. IMHO, servers are
normally running hundreds of days and the log buffer gets rotated,
like the user space logs, ...


>     3.  Execution sequence reordering
> 
>         Reorder __sys_info() so compact, high-signal subsystems (e.g.
>         memory) are collected first, leaving high-volume dumps (all CPU
>         backtraces, full task lists, and ftrace) for last.

This might make sense. I am just afraid that it might be a personal
opinion and we might end up with an endless shuffling here.

My opinion:

IMHO, it does not make much sense to dump sys_info() before kdump
and block consoles. The information is lost when kdump fails.
The information can be extracted from the crashdump when
kdump succeeds.

Best Regards,
Petr