Re: [PATCH v2] stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[Bradley Morgan <brads@mainlining.org>]

On 10 September 2026 04:53:55 BST, Aditya Chillara
<aditya.chillara@oss.qualcomm.com> wrote:
>The cpu stopper thread runs above every other scheduling class, so while a
>stopper callback executes, nothing else on that CPU is scheduled. If such
>a
>callback emits a normal-priority printk(), the legacy console path can
>synchronously flush the pending console backlog. On systems with a slow
>UART and a large backlog, this holds the CPU long enough to starve RT
>kthreads such as the watchdog pet, and for multi_cpu_stop() prevents the
>CPU from advancing the state machine while the other CPUs wait.
Oh!

>The
>resulting delay can prevent watchdog servicing long enough to trigger a
>watchdog bark or bite.
>
>The same can happen from an interrupt taken during the callback:
>multi_cpu_stop() keeps interrupts enabled during MULTI_STOP_PREPARE, and a
>printk() may be emitted from a softirq run on irq exit.
>
>Run CPU stopper callbacks in printk-deferred context to prevent legacy
>console flushes while they execute.
>

Wow! Thanks

Reviewed-by: Bradley Morgan <brads@mainlining.org>


>Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
>---
>A device using a legacy UART console (console=ttyMSM0,115200n8) hit a
>watchdog bark/bite about 40 seconds after boot.
>

Which SOC?

>stop_machine() (used here for kprobe text patching) stops every CPU by
>running multi_cpu_stop() on each of them, through the per-CPU
>"migration/%u" threads. These threads run at a higher priority than the
>msm_watchdog thread. At bite time, all eight CPUs were still spinning in
>multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
>enabled.
>
>Heavy SELinux denial logging had built up a large backlog on the
>console. One CPU took an interrupt while spinning in MULTI_STOP_PREPARE.
>Handling it eventually led to a printk(), and because the console was a
>legacy console, that printk() synchronously drained the whole backlog
>over the slow UART. While the drain was still running, the watchdog bark
>interrupt hit the same CPU, found no recent pet, and escalated to a
>bite.
>
>The captured stack for that CPU, innermost frame first:
>
>  qcom_soc_set_wdt_bite
>  qcom_wdt_bark_handler
>  __handle_irq_event_percpu
>  handle_irq_event
>  handle_fasteoi_irq
>  generic_handle_domain_irq
>  gic_handle_irq
>  do_interrupt_handler
>  el1_interrupt
>  el1h_64_irq_handler
>  el1h_64_irq
>  console_flush_all
>  console_unlock
>  vprintk_emit
>  dev_vprintk_emit
>  dev_printk_emit
>  __dev_printk
>  _dev_err
>  btspi_sleep_timeout_handler
>  call_timer_fn
>  __run_timer_base
>  run_timer_softirq
>  handle_softirqs
>  __do_softirq
>  ____do_softirq
>  call_on_irq_stack
>  do_softirq_own_stack
>  __irq_exit_rcu
>  irq_exit_rcu
>  el1_interrupt
>  el1h_64_irq_handler
>  el1h_64_irq
>  multi_cpu_stop
>  cpu_stopper_thread
>  smpboot_thread_fn
>  kthread
>  ret_from_fork

Same question as above, but a added "Is this a modern SOC?"

>
>Every other CPU stayed parked in the rendezvous the whole time, since
>their stopper threads outrank msm_watchdog. Nothing could pet the
>watchdog until the drain finished.
>
>This was observed through multi_cpu_stop(), but the hazard is not
>specific to it. Every cpu stopper callback runs in stop_sched_class,
>above msm_watchdog and every other thread on the CPU, so a slow flush
>from any of them (including single-CPU callbacks such as the migration
>and task-migration stoppers) can starve the watchdog just as well. The
>fix therefore covers all stopper callbacks, not only multi_cpu_stop().
>
>Fix this by running the CPU stopper callbacks in printk-deferred context.
>
>Reproduced and verified with an out-of-tree test module that triggers
>stop_machine() with a queued console backlog and a printk() inside the
>rendezvous, paired with a kprobe-based script that flags any console
>flush happening while a CPU is inside a stopper callback.

Hmm, interesting, could you provide what you did? (Not in the description,
but to me so I can have a look at it)


>---
>Changes in v2:
>- Use printk_deferred_enter/exit() to defer legacy console flushes instead
>  of in_cpu_stop().
>- Link to v1: https://patch.msgid.link/20260827-defer-legacy-console-write-on-multi_cpu_stop-v1-0-3b9f6bb4679f@oss.qualcomm.com
>---
> kernel/stop_machine.c | 2 ++
> 1 file changed, 2 insertions(+)
>
>diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
>index d085ba1f4b44..31f7af41249f 100644
>--- a/kernel/stop_machine.c
>+++ b/kernel/stop_machine.c
>@@ -507,7 +507,9 @@ static void cpu_stopper_thread(unsigned int cpu)
> 		stopper->caller = work->caller;
> 		stopper->fn = fn;
> 		preempt_count_inc();
>+		printk_deferred_enter();
> 		ret = fn(arg);
>+		printk_deferred_exit();

Clean! :)

> 		if (done) {
> 			if (ret)
> 				done->ret = ret;
>
>---
>base-commit: 77ae27fd98f3b548797c9f22c10ab5cf1c4ada53
>change-id: 20260824-defer-legacy-console-write-on-multi_cpu_stop-d3ef6f6b150b
>
>Best regards,
>--  
>Aditya Chillara <aditya.chillara@oss.qualcomm.com>
>
>
>

--- Thanks!
https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/