[PATCH 0/2] printk/stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[PATCH 0/2] printk/stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[Aditya Chillara]

A device using a legacy UART console (console=ttyMSM0,115200n8) hit a
watchdog bark/bite about 40 seconds after boot.

stop_machine() (used here for kprobe text patching) stops every CPU by
running multi_cpu_stop() on each of them, through the per-CPU
"migration/%u" threads. These threads run at a higher priority than the
msm_watchdog thread. At bite time, all eight CPUs were still spinning in
multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
enabled.

Heavy SELinux denial logging had built up a large backlog on the
console. One CPU took an interrupt while spinning in MULTI_STOP_PREPARE.
Handling it eventually led to a printk(), and because the console was a
legacy console, that printk() synchronously drained the whole backlog
over the slow UART. While the drain was still running, the watchdog bark
interrupt hit the same CPU, found no recent pet, and escalated to a
bite.

The captured stack for that CPU, innermost frame first:

  qcom_soc_set_wdt_bite
  qcom_wdt_bark_handler
  __handle_irq_event_percpu
  handle_irq_event
  handle_fasteoi_irq
  generic_handle_domain_irq
  gic_handle_irq
  do_interrupt_handler
  el1_interrupt
  el1h_64_irq_handler
  el1h_64_irq
  console_flush_all
  console_unlock
  vprintk_emit
  dev_vprintk_emit
  dev_printk_emit
  __dev_printk
  _dev_err
  btspi_sleep_timeout_handler
  call_timer_fn
  __run_timer_base
  run_timer_softirq
  handle_softirqs
  __do_softirq
  ____do_softirq
  call_on_irq_stack
  do_softirq_own_stack
  __irq_exit_rcu
  irq_exit_rcu
  el1_interrupt
  el1h_64_irq_handler
  el1h_64_irq
  multi_cpu_stop
  cpu_stopper_thread
  smpboot_thread_fn
  kthread
  ret_from_fork

Every other CPU stayed parked in the rendezvous the whole time, since
their stopper threads outrank msm_watchdog. Nothing could pet the
watchdog until the drain finished.

This was observed through multi_cpu_stop(), but the hazard is not
specific to it. Every cpu stopper callback runs in stop_sched_class,
above msm_watchdog and every other thread on the CPU, so a slow flush
from any of them (including single-CPU callbacks such as the migration
and task-migration stoppers) can starve the watchdog just as well. The
fix therefore covers all stopper callbacks, not only multi_cpu_stop().

Fix this by having the cpu stopper mark the CPU active while a callback
runs, and having printk use that marker to defer legacy console flushes
until the callback returns:

  1/2 stop_machine: Track when a CPU executes a stopper callback

      Add a per-CPU flag, set in the stopper dispatch path around the
      callback, and an in_cpu_stop() accessor.

  2/2 printk: Defer legacy console flushes while a CPU runs a stopper callback

      Route legacy console output through the offload path instead of
      flushing it directly while a CPU is inside a stopper callback, and
      flush it once the callback returns. Emergency and panic output is
      unaffected.

Reproduced and verified with an out-of-tree test module that triggers
stop_machine() with a queued console backlog and a printk() inside the
rendezvous, paired with a kprobe-based script that flags any console
flush happening while a CPU is inside a stopper callback.

Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
---
Aditya Chillara (2):
      stop_machine: Track when a CPU executes a stopper callback
      printk: Defer legacy console flushes while a CPU runs a stopper callback

 include/linux/printk.h       |  5 +++++
 include/linux/stop_machine.h | 13 +++++++++++++
 kernel/printk/internal.h     |  9 ++++++++-
 kernel/printk/printk.c       | 23 +++++++++++++++++++++++
 kernel/stop_machine.c        | 29 +++++++++++++++++++++++++++++
 5 files changed, 78 insertions(+), 1 deletion(-)
---
base-commit: 77ae27fd98f3b548797c9f22c10ab5cf1c4ada53
change-id: 20260824-defer-legacy-console-write-on-multi_cpu_stop-d3ef6f6b150b

Best regards,
--  
Aditya Chillara <aditya.chillara@oss.qualcomm.com>

[PATCH 1/2] stop_machine: Track when a CPU executes a stopper callback

[Aditya Chillara]

The cpu stopper thread runs in stop_sched_class, above every other
scheduling class. While a stopper callback executes, nothing else on that
CPU is scheduled, including RT kthreads such as a watchdog pet thread. A
callback that performs a long, slow operation can therefore starve those
threads, and for multi_cpu_stop() stall the whole state machine. The same
is true of an interrupt taken during the callback: multi_cpu_stop() keeps
interrupts enabled during MULTI_STOP_PREPARE, and softirqs may run on irq
exit.

Add a per-CPU flag covering execution of a stopper callback, set in the
dispatch path before the callback runs and cleared after it returns, and
expose in_cpu_stop() so callers can detect this context and defer
operations that could hold the CPU.

Assisted-by: LLM
Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
---
 include/linux/stop_machine.h | 13 +++++++++++++
 kernel/stop_machine.c        | 22 ++++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 84e7fb627ba4..ae71a829f036 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -38,6 +38,14 @@ void stop_machine_park(int cpu);
 void stop_machine_unpark(int cpu);
 void stop_machine_yield(const struct cpumask *cpumask);
 
+/*
+ * True while the current CPU is executing a cpu stopper callback. The
+ * stopper runs above every other scheduling class, so nothing else on the
+ * CPU is scheduled until the callback returns. Callers use this to defer
+ * work that could hold the CPU too long (e.g. slow console flushes).
+ */
+bool in_cpu_stop(void);
+
 extern void print_stop_info(const char *log_lvl, struct task_struct *task);
 
 #else	/* CONFIG_SMP */
@@ -84,6 +92,11 @@ static inline void stop_one_cpu_nowait(unsigned int cpu,
 
 static inline void print_stop_info(const char *log_lvl, struct task_struct *task) { }
 
+static inline bool in_cpu_stop(void)
+{
+	return false;
+}
+
 #endif	/* CONFIG_SMP */
 
 /*
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index d085ba1f4b44..d38fabb0b9f1 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -48,8 +48,21 @@ struct cpu_stopper {
 };
 
 static DEFINE_PER_CPU(struct cpu_stopper, cpu_stopper);
+static DEFINE_PER_CPU(bool, cpu_stop_active);
 static bool stop_machine_initialized = false;
 
+bool in_cpu_stop(void)
+{
+	bool ret;
+
+	preempt_disable();
+	ret = READ_ONCE(*this_cpu_ptr(&cpu_stop_active));
+	preempt_enable();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(in_cpu_stop);
+
 void print_stop_info(const char *log_lvl, struct task_struct *task)
 {
 	/*
@@ -507,7 +520,16 @@ static void cpu_stopper_thread(unsigned int cpu)
 		stopper->caller = work->caller;
 		stopper->fn = fn;
 		preempt_count_inc();
+
+		/*
+		 * Set before the callback runs so an interrupt taken during it,
+		 * and any softirq run on irq exit, sees the CPU as inside a
+		 * stopper callback.
+		 */
+		this_cpu_write(cpu_stop_active, true);
 		ret = fn(arg);
+		this_cpu_write(cpu_stop_active, false);
+
 		if (done) {
 			if (ret)
 				done->ret = ret;

-- 
2.34.1

[PATCH 2/2] printk: Defer legacy console flushes while a CPU runs a stopper callback

[Aditya Chillara]

The cpu stopper thread runs above every other scheduling class, so while a
stopper callback executes, nothing else on that CPU is scheduled. If such a
callback emits a normal-priority printk(), the legacy console path can
synchronously flush the pending console backlog. On systems with a slow
UART and a large backlog, this holds the CPU long enough to starve RT
kthreads such as the watchdog pet, and for multi_cpu_stop() prevents the
CPU from advancing the state machine while the other CPUs wait. The
resulting delay can prevent watchdog servicing long enough to trigger a
watchdog bark or bite.

The same can happen from an interrupt taken during the callback:
multi_cpu_stop() keeps interrupts enabled during MULTI_STOP_PREPARE, and a
printk() may be emitted from a softirq run on irq exit.

Defer normal-priority legacy console output while a CPU is executing a
stopper callback (in_cpu_stop()), and reschedule any pending output after
the callback exits. Keep emergency and panic console behavior unchanged.

Assisted-by: LLM
Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
---
 include/linux/printk.h   |  5 +++++
 kernel/printk/internal.h |  9 ++++++++-
 kernel/printk/printk.c   | 23 +++++++++++++++++++++++
 kernel/stop_machine.c    |  7 +++++++
 4 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/include/linux/printk.h b/include/linux/printk.h
index f594c1266bfd..4f3fb4ef5b92 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -166,6 +166,7 @@ __printf(1, 2) __cold int _printk_deferred(const char *fmt, ...);
 
 extern void __printk_deferred_enter(void);
 extern void __printk_deferred_exit(void);
+void printk_defer_console_output(void);
 
 extern void printk_force_console_enter(void);
 extern void printk_force_console_exit(void);
@@ -239,6 +240,10 @@ static inline void printk_deferred_exit(void)
 {
 }
 
+static inline void printk_defer_console_output(void)
+{
+}
+
 static inline void printk_force_console_enter(void)
 {
 }
diff --git a/kernel/printk/internal.h b/kernel/printk/internal.h
index 85fbf1801cbe..8c02f93e798c 100644
--- a/kernel/printk/internal.h
+++ b/kernel/printk/internal.h
@@ -3,6 +3,7 @@
  * internal.h - printk internal definitions
  */
 #include <linux/console.h>
+#include <linux/stop_machine.h>
 #include <linux/types.h>
 #include <linux/sysctl.h>
 
@@ -206,7 +207,13 @@ static inline void printk_get_console_flush_type(struct console_flush_type *ft)
 
 		/* Legacy consoles are flushed directly when possible. */
 		if (have_legacy_console || have_boot_console) {
-			if (!is_printk_legacy_deferred())
+			/*
+			 * Offload while this CPU runs a stopper callback so that a
+			 * slow flush cannot starve RT kthreads (e.g. the watchdog
+			 * pet) or stall the multi_cpu_stop() state machine.
+			 * Flushed once the stopper exits.
+			 */
+			if (!is_printk_legacy_deferred() && !in_cpu_stop())
 				ft->legacy_direct = true;
 			else if (!console_irqwork_blocked)
 				ft->legacy_offload = true;
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 3fcdf4b4e2e5..379fcbe6914c 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -49,6 +49,7 @@
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
 #include <linux/panic.h>
+#include <linux/stop_machine.h>
 
 #include <linux/uaccess.h>
 #include <asm/sections.h>
@@ -4602,6 +4603,16 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
 {
 	int pending = this_cpu_xchg(printk_pending, 0);
 
+	/*
+	 * Don't flush the legacy console from irq_work while this CPU runs a
+	 * stopper callback; keep the bit pending and re-run once it exits (see
+	 * cpu_stopper_thread()).
+	 */
+	if ((pending & PRINTK_PENDING_OUTPUT) && in_cpu_stop()) {
+		this_cpu_or(printk_pending, PRINTK_PENDING_OUTPUT);
+		pending &= ~PRINTK_PENDING_OUTPUT;
+	}
+
 	if (pending & PRINTK_PENDING_OUTPUT) {
 		if (force_legacy_kthread()) {
 			if (printk_legacy_kthread)
@@ -4687,6 +4698,18 @@ void defer_console_output(void)
 	__wake_up_klogd(PRINTK_PENDING_WAKEUP | PRINTK_PENDING_OUTPUT);
 }
 
+void printk_defer_console_output(void)
+{
+	bool pending_output;
+
+	preempt_disable();
+	pending_output = this_cpu_read(printk_pending) & PRINTK_PENDING_OUTPUT;
+	preempt_enable();
+
+	if (pending_output)
+		defer_console_output();
+}
+
 /**
  * printk_trigger_flush - Attempt to flush printk buffer to consoles.
  *
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index d38fabb0b9f1..18ae7ff2d8b8 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -22,6 +22,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/printk.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -530,6 +531,12 @@ static void cpu_stopper_thread(unsigned int cpu)
 		ret = fn(arg);
 		this_cpu_write(cpu_stop_active, false);
 
+		/*
+		 * Flush console output that was deferred while the callback ran,
+		 * now that it is safe to do so from this CPU.
+		 */
+		printk_defer_console_output();
+
 		if (done) {
 			if (ret)
 				done->ret = ret;

-- 
2.34.1

Re: [PATCH 0/2] printk/stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[John Ogness]

On 2026-08-27, Aditya Chillara <aditya.chillara@oss.qualcomm.com> wrote:
> A device using a legacy UART console (console=ttyMSM0,115200n8) hit a
> watchdog bark/bite about 40 seconds after boot.
>
> stop_machine() (used here for kprobe text patching) stops every CPU by
> running multi_cpu_stop() on each of them, through the per-CPU
> "migration/%u" threads. These threads run at a higher priority than the
> msm_watchdog thread. At bite time, all eight CPUs were still spinning in
> multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
> enabled.
>
> Heavy SELinux denial logging had built up a large backlog on the
> console. One CPU took an interrupt while spinning in MULTI_STOP_PREPARE.
> Handling it eventually led to a printk(), and because the console was a
> legacy console, that printk() synchronously drained the whole backlog
> over the slow UART. While the drain was still running, the watchdog bark
> interrupt hit the same CPU, found no recent pet, and escalated to a
> bite.
>
> The captured stack for that CPU, innermost frame first:
>
>   qcom_soc_set_wdt_bite
>   qcom_wdt_bark_handler
>   __handle_irq_event_percpu
>   handle_irq_event
>   handle_fasteoi_irq
>   generic_handle_domain_irq
>   gic_handle_irq
>   do_interrupt_handler
>   el1_interrupt
>   el1h_64_irq_handler
>   el1h_64_irq
>   console_flush_all
>   console_unlock
>   vprintk_emit
>   dev_vprintk_emit
>   dev_printk_emit
>   __dev_printk
>   _dev_err
>   btspi_sleep_timeout_handler
>   call_timer_fn
>   __run_timer_base
>   run_timer_softirq
>   handle_softirqs
>   __do_softirq
>   ____do_softirq
>   call_on_irq_stack
>   do_softirq_own_stack
>   __irq_exit_rcu
>   irq_exit_rcu
>   el1_interrupt
>   el1h_64_irq_handler
>   el1h_64_irq
>   multi_cpu_stop
>   cpu_stopper_thread
>   smpboot_thread_fn
>   kthread
>   ret_from_fork
>
> Every other CPU stayed parked in the rendezvous the whole time, since
> their stopper threads outrank msm_watchdog. Nothing could pet the
> watchdog until the drain finished.
>
> This was observed through multi_cpu_stop(), but the hazard is not
> specific to it. Every cpu stopper callback runs in stop_sched_class,
> above msm_watchdog and every other thread on the CPU, so a slow flush
> from any of them (including single-CPU callbacks such as the migration
> and task-migration stoppers) can starve the watchdog just as well. The
> fix therefore covers all stopper callbacks, not only multi_cpu_stop().
>
> Fix this by having the cpu stopper mark the CPU active while a callback
> runs, and having printk use that marker to defer legacy console flushes
> until the callback returns:
>
>   1/2 stop_machine: Track when a CPU executes a stopper callback
>
>       Add a per-CPU flag, set in the stopper dispatch path around the
>       callback, and an in_cpu_stop() accessor.
>
>   2/2 printk: Defer legacy console flushes while a CPU runs a stopper callback
>
>       Route legacy console output through the offload path instead of
>       flushing it directly while a CPU is inside a stopper callback, and
>       flush it once the callback returns. Emergency and panic output is
>       unaffected.
>
> Reproduced and verified with an out-of-tree test module that triggers
> stop_machine() with a queued console backlog and a printk() inside the
> rendezvous, paired with a kprobe-based script that flags any console
> flush happening while a CPU is inside a stopper callback.

This is a fairly heavy series just to address a problem with legacy
consoles that has always existed.

How about instead a series to switch msm_serial.c/qcom_geni_serial.c
over to NBCON? That is fairly straightforward (especially with the
availability of CON_NBCON_ATOMIC_UNSAFE) and would help to move the
kernel forward rather than improving the parts we are trying to get rid
of.

I would even be willing to convert those 2 drivers if you could provide
the necessary testing for me.

John

Re: [PATCH 0/2] printk/stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[John Ogness]

Hi Aditya,

On 2026-08-27, Aditya Chillara <aditya.chillara@oss.qualcomm.com> wrote:
> A device using a legacy UART console (console=ttyMSM0,115200n8) hit a
> watchdog bark/bite about 40 seconds after boot.
>
> stop_machine() (used here for kprobe text patching) stops every CPU by
> running multi_cpu_stop() on each of them, through the per-CPU
> "migration/%u" threads. These threads run at a higher priority than the
> msm_watchdog thread. At bite time, all eight CPUs were still spinning in
> multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
> enabled.
>
> Heavy SELinux denial logging had built up a large backlog on the
> console. One CPU took an interrupt while spinning in MULTI_STOP_PREPARE.
> Handling it eventually led to a printk(), and because the console was a
> legacy console, that printk() synchronously drained the whole backlog
> over the slow UART. While the drain was still running, the watchdog bark
> interrupt hit the same CPU, found no recent pet, and escalated to a
> bite.
>
> The captured stack for that CPU, innermost frame first:
>
>   qcom_soc_set_wdt_bite
>   qcom_wdt_bark_handler
>   __handle_irq_event_percpu
>   handle_irq_event
>   handle_fasteoi_irq
>   generic_handle_domain_irq
>   gic_handle_irq
>   do_interrupt_handler
>   el1_interrupt
>   el1h_64_irq_handler
>   el1h_64_irq
>   console_flush_all
>   console_unlock
>   vprintk_emit
>   dev_vprintk_emit
>   dev_printk_emit
>   __dev_printk
>   _dev_err
>   btspi_sleep_timeout_handler
>   call_timer_fn
>   __run_timer_base
>   run_timer_softirq
>   handle_softirqs
>   __do_softirq
>   ____do_softirq
>   call_on_irq_stack
>   do_softirq_own_stack
>   __irq_exit_rcu
>   irq_exit_rcu
>   el1_interrupt
>   el1h_64_irq_handler
>   el1h_64_irq
>   multi_cpu_stop
>   cpu_stopper_thread
>   smpboot_thread_fn
>   kthread
>   ret_from_fork
>
> Every other CPU stayed parked in the rendezvous the whole time, since
> their stopper threads outrank msm_watchdog. Nothing could pet the
> watchdog until the drain finished.
>
> This was observed through multi_cpu_stop(), but the hazard is not
> specific to it. Every cpu stopper callback runs in stop_sched_class,
> above msm_watchdog and every other thread on the CPU, so a slow flush
> from any of them (including single-CPU callbacks such as the migration
> and task-migration stoppers) can starve the watchdog just as well. The
> fix therefore covers all stopper callbacks, not only multi_cpu_stop().
>
> Fix this by having the cpu stopper mark the CPU active while a callback
> runs, and having printk use that marker to defer legacy console flushes
> until the callback returns:
>
>   1/2 stop_machine: Track when a CPU executes a stopper callback
>
>       Add a per-CPU flag, set in the stopper dispatch path around the
>       callback, and an in_cpu_stop() accessor.
>
>   2/2 printk: Defer legacy console flushes while a CPU runs a stopper callback
>
>       Route legacy console output through the offload path instead of
>       flushing it directly while a CPU is inside a stopper callback, and
>       flush it once the callback returns. Emergency and panic output is
>       unaffected.
>
> Reproduced and verified with an out-of-tree test module that triggers
> stop_machine() with a queued console backlog and a printk() inside the
> rendezvous, paired with a kprobe-based script that flags any console
> flush happening while a CPU is inside a stopper callback.

Generally speaking, we are not taking the whack-a-mole approach to
workaround all the known legacy console problems (there are a lot of
them!). However, if there are problems that occur during normal usage
(as opposed to crafted tests), then we can insert workarounds.

For workarounds of known legacy console problems we have the deferred
enter/exit functions. These only affect legacy consoles and literally
exist for these purposes. I would expect the following patch would also
solve your problem.

John Ogness

diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index d085ba1f4b44e..31f7af41249f1 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -507,7 +507,9 @@ static void cpu_stopper_thread(unsigned int cpu)
 		stopper->caller = work->caller;
 		stopper->fn = fn;
 		preempt_count_inc();
+		printk_deferred_enter();
 		ret = fn(arg);
+		printk_deferred_exit();
 		if (done) {
 			if (ret)
 				done->ret = ret;

Re: [PATCH 0/2] printk/stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[Aditya Chillara]

On 8/28/2026 2:24 PM, John Ogness wrote:
> Hi Aditya,
> 
> On 2026-08-27, Aditya Chillara <aditya.chillara@oss.qualcomm.com> wrote:
>> A device using a legacy UART console (console=ttyMSM0,115200n8) hit a
>> watchdog bark/bite about 40 seconds after boot.
>>
>> stop_machine() (used here for kprobe text patching) stops every CPU by
>> running multi_cpu_stop() on each of them, through the per-CPU
>> "migration/%u" threads. These threads run at a higher priority than the
>> msm_watchdog thread. At bite time, all eight CPUs were still spinning in
>> multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
>> enabled.
>>
>> Heavy SELinux denial logging had built up a large backlog on the
>> console. One CPU took an interrupt while spinning in MULTI_STOP_PREPARE.
>> Handling it eventually led to a printk(), and because the console was a
>> legacy console, that printk() synchronously drained the whole backlog
>> over the slow UART. While the drain was still running, the watchdog bark
>> interrupt hit the same CPU, found no recent pet, and escalated to a
>> bite.
>>
>> The captured stack for that CPU, innermost frame first:
>>
>>   qcom_soc_set_wdt_bite
>>   qcom_wdt_bark_handler
>>   __handle_irq_event_percpu
>>   handle_irq_event
>>   handle_fasteoi_irq
>>   generic_handle_domain_irq
>>   gic_handle_irq
>>   do_interrupt_handler
>>   el1_interrupt
>>   el1h_64_irq_handler
>>   el1h_64_irq
>>   console_flush_all
>>   console_unlock
>>   vprintk_emit
>>   dev_vprintk_emit
>>   dev_printk_emit
>>   __dev_printk
>>   _dev_err
>>   btspi_sleep_timeout_handler
>>   call_timer_fn
>>   __run_timer_base
>>   run_timer_softirq
>>   handle_softirqs
>>   __do_softirq
>>   ____do_softirq
>>   call_on_irq_stack
>>   do_softirq_own_stack
>>   __irq_exit_rcu
>>   irq_exit_rcu
>>   el1_interrupt
>>   el1h_64_irq_handler
>>   el1h_64_irq
>>   multi_cpu_stop
>>   cpu_stopper_thread
>>   smpboot_thread_fn
>>   kthread
>>   ret_from_fork
>>
>> Every other CPU stayed parked in the rendezvous the whole time, since
>> their stopper threads outrank msm_watchdog. Nothing could pet the
>> watchdog until the drain finished.
>>
>> This was observed through multi_cpu_stop(), but the hazard is not
>> specific to it. Every cpu stopper callback runs in stop_sched_class,
>> above msm_watchdog and every other thread on the CPU, so a slow flush
>> from any of them (including single-CPU callbacks such as the migration
>> and task-migration stoppers) can starve the watchdog just as well. The
>> fix therefore covers all stopper callbacks, not only multi_cpu_stop().
>>
>> Fix this by having the cpu stopper mark the CPU active while a callback
>> runs, and having printk use that marker to defer legacy console flushes
>> until the callback returns:
>>
>>   1/2 stop_machine: Track when a CPU executes a stopper callback
>>
>>       Add a per-CPU flag, set in the stopper dispatch path around the
>>       callback, and an in_cpu_stop() accessor.
>>
>>   2/2 printk: Defer legacy console flushes while a CPU runs a stopper callback
>>
>>       Route legacy console output through the offload path instead of
>>       flushing it directly while a CPU is inside a stopper callback, and
>>       flush it once the callback returns. Emergency and panic output is
>>       unaffected.
>>
>> Reproduced and verified with an out-of-tree test module that triggers
>> stop_machine() with a queued console backlog and a printk() inside the
>> rendezvous, paired with a kprobe-based script that flags any console
>> flush happening while a CPU is inside a stopper callback.
> 
> Generally speaking, we are not taking the whack-a-mole approach to
> workaround all the known legacy console problems (there are a lot of
> them!). However, if there are problems that occur during normal usage
> (as opposed to crafted tests), then we can insert workarounds.
> 
> For workarounds of known legacy console problems we have the deferred
> enter/exit functions. These only affect legacy consoles and literally
> exist for these purposes. I would expect the following patch would also
> solve your problem.

Yes, this fixes the issue.

> 
> John Ogness
> 
> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index d085ba1f4b44e..31f7af41249f1 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -507,7 +507,9 @@ static void cpu_stopper_thread(unsigned int cpu)
>  		stopper->caller = work->caller;
>  		stopper->fn = fn;
>  		preempt_count_inc();
> +		printk_deferred_enter();
>  		ret = fn(arg);
> +		printk_deferred_exit();
>  		if (done) {
>  			if (ret)
>  				done->ret = ret;

Tested-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>

Thank you,
Aditya