mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH] sched/rt: Quiesce RTO push before stopping CPUs on shutdown
@ 2026-11-16  9:20 wangqk1
  0 siblings, 0 replies; only message in thread
From: wangqk1 @ 2026-11-16  9:20 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: linux-kernel, xiedl3


On arm64, machine_power_off(), machine_halt(), and machine_restart()
call smp_send_stop() without running CPU hotplug teardown. A stopped CPU
clears its online bit and parks, while its RT overload bit can remain in
root_domain::rto_mask.

An RTO push relay still in flight can select that stale bit and call
irq_work_queue_on() for the parked CPU, triggering
WARN_ON_ONCE(cpu_is_offline(cpu)). This was observed during normal
poweroff of a PREEMPT_RT kernel under RT load, about 5 ms after the
"Power down" message:

  irq_work_queue_on+0x138/0x148
  rto_push_irq_work_func+0x98
  irq_work_single
  generic_smp_call_function_single_interrupt

A cpu_online() check in rto_next_cpu() cannot close this race because
the target can stop after the check but before irq_work_queue_on().
Checking a shutdown flag only in rto_next_cpu() has the same problem:
a sender can be delayed after the check and resume after another CPU
has been stopped.

Quiesce RTO push from a syscore shutdown hook, before the machine
specific shutdown path can stop any CPU. rto_next_cpu() checks
rto_push_stopped for early termination, but correctness is enforced at
both actual queueing sites. tell_cpu_to_push() starts a relay chain, and
rto_push_irq_work_func() relays work to the next overloaded CPU.

Both sites use rto_push_queue(), which holds a global raw spinlock
across the final rto_push_stopped check and irq_work_queue_on(). The
shutdown hook takes the same lock before setting rto_push_stopped.

This gives two possible interleavings. If a sender gets the lock first,
queueing completes before the shutdown hook returns, hence before
smp_send_stop() can take the target offline. If shutdown gets the lock
first, the sender observes rto_push_stopped and does not queue. A raw
spinlock is required because the relay runs in hardirq context and must
also work on PREEMPT_RT. irq_work_queue_on() only claims and queues the
work and, for a remote CPU, sends an IPI. It does not wait for the remote
callback, so the lock critical section has no callback wait.

When the initial queue operation is rejected, tell_cpu_to_push() drops
the root_domain reference it just acquired. When a relay is rejected,
rto_push_irq_work_func() drops the reference owned by the chain. The
existing cpu < 0 path continues to release that reference as before.

There is no root_domain enumeration and no irq_work_sync(). Enumeration
through cpu_rq(cpu)->rd can miss an old root_domain that cpuset partition
changes detached from every runqueue while an in-flight chain still
holds a reference. Waiting for hard irq_work with irq_work_sync() also
introduces an unbounded shutdown wait. The final queueing gate covers
old and new root_domains without either dependency.

An RCU alternative was considered. The final check and queue operation
could run inside rcu_read_lock(), with shutdown setting
rto_push_stopped and then calling synchronize_rcu(). The grace period
would ensure that a sender which already passed the check finishes
queueing before any CPU is stopped.

RCU was not chosen because shutdown would then depend on a system-wide
RCU grace period, including unrelated readers and quiescent-state
progress, rather than only on short RTO queueing critical sections.
A delayed or stalled grace period would delay poweroff, halt, restart,
or kexec. The lock has the narrower liveness dependency and does not
wait for remote callbacks. Using only READ_ONCE(), an atomic flag, or a
static branch without a grace period or equivalent sender
synchronization would leave the original check-to-queue race open.

Each initial or relayed queue operation takes a global raw spinlock only
around the final flag check and irq_work_queue_on(), not around target
scanning, RT task pushing, or remote callback execution.

A/B testing on a 20-CPU arm64 6.18.14-rt4 PREEMPT_RT system used an RT
overload workload. The A/B latency differences were smaller than the
A/A run-to-run variation and could not be attributed to the patch.
These measurements do not isolate the cost of the lock or establish
zero overhead. Testing covered only the default root_domain.

This change prevents new RTO work from being queued after the shutdown
hook. It does not drain work that was already queued and does not clear
rto_mask. Work left on a CPU that is subsequently parked need not run
because the system is terminating. Panic and crash kexec do not pass
through syscore_shutdown() and retain their existing behavior.

Link: https://lore.kernel.org/linux-kernel/20261013030140.682182-2-wangqk1@xiaopeng.com/
Signed-off-by: wangqk1 <wangqk1@xiaopeng.com>
---
Test details:

The equivalent change was A/B tested on a 20-CPU arm64 6.18.14-rt4
PREEMPT_RT system, with five 60-second runs per kernel:

  cyclictest -t 80 -p 90 -i 50 -d 0 -D 60 -m -h 400 -q
  while :; do hackbench -g 50 -l 5000 -f 25; done  # background load

  Latency (us)          Without patch    With patch       Delta
  Median min                   1.137          1.163       +2.20%
  Median avg                   4.500          4.575       +1.67%
  Median max                  92.000         92.000        0.00%
  Worst max                   95             94          -1.05%

  rto_push_irq_work_func  Without patch    With patch
  Probe hits / 10.16 s              8372         11692
  Probe hits / second                824          1151

  A/A median avg drift, first two vs last two runs: 4.14%

The test binaries were identical, and probes were removed before the
formal runs with tracing disabled. The test did not cover concurrent
cpuset partitions.

This mainline version was build-tested with x86_64 defconfig as
kernel/sched/rt.o. No runtime testing was performed on this mainline
tree. The measurements above used the equivalent arm64 6.18 change.

 kernel/sched/rt.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 70 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 1535046a23ff..5431ca7dc7e9 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -4,6 +4,8 @@
  * policies)
  */
 
+#include <linux/syscore_ops.h>
+
 #include "sched.h"
 #include "pelt.h"
 
@@ -2087,6 +2089,17 @@ static void push_rt_tasks(struct rq *rq)
 
 #ifdef HAVE_RT_PUSH_IPI
 
+/*
+ * The shutdown paths stop secondary CPUs without hotplug teardown, leaving
+ * their RT overload bits in rto_mask. An rto push relay can then select a
+ * stopped CPU and queue irq_work to it.
+ *
+ * Serialize the final stopped check and the queue operation with the syscore
+ * shutdown hook, so no new work can be queued after the hook returns.
+ */
+static bool rto_push_stopped;
+static DEFINE_RAW_SPINLOCK(rto_push_queue_lock);
+
 /*
  * When a high priority task schedules out from a CPU and a lower priority
  * task is scheduled in, a check is made to see if there's any RT tasks
@@ -2134,6 +2147,12 @@ static int rto_next_cpu(struct root_domain *rd)
 	int next;
 	int cpu;
 
+	/* Shutdown has disabled rto push queueing. */
+	if (unlikely(READ_ONCE(rto_push_stopped))) {
+		rd->rto_cpu = -1;
+		return -1;
+	}
+
 	/*
 	 * When starting the IPI RT pushing, the rto_cpu is set to -1,
 	 * rt_next_cpu() will simply return the first CPU found in
@@ -2190,6 +2209,24 @@ static inline void rto_start_unlock(atomic_t *v)
 	atomic_set_release(v, 0);
 }
 
+/*
+ * Return false if the caller must drop the reference held by the push chain.
+ */
+static bool rto_push_queue(struct root_domain *rd, int cpu)
+{
+	unsigned long flags;
+	bool allowed = false;
+
+	raw_spin_lock_irqsave(&rto_push_queue_lock, flags);
+	if (likely(!READ_ONCE(rto_push_stopped))) {
+		irq_work_queue_on(&rd->rto_push_work, cpu);
+		allowed = true;
+	}
+	raw_spin_unlock_irqrestore(&rto_push_queue_lock, flags);
+
+	return allowed;
+}
+
 static void tell_cpu_to_push(struct rq *rq)
 {
 	int cpu = -1;
@@ -2219,7 +2256,8 @@ static void tell_cpu_to_push(struct rq *rq)
 	if (cpu >= 0) {
 		/* Make sure the rd does not get freed while pushing */
 		sched_get_rd(rq->rd);
-		irq_work_queue_on(&rq->rd->rto_push_work, cpu);
+		if (!rto_push_queue(rq->rd, cpu))
+			sched_put_rd(rq->rd);
 	}
 }
 
@@ -2257,8 +2295,38 @@ void rto_push_irq_work_func(struct irq_work *work)
 	}
 
 	/* Try the next RT overloaded CPU */
-	irq_work_queue_on(&rd->rto_push_work, cpu);
+	if (!rto_push_queue(rd, cpu))
+		sched_put_rd(rd);
+}
+
+/*
+ * Called with all CPUs still running, before machine_*() stops them. A sender
+ * either completes queueing before this hook returns, or observes the stopped
+ * state and drops its chain reference without queueing.
+ */
+static void rto_push_shutdown(void *data)
+{
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&rto_push_queue_lock, flags);
+	WRITE_ONCE(rto_push_stopped, true);
+	raw_spin_unlock_irqrestore(&rto_push_queue_lock, flags);
+}
+
+static const struct syscore_ops rto_push_syscore_ops = {
+	.shutdown = rto_push_shutdown,
+};
+
+static struct syscore rto_push_syscore = {
+	.ops = &rto_push_syscore_ops,
+};
+
+static int __init rto_push_syscore_init(void)
+{
+	register_syscore(&rto_push_syscore);
+	return 0;
 }
+device_initcall(rto_push_syscore_init);
 #endif /* HAVE_RT_PUSH_IPI */
 
 static void pull_rt_task(struct rq *this_rq)

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2026-11-16  9:20 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16  9:20 [PATCH] sched/rt: Quiesce RTO push before stopping CPUs on shutdown wangqk1

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®