From: wangqk1 <wangqk1@xiaopeng.com>
To: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>
Cc: linux-kernel@vger.kernel.org, xiedl3@xiaopeng.com
Subject: [PATCH] sched/rt: Quiesce RTO push before stopping CPUs on shutdown
Date: Mon, 16 Nov 2026 17:20:00 +0800 [thread overview]
Message-ID: <20261116092000.120707-1-wangqk1@xiaopeng.com> (raw)
On arm64, machine_power_off(), machine_halt(), and machine_restart()
call smp_send_stop() without running CPU hotplug teardown. A stopped CPU
clears its online bit and parks, while its RT overload bit can remain in
root_domain::rto_mask.
An RTO push relay still in flight can select that stale bit and call
irq_work_queue_on() for the parked CPU, triggering
WARN_ON_ONCE(cpu_is_offline(cpu)). This was observed during normal
poweroff of a PREEMPT_RT kernel under RT load, about 5 ms after the
"Power down" message:
irq_work_queue_on+0x138/0x148
rto_push_irq_work_func+0x98
irq_work_single
generic_smp_call_function_single_interrupt
A cpu_online() check in rto_next_cpu() cannot close this race because
the target can stop after the check but before irq_work_queue_on().
Checking a shutdown flag only in rto_next_cpu() has the same problem:
a sender can be delayed after the check and resume after another CPU
has been stopped.
Quiesce RTO push from a syscore shutdown hook, before the machine
specific shutdown path can stop any CPU. rto_next_cpu() checks
rto_push_stopped for early termination, but correctness is enforced at
both actual queueing sites. tell_cpu_to_push() starts a relay chain, and
rto_push_irq_work_func() relays work to the next overloaded CPU.
Both sites use rto_push_queue(), which holds a global raw spinlock
across the final rto_push_stopped check and irq_work_queue_on(). The
shutdown hook takes the same lock before setting rto_push_stopped.
This gives two possible interleavings. If a sender gets the lock first,
queueing completes before the shutdown hook returns, hence before
smp_send_stop() can take the target offline. If shutdown gets the lock
first, the sender observes rto_push_stopped and does not queue. A raw
spinlock is required because the relay runs in hardirq context and must
also work on PREEMPT_RT. irq_work_queue_on() only claims and queues the
work and, for a remote CPU, sends an IPI. It does not wait for the remote
callback, so the lock critical section has no callback wait.
When the initial queue operation is rejected, tell_cpu_to_push() drops
the root_domain reference it just acquired. When a relay is rejected,
rto_push_irq_work_func() drops the reference owned by the chain. The
existing cpu < 0 path continues to release that reference as before.
There is no root_domain enumeration and no irq_work_sync(). Enumeration
through cpu_rq(cpu)->rd can miss an old root_domain that cpuset partition
changes detached from every runqueue while an in-flight chain still
holds a reference. Waiting for hard irq_work with irq_work_sync() also
introduces an unbounded shutdown wait. The final queueing gate covers
old and new root_domains without either dependency.
An RCU alternative was considered. The final check and queue operation
could run inside rcu_read_lock(), with shutdown setting
rto_push_stopped and then calling synchronize_rcu(). The grace period
would ensure that a sender which already passed the check finishes
queueing before any CPU is stopped.
RCU was not chosen because shutdown would then depend on a system-wide
RCU grace period, including unrelated readers and quiescent-state
progress, rather than only on short RTO queueing critical sections.
A delayed or stalled grace period would delay poweroff, halt, restart,
or kexec. The lock has the narrower liveness dependency and does not
wait for remote callbacks. Using only READ_ONCE(), an atomic flag, or a
static branch without a grace period or equivalent sender
synchronization would leave the original check-to-queue race open.
Each initial or relayed queue operation takes a global raw spinlock only
around the final flag check and irq_work_queue_on(), not around target
scanning, RT task pushing, or remote callback execution.
A/B testing on a 20-CPU arm64 6.18.14-rt4 PREEMPT_RT system used an RT
overload workload. The A/B latency differences were smaller than the
A/A run-to-run variation and could not be attributed to the patch.
These measurements do not isolate the cost of the lock or establish
zero overhead. Testing covered only the default root_domain.
This change prevents new RTO work from being queued after the shutdown
hook. It does not drain work that was already queued and does not clear
rto_mask. Work left on a CPU that is subsequently parked need not run
because the system is terminating. Panic and crash kexec do not pass
through syscore_shutdown() and retain their existing behavior.
Link: https://lore.kernel.org/linux-kernel/20261013030140.682182-2-wangqk1@xiaopeng.com/
Signed-off-by: wangqk1 <wangqk1@xiaopeng.com>
---
Test details:
The equivalent change was A/B tested on a 20-CPU arm64 6.18.14-rt4
PREEMPT_RT system, with five 60-second runs per kernel:
cyclictest -t 80 -p 90 -i 50 -d 0 -D 60 -m -h 400 -q
while :; do hackbench -g 50 -l 5000 -f 25; done # background load
Latency (us) Without patch With patch Delta
Median min 1.137 1.163 +2.20%
Median avg 4.500 4.575 +1.67%
Median max 92.000 92.000 0.00%
Worst max 95 94 -1.05%
rto_push_irq_work_func Without patch With patch
Probe hits / 10.16 s 8372 11692
Probe hits / second 824 1151
A/A median avg drift, first two vs last two runs: 4.14%
The test binaries were identical, and probes were removed before the
formal runs with tracing disabled. The test did not cover concurrent
cpuset partitions.
This mainline version was build-tested with x86_64 defconfig as
kernel/sched/rt.o. No runtime testing was performed on this mainline
tree. The measurements above used the equivalent arm64 6.18 change.
kernel/sched/rt.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 70 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 1535046a23ff..5431ca7dc7e9 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -4,6 +4,8 @@
* policies)
*/
+#include <linux/syscore_ops.h>
+
#include "sched.h"
#include "pelt.h"
@@ -2087,6 +2089,17 @@ static void push_rt_tasks(struct rq *rq)
#ifdef HAVE_RT_PUSH_IPI
+/*
+ * The shutdown paths stop secondary CPUs without hotplug teardown, leaving
+ * their RT overload bits in rto_mask. An rto push relay can then select a
+ * stopped CPU and queue irq_work to it.
+ *
+ * Serialize the final stopped check and the queue operation with the syscore
+ * shutdown hook, so no new work can be queued after the hook returns.
+ */
+static bool rto_push_stopped;
+static DEFINE_RAW_SPINLOCK(rto_push_queue_lock);
+
/*
* When a high priority task schedules out from a CPU and a lower priority
* task is scheduled in, a check is made to see if there's any RT tasks
@@ -2134,6 +2147,12 @@ static int rto_next_cpu(struct root_domain *rd)
int next;
int cpu;
+ /* Shutdown has disabled rto push queueing. */
+ if (unlikely(READ_ONCE(rto_push_stopped))) {
+ rd->rto_cpu = -1;
+ return -1;
+ }
+
/*
* When starting the IPI RT pushing, the rto_cpu is set to -1,
* rt_next_cpu() will simply return the first CPU found in
@@ -2190,6 +2209,24 @@ static inline void rto_start_unlock(atomic_t *v)
atomic_set_release(v, 0);
}
+/*
+ * Return false if the caller must drop the reference held by the push chain.
+ */
+static bool rto_push_queue(struct root_domain *rd, int cpu)
+{
+ unsigned long flags;
+ bool allowed = false;
+
+ raw_spin_lock_irqsave(&rto_push_queue_lock, flags);
+ if (likely(!READ_ONCE(rto_push_stopped))) {
+ irq_work_queue_on(&rd->rto_push_work, cpu);
+ allowed = true;
+ }
+ raw_spin_unlock_irqrestore(&rto_push_queue_lock, flags);
+
+ return allowed;
+}
+
static void tell_cpu_to_push(struct rq *rq)
{
int cpu = -1;
@@ -2219,7 +2256,8 @@ static void tell_cpu_to_push(struct rq *rq)
if (cpu >= 0) {
/* Make sure the rd does not get freed while pushing */
sched_get_rd(rq->rd);
- irq_work_queue_on(&rq->rd->rto_push_work, cpu);
+ if (!rto_push_queue(rq->rd, cpu))
+ sched_put_rd(rq->rd);
}
}
@@ -2257,8 +2295,38 @@ void rto_push_irq_work_func(struct irq_work *work)
}
/* Try the next RT overloaded CPU */
- irq_work_queue_on(&rd->rto_push_work, cpu);
+ if (!rto_push_queue(rd, cpu))
+ sched_put_rd(rd);
+}
+
+/*
+ * Called with all CPUs still running, before machine_*() stops them. A sender
+ * either completes queueing before this hook returns, or observes the stopped
+ * state and drops its chain reference without queueing.
+ */
+static void rto_push_shutdown(void *data)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&rto_push_queue_lock, flags);
+ WRITE_ONCE(rto_push_stopped, true);
+ raw_spin_unlock_irqrestore(&rto_push_queue_lock, flags);
+}
+
+static const struct syscore_ops rto_push_syscore_ops = {
+ .shutdown = rto_push_shutdown,
+};
+
+static struct syscore rto_push_syscore = {
+ .ops = &rto_push_syscore_ops,
+};
+
+static int __init rto_push_syscore_init(void)
+{
+ register_syscore(&rto_push_syscore);
+ return 0;
}
+device_initcall(rto_push_syscore_init);
#endif /* HAVE_RT_PUSH_IPI */
static void pull_rt_task(struct rq *this_rq)
reply other threads:[~2026-11-16 9:20 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116092000.120707-1-wangqk1@xiaopeng.com \
--to=wangqk1@xiaopeng.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=xiedl3@xiaopeng.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®