From: Qiliang Yuan <qiliang.yuan@linux.dev>
To: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Li kunyu <likunyu@open-hieco.net>,
linux-kernel@vger.kernel.org,
Qiliang Yuan <qiliang.yuan@linux.dev>
Subject: [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves
Date: Fri, 20 Nov 2026 21:57:25 -0500 [thread overview]
Message-ID: <20261120-bug-sched-fair-offline-active-v3-2-325e9eefa56d@linux.dev> (raw)
In-Reply-To: <20261120-bug-sched-fair-offline-active-v3-0-325e9eefa56d@linux.dev>
A sched domain rebuild, from CPU hotplug or a cpuset partition change,
calls rq_attach_root() for every active CPU. rq_attach_root() takes the
CPU offline in its old root domain and puts it back online in the new
one, running all rq_offline() and rq_online() callbacks with the rq
lock held and interrupts disabled.
For fair, that walks all task groups twice per CPU: rq_offline_fair()
in clear_tg_offline_cfs_rqs() and rq_online_fair() in
update_runtime_enabled(), although nothing they maintain changes for a
CPU that stays active; tg_set_cfs_bandwidth() already keeps
runtime_enabled in sync on online CPUs. A rebuild thus costs
O(nr_cpus * nr_task_groups), and hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups.
Keep a CPU that stays active online and move only its root domain
state with the rq_attach_root() and rq_detach_root() callbacks. A CPU
that comes up is still put online by the rebuild in
sched_cpu_activate(), and one that is no longer active is still taken
offline there.
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug
operations, min-max in parentheses):
operation before (ms) after (ms) change
partition root 15.97 (14.76-35.88) 1.18 (1.14-2.26) -93%
cpu offline 49.66 (42.11-58.82) 14.49 (12.71-23.19) -71%
cpu online 37.39 (33.40-41.99) 13.70 (9.65-16.53) -63%
Each rq_attach_root() call during those partition switches takes (1280
calls, change of the median):
kernel median (us) p90 (us) max (us) change
before 229 265 1335
after 1 1 26 -99.6%
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/core.c | 27 +++++++++++++++++++++++++++
kernel/sched/sched.h | 2 ++
kernel/sched/topology.c | 20 +++++++++++++++++---
3 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca920a302ecba..49a0cee90a834 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8580,6 +8580,33 @@ void set_rq_offline(struct rq *rq)
}
}
+void rq_attach_root_domain(struct rq *rq)
+{
+ struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+ const struct sched_class *class;
+
+ cpumask_set_cpu(rq->cpu, rd->online);
+
+ for_each_class(class) {
+ if (class->rq_attach_root)
+ class->rq_attach_root(rq);
+ }
+}
+
+void rq_detach_root_domain(struct rq *rq)
+{
+ struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+ const struct sched_class *class;
+
+ update_rq_clock(rq);
+ for_each_class(class) {
+ if (class->rq_detach_root)
+ class->rq_detach_root(rq);
+ }
+
+ cpumask_clear_cpu(rq->cpu, rd->online);
+}
+
static inline void sched_set_rq_online(struct rq *rq, int cpu)
{
struct root_domain *rd;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a24801d2058b4..1ceef4af26bc8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3534,6 +3534,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
extern void set_rq_online (struct rq *rq);
extern void set_rq_offline(struct rq *rq);
+void rq_attach_root_domain(struct rq *rq);
+void rq_detach_root_domain(struct rq *rq);
extern bool sched_smp_initialized;
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..cc2bc832dad1a 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,26 @@ static void free_rootdomain(struct rcu_head *rcu)
void rq_attach_root(struct rq *rq, struct root_domain *rd)
{
struct root_domain *old_rd = NULL;
+ bool active, move = false;
struct rq_flags rf;
rq_lock_irqsave(rq, &rf);
+ active = cpumask_test_cpu(rq->cpu, cpu_active_mask);
+
old_rd = rcu_dereference_root_domain(rq->rd);
if (old_rd) {
- if (cpumask_test_cpu(rq->cpu, old_rd->online))
- set_rq_offline(rq);
+ if (cpumask_test_cpu(rq->cpu, old_rd->online)) {
+ /*
+ * Keep an online CPU that stays active online and move
+ * only its root domain state.
+ */
+ move = active;
+ if (move)
+ rq_detach_root_domain(rq);
+ else
+ set_rq_offline(rq);
+ }
cpumask_clear_cpu(rq->cpu, old_rd->span);
@@ -512,7 +524,9 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
rcu_assign_pointer(rq->rd, rd);
cpumask_set_cpu(rq->cpu, rd->span);
- if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
+ if (move)
+ rq_attach_root_domain(rq);
+ else if (active)
set_rq_online(rq);
/*
--
2.43.0
next prev parent reply other threads:[~2026-11-21 2:57 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-21 2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
2026-11-21 2:57 ` Qiliang Yuan [this message]
2026-11-21 2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120-bug-sched-fair-offline-active-v3-2-325e9eefa56d@linux.dev \
--to=qiliang.yuan@linux.dev \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=likunyu@open-hieco.net \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®