* [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks
2026-11-21 2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
@ 2026-11-21 2:57 ` Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21 2:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: Li kunyu, linux-kernel, Qiliang Yuan
The rq_online() and rq_offline() callbacks run when a CPU comes up or
goes down, and also when rq_attach_root() moves an active CPU to a new
root domain during a sched domain rebuild. rt and dl use them for state
kept in the root domain: the overload masks, cpupri, cpudl and the RT
runtime shared within it. fair uses them for task group state, which
does not depend on the root domain.
With one pair of callbacks for both events, a class cannot do only the
work that a root domain move needs, so every rebuild runs the fair
hotplug work for all active CPUs.
Add rq_attach_root() and rq_detach_root() to struct sched_class for the
root domain state and convert rt and dl to them. Have set_rq_online()
and set_rq_offline() call both kinds of callbacks class by class, so
that every class still runs at the same points and in the same order.
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/core.c | 4 ++++
kernel/sched/cpudeadline.c | 5 +++--
kernel/sched/deadline.c | 8 ++++----
kernel/sched/rt.c | 8 ++++----
kernel/sched/sched.h | 6 ++++++
5 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b4e4a6f4ae02..ca920a302ecba 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8553,6 +8553,8 @@ void set_rq_online(struct rq *rq)
rq->online = 1;
for_each_class(class) {
+ if (class->rq_attach_root)
+ class->rq_attach_root(rq);
if (class->rq_online)
class->rq_online(rq);
}
@@ -8569,6 +8571,8 @@ void set_rq_offline(struct rq *rq)
for_each_class(class) {
if (class->rq_offline)
class->rq_offline(rq);
+ if (class->rq_detach_root)
+ class->rq_detach_root(rq);
}
cpumask_clear_cpu(rq->cpu, rd->online);
diff --git a/kernel/sched/cpudeadline.c b/kernel/sched/cpudeadline.c
index 0a2b7e30fd10c..8e2a95f846278 100644
--- a/kernel/sched/cpudeadline.c
+++ b/kernel/sched/cpudeadline.c
@@ -185,8 +185,9 @@ void cpudl_clear(struct cpudl *cp, int cpu, bool online)
if (old_idx == IDX_INVALID) {
/*
* Nothing to remove if old_idx was invalid.
- * This could happen if rq_online_dl or rq_offline_dl is
- * called for a CPU without -dl tasks running.
+ * This could happen if rq_attach_root_dl() or
+ * rq_detach_root_dl() is called for a CPU without -dl
+ * tasks running.
*/
} else {
new_cpu = cp->elements[cp->size - 1].cpu;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 61b6a6d781715..fcbf53500083e 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3366,7 +3366,7 @@ bool dl_task_needs_bw_move(struct task_struct *p,
}
/* Assumes rq->lock is held */
-static void rq_online_dl(struct rq *rq)
+static void rq_attach_root_dl(struct rq *rq)
{
if (rq->dl.overloaded)
dl_set_overload(rq);
@@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq)
}
/* Assumes rq->lock is held */
-static void rq_offline_dl(struct rq *rq)
+static void rq_detach_root_dl(struct rq *rq)
{
if (rq->dl.overloaded)
dl_clear_overload(rq);
@@ -3664,8 +3664,8 @@ DEFINE_SCHED_CLASS(dl) = {
.select_task_rq = select_task_rq_dl,
.migrate_task_rq = migrate_task_rq_dl,
.set_cpus_allowed = set_cpus_allowed_dl,
- .rq_online = rq_online_dl,
- .rq_offline = rq_offline_dl,
+ .rq_attach_root = rq_attach_root_dl,
+ .rq_detach_root = rq_detach_root_dl,
.task_woken = task_woken_dl,
.find_lock_rq = find_lock_later_rq,
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 81a71aa5f094a..45d10334f7253 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
}
/* Assumes rq->lock is held */
-static void rq_online_rt(struct rq *rq)
+static void rq_attach_root_rt(struct rq *rq)
{
if (rq->rt.overloaded)
rt_set_overload(rq);
@@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq)
}
/* Assumes rq->lock is held */
-static void rq_offline_rt(struct rq *rq)
+static void rq_detach_root_rt(struct rq *rq)
{
if (rq->rt.overloaded)
rt_clear_overload(rq);
@@ -2626,8 +2626,8 @@ DEFINE_SCHED_CLASS(rt) = {
.balance = balance_rt,
.select_task_rq = select_task_rq_rt,
.set_cpus_allowed = set_cpus_allowed_common,
- .rq_online = rq_online_rt,
- .rq_offline = rq_offline_rt,
+ .rq_attach_root = rq_attach_root_rt,
+ .rq_detach_root = rq_detach_root_rt,
.task_woken = task_woken_rt,
.switched_from = switched_from_rt,
.find_lock_rq = find_lock_lowest_rq,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e957524c3ee52..a24801d2058b4 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2799,6 +2799,12 @@ struct sched_class {
void (*rq_online)(struct rq *rq);
void (*rq_offline)(struct rq *rq);
+ /*
+ * Move an online rq to or from a root domain: rq->lock
+ */
+ void (*rq_attach_root)(struct rq *rq);
+ void (*rq_detach_root)(struct rq *rq);
+
/*
* push_cpu_stop: p->pi_lock && rq->lock
*/
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread* [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves
2026-11-21 2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
@ 2026-11-21 2:57 ` Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21 2:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: Li kunyu, linux-kernel, Qiliang Yuan
A sched domain rebuild, from CPU hotplug or a cpuset partition change,
calls rq_attach_root() for every active CPU. rq_attach_root() takes the
CPU offline in its old root domain and puts it back online in the new
one, running all rq_offline() and rq_online() callbacks with the rq
lock held and interrupts disabled.
For fair, that walks all task groups twice per CPU: rq_offline_fair()
in clear_tg_offline_cfs_rqs() and rq_online_fair() in
update_runtime_enabled(), although nothing they maintain changes for a
CPU that stays active; tg_set_cfs_bandwidth() already keeps
runtime_enabled in sync on online CPUs. A rebuild thus costs
O(nr_cpus * nr_task_groups), and hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups.
Keep a CPU that stays active online and move only its root domain
state with the rq_attach_root() and rq_detach_root() callbacks. A CPU
that comes up is still put online by the rebuild in
sched_cpu_activate(), and one that is no longer active is still taken
offline there.
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug
operations, min-max in parentheses):
operation before (ms) after (ms) change
partition root 15.97 (14.76-35.88) 1.18 (1.14-2.26) -93%
cpu offline 49.66 (42.11-58.82) 14.49 (12.71-23.19) -71%
cpu online 37.39 (33.40-41.99) 13.70 (9.65-16.53) -63%
Each rq_attach_root() call during those partition switches takes (1280
calls, change of the median):
kernel median (us) p90 (us) max (us) change
before 229 265 1335
after 1 1 26 -99.6%
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/core.c | 27 +++++++++++++++++++++++++++
kernel/sched/sched.h | 2 ++
kernel/sched/topology.c | 20 +++++++++++++++++---
3 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca920a302ecba..49a0cee90a834 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8580,6 +8580,33 @@ void set_rq_offline(struct rq *rq)
}
}
+void rq_attach_root_domain(struct rq *rq)
+{
+ struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+ const struct sched_class *class;
+
+ cpumask_set_cpu(rq->cpu, rd->online);
+
+ for_each_class(class) {
+ if (class->rq_attach_root)
+ class->rq_attach_root(rq);
+ }
+}
+
+void rq_detach_root_domain(struct rq *rq)
+{
+ struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+ const struct sched_class *class;
+
+ update_rq_clock(rq);
+ for_each_class(class) {
+ if (class->rq_detach_root)
+ class->rq_detach_root(rq);
+ }
+
+ cpumask_clear_cpu(rq->cpu, rd->online);
+}
+
static inline void sched_set_rq_online(struct rq *rq, int cpu)
{
struct root_domain *rd;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a24801d2058b4..1ceef4af26bc8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3534,6 +3534,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
extern void set_rq_online (struct rq *rq);
extern void set_rq_offline(struct rq *rq);
+void rq_attach_root_domain(struct rq *rq);
+void rq_detach_root_domain(struct rq *rq);
extern bool sched_smp_initialized;
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..cc2bc832dad1a 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,26 @@ static void free_rootdomain(struct rcu_head *rcu)
void rq_attach_root(struct rq *rq, struct root_domain *rd)
{
struct root_domain *old_rd = NULL;
+ bool active, move = false;
struct rq_flags rf;
rq_lock_irqsave(rq, &rf);
+ active = cpumask_test_cpu(rq->cpu, cpu_active_mask);
+
old_rd = rcu_dereference_root_domain(rq->rd);
if (old_rd) {
- if (cpumask_test_cpu(rq->cpu, old_rd->online))
- set_rq_offline(rq);
+ if (cpumask_test_cpu(rq->cpu, old_rd->online)) {
+ /*
+ * Keep an online CPU that stays active online and move
+ * only its root domain state.
+ */
+ move = active;
+ if (move)
+ rq_detach_root_domain(rq);
+ else
+ set_rq_offline(rq);
+ }
cpumask_clear_cpu(rq->cpu, old_rd->span);
@@ -512,7 +524,9 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
rcu_assign_pointer(rq->rd, rd);
cpumask_set_cpu(rq->cpu, rd->span);
- if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
+ if (move)
+ rq_attach_root_domain(rq);
+ else if (active)
set_rq_online(rq);
/*
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread* [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs()
2026-11-21 2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
2026-11-21 2:57 ` [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves Qiliang Yuan
@ 2026-11-21 2:57 ` Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21 2:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: Li kunyu, linux-kernel, Qiliang Yuan
Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during
CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active
CPUs, because every sched domain rebuild took all active CPUs offline
and unthrottled their throttled groups.
Rebuilds now keep active CPUs online, so rq_offline_fair() only runs
for a CPU that has already left cpu_active_mask: sched_cpu_deactivate()
clears it before taking the rq offline, and rq_attach_root() takes only
inactive CPUs offline. The check can no longer be true.
Remove it.
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/fair.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 731d78155e654..e366040f4fb4f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7859,10 +7859,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
lockdep_assert_rq_held(rq);
- // Do not unthrottle for an active CPU
- if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
- return;
-
/*
* The rq clock has already been updated in the
* set_rq_offline(), so we should skip updating
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread