* [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds
@ 2026-11-19 0:57 Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: linux-kernel, Qiliang Yuan
A sched domain rebuild, from CPU hotplug or a cpuset partition change,
moves every active CPU to a new root domain through rq_attach_root().
For each of them rq_offline_fair() and rq_online_fair() walk all task
groups with the rq lock held and interrupts disabled, so a rebuild
costs O(nr_cpus * nr_task_groups). Dietmar pointed at the offline side
when f60a631ab9ed was reviewed [1]. Hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups, and their VMs hot-add
vCPUs at run time.
Patch 1 tells the rq_online() and rq_offline() callbacks whether the
CPU actually changes state. Patches 2 and 3 use that to skip the two
fair walks when it doesn't.
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0
(median of 20 partition switches and 16 hotplug operations, all three
kernels measured in the same run):
operation base (ms) patch 1-2 (ms) patch 1-3 (ms) change
partition root 14.86 8.35 1.20 -92%
cpu offline 49.99 33.91 16.49 -67%
cpu online 36.10 31.02 12.87 -64%
Each rq_attach_root() call, which runs with interrupts disabled, takes
(1280 calls over the 20 partition switches):
kernel median (us) p90 (us) max (us) change
base 229 256 6987
patch 1-2 122 162 988 -47%
patch 1-3 2 3 118 -99%
With all three patches, partition switches and onlining no longer grow
with the number of groups; offlining still walks them once for the CPU
that goes down. With RT_GROUP_SCHED enabled, the rt.c runtime walks
remain.
K Prateek Nayak proposed separate rq_attach_root() and rq_detach_root()
callbacks for the root domain moves instead [2]. This series follows
Peter's suggestion of a hotplug argument to rq_online() and rq_offline();
I'm happy to redo it on top of Prateek's approach if that is preferred.
[1] https://lore.kernel.org/r/dbd36b16-4acb-415a-9bbd-258b2e022475@arm.com
[2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
V1 -> V2:
- Pass a hotplug argument to rq_online() and rq_offline() instead of
adding a member to struct rq, as Peter suggested
- Skip both fair task group walks on rebuilds, which leaves no second
loop to merge
- Rebase onto tip/sched/core, where sched_ext no longer implements
rq_online() and rq_offline(), and update the numbers from a new run
CPU hotplug and partition flips were also run with SCHED_FIFO tasks
(rt_group_sched=1, RT_RUNTIME_SHARE on and off), SCHED_DEADLINE tasks
and scx_simple loaded, plus the sched_ext selftests, with the same
results as on tip/sched/core (b251360cb2e6c).
v1: https://lore.kernel.org/r/20261111-bug-sched-fair-offline-active-v1-0-4f54523c5a70@linux.dev
---
Qiliang Yuan (3):
sched: Tell rq_online() and rq_offline() whether the CPU changes state
sched/fair: Skip rq_offline_fair() work for CPUs that stay active
sched/fair: Avoid re-syncing runtime_enabled on root domain changes
kernel/sched/core.c | 12 ++++++------
kernel/sched/deadline.c | 4 ++--
kernel/sched/fair.c | 15 ++++++++-------
kernel/sched/rt.c | 4 ++--
kernel/sched/sched.h | 8 ++++----
kernel/sched/topology.c | 13 ++++++++++---
6 files changed, 32 insertions(+), 24 deletions(-)
---
base-commit: b251360cb2e6c7b74d962831f5797a4520511b36
change-id: 20261111-bug-sched-fair-offline-active-8d063e735d3f
Best regards,
--
Qiliang Yuan <qiliang.yuan@linux.dev>
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state
2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
@ 2026-11-19 0:57 ` Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: linux-kernel, Qiliang Yuan
set_rq_online() and set_rq_offline() run both when a CPU comes up or
goes down and when a sched domain rebuild moves an active CPU to a new
root domain through rq_attach_root(). The class callbacks get the same
call in both cases.
Work such as resyncing CFS bandwidth state or unthrottling cfs_rqs is
only needed when the CPU actually changes state, but the callbacks have
no way to tell, so they repeat it for every active CPU on each rebuild.
Pass a hotplug argument down to the callbacks, and have
sched_set_rq_online() and sched_set_rq_offline() pass true.
rq_attach_root() passes false when taking a CPU offline, as
sched_cpu_deactivate() has already taken a CPU going down offline
before the rebuild. When putting a CPU online it passes true only if
the CPU was not online in its old root domain: sched_cpu_activate()
rebuilds the domains before calling sched_set_rq_online(), so this is
where a CPU coming up first goes online.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/core.c | 12 ++++++------
kernel/sched/deadline.c | 4 ++--
kernel/sched/fair.c | 4 ++--
kernel/sched/rt.c | 4 ++--
kernel/sched/sched.h | 8 ++++----
kernel/sched/topology.c | 13 ++++++++++---
6 files changed, 26 insertions(+), 19 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b4e4a6f4ae02..17ceb9d5046ba 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8542,7 +8542,7 @@ static inline void balance_hotplug_wait(void)
#endif /* !CONFIG_HOTPLUG_CPU */
-void set_rq_online(struct rq *rq)
+void set_rq_online(struct rq *rq, bool hotplug)
{
if (!rq->online) {
const struct sched_class *class;
@@ -8554,12 +8554,12 @@ void set_rq_online(struct rq *rq)
for_each_class(class) {
if (class->rq_online)
- class->rq_online(rq);
+ class->rq_online(rq, hotplug);
}
}
}
-void set_rq_offline(struct rq *rq)
+void set_rq_offline(struct rq *rq, bool hotplug)
{
if (rq->online) {
struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
@@ -8568,7 +8568,7 @@ void set_rq_offline(struct rq *rq)
update_rq_clock(rq);
for_each_class(class) {
if (class->rq_offline)
- class->rq_offline(rq);
+ class->rq_offline(rq, hotplug);
}
cpumask_clear_cpu(rq->cpu, rd->online);
@@ -8585,7 +8585,7 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu)
rd = rcu_dereference_root_domain(rq->rd);
if (rd) {
BUG_ON(!cpumask_test_cpu(cpu, rd->span));
- set_rq_online(rq);
+ set_rq_online(rq, true);
}
rq_unlock_irqrestore(rq, &rf);
}
@@ -8599,7 +8599,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu)
rd = rcu_dereference_root_domain(rq->rd);
if (rd) {
BUG_ON(!cpumask_test_cpu(cpu, rd->span));
- set_rq_offline(rq);
+ set_rq_offline(rq, true);
}
rq_unlock_irqrestore(rq, &rf);
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 61b6a6d781715..73f509b87b8fa 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3366,7 +3366,7 @@ bool dl_task_needs_bw_move(struct task_struct *p,
}
/* Assumes rq->lock is held */
-static void rq_online_dl(struct rq *rq)
+static void rq_online_dl(struct rq *rq, bool hotplug)
{
if (rq->dl.overloaded)
dl_set_overload(rq);
@@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq)
}
/* Assumes rq->lock is held */
-static void rq_offline_dl(struct rq *rq)
+static void rq_offline_dl(struct rq *rq, bool hotplug)
{
if (rq->dl.overloaded)
dl_clear_overload(rq);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 731d78155e654..db9ab8ce36500 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -15082,14 +15082,14 @@ void sched_balance_trigger(struct rq *rq)
nohz_balancer_kick(rq);
}
-static void rq_online_fair(struct rq *rq)
+static void rq_online_fair(struct rq *rq, bool hotplug)
{
update_sysctl();
update_runtime_enabled(rq);
}
-static void rq_offline_fair(struct rq *rq)
+static void rq_offline_fair(struct rq *rq, bool hotplug)
{
update_sysctl();
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 81a71aa5f094a..fc6224a343552 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
}
/* Assumes rq->lock is held */
-static void rq_online_rt(struct rq *rq)
+static void rq_online_rt(struct rq *rq, bool hotplug)
{
if (rq->rt.overloaded)
rt_set_overload(rq);
@@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq)
}
/* Assumes rq->lock is held */
-static void rq_offline_rt(struct rq *rq)
+static void rq_offline_rt(struct rq *rq, bool hotplug)
{
if (rq->rt.overloaded)
rt_clear_overload(rq);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e957524c3ee52..5f7b6ab4dde6d 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2796,8 +2796,8 @@ struct sched_class {
/*
* sched_set_rq_{on,off}line: rq->lock
*/
- void (*rq_online)(struct rq *rq);
- void (*rq_offline)(struct rq *rq);
+ void (*rq_online)(struct rq *rq, bool hotplug);
+ void (*rq_offline)(struct rq *rq, bool hotplug);
/*
* push_cpu_stop: p->pi_lock && rq->lock
@@ -3526,8 +3526,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
raw_spin_rq_unlock(rq1);
}
-extern void set_rq_online (struct rq *rq);
-extern void set_rq_offline(struct rq *rq);
+extern void set_rq_online(struct rq *rq, bool hotplug);
+extern void set_rq_offline(struct rq *rq, bool hotplug);
extern bool sched_smp_initialized;
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..577dc012e5ac5 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,16 @@ static void free_rootdomain(struct rcu_head *rcu)
void rq_attach_root(struct rq *rq, struct root_domain *rd)
{
struct root_domain *old_rd = NULL;
+ bool was_online = false;
struct rq_flags rf;
rq_lock_irqsave(rq, &rf);
old_rd = rcu_dereference_root_domain(rq->rd);
if (old_rd) {
- if (cpumask_test_cpu(rq->cpu, old_rd->online))
- set_rq_offline(rq);
+ was_online = cpumask_test_cpu(rq->cpu, old_rd->online);
+ if (was_online)
+ set_rq_offline(rq, false);
cpumask_clear_cpu(rq->cpu, old_rd->span);
@@ -512,8 +514,13 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
rcu_assign_pointer(rq->rd, rd);
cpumask_set_cpu(rq->cpu, rd->span);
+ /*
+ * A CPU that comes up is first put online here, by the sched domain
+ * rebuild of sched_cpu_activate(). Any other CPU only moves between
+ * root domains.
+ */
if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
- set_rq_online(rq);
+ set_rq_online(rq, !was_online);
/*
* Because the rq is not a task, dl_add_task_root_domain() did not
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active
2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
@ 2026-11-19 0:57 ` Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: linux-kernel, Qiliang Yuan
A sched domain rebuild, triggered by CPU hotplug or a cpuset partition
change, calls rq_attach_root() for every active CPU in the affected
domains. That takes the CPU offline and back online with the rq lock
held and interrupts disabled, so rq_offline_fair() runs for CPUs that
are not going down.
Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during
CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active
CPUs, but clear_tg_offline_cfs_rqs(), added by commit f60a631ab9ed
("sched/fair: Fix tg->load when offlining a CPU"), still walks every
task group and drops the CPU's tg->load_avg contribution, which
update_tg_load_avg() then adds back. With thousands of cpu cgroups each
rebuild costs O(nr_cpus * nr_task_groups).
Return early from rq_offline_fair() unless the CPU is going down, which
covers both walks, and drop the active check from
unthrottle_offline_cfs_rqs().
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):
operation before (ms) after (ms) change
partition root 14.86 8.35 -44%
cpu offline 49.99 33.91 -32%
cpu online 36.10 31.02 -14%
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/fair.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index db9ab8ce36500..1b38c4c47b656 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7859,10 +7859,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
lockdep_assert_rq_held(rq);
- // Do not unthrottle for an active CPU
- if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
- return;
-
/*
* The rq clock has already been updated in the
* set_rq_offline(), so we should skip updating
@@ -15093,6 +15089,9 @@ static void rq_offline_fair(struct rq *rq, bool hotplug)
{
update_sysctl();
+ if (!hotplug)
+ return;
+
/* Ensure any throttled groups are reachable by pick_next_task */
unthrottle_offline_cfs_rqs(rq);
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes
2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
@ 2026-11-19 0:57 ` Qiliang Yuan
2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: linux-kernel, Qiliang Yuan
rq_online_fair() calls update_runtime_enabled(), which walks every task
group and takes each group's cfs_bandwidth lock to copy its quota state
into the CPU's cfs_rq. A sched domain rebuild reaches it for every
active CPU through rq_attach_root(), with the rq lock held and
interrupts disabled.
The walk is only needed when the CPU comes up. While a CPU is online,
tg_set_cfs_bandwidth() updates runtime_enabled on it under the rq lock,
and it holds cpus_read_lock() so it cannot miss a CPU that is coming up.
Call update_runtime_enabled() only when the CPU comes up.
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):
operation before (ms) after (ms) change
partition root 8.35 1.20 -86%
cpu offline 33.91 16.49 -51%
cpu online 31.02 12.87 -59%
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
kernel/sched/fair.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1b38c4c47b656..e42c70e6ebeff 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -15082,7 +15082,9 @@ static void rq_online_fair(struct rq *rq, bool hotplug)
{
update_sysctl();
- update_runtime_enabled(rq);
+ /* tg_set_cfs_bandwidth() keeps an online CPU in sync */
+ if (hotplug)
+ update_runtime_enabled(rq);
}
static void rq_offline_fair(struct rq *rq, bool hotplug)
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-19 0:57 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®