[PATCH 0/2] sched/fair: Cut per-group work from sched domain rebuilds

[PATCH 0/2] sched/fair: Cut per-group work from sched domain rebuilds

[Qiliang Yuan]

A sched domain rebuild, from CPU hotplug or a cpuset partition change,
moves every active CPU to a new root domain through rq_attach_root().
For each of them rq_offline_fair() and rq_online_fair() walk all task
groups with the rq lock held and interrupts disabled, so a rebuild
costs O(nr_cpus * nr_task_groups). Dietmar pointed at the offline side
when f60a631ab9ed was reviewed [1].

Patch 1 skips rq_offline_fair() for CPUs that stay active. Patch 2
skips the runtime_enabled re-sync in rq_online_fair() unless the CPU
actually went down.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0
(median of 20 partition switches and 16 hotplug operations, all three
kernels measured in the same run):

  operation         base (ms)  patch 1 (ms)  patch 1+2 (ms)  change
  partition root        13.28          7.68            1.21    -91%
  cpu offline           44.95         30.96           13.44    -70%
  cpu online            36.84         28.48           12.82    -65%

With both patches these times no longer grow with the number of
groups. With RT_GROUP_SCHED enabled, the rt.c runtime walks remain.

[1] https://lore.kernel.org/r/dbd36b16-4acb-415a-9bbd-258b2e022475@arm.com

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
Qiliang Yuan (2):
      sched/fair: Skip rq_offline_fair() work for CPUs that stay active
      sched/fair: Avoid re-syncing runtime_enabled on root domain changes

 kernel/sched/fair.c  | 20 +++++++++++++++++---
 kernel/sched/sched.h |  2 ++
 2 files changed, 19 insertions(+), 3 deletions(-)
---
base-commit: 067c5c24cef23d913b67eaaf71ae67e7169a6620
change-id: 20261111-bug-sched-fair-offline-active-8d063e735d3f

Best regards,
-- 
Qiliang Yuan <qiliang.yuan@linux.dev>

[PATCH 1/2] sched/fair: Skip rq_offline_fair() work for CPUs that stay active

[Qiliang Yuan]

A sched domain rebuild, triggered by CPU hotplug or a cpuset partition
change, calls rq_attach_root() for every active CPU in the affected
domains. That runs set_rq_offline() and set_rq_online() with the rq lock
held and interrupts disabled, so rq_offline_fair() is called for CPUs
that are not going down.

Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during
CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active
CPUs, but clear_tg_offline_cfs_rqs(), added by commit f60a631ab9ed
("sched/fair: Fix tg->load when offlining a CPU"), still walks every
task group and drops the CPU's tg->load_avg contribution, which
update_tg_load_avg() then adds back. With thousands of cpu cgroups each
rebuild costs O(nr_cpus * nr_task_groups).

Move the active check into rq_offline_fair() so that both walks only run
for a CPU going down.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):

  operation         before (ms)  after (ms)  change
  partition root          13.28        7.68    -42%
  cpu offline             44.95       30.96    -31%
  cpu online              36.84       28.48    -23%

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/fair.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79f6b3a4fd7f3..e245a31b0544e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7816,10 +7816,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
-	// Do not unthrottle for an active CPU
-	if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
-		return;
-
 	/*
 	 * The rq clock has already been updated in the
 	 * set_rq_offline(), so we should skip updating
@@ -15064,6 +15060,13 @@ static void rq_offline_fair(struct rq *rq)
 {
 	update_sysctl();
 
+	/*
+	 * A sched domain rebuild moves every active CPU to a new root domain
+	 * and lands here as well. Only a CPU going down needs the work below.
+	 */
+	if (cpu_active(cpu_of(rq)))
+		return;
+
 	/* Ensure any throttled groups are reachable by pick_next_task */
 	unthrottle_offline_cfs_rqs(rq);
 

-- 
2.43.0

[PATCH 2/2] sched/fair: Avoid re-syncing runtime_enabled on root domain changes

[Qiliang Yuan]

rq_online_fair() calls update_runtime_enabled(), which walks every task
group and takes each group's cfs_bandwidth lock to copy its quota state
into the CPU's cfs_rq. A sched domain rebuild reaches it for every
active CPU through rq_attach_root(), with the rq lock held and
interrupts disabled.

The walk only matters when the cached state may be stale. While a CPU
is online, tg_set_cfs_bandwidth() updates runtime_enabled on it under
the rq lock, and it holds cpus_read_lock() so it cannot miss a CPU that
is coming up. Only unthrottle_offline_cfs_rqs() clears runtime_enabled
behind its back, and a CPU that has never been online has not been
visited at all.

Track this in a per-rq flag that update_runtime_enabled() sets after
the walk and unthrottle_offline_cfs_rqs() clears, and skip the walk
while it is set.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):

  operation         before (ms)  after (ms)  change
  partition root           7.68        1.21    -84%
  cpu offline             30.96       13.44    -57%
  cpu online              28.48       12.82    -55%

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/fair.c  | 11 +++++++++++
 kernel/sched/sched.h |  2 ++
 2 files changed, 13 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e245a31b0544e..e2f67bdc3031e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7798,6 +7798,15 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
+	/*
+	 * tg_set_cfs_bandwidth() keeps runtime_enabled up to date while the
+	 * CPU is online, so only the first activation and the one following
+	 * unthrottle_offline_cfs_rqs() need the walk.
+	 */
+	if (rq->cfsb_runtime_valid)
+		return;
+	rq->cfsb_runtime_valid = true;
+
 	guard(rcu)();
 
 	list_for_each_entry_rcu(tg, &task_groups, list) {
@@ -7816,6 +7825,8 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
+	rq->cfsb_runtime_valid = false;
+
 	/*
 	 * The rq clock has already been updated in the
 	 * set_rq_offline(), so we should skip updating
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1d76950dbb0a1..1a73383204384 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1412,6 +1412,8 @@ struct rq {
 #ifdef CONFIG_CFS_BANDWIDTH
 	call_single_data_t	cfsb_csd;
 	struct list_head	cfsb_csd_list;
+	/* cfs_rq->runtime_enabled is in sync with the quota of every group */
+	bool			cfsb_runtime_valid;
 #endif
 
 	atomic_t		nr_iowait;

-- 
2.43.0

Re: [PATCH 2/2] sched/fair: Avoid re-syncing runtime_enabled on root domain changes

[Peter Zijlstra]

On Wed, Nov 11, 2026 at 08:24:21PM -0500, Qiliang Yuan wrote:
> rq_online_fair() calls update_runtime_enabled(), which walks every task
> group and takes each group's cfs_bandwidth lock to copy its quota state
> into the CPU's cfs_rq. A sched domain rebuild reaches it for every
> active CPU through rq_attach_root(), with the rq lock held and
> interrupts disabled.
> 
> The walk only matters when the cached state may be stale. While a CPU
> is online, tg_set_cfs_bandwidth() updates runtime_enabled on it under
> the rq lock, and it holds cpus_read_lock() so it cannot miss a CPU that
> is coming up. Only unthrottle_offline_cfs_rqs() clears runtime_enabled
> behind its back, and a CPU that has never been online has not been
> visited at all.
> 
> Track this in a per-rq flag that update_runtime_enabled() sets after
> the walk and unthrottle_offline_cfs_rqs() clears, and skip the walk
> while it is set.
> 
> With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
> switching a 4-CPU cpuset to a partition root and offlining or onlining
> one CPU take (median of 20 partition switches and 16 hotplug operations):
> 
>   operation         before (ms)  after (ms)  change
>   partition root           7.68        1.21    -84%
>   cpu offline             30.96       13.44    -57%
>   cpu online              28.48       12.82    -55%

How do you end up with 4k cgroups ?!? That's wild.

Also, why do you care about the rebuild performance? I'm not entirely
sure it makes sense to add data for something that 'never' happens. Do
we really need a new member variable for this?

> Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
> ---
>  kernel/sched/fair.c  | 11 +++++++++++
>  kernel/sched/sched.h |  2 ++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e245a31b0544e..e2f67bdc3031e 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7798,6 +7798,15 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq)
>  
>  	lockdep_assert_rq_held(rq);
>  
> +	/*
> +	 * tg_set_cfs_bandwidth() keeps runtime_enabled up to date while the
> +	 * CPU is online, so only the first activation and the one following
> +	 * unthrottle_offline_cfs_rqs() need the walk.
> +	 */
> +	if (rq->cfsb_runtime_valid)
> +		return;
> +	rq->cfsb_runtime_valid = true;
> +
>  	guard(rcu)();
>  
>  	list_for_each_entry_rcu(tg, &task_groups, list) {
> @@ -7816,6 +7825,8 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
>  
>  	lockdep_assert_rq_held(rq);
>  
> +	rq->cfsb_runtime_valid = false;
> +
>  	/*
>  	 * The rq clock has already been updated in the
>  	 * set_rq_offline(), so we should skip updating
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 1d76950dbb0a1..1a73383204384 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -1412,6 +1412,8 @@ struct rq {
>  #ifdef CONFIG_CFS_BANDWIDTH
>  	call_single_data_t	cfsb_csd;
>  	struct list_head	cfsb_csd_list;
> +	/* cfs_rq->runtime_enabled is in sync with the quota of every group */
> +	bool			cfsb_runtime_valid;
>  #endif
>  
>  	atomic_t		nr_iowait;
> 
> -- 
> 2.43.0
> 

Re: [PATCH 2/2] sched/fair: Avoid re-syncing runtime_enabled on root domain changes

[Qiliang Yuan]

On Thu, Nov 12, 2026 at 11:56:24AM +0100, Peter Zijlstra wrote:
> How do you end up with 4k cgroups ?!? That's wild.

It is common on hosts running AI agent harnesses: every agent sandbox
gets its own container, and each pod and container has a cgroup with
the cpu controller enabled, so a single kernel easily carries over a
thousand task groups. 4000 is the upper end of that, picked to show the
scaling.

> Also, why do you care about the rebuild performance? I'm not entirely
> sure it makes sense to add data for something that 'never' happens. Do
> we really need a new member variable for this?

On those setups it is not rare. Agent harnesses and KubeVirt both
hot-add and remove vCPUs at run time to follow the load, and every
hotplug rebuilds the sched domains of the guest.

A rebuild runs rq_attach_root() on every active CPU, and with the rq
lock held and interrupts disabled it walks all task groups twice. With
4000 cpu cgroups on a 32-CPU VM, one rq_attach_root() call takes 188 us
at the median and up to 938 us, against 2 us with both patches applied
(20 partition switches, 1280 calls). So every rebuild leaves each CPU
with interrupts off for close to a millisecond in the worst case.

The member is there because rq_online_fair() can't tell a rebuild from
a CPU coming up: the CPU is active in both cases, but only the latter
can have runtime_enabled out of date after unthrottle_offline_cfs_rqs().

Re: [PATCH 2/2] sched/fair: Avoid re-syncing runtime_enabled on root domain changes

[Peter Zijlstra]

On Thu, Nov 12, 2026 at 06:10:55AM -0500, Qiliang Yuan wrote:
> On Thu, Nov 12, 2026 at 11:56:24AM +0100, Peter Zijlstra wrote:
> > How do you end up with 4k cgroups ?!? That's wild.
> 
> It is common on hosts running AI agent harnesses: every agent sandbox
> gets its own container, and each pod and container has a cgroup with
> the cpu controller enabled, so a single kernel easily carries over a
> thousand task groups. 4000 is the upper end of that, picked to show the
> scaling.

*groan*

> > Also, why do you care about the rebuild performance? I'm not entirely
> > sure it makes sense to add data for something that 'never' happens. Do
> > we really need a new member variable for this?
> 
> On those setups it is not rare. Agent harnesses and KubeVirt both
> hot-add and remove vCPUs at run time to follow the load, and every
> hotplug rebuilds the sched domains of the guest.

So CPU hotplug is a slow path; there's nothing that's going to 'fix'
that. If it hurts, its one of those things you need to stop doing.

> A rebuild runs rq_attach_root() on every active CPU, and with the rq
> lock held and interrupts disabled it walks all task groups twice. 

Right, that is a bit daft. It should be relatively straight forward to
make that one loop.

> With 4000 cpu cgroups on a 32-CPU VM, one rq_attach_root() call takes
> 188 us at the median and up to 938 us, against 2 us with both patches
> applied (20 partition switches, 1280 calls). So every rebuild leaves
> each CPU with interrupts off for close to a millisecond in the worst
> case.

Not ideal, but again, you did do hotplug so you get to keep the pieces.

> The member is there because rq_online_fair() can't tell a rebuild from
> a CPU coming up: the CPU is active in both cases, but only the latter
> can have runtime_enabled out of date after unthrottle_offline_cfs_rqs().

So the actual hotplug code obviously knows. It should be fairly simple
to add a boolean argument to both functions and set the right value in
rq_attach_root() and sched_set_rq_{on,off}line() respectively.