mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds
@ 2026-11-19  0:57 Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19  0:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: linux-kernel, Qiliang Yuan

A sched domain rebuild, from CPU hotplug or a cpuset partition change,
moves every active CPU to a new root domain through rq_attach_root().
For each of them rq_offline_fair() and rq_online_fair() walk all task
groups with the rq lock held and interrupts disabled, so a rebuild
costs O(nr_cpus * nr_task_groups). Dietmar pointed at the offline side
when f60a631ab9ed was reviewed [1]. Hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups, and their VMs hot-add
vCPUs at run time.

Patch 1 tells the rq_online() and rq_offline() callbacks whether the
CPU actually changes state. Patches 2 and 3 use that to skip the two
fair walks when it doesn't.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0
(median of 20 partition switches and 16 hotplug operations, all three
kernels measured in the same run):

  operation         base (ms)  patch 1-2 (ms)  patch 1-3 (ms)  change
  partition root        14.86            8.35            1.20    -92%
  cpu offline           49.99           33.91           16.49    -67%
  cpu online            36.10           31.02           12.87    -64%

Each rq_attach_root() call, which runs with interrupts disabled, takes
(1280 calls over the 20 partition switches):

  kernel      median (us)  p90 (us)  max (us)  change
  base                229       256      6987
  patch 1-2           122       162       988    -47%
  patch 1-3             2         3       118    -99%

With all three patches, partition switches and onlining no longer grow
with the number of groups; offlining still walks them once for the CPU
that goes down. With RT_GROUP_SCHED enabled, the rt.c runtime walks
remain.

K Prateek Nayak proposed separate rq_attach_root() and rq_detach_root()
callbacks for the root domain moves instead [2]. This series follows
Peter's suggestion of a hotplug argument to rq_online() and rq_offline();
I'm happy to redo it on top of Prateek's approach if that is preferred.

[1] https://lore.kernel.org/r/dbd36b16-4acb-415a-9bbd-258b2e022475@arm.com
[2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
V1 -> V2:
- Pass a hotplug argument to rq_online() and rq_offline() instead of
  adding a member to struct rq, as Peter suggested
- Skip both fair task group walks on rebuilds, which leaves no second
  loop to merge
- Rebase onto tip/sched/core, where sched_ext no longer implements
  rq_online() and rq_offline(), and update the numbers from a new run

CPU hotplug and partition flips were also run with SCHED_FIFO tasks
(rt_group_sched=1, RT_RUNTIME_SHARE on and off), SCHED_DEADLINE tasks
and scx_simple loaded, plus the sched_ext selftests, with the same
results as on tip/sched/core (b251360cb2e6c).

v1: https://lore.kernel.org/r/20261111-bug-sched-fair-offline-active-v1-0-4f54523c5a70@linux.dev

---
Qiliang Yuan (3):
      sched: Tell rq_online() and rq_offline() whether the CPU changes state
      sched/fair: Skip rq_offline_fair() work for CPUs that stay active
      sched/fair: Avoid re-syncing runtime_enabled on root domain changes

 kernel/sched/core.c     | 12 ++++++------
 kernel/sched/deadline.c |  4 ++--
 kernel/sched/fair.c     | 15 ++++++++-------
 kernel/sched/rt.c       |  4 ++--
 kernel/sched/sched.h    |  8 ++++----
 kernel/sched/topology.c | 13 ++++++++++---
 6 files changed, 32 insertions(+), 24 deletions(-)
---
base-commit: b251360cb2e6c7b74d962831f5797a4520511b36
change-id: 20261111-bug-sched-fair-offline-active-8d063e735d3f

Best regards,
-- 
Qiliang Yuan <qiliang.yuan@linux.dev>


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state
  2026-11-19  0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
@ 2026-11-19  0:57 ` Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19  0:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: linux-kernel, Qiliang Yuan

set_rq_online() and set_rq_offline() run both when a CPU comes up or
goes down and when a sched domain rebuild moves an active CPU to a new
root domain through rq_attach_root(). The class callbacks get the same
call in both cases.

Work such as resyncing CFS bandwidth state or unthrottling cfs_rqs is
only needed when the CPU actually changes state, but the callbacks have
no way to tell, so they repeat it for every active CPU on each rebuild.

Pass a hotplug argument down to the callbacks, and have
sched_set_rq_online() and sched_set_rq_offline() pass true.
rq_attach_root() passes false when taking a CPU offline, as
sched_cpu_deactivate() has already taken a CPU going down offline
before the rebuild. When putting a CPU online it passes true only if
the CPU was not online in its old root domain: sched_cpu_activate()
rebuilds the domains before calling sched_set_rq_online(), so this is
where a CPU coming up first goes online.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/core.c     | 12 ++++++------
 kernel/sched/deadline.c |  4 ++--
 kernel/sched/fair.c     |  4 ++--
 kernel/sched/rt.c       |  4 ++--
 kernel/sched/sched.h    |  8 ++++----
 kernel/sched/topology.c | 13 ++++++++++---
 6 files changed, 26 insertions(+), 19 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b4e4a6f4ae02..17ceb9d5046ba 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8542,7 +8542,7 @@ static inline void balance_hotplug_wait(void)
 
 #endif /* !CONFIG_HOTPLUG_CPU */
 
-void set_rq_online(struct rq *rq)
+void set_rq_online(struct rq *rq, bool hotplug)
 {
 	if (!rq->online) {
 		const struct sched_class *class;
@@ -8554,12 +8554,12 @@ void set_rq_online(struct rq *rq)
 
 		for_each_class(class) {
 			if (class->rq_online)
-				class->rq_online(rq);
+				class->rq_online(rq, hotplug);
 		}
 	}
 }
 
-void set_rq_offline(struct rq *rq)
+void set_rq_offline(struct rq *rq, bool hotplug)
 {
 	if (rq->online) {
 		struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
@@ -8568,7 +8568,7 @@ void set_rq_offline(struct rq *rq)
 		update_rq_clock(rq);
 		for_each_class(class) {
 			if (class->rq_offline)
-				class->rq_offline(rq);
+				class->rq_offline(rq, hotplug);
 		}
 
 		cpumask_clear_cpu(rq->cpu, rd->online);
@@ -8585,7 +8585,7 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu)
 	rd = rcu_dereference_root_domain(rq->rd);
 	if (rd) {
 		BUG_ON(!cpumask_test_cpu(cpu, rd->span));
-		set_rq_online(rq);
+		set_rq_online(rq, true);
 	}
 	rq_unlock_irqrestore(rq, &rf);
 }
@@ -8599,7 +8599,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu)
 	rd = rcu_dereference_root_domain(rq->rd);
 	if (rd) {
 		BUG_ON(!cpumask_test_cpu(cpu, rd->span));
-		set_rq_offline(rq);
+		set_rq_offline(rq, true);
 	}
 	rq_unlock_irqrestore(rq, &rf);
 }
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 61b6a6d781715..73f509b87b8fa 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3366,7 +3366,7 @@ bool dl_task_needs_bw_move(struct task_struct *p,
 }
 
 /* Assumes rq->lock is held */
-static void rq_online_dl(struct rq *rq)
+static void rq_online_dl(struct rq *rq, bool hotplug)
 {
 	if (rq->dl.overloaded)
 		dl_set_overload(rq);
@@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq)
 }
 
 /* Assumes rq->lock is held */
-static void rq_offline_dl(struct rq *rq)
+static void rq_offline_dl(struct rq *rq, bool hotplug)
 {
 	if (rq->dl.overloaded)
 		dl_clear_overload(rq);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 731d78155e654..db9ab8ce36500 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -15082,14 +15082,14 @@ void sched_balance_trigger(struct rq *rq)
 	nohz_balancer_kick(rq);
 }
 
-static void rq_online_fair(struct rq *rq)
+static void rq_online_fair(struct rq *rq, bool hotplug)
 {
 	update_sysctl();
 
 	update_runtime_enabled(rq);
 }
 
-static void rq_offline_fair(struct rq *rq)
+static void rq_offline_fair(struct rq *rq, bool hotplug)
 {
 	update_sysctl();
 
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 81a71aa5f094a..fc6224a343552 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
 }
 
 /* Assumes rq->lock is held */
-static void rq_online_rt(struct rq *rq)
+static void rq_online_rt(struct rq *rq, bool hotplug)
 {
 	if (rq->rt.overloaded)
 		rt_set_overload(rq);
@@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq)
 }
 
 /* Assumes rq->lock is held */
-static void rq_offline_rt(struct rq *rq)
+static void rq_offline_rt(struct rq *rq, bool hotplug)
 {
 	if (rq->rt.overloaded)
 		rt_clear_overload(rq);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e957524c3ee52..5f7b6ab4dde6d 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2796,8 +2796,8 @@ struct sched_class {
 	/*
 	 * sched_set_rq_{on,off}line: rq->lock
 	 */
-	void (*rq_online)(struct rq *rq);
-	void (*rq_offline)(struct rq *rq);
+	void (*rq_online)(struct rq *rq, bool hotplug);
+	void (*rq_offline)(struct rq *rq, bool hotplug);
 
 	/*
 	 * push_cpu_stop: p->pi_lock && rq->lock
@@ -3526,8 +3526,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
 	raw_spin_rq_unlock(rq1);
 }
 
-extern void set_rq_online (struct rq *rq);
-extern void set_rq_offline(struct rq *rq);
+extern void set_rq_online(struct rq *rq, bool hotplug);
+extern void set_rq_offline(struct rq *rq, bool hotplug);
 
 extern bool sched_smp_initialized;
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..577dc012e5ac5 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,16 @@ static void free_rootdomain(struct rcu_head *rcu)
 void rq_attach_root(struct rq *rq, struct root_domain *rd)
 {
 	struct root_domain *old_rd = NULL;
+	bool was_online = false;
 	struct rq_flags rf;
 
 	rq_lock_irqsave(rq, &rf);
 
 	old_rd = rcu_dereference_root_domain(rq->rd);
 	if (old_rd) {
-		if (cpumask_test_cpu(rq->cpu, old_rd->online))
-			set_rq_offline(rq);
+		was_online = cpumask_test_cpu(rq->cpu, old_rd->online);
+		if (was_online)
+			set_rq_offline(rq, false);
 
 		cpumask_clear_cpu(rq->cpu, old_rd->span);
 
@@ -512,8 +514,13 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	rcu_assign_pointer(rq->rd, rd);
 
 	cpumask_set_cpu(rq->cpu, rd->span);
+	/*
+	 * A CPU that comes up is first put online here, by the sched domain
+	 * rebuild of sched_cpu_activate(). Any other CPU only moves between
+	 * root domains.
+	 */
 	if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
-		set_rq_online(rq);
+		set_rq_online(rq, !was_online);
 
 	/*
 	 * Because the rq is not a task, dl_add_task_root_domain() did not

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active
  2026-11-19  0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
@ 2026-11-19  0:57 ` Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19  0:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: linux-kernel, Qiliang Yuan

A sched domain rebuild, triggered by CPU hotplug or a cpuset partition
change, calls rq_attach_root() for every active CPU in the affected
domains. That takes the CPU offline and back online with the rq lock
held and interrupts disabled, so rq_offline_fair() runs for CPUs that
are not going down.

Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during
CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active
CPUs, but clear_tg_offline_cfs_rqs(), added by commit f60a631ab9ed
("sched/fair: Fix tg->load when offlining a CPU"), still walks every
task group and drops the CPU's tg->load_avg contribution, which
update_tg_load_avg() then adds back. With thousands of cpu cgroups each
rebuild costs O(nr_cpus * nr_task_groups).

Return early from rq_offline_fair() unless the CPU is going down, which
covers both walks, and drop the active check from
unthrottle_offline_cfs_rqs().

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):

  operation         before (ms)  after (ms)  change
  partition root          14.86        8.35    -44%
  cpu offline             49.99       33.91    -32%
  cpu online              36.10       31.02    -14%

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/fair.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index db9ab8ce36500..1b38c4c47b656 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7859,10 +7859,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
-	// Do not unthrottle for an active CPU
-	if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
-		return;
-
 	/*
 	 * The rq clock has already been updated in the
 	 * set_rq_offline(), so we should skip updating
@@ -15093,6 +15089,9 @@ static void rq_offline_fair(struct rq *rq, bool hotplug)
 {
 	update_sysctl();
 
+	if (!hotplug)
+		return;
+
 	/* Ensure any throttled groups are reachable by pick_next_task */
 	unthrottle_offline_cfs_rqs(rq);
 

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes
  2026-11-19  0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
  2026-11-19  0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
@ 2026-11-19  0:57 ` Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-19  0:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: linux-kernel, Qiliang Yuan

rq_online_fair() calls update_runtime_enabled(), which walks every task
group and takes each group's cfs_bandwidth lock to copy its quota state
into the CPU's cfs_rq. A sched domain rebuild reaches it for every
active CPU through rq_attach_root(), with the rq lock held and
interrupts disabled.

The walk is only needed when the CPU comes up. While a CPU is online,
tg_set_cfs_bandwidth() updates runtime_enabled on it under the rq lock,
and it holds cpus_read_lock() so it cannot miss a CPU that is coming up.

Call update_runtime_enabled() only when the CPU comes up.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug operations):

  operation         before (ms)  after (ms)  change
  partition root           8.35        1.20    -86%
  cpu offline             33.91       16.49    -51%
  cpu online              31.02       12.87    -59%

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1b38c4c47b656..e42c70e6ebeff 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -15082,7 +15082,9 @@ static void rq_online_fair(struct rq *rq, bool hotplug)
 {
 	update_sysctl();
 
-	update_runtime_enabled(rq);
+	/* tg_set_cfs_bandwidth() keeps an online CPU in sync */
+	if (hotplug)
+		update_runtime_enabled(rq);
 }
 
 static void rq_offline_fair(struct rq *rq, bool hotplug)

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-19  0:57 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19  0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-19  0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
2026-11-19  0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan
2026-11-19  0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®