mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds
@ 2026-11-21  2:57 Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21  2:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: Li kunyu, linux-kernel, Qiliang Yuan

A sched domain rebuild, from CPU hotplug or a cpuset partition change,
moves every active CPU to a new root domain through rq_attach_root().
It does so by taking the CPU offline and back online, and
rq_offline_fair() and rq_online_fair() then walk all task groups with
the rq lock held and interrupts disabled, so a rebuild costs
O(nr_cpus * nr_task_groups). Dietmar pointed at the offline side when
f60a631ab9ed was reviewed [1]. Hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups, and their VMs hot-add vCPUs
at run time.

Based on Prateek's patch [2], patch 1 gives root domain moves their
own sched_class callbacks, which only rt and dl implement.
Patch 2 keeps CPUs that stay active online across a rebuild and moves
only their root domain state, so fair is no longer called for them.
Patch 3 drops a check in fair that patch 2 leaves dead.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0
(median of 20 partition switches and 16 hotplug operations, both
kernels measured in the same run):

  operation       base (ms)  patch 1-3 (ms)  change
  partition root      15.97            1.19    -93%
  cpu offline         49.66           15.74    -68%
  cpu online          37.39           13.33    -64%

Each rq_attach_root() call, which runs with interrupts disabled, takes
(1280 calls over the 20 partition switches, change of the median):

  kernel      median (us)  p90 (us)  max (us)  change
  base                229       265      1335
  patch 1-3             1         1        10    -99.6%

Partition switches and onlining no longer grow with the number of
groups; offlining still walks them once for the CPU that goes down.
With RT_GROUP_SCHED enabled, the rt.c runtime walks remain.

[1] https://lore.kernel.org/r/dbd36b16-4acb-415a-9bbd-258b2e022475@arm.com
[2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
V2 -> V3:
- Give root domain moves their own rq_attach_root() and rq_detach_root()
  callbacks, implemented by rt and dl, and keep CPUs that stay active
  online across rebuilds instead of passing a hotplug argument to
  rq_online() and rq_offline() (K Prateek Nayak)
- Keep putting a CPU that comes up online from the rebuild in
  sched_cpu_activate()
- Move the removal of the active CPU check in
  unthrottle_offline_cfs_rqs() into patch 3
- Update the numbers from a new run

V1 -> V2:
- Pass a hotplug argument to rq_online() and rq_offline() instead of
  adding a member to struct rq (Peter Zijlstra)
- Skip both fair task group walks on rebuilds, which leaves no second
  loop to merge
- Rebase onto tip/sched/core, where sched_ext no longer implements
  rq_online() and rq_offline(), and update the numbers from a new run

CPU hotplug and partition flips were also run with SCHED_FIFO tasks
(rt_group_sched=1, RT_RUNTIME_SHARE on and off), SCHED_DEADLINE tasks
and scx_simple loaded, plus the sched_ext selftests, with the same
results as on tip/sched/core (b251360cb2e6c).

v2: https://lore.kernel.org/r/20261118-bug-sched-fair-offline-active-v2-0-b6ede596c6a6@linux.dev
v1: https://lore.kernel.org/r/20261111-bug-sched-fair-offline-active-v1-0-4f54523c5a70@linux.dev

---
Qiliang Yuan (3):
      sched: Give root domain moves their own sched_class callbacks
      sched: Keep CPUs that stay active online across root domain moves
      sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs()

 kernel/sched/core.c        | 31 +++++++++++++++++++++++++++++++
 kernel/sched/cpudeadline.c |  5 +++--
 kernel/sched/deadline.c    |  8 ++++----
 kernel/sched/fair.c        |  4 ----
 kernel/sched/rt.c          |  8 ++++----
 kernel/sched/sched.h       |  8 ++++++++
 kernel/sched/topology.c    | 20 +++++++++++++++++---
 7 files changed, 67 insertions(+), 17 deletions(-)
---
base-commit: b251360cb2e6c7b74d962831f5797a4520511b36
change-id: 20261111-bug-sched-fair-offline-active-8d063e735d3f

Best regards,
-- 
Qiliang Yuan <qiliang.yuan@linux.dev>


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks
  2026-11-21  2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
@ 2026-11-21  2:57 ` Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21  2:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: Li kunyu, linux-kernel, Qiliang Yuan

The rq_online() and rq_offline() callbacks run when a CPU comes up or
goes down, and also when rq_attach_root() moves an active CPU to a new
root domain during a sched domain rebuild. rt and dl use them for state
kept in the root domain: the overload masks, cpupri, cpudl and the RT
runtime shared within it. fair uses them for task group state, which
does not depend on the root domain.

With one pair of callbacks for both events, a class cannot do only the
work that a root domain move needs, so every rebuild runs the fair
hotplug work for all active CPUs.

Add rq_attach_root() and rq_detach_root() to struct sched_class for the
root domain state and convert rt and dl to them. Have set_rq_online()
and set_rq_offline() call both kinds of callbacks class by class, so
that every class still runs at the same points and in the same order.

Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/core.c        | 4 ++++
 kernel/sched/cpudeadline.c | 5 +++--
 kernel/sched/deadline.c    | 8 ++++----
 kernel/sched/rt.c          | 8 ++++----
 kernel/sched/sched.h       | 6 ++++++
 5 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b4e4a6f4ae02..ca920a302ecba 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8553,6 +8553,8 @@ void set_rq_online(struct rq *rq)
 		rq->online = 1;
 
 		for_each_class(class) {
+			if (class->rq_attach_root)
+				class->rq_attach_root(rq);
 			if (class->rq_online)
 				class->rq_online(rq);
 		}
@@ -8569,6 +8571,8 @@ void set_rq_offline(struct rq *rq)
 		for_each_class(class) {
 			if (class->rq_offline)
 				class->rq_offline(rq);
+			if (class->rq_detach_root)
+				class->rq_detach_root(rq);
 		}
 
 		cpumask_clear_cpu(rq->cpu, rd->online);
diff --git a/kernel/sched/cpudeadline.c b/kernel/sched/cpudeadline.c
index 0a2b7e30fd10c..8e2a95f846278 100644
--- a/kernel/sched/cpudeadline.c
+++ b/kernel/sched/cpudeadline.c
@@ -185,8 +185,9 @@ void cpudl_clear(struct cpudl *cp, int cpu, bool online)
 	if (old_idx == IDX_INVALID) {
 		/*
 		 * Nothing to remove if old_idx was invalid.
-		 * This could happen if rq_online_dl or rq_offline_dl is
-		 * called for a CPU without -dl tasks running.
+		 * This could happen if rq_attach_root_dl() or
+		 * rq_detach_root_dl() is called for a CPU without -dl
+		 * tasks running.
 		 */
 	} else {
 		new_cpu = cp->elements[cp->size - 1].cpu;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 61b6a6d781715..fcbf53500083e 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3366,7 +3366,7 @@ bool dl_task_needs_bw_move(struct task_struct *p,
 }
 
 /* Assumes rq->lock is held */
-static void rq_online_dl(struct rq *rq)
+static void rq_attach_root_dl(struct rq *rq)
 {
 	if (rq->dl.overloaded)
 		dl_set_overload(rq);
@@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq)
 }
 
 /* Assumes rq->lock is held */
-static void rq_offline_dl(struct rq *rq)
+static void rq_detach_root_dl(struct rq *rq)
 {
 	if (rq->dl.overloaded)
 		dl_clear_overload(rq);
@@ -3664,8 +3664,8 @@ DEFINE_SCHED_CLASS(dl) = {
 	.select_task_rq		= select_task_rq_dl,
 	.migrate_task_rq	= migrate_task_rq_dl,
 	.set_cpus_allowed       = set_cpus_allowed_dl,
-	.rq_online              = rq_online_dl,
-	.rq_offline             = rq_offline_dl,
+	.rq_attach_root		= rq_attach_root_dl,
+	.rq_detach_root		= rq_detach_root_dl,
 	.task_woken		= task_woken_dl,
 	.find_lock_rq		= find_lock_later_rq,
 
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 81a71aa5f094a..45d10334f7253 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
 }
 
 /* Assumes rq->lock is held */
-static void rq_online_rt(struct rq *rq)
+static void rq_attach_root_rt(struct rq *rq)
 {
 	if (rq->rt.overloaded)
 		rt_set_overload(rq);
@@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq)
 }
 
 /* Assumes rq->lock is held */
-static void rq_offline_rt(struct rq *rq)
+static void rq_detach_root_rt(struct rq *rq)
 {
 	if (rq->rt.overloaded)
 		rt_clear_overload(rq);
@@ -2626,8 +2626,8 @@ DEFINE_SCHED_CLASS(rt) = {
 	.balance		= balance_rt,
 	.select_task_rq		= select_task_rq_rt,
 	.set_cpus_allowed       = set_cpus_allowed_common,
-	.rq_online              = rq_online_rt,
-	.rq_offline             = rq_offline_rt,
+	.rq_attach_root		= rq_attach_root_rt,
+	.rq_detach_root		= rq_detach_root_rt,
 	.task_woken		= task_woken_rt,
 	.switched_from		= switched_from_rt,
 	.find_lock_rq		= find_lock_lowest_rq,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e957524c3ee52..a24801d2058b4 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2799,6 +2799,12 @@ struct sched_class {
 	void (*rq_online)(struct rq *rq);
 	void (*rq_offline)(struct rq *rq);
 
+	/*
+	 * Move an online rq to or from a root domain: rq->lock
+	 */
+	void (*rq_attach_root)(struct rq *rq);
+	void (*rq_detach_root)(struct rq *rq);
+
 	/*
 	 * push_cpu_stop: p->pi_lock && rq->lock
 	 */

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves
  2026-11-21  2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
@ 2026-11-21  2:57 ` Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21  2:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: Li kunyu, linux-kernel, Qiliang Yuan

A sched domain rebuild, from CPU hotplug or a cpuset partition change,
calls rq_attach_root() for every active CPU. rq_attach_root() takes the
CPU offline in its old root domain and puts it back online in the new
one, running all rq_offline() and rq_online() callbacks with the rq
lock held and interrupts disabled.

For fair, that walks all task groups twice per CPU: rq_offline_fair()
in clear_tg_offline_cfs_rqs() and rq_online_fair() in
update_runtime_enabled(), although nothing they maintain changes for a
CPU that stays active; tg_set_cfs_bandwidth() already keeps
runtime_enabled in sync on online CPUs. A rebuild thus costs
O(nr_cpus * nr_task_groups), and hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups.

Keep a CPU that stays active online and move only its root domain
state with the rq_attach_root() and rq_detach_root() callbacks. A CPU
that comes up is still put online by the rebuild in
sched_cpu_activate(), and one that is no longer active is still taken
offline there.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug
operations, min-max in parentheses):

  operation       before (ms)           after (ms)           change
  partition root  15.97 (14.76-35.88)    1.18 (1.14-2.26)      -93%
  cpu offline     49.66 (42.11-58.82)   14.49 (12.71-23.19)    -71%
  cpu online      37.39 (33.40-41.99)   13.70 (9.65-16.53)     -63%

Each rq_attach_root() call during those partition switches takes (1280
calls, change of the median):

  kernel  median (us)  p90 (us)  max (us)  change
  before          229       265      1335
  after             1         1        26    -99.6%

Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/core.c     | 27 +++++++++++++++++++++++++++
 kernel/sched/sched.h    |  2 ++
 kernel/sched/topology.c | 20 +++++++++++++++++---
 3 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca920a302ecba..49a0cee90a834 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8580,6 +8580,33 @@ void set_rq_offline(struct rq *rq)
 	}
 }
 
+void rq_attach_root_domain(struct rq *rq)
+{
+	struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+	const struct sched_class *class;
+
+	cpumask_set_cpu(rq->cpu, rd->online);
+
+	for_each_class(class) {
+		if (class->rq_attach_root)
+			class->rq_attach_root(rq);
+	}
+}
+
+void rq_detach_root_domain(struct rq *rq)
+{
+	struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+	const struct sched_class *class;
+
+	update_rq_clock(rq);
+	for_each_class(class) {
+		if (class->rq_detach_root)
+			class->rq_detach_root(rq);
+	}
+
+	cpumask_clear_cpu(rq->cpu, rd->online);
+}
+
 static inline void sched_set_rq_online(struct rq *rq, int cpu)
 {
 	struct root_domain *rd;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a24801d2058b4..1ceef4af26bc8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3534,6 +3534,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
 
 extern void set_rq_online (struct rq *rq);
 extern void set_rq_offline(struct rq *rq);
+void rq_attach_root_domain(struct rq *rq);
+void rq_detach_root_domain(struct rq *rq);
 
 extern bool sched_smp_initialized;
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..cc2bc832dad1a 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,26 @@ static void free_rootdomain(struct rcu_head *rcu)
 void rq_attach_root(struct rq *rq, struct root_domain *rd)
 {
 	struct root_domain *old_rd = NULL;
+	bool active, move = false;
 	struct rq_flags rf;
 
 	rq_lock_irqsave(rq, &rf);
 
+	active = cpumask_test_cpu(rq->cpu, cpu_active_mask);
+
 	old_rd = rcu_dereference_root_domain(rq->rd);
 	if (old_rd) {
-		if (cpumask_test_cpu(rq->cpu, old_rd->online))
-			set_rq_offline(rq);
+		if (cpumask_test_cpu(rq->cpu, old_rd->online)) {
+			/*
+			 * Keep an online CPU that stays active online and move
+			 * only its root domain state.
+			 */
+			move = active;
+			if (move)
+				rq_detach_root_domain(rq);
+			else
+				set_rq_offline(rq);
+		}
 
 		cpumask_clear_cpu(rq->cpu, old_rd->span);
 
@@ -512,7 +524,9 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	rcu_assign_pointer(rq->rd, rd);
 
 	cpumask_set_cpu(rq->cpu, rd->span);
-	if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
+	if (move)
+		rq_attach_root_domain(rq);
+	else if (active)
 		set_rq_online(rq);
 
 	/*

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs()
  2026-11-21  2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
  2026-11-21  2:57 ` [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves Qiliang Yuan
@ 2026-11-21  2:57 ` Qiliang Yuan
  2 siblings, 0 replies; 4+ messages in thread
From: Qiliang Yuan @ 2026-11-21  2:57 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: Li kunyu, linux-kernel, Qiliang Yuan

Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during
CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active
CPUs, because every sched domain rebuild took all active CPUs offline
and unthrottled their throttled groups.

Rebuilds now keep active CPUs online, so rq_offline_fair() only runs
for a CPU that has already left cpu_active_mask: sched_cpu_deactivate()
clears it before taking the rq offline, and rq_attach_root() takes only
inactive CPUs offline. The check can no longer be true.

Remove it.

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/fair.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 731d78155e654..e366040f4fb4f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7859,10 +7859,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
-	// Do not unthrottle for an active CPU
-	if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
-		return;
-
 	/*
 	 * The rq clock has already been updated in the
 	 * set_rq_offline(), so we should skip updating

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-21  2:57 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-21  2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-21  2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
2026-11-21  2:57 ` [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves Qiliang Yuan
2026-11-21  2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®