mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Qiliang Yuan <qiliang.yuan@linux.dev>
To: Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	 Juri Lelli <juri.lelli@redhat.com>,
	 Vincent Guittot <vincent.guittot@linaro.org>,
	 Dietmar Eggemann <dietmar.eggemann@arm.com>,
	 Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>,  Mel Gorman <mgorman@suse.de>,
	Valentin Schneider <vschneid@redhat.com>,
	 K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Li kunyu <likunyu@open-hieco.net>,
	linux-kernel@vger.kernel.org,
	 Qiliang Yuan <qiliang.yuan@linux.dev>
Subject: [PATCH v3 2/3] sched: Keep CPUs that stay active online across root domain moves
Date: Fri, 20 Nov 2026 21:57:25 -0500	[thread overview]
Message-ID: <20261120-bug-sched-fair-offline-active-v3-2-325e9eefa56d@linux.dev> (raw)
In-Reply-To: <20261120-bug-sched-fair-offline-active-v3-0-325e9eefa56d@linux.dev>

A sched domain rebuild, from CPU hotplug or a cpuset partition change,
calls rq_attach_root() for every active CPU. rq_attach_root() takes the
CPU offline in its old root domain and puts it back online in the new
one, running all rq_offline() and rq_online() callbacks with the rq
lock held and interrupts disabled.

For fair, that walks all task groups twice per CPU: rq_offline_fair()
in clear_tg_offline_cfs_rqs() and rq_online_fair() in
update_runtime_enabled(), although nothing they maintain changes for a
CPU that stays active; tg_set_cfs_bandwidth() already keeps
runtime_enabled in sync on online CPUs. A rebuild thus costs
O(nr_cpus * nr_task_groups), and hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups.

Keep a CPU that stays active online and move only its root domain
state with the rq_attach_root() and rq_detach_root() callbacks. A CPU
that comes up is still put online by the rebuild in
sched_cpu_activate(), and one that is no longer active is still taken
offline there.

With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0,
switching a 4-CPU cpuset to a partition root and offlining or onlining
one CPU take (median of 20 partition switches and 16 hotplug
operations, min-max in parentheses):

  operation       before (ms)           after (ms)           change
  partition root  15.97 (14.76-35.88)    1.18 (1.14-2.26)      -93%
  cpu offline     49.66 (42.11-58.82)   14.49 (12.71-23.19)    -71%
  cpu online      37.39 (33.40-41.99)   13.70 (9.65-16.53)     -63%

Each rq_attach_root() call during those partition switches takes (1280
calls, change of the median):

  kernel  median (us)  p90 (us)  max (us)  change
  before          229       265      1335
  after             1         1        26    -99.6%

Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
 kernel/sched/core.c     | 27 +++++++++++++++++++++++++++
 kernel/sched/sched.h    |  2 ++
 kernel/sched/topology.c | 20 +++++++++++++++++---
 3 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca920a302ecba..49a0cee90a834 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8580,6 +8580,33 @@ void set_rq_offline(struct rq *rq)
 	}
 }
 
+void rq_attach_root_domain(struct rq *rq)
+{
+	struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+	const struct sched_class *class;
+
+	cpumask_set_cpu(rq->cpu, rd->online);
+
+	for_each_class(class) {
+		if (class->rq_attach_root)
+			class->rq_attach_root(rq);
+	}
+}
+
+void rq_detach_root_domain(struct rq *rq)
+{
+	struct root_domain *rd = rcu_dereference_root_domain(rq->rd);
+	const struct sched_class *class;
+
+	update_rq_clock(rq);
+	for_each_class(class) {
+		if (class->rq_detach_root)
+			class->rq_detach_root(rq);
+	}
+
+	cpumask_clear_cpu(rq->cpu, rd->online);
+}
+
 static inline void sched_set_rq_online(struct rq *rq, int cpu)
 {
 	struct root_domain *rd;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a24801d2058b4..1ceef4af26bc8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3534,6 +3534,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)
 
 extern void set_rq_online (struct rq *rq);
 extern void set_rq_offline(struct rq *rq);
+void rq_attach_root_domain(struct rq *rq);
+void rq_detach_root_domain(struct rq *rq);
 
 extern bool sched_smp_initialized;
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 99878da03bce7..cc2bc832dad1a 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -488,14 +488,26 @@ static void free_rootdomain(struct rcu_head *rcu)
 void rq_attach_root(struct rq *rq, struct root_domain *rd)
 {
 	struct root_domain *old_rd = NULL;
+	bool active, move = false;
 	struct rq_flags rf;
 
 	rq_lock_irqsave(rq, &rf);
 
+	active = cpumask_test_cpu(rq->cpu, cpu_active_mask);
+
 	old_rd = rcu_dereference_root_domain(rq->rd);
 	if (old_rd) {
-		if (cpumask_test_cpu(rq->cpu, old_rd->online))
-			set_rq_offline(rq);
+		if (cpumask_test_cpu(rq->cpu, old_rd->online)) {
+			/*
+			 * Keep an online CPU that stays active online and move
+			 * only its root domain state.
+			 */
+			move = active;
+			if (move)
+				rq_detach_root_domain(rq);
+			else
+				set_rq_offline(rq);
+		}
 
 		cpumask_clear_cpu(rq->cpu, old_rd->span);
 
@@ -512,7 +524,9 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	rcu_assign_pointer(rq->rd, rd);
 
 	cpumask_set_cpu(rq->cpu, rd->span);
-	if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
+	if (move)
+		rq_attach_root_domain(rq);
+	else if (active)
 		set_rq_online(rq);
 
 	/*

-- 
2.43.0


  parent reply	other threads:[~2026-11-21  2:57 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-21  2:57 [PATCH v3 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan
2026-11-21  2:57 ` [PATCH v3 1/3] sched: Give root domain moves their own sched_class callbacks Qiliang Yuan
2026-11-21  2:57 ` Qiliang Yuan [this message]
2026-11-21  2:57 ` [PATCH v3 3/3] sched/fair: Drop the active CPU check from unthrottle_offline_cfs_rqs() Qiliang Yuan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120-bug-sched-fair-offline-active-v3-2-325e9eefa56d@linux.dev \
    --to=qiliang.yuan@linux.dev \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=likunyu@open-hieco.net \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®