* [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds
@ 2026-11-19 0:57 Qiliang Yuan
2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: linux-kernel, Qiliang Yuan
A sched domain rebuild, from CPU hotplug or a cpuset partition change,
moves every active CPU to a new root domain through rq_attach_root().
For each of them rq_offline_fair() and rq_online_fair() walk all task
groups with the rq lock held and interrupts disabled, so a rebuild
costs O(nr_cpus * nr_task_groups). Dietmar pointed at the offline side
when f60a631ab9ed was reviewed [1]. Hosts running AI agent harnesses
commonly carry over a thousand cpu cgroups, and their VMs hot-add
vCPUs at run time.
Patch 1 tells the rq_online() and rq_offline() callbacks whether the
CPU actually changes state. Patches 2 and 3 use that to skip the two
fair walks when it doesn't.
With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0
(median of 20 partition switches and 16 hotplug operations, all three
kernels measured in the same run):
operation base (ms) patch 1-2 (ms) patch 1-3 (ms) change
partition root 14.86 8.35 1.20 -92%
cpu offline 49.99 33.91 16.49 -67%
cpu online 36.10 31.02 12.87 -64%
Each rq_attach_root() call, which runs with interrupts disabled, takes
(1280 calls over the 20 partition switches):
kernel median (us) p90 (us) max (us) change
base 229 256 6987
patch 1-2 122 162 988 -47%
patch 1-3 2 3 118 -99%
With all three patches, partition switches and onlining no longer grow
with the number of groups; offlining still walks them once for the CPU
that goes down. With RT_GROUP_SCHED enabled, the rt.c runtime walks
remain.
K Prateek Nayak proposed separate rq_attach_root() and rq_detach_root()
callbacks for the root domain moves instead [2]. This series follows
Peter's suggestion of a hotplug argument to rq_online() and rq_offline();
I'm happy to redo it on top of Prateek's approach if that is preferred.
[1] https://lore.kernel.org/r/dbd36b16-4acb-415a-9bbd-258b2e022475@arm.com
[2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
V1 -> V2:
- Pass a hotplug argument to rq_online() and rq_offline() instead of
adding a member to struct rq, as Peter suggested
- Skip both fair task group walks on rebuilds, which leaves no second
loop to merge
- Rebase onto tip/sched/core, where sched_ext no longer implements
rq_online() and rq_offline(), and update the numbers from a new run
CPU hotplug and partition flips were also run with SCHED_FIFO tasks
(rt_group_sched=1, RT_RUNTIME_SHARE on and off), SCHED_DEADLINE tasks
and scx_simple loaded, plus the sched_ext selftests, with the same
results as on tip/sched/core (b251360cb2e6c).
v1: https://lore.kernel.org/r/20261111-bug-sched-fair-offline-active-v1-0-4f54523c5a70@linux.dev
---
Qiliang Yuan (3):
sched: Tell rq_online() and rq_offline() whether the CPU changes state
sched/fair: Skip rq_offline_fair() work for CPUs that stay active
sched/fair: Avoid re-syncing runtime_enabled on root domain changes
kernel/sched/core.c | 12 ++++++------
kernel/sched/deadline.c | 4 ++--
kernel/sched/fair.c | 15 ++++++++-------
kernel/sched/rt.c | 4 ++--
kernel/sched/sched.h | 8 ++++----
kernel/sched/topology.c | 13 ++++++++++---
6 files changed, 32 insertions(+), 24 deletions(-)
---
base-commit: b251360cb2e6c7b74d962831f5797a4520511b36
change-id: 20261111-bug-sched-fair-offline-active-8d063e735d3f
Best regards,
--
Qiliang Yuan <qiliang.yuan@linux.dev>
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state 2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan @ 2026-11-19 0:57 ` Qiliang Yuan 2026-11-19 8:29 ` Li kunyu 2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan 2 siblings, 1 reply; 7+ messages in thread From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak Cc: linux-kernel, Qiliang Yuan set_rq_online() and set_rq_offline() run both when a CPU comes up or goes down and when a sched domain rebuild moves an active CPU to a new root domain through rq_attach_root(). The class callbacks get the same call in both cases. Work such as resyncing CFS bandwidth state or unthrottling cfs_rqs is only needed when the CPU actually changes state, but the callbacks have no way to tell, so they repeat it for every active CPU on each rebuild. Pass a hotplug argument down to the callbacks, and have sched_set_rq_online() and sched_set_rq_offline() pass true. rq_attach_root() passes false when taking a CPU offline, as sched_cpu_deactivate() has already taken a CPU going down offline before the rebuild. When putting a CPU online it passes true only if the CPU was not online in its old root domain: sched_cpu_activate() rebuilds the domains before calling sched_set_rq_online(), so this is where a CPU coming up first goes online. Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev> --- kernel/sched/core.c | 12 ++++++------ kernel/sched/deadline.c | 4 ++-- kernel/sched/fair.c | 4 ++-- kernel/sched/rt.c | 4 ++-- kernel/sched/sched.h | 8 ++++---- kernel/sched/topology.c | 13 ++++++++++--- 6 files changed, 26 insertions(+), 19 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8b4e4a6f4ae02..17ceb9d5046ba 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8542,7 +8542,7 @@ static inline void balance_hotplug_wait(void) #endif /* !CONFIG_HOTPLUG_CPU */ -void set_rq_online(struct rq *rq) +void set_rq_online(struct rq *rq, bool hotplug) { if (!rq->online) { const struct sched_class *class; @@ -8554,12 +8554,12 @@ void set_rq_online(struct rq *rq) for_each_class(class) { if (class->rq_online) - class->rq_online(rq); + class->rq_online(rq, hotplug); } } } -void set_rq_offline(struct rq *rq) +void set_rq_offline(struct rq *rq, bool hotplug) { if (rq->online) { struct root_domain *rd = rcu_dereference_root_domain(rq->rd); @@ -8568,7 +8568,7 @@ void set_rq_offline(struct rq *rq) update_rq_clock(rq); for_each_class(class) { if (class->rq_offline) - class->rq_offline(rq); + class->rq_offline(rq, hotplug); } cpumask_clear_cpu(rq->cpu, rd->online); @@ -8585,7 +8585,7 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu) rd = rcu_dereference_root_domain(rq->rd); if (rd) { BUG_ON(!cpumask_test_cpu(cpu, rd->span)); - set_rq_online(rq); + set_rq_online(rq, true); } rq_unlock_irqrestore(rq, &rf); } @@ -8599,7 +8599,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu) rd = rcu_dereference_root_domain(rq->rd); if (rd) { BUG_ON(!cpumask_test_cpu(cpu, rd->span)); - set_rq_offline(rq); + set_rq_offline(rq, true); } rq_unlock_irqrestore(rq, &rf); } diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 61b6a6d781715..73f509b87b8fa 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -3366,7 +3366,7 @@ bool dl_task_needs_bw_move(struct task_struct *p, } /* Assumes rq->lock is held */ -static void rq_online_dl(struct rq *rq) +static void rq_online_dl(struct rq *rq, bool hotplug) { if (rq->dl.overloaded) dl_set_overload(rq); @@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq) } /* Assumes rq->lock is held */ -static void rq_offline_dl(struct rq *rq) +static void rq_offline_dl(struct rq *rq, bool hotplug) { if (rq->dl.overloaded) dl_clear_overload(rq); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 731d78155e654..db9ab8ce36500 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -15082,14 +15082,14 @@ void sched_balance_trigger(struct rq *rq) nohz_balancer_kick(rq); } -static void rq_online_fair(struct rq *rq) +static void rq_online_fair(struct rq *rq, bool hotplug) { update_sysctl(); update_runtime_enabled(rq); } -static void rq_offline_fair(struct rq *rq) +static void rq_offline_fair(struct rq *rq, bool hotplug) { update_sysctl(); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 81a71aa5f094a..fc6224a343552 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) } /* Assumes rq->lock is held */ -static void rq_online_rt(struct rq *rq) +static void rq_online_rt(struct rq *rq, bool hotplug) { if (rq->rt.overloaded) rt_set_overload(rq); @@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq) } /* Assumes rq->lock is held */ -static void rq_offline_rt(struct rq *rq) +static void rq_offline_rt(struct rq *rq, bool hotplug) { if (rq->rt.overloaded) rt_clear_overload(rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e957524c3ee52..5f7b6ab4dde6d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2796,8 +2796,8 @@ struct sched_class { /* * sched_set_rq_{on,off}line: rq->lock */ - void (*rq_online)(struct rq *rq); - void (*rq_offline)(struct rq *rq); + void (*rq_online)(struct rq *rq, bool hotplug); + void (*rq_offline)(struct rq *rq, bool hotplug); /* * push_cpu_stop: p->pi_lock && rq->lock @@ -3526,8 +3526,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2) raw_spin_rq_unlock(rq1); } -extern void set_rq_online (struct rq *rq); -extern void set_rq_offline(struct rq *rq); +extern void set_rq_online(struct rq *rq, bool hotplug); +extern void set_rq_offline(struct rq *rq, bool hotplug); extern bool sched_smp_initialized; diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 99878da03bce7..577dc012e5ac5 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -488,14 +488,16 @@ static void free_rootdomain(struct rcu_head *rcu) void rq_attach_root(struct rq *rq, struct root_domain *rd) { struct root_domain *old_rd = NULL; + bool was_online = false; struct rq_flags rf; rq_lock_irqsave(rq, &rf); old_rd = rcu_dereference_root_domain(rq->rd); if (old_rd) { - if (cpumask_test_cpu(rq->cpu, old_rd->online)) - set_rq_offline(rq); + was_online = cpumask_test_cpu(rq->cpu, old_rd->online); + if (was_online) + set_rq_offline(rq, false); cpumask_clear_cpu(rq->cpu, old_rd->span); @@ -512,8 +514,13 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd) rcu_assign_pointer(rq->rd, rd); cpumask_set_cpu(rq->cpu, rd->span); + /* + * A CPU that comes up is first put online here, by the sched domain + * rebuild of sched_cpu_activate(). Any other CPU only moves between + * root domains. + */ if (cpumask_test_cpu(rq->cpu, cpu_active_mask)) - set_rq_online(rq); + set_rq_online(rq, !was_online); /* * Because the rq is not a task, dl_add_task_root_domain() did not -- 2.43.0 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state 2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan @ 2026-11-19 8:29 ` Li kunyu 2026-11-20 1:51 ` Qiliang Yuan 0 siblings, 1 reply; 7+ messages in thread From: Li kunyu @ 2026-11-19 8:29 UTC (permalink / raw) To: Qiliang Yuan, Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak Cc: linux-kernel Hi, On 2026/11/19 8:57, Qiliang Yuan wrote: > -void set_rq_online(struct rq *rq) > +void set_rq_online(struct rq *rq, bool hotplug) > { > if (!rq->online) { > const struct sched_class *class; > @@ -8554,12 +8554,12 @@ void set_rq_online(struct rq *rq) > > for_each_class(class) { > if (class->rq_online) > - class->rq_online(rq); > + class->rq_online(rq, hotplug); > } > } > } > > -void set_rq_offline(struct rq *rq) > +void set_rq_offline(struct rq *rq, bool hotplug) > { > if (rq->online) { > struct root_domain *rd = rcu_dereference_root_domain(rq->rd); > @@ -8568,7 +8568,7 @@ void set_rq_offline(struct rq *rq) > update_rq_clock(rq); > for_each_class(class) { > if (class->rq_offline) > - class->rq_offline(rq); > + class->rq_offline(rq, hotplug); > } > > cpumask_clear_cpu(rq->cpu, rd->online); > @@ -8585,7 +8585,7 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu) > rd = rcu_dereference_root_domain(rq->rd); > if (rd) { > BUG_ON(!cpumask_test_cpu(cpu, rd->span)); > - set_rq_online(rq); > + set_rq_online(rq, true); > } > rq_unlock_irqrestore(rq, &rf); > } > @@ -8599,7 +8599,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu) > rd = rcu_dereference_root_domain(rq->rd); > if (rd) { > BUG_ON(!cpumask_test_cpu(cpu, rd->span)); > - set_rq_offline(rq); > + set_rq_offline(rq, true); > } > rq_unlock_irqrestore(rq, &rf); > } > > +static void rq_online_dl(struct rq *rq, bool hotplug) > { > if (rq->dl.overloaded) > dl_set_overload(rq); > @@ -3379,7 +3379,7 @@ static void rq_online_dl(struct rq *rq) > } > > /* Assumes rq->lock is held */ > -static void rq_offline_dl(struct rq *rq) > +static void rq_offline_dl(struct rq *rq, bool hotplug) > { > if (rq->dl.overloaded) > dl_clear_overload(rq); > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 731d78155e654..db9ab8ce36500 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -15082,14 +15082,14 @@ void sched_balance_trigger(struct rq *rq) > nohz_balancer_kick(rq); > } > > > -static void rq_online_fair(struct rq *rq) > +static void rq_online_fair(struct rq *rq, bool hotplug) > { > update_sysctl(); > > update_runtime_enabled(rq); > } > > -static void rq_offline_fair(struct rq *rq) > +static void rq_offline_fair(struct rq *rq, bool hotplug) > { > update_sysctl(); > > diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c > index 81a71aa5f094a..fc6224a343552 100644 > --- a/kernel/sched/rt.c > +++ b/kernel/sched/rt.c > @@ -2398,7 +2398,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) > } > > /* Assumes rq->lock is held */ > -static void rq_online_rt(struct rq *rq) > +static void rq_online_rt(struct rq *rq, bool hotplug) > { > > if (rq->rt.overloaded) > rt_set_overload(rq); > @@ -2409,7 +2409,7 @@ static void rq_online_rt(struct rq *rq) > } > > /* Assumes rq->lock is held */ > -static void rq_offline_rt(struct rq *rq) > +static void rq_offline_rt(struct rq *rq, bool hotplug) > { > if (rq->rt.overloaded) > rt_clear_overload(rq); > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index e957524c3ee52..5f7b6ab4dde6d 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -2796,8 +2796,8 @@ struct sched_class { > /* > * sched_set_rq_{on,off}line: rq->lock > */ > - void (*rq_online)(struct rq *rq); > - void (*rq_offline)(struct rq *rq); > + void (*rq_online)(struct rq *rq, bool hotplug); > + void (*rq_offline)(struct rq *rq, bool hotplug); I noticed that the functions set_rq_online and set_rq_offline have added a new bool hotplug parameter, and the corresponding scheduler class callback declarations have also been updated accordingly. However, in the actual implementations - such as rq_online_fair(struct rq *rq, bool hotplug) - this parameter does not appear to be used. Is this parameter currently unused, or is it being introduced at the callback layer in advance for future use cases? -- thanks, kunyu ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state 2026-11-19 8:29 ` Li kunyu @ 2026-11-20 1:51 ` Qiliang Yuan 2026-11-20 2:24 ` Li kunyu 0 siblings, 1 reply; 7+ messages in thread From: Qiliang Yuan @ 2026-11-20 1:51 UTC (permalink / raw) To: Li kunyu Cc: Qiliang Yuan, Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, linux-kernel On Thu, Nov 19, 2026 at 04:29:27PM +0800, Li kunyu wrote: > Is this parameter currently unused, or is it being introduced at the callback > layer in advance for future use cases? Only the fair class uses it, later in this series: callback uses hotplug rq_offline_fair() yes, patch 2 returns early when !hotplug rq_online_fair() yes, patch 3 calls update_runtime_enabled() only when hotplug is true rq_online_rt(), rq_offline_rt() no rq_online_dl(), rq_offline_dl() no The rt and dl callbacks take it only because they share the sched_class callback type, and they have no use for it, whatever tasks run on the CPU. They move state that lives in the root domain, so they have to do the same work on hotplug and on a root domain change: callback work why also on !hotplug rq_offline_rt() rt_clear_overload(), rto_mask and cpupri live in __disable_runtime(), rq->rd, so leave the old root cpupri -> CPUPRI_INVALID domain; reclaim RT runtime lent within it rq_online_rt() rt_set_overload(), register the overload state __enable_runtime(), and the highest RT priority in cpupri_set() the new root domain for push/pull rq_offline_dl() dl_clear_overload(), dlo_mask and cpudl live in cpudl_clear() rq->rd, so leave the old root domain rq_online_dl() dl_set_overload(), register the earliest deadline cpudl_set()/cpudl_clear() in the new root domain for push/pull The fair callbacks handle task group state (throttling and the bandwidth switch), which doesn't depend on the root domain, so they can skip it when the CPU stays active. Passing a bool follows Peter's suggestion on v1 [1]. Prateek proposed instead to give root domain changes their own callbacks, implemented by rt and dl only, which leaves rq_online() and rq_offline() to CPU hotplug and needs no parameter [2]. As only the fair class needs the distinction, I'll rework the series along Prateek's lines in v3, so that rt and dl don't carry a parameter they never use. [1] https://lore.kernel.org/r/20261112125947.GZ1813469@noisy.programming.kicks-ass.net [2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com Thanks, Qiliang ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state 2026-11-20 1:51 ` Qiliang Yuan @ 2026-11-20 2:24 ` Li kunyu 0 siblings, 0 replies; 7+ messages in thread From: Li kunyu @ 2026-11-20 2:24 UTC (permalink / raw) To: Qiliang Yuan Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, linux-kernel Hi Qiliang, On 2026/11/20 9:51, Qiliang Yuan wrote: > On Thu, Nov 19, 2026 at 04:29:27PM +0800, Li kunyu wrote: >> Is this parameter currently unused, or is it being introduced at the callback >> layer in advance for future use cases? > > Only the fair class uses it, later in this series: > > callback uses hotplug > rq_offline_fair() yes, patch 2 returns early when !hotplug > rq_online_fair() yes, patch 3 calls update_runtime_enabled() > only when hotplug is true Thank you for the explanation. I have indeed located the relevant calls in Patches 2 and 3. > rq_online_rt(), rq_offline_rt() no > rq_online_dl(), rq_offline_dl() no > > The rt and dl callbacks take it only because they share the > sched_class callback type, and they have no use for it, whatever > tasks run on the CPU. They move state that lives in the root domain, > so they have to do the same work on hotplug and on a root domain > change: > > callback work why also on !hotplug > rq_offline_rt() rt_clear_overload(), rto_mask and cpupri live in > __disable_runtime(), rq->rd, so leave the old root > cpupri -> CPUPRI_INVALID domain; reclaim RT runtime > lent within it > rq_online_rt() rt_set_overload(), register the overload state > __enable_runtime(), and the highest RT priority in > cpupri_set() the new root domain for push/pull > rq_offline_dl() dl_clear_overload(), dlo_mask and cpudl live in > cpudl_clear() rq->rd, so leave the old root > domain > rq_online_dl() dl_set_overload(), register the earliest deadline > cpudl_set()/cpudl_clear() in the new root domain for > push/pull > > The fair callbacks handle task group state (throttling and the > bandwidth switch), which doesn't depend on the root domain, so they > can skip it when the CPU stays active. > > Passing a bool follows Peter's suggestion on v1 [1]. Prateek proposed > instead to give root domain changes their own callbacks, implemented > by rt and dl only, which leaves rq_online() and rq_offline() to CPU > hotplug and needs no parameter [2]. > > As only the fair class needs the distinction, I'll rework the series > along Prateek's lines in v3, so that rt and dl don't carry a > parameter they never use. > > [1] https://lore.kernel.org/r/20261112125947.GZ1813469@noisy.programming.kicks-ass.net > [2] https://lore.kernel.org/r/e1a63a3c-3975-4b05-8ab4-518c37d867d6@amd.com > > Thanks, > Qiliang > Looking forward to the improvements in the v3! -- thanks, kunyu ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active 2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan @ 2026-11-19 0:57 ` Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan 2 siblings, 0 replies; 7+ messages in thread From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak Cc: linux-kernel, Qiliang Yuan A sched domain rebuild, triggered by CPU hotplug or a cpuset partition change, calls rq_attach_root() for every active CPU in the affected domains. That takes the CPU offline and back online with the rq lock held and interrupts disabled, so rq_offline_fair() runs for CPUs that are not going down. Commit af98d8a36a96 ("sched/fair: Fix CPU bandwidth limit bypass during CPU hotplug") made unthrottle_offline_cfs_rqs() return early for active CPUs, but clear_tg_offline_cfs_rqs(), added by commit f60a631ab9ed ("sched/fair: Fix tg->load when offlining a CPU"), still walks every task group and drops the CPU's tg->load_avg contribution, which update_tg_load_avg() then adds back. With thousands of cpu cgroups each rebuild costs O(nr_cpus * nr_task_groups). Return early from rq_offline_fair() unless the CPU is going down, which covers both walks, and drop the active check from unthrottle_offline_cfs_rqs(). With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0, switching a 4-CPU cpuset to a partition root and offlining or onlining one CPU take (median of 20 partition switches and 16 hotplug operations): operation before (ms) after (ms) change partition root 14.86 8.35 -44% cpu offline 49.99 33.91 -32% cpu online 36.10 31.02 -14% Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev> --- kernel/sched/fair.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index db9ab8ce36500..1b38c4c47b656 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7859,10 +7859,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) lockdep_assert_rq_held(rq); - // Do not unthrottle for an active CPU - if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask)) - return; - /* * The rq clock has already been updated in the * set_rq_offline(), so we should skip updating @@ -15093,6 +15089,9 @@ static void rq_offline_fair(struct rq *rq, bool hotplug) { update_sysctl(); + if (!hotplug) + return; + /* Ensure any throttled groups are reachable by pick_next_task */ unthrottle_offline_cfs_rqs(rq); -- 2.43.0 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes 2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan @ 2026-11-19 0:57 ` Qiliang Yuan 2 siblings, 0 replies; 7+ messages in thread From: Qiliang Yuan @ 2026-11-19 0:57 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak Cc: linux-kernel, Qiliang Yuan rq_online_fair() calls update_runtime_enabled(), which walks every task group and takes each group's cfs_bandwidth lock to copy its quota state into the CPU's cfs_rq. A sched domain rebuild reaches it for every active CPU through rq_attach_root(), with the rq lock held and interrupts disabled. The walk is only needed when the CPU comes up. While a CPU is online, tg_set_cfs_bandwidth() updates runtime_enabled on it under the rq lock, and it holds cpus_read_lock() so it cannot miss a CPU that is coming up. Call update_runtime_enabled() only when the CPU comes up. With 4000 cpu cgroups on a 32-CPU VM booted with rt_group_sched=0, switching a 4-CPU cpuset to a partition root and offlining or onlining one CPU take (median of 20 partition switches and 16 hotplug operations): operation before (ms) after (ms) change partition root 8.35 1.20 -86% cpu offline 33.91 16.49 -51% cpu online 31.02 12.87 -59% Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev> --- kernel/sched/fair.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1b38c4c47b656..e42c70e6ebeff 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -15082,7 +15082,9 @@ static void rq_online_fair(struct rq *rq, bool hotplug) { update_sysctl(); - update_runtime_enabled(rq); + /* tg_set_cfs_bandwidth() keeps an online CPU in sync */ + if (hotplug) + update_runtime_enabled(rq); } static void rq_offline_fair(struct rq *rq, bool hotplug) -- 2.43.0 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-20 2:24 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-11-19 0:57 [PATCH v2 0/3] sched/fair: Cut per-group work from sched domain rebuilds Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 1/3] sched: Tell rq_online() and rq_offline() whether the CPU changes state Qiliang Yuan 2026-11-19 8:29 ` Li kunyu 2026-11-20 1:51 ` Qiliang Yuan 2026-11-20 2:24 ` Li kunyu 2026-11-19 0:57 ` [PATCH v2 2/3] sched/fair: Skip rq_offline_fair() work for CPUs that stay active Qiliang Yuan 2026-11-19 0:57 ` [PATCH v2 3/3] sched/fair: Avoid re-syncing runtime_enabled on root domain changes Qiliang Yuan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®