* [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy
@ 2026-06-02 5:00 K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards K Prateek Nayak
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:00 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
v2 addresses comments from Ben, Aaron, and Peter with a copule of small
optimizations on top. Individual patches contain the changelog.
Introduction
============
The current hierarchy is always throttled in __schedule() during the
pick when update_curr() detects a cfs_rq running out of the bandwidth
and issues a resched.
This was necessary prior to per-task throttling where the entire
throttled hierarchy was dequeued at the point of first throttle during
the pick but with per-task throttling, tasks continue to run as usual
until they exit to userspace and dequeue themselves one-by-one until the
hierarchy is deemed fully throttled and the PELT is frozen.
throttle_cfs_rq() is now simply a propagator of throttle indicators and
nothing more.
Implementation
==============
Unify the throttling for current hierarchy under
account_cfs_rq_runtime() which is responsible for the time accounting.
If the bandwidth runs out, account_cfs_rq_runtime() will request for
sched_cfs_bandwidth_slice() and mark the hierarchy as throttled if it
fails to grab bandwidth.
throttle_cfs_rq() will do a task_throttle_setup_work() if it finds the
current task to be on a throttled hierarchy and the task will naturally
dequeue itself when it exits to the userspace without needing an
explicit resched.
First four patches are cleanups and preparation for the final bit that
switches over to using account_cfs_rq_runtime() for throttling which was
provided by Peter in [1].
Benchmarking
============
Following are the results of running hackbench running 3 levels deep
with the setup from "Testing" section on [2] when compared to
tip:sched/core:
kernel : tip tip + series
Min : 207.33 202.20
Max : 210.20 222.47
Median : 207.83 218.33
AMean : 208.29 215.36
GMean : 208.29 215.25
HMean : 208.29 215.13
AMean Stddev : 1.02 7.37
AMean CoefVar : 0.49 pct 3.42 pct
All numbers are in seconds.
There is a slight boot to boot variation for this benchmark but the
utilization numbers in top is more or less similar between the two.
Additional testing and feedback is always appreciated as usual :-)
Patches on top of queue:sched/core at commit ce348f2b1998 ("sched/fair:
Allocate cfs_tg_state with percpu allocator"). All testing was done on a
dual socket 4th Generation EPYC system (2 x 128C/256T).
CONFIG_CFS_BANDWIDTH=n was only build tested.
Changelog
=========
v1..v2:
o Addressed comments from Ben in Patch1 to keep the call to
distribute_cfs_runtime() in do_sched_cfs_slack_timer and used
scoped_guard().
o Instead of adding back "else { throttled = true; }" in
distribute_cfs_runtime(), invert the condition to
"cfs_rq->runtime_remaining <= 0" and break early after setting
"throttled = true". (Ben)
o Added missing update_rq_clock() in tg_set_cfs_bandwidth(). (Aaron,
Peter, Intel Test Robot)
o Added an update_curr() in distribute_cfs_runtime() which was found to
be beneficial to prevent calling unthrottle_cfs_rq() unnecessarily.
o Collected tags from Ben and Aaron (Thanks a ton!).
o Rebased patches on top of:
git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/core
at commit ce348f2b1998 ("sched/fair: Allocate cfs_tg_state with percpu
allocator").
References
==========
[1] https://lore.kernel.org/lkml/20260512110932.GB1889694@noisy.programming.kicks-ass.net/
[2] https://lore.kernel.org/lkml/20250220093257.9380-1-kprateek.nayak@amd.com/
[3] https://lore.kernel.org/lkml/20260522141623.600235-4-zli94@ncsu.edu/
---
K Prateek Nayak (4):
sched/fair: Convert cfs bandwidth throttling to use guards
sched/fair: Use throttled_csd_list for local unthrottle
sched/fair: Call update_curr() before unthrottling the hierarchy
sched/fair: Move the throttled tasks to a local list in
tg_unthrottle_up()
Peter Zijlstra (1):
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
kernel/sched/core.c | 5 +-
kernel/sched/fair.c | 360 +++++++++++++++++++++++---------------------
2 files changed, 193 insertions(+), 172 deletions(-)
base-commit: ce348f2b1998b4c90053a3be407c32102b132800
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
@ 2026-06-02 5:00 ` K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 2/5] sched/fair: Use throttled_csd_list for local unthrottle K Prateek Nayak
` (3 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:00 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
Routine conversion of rcu_read_lock(), spin_lock*, and rq_lock usage
within the cfs bandwidth controller to use class guards.
Only notable changes are:
- Checking for "cfs_rq->runtime_remaining <= 0" instead of the inverse
to spot a throttle and break early. This also saves the need
for extra indentation in the unthrottle case.
- Reordering of list_del_rcu() against throttled_clock indicator update
in unthrottle_cfs_rq(). Both are done with "cfs_b->lock" held after
the "cfs_rq->throttled" is cleared which make the reordering safe
against concurrent list modifications.
No functional changes intended.
Reviewed-by: Ben Segall <bsegall@google.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
v1..v2:
o Addressed comments to keep the call to distribute_cfs_runtime() in
do_sched_cfs_slack_timer() and used scoped_guard() instead. (Ben)
o Instead of adding back "else { throttled = true; }" in
distribute_cfs_runtime(), invert the condition to
"cfs_rq->runtime_remaining <= 0" and break early after setting
"throttled = true". (Indirectly addresses Ben's comment to keep it
clear as to which condition leads to "throttled" being set to true.)
o Collected tags from Ben and Aaron. (Thanks a ton!)
---
kernel/sched/fair.c | 193 +++++++++++++++++++++-----------------------
1 file changed, 90 insertions(+), 103 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d030fccfa4b4..766cd4395e6c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5006,13 +5006,13 @@ static void __maybe_unused clear_tg_offline_cfs_rqs(struct rq *rq)
*/
rq_clock_start_loop_update(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
clear_tg_load_avg(cfs_rq);
}
- rcu_read_unlock();
rq_clock_stop_loop_update(rq);
}
@@ -6511,13 +6511,10 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
{
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
- int ret;
- raw_spin_lock(&cfs_b->lock);
- ret = __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
- raw_spin_unlock(&cfs_b->lock);
+ guard(raw_spinlock)(&cfs_b->lock);
- return ret;
+ return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
}
static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
@@ -6806,33 +6803,32 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
struct rq *rq = rq_of(cfs_rq);
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
- int dequeue = 1;
- raw_spin_lock(&cfs_b->lock);
- /* This will start the period timer if necessary */
- if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) {
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
/*
- * We have raced with bandwidth becoming available, and if we
- * actually throttled the timer might not unthrottle us for an
- * entire period. We additionally needed to make sure that any
- * subsequent check_cfs_rq_runtime calls agree not to throttle
- * us, as we may commit to do cfs put_prev+pick_next, so we ask
- * for 1ns of runtime rather than just check cfs_b.
+ * Check if We have raced with bandwidth becoming available. If
+ * we actually throttled the timer might not unthrottle us for
+ * an entire period. We additionally needed to make sure that
+ * any subsequent check_cfs_rq_runtime calls agree not to
+ * throttle us, as we may commit to do cfs put_prev+pick_next,
+ * so we ask for 1ns of runtime rather than just check cfs_b.
+ *
+ * This will start the period timer if necessary.
+ */
+ if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1))
+ return false;
+
+ /*
+ * No bandwidth available; Add ourselves on the list to be
+ * unthrottled later.
*/
- dequeue = 0;
- } else {
list_add_tail_rcu(&cfs_rq->throttled_list,
&cfs_b->throttled_cfs_rq);
}
- raw_spin_unlock(&cfs_b->lock);
-
- if (!dequeue)
- return false; /* Throttle no longer required. */
/* freeze hierarchy runnable averages while throttled */
- rcu_read_lock();
- walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq);
- rcu_read_unlock();
+ scoped_guard(rcu)
+ walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq);
/*
* Note: distribution will already see us throttled via the
@@ -6865,13 +6861,15 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
update_rq_clock(rq);
- raw_spin_lock(&cfs_b->lock);
- if (cfs_rq->throttled_clock) {
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
+ list_del_rcu(&cfs_rq->throttled_list);
+
+ if (!cfs_rq->throttled_clock)
+ break;
+
cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock;
cfs_rq->throttled_clock = 0;
}
- list_del_rcu(&cfs_rq->throttled_list);
- raw_spin_unlock(&cfs_b->lock);
/* update hierarchical throttle state */
walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq);
@@ -6900,9 +6898,8 @@ static void __cfsb_csd_unthrottle(void *arg)
{
struct cfs_rq *cursor, *tmp;
struct rq *rq = arg;
- struct rq_flags rf;
- rq_lock(rq, &rf);
+ guard(rq_lock)(rq);
/*
* Iterating over the list can trigger several call to
@@ -6919,7 +6916,7 @@ static void __cfsb_csd_unthrottle(void *arg)
* race with group being freed in the window between removing it
* from the list and advancing to the next entry in the list.
*/
- rcu_read_lock();
+ guard(rcu)();
list_for_each_entry_safe(cursor, tmp, &rq->cfsb_csd_list,
throttled_csd_list) {
@@ -6929,10 +6926,7 @@ static void __cfsb_csd_unthrottle(void *arg)
unthrottle_cfs_rq(cursor);
}
- rcu_read_unlock();
-
rq_clock_stop_loop_update(rq);
- rq_unlock(rq, &rf);
}
static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
@@ -6972,11 +6966,11 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
u64 runtime, remaining = 1;
bool throttled = false;
struct cfs_rq *cfs_rq, *tmp;
- struct rq_flags rf;
struct rq *rq;
LIST_HEAD(local_unthrottle);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
throttled_list) {
rq = rq_of(cfs_rq);
@@ -6986,65 +6980,63 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
break;
}
- rq_lock_irqsave(rq, &rf);
+ guard(rq_lock_irqsave)(rq);
+
if (!cfs_rq_throttled(cfs_rq))
- goto next;
+ continue;
/* Already queued for async unthrottle */
if (!list_empty(&cfs_rq->throttled_csd_list))
- goto next;
+ continue;
/* By the above checks, this should never be true */
WARN_ON_ONCE(cfs_rq->runtime_remaining > 0);
- raw_spin_lock(&cfs_b->lock);
- runtime = -cfs_rq->runtime_remaining + 1;
- if (runtime > cfs_b->runtime)
- runtime = cfs_b->runtime;
- cfs_b->runtime -= runtime;
- remaining = cfs_b->runtime;
- raw_spin_unlock(&cfs_b->lock);
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
+ runtime = -cfs_rq->runtime_remaining + 1;
+ if (runtime > cfs_b->runtime)
+ runtime = cfs_b->runtime;
+ cfs_b->runtime -= runtime;
+ remaining = cfs_b->runtime;
+ }
cfs_rq->runtime_remaining += runtime;
- /* we check whether we're throttled above */
- if (cfs_rq->runtime_remaining > 0) {
- if (cpu_of(rq) != this_cpu) {
- unthrottle_cfs_rq_async(cfs_rq);
- } else {
- /*
- * We currently only expect to be unthrottling
- * a single cfs_rq locally.
- */
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
- list_add_tail(&cfs_rq->throttled_csd_list,
- &local_unthrottle);
- }
- } else {
+ /*
+ * Ran out of bandwidth during distribution!
+ * Indicate throttled entities and break early.
+ */
+ if (cfs_rq->runtime_remaining <= 0) {
throttled = true;
+ break;
}
-next:
- rq_unlock_irqrestore(rq, &rf);
+ /* we check whether we're throttled above */
+ if (cpu_of(rq) != this_cpu) {
+ unthrottle_cfs_rq_async(cfs_rq);
+ continue;
+ }
+
+ /*
+ * We currently only expect to be unthrottling
+ * a single cfs_rq locally.
+ */
+ WARN_ON_ONCE(!list_empty(&local_unthrottle));
+ list_add_tail(&cfs_rq->throttled_csd_list, &local_unthrottle);
}
list_for_each_entry_safe(cfs_rq, tmp, &local_unthrottle,
throttled_csd_list) {
struct rq *rq = rq_of(cfs_rq);
- rq_lock_irqsave(rq, &rf);
+ guard(rq_lock_irqsave)(rq);
list_del_init(&cfs_rq->throttled_csd_list);
-
if (cfs_rq_throttled(cfs_rq))
unthrottle_cfs_rq(cfs_rq);
-
- rq_unlock_irqrestore(rq, &rf);
}
WARN_ON_ONCE(!list_empty(&local_unthrottle));
- rcu_read_unlock();
-
return throttled;
}
@@ -7167,7 +7159,8 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
if (slack_runtime <= 0)
return;
- raw_spin_lock(&cfs_b->lock);
+ guard(raw_spinlock)(&cfs_b->lock);
+
if (cfs_b->quota != RUNTIME_INF) {
cfs_b->runtime += slack_runtime;
@@ -7176,7 +7169,6 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
!list_empty(&cfs_b->throttled_cfs_rq))
start_cfs_slack_bandwidth(cfs_b);
}
- raw_spin_unlock(&cfs_b->lock);
/* even if it's not valid for return we don't want to try again */
cfs_rq->runtime_remaining -= slack_runtime;
@@ -7199,25 +7191,21 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
*/
static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
{
- u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
- unsigned long flags;
-
/* confirm we're still not at a refresh boundary */
- raw_spin_lock_irqsave(&cfs_b->lock, flags);
- cfs_b->slack_started = false;
+ scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
+ u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
- if (runtime_refresh_within(cfs_b, min_bandwidth_expiration)) {
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
- return;
- }
+ cfs_b->slack_started = false;
- if (cfs_b->quota != RUNTIME_INF && cfs_b->runtime > slice)
- runtime = cfs_b->runtime;
+ if (runtime_refresh_within(cfs_b, min_bandwidth_expiration))
+ return;
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+ if (cfs_b->quota != RUNTIME_INF && cfs_b->runtime > slice)
+ runtime = cfs_b->runtime;
- if (!runtime)
- return;
+ if (!runtime)
+ return;
+ }
distribute_cfs_runtime(cfs_b);
}
@@ -7306,18 +7294,18 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
container_of(timer, struct cfs_bandwidth, period_timer);
- unsigned long flags;
int overrun;
int idle = 0;
int count = 0;
- raw_spin_lock_irqsave(&cfs_b->lock, flags);
+ CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
+
for (;;) {
overrun = hrtimer_forward_now(timer, cfs_b->period);
if (!overrun)
break;
- idle = do_sched_cfs_period_timer(cfs_b, overrun, flags);
+ idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
if (++count > 3) {
u64 new, old = ktime_to_ns(cfs_b->period);
@@ -7350,11 +7338,13 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
count = 0;
}
}
- if (idle)
+
+ if (idle) {
cfs_b->period_active = 0;
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+ return HRTIMER_NORESTART;
+ }
- return idle ? HRTIMER_NORESTART : HRTIMER_RESTART;
+ return HRTIMER_RESTART;
}
void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth *parent)
@@ -7421,14 +7411,12 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
*/
for_each_possible_cpu(i) {
struct rq *rq = cpu_rq(i);
- unsigned long flags;
if (list_empty(&rq->cfsb_csd_list))
continue;
- local_irq_save(flags);
- __cfsb_csd_unthrottle(rq);
- local_irq_restore(flags);
+ scoped_guard(irqsave)
+ __cfsb_csd_unthrottle(rq);
}
}
@@ -7446,16 +7434,15 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq)
lockdep_assert_rq_held(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
- raw_spin_lock(&cfs_b->lock);
- cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
- raw_spin_unlock(&cfs_b->lock);
+ scoped_guard(raw_spinlock, &cfs_b->lock)
+ cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
}
- rcu_read_unlock();
}
/* cpu offline callback */
@@ -7476,7 +7463,8 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
*/
rq_clock_start_loop_update(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
@@ -7499,7 +7487,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
cfs_rq->runtime_remaining = 1;
unthrottle_cfs_rq(cfs_rq);
}
- rcu_read_unlock();
rq_clock_stop_loop_update(rq);
}
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 2/5] sched/fair: Use throttled_csd_list for local unthrottle
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards K Prateek Nayak
@ 2026-06-02 5:00 ` K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 3/5] sched/fair: Call update_curr() before unthrottling the hierarchy K Prateek Nayak
` (2 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:00 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
When distribute_cfs_runtime() encounters a local cfs_rq, it adds it to a
local list and unthrottles it at the end, when it is done unthrottling
other cfs_rq(s) on cfs_b->throttled_cfs_rq until the bandwidth runs out.
Instead of using a local list, reuse the local CPU's
rq->throttled_csd_list and the __cfsb_csd_unthrottle() path for
unthrottle.
If this is the first cfs_rq to be queued on the "throttled_csd_list", it
prevents the need for a remote CPUs to interrupt this local CPU if they
themselves are performing async unthrottle.
If this is not the first cfs_rq on the list, there is an async unthrottle
operation pending on this local CPU and the unthrottle can be batched
together.
No functional changes intended.
Reviewed-by: Benjamin Segall <bsegall@google.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
changelog v1..v2:
o Collected tags from Ben, and Aaron (Thanks a ton!)
---
kernel/sched/fair.c | 32 +++++++++++++++-----------------
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 766cd4395e6c..825393d3f039 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6962,12 +6962,11 @@ static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
{
+ bool throttled = false, unthrottle_local = false;
int this_cpu = smp_processor_id();
u64 runtime, remaining = 1;
- bool throttled = false;
- struct cfs_rq *cfs_rq, *tmp;
+ struct cfs_rq *cfs_rq;
struct rq *rq;
- LIST_HEAD(local_unthrottle);
guard(rcu)();
@@ -7018,24 +7017,23 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
}
/*
- * We currently only expect to be unthrottling
- * a single cfs_rq locally.
+ * Allow a parallel async unthrottle to unthrottle
+ * this cfs_rq too via __cfsb_csd_unthrottle().
+ * If we are first, do it ourselves at the end and
+ * save on an IPI from remote CPUs.
*/
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
- list_add_tail(&cfs_rq->throttled_csd_list, &local_unthrottle);
+ unthrottle_local = list_empty(&rq->cfsb_csd_list);
+ list_add_tail(&cfs_rq->throttled_csd_list, &rq->cfsb_csd_list);
}
- list_for_each_entry_safe(cfs_rq, tmp, &local_unthrottle,
- throttled_csd_list) {
- struct rq *rq = rq_of(cfs_rq);
-
- guard(rq_lock_irqsave)(rq);
-
- list_del_init(&cfs_rq->throttled_csd_list);
- if (cfs_rq_throttled(cfs_rq))
- unthrottle_cfs_rq(cfs_rq);
+ if (unthrottle_local) {
+ /*
+ * Protect against an IPI that is also trying to flush
+ * the unthrottled cfs_rq(s) from this CPU's csd_list.
+ */
+ scoped_guard(irqsave)
+ __cfsb_csd_unthrottle(cpu_rq(this_cpu));
}
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
return throttled;
}
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 3/5] sched/fair: Call update_curr() before unthrottling the hierarchy
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 2/5] sched/fair: Use throttled_csd_list for local unthrottle K Prateek Nayak
@ 2026-06-02 5:25 ` K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() K Prateek Nayak
4 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:25 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
Subsequent commits will allow update_curr() to throttle the hierarchy
when the runtime accounting exceeds allocated quota. Call update_curr()
before the unthrottle event, and in tg_unthrottle_up() to catch up on
any remaining runtime and stabilize the "runtime_remaining" and
"throttle_count" for that cfs_rq.
Doing an update_curr() early ensures the cfs_rq is not throttled right
back up again when the unthrottle is in progress.
Since all callers of unthrottle_cfs_rq(), except two, already update the
rq_clock and call rq_clock_start_loop_update(), move the
update_rq_clock() from unthrottle_cfs_rq() to the callers that don't
update the rq_clock.
Reviewed-by: Benjamin Segall <bsegall@google.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
changelog v1..v2:
o Added the missing update_rq_clock() in tg_set_cfs_bandwidth() - well,
Peter added it and I just moved it just before the unthrottle and made
the robots happy with the build. (Aaron, Peter, Intel Test Robot)
o Added an update_curr() before replenish in distribute_cfs_runtime()
which helps performance a little bit. Since this was trivial addition,
I've retained the tags.
o Collected tags from Ben, and Aaron (Thanks a ton!)
---
kernel/sched/core.c | 5 ++++-
kernel/sched/fair.c | 21 +++++++++++++++++++--
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index dd031410ab1a..e745c58671ed 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9859,11 +9859,14 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
struct rq *rq = cfs_rq->rq;
guard(rq_lock_irq)(rq);
+
cfs_rq->runtime_enabled = runtime_enabled;
cfs_rq->runtime_remaining = 1;
- if (cfs_rq->throttled)
+ if (cfs_rq->throttled) {
+ update_rq_clock(rq);
unthrottle_cfs_rq(cfs_rq);
+ }
}
if (runtime_was_enabled && !runtime_enabled)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 825393d3f039..16e0ccaffe6e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6711,6 +6711,15 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
struct task_struct *p, *tmp;
+ /*
+ * If cfs_rq->curr is set, the cfs_rq might not have caught up
+ * since the last clock update. Do it now before we begin
+ * queueing task onto it to save the need for unnecessarily
+ * unthrottle the hierarchy for this cfs_rq to be throttled
+ * right back again.
+ */
+ update_curr(cfs_rq);
+
if (--cfs_rq->throttle_count)
return 0;
@@ -6853,14 +6862,16 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
* We can't unthrottle this cfs_rq without any runtime remaining because
* any enqueue in tg_unthrottle_up() will immediately trigger a throttle,
* which is not supposed to happen on unthrottle path.
+ *
+ * Catch up on the remaining runtime since last clock update before
+ * checking runtime remaining.
*/
+ update_curr(cfs_rq);
if (cfs_rq->runtime_enabled && cfs_rq->runtime_remaining <= 0)
return;
cfs_rq->throttled = 0;
- update_rq_clock(rq);
-
scoped_guard(raw_spinlock, &cfs_b->lock) {
list_del_rcu(&cfs_rq->throttled_list);
@@ -6935,6 +6946,7 @@ static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
bool first;
if (rq == this_rq()) {
+ update_rq_clock(rq);
unthrottle_cfs_rq(cfs_rq);
return;
}
@@ -6988,6 +7000,11 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
if (!list_empty(&cfs_rq->throttled_csd_list))
continue;
+ if (cfs_rq->curr) {
+ update_rq_clock(rq);
+ update_curr(cfs_rq);
+ }
+
/* By the above checks, this should never be true */
WARN_ON_ONCE(cfs_rq->runtime_remaining > 0);
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
` (2 preceding siblings ...)
2026-06-02 5:25 ` [PATCH v2 3/5] sched/fair: Call update_curr() before unthrottling the hierarchy K Prateek Nayak
@ 2026-06-02 5:25 ` K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() K Prateek Nayak
4 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:25 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
An update_curr() during the enqueue of throttled task will start
throttling the hierarchy from subsequent commit. This can lead to
tg_throttle_down() seeing non-empty throttled_limbo_list for the cfs_rq
attaching the task from throttled_limbo_list one by one. For example:
R
|
A
/ \
*B C
|
rq->curr
*B is throttled with tasks on hte limbo list. When the tasks are
unthrottled via tg_unthrottle_up() and entity of group B is placed onto
A, update_curr() is called to catch up the vruntime and it may throttle
group A causing the subsequent tg_throttle_down() to see the pending
task's on B's limbo list.
tg_unthrottle_up()
/* --cfs_rq->throttle_count == 0 */
list_for_each_entry_safe(p, cfs_rq->throttled_limbo_list)
enqueue_task_fair()
enqueue_entity(se /* B->se */)
update_curr(cfs_rq /* A->gcfs_rq */)
account_cfs_rq_runtime(cfs_rq)
throttle_cfs_rq(cfs_rq /* A->gcfs_rq */ )
tg_throttle_down()
/* Reaches B->cfs_rq with throttle_count == 0 */
!!! !list_empty(&cfs_rq->throttled_limbo_list)) !!!
Move the tasks from throttled_limbo_list onto a local list before
starting the unthrottle to prevent the splat described above. If the
hierarchy is throttled again in middle of an unthrottle, put the pending
tasks back onto the limbo list to prevent running them unnecessarily.
Reviewed-by: Benjamin Segall <bsegall@google.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
changelog v1..v2:
o Collected tags from Ben, and Aaron (Thanks a ton!)
---
kernel/sched/fair.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 16e0ccaffe6e..8169bcedcfc8 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6710,6 +6710,7 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
struct rq *rq = data;
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
struct task_struct *p, *tmp;
+ LIST_HEAD(throttled_tasks);
/*
* If cfs_rq->curr is set, the cfs_rq might not have caught up
@@ -6740,13 +6741,31 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
cfs_rq->throttled_clock_self_time += delta;
}
+ /*
+ * Move the tasks to a local list since an update_curr() during
+ * enqueue_task_fair() can throttle a higher cfs_rq, and it can
+ * see the "throttled_limbo_list" being non-empty in
+ * tg_throttle_down() if throttle_count turned 0 above.
+ */
+ list_splice_init(&cfs_rq->throttled_limbo_list, &throttled_tasks);
+
/* Re-enqueue the tasks that have been throttled at this level. */
- list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) {
+ list_for_each_entry_safe(p, tmp, &throttled_tasks, throttle_node) {
+ /*
+ * Back to being throttled! Break out and put the remaining
+ * tasks back onto the limbo_list to prevent running them
+ * unnecessarily.
+ */
+ if (cfs_rq->throttle_count)
+ break;
+
list_del_init(&p->throttle_node);
p->throttled = false;
- enqueue_task_fair(rq_of(cfs_rq), p, ENQUEUE_WAKEUP);
+ enqueue_task_fair(rq, p, ENQUEUE_WAKEUP);
}
+ list_splice(&throttled_tasks, &cfs_rq->throttled_limbo_list);
+
/* Add cfs_rq with load or one or more already running entities to the list */
if (!cfs_rq_is_decayed(cfs_rq))
list_add_leaf_cfs_rq(cfs_rq);
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
` (3 preceding siblings ...)
2026-06-02 5:25 ` [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() K Prateek Nayak
@ 2026-06-02 5:25 ` K Prateek Nayak
2026-06-02 7:10 ` [PATCH v2.1 " K Prateek Nayak
4 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 5:25 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
From: Peter Zijlstra <peterz@infradead.org>
assign_cfs_rq_runtime() during update_curr() sets the resched indicator
and relies on check_cfs_rq_runtime() during pick_next_task() /
put_prev_entity() to throttle the hierarchy once current task is
preempted / blocks.
Per-task throttle, on the other hand, uses throttle_cfs_rq() to simply
propagate the throttle signals, and then relies on task work to
individually throttle the runnable tasks on their way out to the
userspace.
Remove check_cfs_rq_runtime() and unify throttling into
account_cfs_rq_runtime() which only sets the cfs_rq->throttled,
cfs_rq->throttle_count indicators via throttle_cfs_rq() and optionally
adds the task work to the current task (donor) it is on the throttled
hierarchy.
throttle_cfs_rq() requests for sched_cfs_bandwidth_slice() worth of
bandwidth for the current hierarchy that enable it to continue running
uninterrupted when selected. For the rest, it requests a bare minimum of
"1" to ensure some bandwidth is available and pass the
"runtime_remaining > 0" checks once selected.
For SCHED_PROXY_EXEC, a mutex holder cannot exit to userspace without
dropping it first and the mutex_unlock() ensures proxy is stopped before
the mutex handoff which preserves the current semantics for running a
throttled task until it exits to the userspace even if it acts as a
donor.
[ prateek: rebased on tip, comments, commit message. ]
Reviewed-by: Benjamin Segall <bsegall@google.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
changelog v1..v2:
o Added S-o-b from Peter matching queue:sched/core.
o Do account_cfs_rq_runtime() selectively in set_next_task_fair() on
"!first" since pick would have ensured sufficient bandwidth is
available if set_next_task_fair() is called after pick which is more
common after the recent pick_next_task removal. (Peter)
o Collected tags from Ben, and Aaron (Thanks a ton!)
---
kernel/sched/fair.c | 116 +++++++++++++++++++++++---------------------
1 file changed, 60 insertions(+), 56 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8169bcedcfc8..ce5cf494b934 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -525,7 +525,7 @@ static int se_is_idle(struct sched_entity *se)
#endif /* !CONFIG_FAIR_GROUP_SCHED */
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
/**************************************************************
* Scheduling class tree data structure manipulation methods:
@@ -6359,8 +6359,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
return se;
}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-
static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
{
/*
@@ -6370,9 +6368,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
if (prev->on_rq)
update_curr(cfs_rq);
- /* throttle cfs_rqs exceeding runtime */
- check_cfs_rq_runtime(cfs_rq);
-
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
/* Put 'current' back into the tree. */
@@ -6507,41 +6502,32 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
return cfs_rq->runtime_remaining > 0;
}
-/* returns 0 on failure to allocate runtime */
-static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
-
- guard(raw_spinlock)(&cfs_b->lock);
-
- return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
-}
+static bool throttle_cfs_rq(struct cfs_rq *cfs_rq);
-static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
/* dock delta_exec before expiring quota (as it could span periods) */
cfs_rq->runtime_remaining -= delta_exec;
if (likely(cfs_rq->runtime_remaining > 0))
- return;
+ return false;
if (cfs_rq->throttled)
- return;
+ return true;
/*
- * if we're unable to extend our runtime we resched so that the active
- * hierarchy can be throttled
+ * throttle_cfs_rq() will try to extend the runtime first
+ * before throttling the hierarchy.
*/
- if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
- resched_curr(rq_of(cfs_rq));
+ return throttle_cfs_rq(cfs_rq);
}
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled)
- return;
+ return false;
- __account_cfs_rq_runtime(cfs_rq, delta_exec);
+ return __account_cfs_rq_runtime(cfs_rq, delta_exec);
}
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
@@ -6829,10 +6815,24 @@ static int tg_throttle_down(struct task_group *tg, void *data)
static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
- struct rq *rq = rq_of(cfs_rq);
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
+ struct sched_entity *curr = cfs_rq->curr;
+ struct rq *rq = rq_of(cfs_rq);
scoped_guard(raw_spinlock, &cfs_b->lock) {
+ u64 target_runtime = 1;
+
+ /*
+ * If cfs_rq->curr is still runnable, we are here from an
+ * update_curr(). Request sysctl_sched_cfs_bandwidth_slice
+ * worth of bandwidth to continue running.
+ *
+ * If the curr is not runnable, just request enough bandwidth
+ * to be runnable next time the pick selects this cfs_rq.
+ */
+ if (curr && curr->on_rq)
+ target_runtime = sched_cfs_bandwidth_slice();
+
/*
* Check if We have raced with bandwidth becoming available. If
* we actually throttled the timer might not unthrottle us for
@@ -6843,7 +6843,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*
* This will start the period timer if necessary.
*/
- if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1))
+ if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, target_runtime))
return false;
/*
@@ -6864,6 +6864,17 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*/
cfs_rq->throttled = 1;
WARN_ON_ONCE(cfs_rq->throttled_clock);
+
+ /*
+ * If current hierarchy was throttled, add throttle work to the
+ * current donor. In case of proxy-execution, the execution
+ * context cannot exit to the userspace while holding a mutex
+ * and the rule of throttle deferral to only throttle the
+ * throttled context at exit to userspace is still preserved.
+ */
+ if (curr && curr->on_rq)
+ task_throttle_setup_work(rq->donor);
+
return true;
}
@@ -7254,7 +7265,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
if (!cfs_bandwidth_used())
return;
- /* an active group must be handled by the update_curr()->put() path */
+ /* an active group must be handled by the update_curr() path */
if (!cfs_rq->runtime_enabled || cfs_rq->curr)
return;
@@ -7264,8 +7275,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
/* update runtime allocation */
account_cfs_rq_runtime(cfs_rq, 0);
- if (cfs_rq->runtime_remaining <= 0)
- throttle_cfs_rq(cfs_rq);
}
static void sync_throttle(struct task_group *tg, int cpu)
@@ -7295,25 +7304,6 @@ static void sync_throttle(struct task_group *tg, int cpu)
cfs_rq->pelt_clock_throttled = 1;
}
-/* conditionally throttle active cfs_rq's from put_prev_entity() */
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- if (!cfs_bandwidth_used())
- return false;
-
- if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
- return false;
-
- /*
- * it's possible for a throttled entity to be forced into a running
- * state (e.g. set_curr_task), in this case we're finished.
- */
- if (cfs_rq_throttled(cfs_rq))
- return true;
-
- return throttle_cfs_rq(cfs_rq);
-}
-
static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
@@ -7567,8 +7557,7 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
#else /* !CONFIG_CFS_BANDWIDTH: */
-static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; }
+static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; }
static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {}
static inline void sync_throttle(struct task_group *tg, int cpu) {}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9903,8 +9892,15 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
/* Might not have done put_prev_entity() */
if (cfs_rq->curr && cfs_rq->curr->on_rq)
update_curr(cfs_rq);
-
- throttled |= check_cfs_rq_runtime(cfs_rq);
+ /*
+ * For the current hierarchy, update_curr() above would
+ * have set the throttle indicators if the cfs_rq has
+ * run out of bandwidth. For others, enqueue / last
+ * update_curr() for the cfs_rq would have ensured the
+ * throttle indicators are set if bandwidth was not
+ * available.
+ */
+ throttled |= cfs_rq_throttled(cfs_rq);
se = pick_next_entity(rq, cfs_rq, true);
if (!se)
@@ -14823,8 +14819,8 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
*/
static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
{
- struct cfs_rq *cfs_rq;
struct sched_entity *se = &curr->se;
+ struct cfs_rq *cfs_rq;
for_each_sched_entity(se) {
cfs_rq = cfs_rq_of(se);
@@ -15006,6 +15002,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
+ bool throttled = false;
for_each_sched_entity(se) {
struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -15015,10 +15012,17 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
break;
set_next_entity(cfs_rq, se, first);
- /* ensure bandwidth has been allocated on our new cfs_rq */
- account_cfs_rq_runtime(cfs_rq, 0);
+ /*
+ * Ensure bandwidth has been allocated on our new cfs_rq
+ * if we've reached here for reasons other than pick.
+ */
+ if (!first)
+ throttled |= account_cfs_rq_runtime(cfs_rq, 0);
}
+ if (throttled)
+ task_throttle_setup_work(p);
+
se = &p->se;
if (task_on_rq_queued(p)) {
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2.1 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
2026-06-02 5:25 ` [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() K Prateek Nayak
@ 2026-06-02 7:10 ` K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
0 siblings, 1 reply; 12+ messages in thread
From: K Prateek Nayak @ 2026-06-02 7:10 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Aaron Lu, Josh Don, K Prateek Nayak,
linux-kernel
From: Peter Zijlstra <peterz@infradead.org>
assign_cfs_rq_runtime() during update_curr() sets the resched indicator
and relies on check_cfs_rq_runtime() during pick_next_task() /
put_prev_entity() to throttle the hierarchy once current task is
preempted / blocks.
Per-task throttle, on the other hand, uses throttle_cfs_rq() to simply
propagate the throttle signals, and then relies on task work to
individually throttle the runnable tasks on their way out to the
userspace.
Remove check_cfs_rq_runtime() and unify throttling into
account_cfs_rq_runtime() which only sets the cfs_rq->throttled,
cfs_rq->throttle_count indicators via throttle_cfs_rq() and optionally
adds the task work to the current task (donor) it is on the throttled
hierarchy.
throttle_cfs_rq() requests for sched_cfs_bandwidth_slice() worth of
bandwidth for the current hierarchy that enable it to continue running
uninterrupted when selected. For the rest, it requests a bare minimum of
"1" to ensure some bandwidth is available and pass the
"runtime_remaining > 0" checks once selected.
For SCHED_PROXY_EXEC, a mutex holder cannot exit to userspace without
dropping it first and the mutex_unlock() ensures proxy is stopped before
the mutex handoff which preserves the current semantics for running a
throttled task until it exits to the userspace even if it acts as a
donor.
[ prateek: rebased on tip, comments, commit message. ]
Reviewed-By: Benjamin Segall <bsegall@google.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
---
changelog v2..v2.1:
o Folded in a the optimization to add task work only in set_next_task()
as pointed out by Peter in
https://lore.kernel.org/lkml/20260601134842.GP343181@noisy.programming.kicks-ass.net/
(Took me a while to wrap my head around it; Diff on top of v2 can be
found on the same thread in
https://lore.kernel.org/lkml/20cc9820-35b2-4d97-bbc2-84af56802dfd@amd.com/)
---
kernel/sched/fair.c | 101 ++++++++++++++++++++------------------------
1 file changed, 46 insertions(+), 55 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8169bcedcfc8..fa8c0b1a1cf1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -525,7 +525,7 @@ static int se_is_idle(struct sched_entity *se)
#endif /* !CONFIG_FAIR_GROUP_SCHED */
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
/**************************************************************
* Scheduling class tree data structure manipulation methods:
@@ -6359,8 +6359,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
return se;
}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-
static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
{
/*
@@ -6370,9 +6368,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
if (prev->on_rq)
update_curr(cfs_rq);
- /* throttle cfs_rqs exceeding runtime */
- check_cfs_rq_runtime(cfs_rq);
-
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
/* Put 'current' back into the tree. */
@@ -6507,41 +6502,32 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
return cfs_rq->runtime_remaining > 0;
}
-/* returns 0 on failure to allocate runtime */
-static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
-
- guard(raw_spinlock)(&cfs_b->lock);
+static bool throttle_cfs_rq(struct cfs_rq *cfs_rq);
- return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
-}
-
-static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
/* dock delta_exec before expiring quota (as it could span periods) */
cfs_rq->runtime_remaining -= delta_exec;
if (likely(cfs_rq->runtime_remaining > 0))
- return;
+ return false;
if (cfs_rq->throttled)
- return;
+ return true;
/*
- * if we're unable to extend our runtime we resched so that the active
- * hierarchy can be throttled
+ * throttle_cfs_rq() will try to extend the runtime first
+ * before throttling the hierarchy.
*/
- if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
- resched_curr(rq_of(cfs_rq));
+ return throttle_cfs_rq(cfs_rq);
}
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled)
- return;
+ return false;
- __account_cfs_rq_runtime(cfs_rq, delta_exec);
+ return __account_cfs_rq_runtime(cfs_rq, delta_exec);
}
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
@@ -6829,10 +6815,24 @@ static int tg_throttle_down(struct task_group *tg, void *data)
static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
- struct rq *rq = rq_of(cfs_rq);
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
+ struct sched_entity *curr = cfs_rq->curr;
+ struct rq *rq = rq_of(cfs_rq);
scoped_guard(raw_spinlock, &cfs_b->lock) {
+ u64 target_runtime = 1;
+
+ /*
+ * If cfs_rq->curr is still runnable, we are here from an
+ * update_curr(). Request sysctl_sched_cfs_bandwidth_slice
+ * worth of bandwidth to continue running.
+ *
+ * If the curr is not runnable, just request enough bandwidth
+ * to be runnable next time the pick selects this cfs_rq.
+ */
+ if (curr && curr->on_rq)
+ target_runtime = sched_cfs_bandwidth_slice();
+
/*
* Check if We have raced with bandwidth becoming available. If
* we actually throttled the timer might not unthrottle us for
@@ -6843,7 +6843,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*
* This will start the period timer if necessary.
*/
- if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1))
+ if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, target_runtime))
return false;
/*
@@ -6864,6 +6864,17 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*/
cfs_rq->throttled = 1;
WARN_ON_ONCE(cfs_rq->throttled_clock);
+
+ /*
+ * If current hierarchy was throttled, add throttle work to the
+ * current donor. In case of proxy-execution, the execution
+ * context cannot exit to the userspace while holding a mutex
+ * and the rule of throttle deferral to only throttle the
+ * throttled context at exit to userspace is still preserved.
+ */
+ if (curr && curr->on_rq)
+ task_throttle_setup_work(rq->donor);
+
return true;
}
@@ -7254,7 +7265,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
if (!cfs_bandwidth_used())
return;
- /* an active group must be handled by the update_curr()->put() path */
+ /* an active group must be handled by the update_curr() path */
if (!cfs_rq->runtime_enabled || cfs_rq->curr)
return;
@@ -7264,8 +7275,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
/* update runtime allocation */
account_cfs_rq_runtime(cfs_rq, 0);
- if (cfs_rq->runtime_remaining <= 0)
- throttle_cfs_rq(cfs_rq);
}
static void sync_throttle(struct task_group *tg, int cpu)
@@ -7295,25 +7304,6 @@ static void sync_throttle(struct task_group *tg, int cpu)
cfs_rq->pelt_clock_throttled = 1;
}
-/* conditionally throttle active cfs_rq's from put_prev_entity() */
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- if (!cfs_bandwidth_used())
- return false;
-
- if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
- return false;
-
- /*
- * it's possible for a throttled entity to be forced into a running
- * state (e.g. set_curr_task), in this case we're finished.
- */
- if (cfs_rq_throttled(cfs_rq))
- return true;
-
- return throttle_cfs_rq(cfs_rq);
-}
-
static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
@@ -7567,8 +7557,7 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
#else /* !CONFIG_CFS_BANDWIDTH: */
-static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; }
+static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; }
static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {}
static inline void sync_throttle(struct task_group *tg, int cpu) {}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9904,8 +9893,6 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
if (cfs_rq->curr && cfs_rq->curr->on_rq)
update_curr(cfs_rq);
- throttled |= check_cfs_rq_runtime(cfs_rq);
-
se = pick_next_entity(rq, cfs_rq, true);
if (!se)
goto again;
@@ -14823,8 +14810,8 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
*/
static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
{
- struct cfs_rq *cfs_rq;
struct sched_entity *se = &curr->se;
+ struct cfs_rq *cfs_rq;
for_each_sched_entity(se) {
cfs_rq = cfs_rq_of(se);
@@ -15006,6 +14993,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
+ bool throttled = false;
for_each_sched_entity(se) {
struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -15016,9 +15004,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
set_next_entity(cfs_rq, se, first);
/* ensure bandwidth has been allocated on our new cfs_rq */
- account_cfs_rq_runtime(cfs_rq, 0);
+ throttled |= account_cfs_rq_runtime(cfs_rq, 0);
}
+ if (throttled)
+ task_throttle_setup_work(p);
+
se = &p->se;
if (task_on_rq_queued(p)) {
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [tip: sched/core] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
2026-06-02 7:10 ` [PATCH v2.1 " K Prateek Nayak
@ 2026-06-04 18:45 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 12+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2026-06-04 18:45 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel), K Prateek Nayak, Aaron Lu, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f666241e6bd5d9a494beca982e1953208dce531c
Gitweb: https://git.kernel.org/tip/f666241e6bd5d9a494beca982e1953208dce531c
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Tue, 02 Jun 2026 07:10:05
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 02 Jun 2026 12:26:13 +02:00
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
assign_cfs_rq_runtime() during update_curr() sets the resched indicator
and relies on check_cfs_rq_runtime() during pick_next_task() /
put_prev_entity() to throttle the hierarchy once current task is
preempted / blocks.
Per-task throttle, on the other hand, uses throttle_cfs_rq() to simply
propagate the throttle signals, and then relies on task work to
individually throttle the runnable tasks on their way out to the
userspace.
Remove check_cfs_rq_runtime() and unify throttling into
account_cfs_rq_runtime() which only sets the cfs_rq->throttled,
cfs_rq->throttle_count indicators via throttle_cfs_rq() and optionally
adds the task work to the current task (donor) it is on the throttled
hierarchy.
throttle_cfs_rq() requests for sched_cfs_bandwidth_slice() worth of
bandwidth for the current hierarchy that enable it to continue running
uninterrupted when selected. For the rest, it requests a bare minimum of
"1" to ensure some bandwidth is available and pass the
"runtime_remaining > 0" checks once selected.
For SCHED_PROXY_EXEC, a mutex holder cannot exit to userspace without
dropping it first and the mutex_unlock() ensures proxy is stopped before
the mutex handoff which preserves the current semantics for running a
throttled task until it exits to the userspace even if it acts as a
donor.
[ prateek: rebased on tip, comments, commit message. ]
Reviewed-By: Benjamin Segall <bsegall@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260602071005.11942-1-kprateek.nayak@amd.com
---
kernel/sched/fair.c | 101 +++++++++++++++++++------------------------
1 file changed, 46 insertions(+), 55 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3f3f09a..f4ed841 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -525,7 +525,7 @@ static int se_is_idle(struct sched_entity *se)
#endif /* !CONFIG_FAIR_GROUP_SCHED */
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
/**************************************************************
* Scheduling class tree data structure manipulation methods:
@@ -6388,8 +6388,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
return se;
}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-
static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
{
/*
@@ -6399,9 +6397,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
if (prev->on_rq)
update_curr(cfs_rq);
- /* throttle cfs_rqs exceeding runtime */
- check_cfs_rq_runtime(cfs_rq);
-
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
/* Put 'current' back into the tree. */
@@ -6536,41 +6531,32 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
return cfs_rq->runtime_remaining > 0;
}
-/* returns 0 on failure to allocate runtime */
-static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
-
- guard(raw_spinlock)(&cfs_b->lock);
+static bool throttle_cfs_rq(struct cfs_rq *cfs_rq);
- return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
-}
-
-static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
/* dock delta_exec before expiring quota (as it could span periods) */
cfs_rq->runtime_remaining -= delta_exec;
if (likely(cfs_rq->runtime_remaining > 0))
- return;
+ return false;
if (cfs_rq->throttled)
- return;
+ return true;
/*
- * if we're unable to extend our runtime we resched so that the active
- * hierarchy can be throttled
+ * throttle_cfs_rq() will try to extend the runtime first
+ * before throttling the hierarchy.
*/
- if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
- resched_curr(rq_of(cfs_rq));
+ return throttle_cfs_rq(cfs_rq);
}
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled)
- return;
+ return false;
- __account_cfs_rq_runtime(cfs_rq, delta_exec);
+ return __account_cfs_rq_runtime(cfs_rq, delta_exec);
}
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
@@ -6858,10 +6844,24 @@ static int tg_throttle_down(struct task_group *tg, void *data)
static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
- struct rq *rq = rq_of(cfs_rq);
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
+ struct sched_entity *curr = cfs_rq->curr;
+ struct rq *rq = rq_of(cfs_rq);
scoped_guard(raw_spinlock, &cfs_b->lock) {
+ u64 target_runtime = 1;
+
+ /*
+ * If cfs_rq->curr is still runnable, we are here from an
+ * update_curr(). Request sysctl_sched_cfs_bandwidth_slice
+ * worth of bandwidth to continue running.
+ *
+ * If the curr is not runnable, just request enough bandwidth
+ * to be runnable next time the pick selects this cfs_rq.
+ */
+ if (curr && curr->on_rq)
+ target_runtime = sched_cfs_bandwidth_slice();
+
/*
* Check if We have raced with bandwidth becoming available. If
* we actually throttled the timer might not unthrottle us for
@@ -6872,7 +6872,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*
* This will start the period timer if necessary.
*/
- if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1))
+ if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, target_runtime))
return false;
/*
@@ -6893,6 +6893,17 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
*/
cfs_rq->throttled = 1;
WARN_ON_ONCE(cfs_rq->throttled_clock);
+
+ /*
+ * If current hierarchy was throttled, add throttle work to the
+ * current donor. In case of proxy-execution, the execution
+ * context cannot exit to the userspace while holding a mutex
+ * and the rule of throttle deferral to only throttle the
+ * throttled context at exit to userspace is still preserved.
+ */
+ if (curr && curr->on_rq)
+ task_throttle_setup_work(rq->donor);
+
return true;
}
@@ -7283,7 +7294,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
if (!cfs_bandwidth_used())
return;
- /* an active group must be handled by the update_curr()->put() path */
+ /* an active group must be handled by the update_curr() path */
if (!cfs_rq->runtime_enabled || cfs_rq->curr)
return;
@@ -7293,8 +7304,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
/* update runtime allocation */
account_cfs_rq_runtime(cfs_rq, 0);
- if (cfs_rq->runtime_remaining <= 0)
- throttle_cfs_rq(cfs_rq);
}
static void sync_throttle(struct task_group *tg, int cpu)
@@ -7324,25 +7333,6 @@ static void sync_throttle(struct task_group *tg, int cpu)
cfs_rq->pelt_clock_throttled = 1;
}
-/* conditionally throttle active cfs_rq's from put_prev_entity() */
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- if (!cfs_bandwidth_used())
- return false;
-
- if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
- return false;
-
- /*
- * it's possible for a throttled entity to be forced into a running
- * state (e.g. set_curr_task), in this case we're finished.
- */
- if (cfs_rq_throttled(cfs_rq))
- return true;
-
- return throttle_cfs_rq(cfs_rq);
-}
-
static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
@@ -7596,8 +7586,7 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
#else /* !CONFIG_CFS_BANDWIDTH: */
-static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; }
+static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; }
static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {}
static inline void sync_throttle(struct task_group *tg, int cpu) {}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9934,8 +9923,6 @@ again:
if (cfs_rq->curr && cfs_rq->curr->on_rq)
update_curr(cfs_rq);
- throttled |= check_cfs_rq_runtime(cfs_rq);
-
se = pick_next_entity(rq, cfs_rq, true);
if (!se)
goto again;
@@ -14853,8 +14840,8 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
*/
static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
{
- struct cfs_rq *cfs_rq;
struct sched_entity *se = &curr->se;
+ struct cfs_rq *cfs_rq;
for_each_sched_entity(se) {
cfs_rq = cfs_rq_of(se);
@@ -15036,6 +15023,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
+ bool throttled = false;
for_each_sched_entity(se) {
struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -15046,9 +15034,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
set_next_entity(cfs_rq, se, first);
/* ensure bandwidth has been allocated on our new cfs_rq */
- account_cfs_rq_runtime(cfs_rq, 0);
+ throttled |= account_cfs_rq_runtime(cfs_rq, 0);
}
+ if (throttled)
+ task_throttle_setup_work(p);
+
se = &p->se;
if (task_on_rq_queued(p)) {
^ permalink raw reply [flat|nested] 12+ messages in thread
* [tip: sched/core] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
2026-06-02 5:25 ` [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() K Prateek Nayak
@ 2026-06-04 18:45 ` tip-bot2 for K Prateek Nayak
0 siblings, 0 replies; 12+ messages in thread
From: tip-bot2 for K Prateek Nayak @ 2026-06-04 18:45 UTC (permalink / raw)
To: linux-tip-commits
Cc: K Prateek Nayak, Peter Zijlstra (Intel),
Benjamin Segall, Aaron Lu, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 102a28344a60e637934ffca62d50ff8319b11165
Gitweb: https://git.kernel.org/tip/102a28344a60e637934ffca62d50ff8319b11165
Author: K Prateek Nayak <kprateek.nayak@amd.com>
AuthorDate: Tue, 02 Jun 2026 05:25:30
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 02 Jun 2026 12:26:12 +02:00
sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
An update_curr() during the enqueue of throttled task will start
throttling the hierarchy from subsequent commit. This can lead to
tg_throttle_down() seeing non-empty throttled_limbo_list for the cfs_rq
attaching the task from throttled_limbo_list one by one. For example:
R
|
A
/ \
*B C
|
rq->curr
*B is throttled with tasks on hte limbo list. When the tasks are
unthrottled via tg_unthrottle_up() and entity of group B is placed onto
A, update_curr() is called to catch up the vruntime and it may throttle
group A causing the subsequent tg_throttle_down() to see the pending
task's on B's limbo list.
tg_unthrottle_up()
/* --cfs_rq->throttle_count == 0 */
list_for_each_entry_safe(p, cfs_rq->throttled_limbo_list)
enqueue_task_fair()
enqueue_entity(se /* B->se */)
update_curr(cfs_rq /* A->gcfs_rq */)
account_cfs_rq_runtime(cfs_rq)
throttle_cfs_rq(cfs_rq /* A->gcfs_rq */ )
tg_throttle_down()
/* Reaches B->cfs_rq with throttle_count == 0 */
!!! !list_empty(&cfs_rq->throttled_limbo_list)) !!!
Move the tasks from throttled_limbo_list onto a local list before
starting the unthrottle to prevent the splat described above. If the
hierarchy is throttled again in middle of an unthrottle, put the pending
tasks back onto the limbo list to prevent running them unnecessarily.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Benjamin Segall <bsegall@google.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260602052531.11450-2-kprateek.nayak@amd.com
---
kernel/sched/fair.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f91d85c..3f3f09a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6739,6 +6739,7 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
struct rq *rq = data;
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
struct task_struct *p, *tmp;
+ LIST_HEAD(throttled_tasks);
/*
* If cfs_rq->curr is set, the cfs_rq might not have caught up
@@ -6769,13 +6770,31 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
cfs_rq->throttled_clock_self_time += delta;
}
+ /*
+ * Move the tasks to a local list since an update_curr() during
+ * enqueue_task_fair() can throttle a higher cfs_rq, and it can
+ * see the "throttled_limbo_list" being non-empty in
+ * tg_throttle_down() if throttle_count turned 0 above.
+ */
+ list_splice_init(&cfs_rq->throttled_limbo_list, &throttled_tasks);
+
/* Re-enqueue the tasks that have been throttled at this level. */
- list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) {
+ list_for_each_entry_safe(p, tmp, &throttled_tasks, throttle_node) {
+ /*
+ * Back to being throttled! Break out and put the remaining
+ * tasks back onto the limbo_list to prevent running them
+ * unnecessarily.
+ */
+ if (cfs_rq->throttle_count)
+ break;
+
list_del_init(&p->throttle_node);
p->throttled = false;
- enqueue_task_fair(rq_of(cfs_rq), p, ENQUEUE_WAKEUP);
+ enqueue_task_fair(rq, p, ENQUEUE_WAKEUP);
}
+ list_splice(&throttled_tasks, &cfs_rq->throttled_limbo_list);
+
/* Add cfs_rq with load or one or more already running entities to the list */
if (!cfs_rq_is_decayed(cfs_rq))
list_add_leaf_cfs_rq(cfs_rq);
^ permalink raw reply [flat|nested] 12+ messages in thread
* [tip: sched/core] sched/fair: Call update_curr() before unthrottling the hierarchy
2026-06-02 5:25 ` [PATCH v2 3/5] sched/fair: Call update_curr() before unthrottling the hierarchy K Prateek Nayak
@ 2026-06-04 18:45 ` tip-bot2 for K Prateek Nayak
0 siblings, 0 replies; 12+ messages in thread
From: tip-bot2 for K Prateek Nayak @ 2026-06-04 18:45 UTC (permalink / raw)
To: linux-tip-commits
Cc: K Prateek Nayak, Peter Zijlstra (Intel),
Benjamin Segall, Aaron Lu, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 28ad5427682bccf06074366f347a6083d6730c1e
Gitweb: https://git.kernel.org/tip/28ad5427682bccf06074366f347a6083d6730c1e
Author: K Prateek Nayak <kprateek.nayak@amd.com>
AuthorDate: Tue, 02 Jun 2026 05:25:29
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 02 Jun 2026 12:26:12 +02:00
sched/fair: Call update_curr() before unthrottling the hierarchy
Subsequent commits will allow update_curr() to throttle the hierarchy
when the runtime accounting exceeds allocated quota. Call update_curr()
before the unthrottle event, and in tg_unthrottle_up() to catch up on
any remaining runtime and stabilize the "runtime_remaining" and
"throttle_count" for that cfs_rq.
Doing an update_curr() early ensures the cfs_rq is not throttled right
back up again when the unthrottle is in progress.
Since all callers of unthrottle_cfs_rq(), except two, already update the
rq_clock and call rq_clock_start_loop_update(), move the
update_rq_clock() from unthrottle_cfs_rq() to the callers that don't
update the rq_clock.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Benjamin Segall <bsegall@google.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260602052531.11450-1-kprateek.nayak@amd.com
---
kernel/sched/core.c | 5 ++++-
kernel/sched/fair.c | 21 +++++++++++++++++++--
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index dd03141..e745c58 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9859,11 +9859,14 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
struct rq *rq = cfs_rq->rq;
guard(rq_lock_irq)(rq);
+
cfs_rq->runtime_enabled = runtime_enabled;
cfs_rq->runtime_remaining = 1;
- if (cfs_rq->throttled)
+ if (cfs_rq->throttled) {
+ update_rq_clock(rq);
unthrottle_cfs_rq(cfs_rq);
+ }
}
if (runtime_was_enabled && !runtime_enabled)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 26a8bbb..f91d85c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6740,6 +6740,15 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
struct task_struct *p, *tmp;
+ /*
+ * If cfs_rq->curr is set, the cfs_rq might not have caught up
+ * since the last clock update. Do it now before we begin
+ * queueing task onto it to save the need for unnecessarily
+ * unthrottle the hierarchy for this cfs_rq to be throttled
+ * right back again.
+ */
+ update_curr(cfs_rq);
+
if (--cfs_rq->throttle_count)
return 0;
@@ -6882,14 +6891,16 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
* We can't unthrottle this cfs_rq without any runtime remaining because
* any enqueue in tg_unthrottle_up() will immediately trigger a throttle,
* which is not supposed to happen on unthrottle path.
+ *
+ * Catch up on the remaining runtime since last clock update before
+ * checking runtime remaining.
*/
+ update_curr(cfs_rq);
if (cfs_rq->runtime_enabled && cfs_rq->runtime_remaining <= 0)
return;
cfs_rq->throttled = 0;
- update_rq_clock(rq);
-
scoped_guard(raw_spinlock, &cfs_b->lock) {
list_del_rcu(&cfs_rq->throttled_list);
@@ -6964,6 +6975,7 @@ static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
bool first;
if (rq == this_rq()) {
+ update_rq_clock(rq);
unthrottle_cfs_rq(cfs_rq);
return;
}
@@ -7017,6 +7029,11 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
if (!list_empty(&cfs_rq->throttled_csd_list))
continue;
+ if (cfs_rq->curr) {
+ update_rq_clock(rq);
+ update_curr(cfs_rq);
+ }
+
/* By the above checks, this should never be true */
WARN_ON_ONCE(cfs_rq->runtime_remaining > 0);
^ permalink raw reply [flat|nested] 12+ messages in thread
* [tip: sched/core] sched/fair: Use throttled_csd_list for local unthrottle
2026-06-02 5:00 ` [PATCH v2 2/5] sched/fair: Use throttled_csd_list for local unthrottle K Prateek Nayak
@ 2026-06-04 18:45 ` tip-bot2 for K Prateek Nayak
0 siblings, 0 replies; 12+ messages in thread
From: tip-bot2 for K Prateek Nayak @ 2026-06-04 18:45 UTC (permalink / raw)
To: linux-tip-commits
Cc: K Prateek Nayak, Peter Zijlstra (Intel),
Benjamin Segall, Aaron Lu, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 253edcf5436c916f2fbf7b880443c7f1ed76101d
Gitweb: https://git.kernel.org/tip/253edcf5436c916f2fbf7b880443c7f1ed76101d
Author: K Prateek Nayak <kprateek.nayak@amd.com>
AuthorDate: Tue, 02 Jun 2026 05:00:02
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 02 Jun 2026 12:26:12 +02:00
sched/fair: Use throttled_csd_list for local unthrottle
When distribute_cfs_runtime() encounters a local cfs_rq, it adds it to a
local list and unthrottles it at the end, when it is done unthrottling
other cfs_rq(s) on cfs_b->throttled_cfs_rq until the bandwidth runs out.
Instead of using a local list, reuse the local CPU's
rq->throttled_csd_list and the __cfsb_csd_unthrottle() path for
unthrottle.
If this is the first cfs_rq to be queued on the "throttled_csd_list", it
prevents the need for a remote CPUs to interrupt this local CPU if they
themselves are performing async unthrottle.
If this is not the first cfs_rq on the list, there is an async unthrottle
operation pending on this local CPU and the unthrottle can be batched
together.
No functional changes intended.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Benjamin Segall <bsegall@google.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260602050005.11160-3-kprateek.nayak@amd.com
---
kernel/sched/fair.c | 32 +++++++++++++++-----------------
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 261e5ce..26a8bbb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6991,12 +6991,11 @@ static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
{
+ bool throttled = false, unthrottle_local = false;
int this_cpu = smp_processor_id();
u64 runtime, remaining = 1;
- bool throttled = false;
- struct cfs_rq *cfs_rq, *tmp;
+ struct cfs_rq *cfs_rq;
struct rq *rq;
- LIST_HEAD(local_unthrottle);
guard(rcu)();
@@ -7047,24 +7046,23 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
}
/*
- * We currently only expect to be unthrottling
- * a single cfs_rq locally.
+ * Allow a parallel async unthrottle to unthrottle
+ * this cfs_rq too via __cfsb_csd_unthrottle().
+ * If we are first, do it ourselves at the end and
+ * save on an IPI from remote CPUs.
*/
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
- list_add_tail(&cfs_rq->throttled_csd_list, &local_unthrottle);
+ unthrottle_local = list_empty(&rq->cfsb_csd_list);
+ list_add_tail(&cfs_rq->throttled_csd_list, &rq->cfsb_csd_list);
}
- list_for_each_entry_safe(cfs_rq, tmp, &local_unthrottle,
- throttled_csd_list) {
- struct rq *rq = rq_of(cfs_rq);
-
- guard(rq_lock_irqsave)(rq);
-
- list_del_init(&cfs_rq->throttled_csd_list);
- if (cfs_rq_throttled(cfs_rq))
- unthrottle_cfs_rq(cfs_rq);
+ if (unthrottle_local) {
+ /*
+ * Protect against an IPI that is also trying to flush
+ * the unthrottled cfs_rq(s) from this CPU's csd_list.
+ */
+ scoped_guard(irqsave)
+ __cfsb_csd_unthrottle(cpu_rq(this_cpu));
}
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
return throttled;
}
^ permalink raw reply [flat|nested] 12+ messages in thread
* [tip: sched/core] sched/fair: Convert cfs bandwidth throttling to use guards
2026-06-02 5:00 ` [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards K Prateek Nayak
@ 2026-06-04 18:45 ` tip-bot2 for K Prateek Nayak
0 siblings, 0 replies; 12+ messages in thread
From: tip-bot2 for K Prateek Nayak @ 2026-06-04 18:45 UTC (permalink / raw)
To: linux-tip-commits
Cc: K Prateek Nayak, Peter Zijlstra (Intel),
Ben Segall, Aaron Lu, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 1abbecd1d2d2fdd96e52f541f07ee2b163631bee
Gitweb: https://git.kernel.org/tip/1abbecd1d2d2fdd96e52f541f07ee2b163631bee
Author: K Prateek Nayak <kprateek.nayak@amd.com>
AuthorDate: Tue, 02 Jun 2026 05:00:01
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 02 Jun 2026 12:26:11 +02:00
sched/fair: Convert cfs bandwidth throttling to use guards
Routine conversion of rcu_read_lock(), spin_lock*, and rq_lock usage
within the cfs bandwidth controller to use class guards.
Only notable changes are:
- Checking for "cfs_rq->runtime_remaining <= 0" instead of the inverse
to spot a throttle and break early. This also saves the need
for extra indentation in the unthrottle case.
- Reordering of list_del_rcu() against throttled_clock indicator update
in unthrottle_cfs_rq(). Both are done with "cfs_b->lock" held after
the "cfs_rq->throttled" is cleared which make the reordering safe
against concurrent list modifications.
No functional changes intended.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ben Segall <bsegall@google.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260602050005.11160-2-kprateek.nayak@amd.com
---
kernel/sched/fair.c | 193 ++++++++++++++++++++-----------------------
1 file changed, 90 insertions(+), 103 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1d4ed88..261e5ce 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5035,13 +5035,13 @@ static void __maybe_unused clear_tg_offline_cfs_rqs(struct rq *rq)
*/
rq_clock_start_loop_update(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
clear_tg_load_avg(cfs_rq);
}
- rcu_read_unlock();
rq_clock_stop_loop_update(rq);
}
@@ -6540,13 +6540,10 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
{
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
- int ret;
- raw_spin_lock(&cfs_b->lock);
- ret = __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
- raw_spin_unlock(&cfs_b->lock);
+ guard(raw_spinlock)(&cfs_b->lock);
- return ret;
+ return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
}
static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
@@ -6835,33 +6832,32 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
struct rq *rq = rq_of(cfs_rq);
struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
- int dequeue = 1;
- raw_spin_lock(&cfs_b->lock);
- /* This will start the period timer if necessary */
- if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) {
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
/*
- * We have raced with bandwidth becoming available, and if we
- * actually throttled the timer might not unthrottle us for an
- * entire period. We additionally needed to make sure that any
- * subsequent check_cfs_rq_runtime calls agree not to throttle
- * us, as we may commit to do cfs put_prev+pick_next, so we ask
- * for 1ns of runtime rather than just check cfs_b.
+ * Check if We have raced with bandwidth becoming available. If
+ * we actually throttled the timer might not unthrottle us for
+ * an entire period. We additionally needed to make sure that
+ * any subsequent check_cfs_rq_runtime calls agree not to
+ * throttle us, as we may commit to do cfs put_prev+pick_next,
+ * so we ask for 1ns of runtime rather than just check cfs_b.
+ *
+ * This will start the period timer if necessary.
+ */
+ if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1))
+ return false;
+
+ /*
+ * No bandwidth available; Add ourselves on the list to be
+ * unthrottled later.
*/
- dequeue = 0;
- } else {
list_add_tail_rcu(&cfs_rq->throttled_list,
&cfs_b->throttled_cfs_rq);
}
- raw_spin_unlock(&cfs_b->lock);
-
- if (!dequeue)
- return false; /* Throttle no longer required. */
/* freeze hierarchy runnable averages while throttled */
- rcu_read_lock();
- walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq);
- rcu_read_unlock();
+ scoped_guard(rcu)
+ walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq);
/*
* Note: distribution will already see us throttled via the
@@ -6894,13 +6890,15 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
update_rq_clock(rq);
- raw_spin_lock(&cfs_b->lock);
- if (cfs_rq->throttled_clock) {
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
+ list_del_rcu(&cfs_rq->throttled_list);
+
+ if (!cfs_rq->throttled_clock)
+ break;
+
cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock;
cfs_rq->throttled_clock = 0;
}
- list_del_rcu(&cfs_rq->throttled_list);
- raw_spin_unlock(&cfs_b->lock);
/* update hierarchical throttle state */
walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq);
@@ -6929,9 +6927,8 @@ static void __cfsb_csd_unthrottle(void *arg)
{
struct cfs_rq *cursor, *tmp;
struct rq *rq = arg;
- struct rq_flags rf;
- rq_lock(rq, &rf);
+ guard(rq_lock)(rq);
/*
* Iterating over the list can trigger several call to
@@ -6948,7 +6945,7 @@ static void __cfsb_csd_unthrottle(void *arg)
* race with group being freed in the window between removing it
* from the list and advancing to the next entry in the list.
*/
- rcu_read_lock();
+ guard(rcu)();
list_for_each_entry_safe(cursor, tmp, &rq->cfsb_csd_list,
throttled_csd_list) {
@@ -6958,10 +6955,7 @@ static void __cfsb_csd_unthrottle(void *arg)
unthrottle_cfs_rq(cursor);
}
- rcu_read_unlock();
-
rq_clock_stop_loop_update(rq);
- rq_unlock(rq, &rf);
}
static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
@@ -7001,11 +6995,11 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
u64 runtime, remaining = 1;
bool throttled = false;
struct cfs_rq *cfs_rq, *tmp;
- struct rq_flags rf;
struct rq *rq;
LIST_HEAD(local_unthrottle);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
throttled_list) {
rq = rq_of(cfs_rq);
@@ -7015,65 +7009,63 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
break;
}
- rq_lock_irqsave(rq, &rf);
+ guard(rq_lock_irqsave)(rq);
+
if (!cfs_rq_throttled(cfs_rq))
- goto next;
+ continue;
/* Already queued for async unthrottle */
if (!list_empty(&cfs_rq->throttled_csd_list))
- goto next;
+ continue;
/* By the above checks, this should never be true */
WARN_ON_ONCE(cfs_rq->runtime_remaining > 0);
- raw_spin_lock(&cfs_b->lock);
- runtime = -cfs_rq->runtime_remaining + 1;
- if (runtime > cfs_b->runtime)
- runtime = cfs_b->runtime;
- cfs_b->runtime -= runtime;
- remaining = cfs_b->runtime;
- raw_spin_unlock(&cfs_b->lock);
+ scoped_guard(raw_spinlock, &cfs_b->lock) {
+ runtime = -cfs_rq->runtime_remaining + 1;
+ if (runtime > cfs_b->runtime)
+ runtime = cfs_b->runtime;
+ cfs_b->runtime -= runtime;
+ remaining = cfs_b->runtime;
+ }
cfs_rq->runtime_remaining += runtime;
- /* we check whether we're throttled above */
- if (cfs_rq->runtime_remaining > 0) {
- if (cpu_of(rq) != this_cpu) {
- unthrottle_cfs_rq_async(cfs_rq);
- } else {
- /*
- * We currently only expect to be unthrottling
- * a single cfs_rq locally.
- */
- WARN_ON_ONCE(!list_empty(&local_unthrottle));
- list_add_tail(&cfs_rq->throttled_csd_list,
- &local_unthrottle);
- }
- } else {
+ /*
+ * Ran out of bandwidth during distribution!
+ * Indicate throttled entities and break early.
+ */
+ if (cfs_rq->runtime_remaining <= 0) {
throttled = true;
+ break;
}
-next:
- rq_unlock_irqrestore(rq, &rf);
+ /* we check whether we're throttled above */
+ if (cpu_of(rq) != this_cpu) {
+ unthrottle_cfs_rq_async(cfs_rq);
+ continue;
+ }
+
+ /*
+ * We currently only expect to be unthrottling
+ * a single cfs_rq locally.
+ */
+ WARN_ON_ONCE(!list_empty(&local_unthrottle));
+ list_add_tail(&cfs_rq->throttled_csd_list, &local_unthrottle);
}
list_for_each_entry_safe(cfs_rq, tmp, &local_unthrottle,
throttled_csd_list) {
struct rq *rq = rq_of(cfs_rq);
- rq_lock_irqsave(rq, &rf);
+ guard(rq_lock_irqsave)(rq);
list_del_init(&cfs_rq->throttled_csd_list);
-
if (cfs_rq_throttled(cfs_rq))
unthrottle_cfs_rq(cfs_rq);
-
- rq_unlock_irqrestore(rq, &rf);
}
WARN_ON_ONCE(!list_empty(&local_unthrottle));
- rcu_read_unlock();
-
return throttled;
}
@@ -7196,7 +7188,8 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
if (slack_runtime <= 0)
return;
- raw_spin_lock(&cfs_b->lock);
+ guard(raw_spinlock)(&cfs_b->lock);
+
if (cfs_b->quota != RUNTIME_INF) {
cfs_b->runtime += slack_runtime;
@@ -7205,7 +7198,6 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
!list_empty(&cfs_b->throttled_cfs_rq))
start_cfs_slack_bandwidth(cfs_b);
}
- raw_spin_unlock(&cfs_b->lock);
/* even if it's not valid for return we don't want to try again */
cfs_rq->runtime_remaining -= slack_runtime;
@@ -7228,25 +7220,21 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
*/
static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
{
- u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
- unsigned long flags;
-
/* confirm we're still not at a refresh boundary */
- raw_spin_lock_irqsave(&cfs_b->lock, flags);
- cfs_b->slack_started = false;
+ scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
+ u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
- if (runtime_refresh_within(cfs_b, min_bandwidth_expiration)) {
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
- return;
- }
+ cfs_b->slack_started = false;
- if (cfs_b->quota != RUNTIME_INF && cfs_b->runtime > slice)
- runtime = cfs_b->runtime;
+ if (runtime_refresh_within(cfs_b, min_bandwidth_expiration))
+ return;
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+ if (cfs_b->quota != RUNTIME_INF && cfs_b->runtime > slice)
+ runtime = cfs_b->runtime;
- if (!runtime)
- return;
+ if (!runtime)
+ return;
+ }
distribute_cfs_runtime(cfs_b);
}
@@ -7335,18 +7323,18 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
container_of(timer, struct cfs_bandwidth, period_timer);
- unsigned long flags;
int overrun;
int idle = 0;
int count = 0;
- raw_spin_lock_irqsave(&cfs_b->lock, flags);
+ CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
+
for (;;) {
overrun = hrtimer_forward_now(timer, cfs_b->period);
if (!overrun)
break;
- idle = do_sched_cfs_period_timer(cfs_b, overrun, flags);
+ idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
if (++count > 3) {
u64 new, old = ktime_to_ns(cfs_b->period);
@@ -7379,11 +7367,13 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
count = 0;
}
}
- if (idle)
+
+ if (idle) {
cfs_b->period_active = 0;
- raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+ return HRTIMER_NORESTART;
+ }
- return idle ? HRTIMER_NORESTART : HRTIMER_RESTART;
+ return HRTIMER_RESTART;
}
void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth *parent)
@@ -7450,14 +7440,12 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
*/
for_each_possible_cpu(i) {
struct rq *rq = cpu_rq(i);
- unsigned long flags;
if (list_empty(&rq->cfsb_csd_list))
continue;
- local_irq_save(flags);
- __cfsb_csd_unthrottle(rq);
- local_irq_restore(flags);
+ scoped_guard(irqsave)
+ __cfsb_csd_unthrottle(rq);
}
}
@@ -7475,16 +7463,15 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq)
lockdep_assert_rq_held(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
- raw_spin_lock(&cfs_b->lock);
- cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
- raw_spin_unlock(&cfs_b->lock);
+ scoped_guard(raw_spinlock, &cfs_b->lock)
+ cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
}
- rcu_read_unlock();
}
/* cpu offline callback */
@@ -7505,7 +7492,8 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
*/
rq_clock_start_loop_update(rq);
- rcu_read_lock();
+ guard(rcu)();
+
list_for_each_entry_rcu(tg, &task_groups, list) {
struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
@@ -7528,7 +7516,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
cfs_rq->runtime_remaining = 1;
unthrottle_cfs_rq(cfs_rq);
}
- rcu_read_unlock();
rq_clock_stop_loop_update(rq);
}
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-06-04 18:45 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-06-02 5:00 [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 1/5] sched/fair: Convert cfs bandwidth throttling to use guards K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:00 ` [PATCH v2 2/5] sched/fair: Use throttled_csd_list for local unthrottle K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 3/5] sched/fair: Call update_curr() before unthrottling the hierarchy K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-06-02 5:25 ` [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() K Prateek Nayak
2026-06-02 7:10 ` [PATCH v2.1 " K Prateek Nayak
2026-06-04 18:45 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®