* [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
Add cpu.max.runtime, a cgroup v2 file that reads and sets the CFS
bandwidth runtime, cfs_b->runtime, in microseconds. Userspace can then
grant a cgroup runtime beyond its quota. Refills still add only the
quota, capped at quota + burst, so the excess is used up over time and
the group returns to its quota per period.
A write is rejected without a quota or above quota + burst, the most
the group can accumulate. A read is capped at quota + burst so it can
be written back; uncapped slack from __return_cfs_rq_runtime() can
exceed that until the next refill. Runtime the CPUs already took, at
most a slice each, is not included and a write leaves it in place.
A write distributes the new runtime to throttled cfs_rqs at once, as
the slack timer does, rather than waiting for the unsynchronized
period timer. It also shifts runtime_snap by the change, so cpu.stat
burst accounting neither counts the write nor hides what the group ran
since the last refill.
Also drop the burst <= quota limit and require only quota + burst <=
max_bw_runtime_us (about 203 days), the quota's own limit, so a group
can accumulate many periods of quota. This applies to cgroup v1
cpu.cfs_burst_us too, which shares tg_set_bandwidth().
The file exists with CONFIG_CFS_BANDWIDTH only.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 22 +-
Documentation/scheduler/sched-bwc.rst | 13 +-
kernel/sched/core.c | 34 ++-
kernel/sched/fair.c | 61 ++++
kernel/sched/sched.h | 2 +
tools/testing/selftests/cgroup/test_cpu.c | 336 ++++++++++++++++++++++
6 files changed, 459 insertions(+), 9 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2e..c5a02057f0242 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1231,12 +1231,32 @@ will be referred to. All time durations are in microseconds.
A read-write single value file which exists on non-root
cgroups. The default is "0".
- The burst in the range [0, $MAX].
+ The burst. The cgroup can accumulate the runtime it does not
+ use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
+ long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
+ days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
the callback actually does.
+ cpu.max.runtime
+ A read-write single value file which exists on non-root
+ cgroups if the kernel is built with CONFIG_CFS_BANDWIDTH.
+
+ The runtime the cgroup has accumulated and not used yet, in
+ microseconds. Each period refills it by $MAX, up to $MAX +
+ cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
+ A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
+ that ran out of runtime run again at once. A write fails with
+ EINVAL while $MAX is "max".
+
+ The runtime does not include what the cgroup's CPUs already took
+ from it, at most a slice each (sched_cfs_bandwidth_slice_us).
+ A write leaves that with the CPUs.
+
+ This file affects only processes under the fair-class scheduler.
+
cpu.pressure
A read-write nested-keyed file.
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index e881a945c1888..c22b3c8c14e29 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -90,8 +90,8 @@ bandwidth restriction in place, such a group is described as an unconstrained
bandwidth group. This represents the traditional work-conserving behavior for
CFS.
-Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
-enact the specified bandwidth limit. The minimum quota allowed for the quota or
+Writing any (valid) positive value(s) will enact the specified bandwidth
+limit. The minimum quota allowed for the quota or
period is 1ms. There is also an upper bound on the period length of 1s.
Additional restrictions exist when bandwidth limits are used in a hierarchical
fashion, these are explained in more detail below.
@@ -101,9 +101,10 @@ and return the group to an unconstrained state once more.
A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
-CFS unchanged. Writing any (valid) positive value(s) no larger than
-cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
-accumulation.
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation: the group can accumulate
+up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
+quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
@@ -243,4 +244,4 @@ Examples
# echo 50000 > cpu.cfs_period_us /* period = 50ms */
# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */
- Larger buffer setting (no larger than quota) allows greater burst capacity.
+ Larger buffer setting allows greater burst capacity.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 175b52a7ea793..76fcd70ff207a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10131,8 +10131,7 @@ static int tg_set_bandwidth(struct task_group *tg,
if (quota_us != RUNTIME_INF && quota_us > max_bw_runtime_us)
return -EINVAL;
- if (quota_us != RUNTIME_INF && (burst_us > quota_us ||
- burst_us + quota_us > max_bw_runtime_us))
+ if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
return -EINVAL;
#ifdef CONFIG_CFS_BANDWIDTH
@@ -10193,6 +10192,29 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css,
tg_bandwidth(tg, &period_us, "a_us, NULL);
return tg_set_bandwidth(tg, period_us, quota_us, burst_us);
}
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype, u64 runtime_us)
+{
+ struct task_group *tg = css_tg(css);
+
+ if (runtime_us > max_bw_runtime_us)
+ return -EINVAL;
+
+ return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+ runtime_us * NSEC_PER_USEC);
+}
+
+static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype)
+{
+ struct task_group *tg = css_tg(css);
+
+ return div_u64(cfs_bandwidth_get_runtime(&tg->cfs_bandwidth),
+ NSEC_PER_USEC);
+}
+#endif /* CONFIG_CFS_BANDWIDTH */
#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_RT_GROUP_SCHED
@@ -10544,7 +10566,15 @@ static struct cftype cpu_files[] = {
.read_u64 = cpu_burst_read_u64,
.write_u64 = cpu_burst_write_u64,
},
+#ifdef CONFIG_CFS_BANDWIDTH
+ {
+ .name = "max.runtime",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .read_u64 = cpu_runtime_read_u64,
+ .write_u64 = cpu_runtime_write_u64,
+ },
#endif /* CONFIG_CFS_BANDWIDTH */
+#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_UCLAMP_TASK_GROUP
{
.name = "uclamp.min",
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d4cc0caee6265..58e948ed1792f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7421,6 +7421,8 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
struct cfs_rq *cfs_rq;
struct rq *rq;
+ lockdep_assert_irqs_disabled();
+
guard(rcu)();
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
@@ -7811,6 +7813,65 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * The runtime of @cfs_b, as cpu.max.runtime reads it.
+ *
+ * __return_cfs_rq_runtime() returns slack without capping it, so the runtime
+ * can exceed quota + burst until the next refill caps it. Report it capped,
+ * so that a value read can always be written back.
+ */
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
+{
+ u64 runtime;
+
+ guard(raw_spinlock_irq)(&cfs_b->lock);
+
+ runtime = cfs_b->runtime;
+ if (cfs_b->quota != RUNTIME_INF)
+ runtime = min(runtime, cfs_b->quota + cfs_b->burst);
+
+ return runtime;
+}
+
+/*
+ * Set the runtime of @cfs_b, as a cpu.max.runtime write does, and hand it to
+ * the throttled cfs_rqs at once rather than at the next period.
+ */
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
+{
+ bool throttled;
+ s64 snap;
+
+ guard(irq)();
+
+ scoped_guard (raw_spinlock, &cfs_b->lock) {
+ if (cfs_b->quota == RUNTIME_INF ||
+ runtime > cfs_b->quota + cfs_b->burst)
+ return -EINVAL;
+
+ /*
+ * The next refill counts runtime_snap - runtime as burst usage.
+ * Move runtime_snap by what the write adds or takes away, so
+ * that the write itself is not counted as burst usage but what
+ * the group ran since the last refill still is.
+ */
+ snap = cfs_b->runtime_snap + runtime - cfs_b->runtime;
+ cfs_b->runtime_snap = max(snap, 0);
+ cfs_b->runtime = runtime;
+
+ throttled = !list_empty(&cfs_b->throttled_cfs_rq) && runtime;
+ }
+
+ /*
+ * distribute_cfs_runtime() takes the rq locks, so it runs without
+ * cfs_b->lock and with interrupts disabled, as from the slack timer.
+ */
+ if (throttled)
+ distribute_cfs_runtime(cfs_b);
+
+ return 0;
+}
+
void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
{
lockdep_assert_held(&cfs_b->lock);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a4df59c9c9b60..1f60c84e0dc38 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -578,6 +578,8 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth
extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 3456e3e988c68..278d7e16eeb2c 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -809,6 +809,337 @@ static int test_cpucg_max_nested(const char *root)
return ret;
}
+/* Wait until the period timer of @cgroup starts a new period. */
+static int wait_period_start(const char *cgroup)
+{
+ long nr_periods = cg_read_key_long(cgroup, "cpu.stat", "nr_periods");
+ int i;
+
+ for (i = 0; i < 3000; i++) {
+ usleep(1000);
+ if (cg_read_key_long(cgroup, "cpu.stat", "nr_periods") != nr_periods)
+ return 0;
+ }
+ return -1;
+}
+
+static long long monotonic_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1000000000LL + ts.tv_nsec;
+}
+
+/* For (long)arg seconds, run for 1ms and then sleep for 10ms. */
+static int run_and_sleep(const char *cgroup, void *arg)
+{
+ long long end = monotonic_ns() + (long)arg * 1000000000LL;
+ long long start;
+
+ while ((start = monotonic_ns()) < end) {
+ while (monotonic_ns() - start < 1000000)
+ ;
+ usleep(10000);
+ }
+ return 0;
+}
+
+static int test_cpucg_max_runtime(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 1000; /* 1ms (minimum) */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 5000000; /* 5s, so cap = 5001ms */
+ long runtime_usec = 2500000; /* 2500ms, half of the 5s run */
+ long duration_sec = 5;
+ /* The injected runtime plus the quota refilled each period of the run */
+ long expected_usec = runtime_usec +
+ duration_sec * USEC_PER_SEC / period_usec * quota_usec;
+ long usage_usec;
+ char *cpucg;
+ char buf[64];
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* Without a quota there is no runtime to set. */
+ if (!cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+
+ /* Start burner, let it settle, then inject credits */
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = duration_sec, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ usleep(100000);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+
+ waitpid(pid, NULL, 0);
+
+ usage_usec = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_usec <= 0)
+ goto cleanup;
+
+ /* At least the injected runtime, at most 5% above the expected usage. */
+ if (usage_usec < runtime_usec ||
+ usage_usec > expected_usec + expected_usec / 20)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write is not burst usage: writing a runtime below the
+ * current one leaves cpu.stat's nr_bursts and burst_usec where they were.
+ */
+static int test_cpucg_max_runtime_burst_stats(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long nr_bursts, burst_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_stats_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* A long period, so the write lands before the timer's first refill. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+
+ /* The idle group refills for a few periods. */
+ sleep(3);
+ nr_bursts = cg_read_key_long(cpucg, "cpu.stat", "nr_bursts");
+ burst_usec = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (nr_bursts != 0 || burst_usec != 0)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write hands the runtime to a throttled group at once,
+ * rather than at the next period: with a 1s period, the group runs again
+ * within 100ms of the write, before the next period starts.
+ */
+static int test_cpucg_max_runtime_unthrottle(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long usage_before, usage_after, nr_periods;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_unthrottle_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 1ms per 1s period, with room for a runtime of up to 1s. */
+ if (cg_write(cpucg, "cpu.max", "1000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 999000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 2, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /*
+ * The period timer's phase is random: start right after a boundary,
+ * so that the next one is some 900ms away. The hog uses up its runtime
+ * and is throttled for the period.
+ */
+ if (wait_period_start(cpucg))
+ goto cleanup_hog;
+ nr_periods = cg_read_key_long(cpucg, "cpu.stat", "nr_periods");
+ usleep(50000);
+ usage_before = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_before < 0 || usage_before > 20000)
+ goto cleanup_hog;
+
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 500000))
+ goto cleanup_hog;
+
+ /* No period started meanwhile: only the write lets the hog run. */
+ usleep(100000);
+ usage_after = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (cg_read_key_long(cpucg, "cpu.stat", "nr_periods") != nr_periods)
+ goto cleanup_hog;
+ if (usage_after - usage_before < 50000)
+ goto cleanup_hog;
+
+ ret = KSFT_PASS;
+
+cleanup_hog:
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * What a group ran beyond its quota before a cpu.max.runtime write is still
+ * burst usage: a write of the runtime the group has left does not hide it.
+ */
+static int test_cpucg_max_runtime_burst_kept(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long burst_before, burst_after, runtime_usec;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_kept_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 10ms per 1s period, and a runtime of 1s to run on. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 990000))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 1000000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 10, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /* Run some 300ms, all in one period, and stop. */
+ if (wait_period_start(cpucg)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+ burst_before = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ usleep(300000);
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+
+ /* Write back what is left: the next refill counts the 300ms run. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+ sleep(1);
+ burst_after = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (burst_before < 0 || burst_after - burst_before < 250000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * cpu.max.runtime reads at most quota + burst, so that a value read can be
+ * written back. A cfs_rq that dequeues returns its unused runtime, which can
+ * take the runtime above quota + burst until the next refill caps it: run
+ * tasks that keep doing that, and read and write back the runtime meanwhile.
+ */
+static int test_cpucg_max_runtime_read_capped(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long cap_usec = 100000; /* quota 100ms, no burst */
+ long long end;
+ long runtime_usec;
+ char *cpucg;
+ int pids[2];
+ int i, n = 0;
+
+ cpucg = cg_name(root, "cpucg_runtime_read_capped_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "100000 100000"))
+ goto cleanup;
+
+ for (n = 0; n < 2; n++) {
+ pids[n] = cg_run_nowait(cpucg, run_and_sleep, (void *)4L);
+ if (pids[n] < 0)
+ goto cleanup_tasks;
+ }
+
+ end = monotonic_ns() + 3000000000LL;
+ while (monotonic_ns() < end) {
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 0 || runtime_usec > cap_usec)
+ goto cleanup_tasks;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup_tasks;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup_tasks:
+ for (i = 0; i < n; i++) {
+ kill(pids[i], SIGKILL);
+ waitpid(pids[i], NULL, 0);
+ }
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -823,6 +1154,11 @@ struct cpucg_test {
T(test_cpucg_nested_weight_underprovisioned),
T(test_cpucg_max),
T(test_cpucg_max_nested),
+ T(test_cpucg_max_runtime),
+ T(test_cpucg_max_runtime_burst_stats),
+ T(test_cpucg_max_runtime_unthrottle),
+ T(test_cpucg_max_runtime_burst_kept),
+ T(test_cpucg_max_runtime_read_capped),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth Fernand Sieber
3 siblings, 0 replies; 5+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
cpu.max.burst lets a group accumulate unused quota up to quota + burst.
But the period timer stops after a period in which the group took no
runtime, and start_cfs_bandwidth() restarts it with hrtimer_forward_now()
and discards the periods that passed. A group that sits idle stops
accumulating, and one that wakes after idling for a long time has only
what it had when the timer stopped.
Count the missed periods instead. While the timer is stopped, its expiry
is the first period boundary it missed, so the overrun at restart is the
number of refills owed. Add quota for each, capped at quota + burst as
__refill_cfs_bandwidth_runtime() caps it.
Without a burst the cap is quota. The timer stops right after a refill
that nothing drew from, so the runtime is at quota then, and groups
without a burst see no change. Groups with a burst do see a change. A
group that has been idle now has up to quota + burst when it runs again.
The documentation of cpu.max.burst, cpu.max.runtime and cpu.cfs_burst_us
says so.
A change of the bandwidth (cpu.max, cpu.max.burst) applies from that
moment, and the group keeps what it earned before it. The periods owed
are first refilled at the bandwidth they were earned under, capped at
its quota + burst, and then the new bandwidth applies, with its own cap.
A group that had no quota earned nothing, so one given a quota after a
long time unconstrained does not start full. A cpu.max.runtime write
also refills the periods owed first, so that it replaces them, and only
the periods after the write refill.
cpu.max.runtime reports the runtime the group has now, owed refills
included, without starting the timer.
The idle periods are not added to cpu.stat's nr_periods, which counts the
periods in which the timer ran. A timer that runs late by several
periods still refills only once, unlike a stopped one; this patch leaves
that as it is.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 15 +-
Documentation/scheduler/sched-bwc.rst | 5 +-
kernel/sched/core.c | 6 +
kernel/sched/fair.c | 79 +++++++++-
kernel/sched/sched.h | 1 +
tools/testing/selftests/cgroup/test_cpu.c | 183 ++++++++++++++++++++++
6 files changed, 278 insertions(+), 11 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index c5a02057f0242..2c220e5998639 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1232,9 +1232,9 @@ will be referred to. All time durations are in microseconds.
cgroups. The default is "0".
The burst. The cgroup can accumulate the runtime it does not
- use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
- long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
- days).
+ use, up to $MAX + cpu.max.burst, also in the periods in which it
+ does not run. The burst may exceed $MAX, as long as $MAX +
+ cpu.max.burst is at most 2^44 - 1 (about 203 days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
@@ -1246,10 +1246,11 @@ will be referred to. All time durations are in microseconds.
The runtime the cgroup has accumulated and not used yet, in
microseconds. Each period refills it by $MAX, up to $MAX +
- cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
- A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
- that ran out of runtime run again at once. A write fails with
- EINVAL while $MAX is "max".
+ cpu.max.burst, whether the cgroup runs in it or not. A read
+ returns at most $MAX + cpu.max.burst. A write sets it, up to
+ $MAX + cpu.max.burst, and lets a cgroup that ran out of runtime
+ run again at once. A write fails with EINVAL while $MAX is
+ "max".
The runtime does not include what the cgroup's CPUs already took
from it, at most a slice each (sched_cfs_bandwidth_slice_us).
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index c22b3c8c14e29..ccf7434ca73f0 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -103,8 +103,9 @@ A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
will enact the cap on unused bandwidth accumulation: the group can accumulate
-up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
-quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
+up to cpu.cfs_quota_us + cpu.cfs_burst_us, also over periods in which it does
+not run. The burst may be larger than the quota, as long as their sum is at
+most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 76fcd70ff207a..75a5f5d1b8f54 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9856,6 +9856,12 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
cfs_bandwidth_usage_inc();
scoped_guard (raw_spinlock_irq, &cfs_b->lock) {
+ /*
+ * The new bandwidth applies from now on. The group keeps what
+ * it earned while idle under the old one.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
cfs_b->period = ns_to_ktime(period);
cfs_b->quota = quota;
cfs_b->burst = burst;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 58e948ed1792f..bc068b3fa4ed1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7813,6 +7813,75 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * Runtime after @periods refills of quota, capped at quota + burst as each
+ * refill in __refill_cfs_bandwidth_runtime() caps it. The runtime can be above
+ * the cap (__return_cfs_rq_runtime() does not cap it), and a refill then cuts
+ * it to the cap.
+ */
+static u64 cfs_bandwidth_refilled(struct cfs_bandwidth *cfs_b, u64 periods)
+{
+ u64 cap = cfs_b->quota + cfs_b->burst;
+ u64 runtime = cfs_b->runtime;
+
+ if (!periods || cfs_b->quota == RUNTIME_INF)
+ return runtime;
+ if (runtime >= cap ||
+ periods >= div64_u64(cap - runtime, cfs_b->quota) + 1)
+ return cap;
+ return runtime + periods * cfs_b->quota;
+}
+
+/*
+ * The period timer stops after a period in which the group took no runtime
+ * (do_sched_cfs_period_timer()), and start_cfs_bandwidth() restarts it on
+ * the next demand. The periods it slept through refill quota too: unused
+ * quota accumulates up to quota + burst whether or not the group ran. While
+ * the timer is stopped its expiry is the first period boundary it missed,
+ * so the boundaries passed since then are the refills still owed.
+ *
+ * Without burst the cap is quota. The timer stops right after a refill that
+ * nothing drew from, so the runtime is at quota then, and this changes nothing
+ * for groups without a burst.
+ *
+ * This is the overrun that hrtimer_forward_now() returns, without moving the
+ * expiry: one for the expiry itself, plus one for each full period since.
+ */
+static u64 cfs_bandwidth_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ ktime_t now = ktime_get();
+ s64 delta;
+
+ if (cfs_b->period_active)
+ return 0;
+
+ delta = ktime_to_ns(ktime_sub(now, hrtimer_get_expires(&cfs_b->period_timer)));
+ if (delta < 0)
+ return 0;
+ return div64_u64(delta, ktime_to_ns(cfs_b->period)) + 1;
+}
+
+/*
+ * Refill the periods a stopped timer slept through, at the bandwidth they
+ * were earned under, and move its expiry past them so that they are not
+ * counted again. A change of the bandwidth or a cpu.max.runtime write calls
+ * this first, so that the change applies from then on and the group keeps
+ * what it earned before. A group without a quota earned nothing.
+ */
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ u64 idle_periods;
+
+ lockdep_assert_held(&cfs_b->lock);
+
+ if (cfs_b->period_active)
+ return;
+
+ idle_periods = hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
+ cfs_b->runtime = cfs_bandwidth_refilled(cfs_b, idle_periods);
+ cfs_b->runtime_snap = cfs_b->runtime;
+}
+
/*
* The runtime of @cfs_b, as cpu.max.runtime reads it.
*
@@ -7826,7 +7895,7 @@ u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
guard(raw_spinlock_irq)(&cfs_b->lock);
- runtime = cfs_b->runtime;
+ runtime = cfs_bandwidth_refilled(cfs_b, cfs_bandwidth_idle_periods(cfs_b));
if (cfs_b->quota != RUNTIME_INF)
runtime = min(runtime, cfs_b->quota + cfs_b->burst);
@@ -7849,6 +7918,12 @@ int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
runtime > cfs_b->quota + cfs_b->burst)
return -EINVAL;
+ /*
+ * Refill the periods that passed before the write, so that the
+ * write replaces them and only the periods after it refill.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
/*
* The next refill counts runtime_snap - runtime as burst usage.
* Move runtime_snap by what the write adds or takes away, so
@@ -7879,8 +7954,8 @@ void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
if (cfs_b->period_active)
return;
+ cfs_bandwidth_refill_idle_periods(cfs_b);
cfs_b->period_active = 1;
- hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
hrtimer_start_expires(&cfs_b->period_timer, HRTIMER_MODE_ABS_PINNED);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1f60c84e0dc38..676e6371d0265 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -580,6 +580,7 @@ extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 278d7e16eeb2c..04a061da514ce 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -1140,6 +1140,186 @@ static int test_cpucg_max_runtime_read_capped(const char *root)
return ret;
}
+/*
+ * A group with a burst keeps earning quota while it is idle, up to
+ * quota + burst, and cpu.max.runtime reads what it has earned. Without a
+ * burst the runtime stays at quota.
+ */
+static int test_cpucg_max_idle_refill(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 990000; /* cap 1s */
+ long runtime_usec;
+ char *cpucg;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+
+ /* No burst: the runtime stays at quota while idle. */
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ /* Burst: the write refills once (20ms), then 1s idle earns 10 more. */
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 110000 || runtime_usec > 130000)
+ goto cleanup;
+
+ /* Capped at quota + burst. */
+ sleep(10);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec + burst_usec)
+ goto cleanup;
+
+ /*
+ * A write sets the runtime from now on: 500ms later, five refills, or
+ * six if the sleep overruns into the next period.
+ */
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+ usleep(500000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max or cpu.max.burst write applies from then on: what a group with
+ * a quota earned while idle is kept, capped at the new quota + burst, and a
+ * group that had no quota has earned nothing while it had none.
+ */
+static int test_cpucg_max_idle_refill_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long runtime_usec;
+ char *cpucg = NULL, *fresh = NULL;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_write_test");
+ fresh = cg_name(root, "cpucg_idle_refill_fresh_test");
+ if (!cpucg || !fresh)
+ goto cleanup;
+
+ if (cg_create(cpucg) || cg_create(fresh))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ /* 3s idle fills the group to quota + burst. */
+ sleep(3);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* Writing the same cpu.max keeps what the group earned. */
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* A lower burst keeps it too, capped at the new quota + burst. */
+ sleep(2);
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 90000))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 100000)
+ goto cleanup;
+
+ /* A group given a quota after idling without one starts with a refill. */
+ if (cg_write_numeric(fresh, "cpu.max.burst", 190000))
+ goto cleanup;
+ sleep(2);
+ if (cg_write(fresh, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(fresh, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (fresh)
+ cg_destroy(fresh);
+ if (cpucg)
+ cg_destroy(cpucg);
+ free(fresh);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write sets the runtime exactly, and only the periods
+ * after it refill.
+ */
+static int test_cpucg_max_idle_refill_runtime_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long runtime_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_idle_refill_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "10000 100000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ sleep(3);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 30000))
+ goto cleanup;
+ /* One refill if a period boundary passes before the read. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 30000 && runtime_usec != 40000)
+ goto cleanup;
+
+ /* 250ms after the write: two or three refills, by the boundaries. */
+ usleep(250000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -1159,6 +1339,9 @@ struct cpucg_test {
T(test_cpucg_max_runtime_unthrottle),
T(test_cpucg_max_runtime_burst_kept),
T(test_cpucg_max_runtime_read_capped),
+ T(test_cpucg_max_idle_refill),
+ T(test_cpucg_max_idle_refill_write),
+ T(test_cpucg_max_idle_refill_runtime_write),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth Fernand Sieber
3 siblings, 0 replies; 5+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
Add a sched_ext_ops callback, and its cid-form variant, that is invoked
when userspace writes cpu.max.runtime. A BPF scheduler that implements
ops.cgroup_set_bandwidth() to enforce a cgroup's CPU bandwidth also
needs the written runtime, so that its budget for the cgroup matches the
fair class's, for example after userspace restores a saved runtime or
adds to it.
The callback is invoked on every write, after the fair class has applied
it. Unlike cgroup_set_bandwidth(), it does not skip a write of the value
last written. The runtime goes down as the cgroup runs, so writing the
same value again is a top-up.
A new tg_bandwidth_mutex is held across the fair class's update and the
callback, by the cpu.max.runtime write and by tg_set_bandwidth(), which
the cpu.max and cgroup v1 cpu.cfs_* writes go through and which calls
cgroup_set_bandwidth(). The runtime passed is then at most the quota +
burst the scheduler was last given, and cpu.max and cpu.max.runtime
writes reach it in the order the fair class applied them. The mutex is
taken before, and the callbacks run after, the fair class's
cpus_read_lock() and cfs_constraints_mutex sections, so
cgroup_set_bandwidth() runs as it did before.
Like cgroup_set_bandwidth(), the callback runs in process context and
may be sleepable.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +++-
kernel/sched/core.c | 25 +++++++++++++++++++++++--
kernel/sched/ext/ext.c | 19 +++++++++++++++++++
kernel/sched/ext/ext.h | 2 ++
kernel/sched/ext/internal.h | 21 +++++++++++++++++++++
5 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 2c220e5998639..b77ca6f8ca548 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1256,7 +1256,9 @@ will be referred to. All time durations are in microseconds.
from it, at most a slice each (sched_cfs_bandwidth_slice_us).
A write leaves that with the CPUs.
- This file affects only processes under the fair-class scheduler.
+ This file affects only processes under the fair-class scheduler and a BPF
+ scheduler with the ``cgroup_set_runtime`` callback depending on what
+ the callback actually does.
cpu.pressure
A read-write nested-keyed file.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 75a5f5d1b8f54..25a4acb72106c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10099,6 +10099,14 @@ static u64 cpu_period_read_u64(struct cgroup_subsys_state *css,
return period_us;
}
+/*
+ * Held across a bandwidth or runtime write and the sched_ext callback that
+ * follows it, so that the BPF scheduler gets the writes in the order the fair
+ * class applied them. The callback runs after the fair class's update has
+ * dropped cpus_read_lock() and cfs_constraints_mutex.
+ */
+static DEFINE_MUTEX(tg_bandwidth_mutex);
+
static int tg_set_bandwidth(struct task_group *tg,
u64 period_us, u64 quota_us, u64 burst_us)
{
@@ -10140,6 +10148,8 @@ static int tg_set_bandwidth(struct task_group *tg,
if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
return -EINVAL;
+ guard(mutex)(&tg_bandwidth_mutex);
+
#ifdef CONFIG_CFS_BANDWIDTH
ret = tg_set_cfs_bandwidth(tg, period_us, quota_us, burst_us);
#endif /* CONFIG_CFS_BANDWIDTH */
@@ -10204,12 +10214,23 @@ static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
struct cftype *cftype, u64 runtime_us)
{
struct task_group *tg = css_tg(css);
+ int ret;
if (runtime_us > max_bw_runtime_us)
return -EINVAL;
- return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
- runtime_us * NSEC_PER_USEC);
+ /*
+ * Serialize against tg_set_bandwidth(), so that the BPF scheduler gets
+ * the runtime checked against the current quota + burst, and this and
+ * cpu.max writes reach it in the order the fair class took them.
+ */
+ guard(mutex)(&tg_bandwidth_mutex);
+
+ ret = cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+ runtime_us * NSEC_PER_USEC);
+ if (!ret)
+ scx_group_set_runtime(tg, runtime_us);
+ return ret;
}
static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index ebdf7b91f9d37..265a80dd93282 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -5817,6 +5817,19 @@ void scx_group_set_bandwidth(struct task_group *tg,
percpu_up_read(&scx_cgroup_ops_rwsem);
}
+
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us)
+{
+ struct scx_sched *sch;
+
+ percpu_down_read(&scx_cgroup_ops_rwsem);
+ sch = scx_tg_knob_sched(tg);
+
+ if (scx_cgroup_enabled && sch && SCX_HAS_OP(sch, cgroup_set_runtime))
+ SCX_CALL_OP(sch, cgroup_set_runtime, NULL, tg_cgrp(tg), runtime_us);
+
+ percpu_up_read(&scx_cgroup_ops_rwsem);
+}
#endif /* CONFIG_EXT_GROUP_SCHED */
#if defined(CONFIG_EXT_GROUP_SCHED) || defined(CONFIG_EXT_SUB_SCHED)
@@ -9230,6 +9243,7 @@ static int bpf_scx_check_member(const struct btf_type *t,
case offsetof(struct sched_ext_ops, cgroup_exit):
case offsetof(struct sched_ext_ops, cgroup_prep_move):
case offsetof(struct sched_ext_ops, cgroup_set_bandwidth):
+ case offsetof(struct sched_ext_ops, cgroup_set_runtime):
#endif
case offsetof(struct sched_ext_ops, cpu_online):
case offsetof(struct sched_ext_ops, cpu_offline):
@@ -9392,6 +9406,7 @@ static void sched_ext_ops__cgroup_cancel_move(struct task_struct *p, struct cgro
static void sched_ext_ops__cgroup_set_weight(struct cgroup *cgrp, u32 weight) {}
static void sched_ext_ops__cgroup_set_bandwidth(struct cgroup *cgrp, u64 period_us, u64 quota_us, u64 burst_us) {}
static void sched_ext_ops__cgroup_set_idle(struct cgroup *cgrp, bool idle) {}
+static void sched_ext_ops__cgroup_set_runtime(struct cgroup *cgrp, u64 runtime_us) {}
#endif /* CONFIG_EXT_GROUP_SCHED */
static s32 sched_ext_ops__sub_attach(struct scx_sub_attach_args *args) { return -EINVAL; }
static void sched_ext_ops__sub_detach(struct scx_sub_detach_args *args) {}
@@ -9434,6 +9449,7 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.cgroup_set_weight = sched_ext_ops__cgroup_set_weight,
.cgroup_set_bandwidth = sched_ext_ops__cgroup_set_bandwidth,
.cgroup_set_idle = sched_ext_ops__cgroup_set_idle,
+ .cgroup_set_runtime = sched_ext_ops__cgroup_set_runtime,
#endif
.sub_attach = sched_ext_ops__sub_attach,
.sub_detach = sched_ext_ops__sub_detach,
@@ -9502,6 +9518,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {
.cpuctl_set_weight = sched_ext_ops__cgroup_set_weight,
.cpuctl_set_bandwidth = sched_ext_ops__cgroup_set_bandwidth,
.cpuctl_set_idle = sched_ext_ops__cgroup_set_idle,
+ .cpuctl_set_runtime = sched_ext_ops__cgroup_set_runtime,
#endif
.sub_attach = sched_ext_ops__sub_attach,
.sub_detach = sched_ext_ops__sub_detach,
@@ -12226,6 +12243,7 @@ static const u32 scx_kf_allow_flags[] = {
[SCX_OP_IDX(cgroup_set_weight)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(cgroup_set_bandwidth)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(cgroup_set_idle)] = SCX_KF_ALLOW_UNLOCKED,
+ [SCX_OP_IDX(cgroup_set_runtime)] = SCX_KF_ALLOW_UNLOCKED,
#endif /* CONFIG_EXT_GROUP_SCHED */
[SCX_OP_IDX(sub_attach)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(sub_detach)] = SCX_KF_ALLOW_UNLOCKED,
@@ -12395,6 +12413,7 @@ static int __init scx_init(void)
CID_OFFSET_MATCH(cgroup_set_weight, cpuctl_set_weight);
CID_OFFSET_MATCH(cgroup_set_bandwidth, cpuctl_set_bandwidth);
CID_OFFSET_MATCH(cgroup_set_idle, cpuctl_set_idle);
+ CID_OFFSET_MATCH(cgroup_set_runtime, cpuctl_set_runtime);
#endif
/* @priv tail must align since both share the same data block */
CID_OFFSET_MATCH(priv, priv);
diff --git a/kernel/sched/ext/ext.h b/kernel/sched/ext/ext.h
index d32bc01071b01..6d1ce582bf724 100644
--- a/kernel/sched/ext/ext.h
+++ b/kernel/sched/ext/ext.h
@@ -94,6 +94,7 @@ void scx_cgroup_cancel_attach(struct cgroup_taskset *tset);
void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight);
void scx_group_set_idle(struct task_group *tg, bool idle);
void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us);
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us);
#else /* CONFIG_EXT_GROUP_SCHED */
static inline void scx_tg_init(struct task_group *tg) {}
static inline int scx_tg_online(struct task_group *tg) { return 0; }
@@ -104,5 +105,6 @@ static inline void scx_cgroup_cancel_attach(struct cgroup_taskset *tset) {}
static inline void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight) {}
static inline void scx_group_set_idle(struct task_group *tg, bool idle) {}
static inline void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us) {}
+static inline void scx_group_set_runtime(struct task_group *tg, u64 runtime_us) {}
#endif /* CONFIG_EXT_GROUP_SCHED */
#endif /* CONFIG_CGROUP_SCHED */
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index ff0a3d508533f..e794507596d6f 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -863,6 +863,26 @@ struct sched_ext_ops {
*/
void (*cgroup_set_idle)(struct cgroup *cgrp, bool idle);
+ /**
+ * @cgroup_set_runtime: A cgroup's runtime is being set
+ * @cgrp: cgroup whose runtime is being set
+ * @runtime_us: runtime in microseconds
+ *
+ * Set the runtime @cgrp has accumulated and not used yet to
+ * @runtime_us. This is from the cpu.max.runtime cgroup interface,
+ * which exists with CONFIG_CFS_BANDWIDTH only. It is called on every
+ * write, also of an unchanged value. The runtime goes down as @cgrp
+ * runs, so a write of the same value is a top-up.
+ *
+ * @cgrp has a quota, and @runtime_us is at most the quota + burst the
+ * scheduler was last given, by @cgroup_init or @cgroup_set_bandwidth:
+ * cpu.max and cpu.max.runtime writes reach the scheduler in the order
+ * the fair class applied them.
+ *
+ * Delivery follows the same rule as cgroup_set_weight().
+ */
+ void (*cgroup_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
+
#endif /* CONFIG_EXT_GROUP_SCHED */
/**
@@ -1239,6 +1259,7 @@ struct sched_ext_ops_cid {
void (*cpuctl_set_bandwidth)(struct cgroup *cgrp, u64 period_us, u64 quota_us,
u64 burst_us);
void (*cpuctl_set_idle)(struct cgroup *cgrp, bool idle);
+ void (*cpuctl_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
#endif /* CONFIG_EXT_GROUP_SCHED */
s32 (*sub_attach)(struct scx_sub_attach_args *args);
void (*sub_detach)(struct scx_sub_detach_args *args);
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
` (2 preceding siblings ...)
2026-11-10 11:08 ` [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
3 siblings, 0 replies; 5+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
A BPF scheduler that enforces cpu.max keeps each cgroup's budget
itself, and no runtime moves between it and the fair class. One loaded
after a cpu.max.runtime write starts the cgroup on a fresh quota,
losing the unspent grant.
Conversely, a read reports the fair class's runtime, which refills to
quota + burst meanwhile. On disable, the fair class takes the tasks
back with that full runtime, so a cgroup that used up its budget runs
over its quota until the period accounting catches up.
Under SCX_OPS_SWITCH_PARTIAL, both budgets apply at once: a cgroup with
tasks in both classes runs up to its quota in each, which no handover
at load or unload can fix. Keep one budget, the fair class's, and let
a BPF scheduler opt in to drawing from it with the new
SCX_OPS_CHARGE_CPU_MAX flag.
With SCX_OPS_CHARGE_CPU_MAX, update_curr_scx() charges a task's runtime
to its cgroups' cfs_rqs as a fair task's would, without throttling it.
A task the scheduler stops late leaves its CPU in debt, which the next
periods pay, also once the task has moved; debt past minus a slice
moves to cfs_b, whose next refill pays up to a quota of it. The new
kfunc scx_bpf_task_cpu_max_runtime() returns the runtime left less that
debt, and the scheduler runs the task only while it is positive.
Cgroups whose period timer is stopped are skipped, as only a charge
restarts it.
Both classes then draw from one budget: under SCX_OPS_SWITCH_PARTIAL a
cgroup gets its quota per period however its tasks are split, and
nothing is handed over at load or unload. A task returning to the fair
class leaves its CPU at most a slice in debt; schedulers without the
flag behave as before.
cfs_b->lock is taken at most twice per millisecond of run time per CPU.
The cgroup_runtime selftest covers these cases.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +-
kernel/sched/ext/ext.c | 34 +
kernel/sched/ext/internal.h | 24 +-
kernel/sched/fair.c | 122 ++++
kernel/sched/sched.h | 12 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/include/scx/compat.h | 1 +
tools/testing/selftests/sched_ext/Makefile | 1 +
.../selftests/sched_ext/cgroup_runtime.bpf.c | 195 ++++++
.../selftests/sched_ext/cgroup_runtime.c | 652 ++++++++++++++++++
tools/testing/selftests/sched_ext/config | 1 +
11 files changed, 1045 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.c
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index b77ca6f8ca548..db9fa6e6dbd49 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1258,7 +1258,9 @@ will be referred to. All time durations are in microseconds.
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_runtime`` callback depending on what
- the callback actually does.
+ the callback actually does. A BPF scheduler with the
+ ``SCX_OPS_CHARGE_CPU_MAX`` flag draws what its processes run from
+ the same runtime, so the cgroup's processes in both share it.
cpu.pressure
A read-write nested-keyed file.
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 265a80dd93282..6ffc9ca2c5c9f 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -1625,6 +1625,11 @@ static void expire_task_slice(struct task_struct *p)
scx_set_task_slice(p, 0);
}
+static bool scx_task_charges_cpu_max(const struct task_struct *p)
+{
+ return scx_task_sched(p)->ops.flags & SCX_OPS_CHARGE_CPU_MAX;
+}
+
static void update_curr_scx(struct rq *rq)
{
struct task_struct *donor;
@@ -1650,6 +1655,9 @@ static void update_curr_scx(struct rq *rq)
if (unlikely(donor == scx_rescuee(rq)))
scx_rescue_charge(rq, delta_exec);
+ if (scx_task_charges_cpu_max(donor))
+ cfs_bandwidth_charge_task(rq, donor, delta_exec);
+
dl_server_update(&rq->ext_server, delta_exec);
}
@@ -2605,6 +2613,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
ops_dequeue(rq, p, deq_flags);
+ /* charge what @p ran, then return what @rq holds for its cgroups */
+ if (scx_task_charges_cpu_max(p)) {
+ if (task_current_donor(rq, p))
+ update_curr_scx(rq);
+ cfs_bandwidth_return_task(rq, p);
+ }
+
/*
* A current scheduling context which is going off @rq first gets
* dequeued and then stops running. As we want running <-> stopping
@@ -11793,6 +11808,24 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
return task_cpu(p);
}
+/**
+ * scx_bpf_task_cpu_max_runtime - Runtime a task has left under cpu.max
+ * @p: task of interest
+ *
+ * Return the runtime in nsecs that @p's cgroup and its ancestors have left
+ * under their cpu.max quotas for @p on its current CPU, the least over those
+ * with a quota: the cgroup's cpu.max.runtime plus what the CPU holds for
+ * the cgroup, less what it owes. U64_MAX if none limits it. Read without
+ * locks.
+ *
+ * With %SCX_OPS_CHARGE_CPU_MAX, the scheduler should run @p only while this
+ * returns more than 0. The runtime grows when a period refills it.
+ */
+__bpf_kfunc u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p)
+{
+ return cfs_bandwidth_task_runtime(p);
+}
+
/**
* scx_bpf_task_cid - cid a task is currently associated with
* @p: task of interest
@@ -12119,6 +12152,7 @@ BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE)
BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_task_cpu_max_runtime, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_task_cid, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_locked_rq, KF_IMPLICIT_ARGS | KF_RET_NULL)
BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED)
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index e794507596d6f..e8c0b573d0a43 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -243,6 +243,27 @@ enum scx_ops_flags {
*/
SCX_OPS_ENQ_BLOCKED = 1LLU << 10,
+ /*
+ * If set, what the scheduler's tasks run is charged to the cpu.max
+ * bandwidth of their cgroups, the runtime the fair class keeps and
+ * cpu.max.runtime reads, as what fair tasks run is. The tasks of a
+ * cgroup in both classes then share one quota, also with
+ * %SCX_OPS_SWITCH_PARTIAL, and nothing is handed over when tasks
+ * switch class.
+ *
+ * The kernel never throttles the scheduler's tasks. The scheduler
+ * enforces the quota by running a task only while
+ * scx_bpf_task_cpu_max_runtime() returns more than 0. What a task runs
+ * beyond that is charged: up to a slice (sched_cfs_bandwidth_slice_us)
+ * per CPU as a debt the cgroup's fair tasks on that CPU pay back, and up
+ * to a quota more from the cgroup's next period. The scheduler does not
+ * need ops.cgroup_set_runtime(), as a write of cpu.max.runtime sets that
+ * runtime.
+ *
+ * Without CONFIG_CFS_BANDWIDTH, this flag has no effect.
+ */
+ SCX_OPS_CHARGE_CPU_MAX = 1LLU << 11,
+
SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE |
SCX_OPS_ENQ_LAST |
SCX_OPS_ENQ_EXITING |
@@ -253,7 +274,8 @@ enum scx_ops_flags {
SCX_OPS_ALWAYS_ENQ_IMMED |
SCX_OPS_TID_TO_TASK |
SCX_OPS_LAZY_RESCHED |
- SCX_OPS_ENQ_BLOCKED,
+ SCX_OPS_ENQ_BLOCKED |
+ SCX_OPS_CHARGE_CPU_MAX,
/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
__SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bc068b3fa4ed1..9ff239f122bda 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6875,6 +6875,11 @@ void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b)
}
cfs_b->runtime = min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
+
+ /* pay the debt cfs_bandwidth_charge_task() left */
+ runtime = min(cfs_b->runtime, cfs_b->debt);
+ cfs_b->runtime -= runtime;
+ cfs_b->debt -= runtime;
cfs_b->runtime_snap = cfs_b->runtime;
}
@@ -8076,6 +8081,123 @@ bool cfs_task_bw_constrained(struct task_struct *p)
return false;
}
+/*
+ * Charge @delta_exec, which @p ran on @rq outside the fair class, to the
+ * bandwidth of its task group and of each ancestor with a quota, as
+ * account_cfs_rq_runtime() charges a fair task. A cfs_rq that runs out takes
+ * another slice from cfs_b, but @p is never throttled: the BPF scheduler that
+ * runs it stops it, and the group's fair tasks are throttled when they run.
+ *
+ * Debt below -slice that cfs_b cannot cover moves to cfs_b, up to a quota,
+ * and the next refill pays it, so that a scheduler which keeps running @p
+ * cannot put the group's fair tasks on @rq in more debt than a slice.
+ */
+void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p,
+ u64 delta_exec)
+{
+ s64 floor = -(s64)sched_cfs_bandwidth_slice();
+ struct task_group *tg;
+
+ lockdep_assert_rq_held(rq);
+
+ if (!cfs_bandwidth_used())
+ return;
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
+
+ if (!cfs_rq->runtime_enabled)
+ continue;
+
+ cfs_rq->runtime_remaining -= delta_exec;
+
+ /*
+ * A throttled cfs_rq gets runtime from distribute_cfs_runtime().
+ * Without runtime, an active period timer refills it, so skip
+ * the lock. A stopped one is restarted by the assign.
+ */
+ if (cfs_rq->runtime_remaining <= 0 && !cfs_rq->throttled &&
+ (READ_ONCE(cfs_b->runtime) || !READ_ONCE(cfs_b->period_active))) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ __assign_cfs_rq_runtime(cfs_b, cfs_rq,
+ sched_cfs_bandwidth_slice());
+ }
+
+ if (cfs_rq->runtime_remaining < floor) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ cfs_b->debt = min_t(u64, cfs_b->debt + floor -
+ cfs_rq->runtime_remaining, cfs_b->quota);
+ cfs_rq->runtime_remaining = floor;
+ }
+ }
+}
+
+/*
+ * @p leaves @rq: give the runtime @rq holds for its groups back, less
+ * min_cfs_rq_runtime, as a fair dequeue does when a cfs_rq empties. Pay
+ * what @rq owes from cfs_b, as @p may not run on @rq again.
+ */
+void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p)
+{
+ struct task_group *tg;
+
+ lockdep_assert_rq_held(rq);
+
+ if (!cfs_bandwidth_used())
+ return;
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
+
+ return_cfs_rq_runtime(cfs_rq);
+
+ if (cfs_rq->runtime_enabled && cfs_rq->runtime_remaining < 0 &&
+ !cfs_rq->throttled && READ_ONCE(cfs_b->runtime)) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ __assign_cfs_rq_runtime(cfs_b, cfs_rq, 0);
+ }
+ }
+}
+
+/*
+ * The runtime @p's groups have left for it on its CPU, without locks: the
+ * least, over the groups with a quota, of what the CPU holds or owes and
+ * cfs_b's runtime. Debt counts up to a quota less 1ns, so that a full period
+ * lets @p run and pay the rest. A group whose period timer is stopped does
+ * not count, as the next charge restarts the timer and refills what it owes.
+ * RUNTIME_INF if no group counts.
+ */
+u64 cfs_bandwidth_task_runtime(struct task_struct *p)
+{
+ u64 runtime = RUNTIME_INF;
+ struct task_group *tg;
+ int cpu = task_cpu(p);
+
+ if (!cfs_bandwidth_used())
+ return runtime;
+
+ guard(rcu)();
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu);
+ s64 held;
+
+ if (!READ_ONCE(cfs_rq->runtime_enabled) ||
+ !READ_ONCE(cfs_b->period_active))
+ continue;
+
+ held = max_t(s64, READ_ONCE(cfs_rq->runtime_remaining),
+ 1 - (s64)READ_ONCE(cfs_b->quota));
+ held += READ_ONCE(cfs_b->runtime);
+ runtime = min_t(u64, runtime, max_t(s64, held, 0));
+ }
+
+ return runtime;
+}
+
#ifdef CONFIG_NO_HZ_FULL
/* called from pick_next_task_fair() */
static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 676e6371d0265..fd98000ffd744 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -456,6 +456,7 @@ struct cfs_bandwidth {
u64 runtime;
u64 burst;
u64 runtime_snap;
+ u64 debt;
s64 hierarchical_quota;
u8 idle;
@@ -621,6 +622,17 @@ static inline bool cfs_task_bw_constrained(struct task_struct *p) { return false
#endif /* !CONFIG_CGROUP_SCHED */
+#ifdef CONFIG_CFS_BANDWIDTH
+void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p, u64 delta_exec);
+void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p);
+u64 cfs_bandwidth_task_runtime(struct task_struct *p);
+#else
+static inline void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p,
+ u64 delta_exec) { }
+static inline void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p) { }
+static inline u64 cfs_bandwidth_task_runtime(struct task_struct *p) { return RUNTIME_INF; }
+#endif
+
/*
* A weight of 0 or 1 can cause arithmetics problems.
* A weight of a cfs_rq is the sum of weights of which entities
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 6194484d4d9a9..cfb7f1b70665f 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -98,6 +98,7 @@ s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags
s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
bool scx_bpf_task_running(const struct task_struct *p) __ksym;
s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
+u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p) __ksym __weak;
struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym __weak;
struct rq *scx_bpf_locked_rq(void) __ksym;
struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index 07fb55e63c57c..60b9972f1a5dc 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -215,6 +215,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
+#define SCX_OPS_CHARGE_CPU_MAX SCX_OPS_FLAG(SCX_OPS_CHARGE_CPU_MAX)
#define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
index c5d3a2eaea7da..4a727b8dc7aab 100644
--- a/tools/testing/selftests/sched_ext/Makefile
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -165,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
auto-test-targets := \
cgroup_nr_cpus \
+ cgroup_runtime \
create_dsq \
dequeue \
dequeue_iter \
diff --git a/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c b/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
new file mode 100644
index 0000000000000..ec98a25d19c29
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A scheduler with SCX_OPS_CHARGE_CPU_MAX, for the cgroup_runtime tests. It
+ * runs a task only while scx_bpf_task_cpu_max_runtime() says it has runtime
+ * left, unless @ignore_runtime is set. A task that has none waits in HELD_DSQ,
+ * and a timer kicks the idle CPUs every millisecond, so that they take it
+ * once a period refills the runtime.
+ *
+ * partial_ops sets SCX_OPS_SWITCH_PARTIAL and full_ops does not. Both pass
+ * the cpu.max and cpu.max.runtime writes of @cgid on to the test.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define HELD_DSQ 0
+#define KICK_NS 1000000
+
+UEI_DEFINE(uei);
+
+/* the cgroup under test, set before attaching */
+u64 cgid;
+
+/* run tasks whether or not they have runtime left */
+bool ignore_runtime;
+
+/* tasks held for lack of runtime */
+u64 nr_held;
+
+/* the cpu.max and cpu.max.runtime writes of @cgid */
+u64 nr_set_bw;
+u64 quota_us;
+u64 burst_us;
+u64 nr_set_runtime;
+/* cgroup_set_runtime() calls above the quota + burst last given */
+u64 nr_bad_order;
+
+struct kick_timer {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct kick_timer);
+} kick_timer SEC(".maps");
+
+static bool runnable(struct task_struct *p)
+{
+ return ignore_runtime || scx_bpf_task_cpu_max_runtime(p);
+}
+
+s32 BPF_STRUCT_OPS(cgroup_runtime_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ if (!runnable(p)) {
+ __sync_fetch_and_add(&nr_held, 1);
+ scx_bpf_dsq_insert(p, HELD_DSQ, SCX_SLICE_DFL, enq_flags);
+ return;
+ }
+ /* the last task of a CPU going idle must go to its local DSQ */
+ if (enq_flags & SCX_ENQ_LAST)
+ scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, enq_flags);
+ else
+ scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_dispatch, s32 cpu, struct task_struct *prev)
+{
+ struct task_struct *p = __COMPAT_scx_bpf_dsq_peek(HELD_DSQ);
+
+ if (p && runnable(p))
+ scx_bpf_dsq_move_to_local(HELD_DSQ, 0);
+}
+
+/*
+ * Stop a task that ran out, so that it goes through enqueue again.
+ * SCX_OPS_ENQ_LAST makes that so also when it is alone on its CPU.
+ */
+void BPF_STRUCT_OPS(cgroup_runtime_tick, struct task_struct *p)
+{
+ if (!runnable(p))
+ scx_bpf_task_set_slice(p, 0);
+}
+
+static int kick_timerfn(void *map, int *key, struct bpf_timer *timer)
+{
+ s32 cpu;
+
+ if (scx_bpf_dsq_nr_queued(HELD_DSQ))
+ bpf_for(cpu, 0, scx_bpf_nr_cpu_ids())
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+
+ bpf_timer_start(timer, KICK_NS, 0);
+ return 0;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_init)
+{
+ struct bpf_timer *timer;
+ u32 key = 0;
+ s32 ret;
+
+ ret = scx_bpf_create_dsq(HELD_DSQ, -1);
+ if (ret)
+ return ret;
+
+ timer = bpf_map_lookup_elem(&kick_timer, &key);
+ if (!timer)
+ return -ESRCH;
+ bpf_timer_init(timer, &kick_timer, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(timer, kick_timerfn);
+ return bpf_timer_start(timer, KICK_NS, 0);
+}
+
+static bool is_target(struct cgroup *cgrp)
+{
+ return cgid && cgrp->kn->id == cgid;
+}
+
+/* sleepable: cpu.max writes call it in process context */
+void BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_cgroup_set_bandwidth,
+ struct cgroup *cgrp, u64 period_us, u64 quota,
+ u64 burst)
+{
+ if (!is_target(cgrp))
+ return;
+
+ __sync_fetch_and_add(&nr_set_bw, 1);
+ quota_us = quota;
+ burst_us = burst;
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_cgroup_set_runtime,
+ struct cgroup *cgrp, u64 runtime_us)
+{
+ if (!is_target(cgrp))
+ return;
+
+ __sync_fetch_and_add(&nr_set_runtime, 1);
+ if (runtime_us > quota_us + burst_us)
+ __sync_fetch_and_add(&nr_bad_order, 1);
+}
+
+s32 BPF_STRUCT_OPS(cgroup_runtime_cgroup_init, struct cgroup *cgrp,
+ struct scx_cgroup_init_args *args)
+{
+ if (is_target(cgrp)) {
+ quota_us = args->bw_quota_us;
+ burst_us = args->bw_burst_us;
+ }
+ return 0;
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(partial_ops,
+ .select_cpu = (void *)cgroup_runtime_select_cpu,
+ .enqueue = (void *)cgroup_runtime_enqueue,
+ .dispatch = (void *)cgroup_runtime_dispatch,
+ .tick = (void *)cgroup_runtime_tick,
+ .init = (void *)cgroup_runtime_init,
+ .cgroup_init = (void *)cgroup_runtime_cgroup_init,
+ .cgroup_set_bandwidth = (void *)cgroup_runtime_cgroup_set_bandwidth,
+ .cgroup_set_runtime = (void *)cgroup_runtime_cgroup_set_runtime,
+ .exit = (void *)cgroup_runtime_exit,
+ .flags = SCX_OPS_SWITCH_PARTIAL |
+ SCX_OPS_ENQ_LAST |
+ SCX_OPS_CHARGE_CPU_MAX,
+ .name = "cgroup_runtime");
+
+SCX_OPS_DEFINE(full_ops,
+ .select_cpu = (void *)cgroup_runtime_select_cpu,
+ .enqueue = (void *)cgroup_runtime_enqueue,
+ .dispatch = (void *)cgroup_runtime_dispatch,
+ .tick = (void *)cgroup_runtime_tick,
+ .init = (void *)cgroup_runtime_init,
+ .cgroup_init = (void *)cgroup_runtime_cgroup_init,
+ .cgroup_set_bandwidth = (void *)cgroup_runtime_cgroup_set_bandwidth,
+ .cgroup_set_runtime = (void *)cgroup_runtime_cgroup_set_runtime,
+ .exit = (void *)cgroup_runtime_exit,
+ .flags = SCX_OPS_ENQ_LAST |
+ SCX_OPS_CHARGE_CPU_MAX,
+ .name = "cgroup_runtime_full");
diff --git a/tools/testing/selftests/sched_ext/cgroup_runtime.c b/tools/testing/selftests/sched_ext/cgroup_runtime.c
new file mode 100644
index 0000000000000..86d50a63a6e73
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/cgroup_runtime.c
@@ -0,0 +1,652 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Verify that a BPF scheduler with SCX_OPS_CHARGE_CPU_MAX shares a cgroup's
+ * cpu.max quota with the fair class: what the cgroup's tasks run in both
+ * classes, with and without SCX_OPS_SWITCH_PARTIAL and while tasks switch
+ * class, stays within the quota; nothing is gained or lost when the
+ * scheduler loads and unloads; and a scheduler that ignores
+ * scx_bpf_task_cpu_max_runtime() leaves the fair tasks at most a quota of
+ * debt and a slice per CPU, while one that stops tasks late pays the overrun
+ * back.
+ * Also verify that cpu.max and cpu.max.runtime writes reach the scheduler
+ * in the order the fair class applied them.
+ *
+ * Usage is cpu.stat's usage_usec. A window of n periods may hold n + 1
+ * refills and start or end with up to a quota, and each CPU may hold up to
+ * a slice (sched_cfs_bandwidth_slice_us), so the checks allow that.
+ */
+
+#define _GNU_SOURCE
+#include <bpf/bpf.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <linux/limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "cgroup_runtime.bpf.skel.h"
+#include "cgroup_util.h"
+#include "scx_test.h"
+
+#ifndef SCHED_EXT
+#define SCHED_EXT 7
+#endif
+
+#define PERIOD_US 100000LL
+#define QUOTA_US 20000LL
+#define NR_PERIODS 10
+#define NR_BURNERS 2
+
+struct ctx {
+ struct cgroup_runtime *skel;
+ struct bpf_link *link;
+ char root[PATH_MAX];
+ char parent[PATH_MAX];
+ char cg[PATH_MAX];
+ bool parent_created;
+ bool cg_created;
+ pid_t burner[NR_BURNERS];
+ long long slice_us;
+ long long saved_slice_us;
+};
+
+static int join_path(char *dst, size_t dst_size, const char *parent, const char *name)
+{
+ int ret;
+
+ ret = snprintf(dst, dst_size, "%s/%s", parent, name);
+ if (ret < 0 || (size_t)ret >= dst_size)
+ return -ENAMETOOLONG;
+ return 0;
+}
+
+static u64 now_us(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000;
+}
+
+static u64 cgroup_id(const char *path)
+{
+ union {
+ u64 id;
+ unsigned char bytes[8];
+ } id = {};
+ struct file_handle *handle;
+ int mount_id, ret;
+
+ handle = calloc(1, sizeof(*handle) + sizeof(id));
+ if (!handle)
+ return 0;
+ handle->handle_bytes = sizeof(id);
+ ret = name_to_handle_at(AT_FDCWD, path, handle, &mount_id, 0);
+ if (!ret && handle->handle_bytes == sizeof(id))
+ memcpy(id.bytes, handle->f_handle, sizeof(id));
+ free(handle);
+
+ return ret ? 0 : id.id;
+}
+
+static long long read_runtime(struct ctx *ctx)
+{
+ return cg_read_long(ctx->cg, "cpu.max.runtime");
+}
+
+static long long read_usage(struct ctx *ctx)
+{
+ return cg_read_key_long(ctx->cg, "cpu.stat", "usage_usec");
+}
+
+static int write_str(const char *cgroup, const char *file, const char *fmt, ...)
+{
+ char buf[64];
+ va_list ap;
+
+ va_start(ap, fmt);
+ vsnprintf(buf, sizeof(buf), fmt, ap);
+ va_end(ap);
+ return cg_write(cgroup, file, buf);
+}
+
+static bool controller_enabled(const char *cgroup, const char *file, const char *controller)
+{
+ char buf[4096], *saveptr, *token;
+
+ if (cg_read(cgroup, file, buf, sizeof(buf)))
+ return false;
+ for (token = strtok_r(buf, "\n ", &saveptr); token;
+ token = strtok_r(NULL, "\n ", &saveptr))
+ if (!strcmp(token, controller))
+ return true;
+ return false;
+}
+
+static int set_policy(pid_t pid, int policy)
+{
+ struct sched_param param = {};
+
+ return sched_setscheduler(pid, policy, ¶m);
+}
+
+static void stop_burners(struct ctx *ctx)
+{
+ int i;
+
+ for (i = 0; i < NR_BURNERS; i++) {
+ if (ctx->burner[i] <= 0)
+ continue;
+ kill(ctx->burner[i], SIGKILL);
+ waitpid(ctx->burner[i], NULL, 0);
+ ctx->burner[i] = 0;
+ }
+}
+
+/* a task in @ctx->cg that spins on CPU @i + 1 under @policy */
+static int start_burner(struct ctx *ctx, int i, int policy)
+{
+ cpu_set_t set;
+ pid_t pid;
+
+ pid = fork();
+ if (pid < 0)
+ return -errno;
+ if (!pid)
+ for (;;)
+ ;
+
+ ctx->burner[i] = pid;
+ CPU_ZERO(&set);
+ CPU_SET((i + 1) % libbpf_num_possible_cpus(), &set);
+ if (sched_setaffinity(pid, sizeof(set), &set) ||
+ cg_enter(ctx->cg, pid) || set_policy(pid, policy))
+ return -1;
+ return 0;
+}
+
+static void cleanup(void *arg)
+{
+ struct ctx *ctx = arg;
+
+ if (!ctx)
+ return;
+ stop_burners(ctx);
+ bpf_link__destroy(ctx->link);
+ cgroup_runtime__destroy(ctx->skel);
+ if (ctx->saved_slice_us)
+ write_str("/proc/sys/kernel", "sched_cfs_bandwidth_slice_us", "%lld",
+ ctx->saved_slice_us);
+ if (ctx->cg_created)
+ cg_destroy(ctx->cg);
+ if (ctx->parent_created)
+ cg_destroy(ctx->parent);
+ free(ctx);
+}
+
+/*
+ * Open the skeleton and make parent/cg under the cgroup2 root, with a quota
+ * of @quota us per @period us, a burst of @burst us and a runtime of
+ * @runtime us on cg.
+ */
+static enum scx_test_status setup_ctx(void **ctxp, long long quota,
+ long long period, long long burst,
+ long long runtime)
+{
+ struct ctx *ctx;
+ char name[64];
+
+ ctx = calloc(1, sizeof(*ctx));
+ if (!ctx)
+ return SCX_TEST_FAIL;
+ *ctxp = ctx;
+
+ /* SCX_ENUM_INIT() exits if vmlinux BTF can't be loaded: before cgroups */
+ ctx->skel = cgroup_runtime__open();
+ if (!ctx->skel) {
+ SCX_ERR("Failed to open skel");
+ return SCX_TEST_FAIL;
+ }
+ SCX_ENUM_INIT(ctx->skel);
+
+ ctx->slice_us = 5000;
+ if (!access("/proc/sys/kernel/sched_cfs_bandwidth_slice_us", R_OK))
+ ctx->slice_us = cg_read_long("/proc/sys/kernel",
+ "sched_cfs_bandwidth_slice_us");
+
+ if (cg_find_unified_root(ctx->root, sizeof(ctx->root), NULL))
+ return SCX_TEST_SKIP;
+ if (!controller_enabled(ctx->root, "cgroup.subtree_control", "cpu") &&
+ cg_write(ctx->root, "cgroup.subtree_control", "+cpu"))
+ return SCX_TEST_SKIP;
+
+ snprintf(name, sizeof(name), "scx_cgroup_runtime_%d", getpid());
+ if (join_path(ctx->parent, sizeof(ctx->parent), ctx->root, name) ||
+ join_path(ctx->cg, sizeof(ctx->cg), ctx->parent, "cg")) {
+ SCX_ERR("Cgroup path is too long");
+ return SCX_TEST_FAIL;
+ }
+
+ if (cg_create(ctx->parent)) {
+ SCX_ERR("Failed to create %s", ctx->parent);
+ return SCX_TEST_FAIL;
+ }
+ ctx->parent_created = true;
+ if (cg_write(ctx->parent, "cgroup.subtree_control", "+cpu")) {
+ SCX_ERR("Failed to enable cpu in %s", ctx->parent);
+ return SCX_TEST_FAIL;
+ }
+ if (cg_create(ctx->cg)) {
+ SCX_ERR("Failed to create %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+ ctx->cg_created = true;
+
+ if (access(cg_control(ctx->cg, "cpu.max.runtime"), F_OK))
+ return SCX_TEST_SKIP;
+
+ if (write_str(ctx->cg, "cpu.max", "%lld %lld", quota, period) ||
+ write_str(ctx->cg, "cpu.max.burst", "%lld", burst) ||
+ write_str(ctx->cg, "cpu.max.runtime", "%lld", runtime)) {
+ SCX_ERR("Failed to configure %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+
+ ctx->skel->bss->cgid = cgroup_id(ctx->cg);
+ if (!ctx->skel->bss->cgid) {
+ SCX_ERR("Failed to read the cgroup ID of %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+ if (libbpf_num_possible_cpus() < NR_BURNERS + 1)
+ return SCX_TEST_SKIP;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status setup_rate(void **ctxp)
+{
+ return setup_ctx(ctxp, QUOTA_US, PERIOD_US, 0, QUOTA_US);
+}
+
+static enum scx_test_status attach_ops(struct ctx *ctx, bool partial)
+{
+ ctx->link = bpf_map__attach_struct_ops(partial ? ctx->skel->maps.partial_ops :
+ ctx->skel->maps.full_ops);
+ SCX_FAIL_IF(!ctx->link, "Failed to attach scheduler");
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status load_ops(struct ctx *ctx, bool partial)
+{
+ bpf_map__set_autoattach(ctx->skel->maps.partial_ops, false);
+ bpf_map__set_autoattach(ctx->skel->maps.full_ops, false);
+ SCX_FAIL_IF(cgroup_runtime__load(ctx->skel), "Failed to load skel");
+ return attach_ops(ctx, partial);
+}
+
+static void detach(struct ctx *ctx)
+{
+ bpf_link__destroy(ctx->link);
+ ctx->link = NULL;
+}
+
+/*
+ * Run two burners in cg under @policy[], the first switching class every
+ * 7 ms if @toggle, and check that cg runs its quota per period, no more and
+ * not much less.
+ */
+static enum scx_test_status run_rate(struct ctx *ctx, bool partial,
+ const int *policy, bool toggle)
+{
+ long long tol = NR_BURNERS * ctx->slice_us + QUOTA_US;
+ long long used, u0;
+ bool ext = false;
+ u64 end;
+ int i;
+
+ if (load_ops(ctx, partial))
+ return SCX_TEST_FAIL;
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, policy[i]), 0);
+
+ usleep(3 * PERIOD_US);
+ u0 = read_usage(ctx);
+ end = now_us() + NR_PERIODS * PERIOD_US;
+ while (now_us() < end) {
+ if (toggle) {
+ ext = !ext;
+ SCX_EQ(set_policy(ctx->burner[0], ext ? SCHED_EXT : SCHED_OTHER), 0);
+ }
+ usleep(7000);
+ }
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us in %d periods of quota %lld us, %lu held\n",
+ used, NR_PERIODS, QUOTA_US, (unsigned long)ctx->skel->bss->nr_held);
+ SCX_GE(used, QUOTA_US * NR_PERIODS - tol);
+ SCX_LE(used, QUOTA_US * NR_PERIODS + tol);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_full(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_OTHER };
+
+ return run_rate(arg, false, policy, false);
+}
+
+static enum scx_test_status run_partial_ext(void *arg)
+{
+ static const int policy[] = { SCHED_EXT, SCHED_EXT };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_partial_fair(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_OTHER };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_partial_mixed(void *arg)
+{
+ static const int policy[] = { SCHED_EXT, SCHED_OTHER };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_switch(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_EXT };
+
+ return run_rate(arg, true, policy, true);
+}
+
+/*
+ * With a quota of 1 ms per second and a runtime of 3 s, what cg runs while
+ * the scheduler loads, runs and unloads is what its runtime loses, up to a
+ * slice per CPU and the few refills.
+ */
+#define LU_QUOTA_US 1000LL
+#define LU_PERIOD_US 1000000LL
+
+static enum scx_test_status setup_load_unload(void **ctxp)
+{
+ return setup_ctx(ctxp, LU_QUOTA_US, LU_PERIOD_US, 4000000, 3000000);
+}
+
+static enum scx_test_status run_load_unload(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long r0, u0, drawn, used, refills, tol;
+ u64 t0, t1;
+ int i;
+
+ bpf_map__set_autoattach(ctx->skel->maps.partial_ops, false);
+ bpf_map__set_autoattach(ctx->skel->maps.full_ops, false);
+ SCX_FAIL_IF(cgroup_runtime__load(ctx->skel), "Failed to load skel");
+
+ t0 = now_us();
+ r0 = read_runtime(ctx);
+ u0 = read_usage(ctx);
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, SCHED_OTHER), 0);
+ usleep(100000);
+ if (attach_ops(ctx, false))
+ return SCX_TEST_FAIL;
+ usleep(200000);
+ detach(ctx);
+ usleep(100000);
+ stop_burners(ctx);
+ drawn = r0 - read_runtime(ctx);
+ used = read_usage(ctx) - u0;
+ t1 = now_us();
+
+ refills = LU_QUOTA_US * ((t1 - t0) / LU_PERIOD_US + 2);
+ tol = NR_BURNERS * ctx->slice_us;
+ printf("ran %lld us, runtime down %lld us\n", used, drawn);
+ SCX_GE(used, 300000);
+ SCX_GE(used, drawn - tol);
+ SCX_LE(used, drawn + refills + tol);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * A scheduler that ignores scx_bpf_task_cpu_max_runtime() runs a task far
+ * past the quota. Once the task is back on the fair class, it runs its quota
+ * again, less at most a quota and a slice of debt.
+ */
+static enum scx_test_status run_clamp(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long u0, ignored, used;
+
+ ctx->skel->bss->ignore_runtime = true;
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ SCX_EQ(start_burner(ctx, 0, SCHED_EXT), 0);
+
+ u0 = read_usage(ctx);
+ usleep(5 * PERIOD_US);
+ ignored = read_usage(ctx) - u0;
+
+ SCX_EQ(set_policy(ctx->burner[0], SCHED_OTHER), 0);
+ u0 = read_usage(ctx);
+ usleep(5 * PERIOD_US);
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us ignoring the quota, then %lld us on the fair class\n",
+ ignored, used);
+ SCX_GE(ignored, 4 * PERIOD_US);
+ SCX_GE(used, 4 * QUOTA_US - 2 * ctx->slice_us);
+ SCX_LE(used, 6 * QUOTA_US + ctx->slice_us);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * The scheduler stops a task only at a tick, so each period it overruns a
+ * quota of 2 ms by up to a tick. With a slice of 1 ms, the next tick's
+ * charge takes the CPU past the -slice floor. The debt must still be paid
+ * from the next periods, so over 100 periods cg runs its quota, with the
+ * second burner moving between CPUs 2 and 3 every 7 ms.
+ */
+#define DEBT_QUOTA_US 2000LL
+#define DEBT_PERIOD_US 10000LL
+#define DEBT_PERIODS 100
+
+static enum scx_test_status setup_debt(void **ctxp)
+{
+ enum scx_test_status ret;
+ struct ctx *ctx;
+
+ ret = setup_ctx(ctxp, DEBT_QUOTA_US, DEBT_PERIOD_US, 0, DEBT_QUOTA_US);
+ if (ret != SCX_TEST_PASS)
+ return ret;
+ ctx = *ctxp;
+ if (write_str("/proc/sys/kernel", "sched_cfs_bandwidth_slice_us", "1000"))
+ return SCX_TEST_SKIP;
+ ctx->saved_slice_us = ctx->slice_us;
+ ctx->slice_us = 1000;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_debt(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long tol = (NR_BURNERS + 1) * ctx->slice_us + DEBT_QUOTA_US;
+ long long want = DEBT_QUOTA_US * DEBT_PERIODS;
+ long long used, u0;
+ bool moved = false;
+ cpu_set_t set;
+ u64 end;
+ int i;
+
+ if (libbpf_num_possible_cpus() < 4)
+ return SCX_TEST_SKIP;
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, SCHED_EXT), 0);
+
+ usleep(3 * DEBT_PERIOD_US);
+ u0 = read_usage(ctx);
+ end = now_us() + DEBT_PERIODS * DEBT_PERIOD_US;
+ while (now_us() < end) {
+ moved = !moved;
+ CPU_ZERO(&set);
+ CPU_SET(moved ? 3 : 2, &set);
+ SCX_EQ(sched_setaffinity(ctx->burner[1], sizeof(set), &set), 0);
+ usleep(7000);
+ }
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us in %d periods of quota %lld us\n", used,
+ DEBT_PERIODS, DEBT_QUOTA_US);
+ SCX_GE(used, want - tol);
+ SCX_LE(used, want + tol);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * One thread alternates cpu.max between quotas of 100ms and 10ms, without a
+ * burst, while another writes a runtime of 90ms, which only the first
+ * allows. The scheduler must never get a runtime above the quota it was last
+ * given, and must end with the quota cpu.max has.
+ */
+#define INTERLEAVE_US 2000000ULL
+
+struct interleave {
+ struct ctx *ctx;
+ long nr_ok;
+ long nr_einval;
+ long nr_err;
+};
+
+static void *interleave_quota(void *arg)
+{
+ struct interleave *il = arg;
+ u64 end = now_us() + INTERLEAVE_US;
+ bool high = false;
+
+ while (now_us() < end) {
+ if (write_str(il->ctx->cg, "cpu.max", "%lld %lld",
+ high ? 100000LL : 10000LL, 1000000LL))
+ il->nr_err++;
+ else
+ il->nr_ok++;
+ high = !high;
+ }
+ return NULL;
+}
+
+static void *interleave_runtime(void *arg)
+{
+ struct interleave *il = arg;
+ u64 end = now_us() + INTERLEAVE_US;
+
+ while (now_us() < end) {
+ int fd = cg_open(il->ctx->cg, "cpu.max.runtime", O_WRONLY);
+
+ if (fd < 0) {
+ il->nr_err++;
+ continue;
+ }
+ if (write(fd, "90000", 5) == 5)
+ il->nr_ok++;
+ else if (errno == EINVAL)
+ il->nr_einval++;
+ else
+ il->nr_err++;
+ close(fd);
+ }
+ return NULL;
+}
+
+static enum scx_test_status setup_interleave(void **ctxp)
+{
+ return setup_ctx(ctxp, 100000, 1000000, 0, 100000);
+}
+
+static enum scx_test_status run_interleave(void *arg)
+{
+ struct ctx *ctx = arg;
+ struct interleave quota = { .ctx = ctx }, runtime = { .ctx = ctx };
+ pthread_t tq, tr;
+ char buf[64];
+
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ SCX_EQ(ctx->skel->bss->quota_us, 100000);
+
+ SCX_EQ(pthread_create(&tq, NULL, interleave_quota, "a), 0);
+ SCX_EQ(pthread_create(&tr, NULL, interleave_runtime, &runtime), 0);
+ pthread_join(tq, NULL);
+ pthread_join(tr, NULL);
+
+ SCX_EQ(quota.nr_err, 0);
+ SCX_EQ(runtime.nr_err, 0);
+ SCX_GT(runtime.nr_ok, 0);
+ SCX_EQ(ctx->skel->bss->nr_bad_order, 0);
+ SCX_EQ(ctx->skel->bss->nr_set_bw, quota.nr_ok);
+ SCX_EQ(ctx->skel->bss->nr_set_runtime, runtime.nr_ok);
+
+ SCX_EQ(cg_read(ctx->cg, "cpu.max", buf, sizeof(buf)), 0);
+ SCX_EQ(ctx->skel->bss->quota_us, atoll(buf));
+ SCX_EQ(ctx->skel->bss->burst_us, 0);
+
+ printf("cpu.max writes %ld, cpu.max.runtime writes %ld (%ld EINVAL)\n",
+ quota.nr_ok, runtime.nr_ok, runtime.nr_einval);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+#define RUNTIME_TEST(__name, __setup, __desc) \
+ struct scx_test cgroup_runtime_##__name = { \
+ .name = "cgroup_runtime_" #__name, \
+ .description = __desc, \
+ .setup = __setup, \
+ .run = run_##__name, \
+ .cleanup = cleanup, \
+ }
+
+RUNTIME_TEST(full, setup_rate, "Without partial switch, the BPF scheduler's tasks run their quota");
+RUNTIME_TEST(partial_ext, setup_rate, "SCHED_EXT tasks of a cgroup run its quota");
+RUNTIME_TEST(partial_fair, setup_rate, "Fair tasks of a cgroup run its quota");
+RUNTIME_TEST(partial_mixed, setup_rate, "Fair and SCHED_EXT tasks of a cgroup share its quota");
+RUNTIME_TEST(switch, setup_rate, "A task switching class every 7 ms stays within the quota");
+RUNTIME_TEST(load_unload, setup_load_unload, "Load and unload neither gain nor lose runtime");
+RUNTIME_TEST(clamp, setup_rate, "A scheduler ignoring the quota leaves a bounded debt");
+RUNTIME_TEST(debt, setup_debt, "Debt past a quota is paid in the next periods");
+RUNTIME_TEST(interleave, setup_interleave, "cpu.max writes reach the scheduler in order");
+
+/* REGISTER_SCX_TEST() can be used once per file */
+__attribute__((constructor))
+static void register_cgroup_runtime_tests(void)
+{
+ scx_test_register(&cgroup_runtime_full);
+ scx_test_register(&cgroup_runtime_partial_ext);
+ scx_test_register(&cgroup_runtime_partial_fair);
+ scx_test_register(&cgroup_runtime_partial_mixed);
+ scx_test_register(&cgroup_runtime_switch);
+ scx_test_register(&cgroup_runtime_load_unload);
+ scx_test_register(&cgroup_runtime_clamp);
+ scx_test_register(&cgroup_runtime_debt);
+ scx_test_register(&cgroup_runtime_interleave);
+}
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
index 8173f9170ffe8..495b140403a18 100644
--- a/tools/testing/selftests/sched_ext/config
+++ b/tools/testing/selftests/sched_ext/config
@@ -9,3 +9,4 @@ CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_EXPERT=y
CONFIG_SCHED_PROXY_EXEC=y
+CONFIG_CFS_BANDWIDTH=y
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 5+ messages in thread