* [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests
@ 2026-11-17 14:10 Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 01/40] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
` (39 more replies)
0 siblings, 40 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel
Hi Ingo, Juri,
This series fixes 36 bugs of SCHED_DEADLINE. Most of them are in its
bandwidth accounting. Then it shows the deadline bandwidth of each
runqueue in sched_debug, and adds three selftests. The fixes come first,
as you asked, so that stable can take patches 1 to 36.
Three of the bugs take the kernel down (patches 4, 8 and 26). One is a
use after free in the rtmutex chain walk (patch 9). One leaves a task
that never runs again (patch 12). One leaves cpuset_mutex locked (patch
6). Most of the others leave total_bw of a root domain, or this_bw of a
runqueue, wrong. The root domain then admits deadline tasks beyond its
capacity, or refuses every one.
Patch 20 is Andrea Parri's. I carry it unchanged, with his agreement.
It is reviewed by Sebastian. It adds an affinity flag and a check next
to the code that patch 21 changes. Take it by whichever route you
prefer.
Juri, your four fixes that Ingo dropped from sched/urgent are still
here, renumbered:
- 1/4 is patch 18.
- 2/4 is patch 13.
- 3/4 is patch 1.
- 4/4 is patch 14.
The fixes
=========
1 Refuse SCHED_FLAG_KEEP_PARAMS requests that would change the policy
or the class.
2 Skip admission for parameters that SCHED_FLAG_KEEP_PARAMS keeps.
3 Let a task boosted by a deadline waiter change its policy.
4 Do not divide by zero active CPUs of a root domain.
5 Refuse a dl_server debugfs write while the runqueue is offline.
6 Take cpuset_mutex for a deadline policy change only with pi, so
that sysrq-n does not leave it locked.
7 Replenish a throttled task switched out and back, so that it runs
again.
8 Do not refill a boost from cleared parameters, so that the owner's
CPU no longer locks up.
9 Continue the rtmutex chain walk when the deadline donor of a waiter
changes, so that no owner keeps a donor that can exit.
10 Reset the deadline and the runtime with the parameters.
11 Start a new period at a deboost in the class, if the boost
refilled the task's runtime from its donor.
12 Replenish a throttled task that a waiter boosts while it sleeps, so
that it runs again.
13 Release the bandwidth of a task that leaves the class asleep.
14 Reject SCHED_DEADLINE on exited tasks (Juri Lelli).
15 Keep the bandwidth of departing tasks across a root domain rebuild.
16 Keep this_bw right when a task changes during a remote wakeup.
17 Rebuild the root domains' accounting from the task list.
18 Record the root domain that holds a task's bandwidth.
19 Leave out of a rebuild a dying task that released its bandwidth.
20 Check the affinity of a deadline task under the runqueue lock, so
that sched_setaffinity() and sched_setattr() cannot both succeed
(Andrea Parri).
21 Move a task's bandwidth only where an attach made room for it.
22 Wait for an attach before rebuilding the sched domains.
23 Make a policy or affinity change of a deadline task wait for a
cpuset attach in progress.
24 Give back the bandwidth an attach did not move.
25 Reserve an attach's bandwidth in the root domain each task moves to.
26 Make a rebuild count a task in the root domain it runs in.
27 Move a task's bandwidth to the root domain it runs in.
28 Pick a CPU for a task in the root domain that counts its
bandwidth, when that root domain has one.
29 Refuse to take offline the last CPU a task has in the root domain
that counts its bandwidth.
30 Refuse a cpuset change that leaves a deadline task no CPU in the
root domain that counts its bandwidth.
31 Cancel the replenishment timer with the parameters.
32 Count in this_bw the bandwidth a boosted task is given.
33 Give a deadline to a task that enters the class while it waits on
an rt_mutex.
34 Cap the runtime left of a constrained task at dl_runtime when its
parameters change.
35 Check the runtime left of a task that comes back to the class
runnable, as a wakeup does.
36 Check the runtime left of a runnable task whose parameters change.
Patches 18, 19 and 21 to 30 go together. 18 records the root domain
that counts a task's bandwidth. 19 and 21 to 27 fix the rebuild, attach
and migration paths that the record exposes. 28 to 30 keep a task in
the root domain that counts it, as long as that root domain has a CPU
for it. Patch 32 completes patch 16. Patch 37 shows this_bw in
sched_debug. Patches 38 to 40 add 112 cases in
tools/testing/selftests/sched/deadline, as Juri suggested.
Peter, patch 9 changes the rtmutex chain walk: a waiter records its
deadline donor with its sort keys. Patch 25 also touches
__set_cpus_allowed_ptr_locked(). A cpuset attach can reserve a deadline
task's bandwidth in one root domain. If the CPU picked for the task is
in the other one, the patch steers the task to the reserved CPU. Patch
23 touches __sched_setscheduler() and sched_setaffinity(). A change of
a deadline task waits there for a cpuset attach in progress. Patch 28
touches select_fallback_rq() and __set_cpus_allowed_ptr_locked() for
deadline tasks.
Waiman, patches 29 and 30 add refusals. They apply when a change would
leave a deadline task no CPU in the root domain that counts it. That
needs a cpuset with CPUs of two root domains: with isolcpus="domain",
or on cgroup v1 with sched_load_balance off. A CPU offline or a write
to cpuset.cpus then fails with EBUSY. cpuset makes a partition root
invalid, with a new reason. Nothing is refused with admission control
off (sched_rt_runtime_us set to -1).
Known limits
============
The fixes leave these cases open. Each patch named below also states
its limit in its message.
- Patch 9: two chain walks can overlap on a chain of three locks or
more. The owners further down keep the old donor until the first walk
reaches them.
- Patch 15: a task that left the class keeps its bandwidth until its
0-lag time. A rebuild still drops it earlier if the CPU of the task
is inactive, or if the task dies during the rebuild.
- Patch 27: a task can run in a root domain that does not count its
bandwidth. The bandwidth then moves there without a check for room,
as dl_task_offline_migration() does upstream.
- Patch 28: a CPU picked while the sched domains are rebuilt has no
preference for a root domain, as upstream. No bandwidth moves
meanwhile. But a wakeup can pick its CPU inside the rebuild and be
queued after it. Its bandwidth can then move to the other root domain
with no check for room. Push and pull between the two root domains
during the rebuild are part of the same limit. This was found by
reading, not reproduced.
- Patches 28 to 30: a task with no CPU left in the root domain that
counts it still moves, and its bandwidth with it. With admission
control on, patches 29 and 30 refuse the CPU offline and the cpuset
changes that I found to lead there. Nothing is refused during a
suspend, or with admission control off.
- A cpuset that holds isolated and housekeeping CPUs still admits a
deadline task. Juri pointed at the XXX in dl_bw_capacity() for this
in the v5 thread. It is left for later.
Found, left for separate patches
================================
The Sashiko review of v6 found two upstream bugs outside the scope of
this series. Both were found by reading. I will post their fixes
separately:
- The 0-lag time in task_non_contending() can overflow with periods of
a few seconds. The bandwidth would then be released seconds late.
- dl_task_offline_migration() and dl_add_task_root_domain() spread
extra_bw over the whole span, offline CPUs included. Tasks with
SCHED_FLAG_RECLAIM would then reclaim a little more than their share,
until the next rebuild.
Testing
=======
The 112 cases pass wherever they run, with lockdep and no kernel
warning, in 19 QEMU guests:
- 5 guests of 8 vCPUs, one with proxy execution off, and 1 of 4 vCPUs.
- 3 guests with PREEMPT_RT, of 8, 8 and 4 vCPUs.
- 3 guests without CONFIG_CPUSETS, of 4, 4 and 5 vCPUs.
- 4 guests of 5 vCPUs booted with isolcpus=domain,1,2 or
isolcpus=domain,0,1, 2 of them with PREEMPT_RT.
- 1 guest of 8 vCPUs booted with isolcpus=domain,6,7.
- 2 guests with CONFIG_RT_GROUP_SCHED, of 8 and 4 vCPUs.
The 23 isolcpus cases run in the fourth group. There, the cases that
need 4 CPUs that are not isolated skip. rebuild_waking_keeps also skips
with isolcpus=domain,1,2: it needs the first CPU isolated. Some of the
23 also run in the fifth group. move_cancelled runs only in the last
group.
No guest combines isolcpus=domain with a kernel without CONFIG_CPUSETS.
Such a kernel hits a BUG at boot on v7.4-rc2 with isolcpus=domain,0,1.
The BUG is in select_fallback_rq() called from workqueue_init(). I will
report it separately.
On v7.4-rc2, 35 cases of deadline_bw_test fail in 8 guests of 8 vCPUs,
as patch 40 lists. overrun_caught fails in part of the runs. The three
cases of deadline_rebuild_test fail there too, and so do
throttled_round_trip, asleep_round_trip and sysrq_normalize of
deadline_switch_test. tip sched/urgent, the base of this series, gives
the same results.
Each case added for a fix fails without that fix. Each fix names the
cases that test it, and patch 40 lists the rest. There are four
exceptions:
- deboost_blocked_keeps_order passes on v7.4-rc2, and fails on v5.
- return_asleep_new_params passes everywhere. It covers the wakeup
path that patches 35 and 36 copy.
- deboost_keeps_own_runtime passes with and without the flag of patch
11. It checks that a task keeps the runtime of its own reservation.
- pi_chain_donor_leaves passes on v7.4-rc2. There, a blocked task that
leaves SCHED_DEADLINE keeps its parameters. It fails on v6.
sysrq_running_owner passes on v6 too. It locks the guest up once the
early return of patch 8 is disabled. Two cases pass on v7.4-rc2 and add
coverage only: frozen_moved_switched and pi_chain_two_owners. So do the
affinity changes in random_mix.
The three sysrq cases only run with DEADLINE_TEST_SYSRQ=1, which the
19 guests set. move_normalized is one of them.
For patches 13, 15 and 22, a kernel built with the series up to that
patch confirms the claim of its message. Each of the 40 patches builds
alone. The whole series also builds without CONFIG_CPUSETS, with
PREEMPT_RT and with CONFIG_RT_GROUP_SCHED. kernel/sched/ also builds
without CONFIG_RT_MUTEXES, which v6 broke. The cpuset selftest
test_cpuset_prs.sh passes, with and without PREEMPT_RT.
Not tested: cgroup v1 cpusets, threaded cgroups (every move is a write
to cgroup.procs), the sched_ext server, and CPUs of unequal capacity.
How this was developed
======================
An LLM coding assistant wrote these patches, their messages and the
tests, under my direction and review. I know the concepts behind
SCHED_DEADLINE, but not its implementation well. So I ask those who
know this code for a careful reading of the fixes. A reproducer that
stops failing shows that the symptom goes away. It does not show that
the fix is the one you would have written.
The bugs came from:
- flaky cgroup selftests of a feature I am developing;
- the review of Juri's four fixes, and his review of v5;
- building deadline_bw_test, and reading the code;
- the Sashiko AI reviews of v1 to v6. Each finding was checked against
the code first.
Changes since v6
================
- The base is unchanged: tip sched/urgent ec6f6731538e.
- Fixes from the Sashiko review of v6. Three are new patches, for bugs
that upstream has:
- New patch 9: the rtmutex chain walk stopped at a waiter whose keys
did not change. The owner further down could then keep a donor
that exits. pi_chain_donor_leaves tests it.
- New patch 12: a task throttled as it blocked, then boosted while it
slept, would never be replenished. It would never run again. This
was found by reading.
- New patch 34: a constrained task given a shorter dl_deadline could
keep more runtime than its new dl_runtime.
change_constrained_new_params and wake_constrained_new_params test
it.
- Fixes to v6 patches, from the same review:
- 8: the donor's runtime, deadline and period are all tested. A
negative runtime is reset to 0. An owner with no parameters of its
own gets a deadline of now.
- 11: it builds without CONFIG_RT_MUTEXES. fork() does not copy
dl_donor_refill.
- 23: after a race with a cpuset update, sched_setaffinity() sets the
mask a second time. That second change waits for an attach too.
- 25: the v2 fast path clears the reservation record. The span of the
default root domain leaves out offline CPUs. move_back_cpu_offline
tests it.
- 26: with admission control off, a task put outside its mask by the
offline fallback is counted where it runs.
- 27: no bandwidth moves to a CPU that is not active.
- 28: the offline migration of a throttled task stays in its root
domain. The span stays known while the recorded CPU goes offline.
hotplug_spinner_keeps, hotplug_waking_keeps and
hotplug_isolated_keeps test it.
- 29: the fallback CPUs follow the affinity set by the user. A dead
task does not block the offline. hotplug_user_mask_refused and
hotplug_dead_accepted test it.
- 30: the new masks are computed from the top down. A scratch mask no
longer overwrites effective_xcpus. An invalid partition gives no
CPUs back. A mask that grows within the user affinity is checked.
Four new cases test it.
- 33: a new donor of a blocked task with no parameters starts a
period.
- 36: a boosted task keeps the mark of its parameter change until an
enqueue after its deboost. change_boosted_new_params tests it.
- Patches 27 and 28: no bandwidth moves and no span is used while the
sched domains are rebuilt. A partition root made and removed while a
task woke up often could move its bandwidth to the isolated root
domain. rebuild_waking_keeps tests it. With isolcpus=domain,0,1, it
fails in most runs without the fix.
- Patch 27: proxy execution can migrate a task blocked on a mutex. Its
bandwidth now stays in the root domain that counts it. This came up
in the review of the proxy execution RFC.
- The claims made "on v7.4-rc1" were run again on v7.4-rc2 and on tip
sched/urgent. They all hold, and the messages now say v7.4-rc2.
Patch 40 lists 35 failing cases, not 32.
- The selftest guards of the hotplug cases are fixed. affinity_vs_setattr
waits for its target to sleep, so its target no longer overruns.
Not changed:
- A departed task that priority inheritance boosts back into the class
is counted once across a rebuild. The rebuild counts it where it
runs. If the deboost races the rebuild, its bandwidth is dropped
until its 0-lag time. Nothing is counted twice.
- At a deboost, the 0-lag time of a task comes from the deadline and
runtime it ran on during the boost, as upstream.
- sched_setattr() still fails with EPERM for a task moved asleep to
another partition. The task is still on a CPU of the old root domain,
and the affinity check of __sched_setscheduler() reads that one. The
check is upstream code, and it refuses but never admits wrongly.
- An attach whose mask keeps a CPU of the task's root domain reserves
nothing. The task then stays in that root domain.
- A write to cpuset.cpus that makes a partition root invalid is not
refused. A deadline task can then run in a root domain that did not
admit it, as upstream. CPU hotplug and a write of "member" do the same
and cannot fail. The next rebuild counts the task where it runs.
Based on tip sched/urgent, ec6f6731538e (v7.4-rc2).
v6: https://lore.kernel.org/all/20261116190549.230119-1-mathieu.desnoyers@efficios.com/
v5: https://lore.kernel.org/all/20261108185406.1549770-1-mathieu.desnoyers@efficios.com/
v4: https://lore.kernel.org/all/20261107014451.1487825-1-mathieu.desnoyers@efficios.com/
Andrea Parri (1):
sched/deadline: Check SCHED_DEADLINE affinity under the rq lock
Juri Lelli (1):
sched/deadline: Reject SCHED_DEADLINE on exited tasks
Mathieu Desnoyers (38):
sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or
the class
sched/deadline: Do not admit bandwidth for parameters that are kept
sched: Let a task boosted by a deadline waiter change its policy
sched/deadline: Do not divide by zero active CPUs of a root domain
sched/deadline: Refuse dl_server debugfs writes while the rq is
offline
sched: Take cpuset_mutex for a deadline policy change only with pi
sched/deadline: Do not lose a throttled task switched out and back
sched/deadline: Do not replenish a boost from cleared parameters
rtmutex: Continue the chain walk when a waiter's deadline donor
changes
sched/deadline: Reset the deadline and the runtime with the parameters
sched/deadline: Reset the donor's runtime at a deboost in the class
sched/deadline: Replenish a throttled task boosted while asleep
sched/deadline: Release the bandwidth of a task that leaves the class
asleep
sched/deadline: Keep departing bandwidth across a root domain rebuild
sched/deadline: Fix this_bw when a task is changed during a remote
wakeup
sched/deadline: Rebuild the root domains' accounting from the task
list
sched/deadline: Record the root domain holding a task's bandwidth
sched/deadline: Leave a dying task out of a rebuild of the accounting
sched/deadline: Move a task's bandwidth only where room was made
cgroup/cpuset: Wait for an attach before rebuilding the sched domains
sched: Make deadline policy and affinity changes wait for a cpuset
attach
cgroup/cpuset: Give back the bandwidth an attach did not move
cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to
sched/deadline: Make a rebuild count a task in the root domain it runs
in
sched/deadline: Move a task's bandwidth to the root domain it runs in
sched/deadline: Keep a task in the root domain that counts its
bandwidth
sched/deadline: Keep the last CPU a task has in its root domain
cgroup/cpuset: Refuse to take a deadline task out of its root domain
sched/deadline: Cancel the replenishment timer with the parameters
sched/deadline: Count the bandwidth a boosted task is given
sched/deadline: Start a period for a task that enters the class
blocked
sched/deadline: Cap the runtime left of a constrained task at
dl_runtime
sched/deadline: Check the runtime of a task that comes back runnable
sched/deadline: Check the runtime of a task whose parameters change
sched/debug: Show the deadline bandwidth of each runqueue
selftests/sched: Add a test of SCHED_DEADLINE policy round trips
selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain
rebuild
selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting
include/linux/cpuset.h | 20 +-
include/linux/sched.h | 43 +
include/linux/sched/deadline.h | 39 +-
kernel/cgroup/cgroup.c | 3 -
kernel/cgroup/cpuset-internal.h | 15 +-
kernel/cgroup/cpuset.c | 549 +-
kernel/locking/rtmutex.c | 22 +-
kernel/locking/rtmutex_api.c | 2 +-
kernel/locking/rtmutex_common.h | 3 +
kernel/sched/core.c | 126 +-
kernel/sched/deadline.c | 1350 ++-
kernel/sched/debug.c | 12 +-
kernel/sched/sched.h | 21 +
kernel/sched/syscalls.c | 177 +-
kernel/sched/topology.c | 11 +
tools/testing/selftests/Makefile | 1 +
.../selftests/sched/deadline/.gitignore | 3 +
.../testing/selftests/sched/deadline/Makefile | 8 +
tools/testing/selftests/sched/deadline/config | 5 +
.../sched/deadline/deadline_bw_test.c | 9597 +++++++++++++++++
.../sched/deadline/deadline_rebuild_test.c | 497 +
.../sched/deadline/deadline_switch_test.c | 1089 ++
.../testing/selftests/sched/deadline/settings | 1 +
23 files changed, 13282 insertions(+), 312 deletions(-)
create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
create mode 100644 tools/testing/selftests/sched/deadline/Makefile
create mode 100644 tools/testing/selftests/sched/deadline/config
create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c
create mode 100644 tools/testing/selftests/sched/deadline/settings
base-commit: ec6f6731538e0d45872eb279fdeb477bed152d6f
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 01/40] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 02/40] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
` (38 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A sched_setattr() request with SCHED_FLAG_KEEP_PARAMS can ask for
another policy. __sched_setscheduler() then applies only half of the
change, and returns 0:
- SCHED_OTHER to SCHED_BATCH leaves the policy as it was.
- SCHED_DEADLINE to SCHED_OTHER sets DEQUEUE_CLASS. switched_from_dl()
then runs on a task that stays in the class. It would release
bandwidth the task still uses.
- SCHED_DEADLINE on a task of another policy reserves bandwidth in
sched_dl_overflow(). The task keeps its policy and a dl_bw of 0, so
nothing releases that bandwidth. For a task boosted into the deadline
class, it is not even a class change.
SCHED_FLAG_KEEP_PARAMS came with utilization clamping. With
SCHED_FLAG_KEEP_POLICY, it lets sched_setattr() change only the clamps
of a task. So __sched_setscheduler() skips __setscheduler_params(),
which sets p->policy, and the sched_class assignment. Nothing checks
that the request keeps the policy.
In testing on v7.4-rc2, a deadline waiter boosted a SCHED_OTHER mutex
owner with a 40ms slice. One such call on the owner left 0.4 CPU in
total_bw, until the next rebuild of the root domains.
Fix this by refusing a request for another policy or class with
EINVAL. Move sched_dl_overflow() after it, as the last check. Nothing
gives back the bandwidth it reserves. A refused request also leaves
sched_reset_on_fork alone.
With this patch, that call is refused and total_bw does not change. The
params_keep_boosted case of deadline_bw_test, added later in this
series, tests it.
Juri Lelli found the class change, from a report by Tejun Heo. The
Sashiko AI review of the first posting of this series found the policy
change and the boosted case.
Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-3-dd761fe005b4@redhat.com/
---
kernel/sched/syscalls.c | 40 +++++++++++++++++++++++++++++-----------
1 file changed, 29 insertions(+), 11 deletions(-)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 0ec401af80d1..89b0d5787f45 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -647,17 +647,6 @@ int __sched_setscheduler(struct task_struct *p,
goto recheck;
}
- /*
- * If setscheduling to SCHED_DEADLINE (or changing the parameters
- * of a SCHED_DEADLINE task) we need to check if enough bandwidth
- * is available.
- */
- if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
- retval = -EBUSY;
- goto unlock;
- }
-
- p->sched_reset_on_fork = reset_on_fork;
oldprio = p->prio;
newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
@@ -677,9 +666,38 @@ int __sched_setscheduler(struct task_struct *p,
prev_class = p->sched_class;
next_class = __setscheduler_class(policy, newprio);
+ /*
+ * With SCHED_FLAG_KEEP_PARAMS, the change below skips
+ * __setscheduler_params() and the sched_class assignment. The
+ * policy and the class stay as they are. Refuse a request for
+ * another policy or class, rather than carry out half of it.
+ *
+ * The class depends on the priority, which priority inheritance
+ * can raise. It also depends on whether a BPF scheduler is in
+ * charge. So test the class too, not only the policy.
+ */
+ if ((attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+ (policy != p->policy || prev_class != next_class)) {
+ retval = -EINVAL;
+ goto unlock;
+ }
+
if (prev_class != next_class)
queue_flags |= DEQUEUE_CLASS;
+ /*
+ * If setscheduling to SCHED_DEADLINE (or changing the parameters
+ * of a SCHED_DEADLINE task) we need to check if enough bandwidth
+ * is available. Keep this check last. If a later check failed,
+ * nothing would give back the bandwidth it reserves.
+ */
+ if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+ retval = -EBUSY;
+ goto unlock;
+ }
+
+ p->sched_reset_on_fork = reset_on_fork;
+
scx_prepare_setscheduler(p, policy);
scoped_guard (sched_change, p, queue_flags) {
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 02/40] sched/deadline: Do not admit bandwidth for parameters that are kept
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 01/40] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 03/40] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
` (37 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
__sched_setscheduler() runs the deadline admission test on a request
with SCHED_FLAG_KEEP_PARAMS. But the task keeps its parameters: the
call skips __setscheduler_params().
Commit f4dddf90d58d ("sched: Skip priority checks with
SCHED_FLAG_KEEP_PARAMS") made sys_sched_setattr() copy the task's own
parameters into such a request. Then the priority checks pass.
get_params() copies them without a lock.
Another call can change the task's parameters before
__sched_setscheduler() takes the runqueue lock. The request is then
stale, and dl_param_changed() takes it for a change. sched_dl_overflow()
would put the stale bandwidth in place of the task's own in total_bw,
and in this_bw for a sleeping task. But the task keeps its own dl_bw.
Its release at the 0-lag time takes that dl_bw, and the difference
stays behind.
Fix this by skipping sched_dl_overflow() for SCHED_FLAG_KEEP_PARAMS.
There is nothing to admit for parameters that are kept.
Found by reading, after the Sashiko AI review of the first posting of
this series raised the boosted case of the previous patch. Not
reproduced: the parameter change must land inside that window.
Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/syscalls.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 89b0d5787f45..6f8ce5fd94a1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -690,8 +690,16 @@ int __sched_setscheduler(struct task_struct *p,
* of a SCHED_DEADLINE task) we need to check if enough bandwidth
* is available. Keep this check last. If a later check failed,
* nothing would give back the bandwidth it reserves.
+ *
+ * With SCHED_FLAG_KEEP_PARAMS, the task keeps its parameters, so
+ * there is nothing to admit. Such a call changes other attributes,
+ * such as the utilization clamps. A deadline task does not use its
+ * clamps, but keeps them for when it leaves the class. The
+ * parameters in attr may also be stale: sys_sched_setattr() read
+ * them from the task without a lock.
*/
- if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+ if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+ (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
retval = -EBUSY;
goto unlock;
}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 03/40] sched: Let a task boosted by a deadline waiter change its policy
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 01/40] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 02/40] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 04/40] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
` (36 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A SCHED_OTHER or SCHED_FIFO task that holds an rt_mutex can be boosted
into the deadline class by a deadline waiter. Then a sched_setattr() or
sched_setscheduler() call on it fails with EBUSY, unless it asks for
SCHED_DEADLINE. sysrq-n would not reset it either.
The admission test runs "if setscheduling to SCHED_DEADLINE (or
changing the parameters of a SCHED_DEADLINE task)", as its comment
says. sched_dl_overflow() then handles three cases, by the task's
policy: the task enters, leaves or stays in SCHED_DEADLINE. But the
call site tests dl_task(p). That tests the priority, so a boosted task
passes it. For such a task and another policy, sched_dl_overflow()
matches no case and returns -1.
Fix this by testing the policy, with task_has_dl_policy(). A boosted
task has no bandwidth of its own to admit or release.
Now sysrq-n can take a boosted task out of the class.
normalize_rt_tasks() calls __sched_setscheduler() without pi, so the
task gets its normal priority. Nothing then drops the boost. At the
unlock, rt_mutex_setprio() sees no change of priority. The pi_se of the
task would keep pointing to the waiter, which can exit. So also drop the
boost in __setscheduler_dl_pi() when the task leaves the class, as
rt_mutex_setprio() does.
The boosted_owner_policy case of deadline_bw_test, added later in this
series, fails with EBUSY on v7.4-rc2. It passes with this series. The
sysrq_boosted_owner case of deadline_switch_test tests the sysrq-n path.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/syscalls.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 6f8ce5fd94a1..cb115aa9b0b3 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -304,6 +304,13 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
scope->flags |= ENQUEUE_REPLENISH;
}
}
+
+ /*
+ * Without pi, as from normalize_rt_tasks(), a boosted task can leave
+ * the class. Drop its boost then, as rt_mutex_setprio() does.
+ */
+ if (!dl_prio(newprio))
+ p->dl.pi_se = &p->dl;
}
#else /* !CONFIG_RT_MUTEXES */
static inline void __setscheduler_dl_pi(int newprio, int policy,
@@ -691,6 +698,10 @@ int __sched_setscheduler(struct task_struct *p,
* is available. Keep this check last. If a later check failed,
* nothing would give back the bandwidth it reserves.
*
+ * Test the policy, not the priority. A task boosted into the class
+ * by priority inheritance has no bandwidth of its own to admit or
+ * release.
+ *
* With SCHED_FLAG_KEEP_PARAMS, the task keeps its parameters, so
* there is nothing to admit. Such a call changes other attributes,
* such as the utilization clamps. A deadline task does not use its
@@ -699,7 +710,8 @@ int __sched_setscheduler(struct task_struct *p,
* them from the task without a lock.
*/
if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
- (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+ (dl_policy(policy) || task_has_dl_policy(p)) &&
+ sched_dl_overflow(p, policy, attr)) {
retval = -EBUSY;
goto unlock;
}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 04/40] sched/deadline: Do not divide by zero active CPUs of a root domain
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (2 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 03/40] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
` (35 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
__dl_sub() and __dl_add() divide by their cpus argument. Most callers
pass dl_bw_cpus(), the number of active CPUs of a root domain. That
number can be 0, and the division then traps.
Commit daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl
tasks") added extra_bw, the bandwidth that deadline tasks can reclaim
on a runqueue. A change of a root domain's total_bw is spread over the
extra_bw of its active CPUs.
During hotplug, a CPU sits in the default root domain, which can then
have no active CPU. This lasts from the rebuild in
sched_cpu_deactivate() to the offline, and from the online to
sched_cpu_activate(). A debugfs write to the runtime or period of its
fair or ext server divides there. A write of the current value passes
admission in the window. In 8-CPU VMs of v7.4-rc2, a loop of such writes
oopses at the first offline and online:
Oops: divide error: 0000 [#1] SMP NOPTI
RIP: 0010:dl_server_apply_params+0x335/0x380
Call Trace:
sched_server_write_common.isra.0+0x1c0/0x260
full_proxy_write+0x58/0x90
vfs_write+0xcb/0x550
By reading, two other paths would divide by zero:
- inactive_task_timer(), for a task that left SCHED_DEADLINE or died,
if its CPU went offline.
- sched_setattr(SCHED_DEADLINE) on a task asleep on an offline CPU,
with admission control off.
Fix this by changing total_bw alone when the root domain has no active
CPU.
With this patch, the reproducer survives 20 rounds with no warning.
Fixes: daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index dfae08625342..b614d5fc1a1a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
}
}
+/*
+ * Spread a change of total_bw over the extra_bw of the active CPUs of the
+ * root domain. @cpus is their number. With no active CPU, as can happen
+ * during CPU hotplug, change total_bw alone.
+ */
static inline
void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
{
dl_b->total_bw -= tsk_bw;
- __dl_update(dl_b, (s32)tsk_bw / cpus);
+ if (cpus)
+ __dl_update(dl_b, (s32)tsk_bw / cpus);
}
static inline
void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
{
dl_b->total_bw += tsk_bw;
- __dl_update(dl_b, -((s32)tsk_bw / cpus));
+ if (cpus)
+ __dl_update(dl_b, -((s32)tsk_bw / cpus));
}
static inline bool
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (3 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 04/40] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:38 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 06/40] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
` (34 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit 4043f5498416 ("sched/deadline: Reject debugfs dl_server writes
for offline CPUs") refuses a write to a dl_server's runtime or period
on an offline CPU. Its message says that such a write has no meaningful
semantics.
It tests cpu_online(). But during hotplug the CPU is still online while
it sits in the default root domain. This lasts from the rebuild in
sched_cpu_deactivate() to the offline, and from the online to
sched_cpu_activate(). That root domain does not count the server's
bandwidth. A write in the window still goes through admission. With
admission control off, it changes total_bw there. On v7.4-rc2, a write
of the current value divides by zero in that window. The previous patch
removes the division, not the write.
Fix this by testing rq->online instead. It is set under the runqueue
lock when the CPU joins a root domain that counts it, after
cpu_active() is set.
A write now fails with EBUSY while the CPU goes offline or comes
online, as a write on an offline CPU already does. Writes at any other
time are unchanged.
Applied without the previous patch, this patch alone also stops the
oops. The reproducer survives 20 rounds with no warning, in 3 VMs of 3.
Fixes: d741f297bcea ("sched/fair: Fair server interface")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/debug.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..e0d6cd3bd607 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -410,7 +410,15 @@ static ssize_t sched_server_write_common(struct file *filp, const char __user *u
return -EINVAL;
}
- if (!cpu_online(cpu_of(rq)))
+ /*
+ * Refuse the write while the runqueue is offline. This
+ * includes the windows where the CPU goes offline or comes
+ * online. The root domain then does not count the server's
+ * bandwidth, and may have no active CPU. rq->online is set
+ * under this lock when the CPU joins a root domain that
+ * counts it. cpu_active() is set before that.
+ */
+ if (!rq->online)
return -EBUSY;
update_rq_clock(rq);
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 06/40] sched: Take cpuset_mutex for a deadline policy change only with pi
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (4 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 07/40] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
` (33 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. On success, it releases it only when pi is set.
normalize_rt_tasks(), for sysrq-n, is the one caller with pi unset. It
runs under read_lock(&tasklist_lock). So sysrq-n takes the mutex in
atomic context for a deadline task, and never releases it. Every later
change that involves SCHED_DEADLINE then blocks forever.
Commit 710da3c8ea7d ("sched/core: Prevent race condition between cpuset
and __sched_setscheduler()") took the cpuset lock only when pi is set.
normalize_rt_tasks() cannot take it under tasklist_lock. Its changelog
accepts this: after sysrq-n, the deadline guarantees are gone anyway.
Commit 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex") then took
the mutex only for deadline tasks. It replaced the pi test by a policy
test, and left the release under pi.
Fix this by taking the mutex for a deadline change only when pi is
set.
The policy test reads p->policy before the runqueue lock. It is checked
again under the lock only when the caller gave no policy. Otherwise,
the task can enter SCHED_DEADLINE in between. The call then takes it
out of the class without the mutex. A rebuild of the root domain
accounting could run at the same time. So check again under the
runqueue lock, and retry with the mutex.
sysrq-n on v7.4-rc2, with a deadline task and lockdep on:
sysrq: Nice All RT Tasks
BUG: sleeping function called from invalid context at kernel/locking/mutex.c:623
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 4356, name: deadline_switch
locks held by deadline_switch/4356: 4, on CPU#3:
...
#3: ffffffff83006098 (tasklist_lock){.+.+}-{3:3}, at: normalize_rt_tasks+0x34/0x150
A sched_setattr(SCHED_DEADLINE) after it never returned, in 8 VMs of 8.
The sysrq_normalize case of deadline_switch_test, added later in this
series, tests this with DEADLINE_TEST_SYSRQ=1. It passes with this
series.
Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/syscalls.c | 26 +++++++++++++++++++++++---
1 file changed, 23 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index cb115aa9b0b3..9b04e77297ae 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -561,9 +561,11 @@ int __sched_setscheduler(struct task_struct *p,
/*
* SCHED_DEADLINE bandwidth accounting relies on stable cpusets
- * information.
+ * information. Do not take cpuset_mutex when pi is unset: the
+ * caller is normalize_rt_tasks(), which holds tasklist_lock and
+ * cannot sleep.
*/
- if (dl_policy(policy) || dl_policy(p->policy)) {
+ if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
cpuset_lock();
}
@@ -643,7 +645,13 @@ int __sched_setscheduler(struct task_struct *p,
}
}
- /* Re-check policy now with rq lock held: */
+ /*
+ * The caller gave no policy (SETPARAM_POLICY): the request was
+ * derived from an unlocked read of p->policy above. If the task
+ * changed policy since, re-derive the request from scratch, and
+ * release cpuset_mutex: the re-derived request re-decides whether
+ * it is needed.
+ */
if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
policy = oldpolicy = -1;
task_rq_unlock(rq, p, &rf);
@@ -654,6 +662,18 @@ int __sched_setscheduler(struct task_struct *p,
goto recheck;
}
+ /*
+ * The caller gave an explicit policy, so the request needs no
+ * re-derivation (oldpolicy is -1 above). But the decision to
+ * take cpuset_mutex used the same unlocked read: the task
+ * entered SCHED_DEADLINE since, and the change below would take
+ * it out of the class. Retry to take the mutex.
+ */
+ if (unlikely(pi && !cpuset_locked && task_has_dl_policy(p))) {
+ task_rq_unlock(rq, p, &rf);
+ goto recheck;
+ }
+
oldprio = p->prio;
newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 07/40] sched/deadline: Do not lose a throttled task switched out and back
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (5 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 06/40] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 08/40] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
` (32 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A SCHED_DEADLINE task can be throttled, switched to another policy, and
switched back before its 0-lag time. It may then never run again, not
even to die of a fatal signal.
A task that leaves SCHED_DEADLINE keeps its parameters until its 0-lag
time, so that leaving and coming back gains nothing. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
their reset from the replenishment timer to the inactive timer, which
switched_to_dl() cancels.
This misses a task that was throttled when it left. An overrun can put
its 0-lag time after its replenishment timer. That timer fires while
the task is out of the class, and does nothing. On the return of the
task, enqueue_dl_entity() leaves it to that timer, which has fired. The
task is then on no dl_rq, with no timer pending, until it leaves
SCHED_DEADLINE again.
Fix this by replenishing the task when the timer finds it out of the
class, as for a throttled task that is not queued. Its return then
enqueues it.
Clearing dl_throttled alone would not do. It would forgive the overrun.
A yield left set, or the old deadline of a constrained task, would
throttle the task again. The Sashiko AI review of the first posting
found this in the first version of this patch.
A cgroup selftest under development hung on this bug about once in
1350 runs. The throttled_round_trip case of deadline_switch_test, added
later in this series, fails 8 times out of 8 on v7.3 and v7.4-rc2. It
passes with this patch.
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b614d5fc1a1a..e4d7cbf4861d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1233,9 +1233,18 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
/*
* The task might have changed its scheduling policy to something
* different than SCHED_DEADLINE (through switched_from_dl()).
+ * It keeps its parameters until its 0-lag time. If it comes back
+ * before then, nothing else would lift its throttle. So replenish
+ * it now, as the !queued case below does. Its return enqueues it.
*/
- if (!dl_task(p))
+ if (!dl_task(p)) {
+ if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
+ sched_clock_tick();
+ update_rq_clock(rq);
+ replenish_dl_entity(dl_se);
+ }
goto unlock;
+ }
/*
* The task might have been boosted by someone else and might be in the
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 08/40] sched/deadline: Do not replenish a boost from cleared parameters
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (6 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 07/40] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
` (31 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
When a SCHED_DEADLINE waiter boosts the owner of an rt_mutex, the owner
runs on the waiter's parameters, through dl.pi_se. Commit 2d3d891d3344
("sched/deadline: Add SCHED_DEADLINE inheritance logic") calls this a
temporary solution. The owner is never throttled. When its runtime runs
out, replenish_dl_entity() refills it at once from the waiter's
dl_runtime.
The waiter can leave SCHED_DEADLINE before the chain walk deboosts the
owner. Its parameters can be cleared in between, at its 0-lag time.
replenish_dl_entity() then loops forever on a dl_runtime of 0. It
holds the rq lock with interrupts off, so the owner's CPU locks up.
O is a SCHED_OTHER task that holds the rt_mutex L. W is a SCHED_DEADLINE
task. T is any other task:
CPU 0 (O) CPU 1 (W) CPU 2 (T)
locks L
blocks on L
rt_mutex_setprio(O, W)
O->dl.pi_se = &W->dl
sleeps
task_non_contending()
arms W's inactive timer
sched_setscheduler(W)
W leaves SCHED_DEADLINE
task_rq_unlock(W)
<W's inactive timer>
inactive_task_timer()
__dl_clear_params()
W->dl.dl_runtime = 0
<tick>
update_curr_dl_se()
enqueue_task_dl()
replenish_dl_entity()
runtime += W->dl.dl_runtime
loops forever
rt_mutex_adjust_pi(W)
rt_mutex_setprio(O, W)
spins on O's rq lock
Fix this by reading the inherited runtime, deadline and period once, and
by not refilling from cleared ones. They are cleared under the waiter's
rq lock, not the owner's. So test all three. The owner stays runnable.
It gets no runtime. A negative runtime is reset to 0: the owner's own
reservation does not pay for the boost. An owner with no parameters of
its own gets a deadline of now. A deadline of 0 would mark its dl_rq as
empty. The wakeup of a boosted task already gives that deadline when its
donor is cleared. The owner runs until the chain walk deboosts it, or
until it releases the lock. That also covers sysrq-n:
normalize_rt_tasks() changes the waiter without a chain walk. The
warning stays for an entity that is not boosted.
The reset bounds the overrun. But a runtime of 0 still counts as
exceeded. So until the deboost, update_curr_dl_se() still requeues the
owner at each update. An owner with SCHED_FLAG_DL_OVERRUN still gets
SIGXCPU.
With a 300ms debug delay before rt_mutex_adjust_pi(), v7.4-rc2 locks
up in 8-CPU VMs. A kernel with this patch does not.
The Sashiko AI review of the third posting found the lockup. Juri Lelli
suggested this fix in place of a larger one.
Fixes: a5e7be3b28a2 ("sched/deadline: Clear dl_entity params when setscheduling to different class")
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/avL8oOtjKqVNR7-l@jlelli-thinkpadt14gen4.remote.csb/
---
kernel/sched/deadline.c | 40 +++++++++++++++++++++++++++++++++++-----
1 file changed, 35 insertions(+), 5 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index e4d7cbf4861d..71b1c155c242 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -807,8 +807,38 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
{
struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
struct rq *rq = rq_of_dl_rq(dl_rq);
+ struct sched_dl_entity *pi_se = pi_of(dl_se);
+ u64 runtime = READ_ONCE(pi_se->dl_runtime);
+ u64 deadline = READ_ONCE(pi_se->dl_deadline);
+ u64 period = READ_ONCE(pi_se->dl_period);
- WARN_ON_ONCE(pi_of(dl_se)->dl_runtime <= 0);
+ /*
+ * A boosted entity runs on the parameters of its donor, the top
+ * waiter of an rt_mutex. The donor can leave the class and have them
+ * cleared before the chain walk deboosts this entity. They are
+ * cleared under the donor's rq lock, which can be another one. So
+ * read each of them once, and check all three. Refilling from a
+ * runtime of 0 would loop forever. Leave the entity runnable
+ * instead: the walk deboosts it, or its task releases the lock.
+ *
+ * Add no runtime. Do not keep a negative one: the overrun comes
+ * from the boost, and the entity's own reservation does not pay for
+ * it. A task with no parameters of its own gets a deadline of now,
+ * as a donor's relative deadline of 0 would give. A deadline of 0
+ * means an empty dl_rq to inc_dl_deadline().
+ */
+ if (unlikely(!runtime || !deadline || !period)) {
+ WARN_ON_ONCE(!is_dl_boosted(dl_se));
+ if (!dl_se->dl_deadline) {
+ dl_se->deadline = rq_clock(rq);
+ dl_se->runtime = 0;
+ }
+ if (dl_se->runtime < 0 || dl_se->dl_yielded)
+ dl_se->runtime = 0;
+ dl_se->dl_yielded = 0;
+ dl_se->dl_throttled = 0;
+ return;
+ }
/*
* This could be the case for a !-dl task that is boosted.
@@ -822,8 +852,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
*/
if (dl_se->dl_deadline == 0 ||
(dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
- dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
- dl_se->runtime = pi_of(dl_se)->dl_runtime;
+ dl_se->deadline = rq_clock(rq) + deadline;
+ dl_se->runtime = runtime;
}
if (dl_se->dl_yielded && dl_se->runtime > 0)
@@ -836,8 +866,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
* arbitrary large.
*/
while (dl_se->runtime <= 0) {
- dl_se->deadline += pi_of(dl_se)->dl_period;
- dl_se->runtime += pi_of(dl_se)->dl_runtime;
+ dl_se->deadline += period;
+ dl_se->runtime += runtime;
}
/*
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (7 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 08/40] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:50 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
` (30 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task boosted by a SCHED_DEADLINE waiter runs on the parameters of a
donor, dl.pi_se. Commit 2279f540ea7d ("sched/deadline: Fix priority
inheritance with multiple scheduling classes") made the donor follow the
chain. rt_mutex_setprio() copies the donor of the top waiter to the
owner. So the owner at the end of a chain points at the deadline task at
its head. That task waits on no lock the owner holds.
The chain walk stops at a task whose waiter has the same sort keys as
the task: the priority and the deadline. Then "there is no further
priority adjustment necessary". Before 2279f540ea7d, this was enough.
The refill read the parameters of the owner's own top waiter, which is
blocked and so alive.
But a boost can change the donor of a blocked task and leave its keys
as they were. The task keeps a deadline priority. It sleeps, so no
refill moves its deadline. The walk then stops one lock too early.
M and O are SCHED_OTHER tasks. W and Z are SCHED_DEADLINE tasks. Z has
the earlier deadline. L1, L1' and L2 are rt_mutexes:
O M W Z
locks L2 locks L1 and L1'
blocks on L1'
rt_mutex_setprio(M, W)
M->dl.pi_se = &W->dl
blocks on L2
rt_mutex_setprio(O, M)
O->dl.pi_se = &W->dl
blocks on L1
rt_mutex_setprio(M, Z)
M->dl.pi_se = &Z->dl
chain walk at M:
keys equal, stops
times out on L1'
exits
O runs on W's parameters, with a later deadline than Z's, while Z waits
for it. After W exits, O's next refill would read W's freed task_struct.
A deboost has the same effect. Take a deadline task that a waiter with
an earlier deadline boosts, and that blocks on a second lock. It keeps
its own deadline. When that waiter leaves, rt_mutex_setprio() makes the
task its own donor again. Its keys stay equal. A policy change in
__setscheduler_dl_pi() can also change the donor of a blocked task with
no change of keys. rt_mutex_adjust_pi() then skips the walk.
Setting up the deadline tasks needs CAP_SYS_NICE. The timeout of a
FUTEX_LOCK_PI needs no privilege. The Sashiko AI review of the sixth
posting of this series found the deboost case.
Fix this by recording the donor in the waiter with the sort keys, in
waiter_update_prio(). rt_waiter_up_to_date() compares both. The walk and
rt_mutex_adjust_pi() use it. A new donor then takes the walk on to the
next owner, and rt_mutex_setprio() gives that owner the new donor. The
waiter's copy is only compared, never dereferenced. A task without a
deadline priority is always its own donor, so its walks do not change.
A known limit remains. Two walks can overlap on a chain of three locks
or more. The owners further down then keep the old donor until the
first walk reaches them.
The case pi_chain_donor_leaves of deadline_bw_test, added later in this
series, tests the missed boost. W leaves SCHED_DEADLINE there, in place
of exiting. Without this patch, another deadline task gets no CPU time.
On v7.4-rc2 the case passes. There, a blocked task that leaves
SCHED_DEADLINE keeps its parameters, so O runs on W's.
Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org # v5.15+
---
kernel/locking/rtmutex.c | 22 ++++++++++++++++++++--
kernel/locking/rtmutex_api.c | 2 +-
kernel/locking/rtmutex_common.h | 3 +++
3 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 5a9534c715b8..c78dfe9c93fc 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -369,10 +369,12 @@ static __always_inline void
waiter_update_prio(struct rt_mutex_waiter *waiter, struct task_struct *task)
{
lockdep_assert_held(&waiter->lock->wait_lock);
+ lockdep_assert_held(&task->pi_lock);
lockdep_assert(RB_EMPTY_NODE(&waiter->tree.entry));
waiter->tree.prio = __waiter_prio(task);
waiter->tree.deadline = task->dl.deadline;
+ waiter->pi_se = task->dl.pi_se;
}
/*
@@ -433,6 +435,21 @@ static __always_inline int rt_waiter_node_equal(struct rt_waiter_node *left,
return 1;
}
+/*
+ * Return true if @waiter is up to date with its task @task. Compare the
+ * sort keys and the deadline donor, which rt_mutex_setprio() copies from
+ * @task to the lock owner. A boost or a deboost of @task can change its
+ * donor and leave its keys as they were. The owner must then get the new
+ * donor: the old one can exit. The caller holds @task->pi_lock, which
+ * protects @task->dl.pi_se. @waiter->pi_se is never dereferenced.
+ */
+static __always_inline bool
+rt_waiter_up_to_date(struct rt_mutex_waiter *waiter, struct task_struct *task)
+{
+ return rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task)) &&
+ waiter->pi_se == task->dl.pi_se;
+}
+
static inline bool rt_mutex_steal(struct rt_mutex_waiter *waiter,
struct rt_mutex_waiter *top_waiter)
{
@@ -824,12 +841,13 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
/*
* If the waiter priority is the same as the task priority
- * then there is no further priority adjustment necessary. If
+ * then there is no further priority adjustment necessary. The
+ * task must also pass on the same deadline donor. If
* deadlock detection is off, we stop the chain walk. If its
* enabled we continue, but stop the requeueing in the chain
* walk.
*/
- if (rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+ if (rt_waiter_up_to_date(waiter, task)) {
if (!detect_deadlock)
goto out_unlock_pi;
else
diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c
index eb18b094473c..6bdbd36bac94 100644
--- a/kernel/locking/rtmutex_api.c
+++ b/kernel/locking/rtmutex_api.c
@@ -510,7 +510,7 @@ void __sched rt_mutex_adjust_pi(struct task_struct *task)
raw_spin_lock_irqsave(&task->pi_lock, flags);
waiter = task->pi_blocked_on;
- if (!waiter || rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+ if (!waiter || rt_waiter_up_to_date(waiter, task)) {
raw_spin_unlock_irqrestore(&task->pi_lock, flags);
return;
}
diff --git a/kernel/locking/rtmutex_common.h b/kernel/locking/rtmutex_common.h
index c38b7bdea7b3..c0f391c352e3 100644
--- a/kernel/locking/rtmutex_common.h
+++ b/kernel/locking/rtmutex_common.h
@@ -43,6 +43,8 @@ struct rt_waiter_node {
* @pi_tree: node to enqueue into the mutex owner waiters tree
* @task: task reference to the blocked task
* @lock: Pointer to the rt_mutex on which the waiter blocks
+ * @pi_se: The deadline donor of @task when @tree was last updated.
+ * Only compared, never dereferenced.
* @wake_state: Wakeup state to use (TASK_NORMAL or TASK_RTLOCK_WAIT)
* @ww_ctx: WW context pointer
*
@@ -54,6 +56,7 @@ struct rt_mutex_waiter {
struct rt_waiter_node pi_tree;
struct task_struct *task;
struct rt_mutex_base *lock;
+ struct sched_dl_entity *pi_se;
unsigned int wake_state;
struct ww_acquire_ctx *ww_ctx;
};
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (8 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
` (29 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task that leaves SCHED_DEADLINE keeps its parameters until its 0-lag
time, so that leaving and coming back gains nothing. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
their reset to that time. Before it, the replenishment timer did the
reset at the deadline. So a task back after the reset found its
deadline passed, and started a new period. But __dl_clear_params()
resets neither the deadline nor the runtime.
Now a task back after its 0-lag time can be enqueued on its old
deadline and runtime. With runtime left, its 0-lag time comes before
its deadline. Back before that deadline, it would first spend the runtime
of the reservation it gave back. Back after it, a constrained task
would be throttled until its old period ends. "sched/deadline: Do not
lose a throttled task switched out and back" moves the deadline of a
throttled task on. This widens the window before the deadline to about
a period.
A task with no parameters of its own, boosted by a deadline waiter,
keeps the donor's deadline and runtime after its deboost. If it then
enters SCHED_DEADLINE before that deadline, it runs on the donor's
runtime. On v7.4-rc2, a task of 10ms per 100ms ran 40ms of its first
period.
Fix this by resetting the deadline and the runtime in
__dl_clear_params(), and in switched_from_dl() at the deboost of a task
with no parameters. The next enqueue then starts a new period.
The Sashiko AI review found the first case in the second posting of
this series, and the deboost in the third. The deboost_resets_period
case of deadline_bw_test, added later in this series, tests the
deboost.
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 71b1c155c242..d6d2657dcddd 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3569,6 +3569,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
if (task_on_rq_queued(p) && p->dl.dl_runtime)
task_non_contending(&p->dl, false);
+ /*
+ * A task with no parameters of its own was boosted into the class.
+ * Its deadline and runtime are the donor's. Reset them, so that its
+ * next entry into the class starts a new period.
+ */
+ if (!p->dl.dl_runtime) {
+ p->dl.deadline = 0;
+ p->dl.runtime = 0;
+ }
+
/*
* In case a task is setscheduled out from SCHED_DEADLINE we need to
* keep track of that on its cpuset (for correct bandwidth tracking).
@@ -3984,7 +3994,8 @@ bool __checkparam_dl(const struct sched_attr *attr)
}
/*
- * This function clears the sched_dl_entity static params.
+ * Clear the static parameters of @dl_se. Also reset its deadline and its
+ * runtime, so that its next enqueue starts a new period.
*/
static void __dl_clear_params(struct sched_dl_entity *dl_se)
{
@@ -3995,6 +4006,9 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_bw = 0;
dl_se->dl_density = 0;
+ dl_se->deadline = 0;
+ dl_se->runtime = 0;
+
dl_se->dl_throttled = 0;
dl_se->dl_yielded = 0;
dl_se->dl_non_contending = 0;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (9 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
` (28 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task boosted by a deadline waiter, its donor, runs on the donor's
reservation. Commit 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE
inheritance logic") uses the waiter's parameters "during the whole
period it holds the pi-lock".
A boosted task that runs out of runtime is not throttled.
replenish_dl_entity() refills it from the waiter's parameters: it adds
the waiter's runtime, and moves the deadline by the waiter's period.
From then on, the runtime and the deadline of the task are not those of
its own reservation.
A task that leaves the class at its deboost does not keep them.
"sched/deadline: Reset the deadline and the runtime with the parameters"
resets them for a task with no parameters. For a task with parameters,
they are cleared with the parameters at its 0-lag time.
But the boosted task can be a SCHED_DEADLINE task itself, when the
waiter has an earlier deadline. It stays in the class at its deboost.
rt_mutex_setprio() only resets dl.pi_se. enqueue_dl_entity() keeps a
deadline that is ahead. So a task that was refilled keeps the waiter's
runtime and deadline after the unlock. A task that sched_setattr() makes
SCHED_DEADLINE while boosted ends its boost the same way.
In one period, the task can run what the waiter left on top of its own
runtime. Other deadline tasks can miss deadlines.
Fix this by recording that refill in a new flag, dl_donor_refill. If
the boost ends in the class with the flag set, start a new period at
the deboost. The next enqueue is too late for a task blocked on another
rt_mutex: the chain walk copies its deadline into its waiter first.
The task can still hold a lock that another deadline task waits on. The
new deadline can be later than that waiter's. rt_mutex_setprio() then
boosts the task again.
A task that was not refilled keeps its runtime and its deadline. They
are those of its own reservation. A new period there would give it a
full runtime at every contended unlock.
The Sashiko AI review of the fourth posting found the sched_setattr()
case. Four cases of deadline_bw_test, added later in this series, test
this fix: deboost_in_class_resets_period, deboost_blocked_keeps_order,
deboost_after_donor_refill and deboost_keeps_own_runtime.
In deboost_after_donor_refill, an owner of 5ms every 100ms runs 8ms with
the lock. Its donor is a task of 2ms every 10ms. A deboost that tests
the runtime against dl_runtime, in place of the flag, misses this
refill: it leaves less than dl_runtime. The owner then held less than
1ms after the unlock, in 3 guests of 3. With the flag, it holds its own
5ms.
Fixes: 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance logic")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 6 ++++++
kernel/sched/core.c | 16 ++++++++++++---
kernel/sched/deadline.c | 43 +++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 10 ++++++++++
kernel/sched/syscalls.c | 2 +-
5 files changed, 73 insertions(+), 4 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..96b498f79484 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -683,6 +683,11 @@ struct sched_dl_entity {
* @dl_overrun tells if the task asked to be informed about runtime
* overruns.
*
+ * @dl_donor_refill tells if a boosted task had its runtime refilled
+ * from the parameters of its donor. Its runtime and deadline are
+ * then not those of its own reservation. The end of the boost
+ * clears it.
+ *
* @dl_server tells if this is a server entity.
*
* @dl_server_active tells if the dlserver is active(started).
@@ -711,6 +716,7 @@ struct sched_dl_entity {
unsigned int dl_yielded : 1;
unsigned int dl_non_contending : 1;
unsigned int dl_overrun : 1;
+ unsigned int dl_donor_refill : 1;
unsigned int dl_server : 1;
unsigned int dl_server_active : 1;
unsigned int dl_defer : 1;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5ed660f099a5..a22df8b43a1a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7834,16 +7834,26 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
} else {
- p->dl.pi_se = &p->dl;
+ dl_deboost(&p->dl);
+ /*
+ * dl_deboost() can start a new period. Its
+ * deadline can then be after the top waiter's.
+ * Boost the task again then.
+ */
+ if (pi_task && dl_prio(pi_task->prio) &&
+ dl_entity_preempt(&pi_task->dl, &p->dl)) {
+ p->dl.pi_se = pi_task->dl.pi_se;
+ scope->flags |= ENQUEUE_REPLENISH;
+ }
}
} else if (rt_prio(prio)) {
if (dl_prio(oldprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
if (oldprio < prio)
scope->flags |= ENQUEUE_HEAD;
} else {
if (dl_prio(oldprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
if (rt_prio(oldprio))
p->rt.timeout = 0;
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index d6d2657dcddd..989e7e7d3f8f 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -93,12 +93,48 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return pi_of(dl_se) != dl_se;
}
+
+/*
+ * Record that @dl_se had its runtime refilled from the parameters of its
+ * donor, if it is boosted.
+ */
+static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
+{
+ if (is_dl_boosted(dl_se))
+ dl_se->dl_donor_refill = 1;
+}
+
+/*
+ * End the boost of @dl_se, whose task stays in the class. If the boost
+ * refilled its runtime from the donor's parameters, start a new period
+ * now. The next enqueue is too late for a task blocked on another
+ * rt_mutex: the chain walk copies its deadline into its waiter node
+ * first. Without a refill, the task keeps the runtime and the deadline
+ * of its own reservation.
+ *
+ * The caller holds the rq lock and the pi_lock of the task, and has
+ * updated the rq clock.
+ */
+void dl_deboost(struct sched_dl_entity *dl_se)
+{
+ struct rq *rq = rq_of_dl_se(dl_se);
+
+ if (dl_se->dl_donor_refill) {
+ dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
+ dl_se->runtime = dl_se->dl_runtime;
+ }
+ dl_drop_boost(dl_se);
+}
#else /* !CONFIG_RT_MUTEXES: */
static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
{
return dl_se;
}
+static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
+{
+}
+
static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return false;
@@ -734,6 +770,7 @@ static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
/* for non-boosted task, pi_of(dl_se) == dl_se */
dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
dl_se->runtime = pi_of(dl_se)->dl_runtime;
+ dl_note_donor_refill(dl_se);
/*
* If it is a deferred reservation, and the server
@@ -832,6 +869,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
if (!dl_se->dl_deadline) {
dl_se->deadline = rq_clock(rq);
dl_se->runtime = 0;
+ dl_note_donor_refill(dl_se);
}
if (dl_se->runtime < 0 || dl_se->dl_yielded)
dl_se->runtime = 0;
@@ -854,11 +892,15 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
(dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
dl_se->deadline = rq_clock(rq) + deadline;
dl_se->runtime = runtime;
+ dl_note_donor_refill(dl_se);
}
if (dl_se->dl_yielded && dl_se->runtime > 0)
dl_se->runtime = 0;
+ if (dl_se->runtime <= 0)
+ dl_note_donor_refill(dl_se);
+
/*
* We keep moving the deadline away until we get some
* available runtime for the entity. This ensures correct
@@ -4013,6 +4055,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_yielded = 0;
dl_se->dl_non_contending = 0;
dl_se->dl_overrun = 0;
+ dl_se->dl_donor_refill = 0;
dl_se->dl_server = 0;
dl_se->dl_defer = 0;
dl_se->dl_defer_running = 0;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 92134ce326a4..d87296abec4c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -356,6 +356,16 @@ extern int sched_dl_global_validate(void);
extern void sched_dl_do_global(void);
extern int sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+#ifdef CONFIG_RT_MUTEXES
+extern void dl_deboost(struct sched_dl_entity *dl_se);
+
+/* End the boost of @dl_se. The caller holds the rq lock of its task. */
+static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
+{
+ dl_se->pi_se = dl_se;
+ dl_se->dl_donor_refill = 0;
+}
+#endif
extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
extern bool __checkparam_dl(const struct sched_attr *attr);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 9b04e77297ae..f346176f1ba1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -310,7 +310,7 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
* the class. Drop its boost then, as rt_mutex_setprio() does.
*/
if (!dl_prio(newprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
}
#else /* !CONFIG_RT_MUTEXES */
static inline void __setscheduler_dl_pi(int newprio, int policy,
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (10 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 13/40] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
` (27 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A deadline task can be throttled as it blocks, with an rt_mutex held.
Its replenishment timer is then armed. A deadline waiter can boost the
task while it sleeps.
Commit feff2e65efd8 ("sched/deadline: Unthrottle PI boosted threads
while enqueuing") lets a boost override the throttle. enqueue_task_dl()
lifts the throttle of a boosted task, and dl_task_timer() ignores a
boosted task. This covers a task that is enqueued while boosted. But a
boost of a sleeping task does not enqueue it.
T is a SCHED_DEADLINE task that holds the rt_mutex L. W is a
SCHED_DEADLINE task with an earlier deadline:
T W T's timer
uses up its runtime, sleeps
dl_throttled = 1
start_dl_timer()
pthread_mutex_timedlock(L)
rt_mutex_setprio(T, W)
T sleeps: no enqueue
dl_task_timer()
T is boosted: returns
times out
rt_mutex_setprio(T, NULL)
wakes up
enqueue_dl_entity()
dl_throttled: waits for the timer
The timer has already fired. T would never run again, not even to die
of SIGKILL. Every later waiter on L would hang too. This was found by
reading.
Fix this by returning early in dl_task_timer() only for a boosted task
that is queued. A boosted task that sleeps then takes the existing path
for a throttled task that is not queued: it is replenished.
The refill reads the parameters of the donor, from the timer. So this
relies on two earlier patches. "sched/deadline: Do not replenish a boost
from cleared parameters" keeps the refill from looping on cleared
parameters. "rtmutex: Continue the chain walk when a waiter's deadline
donor changes" keeps the donor pointer from going stale. After a deboost in
the class, "sched/deadline: Reset the donor's runtime at a deboost in
the class" starts a period of the task's own reservation.
The Sashiko AI review of the sixth posting of this series found this.
Fixes: feff2e65efd8 ("sched/deadline: Unthrottle PI boosted threads while enqueuing")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 989e7e7d3f8f..2b5a2a43addf 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1319,10 +1319,13 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
}
/*
- * The task might have been boosted by someone else and might be in the
- * boosting/deboosting path, its not throttled.
+ * A boost overrides the throttle. enqueue_task_dl() lifts it when the
+ * boost enqueues the task, or when the task wakes up boosted. So a
+ * queued task needs nothing here. A boost of a sleeping task does not
+ * enqueue it, and it can end before the wakeup. Nothing would lift the
+ * throttle then. So let a sleeping task reach the !queued case below.
*/
- if (is_dl_boosted(dl_se))
+ if (is_dl_boosted(dl_se) && task_on_rq_queued(p))
goto unlock;
/*
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 13/40] sched/deadline: Release the bandwidth of a task that leaves the class asleep
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (11 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 14/40] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
` (26 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task that leaves SCHED_DEADLINE while it is not queued can keep its
bandwidth in total_bw until the next rebuild of the root domains.
Repeated, this fills total_bw up to the admission limit. Then
sched_setattr(SCHED_DEADLINE) and CPU offlining fail with EBUSY.
A task that leaves the class keeps its bandwidth until its 0-lag time.
Releasing it earlier would allow temporary overloads. For a queued task,
switched_from_dl() calls task_non_contending(). It releases the
bandwidth if the 0-lag time is past. Otherwise it arms the inactive
timer, which releases it then. For a task that is not queued,
switched_from_dl() relies on the timer armed when the task blocked.
But the timer is not always pending when the task leaves the class:
- The task blocked after its 0-lag time. The timer was not armed.
- The timer fired while the task was still in the class. It only
removed running_bw.
- The task was woken up on another CPU and is not enqueued there yet.
The wakeup cancelled the timer.
- The task came back to the class asleep. switched_to_dl() cancelled
the timer.
Then nothing releases total_bw.
Fix this in switched_from_dl(). For a task that is not queued and has
no timer pending, call task_non_contending(), as for a queued task. It
arms the timer if the 0-lag time is ahead. Otherwise it releases the
bandwidth now and clears the parameters, as inactive_task_timer() does.
Clearing the parameters keeps later paths from releasing the bandwidth
again:
- A timer handler that already started finds a dl_bw of 0.
- The deboost of a later priority inheritance boost finds no
parameters.
An owner that the task boosts as a waiter can find the parameters
cleared too. "sched/deadline: Do not replenish a boost from cleared
parameters" handles this, so a backport needs it first. Without it, the
owner's CPU can lock up as soon as the waiter leaves the class.
Juri Lelli's first fix, since dropped from tip, released the bandwidth
here without clearing dl_bw. The bandwidth of a boosted task was then
released twice.
Juri Lelli suggested the call to task_non_contending() in review.
The Sashiko AI review of the third posting found the two cases with the
0-lag time ahead. deadline_bw_test, added later in this series, tests
this in pi_former_owner_asleep and deboost_released_resets_period.
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/
---
kernel/sched/deadline.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 2b5a2a43addf..67cb1b1abbb8 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3640,6 +3640,14 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
if (p->dl.dl_non_contending)
sub_running_bw(&p->dl, &rq->dl);
sub_rq_bw(&p->dl, &rq->dl);
+ /*
+ * Without an armed timer, nothing releases total_bw. Do as for
+ * a queued task: arm the timer if the 0-lag time is ahead, or
+ * release the bandwidth and clear the parameters now. Call
+ * task_non_contending() after sub_rq_bw(), which needs dl_bw.
+ */
+ if (!p->dl.dl_non_contending)
+ task_non_contending(&p->dl, false);
}
/*
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 14/40] sched/deadline: Reject SCHED_DEADLINE on exited tasks
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (12 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 13/40] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 15/40] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
` (25 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
From: Juri Lelli <juri.lelli@redhat.com>
sched_setattr(SCHED_DEADLINE) on a zombie task succeeds and charges
total_bw via sched_dl_overflow(), but the task has already done its
final schedule() and will never run again. Nothing releases the reserved
bandwidth. Repeating this fills total_bw to the admission cap, after
which sched_setattr(SCHED_DEADLINE) and CPU offline fail with -EBUSY.
Reject sched_setattr(SCHED_DEADLINE) on exited tasks with -ESRCH.
A task of the class that exited, switched to another policy, runs
switched_from_dl(), which decrements the count of deadline tasks of its
cpuset that cgroup_task_dead() decremented already: at 0 with a deadline
task left in it, the cpuset is skipped by the next rebuild of the root
domains' accounting, and that task's bandwidth with it. Reject that as
well. The deadline_bw_test selftest added later in this series has the
case, death_zombie_switched: on v7.4-rc1 the switch is accepted, and
the one-CPU partition root the zombie ran in admits 9 tasks of 10% after
a rebuild, with a task of 30% still in it, where 6 are due.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/3f6e6c87-645b-45fd-8fc6-8da5bc5b82d8@gmail.com/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-4-dd761fe005b4@redhat.com/
Cc: stable@vger.kernel.org
Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
[mathieu: reject a task of the class leaving it too, found by the
Sashiko AI review of the patch]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/syscalls.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index f346176f1ba1..8c72e751ff4e 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -592,6 +592,19 @@ int __sched_setscheduler(struct task_struct *p,
if (retval)
goto unlock;
+ /*
+ * Reject SCHED_DEADLINE on exited tasks. A zombie will never be
+ * scheduled again, so sched_dl_overflow() would charge total_bw
+ * that nothing will ever release. And reject a change of policy of
+ * one in the class: cgroup_task_dead() took it off the count of
+ * deadline tasks of its cpuset, which switched_from_dl() would
+ * decrement a second time.
+ */
+ if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
+ retval = -ESRCH;
+ goto unlock;
+ }
+
/*
* If not changing anything there's no need to proceed further,
* but store a possible modification of reset_on_fork.
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 15/40] sched/deadline: Keep departing bandwidth across a root domain rebuild
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (13 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 14/40] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 16/40] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
` (24 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth in
total_bw until its 0-lag time. This keeps admission control on the safe
side. Then inactive_task_timer() removes it from the root domain of the
task's CPU. Two things break this:
- A rebuild of the accounting clears every root domain, then adds back
the deadline tasks of the cpusets. It misses a task that left
SCHED_DEADLINE, or a dead one. Their bandwidth is dropped, and the
timer removes it again.
- A task that left SCHED_DEADLINE can also move to another root
domain. The timer then takes total_bw from that one, which never
had it.
Either way, a root domain admits deadline tasks beyond its capacity
until the next rebuild.
Fix this with dl_departed_list. A departing task goes on it with the
CPU it is on. A rebuild puts its bandwidth back in that CPU's root
domain. The timer removes it from there, and so does
sched_dl_overflow() for a task that returns to SCHED_DEADLINE.
Priority inheritance can boost a departed task back into the class,
after it moved to another root domain. The boost cancels the timer and
takes the task off the list. Keep the CPU recorded then. When the boost
ends, the bandwidth is taken from the root domain that counts it, not
from the one the task runs in. The pi_departed_moved case of
deadline_bw_test, added later in this series, tests this.
A rebuild still drops the bandwidth of a task whose CPU is inactive, or
that dies during the rebuild. Each task records a count of the rebuilds,
dl_bw_gen, with its bandwidth. The timer removes nothing if a rebuild
dropped the bandwidth since.
Make dl_cookie and dl_bw_visited() static: only deadline.c uses them.
deadline_rebuild_test, added later in this series, fails every time on
v7.3 and v7.4-rc2 kernels. A one-CPU partition root admits two tasks of
600ms every second there, where it has room for one. It passes with
this patch.
Fixes: 387e31300b57 ("sched/deadline: Fix the update of the total -deadline utilization")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 16 +++
include/linux/sched/deadline.h | 5 +-
kernel/cgroup/cpuset.c | 11 +-
kernel/sched/deadline.c | 254 +++++++++++++++++++++++++++++----
4 files changed, 249 insertions(+), 37 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 96b498f79484..063e49690658 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -740,6 +740,22 @@ struct sched_dl_entity {
*/
struct hrtimer inactive_timer;
+ /*
+ * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
+ * time" keeps its bandwidth in its root domain until the inactive
+ * timer removes it. Meanwhile, @dl_departed_node links the task on
+ * the list that a rebuild of the root domains' accounting reads.
+ * @dl_departed_cpu is a CPU of the root domain that counts the
+ * bandwidth. It stays recorded while priority inheritance boosts the
+ * task back into the class. Otherwise @dl_departed_cpu is -1.
+ *
+ * @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
+ * counted. If a rebuild dropped it since, there is nothing to remove.
+ */
+ struct list_head dl_departed_node;
+ int dl_departed_cpu;
+ unsigned int dl_bw_gen;
+
/*
* Bits for DL-server functionality. Also see the comment near
* dl_server_update().
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..bdf39ac1d43d 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -32,7 +32,7 @@ static inline bool dl_time_before(u64 a, u64 b)
struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
-extern void dl_clear_root_domain_cpu(int cpu);
+extern void dl_clear_root_domains(void);
/*
* Return whether moving DL task @p to @new_mask requires moving DL
* bandwidth accounting between root domains. This helper is specific to
@@ -43,9 +43,6 @@ extern void dl_clear_root_domain_cpu(int cpu);
extern bool dl_task_needs_bw_move(struct task_struct *p,
const struct cpumask *new_mask);
-extern u64 dl_cookie;
-extern bool dl_bw_visited(int cpu, u64 cookie);
-
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
return dl_se->dl_server;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..4159abfbbbb5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -985,21 +985,14 @@ void dl_rebuild_rd_accounting(void)
{
struct cpuset *cs = NULL;
struct cgroup_subsys_state *pos_css;
- int cpu;
- u64 cookie = ++dl_cookie;
lockdep_assert_cpuset_lock_held();
lockdep_assert_cpus_held();
lockdep_assert_held(&sched_domains_mutex);
- rcu_read_lock();
-
- for_each_possible_cpu(cpu) {
- if (dl_bw_visited(cpu, cookie))
- continue;
+ dl_clear_root_domains();
- dl_clear_root_domain_cpu(cpu);
- }
+ rcu_read_lock();
cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 67cb1b1abbb8..ddc92cf719ac 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -199,7 +199,13 @@ static inline unsigned long dl_bw_capacity(int i)
}
}
-bool dl_bw_visited(int cpu, u64 cookie)
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+static u64 dl_cookie;
+
+static bool dl_bw_visited(int cpu, u64 cookie)
{
struct root_domain *rd = cpu_rq(cpu)->rd;
@@ -253,6 +259,112 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
}
+/*
+ * A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
+ * its bandwidth in total_bw until that time. Then the inactive timer
+ * removes it. Meanwhile the task may run anywhere, under another policy.
+ * The timer holds a reference to it.
+ *
+ * A rebuild of the root domains' accounting (dl_clear_root_domains() and
+ * dl_add_task_root_domain()) does not find such a task among the deadline
+ * tasks of the cpusets. So dl_departed_list holds these tasks, from their
+ * departure to their timer. The rebuild puts their bandwidth back in the
+ * root domain of the CPU each one was on when it left.
+ *
+ * dl_bw_gen numbers the rebuilds of the accounting: how many times the
+ * root domains were wiped and reconstructed (dl_clear_root_domains()
+ * increments it). A task records the number of its last definite
+ * state, counted or settled. That number dates the state:
+ *
+ * - Current: the bandwidth is counted, or a release settled it since
+ * the rebuild began. A release subtracts; the walk of the cpusets
+ * skips.
+ * - One behind: the clear that just ran wiped it. The rebuild puts
+ * it back.
+ * - Older: an earlier rebuild dropped it. A task on the departed list
+ * stays out, and its release removes nothing. A task that is in the
+ * class is counted again.
+ *
+ * Only equalities are compared, so the wrap of the counter is
+ * harmless.
+ *
+ * A rebuild drops the bandwidth when the CPU recorded is no longer
+ * active. It also drops the bandwidth of a task that dies while the
+ * rebuild runs. The cpuset walk skips that task, and it was not dead yet
+ * when the list was put back.
+ *
+ * dl_departed_lock protects the list and both counts. Take it with
+ * interrupts disabled, under the task's rq lock or pi_lock or in the
+ * rebuild, and before any dl_bw lock.
+ */
+static LIST_HEAD(dl_departed_list);
+static DEFINE_RAW_SPINLOCK(dl_departed_lock);
+static unsigned int dl_bw_gen;
+
+/*
+ * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
+ * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
+ * recorded if @p has one: priority inheritance boosted @p back into the
+ * class and its boost ends. The caller holds the rq lock of @p.
+ */
+static void dl_task_depart(struct task_struct *p, int cpu)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (!list_empty(&dl_se->dl_departed_node))
+ return;
+ if (dl_se->dl_departed_cpu < 0)
+ dl_se->dl_departed_cpu = cpu;
+ list_add(&dl_se->dl_departed_node, &dl_departed_list);
+}
+
+/*
+ * Take @p off dl_departed_list. Priority inheritance boosts @p back into
+ * the deadline class. Keep the CPU recorded: its root domain still counts
+ * the bandwidth, wherever @p runs now. The caller holds the rq lock of
+ * @p.
+ */
+static void dl_task_undepart(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ list_del_init(&dl_se->dl_departed_node);
+}
+
+/*
+ * Remove the bandwidth of @p from the root domain that counts it. That is
+ * the root domain of the CPU recorded when @p departed, or else of @cpu.
+ * Clear the record. Take @p off dl_departed_list. Remove nothing if a
+ * rebuild dropped the bandwidth since it was counted. Then record the
+ * current dl_bw_gen in @p, so that a rebuild in progress does not count it
+ * again. The caller holds the rq lock of @p.
+ */
+static void dl_task_release_bw(struct task_struct *p, int cpu)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+ struct dl_bw *dl_b;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (dl_se->dl_departed_cpu >= 0) {
+ cpu = dl_se->dl_departed_cpu;
+ dl_se->dl_departed_cpu = -1;
+ list_del_init(&dl_se->dl_departed_node);
+ }
+ if (dl_se->dl_bw_gen == dl_bw_gen) {
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ dl_se->dl_bw_gen = dl_bw_gen;
+}
+
static inline
void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
{
@@ -485,13 +597,9 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
sub_running_bw(dl_se, dl_rq);
if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&dl_b->lock);
+ dl_task_release_bw(p, task_cpu(p));
__dl_clear_params(dl_se);
}
}
@@ -500,8 +608,18 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
}
dl_se->dl_non_contending = 1;
- if (!dl_server(dl_se))
- get_task_struct(dl_task_of(dl_se));
+ if (!dl_server(dl_se)) {
+ struct task_struct *p = dl_task_of(dl_se);
+
+ get_task_struct(p);
+ /*
+ * A dying task keeps its bandwidth in the root domain of the
+ * CPU it runs on. Record that CPU. switched_from_dl() records
+ * a task that leaves SCHED_DEADLINE.
+ */
+ if (READ_ONCE(p->__state) == TASK_DEAD)
+ dl_task_depart(p, task_cpu(p));
+ }
hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
}
@@ -2262,17 +2380,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
goto no_task;
if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
dl_se->dl_non_contending = 0;
}
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&dl_b->lock);
+ dl_task_release_bw(p, task_cpu(p));
__dl_clear_params(dl_se);
goto unlock;
@@ -3533,12 +3647,33 @@ void dl_add_task_root_domain(struct task_struct *p)
dl_get_task_effective_cpus(p, msk);
cpu = cpumask_first_and(cpu_active_mask, msk);
BUG_ON(cpu >= nr_cpu_ids);
+
+ raw_spin_lock(&dl_departed_lock);
+ /*
+ * Skip a task whose dl_bw_gen is current. Either
+ * dl_clear_root_domains() put it back as a departed task, and
+ * priority inheritance boosted it into the class since. Or it died
+ * since the rebuild started, and dl_task_release_bw() released its
+ * bandwidth. A task that died since with its 0-lag time ahead is on
+ * the departed list. Count it in the root domain of the CPU
+ * recorded, if that CPU is active.
+ */
+ if (p->dl.dl_bw_gen == dl_bw_gen)
+ goto unlock;
+ if (p->dl.dl_departed_cpu >= 0) {
+ cpu = p->dl.dl_departed_cpu;
+ if (!cpu_active(cpu))
+ goto unlock;
+ }
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
raw_spin_lock(&dl_b->lock);
__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_gen = dl_bw_gen;
+unlock:
+ raw_spin_unlock(&dl_departed_lock);
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
}
@@ -3596,9 +3731,51 @@ void dl_clear_root_domain(struct root_domain *rd)
dl_server_add_bw(rd, i);
}
-void dl_clear_root_domain_cpu(int cpu)
+/*
+ * Clear every root domain before a rebuild of their accounting, and count
+ * the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
+ * back the bandwidth of the tasks on dl_departed_list, which no cpuset
+ * holds as deadline tasks. Put each in the root domain of the CPU
+ * recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ */
+void dl_clear_root_domains(void)
{
- dl_clear_root_domain(cpu_rq(cpu)->rd);
+ struct sched_dl_entity *dl_se;
+ u64 cookie = ++dl_cookie;
+ struct dl_bw *dl_b;
+ int cpu;
+
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ dl_bw_gen++;
+ for_each_possible_cpu(cpu) {
+ if (!dl_bw_visited(cpu, cookie))
+ dl_clear_root_domain(cpu_rq(cpu)->rd);
+ }
+ list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
+ cpu = dl_se->dl_departed_cpu;
+ /*
+ * Skip bandwidth that an earlier rebuild dropped. It is not
+ * counted now, and its release removes nothing
+ * (dl_task_release_bw()).
+ */
+ if (dl_se->dl_bw_gen != dl_bw_gen - 1)
+ continue;
+ /*
+ * Drop the bandwidth of a task whose CPU is inactive. A CPU
+ * that goes offline is in the default root domain by then.
+ * The root domain that counted the task cannot be found
+ * from the recorded number. The release then removes
+ * nothing, and no later rebuild restores it.
+ */
+ if (!cpu_active(cpu))
+ continue;
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ dl_se->dl_bw_gen = dl_bw_gen;
+ }
}
static void switched_from_dl(struct rq *rq, struct task_struct *p)
@@ -3654,9 +3831,20 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
* We cannot use inactive_task_timer() to invoke sub_running_bw()
* at the 0-lag time, because the task could have been migrated
* while SCHED_OTHER in the meanwhile.
+ *
+ * For the same reason, the timer cannot take the task's bandwidth
+ * from the root domain of the CPU the task is on by then. And a
+ * rebuild of the root domains meanwhile would not find the task
+ * among the deadline tasks. So record a CPU of the root domain that
+ * keeps the bandwidth until the timer removes it (dl_task_depart()).
+ * Test dl_non_contending: it tells that the timer is armed and its
+ * handler has not run. hrtimer_active() stays true for a while after
+ * the handler returns.
*/
- if (p->dl.dl_non_contending)
+ if (p->dl.dl_non_contending) {
p->dl.dl_non_contending = 0;
+ dl_task_depart(p, task_cpu(p));
+ }
/*
* Since this might be the only -deadline task on the rq,
@@ -3676,6 +3864,17 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
static void switched_to_dl(struct rq *rq, struct task_struct *p)
{
cancel_inactive_timer(&p->dl);
+ /*
+ * A task on the departed list gets here when priority inheritance
+ * boosts it back into the class before its 0-lag time. (A policy change
+ * took it off the list in sched_dl_overflow().) The inactive timer no
+ * longer removes its bandwidth: the task does when it leaves the class
+ * again. So take it off the list. The task keeps the CPU recorded. It
+ * takes the bandwidth from the root domain of that CPU, even if it
+ * moved to another root domain meanwhile.
+ */
+ if (!list_empty(&p->dl.dl_departed_node))
+ dl_task_undepart(p);
/*
* In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -3790,12 +3989,6 @@ DEFINE_SCHED_CLASS(dl) = {
#endif
};
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
int sched_dl_global_validate(void)
{
u64 runtime = global_rt_runtime();
@@ -3916,8 +4109,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (dl_policy(policy) && !task_has_dl_policy(p) &&
!__dl_overflow(dl_b, cap, 0, new_bw)) {
- if (hrtimer_active(&p->dl.inactive_timer))
- __dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
err = 0;
} else if (dl_policy(policy) && task_has_dl_policy(p) &&
@@ -3943,6 +4134,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
}
raw_spin_unlock(&dl_b->lock);
+ /*
+ * A task that comes back before its 0-lag time still has the
+ * bandwidth it left with. The new bandwidth replaces it, so remove the
+ * old one from the root domain that keeps it (dl_task_release_bw()).
+ * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+ */
+ if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
+ lockdep_assert_cpuset_lock_held();
+ if (p->dl.dl_departed_cpu >= 0)
+ dl_task_release_bw(p, -1);
+ p->dl.dl_bw_gen = dl_bw_gen;
+ }
+
return err;
}
@@ -4083,6 +4287,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
init_dl_task_timer(dl_se);
init_dl_inactive_task_timer(dl_se);
__dl_clear_params(dl_se);
+ INIT_LIST_HEAD(&dl_se->dl_departed_node);
+ dl_se->dl_departed_cpu = -1;
}
bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 16/40] sched/deadline: Fix this_bw when a task is changed during a remote wakeup
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (14 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 15/40] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 17/40] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
` (23 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
The this_bw of a runqueue is the bandwidth of the deadline tasks
assigned to it, running or not. GRUB uses it to compute the inactive
utilization of the runqueue. When a task wakes up on another CPU,
migrate_task_rq_dl() takes its bandwidth off the old runqueue. The
enqueue adds it to the new one.
Between these two steps, no runqueue counts the bandwidth.
try_to_wake_up() can leave the enqueue to the wake list of the new CPU,
and return before it runs. task_cpu() already points at the new CPU. A
change of the task in that window gets this_bw wrong.
In this example, T is a deadline task. S is another task. It changes
the bandwidth of T from "old bw" to "new bw":
CPU 0 CPU 1 CPU 2
T blocks
this_bw of CPU 0 counts T
<timer> try_to_wake_up(T)
set_task_cpu(T, 1)
migrate_task_rq_dl()
sub_rq_bw(): CPU 0 drops T
T on the wake list of CPU 1
return
S: sched_setattr(T)
task_rq_lock(T): CPU 1
T is not queued
dl_change_utilization()
this_bw of CPU 1:
- old bw + new bw
sched_ttwu_pending()
enqueue_task_dl(T)
task_contending()
add_rq_bw():
this_bw of CPU 1:
+ new bw
CPU 1 then counts the new bandwidth twice, less the old one. Repeated,
this makes this_bw drift until it underflows. If T left the class in
that window, switched_from_dl() would take its bandwidth off CPU 1 the
same way. A task that enters the class in that window would be counted
twice, by switched_to_dl() and by the enqueue.
This needs a wakeup that moves the task, so a root domain of more than
one CPU, with busy CPUs. CONFIG_SCHED_PROXY_EXEC=y makes such wakeups
more common. In testing, a task sleeps and wakes in a busy two-CPU
partition root, and another CPU changes its runtime. The kernel warns
within a second in each of 8 VMs:
WARNING: kernel/sched/deadline.c:352 at dl_rq_change_utilization+0x71/0x2e0, CPU#0: wake_race/109
__sub_rq_bw(): this_bw underflow
WARNING: kernel/sched/deadline.c:316 at task_contending+0x244/0x260, CPU#6: swapper/6/0
__add_running_bw(): running_bw overflow
Fix this with a new bit, dl_bw_oncpu, that tells whether a runqueue
counts the bandwidth of the task. add_rq_bw() adds the bandwidth only if
no runqueue counts it. sub_rq_bw() takes it off only if one does.
dl_change_utilization() does nothing while no runqueue counts the
bandwidth. The enqueue adds the new one.
With this patch, none of the 8 VMs warns. deadline_bw_test, added later
in this series, tests this in params_waking and random_mix. The Sashiko
AI review of the first posting found the double count, and a bit that
the first version of this patch left set.
Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 5 +++++
kernel/sched/deadline.c | 39 +++++++++++++++++++++++++++++++++++----
2 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 063e49690658..33e2d9492aa2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -711,6 +711,10 @@ struct sched_dl_entity {
* contributes to the root domain's total_bw. Only meaningful for server
* entities (@dl_server == 1). Allows toggling the reservation on/off
* without losing the configured @dl_runtime/@dl_period.
+ *
+ * @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
+ * @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
+ * runqueue's lock. Not for servers, which keep @dl_bw_attached.
*/
unsigned int dl_throttled : 1;
unsigned int dl_yielded : 1;
@@ -724,6 +728,7 @@ struct sched_dl_entity {
unsigned int dl_defer_running : 1;
unsigned int dl_defer_idle : 1;
unsigned int dl_bw_attached : 1;
+ unsigned int dl_bw_oncpu : 1;
/*
* Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ddc92cf719ac..05be328571e9 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -415,18 +415,35 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
}
+/*
+ * @dl_se->dl_bw_oncpu tells whether the this_bw of a runqueue counts the
+ * bandwidth of @dl_se. add_rq_bw() adds the bandwidth only if no runqueue
+ * counts it. sub_rq_bw() subtracts it only if one does.
+ *
+ * A wakeup on another CPU takes the bandwidth off the old runqueue. The
+ * enqueue on the new CPU adds it back later. A change of the task in
+ * between must not count the bandwidth twice, nor take it off a runqueue
+ * that does not count it.
+ *
+ * The servers track their bandwidth with dl_bw_attached. They call
+ * __add_rq_bw() and __sub_rq_bw() directly.
+ */
static inline
void add_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
{
- if (!dl_entity_is_special(dl_se))
- __add_rq_bw(dl_se->dl_bw, dl_rq);
+ if (dl_entity_is_special(dl_se) || dl_se->dl_bw_oncpu)
+ return;
+ __add_rq_bw(dl_se->dl_bw, dl_rq);
+ dl_se->dl_bw_oncpu = 1;
}
static inline
void sub_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
{
- if (!dl_entity_is_special(dl_se))
- __sub_rq_bw(dl_se->dl_bw, dl_rq);
+ if (dl_entity_is_special(dl_se) || !dl_se->dl_bw_oncpu)
+ return;
+ __sub_rq_bw(dl_se->dl_bw, dl_rq);
+ dl_se->dl_bw_oncpu = 0;
}
static inline
@@ -495,6 +512,14 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
if (task_on_rq_queued(p))
return;
+ /*
+ * The task is waking up on another CPU, and no runqueue counts its
+ * bandwidth. The enqueue adds it, with the new parameters that
+ * __setparam_dl() sets right after this.
+ */
+ if (!p->dl.dl_bw_oncpu)
+ return;
+
dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
}
@@ -2885,6 +2910,11 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
*/
cancel_inactive_timer(&p->dl);
}
+ /*
+ * set_task_cpu() points task_cpu() at the new CPU next. The runqueue
+ * of that CPU counts this bandwidth only once the enqueue there adds
+ * it. Until then, no runqueue counts it, as dl_bw_oncpu now says.
+ */
sub_rq_bw(&p->dl, &rq->dl);
rq_unlock(rq, &rf);
}
@@ -4275,6 +4305,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_defer = 0;
dl_se->dl_defer_running = 0;
dl_se->dl_defer_armed = 0;
+ dl_se->dl_bw_oncpu = 0;
#ifdef CONFIG_RT_MUTEXES
dl_se->pi_se = dl_se;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 17/40] sched/deadline: Rebuild the root domains' accounting from the task list
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (15 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 16/40] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
` (22 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A rebuild of the root domains, as on CPU hotplug, clears their deadline
accounting. dl_rebuild_rd_accounting() then adds back the deadline tasks
of each cpuset. Without CONFIG_CPUSETS, it is a stub. So the first
hotplug drops every deadline task's bandwidth, and their releases wrap
total_bw around. In testing, a root domain admitted 9 tasks of 10ms
every 100ms before a hotplug with deadline sleepers, and none after.
With cpusets, the rebuild skips a cpuset whose nr_deadline_tasks is 0.
This count spares suspend and resume a costly walk of all tasks. An
attach moves it by a tally taken in cpuset_can_attach(). A task that
leaves the class between the tally and the move is taken off the old
cpuset twice. Its count can then reach 0 with a deadline task in it.
Every rebuild would then drop that task's bandwidth.
Fix both by walking the task list, with or without cpusets. The walk
finds every deadline task, so the count and its hooks go. It holds
tasklist_lock for reading, so fork and exit wait meanwhile. It takes
the pi_lock of deadline tasks only. The walk that the count avoided
took the pi_lock of every task. It leaves out an exited task with a
dl_bw of 0. It also leaves out a zombie whose bandwidth an earlier
rebuild dropped, as dl_clear_root_domains() does.
The walk visits every thread, and a suspend or a resume runs one
rebuild for each CPU. So keep one count for the whole system,
dl_nr_bw_tasks: the number of tasks whose dl_bw is not 0.
__setparam_dl() and __dl_clear_params() are the only places that change
the dl_bw of a task, and they update the count. With a count of 0, the
rebuild does not walk the task list, as Juri Lelli suggested.
In testing, a 64-vCPU guest held 10000 threads and no deadline task. A
rebuild took 0.7 to 1.2 ms without the count, and 0.03 ms with it.
Without cpusets, a new dl_accounting_mutex keeps admissions out of a
rebuild, as cpuset_mutex does with them.
Three hotplug cases of deadline_bw_test, added later in this series,
fail without cpusets. With this patch, its five hotplug cases pass, in 4
VMs. Its death_zombie_hotplug case tests the zombie. The Sashiko AI
reviews of v3, and of v4 before posting, found the wrong count and the
exited tasks. Its review of v4 found the zombie.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 9 --
include/linux/sched/deadline.h | 16 ++++
kernel/cgroup/cgroup.c | 3 -
kernel/cgroup/cpuset-internal.h | 6 +-
kernel/cgroup/cpuset.c | 67 +-------------
kernel/sched/deadline.c | 154 ++++++++++++++++++++++++++------
kernel/sched/syscalls.c | 14 +--
kernel/sched/topology.c | 8 ++
8 files changed, 163 insertions(+), 114 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..db1cc2fc0324 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -72,8 +72,6 @@ extern int cpuset_init(void);
extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
extern void cpuset_update_active_cpus(void);
-extern void inc_dl_tasks_cs(struct task_struct *task);
-extern void dec_dl_tasks_cs(struct task_struct *task);
extern void cpuset_lock(void);
extern void cpuset_unlock(void);
extern void lockdep_assert_cpuset_lock_held(void);
@@ -129,7 +127,6 @@ static inline int cpuset_do_page_mem_spread(void)
extern bool current_cpuset_is_being_rebound(void);
-extern void dl_rebuild_rd_accounting(void);
extern void rebuild_sched_domains(void);
extern void cpuset_print_current_mems_allowed(void);
@@ -194,8 +191,6 @@ static inline void cpuset_update_active_cpus(void)
partition_sched_domains(1, NULL, NULL);
}
-static inline void inc_dl_tasks_cs(struct task_struct *task) { }
-static inline void dec_dl_tasks_cs(struct task_struct *task) { }
static inline void cpuset_lock(void) { }
static inline void cpuset_unlock(void) { }
static inline void lockdep_assert_cpuset_lock_held(void) { }
@@ -273,10 +268,6 @@ static inline bool current_cpuset_is_being_rebound(void)
return false;
}
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
-
static inline void rebuild_sched_domains(void)
{
guard(cpus_read_lock)();
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdf39ac1d43d..60be226699bb 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,22 @@ struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
extern void dl_clear_root_domains(void);
+extern void dl_rebuild_rd_accounting(void);
+/*
+ * Keep a rebuild of the root domains' accounting and an admission of
+ * bandwidth from running at the same time. With cpusets, cpuset_mutex
+ * does it, since every rebuild comes from the cpuset code. Without
+ * cpusets, dl_accounting_lock() takes a mutex of its own.
+ */
+#ifdef CONFIG_CPUSETS
+static inline void dl_accounting_lock(void) { }
+static inline void dl_accounting_unlock(void) { }
+static inline void lockdep_assert_dl_accounting_held(void) { }
+#else
+extern void dl_accounting_lock(void);
+extern void dl_accounting_unlock(void);
+extern void lockdep_assert_dl_accounting_held(void);
+#endif
/*
* Return whether moving DL task @p to @new_mask requires moving DL
* bandwidth accounting between root domains. This helper is specific to
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..2cfa770c2dcb 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -7240,9 +7240,6 @@ static void do_cgroup_task_dead(struct task_struct *tsk)
if (thread_group_leader(tsk) && atomic_read(&tsk->signal->live))
list_add_tail(&tsk->cg_list, &cset->dying_tasks);
- if (dl_task(tsk))
- dec_dl_tasks_cs(tsk);
-
WARN_ON_ONCE(cgroup_task_frozen(tsk));
if (unlikely(!(tsk->flags & PF_KTHREAD) &&
test_bit(CGRP_FREEZE, &task_dfl_cgroup(tsk)->flags)))
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 1ba94ffaff9d..bf847953c78d 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,11 +166,7 @@ struct cpuset {
*/
bool remote_partition_updating;
- /*
- * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
- * know when to rebuild associated root domain bandwidth information.
- */
- atomic_t nr_deadline_tasks;
+ /* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
int nr_migrate_dl_tasks;
/* DL bandwidth that needs destination reservation for this attach. */
u64 sum_migrate_dl_bw;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 4159abfbbbb5..7628cf77bdfd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -225,20 +225,6 @@ struct tmpmasks {
cpumask_var_t new_cpus; /* For update_cpumasks_hier() */
};
-void inc_dl_tasks_cs(struct task_struct *p)
-{
- struct cpuset *cs = task_cs(p);
-
- atomic_inc(&cs->nr_deadline_tasks);
-}
-
-void dec_dl_tasks_cs(struct task_struct *p)
-{
- struct cpuset *cs = task_cs(p);
-
- atomic_dec(&cs->nr_deadline_tasks);
-}
-
static inline bool is_partition_valid(const struct cpuset *cs)
{
return cs->partition_root_state > 0;
@@ -965,54 +951,6 @@ static int generate_sched_domains(cpumask_var_t **domains,
return ndoms;
}
-static void dl_update_tasks_root_domain(struct cpuset *cs)
-{
- struct css_task_iter it;
- struct task_struct *task;
-
- if (atomic_read(&cs->nr_deadline_tasks) == 0)
- return;
-
- css_task_iter_start(&cs->css, 0, &it);
-
- while ((task = css_task_iter_next(&it)))
- dl_add_task_root_domain(task);
-
- css_task_iter_end(&it);
-}
-
-void dl_rebuild_rd_accounting(void)
-{
- struct cpuset *cs = NULL;
- struct cgroup_subsys_state *pos_css;
-
- lockdep_assert_cpuset_lock_held();
- lockdep_assert_cpus_held();
- lockdep_assert_held(&sched_domains_mutex);
-
- dl_clear_root_domains();
-
- rcu_read_lock();
-
- cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
-
- if (cpumask_empty(cs->effective_cpus)) {
- pos_css = css_rightmost_descendant(pos_css);
- continue;
- }
-
- css_get(&cs->css);
-
- rcu_read_unlock();
-
- dl_update_tasks_root_domain(cs);
-
- rcu_read_lock();
- css_put(&cs->css);
- }
- rcu_read_unlock();
-}
-
/*
* Rebuild scheduler domains.
*
@@ -3181,9 +3119,7 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
llist_for_each_entry_safe(cs, next, lnode, attach_node) {
init_llist_node(&cs->attach_node);
if (cs->nr_migrate_dl_tasks) {
- if (!cancel)
- atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
- else if (cs->dl_bw_cpu >= 0) /* && cancel */
+ if (cancel && cs->dl_bw_cpu >= 0)
dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
cs->nr_migrate_dl_tasks = 0;
cs->sum_migrate_dl_bw = 0;
@@ -3272,7 +3208,6 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* contribute to sum_migrate_dl_bw.
*/
cs->nr_migrate_dl_tasks++;
- oldcs->nr_migrate_dl_tasks--;
if (dl_task_needs_bw_move(task, cs->effective_cpus))
cs->sum_migrate_dl_bw += task->dl.dl_bw;
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 05be328571e9..8e8c62c574d5 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -266,10 +266,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* The timer holds a reference to it.
*
* A rebuild of the root domains' accounting (dl_clear_root_domains() and
- * dl_add_task_root_domain()) does not find such a task among the deadline
- * tasks of the cpusets. So dl_departed_list holds these tasks, from their
- * departure to their timer. The rebuild puts their bandwidth back in the
- * root domain of the CPU each one was on when it left.
+ * dl_add_task_root_domain()) does not count such a task as a deadline
+ * task. So dl_departed_list holds these tasks, from their departure to
+ * their timer. The rebuild puts their bandwidth back in the root domain
+ * of the CPU each one was on when it left.
*
* dl_bw_gen numbers the rebuilds of the accounting: how many times the
* root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -277,7 +277,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* state, counted or settled. That number dates the state:
*
* - Current: the bandwidth is counted, or a release settled it since
- * the rebuild began. A release subtracts; the walk of the cpusets
+ * the rebuild began. A release subtracts; the walk of the task list
* skips.
* - One behind: the clear that just ran wiped it. The rebuild puts
* it back.
@@ -290,8 +290,8 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
*
* A rebuild drops the bandwidth when the CPU recorded is no longer
* active. It also drops the bandwidth of a task that dies while the
- * rebuild runs. The cpuset walk skips that task, and it was not dead yet
- * when the list was put back.
+ * rebuild runs. The walk of the task list does not find that task, and
+ * it was not dead yet when the list was put back.
*
* dl_departed_lock protects the list and both counts. Take it with
* interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -301,6 +301,22 @@ static LIST_HEAD(dl_departed_list);
static DEFINE_RAW_SPINLOCK(dl_departed_lock);
static unsigned int dl_bw_gen;
+/*
+ * The number of tasks with a bandwidth: those whose dl_bw is not 0. A
+ * special entity and a server are not counted. __setparam_dl() and
+ * __dl_clear_params() are the only places that change the dl_bw of a
+ * task, and they update the count. With a count of 0, a rebuild of the
+ * accounting has no task to count, and does not walk the task list.
+ */
+static atomic_t dl_nr_bw_tasks;
+
+/* Return true if @dl_se is a task that holds a bandwidth. */
+static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
+{
+ return dl_se->dl_bw && !dl_se->dl_server &&
+ !(dl_se->flags & SCHED_FLAG_SUGOV);
+}
+
/*
* Put @p on dl_departed_list with @cpu, a CPU of the root domain that
* keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
@@ -3658,7 +3674,7 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
cpuset_cpus_allowed_locked(p, cpus);
}
-/* The caller should hold cpuset_mutex */
+/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
void dl_add_task_root_domain(struct task_struct *p)
{
struct rq_flags rf;
@@ -3668,7 +3684,13 @@ void dl_add_task_root_domain(struct task_struct *p)
struct cpumask *msk;
raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
- if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
+ /*
+ * Count nothing for a special entity, or for a task with no
+ * bandwidth. The walk of the task list also finds exited tasks. An
+ * exited task whose bandwidth went with its parameters has a dl_bw
+ * of 0.
+ */
+ if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return;
}
@@ -3691,6 +3713,14 @@ void dl_add_task_root_domain(struct task_struct *p)
if (p->dl.dl_bw_gen == dl_bw_gen)
goto unlock;
if (p->dl.dl_departed_cpu >= 0) {
+ /*
+ * The walk also finds a dead task that departed before an
+ * earlier rebuild, if it is not reaped yet. If that rebuild
+ * dropped its bandwidth, dl_clear_root_domains() skipped it,
+ * and its release removes nothing. Skip it here too.
+ */
+ if (p->dl.dl_bw_gen != dl_bw_gen - 1)
+ goto unlock;
cpu = p->dl.dl_departed_cpu;
if (!cpu_active(cpu))
goto unlock;
@@ -3761,13 +3791,80 @@ void dl_clear_root_domain(struct root_domain *rd)
dl_server_add_bw(rd, i);
}
+#ifndef CONFIG_CPUSETS
+/*
+ * With cpusets, cpuset_mutex keeps an admission of bandwidth from running
+ * against a rebuild of the accounting. __sched_setscheduler() takes it for
+ * every change of or into SCHED_DEADLINE. The callers of
+ * partition_sched_domains() hold it from the attachment of the new root
+ * domains to the rebuild. Without cpusets, cpuset_lock() is a stub, and
+ * this mutex takes its place in both.
+ */
+static DEFINE_MUTEX(dl_accounting_mutex);
+
+void dl_accounting_lock(void)
+{
+ mutex_lock(&dl_accounting_mutex);
+}
+
+void dl_accounting_unlock(void)
+{
+ mutex_unlock(&dl_accounting_mutex);
+}
+
+void lockdep_assert_dl_accounting_held(void)
+{
+ lockdep_assert_held(&dl_accounting_mutex);
+}
+#endif /* !CONFIG_CPUSETS */
+
+/*
+ * Rebuild the root domains' accounting. Clear it, then walk all tasks and
+ * count each deadline task again. The caller holds the locks that this
+ * function asserts.
+ */
+void dl_rebuild_rd_accounting(void)
+{
+ struct task_struct *g, *p;
+
+ lockdep_assert_cpus_held();
+ lockdep_assert_held(&sched_domains_mutex);
+ lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
+
+ dl_clear_root_domains();
+
+ /*
+ * Without a task that holds a bandwidth, there is nothing to count.
+ * The walk would still visit every thread of the system. A CPU
+ * that goes offline or comes back runs a rebuild, so a suspend and
+ * a resume run one for each CPU. A task gets a bandwidth under the
+ * locks this rebuild holds, so the count cannot leave 0 meanwhile.
+ */
+ if (!atomic_read(&dl_nr_bw_tasks))
+ return;
+
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, p) {
+ /*
+ * Test the class without the task's locks, and take them only
+ * for a deadline task. A task enters the class under the locks
+ * this rebuild holds, or by a boost. A boost brings no
+ * bandwidth of its own to count.
+ */
+ if (dl_task(p))
+ dl_add_task_root_domain(p);
+ }
+ read_unlock(&tasklist_lock);
+}
+
/*
* Clear every root domain before a rebuild of their accounting, and count
* the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
- * back the bandwidth of the tasks on dl_departed_list, which no cpuset
- * holds as deadline tasks. Put each in the root domain of the CPU
- * recorded for it, or drop it if that CPU is not active.
- * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ * back the bandwidth of the tasks on dl_departed_list, which the walk of
+ * the deadline tasks does not count. Put each in the root domain of the
+ * CPU recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks.
*/
void dl_clear_root_domains(void)
{
@@ -3831,12 +3928,6 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
p->dl.runtime = 0;
}
- /*
- * In case a task is setscheduled out from SCHED_DEADLINE we need to
- * keep track of that on its cpuset (for correct bandwidth tracking).
- */
- dec_dl_tasks_cs(p);
-
if (!task_on_rq_queued(p)) {
/*
* Inactive timer is armed. However, p is leaving DEADLINE and
@@ -3906,12 +3997,6 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
if (!list_empty(&p->dl.dl_departed_node))
dl_task_undepart(p);
- /*
- * In case a task is setscheduled to SCHED_DEADLINE we need to keep
- * track of that on its cpuset (for correct bandwidth tracking).
- */
- inc_dl_tasks_cs(p);
-
/* If p is not queued we will update its parameters at next wakeup. */
if (!task_on_rq_queued(p)) {
add_rq_bw(&p->dl, &rq->dl);
@@ -4168,10 +4253,12 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* A task that comes back before its 0-lag time still has the
* bandwidth it left with. The new bandwidth replaces it, so remove the
* old one from the root domain that keeps it (dl_task_release_bw()).
- * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+ * No rebuild can run meanwhile: the caller holds cpuset_mutex, or
+ * dl_accounting_mutex without cpusets.
*/
if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
if (p->dl.dl_departed_cpu >= 0)
dl_task_release_bw(p, -1);
p->dl.dl_bw_gen = dl_bw_gen;
@@ -4191,12 +4278,18 @@ int sched_dl_overflow(struct task_struct *p, int policy,
void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
{
struct sched_dl_entity *dl_se = &p->dl;
+ bool held;
+ held = dl_se_holds_bw(dl_se);
dl_se->dl_runtime = attr->sched_runtime;
dl_se->dl_deadline = attr->sched_deadline;
dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+ if (dl_se_holds_bw(dl_se) && !held)
+ atomic_inc(&dl_nr_bw_tasks);
+ else if (!dl_se_holds_bw(dl_se) && held)
+ atomic_dec(&dl_nr_bw_tasks);
dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
}
@@ -4286,6 +4379,9 @@ bool __checkparam_dl(const struct sched_attr *attr)
*/
static void __dl_clear_params(struct sched_dl_entity *dl_se)
{
+ if (dl_se_holds_bw(dl_se))
+ WARN_ON_ONCE(atomic_dec_return(&dl_nr_bw_tasks) < 0);
+
dl_se->dl_runtime = 0;
dl_se->dl_deadline = 0;
dl_se->dl_period = 0;
@@ -4317,6 +4413,12 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
RB_CLEAR_NODE(&dl_se->rb_node);
init_dl_task_timer(dl_se);
init_dl_inactive_task_timer(dl_se);
+ /*
+ * A new task has a copy of the entity of its parent. That copy
+ * holds no bandwidth, and dl_nr_bw_tasks does not count it. Clear
+ * dl_bw first, so that __dl_clear_params() leaves the count alone.
+ */
+ dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
INIT_LIST_HEAD(&dl_se->dl_departed_node);
dl_se->dl_departed_cpu = -1;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 8c72e751ff4e..4c013caa2471 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -568,6 +568,7 @@ int __sched_setscheduler(struct task_struct *p,
if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
cpuset_lock();
+ dl_accounting_lock();
}
/*
@@ -596,9 +597,7 @@ int __sched_setscheduler(struct task_struct *p,
* Reject SCHED_DEADLINE on exited tasks. A zombie will never be
* scheduled again, so sched_dl_overflow() would charge total_bw
* that nothing will ever release. And reject a change of policy of
- * one in the class: cgroup_task_dead() took it off the count of
- * deadline tasks of its cpuset, which switched_from_dl() would
- * decrement a second time.
+ * one in the class: its bandwidth went at its last schedule.
*/
if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
retval = -ESRCH;
@@ -669,6 +668,7 @@ int __sched_setscheduler(struct task_struct *p,
policy = oldpolicy = -1;
task_rq_unlock(rq, p, &rf);
if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
cpuset_locked = false;
}
@@ -779,8 +779,10 @@ int __sched_setscheduler(struct task_struct *p,
task_rq_unlock(rq, p, &rf);
if (pi) {
- if (cpuset_locked)
+ if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
+ }
rt_mutex_adjust_pi(p);
}
@@ -792,8 +794,10 @@ int __sched_setscheduler(struct task_struct *p,
unlock:
task_rq_unlock(rq, p, &rf);
- if (cpuset_locked)
+ if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
+ }
return retval;
}
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 9d3a1b8dd393..35e56a45c238 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3528,7 +3528,15 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[],
struct sched_domain_attr *dattr_new)
{
+ /*
+ * The root domains attached here count no task's bandwidth until
+ * dl_rebuild_rd_accounting() at the end. No admission of bandwidth may
+ * run meanwhile. With cpusets, the callers hold cpuset_mutex, which
+ * keeps admissions out. Without cpusets, dl_accounting_lock() does it.
+ */
+ dl_accounting_lock();
sched_domains_mutex_lock();
partition_sched_domains_locked(ndoms_new, doms_new, dattr_new);
sched_domains_mutex_unlock();
+ dl_accounting_unlock();
}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (16 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 17/40] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:14 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 19/40] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
` (21 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Admission, moves and releases of a deadline task's bandwidth find the
root domain that counts it from the CPU the task is on. That CPU can be
in another root domain. Two cases show it.
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the root domain of the
task's runqueue. Since commit 5dd74441cbf4 ("cgroup/cpuset: Reserve DL
bandwidth only for root-domain moves"), both halves move nothing when
that root domain overlaps the destination. But a sleeping task stays on
the CPU it last ran on. So a sleeping task moved to a partition and back
leaves its bandwidth in the partition:
admin task T
makes P a partition root of CPU 3
CPU 3 is now in root domain B
CPUs 0-2 are in root domain A
sched_setattr(SCHED_DEADLINE)
A: total_bw += bw of T
runs on CPU 0, then sleeps
echo T > P/cgroup.procs
cpuset_can_attach()
dl_task_needs_bw_move(): true,
A does not overlap P
B: total_bw += bw of T
set_cpus_allowed_dl()
dl_task_needs_bw_move(): true
A: total_bw -= bw of T
still asleep on CPU 0
echo T > cgroup.procs of the root cgroup
cpuset_can_attach()
dl_task_needs_bw_move(): false,
A overlaps the root cpuset
set_cpus_allowed_dl()
dl_task_needs_bw_move(): false
kill -9 T
wakes up on a CPU of A
exits
A: total_bw -= bw of T
B keeps the bandwidth of T. A does not count T while T is in A, so A
admits beyond its capacity. After T exits, the total_bw of A is short
by the bandwidth of T, and can underflow.
The second case needs no move:
1. A deadline task T sleeps on a CPU of root domain A.
2. That CPU goes offline, and joins the default root domain.
3. The rebuild counts T in A, the root domain of its cpuset.
4. sched_setattr() takes the sleeping T out of SCHED_DEADLINE.
5. The release takes the bandwidth from the root domain of the CPU T
is on: the default one.
A keeps the bandwidth of T until the next rebuild, and refuses deadline
tasks that would fit.
The first case was reproduced on v7.4-rc2. The second needs the earlier
patches of this series: without "sched/deadline: Release the bandwidth
of a task that leaves the class asleep", nothing releases the bandwidth
of that sleeping task. The hotplug_sleeper_switched case of
deadline_bw_test, added later in this series, tests it. Without this
patch and with the CPU still offline, A admitted 63 tasks of 10% where
it had room for 66, in 3 guests of 3.
Fix this by recording, in dl_bw_cpu, a CPU of the root domain that
counts the bandwidth. The record replaces dl_departed_cpu. It is used
as follows:
- Each site that counts the bandwidth sets the record.
- Each release or move reads it. The release clears it.
- If dl_bw_gen shows that a rebuild dropped the bandwidth, nothing is
taken back. set_cpus_allowed_dl() makes that test under
dl_departed_lock, so that a rebuild cannot change dl_bw_gen
meanwhile.
- A move skips a bandwidth of 0, and does not update its record. So
sched_dl_overflow() uses the CPU the task is on for it. The
move_zero_bw case of deadline_bw_test, added later in this series,
tests this.
- A rebuild drops departed bandwidth whose CPU is inactive. So a
departing task records the CPU it departs from, if that CPU is
active and in the same root domain.
The affinity check of sched_setaffinity() reads the record too. A task
moved asleep is still on a CPU of the root domain it left. Testing that
root domain let a new mask take the task out of the one that counts it.
The affinity_moved_asleep case tests this.
A move now leaves out a sugov kthread and a task boosted into the class
with no bandwidth of its own. The rebuild leaves them out too: no root
domain counts theirs. A move would take a sugov kthread's bandwidth
from a root domain that never held it.
Weiheng Qiu reported the moved sleeper, and Juri Lelli analysed it. His
fix, linked below, took the first active CPU of the task's mask. A mask
can have none during CPU hotplug. Tip dropped the fix after a sugov
kthread hit its BUG_ON() there. The Sashiko AI reviews of this series
found the other cases.
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/14508cb2-805f-41ac-9f6b-bfd7c3d7cb1c@gmail.com/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/
---
include/linux/sched.h | 10 +-
kernel/sched/deadline.c | 247 +++++++++++++++++++++++++++-------------
kernel/sched/syscalls.c | 11 +-
3 files changed, 187 insertions(+), 81 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 33e2d9492aa2..4a1cb869facc 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -750,15 +750,17 @@ struct sched_dl_entity {
* time" keeps its bandwidth in its root domain until the inactive
* timer removes it. Meanwhile, @dl_departed_node links the task on
* the list that a rebuild of the root domains' accounting reads.
- * @dl_departed_cpu is a CPU of the root domain that counts the
- * bandwidth. It stays recorded while priority inheritance boosts the
- * task back into the class. Otherwise @dl_departed_cpu is -1.
+ * Otherwise that node is empty.
+ *
+ * @dl_bw_cpu is a CPU of the root domain that counts the bandwidth,
+ * or -1. Each site that counts the bandwidth sets it. Each site that
+ * removes or moves the bandwidth reads it.
*
* @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
* counted. If a rebuild dropped it since, there is nothing to remove.
*/
struct list_head dl_departed_node;
- int dl_departed_cpu;
+ int dl_bw_cpu;
unsigned int dl_bw_gen;
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 8e8c62c574d5..01fbbd02cf8d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -269,7 +269,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* dl_add_task_root_domain()) does not count such a task as a deadline
* task. So dl_departed_list holds these tasks, from their departure to
* their timer. The rebuild puts their bandwidth back in the root domain
- * of the CPU each one was on when it left.
+ * of the CPU recorded with each task's bandwidth.
*
* dl_bw_gen numbers the rebuilds of the accounting: how many times the
* root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -291,7 +291,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* A rebuild drops the bandwidth when the CPU recorded is no longer
* active. It also drops the bandwidth of a task that dies while the
* rebuild runs. The walk of the task list does not find that task, and
- * it was not dead yet when the list was put back.
+ * it was not dead yet when the list was put back. The same happens to a
+ * task that priority inheritance deboosts while the rebuild runs. It was
+ * in the class when the list was put back, and out of it when the walk
+ * reaches it.
*
* dl_departed_lock protects the list and both counts. Take it with
* interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -318,66 +321,75 @@ static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
}
/*
- * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * Put @p on dl_departed_list. The root domain of the CPU recorded with it
* keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
- * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
- * recorded if @p has one: priority inheritance boosted @p back into the
- * class and its boost ends. The caller holds the rq lock of @p.
+ * 0-lag time. Do nothing if @p is on the list already. The caller holds
+ * the rq lock of @p.
*/
-static void dl_task_depart(struct task_struct *p, int cpu)
+static void dl_task_depart(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
+ int cpu = task_cpu(p);
lockdep_assert_irqs_disabled();
guard(raw_spinlock)(&dl_departed_lock);
- if (!list_empty(&dl_se->dl_departed_node))
+ if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
return;
- if (dl_se->dl_departed_cpu < 0)
- dl_se->dl_departed_cpu = cpu;
+ /*
+ * A rebuild drops the bandwidth if the CPU recorded is not active.
+ * That CPU can be one that @p left long ago. Record the CPU @p
+ * departs from instead, if it is active and the same root domain
+ * counts the bandwidth.
+ */
+ if (dl_se->dl_bw_gen == dl_bw_gen && cpu_active(cpu) &&
+ cpu_rq(cpu)->rd == cpu_rq(dl_se->dl_bw_cpu)->rd)
+ dl_se->dl_bw_cpu = cpu;
list_add(&dl_se->dl_departed_node, &dl_departed_list);
}
/*
- * Take @p off dl_departed_list. Priority inheritance boosts @p back into
- * the deadline class. Keep the CPU recorded: its root domain still counts
- * the bandwidth, wherever @p runs now. The caller holds the rq lock of
- * @p.
+ * Take @p off dl_departed_list. @p comes back to the deadline class, and
+ * its bandwidth is that of a deadline task again. The caller holds the rq
+ * lock of @p.
*/
static void dl_task_undepart(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
lockdep_assert_irqs_disabled();
+ if (list_empty(&dl_se->dl_departed_node))
+ return;
guard(raw_spinlock)(&dl_departed_lock);
list_del_init(&dl_se->dl_departed_node);
}
/*
* Remove the bandwidth of @p from the root domain that counts it. That is
- * the root domain of the CPU recorded when @p departed, or else of @cpu.
- * Clear the record. Take @p off dl_departed_list. Remove nothing if a
- * rebuild dropped the bandwidth since it was counted. Then record the
- * current dl_bw_gen in @p, so that a rebuild in progress does not count it
- * again. The caller holds the rq lock of @p.
+ * the root domain of the CPU recorded with @p. Clear the record.
+ * Take @p off dl_departed_list. Remove nothing if a rebuild dropped the
+ * bandwidth since it was counted. Then record the current dl_bw_gen in
+ * @p, so that a rebuild in progress does not count it again. The caller
+ * holds the rq lock of @p.
*/
-static void dl_task_release_bw(struct task_struct *p, int cpu)
+static void dl_task_release_bw(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
struct dl_bw *dl_b;
+ int cpu;
lockdep_assert_irqs_disabled();
guard(raw_spinlock)(&dl_departed_lock);
- if (dl_se->dl_departed_cpu >= 0) {
- cpu = dl_se->dl_departed_cpu;
- dl_se->dl_departed_cpu = -1;
+ if (!list_empty(&dl_se->dl_departed_node))
list_del_init(&dl_se->dl_departed_node);
- }
- if (dl_se->dl_bw_gen == dl_bw_gen) {
+ cpu = dl_se->dl_bw_cpu;
+ WARN_ON_ONCE(dl_se->dl_bw && cpu < 0);
+ if (cpu >= 0 && dl_se->dl_bw_gen == dl_bw_gen) {
dl_b = dl_bw_of(cpu);
raw_spin_lock(&dl_b->lock);
__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
raw_spin_unlock(&dl_b->lock);
}
+ dl_se->dl_bw_cpu = -1;
dl_se->dl_bw_gen = dl_bw_gen;
}
@@ -540,6 +552,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
}
static void __dl_clear_params(struct sched_dl_entity *dl_se);
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p);
/*
* The utilization of a task cannot be immediately removed from
@@ -640,7 +653,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
- dl_task_release_bw(p, task_cpu(p));
+ dl_task_release_bw(p);
__dl_clear_params(dl_se);
}
}
@@ -654,12 +667,13 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
get_task_struct(p);
/*
- * A dying task keeps its bandwidth in the root domain of the
- * CPU it runs on. Record that CPU. switched_from_dl() records
- * a task that leaves SCHED_DEADLINE.
+ * A dying task keeps its bandwidth until its 0-lag time. Put
+ * it on the departed list. The CPU recorded with its bandwidth
+ * tells the root domain that counts it. switched_from_dl() does
+ * the same for a task that leaves SCHED_DEADLINE.
*/
if (READ_ONCE(p->__state) == TASK_DEAD)
- dl_task_depart(p, task_cpu(p));
+ dl_task_depart(p);
}
hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
@@ -897,19 +911,31 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
}
/*
- * And we finally need to fix up root_domain(s) bandwidth accounting,
- * since p is still hanging out in the old (now moved to default) root
- * domain.
+ * And we finally need to fix up root_domain(s) bandwidth accounting.
+ * The bandwidth of @p is counted in the root domain of the CPU
+ * recorded with it. That is not the root domain of @rq if @p was
+ * moved to another cpuset while it was not running. If a rebuild
+ * dropped the bandwidth, there is nothing to move: the next rebuild
+ * counts it.
*/
- dl_b = &rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
-
- dl_b = &later_rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
+ scoped_guard (raw_spinlock, &dl_departed_lock) {
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
+
+ dl_b = &src_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, p->dl.dl_bw,
+ cpumask_weight(src_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
+
+ dl_b = &later_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw,
+ cpumask_weight(later_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ p->dl.dl_bw_cpu = later_rq->cpu;
+ }
set_task_cpu(p, later_rq->cpu);
double_unlock_balance(later_rq, rq);
@@ -2427,7 +2453,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
dl_se->dl_non_contending = 0;
}
- dl_task_release_bw(p, task_cpu(p));
+ dl_task_release_bw(p);
__dl_clear_params(dl_se);
goto unlock;
@@ -3575,14 +3601,26 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
}
}
+/*
+ * Return a CPU of the root domain that counts the bandwidth of @p: the
+ * CPU recorded with it. Without a record, no root domain counts that
+ * bandwidth. Return the CPU @p is on then. Read the record once:
+ * dl_task_check_affinity() holds no lock.
+ */
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
+{
+ int cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+ if (cpu < 0)
+ return task_cpu(p);
+ return cpu;
+}
+
static void set_cpus_allowed_dl(struct task_struct *p,
struct affinity_context *ctx)
{
- struct rq *rq;
-
WARN_ON_ONCE(!dl_task(p));
- rq = task_rq(p);
/*
* Migrating a SCHED_DEADLINE task between exclusive
* cpusets (different root_domains) entails a bandwidth
@@ -3590,17 +3628,31 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* domain (see cpuset_can_attach()).
*/
if (dl_task_needs_bw_move(p, ctx->new_mask)) {
- struct dl_bw *src_dl_b;
+ unsigned int src_cpu = dl_task_root_domain_cpu(p);
+ struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
+ unsigned int cpu;
- src_dl_b = dl_bw_of(cpu_of(rq));
/*
* We now free resources of the root_domain we are migrating
* off. In the worst case, sched_setattr() may temporary fail
- * until we complete the update.
+ * until we complete the update. A rebuild that dropped the
+ * bandwidth of @p leaves nothing to free. dl_departed_lock
+ * keeps a rebuild from changing dl_bw_gen meanwhile.
*/
- raw_spin_lock(&src_dl_b->lock);
- __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&src_dl_b->lock);
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ raw_spin_lock(&src_dl_b->lock);
+ __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+ raw_spin_unlock(&src_dl_b->lock);
+ }
+
+ /*
+ * cpuset_can_attach() made room in the root domain of the new
+ * mask. That root domain counts the bandwidth from now on.
+ */
+ cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
+ p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
+ p->dl.dl_bw_gen = dl_bw_gen;
}
set_cpus_allowed_common(p, ctx);
@@ -3609,10 +3661,18 @@ static void set_cpus_allowed_dl(struct task_struct *p,
bool dl_task_needs_bw_move(struct task_struct *p,
const struct cpumask *new_mask)
{
- if (!dl_task(p))
+ unsigned int cpu;
+
+ /*
+ * Nothing to move for a special entity, or for a task boosted into
+ * the class with no bandwidth of its own. No root domain counts
+ * their bandwidth: dl_add_task_root_domain() leaves them out.
+ */
+ if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
return false;
- return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
+ cpu = dl_task_root_domain_cpu(p);
+ return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
}
/* Assumes rq->lock is held */
@@ -3688,7 +3748,8 @@ void dl_add_task_root_domain(struct task_struct *p)
* Count nothing for a special entity, or for a task with no
* bandwidth. The walk of the task list also finds exited tasks. An
* exited task whose bandwidth went with its parameters has a dl_bw
- * of 0.
+ * of 0. So has a task boosted into the class with no bandwidth of
+ * its own, which gets no record.
*/
if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
@@ -3712,7 +3773,9 @@ void dl_add_task_root_domain(struct task_struct *p)
*/
if (p->dl.dl_bw_gen == dl_bw_gen)
goto unlock;
- if (p->dl.dl_departed_cpu >= 0) {
+ if (!list_empty(&p->dl.dl_departed_node)) {
+ int recorded = p->dl.dl_bw_cpu;
+
/*
* The walk also finds a dead task that departed before an
* earlier rebuild, if it is not reaped yet. If that rebuild
@@ -3721,9 +3784,9 @@ void dl_add_task_root_domain(struct task_struct *p)
*/
if (p->dl.dl_bw_gen != dl_bw_gen - 1)
goto unlock;
- cpu = p->dl.dl_departed_cpu;
- if (!cpu_active(cpu))
+ if (recorded < 0 || !cpu_active(recorded))
goto unlock;
+ cpu = recorded;
}
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
@@ -3731,6 +3794,7 @@ void dl_add_task_root_domain(struct task_struct *p)
raw_spin_lock(&dl_b->lock);
__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_cpu = cpu;
p->dl.dl_bw_gen = dl_bw_gen;
unlock:
raw_spin_unlock(&dl_departed_lock);
@@ -3880,7 +3944,7 @@ void dl_clear_root_domains(void)
dl_clear_root_domain(cpu_rq(cpu)->rd);
}
list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
- cpu = dl_se->dl_departed_cpu;
+ cpu = dl_se->dl_bw_cpu;
/*
* Skip bandwidth that an earlier rebuild dropped. It is not
* counted now, and its release removes nothing
@@ -3895,7 +3959,7 @@ void dl_clear_root_domains(void)
* from the recorded number. The release then removes
* nothing, and no later rebuild restores it.
*/
- if (!cpu_active(cpu))
+ if (cpu < 0 || !cpu_active(cpu))
continue;
dl_b = dl_bw_of(cpu);
raw_spin_lock(&dl_b->lock);
@@ -3956,15 +4020,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
* For the same reason, the timer cannot take the task's bandwidth
* from the root domain of the CPU the task is on by then. And a
* rebuild of the root domains meanwhile would not find the task
- * among the deadline tasks. So record a CPU of the root domain that
- * keeps the bandwidth until the timer removes it (dl_task_depart()).
+ * among the deadline tasks. So put the task on the departed list
+ * until the timer removes the bandwidth (dl_task_depart()). The CPU
+ * recorded with the bandwidth tells the root domain that keeps it.
* Test dl_non_contending: it tells that the timer is armed and its
* handler has not run. hrtimer_active() stays true for a while after
* the handler returns.
*/
if (p->dl.dl_non_contending) {
p->dl.dl_non_contending = 0;
- dl_task_depart(p, task_cpu(p));
+ dl_task_depart(p);
}
/*
@@ -3992,10 +4057,11 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
* longer removes its bandwidth: the task does when it leaves the class
* again. So take it off the list. The task keeps the CPU recorded. It
* takes the bandwidth from the root domain of that CPU, even if it
- * moved to another root domain meanwhile.
+ * moved to another root domain meanwhile. A rebuild while it is
+ * boosted counts it like any deadline task, and records the CPU it
+ * picks.
*/
- if (!list_empty(&p->dl.dl_departed_node))
- dl_task_undepart(p);
+ dl_task_undepart(p);
/* If p is not queued we will update its parameters at next wakeup. */
if (!task_on_rq_queued(p)) {
@@ -4202,9 +4268,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
u64 period = attr->sched_period ?: attr->sched_deadline;
u64 runtime = attr->sched_runtime;
u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
- int cpus, err = -1, cpu = task_cpu(p);
- struct dl_bw *dl_b = dl_bw_of(cpu);
+ int cpus, err = -1, cpu;
+ struct dl_bw *dl_b;
unsigned long cap;
+ bool counted;
if (attr->sched_flags & SCHED_FLAG_SUGOV)
return 0;
@@ -4213,6 +4280,23 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
return 0;
+ /*
+ * Change the bandwidth of a deadline task where it is counted: in
+ * the root domain of the CPU recorded with it. A task moved asleep
+ * has not run there yet, so the CPU it is on can be elsewhere. No
+ * bandwidth is counted if a rebuild dropped it, or if the task
+ * enters the class. A runtime below about a millionth of the period
+ * gives a bandwidth of 0. It counts for nothing, and no move updates
+ * its record. In these three cases, the root domain of the CPU the
+ * task is on takes the new bandwidth. The caller holds cpuset_mutex,
+ * or dl_accounting_mutex without cpusets, so no rebuild runs
+ * meanwhile.
+ */
+ counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
+ p->dl.dl_bw_gen == dl_bw_gen;
+ cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
+ dl_b = dl_bw_of(cpu);
+
/*
* Either if a task, enters, leave, or stays -deadline but changes
* its parameters, we may need to update accordingly the total
@@ -4227,7 +4311,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
__dl_add(dl_b, new_bw, cpus);
err = 0;
} else if (dl_policy(policy) && task_has_dl_policy(p) &&
- !__dl_overflow(dl_b, cap, p->dl.dl_bw, new_bw)) {
+ !__dl_overflow(dl_b, cap, counted ? p->dl.dl_bw : 0,
+ new_bw)) {
/*
* XXX this is slightly incorrect: when the task
* utilization decreases, we should delay the total
@@ -4235,7 +4320,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* But this would require to set the task's "inactive
* timer" when the task is not inactive.
*/
- __dl_sub(dl_b, p->dl.dl_bw, cpus);
+ if (counted)
+ __dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
dl_change_utilization(p, new_bw);
err = 0;
@@ -4256,11 +4342,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* No rebuild can run meanwhile: the caller holds cpuset_mutex, or
* dl_accounting_mutex without cpusets.
*/
- if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
- lockdep_assert_cpuset_lock_held();
- lockdep_assert_dl_accounting_held();
- if (p->dl.dl_departed_cpu >= 0)
- dl_task_release_bw(p, -1);
+ if (!err && dl_policy(policy)) {
+ if (!task_has_dl_policy(p)) {
+ lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
+ /*
+ * The new bandwidth replaces any the task still has
+ * counted. That one may be on the departed list, or
+ * taken off it by a boost into the class
+ * (switched_to_dl()). The record says where it is.
+ */
+ if (p->dl.dl_bw_cpu >= 0)
+ dl_task_release_bw(p);
+ }
+ p->dl.dl_bw_cpu = cpu;
p->dl.dl_bw_gen = dl_bw_gen;
}
@@ -4420,8 +4515,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
*/
dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
+ dl_se->dl_bw_cpu = -1;
INIT_LIST_HEAD(&dl_se->dl_departed_node);
- dl_se->dl_departed_cpu = -1;
}
bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4c013caa2471..306cd8d32173 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1185,6 +1185,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
{
+ unsigned int cpu;
+
/*
* If the task isn't a deadline task or admission control is
* disabled then we don't care about affinity changes.
@@ -1204,9 +1206,16 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
* if admission test is enabled, we only admit -deadline
* tasks allowed to run on all the CPUs in the task's
* root_domain.
+ *
+ * That is the root domain that counts the bandwidth of the task:
+ * the one of the CPU recorded with it. A task moved asleep to
+ * another cpuset is still on a CPU of the root domain it left. A
+ * bandwidth of 0 is counted nowhere, and no move updates its
+ * record: use the CPU the task is on.
*/
+ cpu = READ_ONCE(p->dl.dl_bw) ? dl_task_root_domain_cpu(p) : task_cpu(p);
guard(rcu)();
- if (!cpumask_subset(task_rq(p)->rd->span, mask))
+ if (!cpumask_subset(cpu_rq(cpu)->rd->span, mask))
return -EBUSY;
return 0;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 19/40] sched/deadline: Leave a dying task out of a rebuild of the accounting
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (17 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 20/40] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
` (20 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A rebuild of the root domains' accounting takes the pi_lock of each
deadline task, not its rq lock. A dying task past its 0-lag time
releases its bandwidth at its last schedule. It holds its rq lock there,
not its pi_lock. Then it clears dl_bw. The rebuild can find the task
between the two steps.
In this diagram, T is a deadline task that exits on CPU 0 past its 0-lag
time. bw is its dl_bw. CPU 1 rebuilds the accounting:
CPU 0 (T) CPU 1 (rebuild)
do_task_dead(): TASK_DEAD, under pi_lock
__schedule(): T's rq lock taken
task_non_contending(T)
dl_task_release_bw(T)
total_bw -= bw
dl_bw_gen recorded
record cleared
dl_rebuild_rd_accounting()
dl_clear_root_domains()
dl_bw_gen++
total_bw reset
dl_add_task_root_domain(T)
T's pi_lock taken
dl_bw != 0, dl_bw_gen old
total_bw += bw
__dl_clear_params(T): dl_bw = 0
dl_bw_gen does not help, since the release came before the rebuild. The
rebuild would count a bandwidth that nothing releases again. The root
domain would then refuse deadline tasks it has room for, until the next
rebuild.
Fix this with the record of the bandwidth, dl_bw_cpu. The release
clears it under dl_departed_lock, which the walk holds. So leave out a
dead task with no record. A dead task with a record has not reached its
last schedule. The walk counts it as a live task. That schedule then
releases the bandwidth, or puts the task on the departed list. A dead
task on the departed list keeps its bandwidth until its timer, and the
rebuild counts it as before.
The Sashiko AI review of v2 of this series found this window. Its review
of v4 found a gap in the first version of this fix. That version also
left out a task between TASK_DEAD and its last schedule, with its 0-lag
time ahead.
This patch completes "sched/deadline: Rebuild the root domains'
accounting from the task list". The walk of that patch finds exiting
tasks, which the cpuset iterator skipped. The fix needs the record
that "sched/deadline: Record the root domain holding a task's
bandwidth" adds. The three go to stable together.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 01fbbd02cf8d..779acb9836dc 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3787,6 +3787,17 @@ void dl_add_task_root_domain(struct task_struct *p)
if (recorded < 0 || !cpu_active(recorded))
goto unlock;
cpu = recorded;
+ } else if (READ_ONCE(p->__state) == TASK_DEAD && p->dl.dl_bw_cpu < 0) {
+ /*
+ * Count nothing for a dead task with no record. Its last
+ * schedule released the bandwidth under its rq lock, and has
+ * not cleared dl_bw yet. The release clears the record under
+ * dl_departed_lock, which this walk holds. A dead task with a
+ * record has not reached its last schedule. Count it as a
+ * live task. That schedule then releases what the walk
+ * counted, or puts the task on the departed list.
+ */
+ goto unlock;
}
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 20/40] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (18 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 19/40] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 21/40] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
` (19 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
From: Andrea Parri <parri.andrea@gmail.com>
A task can end up SCHED_DEADLINE with an affinity narrower than its root
domain when sched_setaffinity() and sched_setattr() race on it. Both
calls succeed, although each fails when made after the other:
sched_setattr() with -EPERM, sched_setaffinity() with -EBUSY.
__sched_setaffinity() calls dl_task_check_affinity() before
__set_cpus_allowed_ptr() takes task_rq_lock(), whereas
__sched_setscheduler() runs its admission test and switches the policy
under a single task_rq_lock(). So both calls can pass their checks:
CPU 1: sched_setaffinity(p, cpu0) CPU 2: sched_setattr(p, DEADLINE)
dl_task_check_affinity()
!task_has_dl_policy(p): return 0
task_rq_lock()
admission test passes
p->policy = SCHED_DEADLINE
task_rq_unlock()
__set_cpus_allowed_ptr()
task_rq_lock()
p->cpus_mask = cpu0
The window includes CPU 1 spinning on task_rq_lock() while CPU 2 holds
it. With one thread narrowing a sleeping task to CPU 0 while another
makes it SCHED_DEADLINE, 923514, 959547 and 983997 of 1000000 rounds
ended with both calls succeeding on three boots of a 4-CPU guest.
Move the check into __set_cpus_allowed_ptr_locked(), next to the
PF_NO_SETAFFINITY re-check, under the same lock as the admission test.
Whichever call takes task_rq_lock() second then sees the other's
result: sched_setaffinity() fails with -EBUSY if p is already
SCHED_DEADLINE, and sched_setattr() fails with -EPERM if p's affinity
is already cpu0.
A new SCA_DL_CHECK flag selects the check, and only
__sched_setaffinity() sets it, so sched_setaffinity() and
relax_compatible_cpus_allowed_ptr() keep the check and other callers
stay unchecked. restrict_cpus_allowed_ptr() already does its deadline
check this way.
__sched_setaffinity() clears SCA_DL_CHECK before the second
__set_cpus_allowed_ptr() call it makes after racing with a cpuset
update, so the cpuset mask still wins there, as it does when
cpuset_update_tasks_cpumask() narrows a deadline task. SCA_CHECK stays
set for the PF_NO_SETAFFINITY re-check.
With this applied, the same test ends with both calls succeeding in 0
of 1000000 rounds on each of three boots, and sched_setaffinity()
fails with -EBUSY whenever it loses.
The race was reported by Sebastian. The fix was written with the help
of an LLM.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Closes: https://lore.kernel.org/20261103081149.DuyWNhVe@linutronix.de
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/core.c | 11 +++++++++++
kernel/sched/sched.h | 1 +
kernel/sched/syscalls.c | 12 +++++++-----
3 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index a22df8b43a1a..c77cae5d1be1 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,17 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * Check under the same lock that __sched_setscheduler() holds for
+ * its admission test, so that p cannot become SCHED_DEADLINE
+ * between this check and the mask update below.
+ */
+ if (ctx->flags & SCA_DL_CHECK) {
+ ret = dl_task_check_affinity(p, ctx->new_mask);
+ if (ret)
+ goto out;
+ }
+
if (!(ctx->flags & SCA_MIGRATE_ENABLE)) {
if (cpumask_equal(&p->cpus_mask, ctx->new_mask)) {
if (ctx->flags & SCA_USER)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d87296abec4c..fd6d36b6b1e2 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2982,6 +2982,7 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_MIGRATE_DISABLE 0x02
#define SCA_MIGRATE_ENABLE 0x04
#define SCA_USER 0x08
+#define SCA_DL_CHECK 0x10
extern void update_group_capacity(struct sched_domain *sd, int cpu);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 306cd8d32173..4f47b17151ed 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1238,11 +1238,7 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
cpumask_and(new_mask, ctx->new_mask, cpus_allowed);
ctx->new_mask = new_mask;
- ctx->flags |= SCA_CHECK;
-
- retval = dl_task_check_affinity(p, new_mask);
- if (retval)
- goto out_free_new_mask;
+ ctx->flags |= SCA_CHECK | SCA_DL_CHECK;
retval = __set_cpus_allowed_ptr(p, ctx);
if (retval)
@@ -1271,6 +1267,12 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
if (empty)
cpumask_copy(new_mask, cpus_allowed);
}
+
+ /*
+ * The cpuset mask must win here, as it does when
+ * cpuset_update_tasks_cpumask() sets it on a deadline task.
+ */
+ ctx->flags &= ~SCA_DL_CHECK;
__set_cpus_allowed_ptr(p, ctx);
retval = -EINVAL;
}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 21/40] sched/deadline: Move a task's bandwidth only where room was made
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (19 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 20/40] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 22/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
` (18 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination root domain. set_cpus_allowed_dl() then frees it in the
source, when the task's affinity changes. Its comment relies on the
first half: "We already made space for us in the destination domain".
But set_cpus_allowed_dl() runs on every change of affinity. It frees the
bandwidth whenever the new mask leaves the task's root domain, with or
without a reservation. sched_setaffinity() gets there too, if the task's
cpuset spans two root domains. With isolcpus="domain", any cpuset that
is not a partition root can hold isolated CPUs and other CPUs, which are
in two root domains. With admission control on, __sched_setaffinity()
refuses such a mask. With sched_rt_runtime_us at -1, nothing refuses it.
The bandwidth is then freed in the source and added nowhere: no root
domain counts it any more. The destination would underflow when the task
leaves.
Fix this by flagging the affinity changes that come with a reservation,
as SCA_DL_BW_RESERVED. cpuset_attach_task() sets it, through the new
set_cpus_allowed_bw_reserved(). set_cpus_allowed_dl() moves the
bandwidth only with the flag. Without it, the bandwidth stays where it
is counted, until the next rebuild of the accounting counts it anew.
The flag also skips the early exit of __set_cpus_allowed_ptr_locked()
for a mask that does not change. The task may already have that mask
from a change that moved no bandwidth, such as a sched_setaffinity()
without admission control.
The attach still flags a task that sched_setattr() made a deadline task
after cpuset_can_attach(), with nothing reserved for it. "sched: Make
deadline policy and affinity changes wait for a cpuset attach" fixes
that.
No case of deadline_bw_test fails before this patch and passes with it.
no_admission_affinity_moves, added later in this series, covers this
path together with "sched/deadline: Move a task's bandwidth to the root
domain it runs in".
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 2 ++
kernel/cgroup/cpuset.c | 13 ++++++++++++-
kernel/sched/core.c | 26 +++++++++++++++++++++++++-
kernel/sched/deadline.c | 14 +++++++++-----
kernel/sched/sched.h | 2 ++
5 files changed, 50 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4a1cb869facc..dee4d4df624c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1972,6 +1972,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
* Return: zero if successful, or a negative error code
*/
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
+extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
+ const struct cpumask *new_mask);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7628cf77bdfd..194a438e25c3 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3261,6 +3261,7 @@ static cpumask_var_t cpus_attach;
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
+ bool reserved = false;
struct mm_struct *mm;
lockdep_assert_cpuset_lock_held();
@@ -3270,11 +3271,21 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
else
cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
subpartitions_cpus);
+ /*
+ * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
+ * deadline tasks that cpuset_can_attach() counted. Flag the move, so
+ * that the bandwidth of this task moves with it.
+ */
+ if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+ reserved = true;
/*
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ if (reserved)
+ WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
+ else
+ WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
if (cpuset_v2() && !attach_ctx.mems_updated)
return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c77cae5d1be1..2a964286a7cd 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3207,7 +3207,15 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
}
if (!(ctx->flags & SCA_MIGRATE_ENABLE)) {
- if (cpumask_equal(&p->cpus_mask, ctx->new_mask)) {
+ /*
+ * A move with SCA_DL_BW_RESERVED must reach
+ * set_cpus_allowed_dl(), even if the mask does not change.
+ * @p may have this mask from a change that moved no
+ * bandwidth, such as sched_setaffinity() without admission
+ * control.
+ */
+ if (cpumask_equal(&p->cpus_mask, ctx->new_mask) &&
+ !(ctx->flags & SCA_DL_BW_RESERVED)) {
if (ctx->flags & SCA_USER)
swap(p->user_cpus_ptr, ctx->user_mask);
goto out;
@@ -3280,6 +3288,22 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
}
EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
+/*
+ * Like set_cpus_allowed_ptr(), for a caller that has made room for the
+ * deadline bandwidth of @p in the root domain of @new_mask. Move that
+ * bandwidth there with @p. Return 0 on success, or a negative error
+ * code.
+ */
+int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+{
+ struct affinity_context ac = {
+ .new_mask = new_mask,
+ .flags = SCA_DL_BW_RESERVED,
+ };
+
+ return __set_cpus_allowed_ptr(p, &ac);
+}
+
/*
* Change a given task's CPU affinity to the intersection of its current
* affinity mask and @subset_mask, writing the resulting mask to @new_mask.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 779acb9836dc..aba03fd6b381 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3622,12 +3622,16 @@ static void set_cpus_allowed_dl(struct task_struct *p,
WARN_ON_ONCE(!dl_task(p));
/*
- * Migrating a SCHED_DEADLINE task between exclusive
- * cpusets (different root_domains) entails a bandwidth
- * update. We already made space for us in the destination
- * domain (see cpuset_can_attach()).
+ * Migrating a SCHED_DEADLINE task between exclusive cpusets
+ * (different root_domains) entails a bandwidth update. Move the
+ * bandwidth only if the caller made room for it in the destination,
+ * as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
+ * Otherwise, the bandwidth stays where it is counted, until the next
+ * rebuild of the accounting. Moving it without room would leave no
+ * root domain counting it.
*/
- if (dl_task_needs_bw_move(p, ctx->new_mask)) {
+ if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+ dl_task_needs_bw_move(p, ctx->new_mask)) {
unsigned int src_cpu = dl_task_root_domain_cpu(p);
struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
unsigned int cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index fd6d36b6b1e2..9e3ca69abc36 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2983,6 +2983,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_MIGRATE_ENABLE 0x04
#define SCA_USER 0x08
#define SCA_DL_CHECK 0x10
+/* The caller made room for the deadline bandwidth in the new root domain. */
+#define SCA_DL_BW_RESERVED 0x20
extern void update_group_capacity(struct sched_domain *sd, int cpu);
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 22/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (20 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 21/40] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
` (17 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
cpuset_can_attach() reserves the bandwidth of the deadline tasks it
moves in the destination root domain. cpuset_attach() moves their
bandwidth later, on the strength of that reservation.
Commit 75f7a25ec6bb ("cgroup/cpuset: Prevent race between task attach
and cpuset state change") lets cpuset_attach() rely on what
cpuset_can_attach() set up. The writes of cpus, mems and cpus.partition
wait for an attach in progress, in wait_attach_done_lock(). CPU hotplug
cannot run in between either: it takes cpus_write_lock, and the attach
holds cpus_read_lock throughout.
But a rebuild of the sched domains does not wait. A write of
sched_rt_runtime_us and the energy model rebuild them through
rebuild_sched_domains(). The cgroup v1 writes of sched_load_balance and
sched_relax_domain_level, and hk_sd_workfn(), use cpuset_full_lock().
In this example, a rebuild drops the reservation. T is a deadline task
asleep in root domain R. Thread A moves T to a partition root P, which
has a root domain of its own. Thread B writes sched_rt_runtime_us:
A B
cpuset_can_attach()
takes cpuset_mutex
cpuset_reserve_dl_bw()
P's total_bw += T's bandwidth
drops cpuset_mutex
rebuild_sched_domains()
takes cpuset_mutex
dl_rebuild_rd_accounting()
clears R's and P's total_bw
counts T in R (not moved yet)
drops cpuset_mutex
cpuset_attach()
takes cpuset_mutex
cpuset_attach_task(T)
set_cpus_allowed_dl(T)
R's total_bw -= T's bandwidth
records a CPU of P
P now lacks the bandwidth of T, and admits deadline tasks beyond its
capacity. When T leaves P, P's total_bw underflows. In testing, the
race left a destination admitting 3 tasks of 10% too many, or P
admitting no task at all.
Fix this by waiting for an attach in progress in
rebuild_sched_domains_cpuslocked() and in cpuset_full_lock(). The
latter then takes its locks in the order of the writes of cpus and
cpus.partition. A rebuild from CPU hotplug finds no attach in progress.
The online, offline and kill of a cpuset never wait: they hold
cgroup_mutex, which the attach holds throughout.
The move_vs_rebuild case of deadline_bw_test, added later in this
series, tests this race. Without this patch, it fails in 5 of 8 VMs with
8 CPUs. With it, the case passes in 8 of 8. The next patch needs this
wait too: without it, it would give back a reservation that a rebuild
dropped already.
Found by the Sashiko AI reviews of v2 (the race) and v3 (the cgroup v1
writes) of this series.
Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/cgroup/cpuset.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 194a438e25c3..51b6be460794 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -296,17 +296,20 @@ void lockdep_assert_cpuset_lock_held(void)
lockdep_assert_held(&cpuset_mutex);
}
+static void wait_attach_done_lock(void) __acquires(&cpuset_mutex);
+
/**
* cpuset_full_lock - Acquire full protection for cpuset modification
*
* Takes both CPU hotplug read lock (cpus_read_lock()) and cpuset mutex
- * to safely modify cpuset data.
+ * to safely modify cpuset data. Waits for an attach in progress before
+ * it takes cpuset mutex.
*/
void cpuset_full_lock(void)
{
mutex_lock(&cpuset_top_mutex);
cpus_read_lock();
- mutex_lock(&cpuset_mutex);
+ wait_attach_done_lock();
}
void cpuset_full_unlock(void)
@@ -1005,7 +1008,14 @@ void rebuild_sched_domains_locked(void)
static void rebuild_sched_domains_cpuslocked(void)
{
- mutex_lock(&cpuset_mutex);
+ /*
+ * Wait for an attach in progress. cpuset_can_attach() has reserved
+ * the deadline bandwidth of the moving tasks in their new root
+ * domain, and cpuset_attach() moves it on the strength of that
+ * reservation. A rebuild of the accounting in between would drop
+ * the reservation.
+ */
+ wait_attach_done_lock();
rebuild_sched_domains_locked();
mutex_unlock(&cpuset_mutex);
}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (21 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 22/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:18 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 24/40] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
` (16 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. Commit 710da3c8ea7d ("sched/core: Prevent race condition
between cpuset and __sched_setscheduler()") added that lock. It keeps
the cpuset code from changing the root domains while the deadline
admission test runs.
But a cpuset attach drops cpuset_mutex between cpuset_can_attach() and
cpuset_attach(). A sched_setattr() can run in between. Take a process
in the root cgroup. It has deadline threads, and a thread T that is not
a deadline task. The process is moved into a partition root P:
echo PID > P/cgroup.procs sched_setattr(T, SCHED_DEADLINE)
cpuset_can_attach()
lock cpuset_mutex
reserve room in P for the deadline
threads: T is not one of them
unlock cpuset_mutex
lock cpuset_mutex
admit T in the root domain it
is still in
unlock cpuset_mutex
cpuset_attach()
lock cpuset_mutex
move T to P, where nothing is
reserved for it
unlock cpuset_mutex
The attach still moves the bandwidth of T. It is freed in the root
domain T left, and P has no room reserved to count it. No root domain
counts it any more, and P can admit deadline tasks beyond its capacity.
In testing, P was a one-CPU partition root, which admits 95%. The
process had two sleeping deadline threads of 30% each, and
sched_setattr() made T a deadline task of 40%. The move and the
sched_setattr() both succeeded, with no room in P for the three. With
threads of 20% and a T of 30%, P then admitted five more tasks of 10%,
where it had room for two.
Fix this by taking cpuset_mutex in __sched_setscheduler() only when no
attach is in progress. cpuset_partition_write() and
cpuset_hotplug_update_tasks() already wait for an attach that way. A
deadline policy change then comes before the attach, and is part of its
reservation, or after it.
With this patch, and no room in P for the three, the two never both
succeed. If the sched_setattr() comes first, the move is refused with
EBUSY. If it comes after, T has its new CPUs and has not run on them.
The call is then refused with EPERM, as for any sleeping task moved to
another partition.
sched_setaffinity() on a deadline task can run in between as well. A
later patch of this series reserves the bandwidth of the task on one CPU
of its new affinity, and records that CPU with the task. It is
"cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to". A new affinity in between could leave that CPU out. cpuset_attach()
would then move no bandwidth, and the task would stay counted in the
root domain it left. So sched_setaffinity() waits the same way when the
task is a deadline task.
sched_setaffinity() tests the policy with no lock, before it decides to
wait. A sched_setattr() can make the task a deadline task right after
that test. So test the policy again under the runqueue lock, where the
mask changes. If the task is a deadline task there and the caller did
not wait, the call starts over, and waits. The new SCA_DL_NOWAIT flag
asks for that test. After a race with a cpuset update,
__sched_setaffinity() sets the mask a second time, within the cpuset.
That second change waits for an attach too, so that the test cannot
skip it.
For user-space, these calls now wait until a cpuset attach in progress
ends:
- sched_setattr() and sched_setscheduler() on a deadline task
- sched_setattr() that makes a task a deadline task
- sched_setaffinity() on a deadline task
The attach can be the move of any process. The wait is not
interruptible: a signal does not end it. Outside that window, the
results of these calls are unchanged.
Three cases of deadline_bw_test, added later in this series, test this.
move_became_deadline and move_became_deadline_no_room fail without this
patch. move_affinity_changed fails once the bandwidth is reserved on
one CPU, if sched_setaffinity() does not wait.
A fourth case, move_affinity_became_deadline, covers the second test of
the policy. Its window is a few instructions wide. A delay added after
the first test, for testing only, widens it. The case then fails without
the second test in 3 guests of 3. It passes with it.
Fixes: 710da3c8ea7d ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 11 +++++++++
kernel/sched/core.c | 10 ++++++++
kernel/sched/sched.h | 2 ++
kernel/sched/syscalls.c | 54 ++++++++++++++++++++++++++++++++++-------
5 files changed, 70 insertions(+), 9 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index db1cc2fc0324..5d0c01a25354 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -73,6 +73,7 @@ extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
extern void cpuset_update_active_cpus(void);
extern void cpuset_lock(void);
+extern void cpuset_lock_no_attach(void);
extern void cpuset_unlock(void);
extern void lockdep_assert_cpuset_lock_held(void);
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
@@ -192,6 +193,7 @@ static inline void cpuset_update_active_cpus(void)
}
static inline void cpuset_lock(void) { }
+static inline void cpuset_lock_no_attach(void) { }
static inline void cpuset_unlock(void) { }
static inline void lockdep_assert_cpuset_lock_held(void) { }
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 51b6be460794..ce9c57617f49 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -378,6 +378,17 @@ static void wait_attach_done_lock(void)
}
}
+/**
+ * cpuset_lock_no_attach - Acquire cpuset_mutex with no task attach in progress
+ *
+ * A task attach drops cpuset_mutex between cpuset_can_attach() and
+ * cpuset_attach(). Use this to keep a change out of that window.
+ */
+void cpuset_lock_no_attach(void)
+{
+ wait_attach_done_lock();
+}
+
static inline void check_insane_mems_config(nodemask_t *nodes)
{
if (!cpusets_insane_config() &&
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 2a964286a7cd..52c291de384e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,16 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * sched_setaffinity() tested the policy with no lock, and did not
+ * wait for a cpuset attach in progress. @p became a deadline task
+ * since: let the caller wait and try again.
+ */
+ if ((ctx->flags & SCA_DL_NOWAIT) && task_has_dl_policy(p)) {
+ ret = -EAGAIN;
+ goto out;
+ }
+
/*
* Check under the same lock that __sched_setscheduler() holds for
* its admission test, so that p cannot become SCHED_DEADLINE
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9e3ca69abc36..2087ad70c56e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2985,6 +2985,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_DL_CHECK 0x10
/* The caller made room for the deadline bandwidth in the new root domain. */
#define SCA_DL_BW_RESERVED 0x20
+/* The caller did not wait for a cpuset attach: -EAGAIN for a deadline task. */
+#define SCA_DL_NOWAIT 0x40
extern void update_group_capacity(struct sched_domain *sd, int cpu);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4f47b17151ed..4aa3ccbff320 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -564,10 +564,15 @@ int __sched_setscheduler(struct task_struct *p,
* information. Do not take cpuset_mutex when pi is unset: the
* caller is normalize_rt_tasks(), which holds tasklist_lock and
* cannot sleep.
+ *
+ * Wait for a task attach in progress. cpuset_can_attach() reserved
+ * the bandwidth of the deadline tasks it saw, and cpuset_attach()
+ * moves them. A task that enters or leaves the class in between
+ * would not match that reservation.
*/
if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
- cpuset_lock();
+ cpuset_lock_no_attach();
dl_accounting_lock();
}
@@ -1271,9 +1276,21 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
/*
* The cpuset mask must win here, as it does when
* cpuset_update_tasks_cpumask() sets it on a deadline task.
+ *
+ * A sched_setattr() may have made @p a deadline task since
+ * the first call. SCA_DL_NOWAIT would then skip this second
+ * call. Wait for a cpuset attach in progress instead, as
+ * sched_setaffinity() does for a deadline task.
*/
ctx->flags &= ~SCA_DL_CHECK;
- __set_cpus_allowed_ptr(p, ctx);
+ if (ctx->flags & SCA_DL_NOWAIT) {
+ ctx->flags &= ~SCA_DL_NOWAIT;
+ cpuset_lock_no_attach();
+ __set_cpus_allowed_ptr(p, ctx);
+ cpuset_unlock();
+ } else {
+ __set_cpus_allowed_ptr(p, ctx);
+ }
retval = -EINVAL;
}
@@ -1289,6 +1306,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
struct affinity_context ac;
struct cpumask *user_mask;
int retval;
+ bool dl;
CLASS(find_get_task, p)(pid);
if (!p)
@@ -1318,13 +1336,31 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
return -ENOMEM;
}
- ac = (struct affinity_context){
- .new_mask = in_mask,
- .user_mask = user_mask,
- .flags = SCA_USER,
- };
-
- retval = __sched_setaffinity(p, &ac);
+ /*
+ * Wait for a cpuset attach in progress before changing the affinity
+ * of a deadline task. cpuset_can_attach() reserved the bandwidth of
+ * the task for the CPUs it moves to, and cpuset_attach() moves it
+ * there. A new affinity in between could change those CPUs.
+ *
+ * The policy is tested here with no lock. A task that becomes a
+ * deadline task after this test is found under its runqueue lock,
+ * with SCA_DL_NOWAIT. Wait and try again then.
+ */
+ for (;;) {
+ dl = task_has_dl_policy(p);
+ ac = (struct affinity_context){
+ .new_mask = in_mask,
+ .user_mask = user_mask,
+ .flags = SCA_USER | (dl ? 0 : SCA_DL_NOWAIT),
+ };
+ if (dl)
+ cpuset_lock_no_attach();
+ retval = __sched_setaffinity(p, &ac);
+ if (dl)
+ cpuset_unlock();
+ if (dl || retval != -EAGAIN)
+ break;
+ }
kfree(ac.user_mask);
return retval;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 24/40] cgroup/cpuset: Give back the bandwidth an attach did not move
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (22 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 25/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
` (15 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach()
fails") reserves the bandwidth of an attach as one sum.
cpuset_can_attach() adds up the bandwidth of the deadline tasks it
moves. Once all tasks pass their checks, it reserves the sum in the
destination root domain. If another controller refuses the attach,
cpuset_cancel_attach() frees the sum. If the attach goes through, the
move of each task uses its part of the sum.
But a task can leave SCHED_DEADLINE between cpuset_can_attach() and
cpuset_attach(). The previous patch makes sched_setattr() wait for the
attach. sysrq-n does not wait: it cannot sleep. The task then moves no
bandwidth. Its part of the reservation stays in the destination's
total_bw, until the next rebuild of the accounting. The destination can
then refuse deadline tasks that would fit.
Fix this by keeping the tally of what the attach moves, in
sum_moved_dl_bw. Once the attach is through, give back what it did not
move.
set_cpus_allowed_dl() decides the move under the runqueue lock.
cpuset_attach_task() does not hold it. So the tally counts what
set_cpus_allowed_dl() reports in the affinity context.
The random_mix case of deadline_bw_test, added later in this series,
found the bug. It could no longer lower a fair server in a partition
root with no room left.
The move_normalized case, added with that test, fires sysrq-n while a
process with a sleeping deadline thread of 30% moves into a one-CPU
partition root. Without this patch, the partition kept the 30% in 3
guests of 3. It admitted 6 tasks of 10% where it had room for 9.
Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 3 ++-
kernel/cgroup/cpuset-internal.h | 2 ++
kernel/cgroup/cpuset.c | 30 +++++++++++++++++++++++++-----
kernel/sched/core.c | 12 ++++++++----
kernel/sched/deadline.c | 4 ++++
kernel/sched/sched.h | 2 ++
6 files changed, 43 insertions(+), 10 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index dee4d4df624c..471cd27894ff 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1973,7 +1973,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
*/
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
- const struct cpumask *new_mask);
+ const struct cpumask *new_mask,
+ u64 *moved);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index bf847953c78d..93aa46d76b04 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -170,6 +170,8 @@ struct cpuset {
int nr_migrate_dl_tasks;
/* DL bandwidth that needs destination reservation for this attach. */
u64 sum_migrate_dl_bw;
+ /* The part of sum_migrate_dl_bw moved. The rest is given back. */
+ u64 sum_moved_dl_bw;
/*
* CPU used for temporary DL bandwidth allocation during attach;
* -1 if no DL bandwidth was allocated in the current attach.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ce9c57617f49..2f39c3d12550 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3140,10 +3140,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
llist_for_each_entry_safe(cs, next, lnode, attach_node) {
init_llist_node(&cs->attach_node);
if (cs->nr_migrate_dl_tasks) {
- if (cancel && cs->dl_bw_cpu >= 0)
- dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+ /*
+ * Give back the bandwidth the attach did not move.
+ * If the attach is cancelled, that is all of the
+ * reservation. Otherwise, it is what no move used,
+ * such as the part for a task that left
+ * SCHED_DEADLINE. The class it left releases its own
+ * bandwidth, from the root domain that counts it.
+ */
+ if (cs->dl_bw_cpu >= 0) {
+ u64 bw = cancel ? cs->sum_migrate_dl_bw :
+ cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+
+ if (bw)
+ dl_bw_free(cs->dl_bw_cpu, bw);
+ }
cs->nr_migrate_dl_tasks = 0;
cs->sum_migrate_dl_bw = 0;
+ cs->sum_moved_dl_bw = 0;
cs->dl_bw_cpu = -1;
}
}
@@ -3303,10 +3317,16 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- if (reserved)
- WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
- else
+ if (reserved) {
+ u64 moved;
+
+ /* Count what the move reports it moved. */
+ WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
+ &moved));
+ cs->sum_moved_dl_bw += moved;
+ } else {
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ }
if (cpuset_v2() && !attach_ctx.mems_updated)
return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 52c291de384e..229b7bfa9634 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3301,17 +3301,21 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
/*
* Like set_cpus_allowed_ptr(), for a caller that has made room for the
* deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p. Return 0 on success, or a negative error
- * code.
+ * bandwidth there with @p, and set @moved to the bandwidth moved, 0
+ * if none. Return 0 on success, or a negative error code.
*/
-int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+int set_cpus_allowed_bw_reserved(struct task_struct *p,
+ const struct cpumask *new_mask, u64 *moved)
{
struct affinity_context ac = {
.new_mask = new_mask,
.flags = SCA_DL_BW_RESERVED,
};
+ int ret;
- return __set_cpus_allowed_ptr(p, &ac);
+ ret = __set_cpus_allowed_ptr(p, &ac);
+ *moved = ac.dl_bw_moved;
+ return ret;
}
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index aba03fd6b381..79c99e3bc156 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3629,6 +3629,9 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* Otherwise, the bandwidth stays where it is counted, until the next
* rebuild of the accounting. Moving it without room would leave no
* root domain counting it.
+ *
+ * Report in @ctx the bandwidth moved. The caller made its own test
+ * without the runqueue lock, so it must count only what this reports.
*/
if ((ctx->flags & SCA_DL_BW_RESERVED) &&
dl_task_needs_bw_move(p, ctx->new_mask)) {
@@ -3657,6 +3660,7 @@ static void set_cpus_allowed_dl(struct task_struct *p,
cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
p->dl.dl_bw_gen = dl_bw_gen;
+ ctx->dl_bw_moved = p->dl.dl_bw;
}
set_cpus_allowed_common(p, ctx);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 2087ad70c56e..661da2b5c278 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2688,6 +2688,8 @@ extern const u32 sched_prio_to_wmult[40];
struct affinity_context {
const struct cpumask *new_mask;
struct cpumask *user_mask;
+ /* Out: the deadline bandwidth moved to the new root domain. */
+ u64 dl_bw_moved;
unsigned int flags;
};
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 25/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (23 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 24/40] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
` (14 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the source when the
task's affinity changes. The reservation goes to the root domain of the
first active CPU of the destination cpuset. So cpuset_can_attach()
treats the cpuset as one root domain.
But a cpuset can span more than one root domain. With isolcpus=domain,
the root cpuset holds the isolated CPUs, which are in the default root
domain. A task also keeps the affinity set by sched_setaffinity(). Its
new CPUs can then all be in a root domain that got no reservation. The
task runs there, and its bandwidth is reserved in the other root domain.
Take a machine with CPUs 0-4, booted with isolcpus=domain,1,2. P is a
partition root of CPU 4. A deadline task T, pinned to CPUs 1-2, is moved
into P and back to the root cgroup:
total_bw of the root domain of
admin, task T CPUs 0, 3 CPUs 1, 2 CPU 4 (P)
sched_setaffinity(T, CPUs 1-2)
sched_setattr(T, SCHED_DEADLINE) + bw
echo T > P/cgroup.procs
cpuset_can_attach()
reserves on CPU 4 + bw
set_cpus_allowed_dl()
frees where recorded, CPU 1 - bw
records CPU 4
T runs on CPU 4
echo T > cgroup.procs of the root
cpuset_can_attach()
reserves on CPU 0 + bw
__set_cpus_allowed_ptr()
keeps T's affinity: CPUs 1-2
set_cpus_allowed_dl()
frees where recorded, CPU 4 - bw
records CPU 1
T runs on CPU 1
T exits
releases where recorded, CPU 1 - bw
--------- --------- ---------
left after T exits + bw - bw 0
T ran in the root domain of CPUs 1 and 2 without being counted there.
In testing, the total_bw of that root domain went below zero when T
died. It then refused every deadline task, until the next rebuild of
the accounting.
Fix this by reserving the bandwidth on the CPU that the move records
with the task. The new dl_task_bw_reserve_cpu() works out that CPU and
records it with the task. It is the CPU the task is on, if the task can
stay there. Else it is the first active CPU of the new mask, a
housekeeping one if any. The rebuild of the accounting makes the same
choice, later in this series. cpuset_can_attach() reserves the bandwidth
per such CPU, and refuses with EBUSY as before. cpuset_attach_task()
moves the bandwidth to the CPU recorded, and does not work it out again:
the task may have run on another CPU since. cpuset_attach() gives back
the rest.
The record is a new field of the task, dl_bw_resv_cpu.
cpuset_attach_task() moves the bandwidth of a task only if the task has
that record. Juri Lelli suggested to track the tasks that
cpuset_can_attach() counts. A task it did not count cannot use the room
made for another task.
A new mask can also span both root domains. A running task would then go
to the CPU that cpumask_any_and_distribute() picks. A sleeping one would
wake up where select_fallback_rq() picks. Either CPU can be in the root
domain without the reservation. So __set_cpus_allowed_ptr_locked() takes
the CPU with the reservation, if cpumask_any_and_distribute() picked a
CPU of the other root domain. A sleeping task gets the CPU chosen as its
wakeup CPU.
A new mask that still holds an active CPU of the root domain counting
the task moves no bandwidth at the attach. The task can still land in
the other root domain, if it must leave its CPU. A later patch of this
series moves the bandwidth there, without a check for room. It is
"sched/deadline: Move a task's bandwidth to the root domain it runs in".
The sums are kept per CPU, not per cpuset: one cpuset can need room in
two root domains. One set of sums is enough, since cgroup_mutex lets
one attach run at a time. guarantee_active_cpus() now takes the cpuset
to start from.
Later in this series, deadline_bw_test gets four cases for this:
move_isolated, move_isolated_refused, move_isolated_asleep and
move_spanning. Without this patch, all four fail with
isolcpus=domain,1,2. With isolcpus=domain,0,1, move_isolated,
move_spanning and six other move cases fail. The case
move_back_cpu_offline tests a mask that holds an offline CPU.
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 8 +-
include/linux/sched/deadline.h | 15 ++-
kernel/cgroup/cpuset-internal.h | 12 --
kernel/cgroup/cpuset.c | 216 +++++++++++++++++++-------------
kernel/sched/core.c | 27 +++-
kernel/sched/deadline.c | 133 ++++++++++++++++----
kernel/sched/sched.h | 2 +
kernel/sched/syscalls.c | 5 +-
8 files changed, 282 insertions(+), 136 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 471cd27894ff..2f2f59402f09 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -758,10 +758,16 @@ struct sched_dl_entity {
*
* @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
* counted. If a rebuild dropped it since, there is nothing to remove.
+ *
+ * @dl_bw_resv_cpu is a CPU of the root domain where a cpuset attach
+ * in progress reserved room for the bandwidth, or -1.
+ * cpuset_can_attach() sets it, and cpuset_attach() moves the
+ * bandwidth there.
*/
struct list_head dl_departed_node;
int dl_bw_cpu;
unsigned int dl_bw_gen;
+ int dl_bw_resv_cpu;
/*
* Bits for DL-server functionality. Also see the comment near
@@ -1974,7 +1980,7 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
const struct cpumask *new_mask,
- u64 *moved);
+ int cpu, u64 *moved);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 60be226699bb..cbb7cd71db58 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -50,14 +50,17 @@ extern void dl_accounting_unlock(void);
extern void lockdep_assert_dl_accounting_held(void);
#endif
/*
- * Return whether moving DL task @p to @new_mask requires moving DL
- * bandwidth accounting between root domains. This helper is specific to
- * DL bandwidth move accounting semantics and is shared by
- * cpuset_can_attach() and set_cpus_allowed_dl() so both paths use the
- * same source root-domain test.
+ * Return the CPU that a move of DL task @p to the CPUs of @new_mask
+ * records with its DL bandwidth. Return -1 if the bandwidth stays in the
+ * root domain that counts it. Like the move, keep the CPUs within the
+ * affinity set by the user. cpuset_can_attach() reserves the bandwidth in
+ * the root domain of that CPU. cpuset_attach_task() then moves it there,
+ * and set_cpus_allowed_dl() makes the same test.
*/
-extern bool dl_task_needs_bw_move(struct task_struct *p,
+extern int dl_task_bw_reserve_cpu(struct task_struct *p,
const struct cpumask *new_mask);
+extern int dl_task_bw_reserved_cpu(struct task_struct *p);
+extern void dl_task_bw_unreserve(struct task_struct *p);
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 93aa46d76b04..a83443dfb788 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,18 +166,6 @@ struct cpuset {
*/
bool remote_partition_updating;
- /* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
- int nr_migrate_dl_tasks;
- /* DL bandwidth that needs destination reservation for this attach. */
- u64 sum_migrate_dl_bw;
- /* The part of sum_migrate_dl_bw moved. The rest is given back. */
- u64 sum_moved_dl_bw;
- /*
- * CPU used for temporary DL bandwidth allocation during attach;
- * -1 if no DL bandwidth was allocated in the current attach.
- */
- int dl_bw_cpu;
-
/* Invalid partition error code, not lock protected */
enum prs_errcode prs_err;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 2f39c3d12550..7bdb9e21a7ed 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -272,7 +272,6 @@ struct cpuset top_cpuset = {
.flags = BIT(CS_CPU_EXCLUSIVE) |
BIT(CS_MEM_EXCLUSIVE) | BIT(CS_SCHED_LOAD_BALANCE),
.partition_root_state = PRS_ROOT,
- .dl_bw_cpu = -1,
};
/**
@@ -487,28 +486,24 @@ static inline bool partition_is_populated(struct cpuset *cs,
}
/*
- * Return in pmask the portion of a task's cpusets's cpus_allowed that
- * are online and are capable of running the task. If none are found,
- * walk up the cpuset hierarchy until we find one that does have some
- * appropriate cpus.
+ * Return in pmask the portion of the cpus_allowed of @cs that are online
+ * and are capable of running @tsk. If none are found, walk up the cpuset
+ * hierarchy until we find one that does have some appropriate cpus.
*
* One way or another, we guarantee to return some non-empty subset
* of cpu_active_mask.
*
* Call with callback_lock or cpuset_mutex held.
*/
-static void guarantee_active_cpus(struct task_struct *tsk,
+static void guarantee_active_cpus(struct cpuset *cs, struct task_struct *tsk,
struct cpumask *pmask)
{
const struct cpumask *possible_mask = task_cpu_possible_mask(tsk);
- struct cpuset *cs;
if (WARN_ON(!cpumask_and(pmask, possible_mask, cpu_active_mask)))
cpumask_copy(pmask, cpu_active_mask);
rcu_read_lock();
- cs = task_cs(tsk);
-
while (!cpumask_intersects(cs->effective_cpus, pmask)) {
cs = parent_cs(cs);
if (unlikely(!cs)) {
@@ -630,7 +625,6 @@ struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)
if (!trial)
return NULL;
- trial->dl_bw_cpu = -1;
init_llist_node(&trial->attach_node);
/* Setup cpumask pointer array */
@@ -3106,61 +3100,89 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
return 0;
}
+/*
+ * The deadline bandwidth of the attach in progress, per CPU. When the
+ * bandwidth of a deadline task moves to another root domain,
+ * dl_task_bw_reserve_cpu() returns the CPU that the move records with it.
+ * It also records that CPU with the task. cpuset_can_attach() sums the
+ * bandwidth of these tasks per such CPU. It reserves each sum in the root
+ * domain of its CPU. cpuset_attach_task() moves the bandwidth of each task
+ * to the CPU recorded with it. It counts what it moves, and cpuset_attach()
+ * gives back the rest.
+ * dl_attach_cpus holds the CPUs with a sum. cgroup_mutex lets one attach
+ * run at a time, and cpuset_mutex protects these.
+ */
+struct cpuset_dl_bw {
+ u64 sum;
+ u64 moved; /* the part of sum moved */
+ bool reserved; /* dl_bw_alloc() took sum */
+};
+
+static DEFINE_PER_CPU(struct cpuset_dl_bw, cpuset_dl_bw);
+static cpumask_var_t dl_attach_cpus;
+
static int cpuset_reserve_dl_bw(void)
{
- struct cpuset *cs;
int cpu, ret;
- llist_for_each_entry(cs, dst_cs_head.first, attach_node) {
- if (!cs->sum_migrate_dl_bw)
- continue;
-
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids))
- return -EINVAL;
+ for_each_cpu(cpu, dl_attach_cpus) {
+ struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ ret = dl_bw_alloc(cpu, b->sum);
if (ret)
return ret;
-
- cs->dl_bw_cpu = cpu;
+ b->reserved = true;
}
return 0;
}
/*
- * Clear and optionally apply (@cancel is false) the attach related data in the
- * source or destination cpuset.
+ * Give back the bandwidth the attach reserved and did not move. If the
+ * attach is cancelled, that is all of the reservation. Otherwise, it is
+ * what no move used, such as the part for a task that left
+ * SCHED_DEADLINE. The class it left releases its own bandwidth, from the
+ * root domain that counts it.
*/
-static void clear_attach_data(struct llist_head *head, bool cancel)
+static void cpuset_free_dl_bw(void)
+{
+ int cpu;
+
+ for_each_cpu(cpu, dl_attach_cpus) {
+ struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
+
+ if (b->reserved && b->sum != b->moved)
+ dl_bw_free(cpu, b->sum - b->moved);
+ *b = (struct cpuset_dl_bw){ };
+ }
+ cpumask_clear(dl_attach_cpus);
+}
+
+/* Empty @head, the list of the source or destination cpusets. */
+static void clear_attach_data(struct llist_head *head)
{
struct cpuset *cs, *next;
struct llist_node *lnode = __llist_del_all(head);
- llist_for_each_entry_safe(cs, next, lnode, attach_node) {
+ llist_for_each_entry_safe(cs, next, lnode, attach_node)
init_llist_node(&cs->attach_node);
- if (cs->nr_migrate_dl_tasks) {
- /*
- * Give back the bandwidth the attach did not move.
- * If the attach is cancelled, that is all of the
- * reservation. Otherwise, it is what no move used,
- * such as the part for a task that left
- * SCHED_DEADLINE. The class it left releases its own
- * bandwidth, from the root domain that counts it.
- */
- if (cs->dl_bw_cpu >= 0) {
- u64 bw = cancel ? cs->sum_migrate_dl_bw :
- cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+}
- if (bw)
- dl_bw_free(cs->dl_bw_cpu, bw);
- }
- cs->nr_migrate_dl_tasks = 0;
- cs->sum_migrate_dl_bw = 0;
- cs->sum_moved_dl_bw = 0;
- cs->dl_bw_cpu = -1;
- }
- }
+/*
+ * Protected by cpuset_mutex. cpus_attach is used by cpuset_can_attach() and
+ * cpuset_attach_task(), but we can't allocate it dynamically there. Define
+ * it global and allocate from cpuset_init().
+ */
+static cpumask_var_t cpus_attach;
+
+/* Set @mask to the CPUs that an attach to @cs gives @task. */
+static void cpuset_attach_cpus(struct cpuset *cs, struct task_struct *task,
+ struct cpumask *mask)
+{
+ if (cs != &top_cpuset)
+ guarantee_active_cpus(cs, task, mask);
+ else
+ cpumask_andnot(mask, task_cpu_possible_mask(task),
+ subpartitions_cpus);
}
/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
@@ -3170,7 +3192,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
struct cpuset *cs, *oldcs;
struct task_struct *task;
bool setsched_check;
- int ret;
+ int cpu, ret;
cs = oldcs = NULL;
mutex_lock(&cpuset_mutex);
@@ -3236,15 +3258,20 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
}
- if (dl_task(task)) {
- /*
- * Count all migrating DL tasks for cpuset task accounting.
- * Only tasks that need a root-domain bandwidth move
- * contribute to sum_migrate_dl_bw.
- */
- cs->nr_migrate_dl_tasks++;
- if (dl_task_needs_bw_move(task, cs->effective_cpus))
- cs->sum_migrate_dl_bw += task->dl.dl_bw;
+ if (!dl_task(task))
+ continue;
+ /*
+ * Sum the bandwidth of the deadline tasks that move to another
+ * root domain, per CPU that the move records. That CPU depends
+ * on the CPUs the task gets. A cpuset can span more than one
+ * root domain, and the affinity of a task can keep it in any
+ * of them.
+ */
+ cpuset_attach_cpus(cs, task, cpus_attach);
+ cpu = dl_task_bw_reserve_cpu(task, cpus_attach);
+ if (cpu >= 0) {
+ per_cpu_ptr(&cpuset_dl_bw, cpu)->sum += task->dl.dl_bw;
+ cpumask_set_cpu(cpu, dl_attach_cpus);
}
}
@@ -3268,8 +3295,11 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
out_unlock:
if (ret) {
- clear_attach_data(&src_cs_head, true);
- clear_attach_data(&dst_cs_head, true);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
} else {
attach_ctx.in_progress++;
}
@@ -3280,50 +3310,59 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
static void cpuset_cancel_attach(struct cgroup_taskset *tset)
{
+ struct cgroup_subsys_state *css;
+ struct task_struct *task;
+
mutex_lock(&cpuset_mutex);
dec_attach_in_progress_locked();
- clear_attach_data(&src_cs_head, true);
- clear_attach_data(&dst_cs_head, true);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
mutex_unlock(&cpuset_mutex);
}
-/*
- * Protected by cpuset_mutex. cpus_attach is used only by cpuset_attach_task()
- * but we can't allocate it dynamically there. Define it global and
- * allocate from cpuset_init().
- */
-static cpumask_var_t cpus_attach;
-
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
- bool reserved = false;
+ struct cpuset_dl_bw *b = NULL;
struct mm_struct *mm;
+ int cpu = -1;
lockdep_assert_cpuset_lock_held();
- if (cs != &top_cpuset)
- guarantee_active_cpus(task, cpus_attach);
- else
- cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
- subpartitions_cpus);
+ cpuset_attach_cpus(cs, task, cpus_attach);
/*
- * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
- * deadline tasks that cpuset_can_attach() counted. Flag the move, so
- * that the bandwidth of this task moves with it.
+ * cpuset_can_attach() made room for the bandwidth of each deadline
+ * task it moves, in the root domain of the CPU it recorded with the
+ * task. Flag the move with that CPU, so that the bandwidth goes
+ * there. The task may have run on another CPU since: the CPU is not
+ * worked out again.
+ *
+ * A task with no CPU recorded needs no move, or was not a deadline
+ * task in cpuset_can_attach(). Its move is not flagged, so that it
+ * cannot use the room made for another task. Its bandwidth stays
+ * where it is counted.
*/
- if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
- reserved = true;
+ cpu = dl_task_bw_reserved_cpu(task);
+ if (cpu >= 0)
+ b = per_cpu_ptr(&cpuset_dl_bw, cpu);
/*
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- if (reserved) {
+ if (b) {
u64 moved;
/* Count what the move reports it moved. */
WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
- &moved));
- cs->sum_moved_dl_bw += moved;
+ cpu, &moved));
+ b->moved += moved;
+ /*
+ * set_cpus_allowed_dl() cleared the record, unless the task
+ * left SCHED_DEADLINE. Clear it then too.
+ */
+ dl_task_bw_unreserve(task);
} else {
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
}
@@ -3397,6 +3436,13 @@ static void cpuset_attach(struct cgroup_taskset *tset)
if (cpuset_v2() && !attach_ctx.cpus_updated && !attach_ctx.mems_updated) {
llist_for_each_entry(cs, dst_cs_head.first, attach_node)
cs->old_mems_allowed = attach_ctx.nodemask_to;
+ /*
+ * No mask changes, so no bandwidth moves. Forget the CPU
+ * that cpuset_can_attach() recorded with a deadline task, as
+ * cpuset_attach_task() does. A later attach must not find it.
+ */
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
goto out;
}
@@ -3407,8 +3453,9 @@ static void cpuset_attach(struct cgroup_taskset *tset)
schedule_flush_migrate_mm();
cs->old_mems_allowed = attach_ctx.nodemask_to;
out:
- clear_attach_data(&src_cs_head, false);
- clear_attach_data(&dst_cs_head, false);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
dec_attach_in_progress_locked();
mutex_unlock(&cpuset_mutex);
@@ -3913,6 +3960,7 @@ int __init cpuset_init(void)
cpuset1_init(&top_cpuset);
BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&dl_attach_cpus, GFP_KERNEL));
if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT))
cpumask_andnot(isolated_cpus, cpu_possible_mask,
@@ -4211,7 +4259,7 @@ static void __cpuset_cpus_allowed_locked(struct task_struct *tsk, struct cpumask
cs = task_cs(tsk);
if (cs != &top_cpuset)
- guarantee_active_cpus(tsk, pmask);
+ guarantee_active_cpus(cs, tsk, pmask);
/*
* Tasks in the top cpuset won't get update to their cpumasks
* when a hotplug online/offline event happens. So we include all
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 229b7bfa9634..e9f80407f75b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3250,6 +3250,22 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * The caller made room for the deadline bandwidth of @p in the root
+ * domain of ctx->dl_bw_cpu. The new mask can span two root domains:
+ * keep @p in that one. Also wake a sleeping @p up on dest_cpu, since
+ * select_fallback_rq() could pick a CPU of the other one.
+ */
+ if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, cpu_valid_mask) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask)) {
+ if (!cpumask_test_cpu(dest_cpu,
+ cpu_rq(ctx->dl_bw_cpu)->rd->span))
+ dest_cpu = ctx->dl_bw_cpu;
+ if (!task_on_rq_queued(p))
+ p->wake_cpu = dest_cpu;
+ }
+
do_set_cpus_allowed(p, ctx);
return affine_move_task(rq, p, rf, dest_cpu, ctx->flags);
@@ -3300,15 +3316,18 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
/*
* Like set_cpus_allowed_ptr(), for a caller that has made room for the
- * deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p, and set @moved to the bandwidth moved, 0
- * if none. Return 0 on success, or a negative error code.
+ * deadline bandwidth of @p in the root domain of @cpu. If the move records
+ * @cpu with that bandwidth, move the bandwidth there with @p. Set @moved
+ * to the bandwidth moved, 0 if none. Return 0 on success, or a negative
+ * error code.
*/
int set_cpus_allowed_bw_reserved(struct task_struct *p,
- const struct cpumask *new_mask, u64 *moved)
+ const struct cpumask *new_mask, int cpu,
+ u64 *moved)
{
struct affinity_context ac = {
.new_mask = new_mask,
+ .dl_bw_cpu = cpu,
.flags = SCA_DL_BW_RESERVED,
};
int ret;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 79c99e3bc156..a9b8bd019d4b 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3630,44 +3630,63 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* rebuild of the accounting. Moving it without room would leave no
* root domain counting it.
*
- * Report in @ctx the bandwidth moved. The caller made its own test
- * without the runqueue lock, so it must count only what this reports.
+ * cpuset_can_attach() made room for @p in the root domain of
+ * ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
+ * there. Do not work out the CPU again: @p may have run on another
+ * CPU since. __set_cpus_allowed_ptr_locked() keeps @p in that root
+ * domain.
+ *
+ * Report in @ctx the bandwidth moved. The caller gives back the room
+ * that no move used.
*/
if ((ctx->flags & SCA_DL_BW_RESERVED) &&
- dl_task_needs_bw_move(p, ctx->new_mask)) {
+ p->dl.dl_bw_resv_cpu == ctx->dl_bw_cpu) {
unsigned int src_cpu = dl_task_root_domain_cpu(p);
struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
- unsigned int cpu;
+ p->dl.dl_bw_resv_cpu = -1;
/*
- * We now free resources of the root_domain we are migrating
- * off. In the worst case, sched_setattr() may temporary fail
- * until we complete the update. A rebuild that dropped the
- * bandwidth of @p leaves nothing to free. dl_departed_lock
- * keeps a rebuild from changing dl_bw_gen meanwhile.
+ * Nothing moves if @p left the CPUs it was to move to, or if
+ * its root domain already counts it: @p ran there since.
*/
- guard(raw_spinlock)(&dl_departed_lock);
- if (p->dl.dl_bw_gen == dl_bw_gen) {
- raw_spin_lock(&src_dl_b->lock);
- __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
- raw_spin_unlock(&src_dl_b->lock);
- }
+ if (p->dl.dl_bw && !dl_entity_is_special(&p->dl) &&
+ cpu_active(ctx->dl_bw_cpu) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask) &&
+ cpu_rq(src_cpu)->rd != cpu_rq(ctx->dl_bw_cpu)->rd) {
+ /*
+ * We now free resources of the root_domain we are
+ * migrating off. In the worst case, sched_setattr()
+ * may temporary fail until we complete the update. A
+ * rebuild that dropped the bandwidth of @p leaves
+ * nothing to free. dl_departed_lock keeps a rebuild
+ * from changing dl_bw_gen meanwhile.
+ */
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ raw_spin_lock(&src_dl_b->lock);
+ __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+ raw_spin_unlock(&src_dl_b->lock);
+ }
- /*
- * cpuset_can_attach() made room in the root domain of the new
- * mask. That root domain counts the bandwidth from now on.
- */
- cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
- p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
- p->dl.dl_bw_gen = dl_bw_gen;
- ctx->dl_bw_moved = p->dl.dl_bw;
+ /* Record the CPU the caller made room on. */
+ p->dl.dl_bw_cpu = ctx->dl_bw_cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
+ ctx->dl_bw_moved = p->dl.dl_bw;
+ }
}
set_cpus_allowed_common(p, ctx);
}
-bool dl_task_needs_bw_move(struct task_struct *p,
- const struct cpumask *new_mask)
+/*
+ * Return the CPU that a move of @p to the CPUs of @new_mask records with
+ * its bandwidth. That is the CPU @p is on, if @p can stay there. Else it
+ * is the first active CPU of @new_mask, a housekeeping one if there is
+ * one. Return -1 if the bandwidth stays in the root domain that counts
+ * it. The caller holds the rq lock of @p.
+ */
+static int __dl_task_bw_move_cpu(struct task_struct *p,
+ const struct cpumask *new_mask)
{
unsigned int cpu;
@@ -3677,10 +3696,69 @@ bool dl_task_needs_bw_move(struct task_struct *p,
* their bandwidth: dl_add_task_root_domain() leaves them out.
*/
if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
- return false;
+ return -1;
+ /*
+ * Test the active CPUs only. The default root domain also holds the
+ * offline CPUs, and the mask of the top cpuset holds them too.
+ */
cpu = dl_task_root_domain_cpu(p);
- return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
+ if (cpumask_intersects_and(cpu_rq(cpu)->rd->span, new_mask,
+ cpu_active_mask))
+ return -1;
+ /*
+ * A task moved asleep is still on a CPU of the root domain it left.
+ * If the new mask holds that CPU, the task wakes up there.
+ */
+ cpu = task_cpu(p);
+ if (cpu_active(cpu) && cpumask_test_cpu(cpu, new_mask))
+ return cpu;
+ cpu = cpumask_first_and_and(new_mask, cpu_active_mask,
+ housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_any_and(new_mask, cpu_active_mask);
+ return cpu < nr_cpu_ids ? (int)cpu : -1;
+}
+
+/*
+ * Work out the CPU that a move of @p to the CPUs of @new_mask records
+ * with its bandwidth. Record it with @p: a cpuset attach reserves room
+ * on it. Return it, or -1 if the bandwidth stays where it is.
+ */
+int dl_task_bw_reserve_cpu(struct task_struct *p, const struct cpumask *new_mask)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+ int cpu;
+
+ /* Like __set_cpus_allowed_ptr(), keep within the user affinity. */
+ rq = task_rq_lock(p, &rf);
+ if (p->user_cpus_ptr &&
+ cpumask_and(rq->scratch_mask, new_mask, p->user_cpus_ptr))
+ new_mask = rq->scratch_mask;
+ cpu = __dl_task_bw_move_cpu(p, new_mask);
+ p->dl.dl_bw_resv_cpu = cpu;
+ task_rq_unlock(rq, p, &rf);
+ return cpu;
+}
+
+/* Return the CPU that dl_task_bw_reserve_cpu() recorded with @p, or -1. */
+int dl_task_bw_reserved_cpu(struct task_struct *p)
+{
+ return READ_ONCE(p->dl.dl_bw_resv_cpu);
+}
+
+/* Forget the CPU that dl_task_bw_reserve_cpu() recorded with @p. */
+void dl_task_bw_unreserve(struct task_struct *p)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+
+ if (READ_ONCE(p->dl.dl_bw_resv_cpu) < 0)
+ return;
+ rq = task_rq_lock(p, &rf);
+ p->dl.dl_bw_resv_cpu = -1;
+ task_rq_unlock(rq, p, &rf);
}
/* Assumes rq->lock is held */
@@ -4535,6 +4613,7 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
dl_se->dl_bw_cpu = -1;
+ dl_se->dl_bw_resv_cpu = -1;
INIT_LIST_HEAD(&dl_se->dl_departed_node);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 661da2b5c278..3cc287785b91 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2690,6 +2690,8 @@ struct affinity_context {
struct cpumask *user_mask;
/* Out: the deadline bandwidth moved to the new root domain. */
u64 dl_bw_moved;
+ /* In, with SCA_DL_BW_RESERVED: the CPU the caller made room on. */
+ int dl_bw_cpu;
unsigned int flags;
};
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4aa3ccbff320..241dedd2aaac 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1339,8 +1339,9 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
/*
* Wait for a cpuset attach in progress before changing the affinity
* of a deadline task. cpuset_can_attach() reserved the bandwidth of
- * the task for the CPUs it moves to, and cpuset_attach() moves it
- * there. A new affinity in between could change those CPUs.
+ * the task on one CPU, and recorded that CPU with the task.
+ * cpuset_attach() moves the bandwidth there. A new affinity in
+ * between could leave that CPU out, and the bandwidth would not move.
*
* The policy is tested here with no lock. A task that becomes a
* deadline task after this test is found under its runqueue lock,
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (24 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 25/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:31 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
` (13 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A rebuild of the accounting counts each deadline task on the first
active CPU of its cpuset. Commit 318e18ed22e8 ("sched/deadline: Walk up
cpuset hierarchy to decide root domain when hot-unplug") made this
choice. It relies on the bandwidth check to keep such a CPU active.
This guess would go wrong in three ways:
- A cpuset, like the root one, can hold CPUs isolated by
isolcpus="domain" beside housekeeping CPUs. If an isolated CPU comes
first, the task would be counted in the default root domain. Its own
root domain would then be short of that bandwidth.
- A task confined to isolated CPUs can see them all go offline. The
rebuild would then find no active CPU, and hit its BUG_ON().
- dl_task_offline_migration() moves a throttled task, with its
bandwidth, off CPUs that went offline. The rebuild would count it in
its cpuset, not where it moved.
In the first and third cases, a root domain would admit deadline tasks
beyond its capacity.
Fix this by choosing the CPU in this order:
1. The CPU the task is on, if it is active and in the task's mask. A
mask can hold CPUs of two root domains, for instance after
select_fallback_rq() widened it. Also the CPU the task is queued on
out of its mask, if it is active and recorded with the bandwidth.
Without admission control, dl_task_offline_migration() can put a
throttled task there. It runs there until it blocks.
2. dl_bw_cpu, the CPU recorded with the bandwidth, if it is active and
in the mask. A task moved asleep is not on its new CPUs yet.
3. The first active CPU of the mask, a housekeeping one if any.
4. With no active CPU in the mask: dl_bw_cpu if it is active, else an
active CPU of the cpuset.
The mask of the task comes before its cpuset. A partition change sets
the task masks before its rebuild, and the housekeeping mask after.
Leave out a task with no active CPU, instead of the BUG_ON(). dl_bw_gen
then keeps its release from removing anything.
dl_task_offline_migration() counts it again when it moves it.
A task that select_fallback_rq() moves to another root domain still runs
where its bandwidth is not counted. The next patch moves the bandwidth
with it.
The Sashiko AI reviews of v1 to v4 found these cases.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Fixes: 318e18ed22e8 ("sched/deadline: Walk up cpuset hierarchy to decide root domain when hot-unplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 88 +++++++++++++++++++++++++++++++++++------
1 file changed, 77 insertions(+), 11 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a9b8bd019d4b..9ff79141c630 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -927,14 +927,20 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
__dl_sub(dl_b, p->dl.dl_bw,
cpumask_weight(src_rq->rd->span));
raw_spin_unlock(&dl_b->lock);
-
- dl_b = &later_rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_add(dl_b, p->dl.dl_bw,
- cpumask_weight(later_rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
}
+ /*
+ * Count the bandwidth in the root domain the task moves to.
+ * Do it also when a rebuild left the task out, for want of an
+ * active CPU. A rebuild that reaches the task after this finds
+ * its dl_bw_gen current, and leaves it alone.
+ */
+ dl_b = &later_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw,
+ cpumask_weight(later_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
p->dl.dl_bw_cpu = later_rq->cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
}
set_task_cpu(p, later_rq->cpu);
@@ -3820,6 +3826,60 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
cpuset_cpus_allowed_locked(p, cpus);
}
+/*
+ * Return a CPU of the root domain where a rebuild of the accounting should
+ * count the bandwidth of @p, or nr_cpu_ids if there is none.
+ *
+ * Prefer the CPU @p is on, if it is active and in the mask of @p. The
+ * task runs, or wakes up, in the root domain of that CPU. A mask can hold
+ * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
+ * isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * Prefer that CPU also if @p is queued there, out of its mask, and the
+ * CPU is the one recorded. Without admission control,
+ * dl_task_offline_migration() can put a throttled task and its bandwidth
+ * on any active CPU. The task runs there until it blocks.
+ *
+ * Else prefer the CPU recorded with the bandwidth, if it is active and in
+ * the mask. It is in the root domain that admitted the bandwidth, or that
+ * its last move put it in. A task moved asleep is still on a CPU of the
+ * root domain it left, out of its mask. Else take the first active CPU of
+ * the mask, a housekeeping one if there is one. A task on isolated CPUs
+ * that all went offline can have no active CPU at all.
+ */
+static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
+{
+ struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+ int recorded = p->dl.dl_bw_cpu;
+ unsigned int cpu;
+
+ /*
+ * Look at the mask of the task first. A change of the cpusets sets
+ * the mask of each task before its rebuild. The housekeeping mask
+ * that dl_get_task_effective_cpus() reads is updated after it. If no
+ * CPU of the mask is active, take the CPU recorded if it is active.
+ * dl_task_offline_migration() moved the task there, or an earlier
+ * rebuild counted it there. Else ask the cpuset.
+ */
+ cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+ cpu = task_cpu(p);
+ if (cpumask_test_cpu(cpu, msk) ||
+ (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
+ return cpu;
+ if (cpumask_empty(msk)) {
+ if (recorded >= 0 && cpu_active(recorded))
+ return recorded;
+ dl_get_task_effective_cpus(p, msk);
+ cpumask_and(msk, msk, cpu_active_mask);
+ } else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
+ return recorded;
+ }
+ cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+ if (cpu < nr_cpu_ids)
+ return cpu;
+ return cpumask_first(msk);
+}
+
/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
void dl_add_task_root_domain(struct task_struct *p)
{
@@ -3827,7 +3887,6 @@ void dl_add_task_root_domain(struct task_struct *p)
struct rq *rq;
struct dl_bw *dl_b;
unsigned int cpu;
- struct cpumask *msk;
raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
/*
@@ -3842,10 +3901,17 @@ void dl_add_task_root_domain(struct task_struct *p)
return;
}
- msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
- dl_get_task_effective_cpus(p, msk);
- cpu = cpumask_first_and(cpu_active_mask, msk);
- BUG_ON(cpu >= nr_cpu_ids);
+ cpu = dl_task_rebuild_cpu(p);
+ if (cpu >= nr_cpu_ids) {
+ /*
+ * No active CPU can count the bandwidth of this task. Leave it
+ * out of this rebuild. dl_bw_gen then keeps its release from
+ * removing anything. A later rebuild, or
+ * dl_task_offline_migration(), counts it again.
+ */
+ raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+ return;
+ }
raw_spin_lock(&dl_departed_lock);
/*
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to the root domain it runs in
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (25 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:29 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
` (12 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
When a CPU goes offline, select_fallback_rq() picks a new CPU for a
deadline task asleep on it. balance_push() moves a queued task the same
way. The new CPU can be in another root domain. With isolcpus="domain",
the isolated CPUs are in the default root domain. A cpuset that is not
a partition can hold CPUs of both root domains.
Commit 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all
levels after offline migration") fixed this for a throttled task.
dl_task_offline_migration() moves its bandwidth with it. A task that
select_fallback_rq() places still moves no bandwidth. So it runs in a
root domain that does not count its bandwidth, until the next rebuild.
That root domain can admit deadline tasks beyond its capacity.
Fix this in migrate_task_rq_dl(). The CPU recorded with the bandwidth
gives the root domain that counts it. If the new CPU is in another root
domain, move the bandwidth there. Record the new CPU. As
dl_task_offline_migration() does, add the bandwidth even without room,
since the task runs there. Add it also if a rebuild left the task out.
Do not move it to a CPU that is not active. A migration-disabled task
can wake up on a CPU that goes offline. That CPU is in the default root
domain by then. The task leaves it before it runs anywhere else.
Compare the root domains, not the CPU the task leaves. That CPU can
still be in the mask when select_fallback_rq() picks a CPU of the other
root domain. A task can also wake up on its CPU with no migration while
another root domain counts its bandwidth. A cpuset move or a rebuild can
move the bandwidth while the task sleeps. A change of the mask can then
give that CPU back. So task_woken_dl() makes the same check.
Both checks read the record without dl_departed_lock, so a move inside
a root domain takes no lock. dl_task_move_bw() checks again under
dl_departed_lock. Its caller holds the pi_lock or the rq lock of the
task. Together, the two locks keep the record stable.
Move no bandwidth while the sched domains are rebuilt. The CPUs then
change root domains one by one. Two CPUs can be in different root
domains only because one of them was attached to its new root domain
first. The rebuild of the accounting that follows counts the task.
dl_bw_rebuild_begin() opens this window in
partition_sched_domains_locked(). dl_rebuild_rd_accounting() closes it.
Move no bandwidth for a task blocked on a mutex while it is migrated.
Proxy execution queues such a task on the CPU of the mutex owner, which
can be in another root domain. The task does not run there. It goes
back when it is woken. proxy_set_task_cpu() does not call
migrate_task_rq_dl() today. A proxy execution fix makes it do so, as
set_task_cpu() does [1]. If the task then wakes up in another root
domain, the wake-up moves the bandwidth.
These cases of deadline_bw_test, added later in this series, fail
without this patch:
- hotplug_fallback, hotplug_fallback_moved and
hotplug_fallback_in_mask: a task woken on an isolated CPU stays
counted in the housekeeping root domain.
- no_admission_affinity_moves: without admission control, a task
given an affinity of one isolated CPU stays counted in the
housekeeping root domain.
This follows the Sashiko AI reviews of v4 and v5.
[1] https://lore.kernel.org/all/20261116192638.233125-5-mathieu.desnoyers@efficios.com/
Fixes: 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all levels after offline migration")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched/deadline.h | 1 +
kernel/sched/deadline.c | 152 ++++++++++++++++++++++++++++++---
kernel/sched/topology.c | 3 +
3 files changed, 144 insertions(+), 12 deletions(-)
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index cbb7cd71db58..bdabd54af729 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,7 @@ struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
extern void dl_clear_root_domains(void);
+extern void dl_bw_rebuild_begin(void);
extern void dl_rebuild_rd_accounting(void);
/*
* Keep a rebuild of the root domains' accounting and an admission of
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 9ff79141c630..6c6a4d9cd054 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -304,6 +304,44 @@ static LIST_HEAD(dl_departed_list);
static DEFINE_RAW_SPINLOCK(dl_departed_lock);
static unsigned int dl_bw_gen;
+/*
+ * Odd while the sched domains are rebuilt: from before the first CPU
+ * leaves its root domain until the accounting is rebuilt. Meanwhile the
+ * CPUs change root domains one by one. Housekeeping CPUs pass through
+ * def_root_domain, next to the CPUs that isolcpus="domain" isolates. So
+ * the root domain of the CPU recorded with a task says nothing about
+ * where its bandwidth is counted. The new root domains count no task
+ * yet.
+ *
+ * Changed under dl_departed_lock. dl_task_bw_elsewhere() reads it
+ * without the lock.
+ */
+static unsigned int dl_bw_rebuild_seq;
+
+/* Return true while the sched domains are rebuilt. */
+static inline bool dl_bw_rebuilding(void)
+{
+ return READ_ONCE(dl_bw_rebuild_seq) & 1;
+}
+
+/*
+ * Start a rebuild of the sched domains. Call it before the first CPU
+ * leaves its root domain. dl_rebuild_rd_accounting() ends the rebuild.
+ */
+void dl_bw_rebuild_begin(void)
+{
+ lockdep_assert_held(&sched_domains_mutex);
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+}
+
+static void dl_bw_rebuild_end(void)
+{
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ WARN_ON_ONCE(!(dl_bw_rebuild_seq & 1));
+ WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+}
+
/*
* The number of tasks with a bandwidth: those whose dl_bw is not 0. A
* special entity and a server are not counted. __setparam_dl() and
@@ -2930,11 +2968,86 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
return cpu;
}
-static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
+/*
+ * Return true if the root domain of @cpu may not count the bandwidth of
+ * @p. Either another root domain counts it, or a rebuild left @p out.
+ * Test without dl_departed_lock, so that a move inside a root domain
+ * takes no lock. dl_task_move_bw() tests again under it.
+ *
+ * Return false if @cpu is not active. A migration-disabled task can wake
+ * up on a CPU that goes offline, after it joined the default root domain.
+ * Keep the bandwidth where it is counted. balance_push() moves the task
+ * off that CPU once it enables migration, or its next wakeup does once
+ * the CPU is offline. That migration moves the bandwidth if needed.
+ *
+ * Return false while the sched domains are rebuilt. The CPUs then move
+ * to their new root domains one by one. That alone can make the two root
+ * domains differ. The rebuild of the accounting
+ * that follows counts @p.
+ */
+static bool dl_task_bw_elsewhere(struct task_struct *p, int cpu)
+{
+ int bw_cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+ if (bw_cpu < 0 || !p->dl.dl_bw || !cpu_active(cpu) || dl_bw_rebuilding())
+ return false;
+ return READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen) ||
+ cpu_rq(bw_cpu)->rd != cpu_rq(cpu)->rd;
+}
+
+/*
+ * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
+ * also if a rebuild left @p out. Add it even without room, since @p runs
+ * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
+ */
+static void dl_task_move_bw(struct task_struct *p, int new_cpu)
+{
+ struct dl_bw *dl_b;
+ int cpu;
+
+ if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
+ return;
+
+ guard(raw_spinlock)(&dl_departed_lock);
+ cpu = p->dl.dl_bw_cpu;
+ if (cpu < 0 || dl_bw_rebuilding())
+ return;
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
+ return;
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ dl_b = dl_bw_of(new_cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
+ raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_cpu = new_cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
+}
+
+static void migrate_task_rq_dl(struct task_struct *p, int new_cpu)
{
struct rq_flags rf;
struct rq *rq;
+ /*
+ * The new CPU can be in another root domain than the one that counts
+ * the bandwidth. A mask can hold CPUs of two root domains, and
+ * select_fallback_rq() can widen it. Move the bandwidth there, as
+ * dl_task_offline_migration() does for a throttled task.
+ *
+ * Not for a task that is blocked on a mutex and only queued for
+ * proxy execution. It does not run on the new CPU, so its bandwidth
+ * stays in its root domain. If its wake-up then picks a CPU of another
+ * root domain, the bandwidth moves at that time.
+ */
+ if (unlikely(dl_task_bw_elsewhere(p, new_cpu)) &&
+ !(p->is_blocked && task_on_rq_migrating(p)))
+ dl_task_move_bw(p, new_cpu);
+
if (READ_ONCE(p->__state) != TASK_WAKING)
return;
@@ -3597,6 +3710,15 @@ static void pull_dl_task(struct rq *this_rq)
*/
static void task_woken_dl(struct rq *rq, struct task_struct *p)
{
+ /*
+ * A task can wake up on the CPU it slept on, with no migration,
+ * while another root domain counts its bandwidth. The bandwidth
+ * moved while the task slept out of its mask, and the mask then got
+ * that CPU back. Move the bandwidth to this root domain.
+ */
+ if (unlikely(dl_task_bw_elsewhere(p, cpu_of(rq))))
+ dl_task_move_bw(p, cpu_of(rq));
+
if (!task_on_cpu(rq, p) &&
!test_tsk_need_resched(rq->curr) &&
p->nr_cpus_allowed > 1 &&
@@ -3632,9 +3754,10 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* (different root_domains) entails a bandwidth update. Move the
* bandwidth only if the caller made room for it in the destination,
* as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
- * Otherwise, the bandwidth stays where it is counted, until the next
- * rebuild of the accounting. Moving it without room would leave no
- * root domain counting it.
+ * Otherwise, the bandwidth stays where it is counted, until the task
+ * runs in another root domain (migrate_task_rq_dl(), task_woken_dl()),
+ * or the next rebuild of the accounting. Freeing it here without room
+ * would leave no root domain counting it.
*
* cpuset_can_attach() made room for @p in the root domain of
* ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
@@ -3831,10 +3954,11 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
* count the bandwidth of @p, or nr_cpu_ids if there is none.
*
* Prefer the CPU @p is on, if it is active and in the mask of @p. The
- * task runs, or wakes up, in the root domain of that CPU. A mask can hold
- * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
- * isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * task runs, or most often wakes up, in the root domain of that CPU. A
+ * mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
+ * isolated by isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. A task that then runs in
+ * the other root domain takes its bandwidth along.
* Prefer that CPU also if @p is queued there, out of its mask, and the
* CPU is the one recorded. Without admission control,
* dl_task_offline_migration() can put a throttled task and its bandwidth
@@ -3906,8 +4030,8 @@ void dl_add_task_root_domain(struct task_struct *p)
/*
* No active CPU can count the bandwidth of this task. Leave it
* out of this rebuild. dl_bw_gen then keeps its release from
- * removing anything. A later rebuild, or
- * dl_task_offline_migration(), counts it again.
+ * removing anything. A later rebuild counts it again. So does
+ * the move of the task to an active CPU.
*/
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return;
@@ -4068,8 +4192,10 @@ void dl_rebuild_rd_accounting(void)
* a resume run one for each CPU. A task gets a bandwidth under the
* locks this rebuild holds, so the count cannot leave 0 meanwhile.
*/
- if (!atomic_read(&dl_nr_bw_tasks))
+ if (!atomic_read(&dl_nr_bw_tasks)) {
+ dl_bw_rebuild_end();
return;
+ }
read_lock(&tasklist_lock);
for_each_process_thread(g, p) {
@@ -4083,6 +4209,7 @@ void dl_rebuild_rd_accounting(void)
dl_add_task_root_domain(p);
}
read_unlock(&tasklist_lock);
+ dl_bw_rebuild_end();
}
/*
@@ -4101,7 +4228,8 @@ void dl_clear_root_domains(void)
int cpu;
guard(raw_spinlock_irqsave)(&dl_departed_lock);
- dl_bw_gen++;
+ /* dl_task_bw_elsewhere() reads it without the lock. */
+ WRITE_ONCE(dl_bw_gen, dl_bw_gen + 1);
for_each_possible_cpu(cpu) {
if (!dl_bw_visited(cpu, cookie))
dl_clear_root_domain(cpu_rq(cpu)->rd);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 35e56a45c238..9ec5f38afd28 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3430,6 +3430,9 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
n = ndoms_new;
}
+ /* dl_rebuild_rd_accounting() ends it. */
+ dl_bw_rebuild_begin();
+
/* Destroy deleted domains: */
for (i = 0; i < ndoms_cur; i++) {
for (j = 0; j < n && !new_topology; j++) {
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (26 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:32 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
` (11 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
The root domain that admits a deadline task counts its bandwidth. The
mask of the task can hold CPUs of another root domain. With
isolcpus="domain", the isolated CPUs are in the default root domain. A
cpuset that is not a partition root can hold isolated and housekeeping
CPUs.
The code that picks a CPU for a task does not look at the root domain
that counts it:
- select_fallback_rq() and dl_task_offline_migration() pick a CPU when
the CPU of the task goes offline.
- __set_cpus_allowed_ptr_locked() picks one when the mask changes.
- select_task_rq_dl() can leave a task on the CPU it slept on, which
can be in the other root domain.
- dl_task_rebuild_cpu() picks the CPU that a rebuild counts a task
with. It prefers the CPU the task is on, then a housekeeping CPU.
Each can take the task, or its bandwidth, to the other root domain
while its own root domain still has a CPU for it. The bandwidth then
moves with no admission test, and the root domain it arrives in can
exceed its limit.
Add dl_task_bw_span(), which returns the CPUs of the root domain that
counts a task. Make each of these places try one of those CPUs first.
After a rebuild the root domains are new, and the CPU recorded with the
bandwidth can be offline. So dl_task_rebuild_cpu() cannot use the root
domain. It looks at the kind of the CPU recorded, housekeeping or
isolated. It prefers a CPU of the mask of the same kind.
The CPU recorded can be the one going offline. Its root domain still
counts the task until the rebuild attaches that CPU to the default root
domain. So dl_task_bw_span() still returns that root domain until then.
This covers the common case: a task that runs on the CPU when it goes
offline, which balance_push() moves.
dl_task_bw_span() returns NULL while the sched domains are rebuilt. The
CPUs then change root domains one by one. A housekeeping CPU passes
through the default root domain, which holds the isolated CPUs. Its
span would send a housekeeping task to an isolated CPU. So inside a
rebuild, the CPU is picked as it was before this patch. If that CPU is
in the other root domain, the next wakeup brings the task back.
A known limit remains. A wakeup can pick its CPU inside a rebuild, and
be queued after the rebuild. Its bandwidth can then still move to the
other root domain, with no admission test. This was found by reading,
not reproduced.
A task with no CPU left in its root domain still moves, and its
bandwidth with it. The next two patches refuse the changes that lead
to that.
The case hotplug_keeps_root_domain of deadline_bw_test, added later in
this series, tests this. A cpuset holds two isolated CPUs and two
housekeeping CPUs, H and K. A deadline task counted in the housekeeping
root domain sleeps on H. Then H goes offline. The task must wake up on
K. Without this patch it woke up on an isolated CPU: CPU 0, in a guest
with isolcpus=domain,0,1.
These cases test the other paths:
- hotplug_spinner_keeps: a running task whose CPU goes offline;
- hotplug_waking_keeps: a task that wakes up often, at the offline;
- hotplug_isolated_keeps: the same, for a task counted on the isolated
side;
- rebuild_waking_keeps: a wakeup inside a rebuild. It fails in most
runs if dl_task_bw_span() does not return NULL there. That needs
isolcpus=domain,0,1.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/core.c | 38 ++++++++++
kernel/sched/deadline.c | 156 +++++++++++++++++++++++++++++++++-------
kernel/sched/sched.h | 1 +
3 files changed, 171 insertions(+), 24 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e9f80407f75b..dc270a23179d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3264,6 +3264,23 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
dest_cpu = ctx->dl_bw_cpu;
if (!task_on_rq_queued(p))
p->wake_cpu = dest_cpu;
+ } else {
+ const struct cpumask *span = dl_task_bw_span(p);
+ unsigned int cpu = nr_cpu_ids;
+
+ /*
+ * No room was made for @p in another root domain. Keep @p
+ * in the root domain that counts its bandwidth, if the new
+ * mask holds one of its CPUs.
+ */
+ if (span && !cpumask_test_cpu(dest_cpu, span))
+ cpu = cpumask_first_and_and(cpu_valid_mask,
+ ctx->new_mask, span);
+ if (cpu < nr_cpu_ids) {
+ dest_cpu = cpu;
+ if (!task_on_rq_queued(p))
+ p->wake_cpu = dest_cpu;
+ }
}
do_set_cpus_allowed(p, ctx);
@@ -3668,8 +3685,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
int nid = cpu_to_node(cpu);
const struct cpumask *nodemask = NULL;
enum { cpuset, possible, fail } state = cpuset;
+ const struct cpumask *span;
int dest_cpu;
+ /*
+ * Keep a deadline task in the root domain that counts its
+ * bandwidth, if one of its CPUs is allowed and online.
+ */
+ span = dl_task_bw_span(p);
+ if (span) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ return dest_cpu;
+ }
+ }
+
/*
* If the node that the CPU is on has been offlined, cpu_to_node()
* will return -1. There is no CPU on the node, and we should
@@ -3686,6 +3716,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
}
for (;;) {
+ /* The mask is wider now: look again in that root domain. */
+ if (span && state != cpuset) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ goto out;
+ }
+ }
+
/* Any allowed, online CPU? */
for_each_cpu(dest_cpu, p->cpus_ptr) {
if (!is_cpu_allowed(p, dest_cpu))
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 6c6a4d9cd054..e5f77d3c5fa0 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -313,8 +313,8 @@ static unsigned int dl_bw_gen;
* where its bandwidth is counted. The new root domains count no task
* yet.
*
- * Changed under dl_departed_lock. dl_task_bw_elsewhere() reads it
- * without the lock.
+ * Changed under dl_departed_lock. dl_task_bw_span() and
+ * dl_task_bw_elsewhere() read it without the lock.
*/
static unsigned int dl_bw_rebuild_seq;
@@ -333,6 +333,8 @@ void dl_bw_rebuild_begin(void)
lockdep_assert_held(&sched_domains_mutex);
guard(raw_spinlock_irqsave)(&dl_departed_lock);
WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+ /* Before the first write of rq->rd: see dl_task_bw_span(). */
+ smp_wmb();
}
static void dl_bw_rebuild_end(void)
@@ -901,18 +903,36 @@ static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *rq);
static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p)
{
+ const struct cpumask *span;
struct rq *later_rq = NULL;
struct dl_bw *dl_b;
later_rq = find_lock_later_rq(p, rq);
+ /*
+ * find_lock_later_rq() looks in the root domain of @rq. Once @rq is
+ * offline, a rebuild attaches it to def_root_domain. That root domain
+ * may not count @p. Do not take @p out of the root domain that counts
+ * it while @p has an active CPU there.
+ */
+ span = dl_task_bw_span(p);
+ if (later_rq && span && !cpumask_test_cpu(later_rq->cpu, span) &&
+ cpumask_first_and_and(cpu_active_mask, p->cpus_ptr, span) < nr_cpu_ids) {
+ double_unlock_balance(rq, later_rq);
+ later_rq = NULL;
+ }
if (!later_rq) {
int cpu;
/*
* If we cannot preempt any rq, fall back to pick any
- * online CPU:
+ * online CPU. Try the root domain that counts @p first.
*/
- cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
+ cpu = nr_cpu_ids;
+ if (span)
+ cpu = cpumask_first_and_and(cpu_active_mask,
+ p->cpus_ptr, span);
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
if (cpu >= nr_cpu_ids) {
/*
* Failed to find any suitable CPU.
@@ -2923,6 +2943,7 @@ static int
select_task_rq_dl(struct task_struct *p, int cpu, int flags)
{
struct task_struct *curr, *donor;
+ const struct cpumask *span;
bool select_rq;
struct rq *rq;
@@ -2963,6 +2984,20 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
dl_task_is_earliest_deadline(p, cpu_rq(target)))
cpu = target;
}
+
+ /*
+ * Keep @p in the root domain that counts its bandwidth. Its mask
+ * can hold CPUs of another root domain, with isolcpus="domain",
+ * and @p can sleep on one of them.
+ */
+ span = dl_task_bw_span(p);
+ if (span && !cpumask_test_cpu(cpu, span)) {
+ unsigned int in_span;
+
+ in_span = cpumask_first_and_and(span, p->cpus_ptr, cpu_active_mask);
+ if (in_span < nr_cpu_ids)
+ cpu = in_span;
+ }
rcu_read_unlock();
return cpu;
@@ -3744,6 +3779,58 @@ static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
return cpu;
}
+/*
+ * Return the CPUs of the root domain that counts the bandwidth of @p, or
+ * NULL if none does. The mask of @p can hold CPUs of another root domain,
+ * with isolcpus="domain". Where a CPU is picked for @p, a CPU of this
+ * root domain comes first: no other root domain admitted @p.
+ *
+ * Return NULL while the sched domains are rebuilt. The CPU recorded can
+ * then sit in def_root_domain on its way to a new root domain. The span
+ * of def_root_domain holds the isolated CPUs, and no longer the
+ * housekeeping CPUs that already left it. A caller would send a
+ * housekeeping task to an isolated CPU.
+ *
+ * The caller holds the pi_lock or the rq lock of @p. With interrupts
+ * off, the root domain is not freed.
+ */
+const struct cpumask *dl_task_bw_span(struct task_struct *p)
+{
+ unsigned int seq = READ_ONCE(dl_bw_rebuild_seq);
+ struct root_domain *rd;
+ int cpu;
+
+ if (!task_has_dl_policy(p) || dl_entity_is_special(&p->dl) ||
+ !p->dl.dl_bw || (seq & 1))
+ return NULL;
+ cpu = READ_ONCE(p->dl.dl_bw_cpu);
+ if (cpu < 0 || READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen))
+ return NULL;
+ /*
+ * The CPU recorded can be going offline. It keeps its root domain
+ * until the rebuild attaches it to def_root_domain. The rebuild then
+ * records an active CPU with @p, or leaves @p out. Until then, the
+ * root domain of the CPU recorded still counts @p.
+ *
+ * A CPU that isolcpus="domain" isolates is always in
+ * def_root_domain, which then counts @p. A housekeeping CPU that is
+ * not active and in def_root_domain was attached there by the
+ * rebuild. def_root_domain does not count @p then.
+ */
+ rd = cpu_rq(cpu)->rd;
+ /*
+ * A rebuild that started since writes rq->rd after it made
+ * dl_bw_rebuild_seq odd. Use this root domain only if none did.
+ */
+ smp_rmb();
+ if (READ_ONCE(dl_bw_rebuild_seq) != seq)
+ return NULL;
+ if (!cpu_active(cpu) && rd == &def_root_domain &&
+ housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN_BOOT))
+ return NULL;
+ return rd->span;
+}
+
static void set_cpus_allowed_dl(struct task_struct *p,
struct affinity_context *ctx)
{
@@ -3953,23 +4040,24 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
* Return a CPU of the root domain where a rebuild of the accounting should
* count the bandwidth of @p, or nr_cpu_ids if there is none.
*
- * Prefer the CPU @p is on, if it is active and in the mask of @p. The
- * task runs, or most often wakes up, in the root domain of that CPU. A
- * mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
- * isolated by isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. A task that then runs in
- * the other root domain takes its bandwidth along.
- * Prefer that CPU also if @p is queued there, out of its mask, and the
- * CPU is the one recorded. Without admission control,
- * dl_task_offline_migration() can put a throttled task and its bandwidth
- * on any active CPU. The task runs there until it blocks.
+ * A mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
+ * isolated by isolcpus="domain". Keep the bandwidth on the side of the
+ * CPU recorded with it. That CPU is in the root domain that admitted the
+ * bandwidth, or that its last move put it in. Take the CPU @p is on if it
+ * is active, in the mask of @p and on that side. Else take the CPU
+ * recorded, if it is active and in the mask. Else take a CPU of the mask
+ * on that side.
+ *
+ * With no CPU on that side, or no CPU recorded, prefer the CPU @p is on.
+ * Else take the first active CPU of the mask, a housekeeping one if there
+ * is one. A task on isolated CPUs that all went offline can have no
+ * active CPU at all.
*
- * Else prefer the CPU recorded with the bandwidth, if it is active and in
- * the mask. It is in the root domain that admitted the bandwidth, or that
- * its last move put it in. A task moved asleep is still on a CPU of the
- * root domain it left, out of its mask. Else take the first active CPU of
- * the mask, a housekeeping one if there is one. A task on isolated CPUs
- * that all went offline can have no active CPU at all.
+ * One case comes before all of these. Take the CPU @p is on if @p is
+ * queued there, out of its mask, and the CPU is the one recorded. Without
+ * admission control, dl_task_offline_migration() can put a throttled task
+ * and its bandwidth on any active CPU. The task runs there until it
+ * blocks.
*/
static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
{
@@ -3987,17 +4075,37 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
*/
cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
cpu = task_cpu(p);
- if (cpumask_test_cpu(cpu, msk) ||
- (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
+ if (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p))
return cpu;
if (cpumask_empty(msk)) {
if (recorded >= 0 && cpu_active(recorded))
return recorded;
dl_get_task_effective_cpus(p, msk);
cpumask_and(msk, msk, cpu_active_mask);
- } else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
- return recorded;
}
+ /*
+ * Keep the bandwidth where it was counted: with the housekeeping
+ * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
+ * domains are new, and the CPU recorded can be offline. So prefer a
+ * CPU of the mask of the same kind as the CPU recorded: housekeeping
+ * or not.
+ */
+ if (recorded >= 0) {
+ const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+ bool rec_hk = cpumask_test_cpu(recorded, hk);
+ unsigned int same;
+
+ if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
+ return cpu;
+ if (cpumask_test_cpu(recorded, msk))
+ return recorded;
+ for_each_cpu(same, msk) {
+ if (cpumask_test_cpu(same, hk) == rec_hk)
+ return same;
+ }
+ }
+ if (cpumask_test_cpu(cpu, msk))
+ return cpu;
cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
if (cpu < nr_cpu_ids)
return cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3cc287785b91..f80ea5541b3f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -371,6 +371,7 @@ extern bool __checkparam_dl(const struct sched_attr *attr);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
extern int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
extern int dl_bw_deactivate(int cpu);
+extern const struct cpumask *dl_task_bw_span(struct task_struct *p);
extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec);
/*
* SCHED_DEADLINE supports servers (nested scheduling) with the following
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (27 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:41 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 30/40] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
` (10 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
dl_bw_deactivate() refuses to take a CPU offline when the rest of its
root domain cannot hold the deadline bandwidth counted there. It also
keeps the last CPU of a root domain that counts a deadline task. That
test comes from commit d4742f6ed7ea ("sched/deadline: Correctly account
for allocated bandwidth during hotplug").
That test looks at the root domain, not at the mask of each task. With
isolcpus="domain", a cpuset that is not a partition root can hold one
housekeeping CPU and some isolated CPUs. A deadline task in it can be
counted in the housekeeping root domain. When that housekeeping CPU
goes offline, the root domain keeps its other CPUs, so the offline is
accepted. But the task has only isolated CPUs left. It runs in the
default root domain, which did not admit it. Its bandwidth follows with
no admission test, and the default root domain can exceed its limit.
Refuse the offline with EBUSY in that case. Walk the deadline tasks
counted in the root domain of the CPU. Refuse if one of them has CPUs
left to run on, but none in that root domain. Skip the walk if no task
holds a bandwidth. A task whose mask holds no other CPU gets the CPUs of
an ancestor of its cpuset: add cpuset_cpus_allowed_without() to tell
which. Like __set_cpus_allowed_ptr(), it keeps them within the affinity
set by the user, if they intersect. Skip a dead task: it runs no more.
Do not refuse during a suspend. It takes the CPUs offline with the
tasks frozen, and the resume puts them back. Do not refuse without
admission control either: there is no limit to exceed.
An administrator who needs the CPU offline can first give the cpuset of
the task another CPU of that root domain. The offline is then accepted,
and the task wakes up on that CPU: the hotplug_keeps_root_domain case
tests this. The administrator can also turn admission control off, by
writing -1 to /proc/sys/kernel/sched_rt_runtime_us.
The walk cannot miss a task. sched_setattr() only makes a deadline task
of a task whose mask holds its whole root domain. sched_setaffinity()
only gives a deadline task such a mask. A mask also gets smaller by a
cpuset change or a cpuset move, and both exclude CPU hotplug.
For user-space, the refused offline is a write of 0 to
/sys/devices/system/cpu/cpuN/online that fails with EBUSY. An offline
that leaves each deadline task a CPU in its root domain is accepted as
before.
The case hotplug_last_refused of deadline_bw_test, added later in this
series, tests this. Without this patch the offline is accepted. The
cases hotplug_user_mask_refused and hotplug_dead_accepted test the
affinity set by the user and a dead task.
Fixes: d4742f6ed7ea ("sched/deadline: Correctly account for allocated bandwidth during hotplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 9 +++++++
kernel/cgroup/cpuset.c | 26 ++++++++++++++++++
kernel/sched/deadline.c | 60 ++++++++++++++++++++++++++++++++++++++++-
3 files changed, 94 insertions(+), 1 deletion(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 5d0c01a25354..68849539cc8b 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -79,6 +79,8 @@ extern void lockdep_assert_cpuset_lock_held(void);
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern void cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+ struct cpumask *mask);
extern int cpuset_num_cpus(struct cgroup *cgroup);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#define cpuset_current_mems_allowed (current->mems_allowed)
@@ -214,6 +216,13 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)
return false;
}
+static inline void cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+ struct cpumask *mask)
+{
+ cpumask_and(mask, task_cpu_possible_mask(p), cpu_active_mask);
+ cpumask_clear_cpu(cpu, mask);
+}
+
static inline int cpuset_num_cpus(struct cgroup *cgroup)
{
return num_online_cpus();
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7bdb9e21a7ed..3ccd61a37d30 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4312,6 +4312,32 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)
spin_unlock_irqrestore(&callback_lock, flags);
}
+/*
+ * Set @pmask to the active CPUs that the cpuset of @tsk gives it once @cpu
+ * is offline: those of the nearest ancestor that keeps one. Leave it empty
+ * if there is none. As cpuset_update_tasks_cpumask() does, keep the CPUs
+ * possible for @tsk. As __set_cpus_allowed_ptr() does, keep those of the
+ * affinity the user set, if one is left. The caller holds the CPU hotplug
+ * lock for writing, which keeps the cpusets and their tasks as they are.
+ * It holds the pi_lock of @tsk, which keeps its user affinity.
+ */
+void cpuset_cpus_allowed_without(struct task_struct *tsk, int cpu,
+ struct cpumask *pmask)
+{
+ struct cpuset *cs;
+
+ guard(rcu)();
+ for (cs = task_cs(tsk); cs; cs = parent_cs(cs)) {
+ cpumask_and(pmask, cs->effective_cpus, cpu_active_mask);
+ cpumask_clear_cpu(cpu, pmask);
+ if (!cpumask_empty(pmask))
+ break;
+ }
+ cpumask_and(pmask, pmask, task_cpu_possible_mask(tsk));
+ if (tsk->user_cpus_ptr && cpumask_intersects(pmask, tsk->user_cpus_ptr))
+ cpumask_and(pmask, pmask, tsk->user_cpus_ptr);
+}
+
/**
* cpuset_cpus_allowed_fallback - final fallback before complete catastrophe.
* @tsk: pointer to task_struct with which the scheduler is struggling
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index e5f77d3c5fa0..47264640de14 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -5028,9 +5028,67 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
return overflow ? -EBUSY : 0;
}
+/*
+ * Return true if @p has CPUs to run on once @cpu is offline, but none in
+ * the root domain of @cpu, which counts its bandwidth. A dead task runs no
+ * more. It can wait for its 0-lag time, or for its parent to reap it.
+ * Return false for it.
+ */
+static bool dl_task_leaves_span(struct task_struct *p, int cpu,
+ const struct cpumask *span)
+{
+ struct cpumask *msk;
+
+ guard(raw_spinlock_irqsave)(&p->pi_lock);
+ if (READ_ONCE(p->__state) == TASK_DEAD || dl_task_bw_span(p) != span)
+ return false;
+ msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+ /* A task in a migrate_disable() section has a cpus_ptr of one CPU. */
+ cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+ cpumask_clear_cpu(cpu, msk);
+ /* With no CPU left, the cpuset gives @p the CPUs of an ancestor. */
+ if (cpumask_empty(msk))
+ cpuset_cpus_allowed_without(p, cpu, msk);
+ return !cpumask_empty(msk) && !cpumask_intersects(msk, span);
+}
+
+/*
+ * Return true if @cpu is the last CPU that a deadline task can run on in
+ * the root domain that counts its bandwidth. With @cpu offline, the task
+ * would run in another root domain, which did not admit it. The mask of
+ * a task can hold CPUs of two root domains: with isolcpus="domain", or on
+ * cgroup v1 with sched_load_balance off.
+ *
+ * A suspend takes the CPUs offline with the tasks frozen, and the resume
+ * puts them back. Without admission control, there is no limit to exceed.
+ * Return false in both cases.
+ */
+static bool dl_cpu_last_of_task(int cpu)
+{
+ const struct cpumask *span;
+ struct task_struct *g, *p;
+
+ if (cpuhp_tasks_frozen || !dl_bandwidth_enabled())
+ return false;
+ /* No task holds a bandwidth: do not walk the task list. */
+ if (!atomic_read(&dl_nr_bw_tasks))
+ return false;
+ guard(rcu)();
+ span = cpu_rq(cpu)->rd->span;
+ for_each_process_thread(g, p) {
+ if (task_has_dl_policy(p) && dl_task_leaves_span(p, cpu, span))
+ return true;
+ }
+ return false;
+}
+
int dl_bw_deactivate(int cpu)
{
- return dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+ int ret = dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+
+ if (!ret && dl_cpu_last_of_task(cpu))
+ ret = -EBUSY;
+ return ret;
}
int dl_bw_alloc(int cpu, u64 dl_bw)
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 30/40] cgroup/cpuset: Refuse to take a deadline task out of its root domain
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (28 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 31/40] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
` (9 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A cpuset attach asks the destination root domain for room before it
moves a deadline task there. A change of the CPUs of a cpuset asks
nothing. validate_change() only checks that a partition root keeps
enough capacity for its deadline bandwidth. That check comes from
commit f82f80426f7a ("sched/deadline: Ensure that updates to exclusive
cpusets don't break AC").
With isolcpus="domain", a cpuset that is not a partition root can hold
housekeeping CPUs and isolated CPUs, which are in two root domains. Two
changes can take every CPU of one root domain away from that cpuset:
- a write to cpuset.cpus, of that cpuset or of an ancestor;
- a partition root that takes those CPUs, when it is created or when
it grows.
A deadline task counted in that root domain then runs in the other one,
which did not admit it. Its bandwidth follows with no admission test,
and that root domain can exceed its limit.
Refuse both:
- A write to cpuset.cpus fails with EBUSY, and changes nothing. The
cpuset.cpus file of cgroup v1 gets the same check.
- The write that makes the partition root, or makes it grow, still
succeeds. cpuset makes the partition root invalid. For a "root"
partition, cpuset.cpus.partition then reads "root invalid (Deadline
task would leave its root domain)".
A task goes to another root domain by a move to another cpuset, which
asks for room.
A change that leaves each deadline task a CPU in its root domain is
accepted as before.
The check follows __set_cpus_allowed_ptr(). That keeps the new mask
within the affinity set by the user, if they intersect. So a mask that
grows can also take a task out of its root domain, and it is checked
too. Changes that cannot fail are not checked: a partition that is
disabled or becomes invalid, and a CPU that comes online.
dl_task_keeps_root_domain() tells whether a new mask keeps a task in
the root domain that counts it. Without admission control it accepts
any mask: there is no limit to exceed. An administrator who needs the
change can turn admission control off, by writing -1 to
/proc/sys/kernel/sched_rt_runtime_us.
The cases mask_out_refused and partition_takes_refused of
deadline_bw_test, added later in this series, test this. Without this
patch the write is accepted and the partition root is valid. These
cases test the other paths:
- partition_takes_fallback_refused: the CPUs of an ancestor;
- mask_grows_refused: a mask that grows;
- partition_swap_refused: a partition root that takes and gives CPUs in
one change;
- partition_refused_no_warn: a refusal with no warning.
Fixes: f82f80426f7a ("sched/deadline: Ensure that updates to exclusive cpusets don't break AC")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched/deadline.h | 2 +
kernel/cgroup/cpuset-internal.h | 1 +
kernel/cgroup/cpuset.c | 219 +++++++++++++++++++++++++++++++-
kernel/sched/deadline.c | 25 ++++
4 files changed, 244 insertions(+), 3 deletions(-)
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdabd54af729..67426834a9a4 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -62,6 +62,8 @@ extern int dl_task_bw_reserve_cpu(struct task_struct *p,
const struct cpumask *new_mask);
extern int dl_task_bw_reserved_cpu(struct task_struct *p);
extern void dl_task_bw_unreserve(struct task_struct *p);
+extern bool dl_task_keeps_root_domain(struct task_struct *p,
+ const struct cpumask *new_mask);
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index a83443dfb788..8fe0225704eb 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -36,6 +36,7 @@ enum prs_errcode {
PERR_ACCESS,
PERR_REMOTE,
PERR_ISOPARENT,
+ PERR_DLTASK,
};
/* bits in struct cpuset flags field */
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3ccd61a37d30..d604b0267cc7 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -61,6 +61,7 @@ static const char * const perr_strings[] = {
[PERR_ACCESS] = "Enable partition not permitted",
[PERR_REMOTE] = "Have remote partition underneath",
[PERR_ISOPARENT] = "Root partition is not allowed with isolated parent",
+ [PERR_DLTASK] = "Deadline task would leave its root domain",
};
/*
@@ -1126,6 +1127,130 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
cpuset_force_rebuild();
}
+/*
+ * Return false if the CPUs of @new_cpus make a deadline task of @cp leave
+ * the root domain that counts its bandwidth. Called under rcu_read_lock(),
+ * which it drops and takes again.
+ */
+static bool dl_tasks_keep_cpus(struct cpuset *cp, const struct cpumask *new_cpus)
+{
+ struct css_task_iter it;
+ struct task_struct *task;
+ bool keep = true;
+
+ if (!css_tryget_online(&cp->css))
+ return true;
+ rcu_read_unlock();
+
+ css_task_iter_start(&cp->css, 0, &it);
+ while (keep && (task = css_task_iter_next(&it))) {
+ /* The top cpuset leaves the mask of these tasks alone. */
+ if (cp == &top_cpuset &&
+ (task->flags & (PF_KTHREAD | PF_NO_SETAFFINITY)))
+ continue;
+ keep = dl_task_keeps_root_domain(task, new_cpus);
+ }
+ css_task_iter_end(&it);
+
+ rcu_read_lock();
+ css_put(&cp->css);
+ return keep;
+}
+
+/*
+ * Set @new_cpus to the effective CPUs that @cp gets. On entry, @new_cpus
+ * holds those that @from gets. @from is @cp or one of its ancestors. Work
+ * down from the child of @from to @cp, as compute_effective_cpumask() does
+ * for a cpuset that is not a valid partition root.
+ */
+static void effective_cpus_below(struct cpuset *cp, struct cpuset *from,
+ struct cpumask *new_cpus)
+{
+ struct cpuset *a;
+ int depth = 0, i;
+
+ for (a = cp; a != from; a = parent_cs(a))
+ depth++;
+ while (depth--) {
+ for (a = cp, i = 0; i < depth; i++)
+ a = parent_cs(a);
+ if (cpumask_intersects(new_cpus, a->cpus_allowed) || !is_in_v2_mode())
+ cpumask_and(new_cpus, new_cpus, a->cpus_allowed);
+ }
+}
+
+/*
+ * A scratch mask for dl_tasks_keep_without(). Its callers still need each
+ * mask of their struct tmpmasks after the call. Protected by
+ * cpuset_mutex.
+ */
+static cpumask_var_t dl_keep_cpus;
+
+/*
+ * Return false if a partition that takes the CPUs of @xcpus makes a
+ * deadline task leave the root domain that counts its bandwidth. A cpuset
+ * can hold CPUs of two root domains: with isolcpus="domain", or on cgroup
+ * v1 with sched_load_balance off. Such a task would run in a root domain
+ * that did not admit it.
+ *
+ * @root is the partition root that gives the CPUs. @part is the cpuset
+ * that takes them. In the same change, @part can give the CPUs of @gives
+ * back to @root. @gives is NULL if it gives none back.
+ *
+ * Work out the new CPUs of each cpuset of @root from the top down, as
+ * update_cpumasks_hier() does, down to the next partition roots. If
+ * @enable, @part becomes a partition root. Then do the same for its own
+ * cpusets, starting from the active CPUs of @xcpus.
+ */
+static bool dl_tasks_keep_without(struct cpuset *root, struct cpuset *part,
+ const struct cpumask *xcpus,
+ const struct cpumask *gives, bool enable)
+{
+ struct cpumask *new_cpus = dl_keep_cpus;
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp;
+ bool keep = true;
+
+ lockdep_assert_cpuset_lock_held();
+ rcu_read_lock();
+ cpuset_for_each_descendant_pre(cp, pos_css, root) {
+ if (cp == part || (cp != root && is_partition_valid(cp))) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ /* The CPUs that @root has after the change. */
+ cpumask_copy(new_cpus, root->effective_cpus);
+ if (gives) {
+ cpumask_or(new_cpus, new_cpus, gives);
+ cpumask_and(new_cpus, new_cpus, cpu_active_mask);
+ }
+ cpumask_andnot(new_cpus, new_cpus, xcpus);
+ effective_cpus_below(cp, root, new_cpus);
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ if (keep && enable) {
+ cpuset_for_each_descendant_pre(cp, pos_css, part) {
+ if (cp != part && is_partition_valid(cp)) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ cpumask_and(new_cpus, xcpus, cpu_active_mask);
+ effective_cpus_below(cp, part, new_cpus);
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ }
+ rcu_read_unlock();
+ return keep;
+}
+
/*
* tasks_nocpu_error - Return true if tasks will have no effective_cpus
*/
@@ -1676,6 +1801,8 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
return PERR_NOCPUS;
if (prstate_housekeeping_conflict(new_prs, PRS_ROOT, tmp->new_cpus, NULL))
return PERR_HKEEPING;
+ if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->new_cpus, NULL, true))
+ return PERR_DLTASK;
spin_lock_irq(&callback_lock);
partition_xcpus_add(new_prs, NULL, tmp->new_cpus);
@@ -1779,6 +1906,10 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
excpus, tmp->delmask))
WRITE_ONCE(cs->prs_err, PERR_HKEEPING);
+ else if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->addmask,
+ deleting ? tmp->delmask : NULL,
+ false))
+ WRITE_ONCE(cs->prs_err, PERR_DLTASK);
if (cs->prs_err)
goto invalidate;
}
@@ -1926,6 +2057,9 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
if (nocpu)
return PERR_NOCPUS;
+ if (!dl_tasks_keep_without(parent, cs, tmp->delmask, NULL, true))
+ return PERR_DLTASK;
+
/*
* This function will only be called when all the preliminary
* checks have passed. At this point, the following condition
@@ -2013,6 +2147,20 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
deleting = false;
adding = cpumask_and(tmp->addmask,
cs->effective_xcpus, parent->effective_xcpus);
+ } else if (deleting &&
+ !dl_tasks_keep_without(parent, cs, tmp->delmask,
+ adding ? tmp->addmask : NULL,
+ is_partition_invalid(cs))) {
+ /*
+ * As above: the partition is or becomes invalid. Only
+ * a valid one holds CPUs of its parent to give back.
+ */
+ part_error = PERR_DLTASK;
+ deleting = false;
+ if (is_partition_valid(cs))
+ adding = cpumask_and(tmp->addmask,
+ cs->effective_xcpus,
+ parent->effective_xcpus);
}
} else {
/*
@@ -2084,10 +2232,13 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
}
rcu_read_unlock();
}
- if (exclusive)
- deleting = !cpumask_empty(tmp->delmask);
- else
+ if (!exclusive)
part_error = PERR_NOTEXCL;
+ else if (!dl_tasks_keep_without(parent, cs, tmp->delmask,
+ NULL, true))
+ part_error = PERR_DLTASK;
+ else
+ deleting = !cpumask_empty(tmp->delmask);
}
/*
* When both cs and parent remain valid, parent's
@@ -2496,6 +2647,62 @@ static void partition_cpus_change(struct cpuset *cs, struct cpuset *trialcs,
}
}
+/*
+ * Set @new_cpus to the effective CPUs that @cp gets if @cs takes the CPUs
+ * of @trialcs. @cp is @cs or one of its descendants.
+ */
+static void trial_effective_cpus(struct cpuset *cp, struct cpuset *cs,
+ struct cpuset *trialcs, struct cpumask *new_cpus)
+{
+ struct cpuset *parent = parent_cs(cs);
+
+ if (!cpumask_and(new_cpus, trialcs->cpus_allowed, parent->effective_cpus) &&
+ is_in_v2_mode())
+ cpumask_copy(new_cpus, parent->effective_cpus);
+ effective_cpus_below(cp, cs, new_cpus);
+}
+
+/*
+ * Return false if the CPUs of @trialcs make a deadline task of @cs, or of
+ * a descendant, leave the root domain that counts its bandwidth. A cpuset
+ * can hold CPUs of two root domains: with isolcpus="domain", or on cgroup
+ * v1 with sched_load_balance off. The task would run in a root domain that
+ * did not admit it. A move of the task to another cpuset is the way there:
+ * it asks the other root domain for room.
+ *
+ * A partition root takes its CPUs from its parent, which checks the same
+ * for its own tasks: see dl_tasks_keep_without().
+ */
+static bool dl_tasks_keep_root_domain(struct cpuset *cs, struct cpuset *trialcs,
+ struct cpumask *new_cpus)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp;
+ bool keep = true;
+
+ if (is_partition_valid(cs))
+ return true;
+ rcu_read_lock();
+ cpuset_for_each_descendant_pre(cp, pos_css, cs) {
+ if (is_partition_valid(cp)) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ trial_effective_cpus(cp, cs, trialcs, new_cpus);
+ /*
+ * Test a mask that grows too. A task whose user affinity
+ * holds none of the old CPUs gets only the new ones.
+ */
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ rcu_read_unlock();
+ return keep;
+}
+
/**
* update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it
* @cs: the cpuset to consider
@@ -2528,6 +2735,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,
if (alloc_tmpmasks(&tmp))
return -ENOMEM;
+ if (!dl_tasks_keep_root_domain(cs, trialcs, tmp.new_cpus)) {
+ free_tmpmasks(&tmp);
+ return -EBUSY;
+ }
+
/*
* Check all the descendants in update_cpumasks_hier() if
* effective_xcpus is to be changed.
@@ -3961,6 +4173,7 @@ int __init cpuset_init(void)
BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&dl_attach_cpus, GFP_KERNEL));
+ BUG_ON(!alloc_cpumask_var(&dl_keep_cpus, GFP_KERNEL));
if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT))
cpumask_andnot(isolated_cpus, cpu_possible_mask,
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 47264640de14..ff559f2729fb 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -5091,6 +5091,31 @@ int dl_bw_deactivate(int cpu)
return ret;
}
+/*
+ * Return false if @p has CPUs in the root domain that counts its
+ * bandwidth and @new_mask holds none of them. A cpuset refuses such a
+ * mask: only a move to another cpuset makes room in another root domain.
+ * Like __set_cpus_allowed_ptr(), keep @new_mask within the affinity set by
+ * the user, if they intersect.
+ */
+bool dl_task_keeps_root_domain(struct task_struct *p,
+ const struct cpumask *new_mask)
+{
+ const struct cpumask *span;
+
+ if (!dl_bandwidth_enabled() || !task_has_dl_policy(p))
+ return true;
+ /* pi_lock also protects p->user_cpus_ptr. */
+ guard(raw_spinlock_irqsave)(&p->pi_lock);
+ guard(rcu)();
+ span = dl_task_bw_span(p);
+ if (!span || !cpumask_intersects(span, p->cpus_ptr))
+ return true;
+ if (p->user_cpus_ptr && cpumask_intersects(new_mask, p->user_cpus_ptr))
+ return cpumask_intersects_and(span, new_mask, p->user_cpus_ptr);
+ return cpumask_intersects(span, new_mask);
+}
+
int dl_bw_alloc(int cpu, u64 dl_bw)
{
return dl_bw_manage(dl_bw_req_alloc, cpu, dl_bw);
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 31/40] sched/deadline: Cancel the replenishment timer with the parameters
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (29 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 30/40] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 32/40] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
` (8 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
A task that leaves SCHED_DEADLINE has its parameters cleared at its
0-lag time. Its replenishment timer can still be armed then. If the
task comes back and throttles before that timer fires, start_dl_timer()
does not restart it, since it is queued. The replenishment would then
come at the period of the reservation the task left. Too early, it would
give the task runtime it has not earned. Too late, it would delay the
task by up to a period.
Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole") armed
the replenishment timer when a task leaves the class. The timer then
reset the task if it was still out. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
that reset to the inactive timer, at the 0-lag time. Nothing cancels
the replenishment timer there.
The timer of a throttled task fires at its next period, which can come
after the 0-lag time:
- A task that yielded is throttled with runtime left. Its 0-lag time
comes before its deadline.
- A constrained-deadline task that overran by little reaches its 0-lag
time before the end of its period.
Fix this by cancelling the timer where the parameters are cleared. This
includes task_non_contending(). switched_from_dl() calls it to release
the bandwidth of a task that leaves asleep past its 0-lag time. A
constrained task that overran as it blocked has the timer armed there
too.
The Sashiko AI review of the first posting of this series found this.
It has no reproducer.
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ff559f2729fb..b9268c52935d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -694,6 +694,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
dl_task_release_bw(p);
+ cancel_replenish_timer(dl_se);
__dl_clear_params(dl_se);
}
}
@@ -2518,6 +2519,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
}
dl_task_release_bw(p);
+ cancel_replenish_timer(dl_se);
__dl_clear_params(dl_se);
goto unlock;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 32/40] sched/deadline: Count the bandwidth a boosted task is given
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (30 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 31/40] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
` (7 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
sched_setattr(SCHED_DEADLINE) on a sleeping task that priority
inheritance boosted into the class leaves the new bandwidth out of
this_bw. switched_to_dl() does not run, since the task is in the class
already. sched_dl_overflow() changes this_bw only for a task of the
deadline policy. When the task wakes up, __add_running_bw() warns:
WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
When the task leaves the runqueue, this_bw would stay short by that
bandwidth, and skew GRUB reclaiming on that CPU.
Fix this by changing this_bw in __setparam_dl(), which every task that
gains deadline parameters goes through. If no runqueue counts the
bandwidth of the task, dl_change_utilization() now adds the new one. It
now also runs when the bandwidth does not change. It then returns early,
so that a sleeping task keeps its running_bw and its inactive timer.
A sugov kthread given a reservation of its own now has its bandwidth
counted in this_bw. It reaches dl_change_utilization(), so the
WARN_ON_ONCE() for SCHED_FLAG_SUGOV there goes. sched_dl_overflow()
returned early for an unchanged bandwidth, even if no root domain counts
it. That is the case for such a kthread, and for a task that a rebuild
left out. Admit such a bandwidth as new. Without room, that fails with
EBUSY. The Sashiko AI review of the third posting found this early
return.
A reproducer shows the warning, and this_bw unchanged by the change.
With this patch, this_bw takes the new bandwidth, and nothing warns.
Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 54 ++++++++++++++++++++++++++++++++---------
1 file changed, 43 insertions(+), 11 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b9268c52935d..0b3005948313 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -573,22 +573,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
cancel_dl_timer(dl_se, &dl_se->inactive_timer);
}
+/*
+ * Make the runqueue of a sleeping task count @new_bw as the bandwidth of
+ * the task. Do nothing for a queued task. The dequeue and the enqueue
+ * around the change of its parameters update this_bw.
+ */
static void dl_change_utilization(struct task_struct *p, u64 new_bw)
{
- WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+ struct sched_dl_entity *dl_se = &p->dl;
+ struct rq *rq = task_rq(p);
if (task_on_rq_queued(p))
return;
/*
- * The task is waking up on another CPU, and no runqueue counts its
- * bandwidth. The enqueue adds it, with the new parameters that
- * __setparam_dl() sets right after this.
+ * No runqueue counts the bandwidth of the task. Add the new one to
+ * this runqueue, so that add_rq_bw() adds nothing later. A task that
+ * enters the class asleep comes here, and so does a special entity
+ * given a reservation of its own. So does a task waking up on
+ * another CPU: task_cpu() already points at the CPU it wakes on.
*/
- if (!p->dl.dl_bw_oncpu)
+ if (!dl_se->dl_bw_oncpu) {
+ __add_rq_bw(new_bw, &rq->dl);
+ dl_se->dl_bw_oncpu = 1;
+ return;
+ }
+
+ /* Same bandwidth: leave running_bw and the inactive timer alone. */
+ if (new_bw == dl_se->dl_bw)
return;
- dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+ dl_rq_change_utilization(rq, dl_se, new_bw);
}
static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4677,10 +4692,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (attr->sched_flags & SCHED_FLAG_SUGOV)
return 0;
- /* !deadline task may carry old deadline bandwidth */
- if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
- return 0;
-
/*
* Change the bandwidth of a deadline task where it is counted: in
* the root domain of the CPU recorded with it. A task moved asleep
@@ -4695,6 +4706,16 @@ int sched_dl_overflow(struct task_struct *p, int policy,
*/
counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
p->dl.dl_bw_gen == dl_bw_gen;
+
+ /*
+ * !deadline task may carry old deadline bandwidth. The same
+ * bandwidth needs no admission if it is counted. If it is not, admit
+ * it as new below. A special entity given a reservation of its own
+ * has no bandwidth counted, nor has a task that a rebuild left out.
+ */
+ if (new_bw == p->dl.dl_bw && counted)
+ return 0;
+
cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
dl_b = dl_bw_of(cpu);
@@ -4724,7 +4745,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (counted)
__dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
- dl_change_utilization(p, new_bw);
err = 0;
} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
/*
@@ -4774,8 +4794,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
{
struct sched_dl_entity *dl_se = &p->dl;
+ u64 period = attr->sched_period ?: attr->sched_deadline;
bool held;
+ /*
+ * Change the bandwidth that the runqueue counts here, with the
+ * parameters. A task that priority inheritance boosted into the
+ * class is in it already, so switched_to_dl() does not run for it.
+ *
+ * A special entity has no bandwidth on a runqueue. Test the flags
+ * of the request, since those of the entity are still the old ones.
+ */
+ if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+ dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
+
held = dl_se_holds_bw(dl_se);
dl_se->dl_runtime = attr->sched_runtime;
dl_se->dl_deadline = attr->sched_deadline;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (31 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 32/40] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:46 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 34/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
` (6 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit fb00aca47440 ("rtmutex: Turn the plist into an rb-tree") sorts
the waiters of an rt_mutex by priority. For deadline tasks, "the one
with the earliest deadline wins". A waiter takes the deadline of its
task.
A task can enter the class while it waits on an rt_mutex.
sched_setattr() can make it SCHED_DEADLINE. A priority inheritance
boost can also put it in the class, if it owns another rt_mutex.
switched_to_dl() leaves a task that is not queued to its wakeup. But
the chain walk reads the deadline first, and copies it into the waiter.
A task new to the class has a deadline of 0.
So the waiter goes before every other deadline waiter. It takes the
rt_mutex first, whatever their deadlines. Those waiters can then miss
their deadlines. It also boosts a deadline owner whose own deadline is
earlier.
Fix this by starting a new period in switched_to_dl() for a task that
waits on an rt_mutex, if its deadline has passed. The period uses the
reservation the task runs on: its own, or the copy a boost made. This
is what setup_new_dl_entity() does at an enqueue. At the wakeup,
update_dl_entity() then checks that period as for any wakeup.
A new donor of a task with no parameters of its own needs the same. For
a queued task, the enqueue of the boost starts a period from the new
donor. For a blocked task, dl_boost_blocked() starts it at the boost.
The setattr_blocked_keeps_order and boost_blocked_keeps_order cases of
deadline_bw_test, added later in this series, test this. Both fail on
v7.4-rc2 and on the series without this patch. The triage of the Sashiko
AI review of the fifth posting found the bug.
Fixes: fb00aca47440 ("rtmutex: Turn the plist into an rb-tree")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/core.c | 2 ++
kernel/sched/deadline.c | 42 +++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 1 +
3 files changed, 45 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index dc270a23179d..2d3409a1a12a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7939,6 +7939,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
dl_entity_preempt(&pi_task->dl, &p->dl))) {
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
+ if (!scope->queued)
+ dl_boost_blocked(p);
} else {
dl_deboost(&p->dl);
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 0b3005948313..90b17969554b 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -125,6 +125,12 @@ void dl_deboost(struct sched_dl_entity *dl_se)
}
dl_drop_boost(dl_se);
}
+
+/* Return true if @p waits on an rt_mutex. The caller holds its pi_lock. */
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+ return p->pi_blocked_on;
+}
#else /* !CONFIG_RT_MUTEXES: */
static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
{
@@ -139,6 +145,11 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return false;
}
+
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+ return false;
+}
#endif /* !CONFIG_RT_MUTEXES */
static inline u8 dl_get_type(struct sched_dl_entity *dl_se, struct rq *rq)
@@ -1048,6 +1059,26 @@ static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
trace_sched_dl_replenish_tp(dl_se, cpu_of(rq), dl_get_type(dl_se, rq));
}
+#ifdef CONFIG_RT_MUTEXES
+/*
+ * Start a period from the new donor of @p, if @p has no reservation of its
+ * own and waits on an rt_mutex. The enqueue does this for a queued task, in
+ * replenish_dl_entity(). A blocked task is not enqueued, but the chain walk
+ * copies its deadline into its waiter.
+ *
+ * The caller holds the rq lock and the pi_lock of @p, and has updated the
+ * rq clock.
+ */
+void dl_boost_blocked(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ if (!dl_se->dl_deadline && !dl_se->dl_throttled &&
+ dl_task_pi_blocked(p))
+ replenish_dl_new_period(dl_se, task_rq(p));
+}
+#endif
+
/*
* We are being explicitly informed that a new instance is starting,
* and this means that:
@@ -4483,6 +4514,17 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
if (!task_on_rq_queued(p)) {
add_rq_bw(&p->dl, &rq->dl);
+ /*
+ * A task that waits on an rt_mutex cannot wait for its wakeup.
+ * The chain walk sorts its waiter by its deadline first. A task
+ * new to the class has a deadline of 0, which goes before every
+ * real one. Start a new period now, as setup_new_dl_entity()
+ * does at an enqueue.
+ */
+ if (dl_task_pi_blocked(p) && !p->dl.dl_throttled &&
+ dl_time_before(p->dl.deadline, rq_clock(rq)))
+ replenish_dl_new_period(&p->dl, rq);
+
return;
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index f80ea5541b3f..717ced2bb487 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -358,6 +358,7 @@ extern int sched_dl_overflow(struct task_struct *p, int policy, const struct sc
extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
#ifdef CONFIG_RT_MUTEXES
extern void dl_deboost(struct sched_dl_entity *dl_se);
+extern void dl_boost_blocked(struct task_struct *p);
/* End the boost of @dl_se. The caller holds the rq lock of its task. */
static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 34/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (32 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 35/40] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
` (5 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
sched_setattr() can change the parameters of a SCHED_DEADLINE task. The
task keeps its runtime and its absolute deadline: the new parameters
take effect at the next period. At the next wakeup, update_dl_entity()
checks the runtime left against the new parameters.
For a constrained task, that check uses the density, dl_runtime /
dl_deadline. Commit 2317d5f1c349 ("sched/deadline: Use deadline instead
of period when calculating overflow") made dl_entity_overflow() compare
with it: "We care about if the runtime can make its deadline, not its
period." Commit 3effcb4247e7 ("sched/deadline: Use the revised wakeup
rule for suspending constrained dl tasks") then cut the runtime of a
constrained task that overflows to density * laxity. Both bound the
runtime by dl_runtime only while the deadline is at most dl_deadline
ahead.
A change to a shorter dl_deadline breaks that. The old deadline can be
farther ahead. The check then keeps more runtime than the new
dl_runtime, or computes more. For example, take a task of 50ms every
100ms with 49ms left. Changed to 8ms in 20ms every 100ms, it would wake
with 39.6ms. It would then run far more than its new runtime in this
period. Admission control already counts the new, smaller bandwidth. So
other deadline tasks could miss their deadlines.
An implicit task is not affected. Its check compares with its
bandwidth, which bounds the runtime for any deadline.
Fix this by keeping no more than dl_runtime before the check, for a
constrained task. A boosted task runs on the parameters of its donor,
and is left alone. The runtime is signed and can be negative there, so
compare it as signed.
The cases change_constrained_new_params and wake_constrained_new_params
of deadline_bw_test, added later in this series, test this. The first
changes a running task, the second a sleeping one. Both fail without
this patch.
The Sashiko AI review of the sixth posting of this series found this.
Fixes: 2317d5f1c349 ("sched/deadline: Use deadline instead of period when calculating overflow")
Fixes: 3effcb4247e7 ("sched/deadline: Use the revised wakeup rule for suspending constrained dl tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 90b17969554b..1a07f4109cc5 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1389,6 +1389,25 @@ static void update_dl_entity(struct sched_dl_entity *dl_se)
{
struct rq *rq = rq_of_dl_se(dl_se);
+ /*
+ * The checks below compare the runtime left with the density,
+ * dl_runtime / dl_deadline. That bounds the runtime by dl_runtime
+ * only while the deadline is at most dl_deadline ahead. After a
+ * change to a shorter dl_deadline, the deadline can be farther
+ * ahead, and the runtime left larger than dl_runtime. A constrained
+ * task would then run more than dl_runtime in a period. Keep no
+ * more than dl_runtime.
+ *
+ * An implicit task is checked against its bandwidth, which holds
+ * for any deadline. A boosted task runs on the parameters of its
+ * donor. Leave both alone.
+ *
+ * The runtime is signed, and can be negative here.
+ */
+ if (!dl_is_implicit(dl_se) && !is_dl_boosted(dl_se) &&
+ dl_se->runtime > (s64)dl_se->dl_runtime)
+ dl_se->runtime = dl_se->dl_runtime;
+
if (dl_time_before(dl_se->deadline, rq_clock(rq)) ||
dl_entity_overflow(dl_se, rq_clock(rq))) {
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 35/40] sched/deadline: Check the runtime of a task that comes back runnable
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (33 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 34/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
` (4 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
keeps the runtime and the deadline of a task that leaves
SCHED_DEADLINE. A task that comes back before its deadline goes on with
them. So leaving the class and coming back gives no new runtime.
But the task can come back with new parameters. If it is asleep, its
wakeup checks the runtime left against them, with dl_entity_overflow().
If it is runnable, enqueue_dl_entity() starts a new period only if the
deadline has passed. The runtime left is not checked.
So a task can come back with a smaller runtime, and still run on the
old one. Admission control counts only the new one. Juri Lelli found
this case. On v7.4-rc2, a task of 50ms every 100ms came back through
SCHED_OTHER with 10ms every 100ms. It ran 39ms of the next 100ms.
Fix this by applying the wakeup rule to a task that enters the class
runnable, with update_dl_entity(). It starts a new period, or reduces
the runtime of a constrained task, if the runtime left does not fit the
new parameters. The task gets no more than a wakeup would give it.
The return_running_new_params case of deadline_bw_test, added later in
this series, tests this. It fails on v7.4-rc2 and passes with this
patch. The return_asleep_new_params case tests the wakeup, and passes
on both.
Fixes: a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
Reported-by: Juri Lelli <juri.lelli@redhat.com>
Closes: https://lore.kernel.org/all/avHvPjz4dL0zNyGb@jlelli-thinkpadt14gen4.remote.csb/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 1a07f4109cc5..b650c37ae561 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2834,6 +2834,14 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
update_dl_entity(dl_se);
} else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se);
+ } else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+ /*
+ * The task enters the class runnable. It may come back before
+ * its deadline, with the runtime left from older parameters.
+ * Check that runtime against the new parameters, as a wakeup
+ * does.
+ */
+ update_dl_entity(dl_se);
} else if ((flags & ENQUEUE_MOVE) &&
!is_dl_boosted(dl_se) &&
dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (34 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 35/40] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:45 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 37/40] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
` (3 subsequent siblings)
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel, stable
Commit 40767b0dc768 ("sched/deadline: Fix deadline parameter
modification handling") lets new parameters take effect at the next
period. So sched_setattr() cannot give a task a fresh runtime.
But admission control counts the new bandwidth at once. When a task
shrinks its reservation, the runtime left from the old one is not
checked against the new one. A comment in sched_dl_overflow() names
the gap: the old bandwidth should stay counted until the 0-lag time.
So a running task can shrink its reservation, and still run on the old
one. Other tasks can meanwhile be admitted into the bandwidth it gave
back. On v7.4-rc2, a task of 50ms every 100ms changed to 10ms every
100ms. It ran 39ms of the next 100ms.
Fix this by applying the wakeup rule after a change of parameters, as
the previous patch does for a task that enters the class.
__setparam_dl() marks a change of the runtime, the deadline or the
period. The next enqueue then calls update_dl_entity(). It starts a new
period, or reduces the runtime of a constrained task, if the runtime
left does not fit. The task gets no more than a wakeup would give it. A
boosted task runs on the parameters of its donor. So it keeps the mark
until an enqueue after its deboost.
A larger reservation still takes effect at the next period.
The enqueue flags cannot tell a change of parameters: a cgroup move
also uses ENQUEUE_MOVE. Without the mark, a task that waited behind
others could fail the check, and get a later deadline for no reason.
The change_running_new_params case of deadline_bw_test, added later in
this series, tests this. change_boosted_new_params tests a task that
changes its parameters while boosted. Both fail on v7.4-rc2 and pass
with this patch.
Fixes: 40767b0dc768 ("sched/deadline: Fix deadline parameter modification handling")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 5 +++++
kernel/sched/deadline.c | 26 +++++++++++++++++++++-----
2 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2f2f59402f09..5bb41f350936 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -715,6 +715,10 @@ struct sched_dl_entity {
* @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
* @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
* runqueue's lock. Not for servers, which keep @dl_bw_attached.
+ *
+ * @dl_new_params tells if sched_setattr() changed the runtime, the
+ * deadline or the period. The next enqueue that is not boosted checks
+ * the runtime left against them, and clears it.
*/
unsigned int dl_throttled : 1;
unsigned int dl_yielded : 1;
@@ -729,6 +733,7 @@ struct sched_dl_entity {
unsigned int dl_defer_idle : 1;
unsigned int dl_bw_attached : 1;
unsigned int dl_bw_oncpu : 1;
+ unsigned int dl_new_params : 1;
/*
* Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b650c37ae561..f1d259df8f2a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2785,7 +2785,16 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
static void
enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
{
+ bool new_params = dl_se->dl_new_params;
+
WARN_ON_ONCE(on_dl_rq(dl_se));
+ /*
+ * A boosted task runs on the parameters of its donor. Keep the mark
+ * until an enqueue checks the runtime against its own: the one at
+ * the deboost, or at a wakeup.
+ */
+ if (!is_dl_boosted(dl_se))
+ dl_se->dl_new_params = 0;
update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
@@ -2834,12 +2843,13 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
update_dl_entity(dl_se);
} else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se);
- } else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+ } else if (((flags & ENQUEUE_CLASS) || new_params) &&
+ !is_dl_boosted(dl_se)) {
/*
- * The task enters the class runnable. It may come back before
- * its deadline, with the runtime left from older parameters.
- * Check that runtime against the new parameters, as a wakeup
- * does.
+ * The task enters the class, or its parameters change, while
+ * it is runnable. It may have runtime left from older
+ * parameters. Check that runtime against the new parameters,
+ * as a wakeup does.
*/
update_dl_entity(dl_se);
} else if ((flags & ENQUEUE_MOVE) &&
@@ -4866,6 +4876,11 @@ void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
u64 period = attr->sched_period ?: attr->sched_deadline;
bool held;
+ if (attr->sched_runtime != dl_se->dl_runtime ||
+ attr->sched_deadline != dl_se->dl_deadline ||
+ period != dl_se->dl_period)
+ dl_se->dl_new_params = 1;
+
/*
* Change the bandwidth that the runqueue counts here, with the
* parameters. A task that priority inheritance boosted into the
@@ -4999,6 +5014,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_defer_running = 0;
dl_se->dl_defer_armed = 0;
dl_se->dl_bw_oncpu = 0;
+ dl_se->dl_new_params = 0;
#ifdef CONFIG_RT_MUTEXES
dl_se->pi_se = dl_se;
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 37/40] sched/debug: Show the deadline bandwidth of each runqueue
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (35 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
` (2 subsequent siblings)
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel
For each CPU, sched_debug prints the deadline bandwidth of its root
domain. It does not print the two sums of its runqueue. this_bw is the
bandwidth of the deadline tasks assigned to the CPU. running_bw is the
part of it that belongs to active tasks.
GRUB reclaiming depends on both sums. grub_reclaim() takes this_bw -
running_bw as the inactive bandwidth of the CPU. The more inactive
bandwidth, the slower a reclaiming task uses up its runtime.
The kernel warns when a sum wraps around, and when running_bw exceeds
this_bw. Bandwidth left behind in this_bw triggers no warning.
Print both sums. The deadline_bw_test selftest added later in this
series reads this_bw in its params_waking and random_mix cases. It
checks that this_bw is back to its first value once their tasks are
gone.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/debug.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e0d6cd3bd607..3dbe1d7d2d77 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1174,6 +1174,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
SEQ_printf(m, " .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
PU(dl_nr_running);
+ PU(running_bw);
+ PU(this_bw);
dl_bw = &cpu_rq(cpu)->rd->dl_bw;
SEQ_printf(m, " .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
SEQ_printf(m, " .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (36 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 37/40] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
@ 2026-11-17 14:10 ` Mathieu Desnoyers
2026-11-17 15:23 ` sashiko-bot
2026-11-17 14:11 ` [PATCH v7 39/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-17 14:11 ` [PATCH v7 40/40] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
39 siblings, 1 reply; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:10 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel
No selftest runs a SCHED_DEADLINE task.
Add deadline_switch_test, with five cases:
- throttled_round_trip: a task of 10ms every 2s spins, so it is
throttled in every period. Right after a throttle, the test switches
it to SCHED_OTHER. The test switches it back 200us after its
deadline, between its replenishment timer and its 0-lag time. It must
then run again.
- asleep_round_trip: a task of 9ms every 10ms runs 0.2ms, then sleeps
50ms. While it sleeps past its 0-lag time, the test switches it to
SCHED_OTHER and back. A leak at each round trip fills the root
domain, and the switch back fails with EBUSY.
- sysrq_normalize: sysrq-n takes a deadline task out of the class.
Another task must then still enter SCHED_DEADLINE.
- sysrq_boosted_owner: sysrq-n takes a SCHED_OTHER task out of the
deadline class while a deadline waiter boosts it. After the unlock,
the task enters SCHED_DEADLINE at 10ms every 100ms. It must not run
more than its runtime in its first period.
- sysrq_running_owner: a SCHED_OTHER task spins alone on a CPU, with a
lock held. A deadline waiter of 100us every 1ms boosts it. The test
starts 2048 sleeping SCHED_FIFO threads, then fires sysrq-n. Every
thread must end, and the owner must no longer be boosted.
The cases need CAP_SYS_NICE, and skip without it. asleep_round_trip
also skips without admission control, when sched_rt_runtime_us is -1.
sysrq-n changes the policy of every real-time user task. So the three
sysrq cases run only with DEADLINE_TEST_SYSRQ set. They also need root
and CONFIG_MAGIC_SYSRQ. The two owner cases run only if
sysrq_normalize passed.
On a kernel with the bugs, asleep_round_trip and sysrq_normalize keep
new tasks out of SCHED_DEADLINE. So they run last, followed only by
the two owner cases.
On v7.4-rc2, in 8-CPU VMs, three of the five cases fail:
- throttled_round_trip. "sched/deadline: Do not lose a throttled task
switched out and back" fixes it.
- asleep_round_trip. "sched/deadline: Release the bandwidth of a task
that leaves the class asleep" fixes it.
- sysrq_normalize. "sched: Take cpuset_mutex for a deadline policy
change only with pi" fixes it.
sysrq_boosted_owner tests "sched: Let a task boosted by a deadline
waiter change its policy". Without the sysrq-n part of that fix, the
owner ran 99-100ms of its first period, in 4 runs of 4. With it, it ran
10-11ms.
sysrq_running_owner tests "sched/deadline: Do not replenish a boost
from cleared parameters". With the early return of that fix disabled,
the guest locked up in 3 runs of 3.
With the whole series, the four cases pass, with and without
PREEMPT_RT.
The test goes in a new selftest target, sched/deadline.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
tools/testing/selftests/Makefile | 1 +
.../selftests/sched/deadline/.gitignore | 1 +
.../testing/selftests/sched/deadline/Makefile | 8 +
tools/testing/selftests/sched/deadline/config | 1 +
.../sched/deadline/deadline_switch_test.c | 1089 +++++++++++++++++
5 files changed, 1100 insertions(+)
create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
create mode 100644 tools/testing/selftests/sched/deadline/Makefile
create mode 100644 tools/testing/selftests/sched/deadline/config
create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 07c3719f8b04..35b0c5e86ba7 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -120,6 +120,7 @@ TARGETS += rlimits
TARGETS += rseq
TARGETS += rtc
TARGETS += rust
+TARGETS += sched/deadline
TARGETS += sched_ext
TARGETS += seccomp
TARGETS += sgx
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
new file mode 100644
index 000000000000..9f203c455b91
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -0,0 +1 @@
+deadline_switch_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
new file mode 100644
index 000000000000..3f4dc1ec6119
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
+LDLIBS += -lpthread
+
+TEST_GEN_PROGS := deadline_switch_test
+
+include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
new file mode 100644
index 000000000000..3a3a7285a708
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/config
@@ -0,0 +1 @@
+CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_switch_test.c b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
new file mode 100644
index 000000000000..daf10f81cb96
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
@@ -0,0 +1,1089 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test SCHED_DEADLINE tasks that switch to SCHED_OTHER and back:
+ *
+ * - throttled_round_trip: the test switches a throttled task to
+ * SCHED_OTHER. The task's replenishment timer fires while it is out of
+ * SCHED_DEADLINE. The test switches it back after its deadline, before
+ * its 0-lag time. The task must run again.
+ * - asleep_round_trip: a task is switched out and back while it sleeps
+ * past its 0-lag time. Its bandwidth must leave the root domain when it
+ * leaves SCHED_DEADLINE. Otherwise each switch back adds it again, and
+ * sched_setattr() soon fails with EBUSY.
+ * - sysrq_normalize: sysrq-n takes a deadline task out of the class.
+ * Another task must then still enter SCHED_DEADLINE.
+ * - sysrq_boosted_owner: a deadline waiter boosts a sleeping SCHED_OTHER
+ * owner into the deadline class. sysrq-n must end the boost. When the
+ * owner later calls sched_setattr() to enter SCHED_DEADLINE, it must be
+ * throttled at its own runtime.
+ * - sysrq_running_owner: a deadline waiter boosts a running SCHED_OTHER
+ * owner. sysrq-n clears the waiter's parameters before it ends the
+ * boost. The owner must keep running until then.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE 6
+#endif
+
+#define NSEC_PER_USEC 1000ULL
+#define NSEC_PER_MSEC 1000000ULL
+#define NSEC_PER_SEC 1000000000ULL
+
+struct dl_attr {
+ uint32_t size;
+ uint32_t sched_policy;
+ uint64_t sched_flags;
+ int32_t sched_nice;
+ uint32_t sched_priority;
+ uint64_t sched_runtime;
+ uint64_t sched_deadline;
+ uint64_t sched_period;
+};
+
+/* Written by the child, read by the parent. */
+struct shared {
+ uint64_t stamp; /* last time the child ran */
+ uint64_t run_start; /* when it last resumed after a throttle */
+ uint64_t runs; /* resumptions after a throttle */
+ uint64_t sleeps; /* sleeps begun */
+};
+
+static uint64_t now_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+ struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+ while (nanosleep(&ts, &ts) && errno == EINTR)
+ ;
+}
+
+static void sleep_until(uint64_t t)
+{
+ struct timespec ts = { .tv_sec = t / NSEC_PER_SEC, .tv_nsec = t % NSEC_PER_SEC };
+
+ while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL) == EINTR)
+ ;
+}
+
+static int set_deadline(pid_t pid, uint64_t runtime, uint64_t period)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_runtime = runtime,
+ .sched_deadline = period,
+ .sched_period = period,
+ };
+
+ return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+ struct sched_param sp = { .sched_priority = 0 };
+
+ return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+static struct shared *sh;
+
+/*
+ * Fork a child that is killed when the test dies, so that it cannot keep
+ * its bandwidth. Return what fork() returns.
+ */
+static pid_t fork_child(void)
+{
+ pid_t parent = getpid();
+ pid_t pid = fork();
+
+ if (!pid && (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent))
+ _exit(KSFT_FAIL);
+ return pid;
+}
+
+static void reap(pid_t pid)
+{
+ int status;
+
+ if (pid <= 0)
+ return;
+ if (kill(pid, SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+ /*
+ * A deadline task left throttled for good never runs, even to die.
+ * Switched to SCHED_OTHER, it runs and dies. Ignore the result: the
+ * task may be dead already.
+ */
+ set_other(pid);
+ while (waitpid(pid, &status, 0) != pid) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+ break;
+ }
+}
+
+/*
+ * Report a child that died before the end of its case. Return KSFT_SKIP
+ * if it could not enter SCHED_DEADLINE, and KSFT_FAIL otherwise.
+ */
+static int gone(int status)
+{
+ if (WIFEXITED(status) && WEXITSTATUS(status) == KSFT_SKIP) {
+ ksft_print_msg("SCHED_DEADLINE refused\n");
+ return KSFT_SKIP;
+ }
+ if (WIFSIGNALED(status))
+ ksft_print_msg("the task died of signal %d\n", WTERMSIG(status));
+ else
+ ksft_print_msg("the task exited with %d\n", WEXITSTATUS(status));
+ return KSFT_FAIL;
+}
+
+/*
+ * Spin, so that the task uses up its runtime and is throttled in every
+ * period. Record in sh when it runs, and when it resumes after a
+ * throttle. A throttle stops the task for most of its 2s period. A gap
+ * of 1s or less, such as a preemption, is not a throttle.
+ */
+static void spinner(void)
+{
+ uint64_t last = now_ns();
+
+ for (;;) {
+ uint64_t t = now_ns();
+
+ if (t - last > NSEC_PER_SEC) {
+ __atomic_store_n(&sh->run_start, t, __ATOMIC_RELAXED);
+ __atomic_fetch_add(&sh->runs, 1, __ATOMIC_RELEASE);
+ }
+ __atomic_store_n(&sh->stamp, t, __ATOMIC_RELEASE);
+ last = t;
+ }
+}
+
+/*
+ * Return the runtime @pid has left, from dl.runtime in /proc/<pid>/sched.
+ * It is below zero for a throttled task: the task overran by that much.
+ * Return 1 if it cannot be read.
+ */
+static long long dl_runtime_left(pid_t pid)
+{
+ char path[64], line[256];
+ long long left = 1;
+ FILE *f;
+
+ snprintf(path, sizeof(path), "/proc/%d/sched", pid);
+ f = fopen(path, "r");
+ if (!f)
+ return 1;
+ while (fgets(line, sizeof(line), f)) {
+ char *val = strchr(line, ':');
+
+ if (!strncmp(line, "dl.runtime", 10) && val) {
+ left = atoll(val + 1);
+ break;
+ }
+ }
+ fclose(f);
+ return left;
+}
+
+#define LOST_RUNTIME (10 * NSEC_PER_MSEC)
+#define LOST_PERIOD (2 * NSEC_PER_SEC)
+#define LOST_MARGIN (200 * NSEC_PER_USEC)
+
+/*
+ * The child is a deadline task of 10ms every 2s, and spins. Three times,
+ * the parent does this:
+ * - Wait until the child resumes at a replenishment, at time r, and is
+ * throttled again.
+ * - Switch it to SCHED_OTHER.
+ * - Sleep until r + LOST_PERIOD + LOST_MARGIN. r + LOST_PERIOD is the
+ * child's deadline. Its replenishment timer fires there, on a task out
+ * of SCHED_DEADLINE.
+ * - Switch it back to SCHED_DEADLINE.
+ * - Check that it runs again within 1.5 periods.
+ *
+ * The switch back must come before the 0-lag time. The 0-lag time is after
+ * the deadline by the overrun times period/runtime, which is 200 here.
+ * The overrun is how far the task ran past its runtime before the kernel
+ * throttled it. The parent reads it from /proc. A round whose switch back
+ * comes after the 0-lag time tests nothing, and does not count. The check
+ * keeps half of LOST_MARGIN as a margin. The case is skipped if no round
+ * counts. If the overrun cannot be read, every round counts.
+ */
+static int test_throttled_round_trip(void)
+{
+ int attempt, ret = KSFT_PASS, err, counted = 0;
+ pid_t pid;
+
+ memset(sh, 0, sizeof(*sh));
+ pid = fork_child();
+ if (pid < 0) {
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ if (!pid) {
+ if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+ _exit(KSFT_SKIP);
+ spinner();
+ }
+ for (attempt = 0; attempt < 3 && ret == KSFT_PASS; attempt++) {
+ uint64_t runs, r, s, t;
+ long long left;
+
+ /*
+ * Wait until the task resumes at a replenishment, at time r.
+ * Then wait until it stops running: it is throttled.
+ */
+ runs = __atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE);
+ t = now_ns();
+ while (__atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE) == runs) {
+ if (now_ns() - t > 3 * LOST_PERIOD) {
+ int status;
+
+ if (waitpid(pid, &status, WNOHANG) == pid)
+ return gone(status);
+ ksft_print_msg("the task did not resume within 3 periods\n");
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ r = __atomic_load_n(&sh->run_start, __ATOMIC_RELAXED);
+ do {
+ s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+ sleep_ns(2 * NSEC_PER_MSEC);
+ } while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) != s);
+ /*
+ * The task is throttled until r + LOST_PERIOD. Switch it out
+ * now, and back LOST_MARGIN after that.
+ */
+ left = dl_runtime_left(pid);
+ err = set_other(pid);
+ if (!err) {
+ sleep_until(r + LOST_PERIOD + LOST_MARGIN);
+ err = set_deadline(pid, LOST_RUNTIME, LOST_PERIOD);
+ t = now_ns();
+ }
+ if (err) {
+ ksft_print_msg("policy switch: %s\n", strerror(-err));
+ ret = KSFT_FAIL;
+ break;
+ }
+ /* The deadline is at most r + LOST_PERIOD: r is read after it. */
+ if (left > 0 ||
+ t + LOST_MARGIN / 2 <
+ r + LOST_PERIOD + (uint64_t)-left * (LOST_PERIOD / LOST_RUNTIME))
+ counted++;
+ else
+ ksft_print_msg("an overrun of %lld ns: switched back after the 0-lag time\n",
+ -left);
+ sleep_ns(10 * NSEC_PER_MSEC);
+ s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+ t = now_ns();
+ while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s &&
+ now_ns() - t < LOST_PERIOD + LOST_PERIOD / 2)
+ sleep_ns(NSEC_PER_MSEC);
+ if (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s) {
+ ksft_print_msg("throttled, switched out and back: never ran again\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ if (ret == KSFT_PASS && !counted)
+ ret = KSFT_SKIP;
+ return ret;
+}
+
+#define LEAK_RUNTIME (9 * NSEC_PER_MSEC)
+#define LEAK_PERIOD (10 * NSEC_PER_MSEC)
+
+/* Run 0.2ms, then sleep 50ms, forever. Count the sleeps in sh. */
+static void sleeper(void)
+{
+ for (;;) {
+ uint64_t end = now_ns() + 200 * NSEC_PER_USEC;
+
+ while (now_ns() < end)
+ ;
+ __atomic_fetch_add(&sh->sleeps, 1, __ATOMIC_RELEASE);
+ sleep_ns(50 * NSEC_PER_MSEC);
+ }
+}
+
+/*
+ * Switch a sleeping task out of SCHED_DEADLINE and back, over and over.
+ * The task has 90% of a CPU. If each round trip leaks it, the root domain
+ * is full after about as many round trips as it has CPUs.
+ */
+static int test_asleep_round_trip(void)
+{
+ int i, trips = get_nprocs() * 10 / 9 + 3, ret = KSFT_PASS, err;
+ pid_t pid;
+ FILE *f;
+
+ f = fopen("/proc/sys/kernel/sched_rt_runtime_us", "r");
+ if (f) {
+ long rt = 0;
+
+ if (fscanf(f, "%ld", &rt) == 1 && rt < 0) {
+ fclose(f);
+ ksft_print_msg("no admission control: a leak cannot be seen\n");
+ return KSFT_SKIP;
+ }
+ fclose(f);
+ }
+ memset(sh, 0, sizeof(*sh));
+ pid = fork_child();
+ if (pid < 0) {
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ if (!pid) {
+ if (set_deadline(0, LEAK_RUNTIME, LEAK_PERIOD))
+ _exit(KSFT_SKIP);
+ sleeper();
+ }
+ for (i = 0; i < trips; i++) {
+ uint64_t sleeps = __atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE);
+ uint64_t t = now_ns();
+
+ while (__atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE) == sleeps) {
+ if (now_ns() - t > NSEC_PER_SEC) {
+ int status;
+
+ if (waitpid(pid, &status, WNOHANG) == pid)
+ return gone(status);
+ ksft_print_msg("the task did not run within 1s\n");
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ /*
+ * Wait 5ms. The task is still asleep, and past its 0-lag
+ * time.
+ */
+ sleep_ns(5 * NSEC_PER_MSEC);
+ err = set_other(pid);
+ if (!err)
+ err = set_deadline(pid, LEAK_RUNTIME, LEAK_PERIOD);
+ if (err) {
+ ksft_print_msg("switch back refused after %d of %d round trips: %s\n",
+ i, trips, strerror(-err));
+ ret = KSFT_FAIL;
+ break;
+ }
+ }
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Take a deadline task out of the class with sysrq-n. Then check that
+ * another task can still enter SCHED_DEADLINE.
+ *
+ * sysrq-n calls __sched_setscheduler() on every real-time and deadline
+ * user task, under tasklist_lock, to make it SCHED_OTHER. On a kernel
+ * with the bug, that call takes cpuset_mutex for a deadline task and
+ * never releases it. No task can enter SCHED_DEADLINE after that.
+ *
+ * sysrq-n changes the whole system. So the sysrq cases run last, and only
+ * with DEADLINE_TEST_SYSRQ set.
+ */
+static int test_sysrq_normalize(void)
+{
+ int fd, status, ret = KSFT_PASS;
+ pid_t pid, prober, w;
+ uint64_t t;
+
+ if (!getenv("DEADLINE_TEST_SYSRQ")) {
+ ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+ return KSFT_SKIP;
+ }
+ fd = open("/proc/sysrq-trigger", O_WRONLY);
+ if (fd < 0) {
+ ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ pid = fork_child();
+ if (pid < 0) {
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ close(fd);
+ return KSFT_FAIL;
+ }
+ if (!pid) {
+ if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+ _exit(KSFT_SKIP);
+ for (;;)
+ pause();
+ }
+ t = now_ns();
+ while (sched_getscheduler(pid) != SCHED_DEADLINE) {
+ if (waitpid(pid, &status, WNOHANG) == pid) {
+ close(fd);
+ return gone(status);
+ }
+ if (now_ns() - t > NSEC_PER_SEC) {
+ ksft_print_msg("the task did not enter SCHED_DEADLINE within 1s\n");
+ close(fd);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ if (write(fd, "n", 1) != 1) {
+ ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+ close(fd);
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ close(fd);
+ t = now_ns();
+ while (sched_getscheduler(pid) != SCHED_OTHER && now_ns() - t < NSEC_PER_SEC)
+ sleep_ns(NSEC_PER_MSEC);
+ if (sched_getscheduler(pid) != SCHED_OTHER) {
+ ksft_print_msg("still SCHED_DEADLINE after sysrq-n\n");
+ ret = KSFT_FAIL;
+ }
+ /* A new task must still enter SCHED_DEADLINE within 5s. */
+ prober = fork_child();
+ if (prober < 0) {
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (!prober)
+ _exit(set_deadline(0, LOST_RUNTIME, LOST_PERIOD) ? 1 : 0);
+ t = now_ns();
+ while ((w = waitpid(prober, &status, WNOHANG)) == 0) {
+ if (now_ns() - t > 5 * NSEC_PER_SEC) {
+ ksft_print_msg("SCHED_DEADLINE admission blocked after sysrq-n\n");
+ reap(pid);
+ /*
+ * The prober is blocked on a mutex. Do not wait for
+ * it.
+ */
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ if (w != prober) {
+ ksft_print_msg("waitpid %d: %s\n", prober, strerror(errno));
+ ret = KSFT_FAIL;
+ } else if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+ ksft_print_msg("SCHED_DEADLINE refused after sysrq-n\n");
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ return ret;
+}
+
+/* The priority /proc shows for a deadline task, or a task boosted by one. */
+#define DL_TASK_PRIO (-101)
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_locked, owner_go;
+static pid_t owner_tid;
+static int owner_ms; /* the owner's CPU time in its first period */
+
+static uint64_t thread_time_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/*
+ * Wait on @s for at most @ms, retrying on signals. Return 0, or -1 on a
+ * timeout or an error.
+ */
+static int sem_wait_for(sem_t *s, int ms)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+ ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ while (sem_timedwait(s, &ts)) {
+ if (errno != EINTR)
+ return -1;
+ }
+ return 0;
+}
+
+/* Return the priority @tid runs at, from /proc, or 0. */
+static int prio_of(pid_t tid)
+{
+ char path[64], buf[1024], *p;
+ ssize_t n;
+ int fd, i;
+
+ snprintf(path, sizeof(path), "/proc/%d/stat", tid);
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return 0;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n <= 0)
+ return 0;
+ buf[n] = 0;
+ /* The priority is field 18, the 16th after the command name. */
+ p = strrchr(buf, ')');
+ for (i = 0; p && i < 16; i++)
+ p = strchr(p + 1, ' ');
+ return p ? atoi(p + 1) : 0;
+}
+
+/* Join @t. Return -1 if the join fails or @t returned an error. */
+static int join(pthread_t t)
+{
+ void *res;
+ int err = pthread_join(t, &res);
+
+ if (err)
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ return err || res ? -1 : 0;
+}
+
+/*
+ * Run as the owner thread. Take pi_lock, post owner_locked, and sleep on
+ * owner_go while holding the lock. A deadline waiter boosts the owner
+ * during that sleep. Then drop the lock, become a deadline task of 10ms
+ * every 100ms, and spin for 100ms. Store the CPU time used, in ms, in
+ * owner_ms. Return NULL, or 1 on an error.
+ */
+static void *sleeping_owner(void *arg)
+{
+ uint64_t t0, cpu0;
+ int err;
+
+ owner_tid = syscall(SYS_gettid);
+ err = pthread_mutex_lock(&pi_lock);
+ if (err) {
+ ksft_print_msg("the owner cannot take the lock: %s\n", strerror(err));
+ return (void *)1;
+ }
+ sem_post(&owner_locked);
+ while (sem_wait(&owner_go) && errno == EINTR)
+ ;
+ err = pthread_mutex_unlock(&pi_lock);
+ if (err) {
+ ksft_print_msg("the owner cannot unlock: %s\n", strerror(err));
+ return (void *)1;
+ }
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, 100 * NSEC_PER_MSEC);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ t0 = now_ns();
+ cpu0 = thread_time_ns();
+ while (now_ns() - t0 < 100 * NSEC_PER_MSEC)
+ ;
+ owner_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+ set_other(0);
+ return NULL;
+}
+
+/*
+ * The waiter thread: become a deadline task of 5ms every 50ms, and wait
+ * up to 300ms for pi_lock. The owner holds the lock all along. Return
+ * NULL if the wait timed out, or 1.
+ */
+static void *boosting_waiter(void *arg)
+{
+ struct timespec ts;
+ int err;
+
+ err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+ if (err) {
+ ksft_print_msg("the waiter refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += 300 * 1000000L;
+ ts.tv_sec += ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ err = pthread_mutex_timedlock(&pi_lock, &ts);
+ if (!err)
+ pthread_mutex_unlock(&pi_lock);
+ set_other(0); /* sysrq-n may have done it */
+ if (err != ETIMEDOUT) {
+ ksft_print_msg("the waiter of the lock: %s, not a timeout\n",
+ err ? strerror(err) : "got it");
+ return (void *)1;
+ }
+ return NULL;
+}
+
+/*
+ * Check that sysrq-n ends the boost of a SCHED_OTHER lock owner.
+ *
+ * M is the main thread, O the owner, W the waiter. All three start as
+ * SCHED_OTHER. pi_lock is a priority inheritance mutex.
+ *
+ * M O W
+ * start O
+ * lock pi_lock
+ * post owner_locked
+ * sleep on owner_go
+ * start W
+ * sched_setattr():
+ * deadline,
+ * 5ms every 50ms
+ * timedlock pi_lock,
+ * 300ms: boosts O
+ * see O at deadline priority
+ * write "n" to sysrq-trigger:
+ * O and W leave the class
+ * timedlock times out:
+ * deboosts O
+ * join W
+ * post owner_go
+ * unlock pi_lock
+ * sched_setattr(): deadline,
+ * 10ms every 100ms
+ * spin 100ms, measure
+ * CPU time
+ * join O
+ *
+ * sysrq-n takes O out of the deadline class without a chain walk. The
+ * boost must end there. On a kernel with the bug it does not, and the
+ * deboost at W's timeout sees no change of priority, so it leaves the
+ * boost too. O would then enter SCHED_DEADLINE as a boosted task, and
+ * never be throttled. O must get about 10ms of CPU time in its 100ms. The
+ * case fails above 20ms.
+ *
+ * O sleeps while it is boosted, on purpose. A boosted task runs on its
+ * waiter's parameters. sysrq-n takes W out of the class too, and W's
+ * parameters can then be cleared. On a kernel that refills the runtime
+ * from cleared parameters, a running O would loop in
+ * replenish_dl_entity().
+ *
+ * Run the case only if sysrq_normalize passed. @normalized is its result.
+ * Otherwise sysrq-n may leave cpuset_mutex held, and O would block in
+ * sched_setattr().
+ */
+static int test_sysrq_boosted_owner(int normalized)
+{
+ int fd, err, ret = KSFT_PASS;
+ pthread_mutexattr_t a;
+ pthread_t o, w;
+ uint64_t t;
+
+ if (!getenv("DEADLINE_TEST_SYSRQ")) {
+ ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+ return KSFT_SKIP;
+ }
+ if (normalized != KSFT_PASS) {
+ ksft_print_msg("sysrq_normalize did not pass: sched_setattr() may block\n");
+ return KSFT_SKIP;
+ }
+ fd = open("/proc/sysrq-trigger", O_WRONLY);
+ if (fd < 0) {
+ ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ err = pthread_mutexattr_init(&a);
+ if (!err) {
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutex_init(&pi_lock, &a);
+ pthread_mutexattr_destroy(&a);
+ }
+ if (err) {
+ ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+ close(fd);
+ return KSFT_SKIP;
+ }
+ if (sem_init(&owner_locked, 0, 0) || sem_init(&owner_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ owner_ms = -1;
+ if (pthread_create(&o, NULL, sleeping_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (sem_wait_for(&owner_locked, 5000)) {
+ ksft_print_msg("the owner never took the lock\n");
+ sem_post(&owner_go); /* in case it gets there late */
+ join(o);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
+ ksft_print_msg("the waiter thread could not start\n");
+ ret = KSFT_FAIL;
+ goto unlock;
+ }
+ t = now_ns();
+ while (prio_of(owner_tid) != DL_TASK_PRIO && now_ns() - t < 200 * NSEC_PER_MSEC)
+ sleep_ns(NSEC_PER_MSEC);
+ if (prio_of(owner_tid) != DL_TASK_PRIO) {
+ ksft_print_msg("the waiter never boosted the owner\n");
+ ret = KSFT_FAIL;
+ } else if (write(fd, "n", 1) != 1) {
+ ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+ ret = KSFT_SKIP;
+ }
+ if (join(w))
+ ret = KSFT_FAIL; /* the waiter said why */
+unlock:
+ sem_post(&owner_go);
+ if (join(o)) {
+ ret = KSFT_FAIL; /* the owner said why */
+ } else if (ret == KSFT_PASS && owner_ms > 20) {
+ ksft_print_msg("the owner ran %d ms of its first 100: the boost outlived sysrq-n\n",
+ owner_ms);
+ ret = KSFT_FAIL;
+ }
+out:
+ close(fd);
+ pthread_mutex_destroy(&pi_lock);
+ return ret;
+}
+
+#define FILLERS 2048
+#define FILLER_STACK (64 * 1024)
+
+static atomic_int running_stop;
+static sem_t filler_ready, waiter_go;
+static int filler_pipe[2];
+static int running_cpu;
+
+/*
+ * Run as a filler thread: become SCHED_FIFO, and sleep until the write end
+ * of filler_pipe is closed. Return NULL, or 1 on an error.
+ */
+static void *filler(void *arg)
+{
+ struct sched_param sp = { .sched_priority = 1 };
+ int err = sched_setscheduler(0, SCHED_FIFO, &sp) ? errno : 0;
+ char c;
+
+ sem_post(&filler_ready);
+ if (err)
+ return (void *)1;
+ while (read(filler_pipe[0], &c, 1) < 0 && errno == EINTR)
+ ;
+ return NULL;
+}
+
+/*
+ * Run as the owner thread O of sysrq_running_owner, on running_cpu only.
+ * Take pi_lock and post owner_locked. Spin until running_stop is set, or
+ * for 5s. Then unlock. Return NULL, or 1 on an error.
+ */
+static void *running_owner(void *arg)
+{
+ uint64_t t0;
+ cpu_set_t one;
+ int err;
+
+ owner_tid = syscall(SYS_gettid);
+ CPU_ZERO(&one);
+ CPU_SET(running_cpu, &one);
+ if (sched_setaffinity(0, sizeof(one), &one)) {
+ ksft_print_msg("the owner cannot run on CPU %d: %s\n", running_cpu,
+ strerror(errno));
+ sem_post(&owner_locked);
+ return (void *)1;
+ }
+ err = pthread_mutex_lock(&pi_lock);
+ if (err) {
+ ksft_print_msg("the owner cannot take the lock: %s\n", strerror(err));
+ sem_post(&owner_locked);
+ return (void *)1;
+ }
+ sem_post(&owner_locked);
+ t0 = now_ns();
+ while (!atomic_load(&running_stop) && now_ns() - t0 < 5 * NSEC_PER_SEC)
+ ;
+ err = pthread_mutex_unlock(&pi_lock);
+ if (err) {
+ ksft_print_msg("the owner cannot unlock: %s\n", strerror(err));
+ return (void *)1;
+ }
+ return NULL;
+}
+
+/*
+ * Run as the waiter thread W of sysrq_running_owner. Wait for waiter_go.
+ * Then become a deadline task of 100us every 1ms, and wait for pi_lock.
+ * Return NULL once it got the lock, or 1.
+ */
+static void *running_waiter(void *arg)
+{
+ int err;
+
+ while (sem_wait(&waiter_go) && errno == EINTR)
+ ;
+ err = set_deadline(0, 100 * NSEC_PER_USEC, NSEC_PER_MSEC);
+ if (err) {
+ ksft_print_msg("the waiter refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ err = pthread_mutex_lock(&pi_lock);
+ if (!err)
+ pthread_mutex_unlock(&pi_lock);
+ set_other(0); /* sysrq-n did it already */
+ if (err) {
+ ksft_print_msg("the waiter cannot take the lock: %s\n", strerror(err));
+ return (void *)1;
+ }
+ return NULL;
+}
+
+/*
+ * Check that a boosted owner that runs survives the clearing of its
+ * donor's parameters.
+ *
+ * M is the main thread, O the owner, W the waiter. pi_lock is a priority
+ * inheritance mutex. O runs alone on the last CPU. M and the fillers run
+ * on the other CPUs. The fillers are 2048 sleeping SCHED_FIFO threads,
+ * created after W and before O.
+ *
+ * M O W
+ * start W, the fillers, O
+ * lock pi_lock
+ * post owner_locked
+ * spin
+ * post waiter_go
+ * sched_setattr():
+ * deadline,
+ * 100us every 1ms
+ * lock pi_lock:
+ * boosts O
+ * runs on W's runtime,
+ * refilled as it runs out
+ * see O at deadline priority
+ * sleep 20ms
+ * write "n" to sysrq-trigger:
+ * W leaves the class.
+ * Its parameters are
+ * cleared.
+ * Then the fillers leave
+ * their class, one by one.
+ * refills from W's cleared
+ * parameters
+ * Then O leaves the class.
+ * see O not boosted
+ * set running_stop
+ * unlock pi_lock
+ * take pi_lock, unlock
+ * join O and W
+ * end the fillers
+ *
+ * sysrq-n changes the tasks in the order of the task list, with no chain
+ * walk. The fillers keep O boosted from cleared parameters for a few ms.
+ * O refills its runtime in that time. On a kernel with the bug, that
+ * refill loops forever with O's rq lock held. sysrq-n then waits forever
+ * for that lock. The case passes if every thread ends.
+ *
+ * Run the case only if sysrq_normalize passed. @normalized is its result.
+ */
+static int test_sysrq_running_owner(int normalized)
+{
+ int fd, err, i, started = 0, ret = KSFT_PASS;
+ pthread_t o, w, *fill = NULL;
+ cpu_set_t old, others;
+ pthread_mutexattr_t a;
+ pthread_attr_t fa;
+ uint64_t t;
+
+ if (!getenv("DEADLINE_TEST_SYSRQ")) {
+ ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+ return KSFT_SKIP;
+ }
+ if (normalized != KSFT_PASS) {
+ ksft_print_msg("sysrq_normalize did not pass: sched_setattr() may block\n");
+ return KSFT_SKIP;
+ }
+ if (get_nprocs() < 2 || sched_getaffinity(0, sizeof(old), &old)) {
+ ksft_print_msg("needs 2 CPUs\n");
+ return KSFT_SKIP;
+ }
+ for (running_cpu = CPU_SETSIZE - 1; running_cpu >= 0; running_cpu--)
+ if (CPU_ISSET(running_cpu, &old))
+ break;
+ others = old;
+ CPU_CLR(running_cpu, &others);
+ if (!CPU_COUNT(&others)) {
+ ksft_print_msg("needs 2 CPUs\n");
+ return KSFT_SKIP;
+ }
+ fd = open("/proc/sysrq-trigger", O_WRONLY);
+ if (fd < 0) {
+ ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ err = pthread_mutexattr_init(&a);
+ if (!err) {
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutex_init(&pi_lock, &a);
+ pthread_mutexattr_destroy(&a);
+ }
+ if (err) {
+ ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+ close(fd);
+ return KSFT_SKIP;
+ }
+ fill = calloc(FILLERS, sizeof(*fill));
+ if (!fill || pipe(filler_pipe) || sem_init(&owner_locked, 0, 0) ||
+ sem_init(&filler_ready, 0, 0) || sem_init(&waiter_go, 0, 0) ||
+ pthread_attr_init(&fa) || pthread_attr_setstacksize(&fa, FILLER_STACK)) {
+ ksft_print_msg("setup: %s\n", strerror(errno));
+ free(fill);
+ close(fd);
+ pthread_mutex_destroy(&pi_lock);
+ return KSFT_FAIL;
+ }
+ running_stop = 0;
+ /* W, then the fillers, then O: the order of the task list */
+ if (pthread_create(&w, NULL, running_waiter, NULL)) {
+ ksft_print_msg("the waiter thread could not start\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ /*
+ * Keep M and the fillers off O's CPU. W keeps every CPU: a deadline
+ * task must be allowed on its whole root domain.
+ */
+ if (sched_setaffinity(0, sizeof(others), &others)) {
+ ksft_print_msg("sched_setaffinity: %s\n", strerror(errno));
+ sem_post(&waiter_go);
+ ret = KSFT_FAIL;
+ goto join_w;
+ }
+ for (started = 0; started < FILLERS; started++) {
+ if (pthread_create(&fill[started], &fa, filler, NULL)) {
+ ksft_print_msg("filler %d could not start\n", started);
+ ret = KSFT_FAIL;
+ break;
+ }
+ }
+ for (i = 0; i < started; i++)
+ if (sem_wait_for(&filler_ready, 5000))
+ ret = KSFT_FAIL;
+ if (ret != KSFT_PASS) {
+ sem_post(&waiter_go);
+ goto join_w;
+ }
+ if (pthread_create(&o, NULL, running_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ sem_post(&waiter_go);
+ ret = KSFT_FAIL;
+ goto join_w;
+ }
+ if (sem_wait_for(&owner_locked, 5000)) {
+ ksft_print_msg("the owner never took the lock\n");
+ ret = KSFT_FAIL;
+ }
+ sem_post(&waiter_go);
+ t = now_ns();
+ while (ret == KSFT_PASS && prio_of(owner_tid) != DL_TASK_PRIO &&
+ now_ns() - t < 200 * NSEC_PER_MSEC)
+ sleep_ns(NSEC_PER_MSEC);
+ if (ret == KSFT_PASS && prio_of(owner_tid) != DL_TASK_PRIO) {
+ ksft_print_msg("the waiter never boosted the owner\n");
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS) {
+ sleep_ns(20 * NSEC_PER_MSEC);
+ if (write(fd, "n", 1) != 1) {
+ ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+ ret = KSFT_SKIP;
+ } else if (prio_of(owner_tid) == DL_TASK_PRIO) {
+ ksft_print_msg("the owner is still boosted after sysrq-n\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ running_stop = 1;
+ if (join(o))
+ ret = KSFT_FAIL; /* the owner said why */
+join_w:
+ if (join(w))
+ ret = KSFT_FAIL; /* the waiter said why */
+out:
+ close(filler_pipe[1]);
+ for (i = 0; i < started; i++)
+ if (join(fill[i]))
+ ret = KSFT_FAIL;
+ close(filler_pipe[0]);
+ sched_setaffinity(0, sizeof(old), &old);
+ pthread_attr_destroy(&fa);
+ free(fill);
+ close(fd);
+ pthread_mutex_destroy(&pi_lock);
+ return ret;
+}
+
+static void report(const char *name, int ret)
+{
+ if (ret == KSFT_PASS)
+ ksft_test_result_pass("%s\n", name);
+ else if (ret == KSFT_SKIP)
+ ksft_test_result_skip("%s\n", name);
+ else
+ ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+ int ret;
+
+ sh = mmap(NULL, sizeof(*sh), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (sh == MAP_FAILED)
+ ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+ ksft_print_header();
+ ksft_set_plan(5);
+ /*
+ * Keep this order. On a kernel with the bugs, asleep_round_trip fills
+ * the root domain, and sysrq_normalize blocks every later admission.
+ * So sysrq_boosted_owner runs only if sysrq_normalize passed.
+ */
+ report("throttled_round_trip", test_throttled_round_trip());
+ report("asleep_round_trip", test_asleep_round_trip());
+ ret = test_sysrq_normalize();
+ report("sysrq_normalize", ret);
+ report("sysrq_boosted_owner", test_sysrq_boosted_owner(ret));
+ report("sysrq_running_owner", test_sysrq_running_owner(ret));
+ ksft_finished();
+}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 39/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (37 preceding siblings ...)
2026-11-17 14:10 ` [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
@ 2026-11-17 14:11 ` Mathieu Desnoyers
2026-11-17 14:11 ` [PATCH v7 40/40] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:11 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel
A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth until
its 0-lag time. No selftest checks what happens to this bandwidth
meanwhile.
Add deadline_rebuild_test, with three cases. In each case, a task of
400ms every second runs 380ms, then leaves SCHED_DEADLINE. Its 0-lag
time is then about 570ms ahead. P is a partition root of one CPU.
Before the 0-lag time:
- dies_before_rebuild: the task exits in P. Then a second partition
root is made, which rebuilds the accounting of every root domain.
- switched_out_before_rebuild: the same, but the task switches to
SCHED_OTHER and sleeps in P.
- moved_after_switch_out: the task switches to SCHED_OTHER outside P.
Then it moves into P and dies there.
After the 0-lag time, P must admit one task of 600ms every second, and
refuse a second one with EBUSY. A kernel with the bug removes the
bandwidth from P's root domain at the 0-lag time. But the rebuild
dropped it already, or P never had it. total_bw then wraps around, and
P admits both tasks.
The test skips without root, 3 CPUs, the cgroup v2 cpuset controller or
deadline admission control. Admission control is off when
sched_rt_runtime_us is -1.
On v7.4-rc2, in 8-CPU VMs, the three cases fail. "sched/deadline: Keep
departing bandwidth across a root domain rebuild" fixes them. With the
whole series, the three cases pass, with and without PREEMPT_RT.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../selftests/sched/deadline/.gitignore | 1 +
.../testing/selftests/sched/deadline/Makefile | 2 +-
tools/testing/selftests/sched/deadline/config | 2 +
.../sched/deadline/deadline_rebuild_test.c | 497 ++++++++++++++++++
4 files changed, 501 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index 9f203c455b91..bf7dd0da81a9 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1 +1,2 @@
deadline_switch_test
+deadline_rebuild_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index 3f4dc1ec6119..f3a1d1746d22 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
LDLIBS += -lpthread
-TEST_GEN_PROGS := deadline_switch_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
index 3a3a7285a708..a711e8af46bf 100644
--- a/tools/testing/selftests/sched/deadline/config
+++ b/tools/testing/selftests/sched/deadline/config
@@ -1 +1,3 @@
+CONFIG_CGROUPS=y
+CONFIG_CPUSETS=y
CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
new file mode 100644
index 000000000000..714a0ff03f1b
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the bandwidth of a SCHED_DEADLINE task that leaves the policy, or
+ * dies, before its 0-lag time. The bandwidth stays in the task's root
+ * domain until the inactive timer removes it. It must be removed once,
+ * from the root domain that holds it.
+ *
+ * P is a partition root of one CPU. In each case the task is a deadline
+ * task of 400ms every second, which first runs 380ms. The cases are:
+ * - dies_before_rebuild: the task is in P and exits. Before its 0-lag
+ * time, the test makes a second partition root, Q. This rebuilds the
+ * accounting of every root domain.
+ * - switched_out_before_rebuild: the same, but the task switches to
+ * SCHED_OTHER and sleeps instead of exiting.
+ * - moved_after_switch_out: the task is outside P. It switches to
+ * SCHED_OTHER and sleeps. Before its 0-lag time, the test moves it into
+ * P and kills it there.
+ *
+ * Once the inactive timer has fired, P must admit one task of 600ms
+ * every second, and refuse a second one with EBUSY. A kernel with the
+ * bug removes the bandwidth from P's root domain after the rebuild
+ * dropped it, or when P never had it. total_bw of that root domain then
+ * wraps around, and P admits both tasks.
+ *
+ * The test needs root, 3 CPUs, the cgroup v2 cpuset controller and
+ * deadline admission control.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE 6
+#endif
+
+#define NSEC_PER_MSEC 1000000ULL
+#define NSEC_PER_SEC 1000000000ULL
+
+struct dl_attr {
+ uint32_t size;
+ uint32_t sched_policy;
+ uint64_t sched_flags;
+ int32_t sched_nice;
+ uint32_t sched_priority;
+ uint64_t sched_runtime;
+ uint64_t sched_deadline;
+ uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+static int set_deadline(pid_t pid, uint64_t runtime)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_runtime = runtime,
+ .sched_deadline = NSEC_PER_SEC,
+ .sched_period = NSEC_PER_SEC,
+ };
+
+ return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+ struct sched_param sp = { .sched_priority = 0 };
+
+ return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+/*
+ * Return the CLOCK_MONOTONIC time in ns. clock_gettime() cannot fail: the
+ * clock is always supported, and the pointer is valid.
+ */
+static uint64_t now_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+ struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+ while (nanosleep(&ts, &ts) && errno == EINTR)
+ ;
+}
+
+static void reap(pid_t pid)
+{
+ int status;
+
+ if (pid <= 0)
+ return;
+ if (kill(pid, SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+ /* Ignore the result: the task may be dead already. */
+ set_other(pid);
+ while (waitpid(pid, &status, 0) != pid) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+ break;
+ }
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+ char path[PATH_MAX];
+ int fd, err = 0;
+ ssize_t n;
+
+ if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+ return -ENAMETOOLONG;
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return -errno;
+ n = write(fd, val, strlen(val));
+ if (n < 0)
+ err = -errno;
+ else if (n != (ssize_t)strlen(val))
+ err = -EIO;
+ if (close(fd) && !err)
+ err = -errno;
+ return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+ int fd = open(path, O_RDONLY);
+ ssize_t n;
+
+ if (fd < 0)
+ return -1;
+ do {
+ n = read(fd, buf, len - 1);
+ } while (n < 0 && errno == EINTR);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = 0;
+ return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+ char buf[16];
+
+ snprintf(buf, sizeof(buf), "%d", pid);
+ return write_file(cg, "cgroup.procs", buf);
+}
+
+/* Make @cg a partition root of CPU @cpu alone. Return 0, or -1 on error. */
+static int make_partition(const char *cg, int cpu)
+{
+ char path[PATH_MAX], buf[32];
+
+ if (mkdir(cg, 0755) && errno != EEXIST)
+ return -1;
+ snprintf(buf, sizeof(buf), "%d", cpu);
+ if (write_file(cg, "cpuset.cpus", buf) || write_file(cg, "cpuset.cpus.partition", "root"))
+ return -1;
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+ if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+ return -1;
+ return 0;
+}
+
+static void remove_partition(const char *cg)
+{
+ int err = write_file(cg, "cpuset.cpus.partition", "member");
+ uint64_t end;
+
+ if (err && err != -ENOENT)
+ ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+ /* A cgroup whose last task just died takes a moment to empty. */
+ end = now_ns() + NSEC_PER_SEC;
+ while (rmdir(cg)) {
+ if (errno == ENOENT)
+ return;
+ if (errno != EBUSY || now_ns() > end) {
+ ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+ return;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC);
+ }
+}
+
+/*
+ * Return the CPU time of the calling thread in ns. clock_gettime() cannot
+ * fail: the clock is always supported, and the pointer is valid.
+ */
+static uint64_t cpu_time_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+enum depart { DIES, SWITCHED_OUT };
+
+/*
+ * Fork a child that enters @cg, if given, then SCHED_DEADLINE at 400ms
+ * every second. The child runs 380ms of its runtime. Then it dies if @how
+ * is DIES. Otherwise it switches to SCHED_OTHER, writes a byte to @fd and
+ * sleeps. Its 0-lag time is then about 570ms ahead. Return the pid of the
+ * child, or -1. Only the read end of @fd stays open in the caller.
+ */
+static pid_t departing(const char *cg, enum depart how, int fd[2])
+{
+ pid_t pid = fork();
+
+ if (pid < 0)
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ if (!pid) {
+ uint64_t t0;
+
+ close(fd[0]);
+ if (cg && enter(cg, getpid()))
+ _exit(2);
+ if (set_deadline(0, 400 * NSEC_PER_MSEC))
+ _exit(3);
+ t0 = cpu_time_ns();
+ while (cpu_time_ns() - t0 < 380 * NSEC_PER_MSEC)
+ ;
+ if (how == DIES)
+ _exit(0);
+ if (set_other(0) || write(fd[1], "x", 1) != 1)
+ _exit(4);
+ for (;;)
+ pause();
+ }
+ close(fd[1]);
+ fd[1] = -1;
+ return pid;
+}
+
+/*
+ * Wait until the child is out of SCHED_DEADLINE: dead or switched out.
+ * Return 0 on success. If the child failed, return its exit status, or
+ * -1. Set *@pid to -1 once the child is reaped.
+ */
+static int departed(pid_t *pid, enum depart how, int fd)
+{
+ int status;
+ char c;
+
+ if (*pid < 0)
+ return -1;
+ /*
+ * A child that switches out sends a byte. Without the byte, the
+ * child died or failed, and its exit status tells which.
+ */
+ if (how == SWITCHED_OUT) {
+ ssize_t n;
+
+ do {
+ n = read(fd, &c, 1);
+ } while (n < 0 && errno == EINTR);
+ if (n == 1)
+ return 0;
+ }
+ if (waitpid(*pid, &status, 0) != *pid)
+ return -1;
+ *pid = -1;
+ return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
+}
+
+/*
+ * Fork a child that enters @cg and sleeps there. The child moves itself,
+ * so that it last ran on a CPU of @cg. Wait until it is there. Return its
+ * pid, or -1.
+ */
+static pid_t sleeper(const char *cg)
+{
+ int fd[2];
+ pid_t pid;
+ char c;
+
+ if (pipe(fd))
+ return -1;
+ pid = fork();
+ if (pid < 0)
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ if (!pid) {
+ close(fd[0]);
+ if (enter(cg, getpid()) || write(fd[1], "x", 1) != 1)
+ _exit(1);
+ for (;;)
+ pause();
+ }
+ close(fd[1]);
+ if (pid > 0 && read(fd[0], &c, 1) != 1) {
+ reap(pid);
+ pid = -1;
+ }
+ close(fd[0]);
+ return pid;
+}
+
+/*
+ * Check that P admits one task of 600ms every second, but not two. The
+ * caller must wait until every inactive timer has fired.
+ */
+static int check_p(void)
+{
+ pid_t a = sleeper(cg_p), b = sleeper(cg_p);
+ int ret = KSFT_FAIL, e1, e2;
+
+ if (a < 0 || b < 0) {
+ ksft_print_msg("no task could enter P\n");
+ goto out;
+ }
+ e1 = set_deadline(a, 600 * NSEC_PER_MSEC);
+ e2 = set_deadline(b, 600 * NSEC_PER_MSEC);
+ if (e1)
+ ksft_print_msg("a first task refused in P: %s\n", strerror(-e1));
+ else if (!e2)
+ ksft_print_msg("1.2 CPUs of deadline tasks admitted in a partition of one CPU\n");
+ else if (e2 != -EBUSY)
+ ksft_print_msg("the second task refused with %s, not EBUSY\n", strerror(-e2));
+ else
+ ret = KSFT_PASS;
+out:
+ reap(a);
+ reap(b);
+ return ret;
+}
+
+/*
+ * Have a task of P die or switch out, as @how says. Make Q before its
+ * 0-lag time, then check P.
+ */
+static int test_rebuild(enum depart how)
+{
+ int ret = KSFT_FAIL, fd[2], err;
+ pid_t pid = -1;
+
+ if (pipe(fd))
+ return KSFT_FAIL;
+ if (make_partition(cg_p, ncpus - 1)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ pid = departing(cg_p, how, fd);
+ err = departed(&pid, how, fd[0]);
+ if (err) {
+ ksft_print_msg("the deadline task failed (%d)\n", err);
+ goto out;
+ }
+ if (make_partition(cg_q, ncpus - 2)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 2);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ /*
+ * Wait past the 0-lag time. In the SWITCHED_OUT case the task is still
+ * alive then: it sleeps in P as a SCHED_OTHER task.
+ */
+ sleep_ns(1500 * NSEC_PER_MSEC);
+ ret = check_p();
+out:
+ reap(pid);
+ remove_partition(cg_q);
+ remove_partition(cg_p);
+ close(fd[0]);
+ if (fd[1] >= 0)
+ close(fd[1]);
+ return ret;
+}
+
+/*
+ * Have a task outside P switch to SCHED_OTHER and sleep. Before its 0-lag
+ * time, move it into P and kill it. Then check P.
+ */
+static int test_move(void)
+{
+ int ret = KSFT_FAIL, fd[2], err;
+ pid_t pid = -1;
+
+ if (pipe(fd))
+ return KSFT_FAIL;
+ if (make_partition(cg_p, ncpus - 1)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ pid = departing(NULL, SWITCHED_OUT, fd);
+ err = departed(&pid, SWITCHED_OUT, fd[0]);
+ if (err) {
+ ksft_print_msg("the deadline task failed (%d)\n", err);
+ goto out;
+ }
+ if (enter(cg_p, pid)) {
+ ksft_print_msg("the task cannot move into P\n");
+ goto out;
+ }
+ /* The task wakes up and dies on P's CPU, before its 0-lag time. */
+ reap(pid);
+ pid = -1;
+ sleep_ns(1500 * NSEC_PER_MSEC);
+ ret = check_p();
+out:
+ reap(pid);
+ remove_partition(cg_p);
+ close(fd[0]);
+ if (fd[1] >= 0)
+ close(fd[1]);
+ return ret;
+}
+
+static int find_cgroup2(void)
+{
+ char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+ FILE *f = fopen("/proc/self/mounts", "r");
+ int found = 0;
+
+ while (f && !found && fgets(line, sizeof(line), f)) {
+ if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+ !strcmp(type, "cgroup2")) {
+ snprintf(root, sizeof(root), "%s", dir);
+ found = 1;
+ }
+ }
+ if (f)
+ fclose(f);
+ return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+ if (ret == KSFT_PASS)
+ ksft_test_result_pass("%s\n", name);
+ else if (ret == KSFT_SKIP)
+ ksft_test_result_skip("%s\n", name);
+ else
+ ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+ char buf[256], path[PATH_MAX + 32];
+ int added;
+
+ ksft_print_header();
+ if (geteuid())
+ ksft_exit_skip("needs root\n");
+ ncpus = get_nprocs();
+ if (ncpus < 3)
+ ksft_exit_skip("needs 3 CPUs\n");
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+ ksft_exit_skip("no deadline admission control\n");
+ if (find_cgroup2())
+ ksft_exit_skip("cgroup v2 is not mounted\n");
+ /*
+ * Enable the cpuset controller for the root's children, if it is
+ * off. It is turned off again at the end.
+ */
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ ksft_exit_skip("no cgroup.subtree_control\n");
+ added = !strstr(buf, "cpuset");
+ if (added && write_file(root, "cgroup.subtree_control", "+cpuset"))
+ ksft_exit_skip("no cpuset controller\n");
+ if (snprintf(cg_p, sizeof(cg_p), "%s/deadline_rebuild_p", root) >= (int)sizeof(cg_p) ||
+ snprintf(cg_q, sizeof(cg_q), "%s/deadline_rebuild_q", root) >= (int)sizeof(cg_q))
+ ksft_exit_skip("cgroup v2 mount path too long\n");
+
+ ksft_set_plan(3);
+ report("dies_before_rebuild", test_rebuild(DIES));
+ report("switched_out_before_rebuild", test_rebuild(SWITCHED_OUT));
+ report("moved_after_switch_out", test_move());
+ if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+ ksft_finished();
+}
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* [PATCH v7 40/40] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (38 preceding siblings ...)
2026-11-17 14:11 ` [PATCH v7 39/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-17 14:11 ` Mathieu Desnoyers
39 siblings, 0 replies; 55+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:11 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, luca.abeni, mathieu.poirier, longman,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
linux-kselftest, linux-rt-devel
Upstream has no selftest of the SCHED_DEADLINE bandwidth accounting. A
leak or a double release does not show at once. Later, the root domain
refuses deadline tasks, or admits more than it can run.
Add deadline_bw_test, with 104 cases. Most cases count the tasks of 10ms
every 100ms that a root domain admits, before and after they run. Once
the tasks of the case are gone and past their 0-lag time, the two counts
must be equal. The cases cover:
- priority inheritance, boost and deboost
- parameter changes, SCHED_FLAG_KEEP_PARAMS among them
- cpuset moves
- what can happen during a cpuset move: a rebuild of the sched domains,
a policy change, an affinity change, a wakeup, the death of a thread,
a refusal by another controller
- deaths, sched_yield() and runtime overruns
- a stopped task and a frozen task
- bandwidth reclaiming and fair server changes
- CPU hotplug, also with admission control turned off
- sched_setaffinity() racing with sched_setattr()
- parameter changes while a task wakes on another CPU (params_waking)
- a random mix of all of the above, with affinity changes (random_mix)
The comment above each case gives its scenario. params_waking and
random_mix also compare the this_bw of their CPUs, which "sched/debug:
Show the deadline bandwidth of each runqueue" prints. Four cases follow
reproducers of Juri Lelli's sched-deadline-tests: stopped_switched_out,
hotplug_sleeper_switched, hotplug_no_admission_pinned and
pi_chain_two_owners.
The test needs root, deadline admission control and 3 CPUs. Most cases
need the cgroup v2 cpuset controller, and some need debugfs, CPU hotplug
or 4 CPUs. Twenty-three cases need CPUs isolated by isolcpus=domain.
move_cancelled needs CONFIG_RT_GROUP_SCHED. move_normalized fires
sysrq-n, which changes the whole system, so it needs
DEADLINE_TEST_SYSRQ=1. A case skips without what it needs.
The message of each fix in this series names the cases that test it.
These cases are not named there. Each fails up to the patch before its
fix, and passes from the fix:
- overrun_caught and stopped_switched_out: "sched/deadline: Release the
bandwidth of a task that leaves the class asleep"
- hotplug_departed and hotplug_no_admission_pinned: "sched/deadline:
Keep departing bandwidth across a root domain rebuild". Before it,
the second one fails in part of the runs.
- move_asleep, move_non_contending, move_died, move_woken and
move_woken_full: "sched/deadline: Record the root domain holding a
task's bandwidth"
- affinity_vs_setattr: "sched/deadline: Check SCHED_DEADLINE affinity
under the rq lock"
- params_keep_class_change: "sched: Refuse SCHED_FLAG_KEEP_PARAMS that
would change the policy or the class"
On v7.4-rc2, in 8 VMs of 8 CPUs, 35 cases fail in every VM.
overrun_caught fails in part of the runs. The same holds on tip
sched/urgent, the base of this series. The 35 cases are:
affinity_vs_setattr move_affinity_changed
boost_blocked_keeps_order move_asleep
boosted_owner_policy move_back_cpu_offline
change_boosted_new_params move_became_deadline
change_constrained_new_params move_became_deadline_no_room
change_running_new_params move_died
death_zombie_hotplug move_non_contending
death_zombie_switched move_vs_rebuild
deboost_after_donor_refill move_woken
deboost_in_class_resets_period move_woken_full
deboost_released_resets_period params_keep_boosted
deboost_resets_period params_keep_class_change
hotplug_departed pi_departed_moved
hotplug_no_admission_pinned pi_former_owner_asleep
hotplug_sleeper_switched random_mix
move_affinity_became_deadline return_running_new_params
setattr_blocked_keeps_order stopped_switched_out
wake_constrained_new_params
28 cases skip there. 25 lack what they need. hotplug_refused and
hotplug_server cannot set up their state. move_normalized was not run:
on v7.4-rc2, sysrq-n leaves cpuset_mutex locked, which "sched: Take
cpuset_mutex for a deadline policy change only with pi" fixes.
With the whole series, every case passes wherever it runs, PREEMPT_RT
included. The twenty-three isolcpus cases ran in guests booted with
isolcpus=domain,1,2 or isolcpus=domain,0,1. move_cancelled ran in two
guests built with CONFIG_RT_GROUP_SCHED. No kernel warning is logged.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://github.com/jlelli/sched-deadline-tests
---
.../selftests/sched/deadline/.gitignore | 1 +
.../testing/selftests/sched/deadline/Makefile | 2 +-
tools/testing/selftests/sched/deadline/config | 2 +
.../sched/deadline/deadline_bw_test.c | 9597 +++++++++++++++++
.../testing/selftests/sched/deadline/settings | 1 +
5 files changed, 9602 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
create mode 100644 tools/testing/selftests/sched/deadline/settings
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index bf7dd0da81a9..42f6b2945e96 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1,2 +1,3 @@
deadline_switch_test
deadline_rebuild_test
+deadline_bw_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index f3a1d1746d22..15f1e96a21e6 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
LDLIBS += -lpthread
-TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test deadline_bw_test
include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
index a711e8af46bf..b19db0c962e0 100644
--- a/tools/testing/selftests/sched/deadline/config
+++ b/tools/testing/selftests/sched/deadline/config
@@ -1,3 +1,5 @@
CONFIG_CGROUPS=y
CONFIG_CPUSETS=y
+CONFIG_DEBUG_FS=y
+CONFIG_HOTPLUG_CPU=y
CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
new file mode 100644
index 000000000000..5613f866d473
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -0,0 +1,9597 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the bandwidth accounting of SCHED_DEADLINE. The deadline tasks
+ * of the cases go through:
+ * - priority inheritance
+ * - parameter changes
+ * - cpuset moves
+ * - affinity changes
+ * - deaths, yields and runtime overruns
+ * - bandwidth reclaiming and fair server changes
+ * - CPU hotplug
+ * - a random mix of most of these
+ *
+ * Most cases measure the free bandwidth of a root domain before and after
+ * they run. probe() counts the tasks of 10ms every 100ms that the root
+ * domain admits. Once the tasks of the case are gone and their 0-lag time
+ * is past, the two counts must be equal. Fewer means leaked bandwidth.
+ * More means bandwidth released twice, which lets the root domain admit
+ * deadline tasks beyond its capacity. A deadline task that a case keeps
+ * running must also make progress.
+ *
+ * The reclaiming cases measure the share of a CPU that a deadline task
+ * with SCHED_FLAG_RECLAIM gets. That share follows extra_bw: the free
+ * bandwidth that the task's runqueue can reclaim. probe() follows the
+ * root domain's total_bw instead.
+ *
+ * The CPUs that isolcpus="domain" isolates are in the default root
+ * domain: the root domain of the isolated CPUs. The other CPUs are the
+ * housekeeping CPUs, in the housekeeping root domain. The test leaves the
+ * isolated CPUs out, and keeps its tasks off them. Only the cases that
+ * need them use them.
+ *
+ * The cases of p_cases[] run with P, a partition root of the last
+ * housekeeping CPU. P is made anew for each case. Making it rebuilds the
+ * accounting, which clears what an earlier case left.
+ *
+ * The cases of hp_cases[] run after them, without P. These are the
+ * hotplug cases, and nine other cases: mask_out_refused,
+ * partition_takes_refused, no_admission_affinity_moves,
+ * affinity_vs_setattr, rebuild_waking_keeps,
+ * partition_takes_fallback_refused, mask_grows_refused,
+ * partition_swap_refused and partition_refused_no_warn. The test skips
+ * the hotplug cases if no CPU can go offline. It still runs these nine.
+ * death_zombie_hotplug takes CPUs offline too, but it is in p_cases[]: it
+ * needs the cpuset setup of those cases.
+ *
+ * The server cases change the fair server of a CPU through debugfs, and
+ * the test then puts it back.
+ *
+ * The test needs root, deadline admission control and 3 housekeeping
+ * CPUs. Some cases need 4. Most also need the cgroup v2 cpuset
+ * controller. Some need debugfs or CPU hotplug. Twenty-three need
+ * isolcpus="domain" at boot.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+#include <linux/futex.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE 6
+#endif
+#ifndef SCHED_FLAG_RECLAIM
+#define SCHED_FLAG_RECLAIM 0x02
+#endif
+#ifndef SCHED_FLAG_DL_OVERRUN
+#define SCHED_FLAG_DL_OVERRUN 0x04
+#endif
+#ifndef SCHED_FLAG_KEEP_PARAMS
+#define SCHED_FLAG_KEEP_PARAMS 0x10
+#endif
+
+#define NSEC_PER_USEC 1000ULL
+#define NSEC_PER_MSEC 1000000ULL
+#define NSEC_PER_SEC 1000000000ULL
+
+/* The threads of probe() and hold() do nothing: a small stack will do. */
+#define THREAD_STACK (64 * 1024 > PTHREAD_STACK_MIN ? 64 * 1024 : PTHREAD_STACK_MIN)
+
+#define PROBE_RUNTIME (10 * NSEC_PER_MSEC)
+#define PROBE_PERIOD (100 * NSEC_PER_MSEC)
+#define PERIOD (100 * NSEC_PER_MSEC)
+
+struct dl_attr {
+ uint32_t size;
+ uint32_t sched_policy;
+ uint64_t sched_flags;
+ int32_t sched_nice;
+ uint32_t sched_priority;
+ uint64_t sched_runtime;
+ uint64_t sched_deadline;
+ uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+/* The online CPUs at the start, ascending, except the isolated ones */
+static int ncpus, *online;
+static int nisol, *isol; /* the online CPUs that isolcpus="domain" isolates */
+static char isolated[4096]; /* the CPU list of /sys/devices/system/cpu/isolated */
+
+/* Return the @n-th highest online CPU, counting from 1. */
+static int top_cpu(int n)
+{
+ return online[ncpus - n];
+}
+
+/*
+ * clock_gettime() cannot fail in now_ns() and thread_time_ns(). Both
+ * clocks are always supported, and the pointer is valid.
+ */
+static uint64_t now_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static uint64_t thread_time_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+ struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+ while (nanosleep(&ts, &ts) && errno == EINTR)
+ ;
+}
+
+static void spin_ns(uint64_t ns)
+{
+ uint64_t end = thread_time_ns() + ns;
+
+ while (thread_time_ns() < end)
+ ;
+}
+
+/*
+ * Poll @cond every @step ns, for at most @ms. Return 0 once it holds, or
+ * -1 on a timeout. The bound makes a child that never started, or was
+ * refused SCHED_DEADLINE, fail its case instead of hanging it.
+ */
+#define wait_for_step(ms, step, cond) \
+({ \
+ uint64_t wait_end = now_ns() + (uint64_t)(ms) * NSEC_PER_MSEC; \
+ int wait_ret = -1; \
+ \
+ while (1) { \
+ if (cond) { \
+ wait_ret = 0; \
+ break; \
+ } \
+ if (now_ns() >= wait_end) \
+ break; \
+ sleep_ns(step); \
+ } \
+ wait_ret; \
+})
+
+#define wait_for(ms, cond) wait_for_step(ms, NSEC_PER_MSEC, cond)
+
+/* Wait on @s. Retry when a signal interrupts the wait. */
+static void sem_wait_sig(sem_t *s)
+{
+ while (sem_wait(s)) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("sem_wait: %s\n", strerror(errno));
+ return;
+ }
+}
+
+/*
+ * Wait on @s for at most @ms, retrying on signals. Return 0, or -1 on a
+ * timeout or an error.
+ */
+static int sem_wait_for(sem_t *s, int ms)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+ ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ while (sem_timedwait(s, &ts)) {
+ if (errno == EINTR)
+ continue;
+ if (errno != ETIMEDOUT)
+ ksft_print_msg("sem_timedwait: %s\n", strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Fork a child that is killed when the thread that forked it dies, so
+ * that it cannot keep its bandwidth. Every caller forks from the main
+ * thread of its process. Print a message with @what if fork() fails.
+ * Return what fork() returns. The caller must handle -1.
+ */
+static pid_t fork_child(const char *what)
+{
+ pid_t parent = getpid();
+ pid_t pid = fork();
+
+ if (pid < 0)
+ ksft_print_msg("fork for %s: %s\n", what, strerror(errno));
+ else if (!pid && (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent))
+ _exit(KSFT_FAIL);
+ return pid;
+}
+
+static void reap(pid_t pid)
+{
+ int status;
+
+ if (pid <= 0)
+ return;
+ if (kill(pid, SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+ while (waitpid(pid, &status, 0) != pid) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+ break;
+ }
+}
+
+static void join(pthread_t t)
+{
+ int err = pthread_join(t, NULL);
+
+ if (err)
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+}
+
+static pid_t gettid_(void)
+{
+ return syscall(SYS_gettid);
+}
+
+static int set_deadline_flags(pid_t tid, uint64_t runtime, uint64_t period,
+ uint64_t flags)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_flags = flags,
+ .sched_runtime = runtime,
+ .sched_deadline = period,
+ .sched_period = period,
+ };
+
+ return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int set_deadline(pid_t tid, uint64_t runtime, uint64_t period)
+{
+ return set_deadline_flags(tid, runtime, period, 0);
+}
+
+/* Make @tid a deadline task with a relative deadline shorter than @period. */
+static int set_constrained(pid_t tid, uint64_t runtime, uint64_t deadline, uint64_t period)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_runtime = runtime,
+ .sched_deadline = deadline,
+ .sched_period = period,
+ };
+
+ return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t tid)
+{
+ struct dl_attr attr = { .size = sizeof(attr), .sched_policy = SCHED_OTHER };
+
+ return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+ char path[PATH_MAX];
+ int fd, err = 0;
+ ssize_t n;
+
+ if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+ return -ENAMETOOLONG;
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return -errno;
+ n = write(fd, val, strlen(val));
+ if (n < 0)
+ err = -errno;
+ else if (n != (ssize_t)strlen(val))
+ err = -EIO;
+ if (close(fd) && !err)
+ err = -errno;
+ return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+ int fd = open(path, O_RDONLY);
+ ssize_t n;
+
+ if (fd < 0)
+ return -1;
+ do {
+ n = read(fd, buf, len - 1);
+ } while (n < 0 && errno == EINTR);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = 0;
+ return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+ char buf[16];
+
+ snprintf(buf, sizeof(buf), "%d", pid);
+ return write_file(cg, "cgroup.procs", buf);
+}
+
+/* Make @cg a partition root of the CPUs in @cpus. Return 0, or -1. */
+static int make_partition_cpus(const char *cg, const char *cpus)
+{
+ char path[PATH_MAX + 32], buf[32];
+
+ if (mkdir(cg, 0755) && errno != EEXIST)
+ return -1;
+ if (write_file(cg, "cpuset.cpus", cpus) ||
+ write_file(cg, "cpuset.cpus.partition", "root"))
+ return -1;
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+ if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+ return -1;
+ return 0;
+}
+
+/* Make @cg a partition root of CPU @cpu. Return 0, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", cpu);
+ return make_partition_cpus(cg, buf);
+}
+
+static void remove_partition(const char *cg)
+{
+ int err = write_file(cg, "cpuset.cpus.partition", "member");
+
+ if (err && err != -ENOENT)
+ ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+ /* A cgroup whose last task just died takes a moment to empty. */
+ if (wait_for(1000, !rmdir(cg) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+}
+
+/* Make P a partition root of the last online CPU. Return 0, or -1. */
+static int make_p(void)
+{
+ return make_partition(cg_p, top_cpu(1));
+}
+
+static void remove_p(void)
+{
+ remove_partition(cg_p);
+}
+
+/*
+ * Return field @n of /proc/@pid/stat, or @fail. Fields are numbered as
+ * in proc(5), and @n must be 3 or more: a field after the command name.
+ */
+static int stat_field(pid_t pid, int n, int fail)
+{
+ char path[64], buf[1024], *p;
+ int i;
+
+ snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+ if (read_file(path, buf, sizeof(buf)))
+ return fail;
+ p = strrchr(buf, ')');
+ for (i = 0; p && i < n - 2; i++)
+ p = strchr(p + 1, ' ');
+ return p ? atoi(p + 1) : fail;
+}
+
+/* Return the CPU @pid last ran on, from /proc, or -1. */
+static int task_cpu_of(pid_t pid)
+{
+ return stat_field(pid, 39, -1);
+}
+
+/* The priority /proc shows for a deadline task, or a task boosted by one. */
+#define DL_TASK_PRIO (-101)
+
+/* Return the priority @pid runs at, from /proc, or 0. */
+static int prio_of(pid_t pid)
+{
+ return stat_field(pid, 18, 0);
+}
+
+/* Return the CPU time @t has used, or 0 if it cannot be read. */
+static uint64_t cpu_time_of(pthread_t t)
+{
+ static int told;
+ struct timespec ts;
+ clockid_t c;
+ int err;
+
+ err = pthread_getcpuclockid(t, &c);
+ if (!err && clock_gettime(c, &ts))
+ err = errno;
+ if (err) {
+ if (!told++)
+ ksft_print_msg("the CPU time of a thread cannot be read: %s\n",
+ strerror(err));
+ return 0;
+ }
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/* Return the state letter of @pid in /proc, or 0. */
+static char proc_state_of(pid_t pid)
+{
+ char path[64], buf[1024], *p;
+
+ snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+ if (read_file(path, buf, sizeof(buf)))
+ return 0;
+ p = strrchr(buf, ')');
+ if (!p || p[1] != ' ')
+ return 0;
+ return p[2];
+}
+
+/*
+ * Wait a period. Then return the CPU share, in percent, that @t uses
+ * over the next @ns.
+ */
+static int share_of(pthread_t t, uint64_t ns)
+{
+ uint64_t cpu, wall;
+
+ sleep_ns(PERIOD); /* a period of its own to settle in */
+ cpu = cpu_time_of(t);
+ wall = now_ns();
+ sleep_ns(ns);
+ cpu = cpu_time_of(t) - cpu;
+ wall = now_ns() - wall;
+ return wall ? (int)(100 * cpu / wall) : -1;
+}
+
+struct probe_slot {
+ pthread_t t;
+ int err;
+};
+
+static sem_t probe_done;
+static int probe_hold[2];
+
+static void *probe_thread(void *arg)
+{
+ struct probe_slot *s = arg;
+ char c;
+
+ s->err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+ if (sem_post(&probe_done))
+ ksft_print_msg("sem_post: %s\n", strerror(errno));
+ /* Sleep until the probe closes the write end. */
+ while (!s->err) {
+ ssize_t r = read(probe_hold[0], &c, 1);
+
+ if (r >= 0)
+ break;
+ if (errno != EINTR) {
+ s->err = -errno;
+ break;
+ }
+ }
+ return NULL;
+}
+
+/*
+ * Count the deadline tasks of PROBE_RUNTIME every PROBE_PERIOD that the
+ * root domain of @cg admits. A child process joins @cg, or stays in the
+ * caller's cgroup if @cg is NULL. Its threads become deadline tasks one
+ * at a time, until one is refused. Then they all exit. Return the number
+ * admitted, or -1 on an error.
+ *
+ * The threads barely run, so their bandwidth is released when they exit.
+ * probe() still waits three periods for their 0-lag times.
+ */
+static int probe(const char *cg)
+{
+ int fd[2], n = -1, max = 16 * ncpus + 16;
+ pid_t pid;
+
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ return -1;
+ }
+ pid = fork_child("the admission check");
+ if (!pid) {
+ struct probe_slot *s = calloc(max, sizeof(*s));
+ pthread_attr_t a;
+ int i, k = 0;
+
+ close(fd[0]);
+ if (!s || sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+ (cg && enter(cg, getpid())))
+ _exit(1);
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ _exit(1);
+ for (i = 0; i < max; i++) {
+ if (pthread_create(&s[i].t, &a, probe_thread, &s[i])) {
+ k = -1; /* an error, not a refusal */
+ break;
+ }
+ if (sem_wait_for(&probe_done, 10000)) {
+ k = -1;
+ i++; /* join that thread too */
+ break;
+ }
+ if (s[i].err) {
+ if (s[i].err != -EBUSY)
+ k = -1;
+ join(s[i].t);
+ break;
+ }
+ if (k >= 0)
+ k++;
+ }
+ if (i == max)
+ k = -1;
+ close(probe_hold[1]);
+ while (--i >= 0)
+ join(s[i].t);
+ if (write(fd[1], &k, sizeof(k)) != sizeof(k))
+ _exit(1);
+ _exit(0);
+ }
+ close(fd[1]);
+ if (pid < 0 || read(fd[0], &n, sizeof(n)) != sizeof(n))
+ n = -1;
+ close(fd[0]);
+ reap(pid);
+ sleep_ns(3 * PROBE_PERIOD);
+ return n;
+}
+
+/*
+ * Run @fn in a child process in @cg. Every thread of the case is then in
+ * @cg, and dies with the child. Return KSFT_PASS, KSFT_FAIL or KSFT_SKIP.
+ */
+static int run_in(const char *cg, int (*fn)(void))
+{
+ pid_t pid = fork_child("a case");
+ int status;
+
+ if (!pid) {
+ int err = cg ? enter(cg, getpid()) : 0;
+
+ if (err) {
+ ksft_print_msg("the case cannot enter %s: %s\n", cg, strerror(-err));
+ _exit(KSFT_SKIP);
+ }
+ _exit(fn());
+ }
+ if (pid < 0)
+ return KSFT_FAIL;
+ while (waitpid(pid, &status, 0) != pid) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+ return KSFT_FAIL;
+ }
+ if (!WIFEXITED(status)) {
+ ksft_print_msg("the case died of signal %d\n",
+ WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+ return KSFT_FAIL;
+ }
+ return WEXITSTATUS(status);
+}
+
+/*
+ * Run @fn in @cg with run_in(), and probe the root domain of @cg before
+ * and after. Return KSFT_SKIP if the first probe admits no task. Return
+ * KSFT_FAIL if a probe fails, or if @fn passed and the two probes differ.
+ * Otherwise return the result of @fn.
+ */
+static int measured(const char *name, const char *cg, int (*fn)(void))
+{
+ int before, after, ret;
+
+ before = probe(cg);
+ if (before <= 0) {
+ ksft_print_msg("%s: %s\n", name, before ? "the probe failed" : "no room to probe");
+ return before ? KSFT_FAIL : KSFT_SKIP;
+ }
+ ret = run_in(cg, fn);
+ sleep_ns(3 * PERIOD);
+ after = probe(cg);
+ if (after < 0) {
+ ksft_print_msg("%s: the probe failed after the case\n", name);
+ ret = KSFT_FAIL;
+ } else if (ret == KSFT_PASS && after != before) {
+ ksft_print_msg("%s: %d probe tasks admitted before, %d after: %s\n",
+ name, before, after,
+ after < before ? "bandwidth leaked" : "bandwidth released twice");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Priority inheritance */
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_ready, owner_go;
+static atomic_int owner_holds, owner_tid;
+
+/*
+ * Set up pi_lock, a priority inheritance mutex, and the owner's
+ * semaphores. Return 0, or -1 if they cannot be set up.
+ */
+static int pi_init(void)
+{
+ pthread_mutexattr_t a;
+ int err;
+
+ err = pthread_mutexattr_init(&a);
+ if (!err) {
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutex_init(&pi_lock, &a);
+ pthread_mutexattr_destroy(&a);
+ }
+ if (err) {
+ ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+ return -1;
+ }
+ if (sem_init(&owner_ready, 0, 0) || sem_init(&owner_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ return -1;
+ }
+ owner_holds = 0;
+ return 0;
+}
+
+/* Undo pi_init(). The owner and the waiters must be gone. */
+static void pi_fini(void)
+{
+ pthread_mutex_destroy(&pi_lock);
+ sem_destroy(&owner_ready);
+ sem_destroy(&owner_go);
+}
+
+/*
+ * Make the calling thread a deadline task, and wait up to @ms for
+ * pi_lock. Then switch back to SCHED_OTHER. Return 0 if the wait ends as
+ * @timeout expects: a timeout if it is set, the lock if not. Return -1
+ * otherwise.
+ */
+static int wait_lock_for(int ms, int timeout)
+{
+ struct timespec ts;
+ int err;
+
+ err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+ if (err) {
+ ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+ return -1;
+ }
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += (long)ms * 1000000L;
+ ts.tv_sec += ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ err = pthread_mutex_timedlock(&pi_lock, &ts);
+ if (!err) {
+ int e = pthread_mutex_unlock(&pi_lock);
+
+ if (e)
+ ksft_print_msg("a waiter cannot unlock: %s\n", strerror(e));
+ }
+ if (set_other(0)) {
+ ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+ return -1;
+ }
+ if (err != (timeout ? ETIMEDOUT : 0)) {
+ ksft_print_msg("a waiter of the lock: %s, not %s\n", err ? strerror(err) : "got it",
+ timeout ? "a timeout" : "the lock");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Take pi_lock as the owner, and return 0. On a failure, report it, set
+ * owner_holds to -1 and return -1.
+ */
+static int take_pi_lock(void)
+{
+ int err = pthread_mutex_lock(&pi_lock);
+
+ if (err) {
+ ksft_print_msg("an owner cannot take the lock: %s\n", strerror(err));
+ owner_holds = -1;
+ }
+ return err ? -1 : 0;
+}
+
+static void drop_pi_lock(void)
+{
+ int err = pthread_mutex_unlock(&pi_lock);
+
+ if (err)
+ ksft_print_msg("an owner cannot unlock: %s\n", strerror(err));
+}
+
+/*
+ * A chain of two priority inheritance mutexes, M1 and M2. The comment of
+ * pi_chain_two_owners() describes the threads D1, N1 and N2. chain_tid[]
+ * holds their tids, in that order. chain_step says how far the case is:
+ * - 1: N2 holds M2. N2 sets it.
+ * - 2: the main thread has checked the boost of N1. N1 may now block on
+ * M2.
+ * - 3: the main thread is done. N2 may now unlock M2.
+ */
+static pthread_mutex_t chain_m1, chain_m2;
+static atomic_int chain_tid[3], chain_step, chain_err;
+
+/*
+ * Wait up to 2s for chain_step to reach @step. Return 0 once it does, or
+ * once a thread of the chain has set chain_err. Return -1 on a timeout.
+ */
+static int chain_wait(int step)
+{
+ return wait_for(2000, chain_step >= step || chain_err);
+}
+
+/* N2: take M2, and hold it until step 3. */
+static void *chain_n2(void *arg)
+{
+ chain_tid[2] = gettid_();
+ if (pthread_mutex_lock(&chain_m2)) {
+ chain_err = 1;
+ return arg;
+ }
+ chain_step = 1;
+ if (chain_wait(3))
+ chain_err = 1;
+ pthread_mutex_unlock(&chain_m2);
+ return arg;
+}
+
+/* N1: take M1. At step 2, block on M2, which N2 holds. */
+static void *chain_n1(void *arg)
+{
+ if (pthread_mutex_lock(&chain_m1)) {
+ chain_err = 1;
+ return arg;
+ }
+ chain_tid[1] = gettid_(); /* N1 holds M1 */
+ if (chain_wait(2) || pthread_mutex_lock(&chain_m2)) {
+ chain_err = 1;
+ } else {
+ pthread_mutex_unlock(&chain_m2);
+ }
+ pthread_mutex_unlock(&chain_m1);
+ return arg;
+}
+
+/* D1: become a deadline task and block on M1, which N1 holds. */
+static void *chain_d1(void *arg)
+{
+ struct timespec ts;
+
+ chain_tid[0] = gettid_();
+ if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+ chain_err = 1;
+ return arg;
+ }
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_sec += 5;
+ if (pthread_mutex_timedlock(&chain_m1, &ts))
+ chain_err = 1;
+ else
+ pthread_mutex_unlock(&chain_m1);
+ if (set_other(0))
+ chain_err = 1;
+ return arg;
+}
+
+/*
+ * Boost two SCHED_OTHER tasks through a chain of two mutexes:
+ *
+ * D1 (deadline) waits for M1, held by N1
+ * N1 (boosted) waits for M2, held by N2
+ *
+ * N1 is boosted when D1 blocks. N2 is boosted when N1 then blocks: the
+ * boost goes through N1, which has no deadline parameters of its own.
+ * Both must be in the deadline class then. Once N2 unlocks, the three
+ * must finish.
+ */
+static int pi_chain_two_owners(void)
+{
+ pthread_mutexattr_t a;
+ int ret = KSFT_PASS, i;
+ pthread_t t[3];
+
+ if (pthread_mutexattr_init(&a) ||
+ pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT) ||
+ pthread_mutex_init(&chain_m1, &a) || pthread_mutex_init(&chain_m2, &a)) {
+ ksft_print_msg("no priority inheritance mutex\n");
+ return KSFT_SKIP;
+ }
+ chain_step = 0;
+ chain_err = 0;
+ for (i = 0; i < 3; i++)
+ chain_tid[i] = 0;
+ if (pthread_create(&t[2], NULL, chain_n2, NULL) || chain_wait(1) ||
+ pthread_create(&t[1], NULL, chain_n1, NULL) ||
+ wait_for(2000, chain_tid[1]) ||
+ pthread_create(&t[0], NULL, chain_d1, NULL)) {
+ ksft_print_msg("the threads of the chain could not start\n");
+ return KSFT_FAIL;
+ }
+ if (wait_for(2000, prio_of(chain_tid[1]) == DL_TASK_PRIO || chain_err)) {
+ ksft_print_msg("N1 is not boosted by D1\n");
+ ret = KSFT_FAIL;
+ }
+ chain_step = 2;
+ if (ret == KSFT_PASS &&
+ wait_for(2000, prio_of(chain_tid[2]) == DL_TASK_PRIO || chain_err)) {
+ ksft_print_msg("N2 is not boosted through N1\n");
+ ret = KSFT_FAIL;
+ }
+ chain_step = 3;
+ for (i = 0; i < 3; i++)
+ join(t[i]);
+ if (chain_err) {
+ ksft_print_msg("a thread of the chain failed\n");
+ ret = KSFT_FAIL;
+ }
+ pthread_mutex_destroy(&chain_m1);
+ pthread_mutex_destroy(&chain_m2);
+ return ret;
+}
+
+/*
+ * Run as the owner thread. With @arg (was_dl) set, first become a
+ * deadline task of 30ms every 100ms, run 1ms, and sleep on owner_go. The
+ * main thread switches the owner to SCHED_OTHER during that sleep. Then
+ * take pi_lock, and sleep 400ms while holding it. Return NULL, or 1 on an
+ * error.
+ */
+static void *sleeping_owner(void *arg)
+{
+ int was_dl = (long)arg;
+
+ owner_tid = gettid_();
+ if (was_dl) {
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ owner_holds = -1;
+ sem_post(&owner_ready);
+ return (void *)1;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ /*
+ * Sleep. The main thread switches this thread to SCHED_OTHER
+ * 200ms later, past its 0-lag time.
+ */
+ sem_post(&owner_ready);
+ sem_wait_sig(&owner_go);
+ }
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ sleep_ns(400 * NSEC_PER_MSEC);
+ drop_pi_lock();
+ return NULL;
+}
+
+/*
+ * A SCHED_OTHER owner takes pi_lock and sleeps 400ms while holding it.
+ * The main thread then waits 20ms for the lock, @rounds times. For each
+ * wait it becomes a deadline task of 5ms every 50ms. Each wait boosts the
+ * sleeping owner, and its timeout deboosts it. Every wait must time out.
+ *
+ * With @was_dl, the owner was a deadline task of 30ms every 100ms before.
+ * It ran 1ms and slept. The main thread switched it to SCHED_OTHER 200ms
+ * later, past its 0-lag time.
+ */
+static int pi_asleep(int was_dl, int rounds)
+{
+ pthread_t o;
+ int r, ret = KSFT_PASS;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pthread_create(&o, NULL, sleeping_owner, (void *)(long)was_dl)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (was_dl) {
+ if (sem_wait_for(&owner_ready, 5000)) {
+ ksft_print_msg("the owner did not become a deadline task\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(200 * NSEC_PER_MSEC);
+ if (set_other(owner_tid)) {
+ ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ sem_post(&owner_go);
+ }
+ if (ret == KSFT_PASS && (wait_for(5000, owner_holds) || owner_holds != 2)) {
+ ksft_print_msg("the owner never took the lock\n");
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS)
+ sleep_ns(20 * NSEC_PER_MSEC);
+ for (r = 0; r < rounds && ret == KSFT_PASS; r++) {
+ if (wait_lock_for(20, 1)) {
+ ksft_print_msg("round %d of %d failed\n", r + 1, rounds);
+ ret = KSFT_FAIL;
+ }
+ }
+ join(o);
+ return ret;
+}
+
+static int pi_other_owner_asleep(void)
+{
+ return pi_asleep(0, 5);
+}
+
+static int pi_former_owner_asleep(void)
+{
+ return pi_asleep(1, 5);
+}
+
+/*
+ * Run as the owner thread. Become a deadline task of 30ms every 100ms,
+ * and take pi_lock. Hold it for 300ms, spinning 2ms and sleeping 5ms in
+ * turn. A waiter boosts the owner when its deadline is the earlier one.
+ */
+static void *dl_owner(void *arg)
+{
+ uint64_t end;
+
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ owner_holds = -1;
+ return (void *)1;
+ }
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ end = now_ns() + 300 * NSEC_PER_MSEC;
+ while (now_ns() < end) {
+ spin_ns(2 * NSEC_PER_MSEC);
+ sleep_ns(5 * NSEC_PER_MSEC);
+ }
+ drop_pi_lock();
+ return NULL;
+}
+
+/*
+ * A deadline owner of 30ms every 100ms holds pi_lock for 300ms. The main
+ * thread waits 20ms for the lock, five times. For each wait it becomes a
+ * deadline task of 5ms every 50ms. Every wait must time out, and the
+ * owner must end without an error.
+ */
+static int pi_deadline_owner(void)
+{
+ int r, ret = KSFT_PASS, err;
+ pthread_t o;
+ void *res;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pthread_create(&o, NULL, dl_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the deadline owner never took the lock\n");
+ ret = KSFT_FAIL;
+ }
+ for (r = 0; r < 5 && ret == KSFT_PASS; r++)
+ if (wait_lock_for(20, 1))
+ ret = KSFT_FAIL;
+ err = pthread_join(o, &res);
+ if (err)
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ return res || err ? KSFT_FAIL : ret;
+}
+
+/*
+ * The owner thread: take pi_lock and spin 50ms with it. The unlock
+ * deboosts it while it runs.
+ */
+static void *running_owner(void *arg)
+{
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ spin_ns(50 * NSEC_PER_MSEC);
+ drop_pi_lock();
+ return NULL;
+}
+
+/*
+ * A SCHED_OTHER owner takes pi_lock and spins 50ms. The main thread waits
+ * for the lock as a deadline task of 5ms every 50ms, which boosts the
+ * running owner. The owner's unlock deboosts it while it runs. The main
+ * thread must get the lock within 1s. Do this five times.
+ */
+static int pi_unlock_running(void)
+{
+ pthread_t o;
+ int r, ret = KSFT_PASS;
+
+ for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pthread_create(&o, NULL, running_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (wait_for_step(5000, 100 * NSEC_PER_USEC, owner_holds) ||
+ owner_holds != 2) {
+ ksft_print_msg("the owner never took the lock\n");
+ ret = KSFT_FAIL;
+ } else if (wait_lock_for(1000, 0)) {
+ ret = KSFT_FAIL;
+ }
+ join(o);
+ pi_fini();
+ }
+ return ret;
+}
+
+/* Parameter changes */
+
+static atomic_int t_state; /* 1: about to sleep, or spinning */
+static _Atomic pid_t t_tid;
+static sem_t t_exit;
+
+/*
+ * The sleeper thread: become a deadline task of 20ms every 100ms. Run
+ * @arg ns of a fresh period, then sleep until t_exit is posted.
+ */
+static void *runs_then_sleeps(void *arg)
+{
+ uint64_t run = (uint64_t)(long)arg;
+
+ t_tid = gettid_();
+ if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD)) {
+ t_state = -1;
+ return (void *)1;
+ }
+ sleep_ns(PERIOD); /* a fresh period */
+ spin_ns(run);
+ t_state = 1;
+ sem_wait_sig(&t_exit);
+ return NULL;
+}
+
+/*
+ * Start runs_then_sleeps() in @t, to run @run ns. Return 0 once it is
+ * about to sleep, or -1.
+ */
+static int start_sleeper(pthread_t *t, uint64_t run)
+{
+ t_state = 0;
+ if (sem_init(&t_exit, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ return -1;
+ }
+ if (pthread_create(t, NULL, runs_then_sleeps, (void *)(long)run)) {
+ ksft_print_msg("the sleeper thread could not start\n");
+ return -1;
+ }
+ if (wait_for_step(5000, 100 * NSEC_PER_USEC, t_state) || t_state != 1) {
+ ksft_print_msg("the sleeper was refused SCHED_DEADLINE\n");
+ sem_post(&t_exit); /* in case it gets there late */
+ join(*t);
+ return -1;
+ }
+ return 0;
+}
+
+static void stop_sleeper(pthread_t t)
+{
+ if (sem_post(&t_exit))
+ ksft_print_msg("sem_post: %s\n", strerror(errno));
+ join(t);
+}
+
+/* Return 0 if the root domain of @cg admits @want probe tasks now, or -1. */
+static int admits_in(const char *cg, int want, const char *what)
+{
+ int n = probe(cg);
+
+ if (n != want) {
+ ksft_print_msg("%s: %d probe tasks admitted, %d expected\n", what, n, want);
+ return -1;
+ }
+ return 0;
+}
+
+/* Do the same check in the root domain of the caller's cgroup. */
+static int admits(int want, const char *what)
+{
+ return admits_in(NULL, want, what);
+}
+
+/*
+ * A deadline task of 20ms every 100ms sleeps in P, past its 0-lag time.
+ * The main thread changes its runtime to 40ms, 10ms, 30ms and 20ms. After
+ * each change, P must admit one probe task fewer for each 10ms of the new
+ * runtime.
+ */
+static int params_asleep(void)
+{
+ static const uint64_t rt[] = { 40, 10, 30, 20 };
+ int base, i, ret = KSFT_PASS;
+ pthread_t t;
+
+ if (start_sleeper(&t, NSEC_PER_MSEC))
+ return KSFT_FAIL;
+ sleep_ns(2 * PERIOD);
+ /*
+ * probe(NULL) probes P: this process is in P. The sleeper holds 20%,
+ * which is two probe tasks. base is what P admits without it.
+ */
+ base = probe(NULL) + 2;
+ for (i = 0; i < 4 && ret == KSFT_PASS; i++) {
+ char what[64];
+
+ if (set_deadline(t_tid, rt[i] * NSEC_PER_MSEC, PERIOD)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ snprintf(what, sizeof(what), "asleep, changed to %llums every 100ms",
+ (unsigned long long)rt[i]);
+ if (admits(base - (int)(rt[i] / 10), what))
+ ret = KSFT_FAIL;
+ }
+ stop_sleeper(t);
+ return ret;
+}
+
+/*
+ * A deadline task of 20ms every 100ms runs 5ms in a new period, then
+ * sleeps. Its 0-lag time is then about 20ms ahead. 1ms later, the main
+ * thread changes its runtime to 40ms, then to 10ms. measured() checks
+ * that P admits as many probe tasks after the case as before.
+ */
+static int params_non_contending(void)
+{
+ pthread_t t;
+ int ret = KSFT_PASS;
+
+ if (start_sleeper(&t, 5 * NSEC_PER_MSEC))
+ return KSFT_FAIL;
+ sleep_ns(NSEC_PER_MSEC);
+ if (set_deadline(t_tid, 40 * NSEC_PER_MSEC, PERIOD) ||
+ set_deadline(t_tid, 10 * NSEC_PER_MSEC, PERIOD))
+ ret = KSFT_FAIL;
+ stop_sleeper(t);
+ return ret;
+}
+
+static atomic_int spin_stop;
+/* Set before dl_spinner() starts. */
+static uint64_t spin_runtime, spin_flags;
+
+static void *dl_spinner(void *arg)
+{
+ t_tid = gettid_();
+ if (set_deadline_flags(0, spin_runtime, PERIOD, spin_flags)) {
+ t_state = -1;
+ return (void *)1;
+ }
+ t_state = 1;
+ while (!spin_stop)
+ ;
+ return NULL;
+}
+
+/*
+ * Start dl_spinner() in @t, a deadline task of @runtime every 100ms with
+ * @flags. Return 0, or -1.
+ */
+static int start_spinner(pthread_t *t, uint64_t runtime, uint64_t flags)
+{
+ t_state = 0;
+ spin_stop = 0;
+ spin_runtime = runtime;
+ spin_flags = flags;
+ if (pthread_create(t, NULL, dl_spinner, NULL)) {
+ ksft_print_msg("the spinner thread could not start\n");
+ return -1;
+ }
+ if (!wait_for(5000, t_state) && t_state == 1)
+ return 0;
+ spin_stop = 1; /* in case it gets there late */
+ join(*t);
+ return -1;
+}
+
+static void stop_spinner(pthread_t t)
+{
+ spin_stop = 1;
+ /* The thread may have left the class already. That is fine. */
+ set_other(t_tid);
+ join(t);
+}
+
+/*
+ * Wait until @t stops using CPU time: it is throttled. Return 0, or -1
+ * if that takes over 1s or its CPU time cannot be read.
+ */
+static int wait_throttled(pthread_t t)
+{
+ uint64_t end = now_ns() + NSEC_PER_SEC, last = cpu_time_of(t);
+
+ while (last && now_ns() < end) {
+ uint64_t cur;
+
+ sleep_ns(3 * NSEC_PER_MSEC);
+ cur = cpu_time_of(t);
+ if (cur == last)
+ return 0;
+ last = cur;
+ }
+ return -1;
+}
+
+/* Return 0 if @t runs 1ms more within @ns, or -1. */
+static int progresses(pthread_t t, uint64_t ns)
+{
+ uint64_t start = cpu_time_of(t), end = now_ns() + ns;
+
+ while (now_ns() < end) {
+ if (cpu_time_of(t) > start + NSEC_PER_MSEC)
+ return 0;
+ sleep_ns(2 * NSEC_PER_MSEC);
+ }
+ return -1;
+}
+
+/*
+ * A deadline task of 10ms every 100ms spins. Five times, wait until it is
+ * throttled, then change its runtime from the main thread: to 20ms, then
+ * back to 10ms, in turn. After each change it must run 1ms more within
+ * three periods.
+ */
+static int params_throttled(void)
+{
+ int r, ret = KSFT_PASS;
+ pthread_t t;
+
+ if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0))
+ return KSFT_FAIL;
+ for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+ if (wait_throttled(t) ||
+ set_deadline(t_tid, (r & 1 ? 10 : 20) * NSEC_PER_MSEC, PERIOD)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ if (progresses(t, 3 * PERIOD)) {
+ ksft_print_msg("throttled, parameters changed: never ran again\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ stop_spinner(t);
+ return ret;
+}
+
+/*
+ * A deadline task of 20ms every 100ms sleeps in P. The main thread tries
+ * to raise its runtime by more than P has free. sched_setattr() must fail
+ * with EBUSY, and P must admit as many probe tasks after as before.
+ */
+static int params_refused(void)
+{
+ pthread_t t;
+ int ret = KSFT_PASS, base, err;
+
+ if (start_sleeper(&t, NSEC_PER_MSEC))
+ return KSFT_FAIL;
+ sleep_ns(2 * PERIOD);
+ base = probe(NULL);
+ /*
+ * P has room for base probe tasks of 10%, and not for one more. So
+ * growing the task's runtime by (base + 1) * 10ms must be refused,
+ * whatever the admission limit.
+ */
+ if (20 + 10 * (base + 1) > 100) {
+ stop_sleeper(t);
+ ksft_print_msg("P has more room than one of its tasks can take\n");
+ return KSFT_SKIP;
+ }
+ err = set_deadline(t_tid, (20 + 10 * (uint64_t)(base + 1)) * NSEC_PER_MSEC, PERIOD);
+ if (err != -EBUSY) {
+ ksft_print_msg("an increase beyond P's room: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ } else if (admits(base, "after a refused increase")) {
+ ret = KSFT_FAIL;
+ }
+ stop_sleeper(t);
+ return ret;
+}
+
+/*
+ * A SCHED_OTHER task with a slice of 40ms asks for SCHED_DEADLINE with
+ * SCHED_FLAG_KEEP_PARAMS. That changes the class, so it must be refused
+ * with EINVAL. It must not reserve bandwidth for the parameters it would
+ * keep, which take the slice as their runtime.
+ */
+static int params_keep_class_change(void)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_OTHER,
+ .sched_runtime = 40 * NSEC_PER_MSEC,
+ };
+ int err;
+
+ if (syscall(SYS_sched_setattr, 0, &attr, 0)) {
+ ksft_print_msg("a slice of its own is refused: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ attr.sched_policy = SCHED_DEADLINE;
+ attr.sched_flags = SCHED_FLAG_KEEP_PARAMS;
+ attr.sched_deadline = PERIOD;
+ attr.sched_period = PERIOD;
+ err = syscall(SYS_sched_setattr, 0, &attr, 0) ? -errno : 0;
+ if (err != -EINVAL) {
+ ksft_print_msg("to SCHED_DEADLINE with SCHED_FLAG_KEEP_PARAMS: %s, not EINVAL\n",
+ err ? strerror(-err) : "accepted");
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+/*
+ * The waiter thread: wait up to 300ms for pi_lock as a deadline task,
+ * which boosts the owner. Return NULL if the wait timed out.
+ */
+static void *boosting_waiter(void *arg)
+{
+ return (void *)(long)wait_lock_for(300, 1);
+}
+
+/*
+ * Start a boosting_waiter() thread in @w. Wait up to 200ms for it to boost
+ * the owner. Return 0 once the owner is boosted. Otherwise report the
+ * failure, join the waiter and return -1.
+ */
+static int start_boost(pthread_t *w)
+{
+ if (pthread_create(w, NULL, boosting_waiter, NULL)) {
+ ksft_print_msg("the waiter thread could not start\n");
+ return -1;
+ }
+ if (!wait_for(200, prio_of(owner_tid) == DL_TASK_PRIO))
+ return 0;
+ ksft_print_msg("the waiter never boosted the owner\n");
+ join(*w);
+ return -1;
+}
+
+/*
+ * Join the boosting_waiter() thread @w. Return -1 if its wait did not
+ * time out.
+ */
+static int join_waiter(pthread_t w)
+{
+ void *res;
+ int err = pthread_join(w, &res);
+
+ if (err)
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ return err || res ? -1 : 0;
+}
+
+/*
+ * A SCHED_OTHER owner holds pi_lock and sleeps. The main thread gives it
+ * a slice of 40ms with sched_setattr(). A deadline waiter then boosts the
+ * owner into the deadline class. The main thread then calls
+ * sched_setattr() on the owner, with SCHED_DEADLINE and
+ * SCHED_FLAG_KEEP_PARAMS.
+ *
+ * The call must fail with EINVAL. The boost put the owner in the deadline
+ * class, but its policy is still SCHED_OTHER. So the call asks for another
+ * policy, which SCHED_FLAG_KEEP_PARAMS cannot give. If the call were
+ * admitted, it would reserve bandwidth for the 40ms slice, and nothing
+ * would release it.
+ */
+static int params_keep_boosted(void)
+{
+ struct dl_attr slice = {
+ .size = sizeof(slice),
+ .sched_policy = SCHED_OTHER,
+ .sched_runtime = 40 * NSEC_PER_MSEC, /* the slice to keep */
+ };
+ struct dl_attr keep = {
+ .size = sizeof(keep),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_flags = SCHED_FLAG_KEEP_PARAMS,
+ .sched_deadline = PERIOD,
+ .sched_period = PERIOD,
+ };
+ int ret = KSFT_PASS, err;
+ pthread_t o, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the owner never took the lock\n");
+ join(o);
+ return KSFT_FAIL;
+ }
+ if (syscall(SYS_sched_setattr, owner_tid, &slice, 0)) {
+ ksft_print_msg("the owner cannot take a slice of its own: %s\n",
+ strerror(errno));
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ if (start_boost(&w)) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ err = syscall(SYS_sched_setattr, owner_tid, &keep, 0) ? -errno : 0;
+ if (err != -EINVAL) {
+ ksft_print_msg("KEEP_PARAMS to SCHED_DEADLINE on a boosted owner: %s, not EINVAL\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ }
+ if (join_waiter(w))
+ ret = KSFT_FAIL; /* the waiter said why */
+out:
+ join(o);
+ return ret;
+}
+
+/*
+ * A deadline waiter boosts a sleeping SCHED_OTHER owner. The main thread
+ * then calls sched_setattr() on the owner, with SCHED_OTHER. The call
+ * must succeed while the owner is boosted.
+ */
+static int boosted_owner_policy(void)
+{
+ int ret = KSFT_PASS, err;
+ pthread_t o, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the owner never took the lock\n");
+ join(o);
+ return KSFT_FAIL;
+ }
+ if (start_boost(&w)) {
+ join(o);
+ return KSFT_FAIL;
+ }
+ err = set_other(owner_tid);
+ if (err) {
+ ksft_print_msg("SCHED_OTHER on a boosted owner: %s\n", strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ if (join_waiter(w))
+ ret = KSFT_FAIL; /* the waiter said why */
+ join(o);
+ return ret;
+}
+
+static atomic_int owner_first_period_ms;
+static _Atomic pid_t donor_tid;
+
+/*
+ * Run as the owner O of deboost_case(). @arg is its @mode. Store the CPU
+ * time of step 4, in ms, in owner_first_period_ms. Return NULL, 1 on an
+ * error, or 2 if no waiter boosted it within 1s.
+ */
+static void *deboosted_owner(void *arg)
+{
+ uint64_t t0, cpu0;
+ int err;
+
+ owner_tid = gettid_();
+ if (arg == (void *)1L) {
+ err = set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n",
+ strerror(-err));
+ return (void *)1;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ sem_post(&owner_ready);
+ sem_wait_sig(&owner_go);
+ }
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ t0 = now_ns();
+ while (prio_of(owner_tid) != DL_TASK_PRIO) {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ drop_pi_lock();
+ return (void *)2; /* never boosted */
+ }
+ }
+ if (arg == (void *)2L) {
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the boosted owner refused as a deadline task: %s\n",
+ strerror(-err));
+ drop_pi_lock();
+ return (void *)1;
+ }
+ }
+ spin_ns(5 * NSEC_PER_MSEC);
+ drop_pi_lock();
+ err = arg == (void *)2L ? 0 : set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ t0 = now_ns();
+ cpu0 = thread_time_ns();
+ while (now_ns() - t0 < PERIOD)
+ ;
+ owner_first_period_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+ return NULL;
+}
+
+/*
+ * The waiter thread: take pi_lock as a deadline task of 40ms every
+ * 100ms. It boosts the owner, which then runs on the waiter's runtime.
+ * Return NULL once it got the lock.
+ */
+static void *donor_waiter(void *arg)
+{
+ int err;
+
+ donor_tid = gettid_();
+ err = set_deadline(0, 40 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ err = take_pi_lock();
+ if (!err)
+ drop_pi_lock();
+ if (set_other(0))
+ ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+ return (void *)(long)err;
+}
+
+/*
+ * Run a deboost case. The owner O starts as SCHED_OTHER. The donor D is
+ * the waiter that boosts it: a deadline task of 40ms every 100ms.
+ * 1. O takes pi_lock and spins.
+ * 2. D blocks on pi_lock. This boosts O, which then runs on D's runtime.
+ * 3. O spins 5ms more and unlocks. The unlock deboosts O.
+ * 4. O becomes a deadline task of 10ms every 100ms. It spins for 100ms
+ * and measures its CPU time.
+ *
+ * O must get about 10ms in step 4. The case fails above 20ms: O then
+ * still runs on the runtime that the boost left.
+ *
+ * @mode changes one step:
+ * - 0: nothing.
+ * - 1: before step 1, O runs 1ms as a deadline task of 60ms every 100ms,
+ * and sleeps. The main thread switches it to SCHED_OTHER 300ms later,
+ * past its 0-lag time.
+ * - 2: O becomes the deadline task of step 4 in step 3 instead, before
+ * the 5ms. It stays in the deadline class at the deboost.
+ */
+static int deboost_case(long mode)
+{
+ int err, ret = KSFT_PASS;
+ void *o_ret, *w_ret;
+ pthread_t o, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ owner_first_period_ms = -1;
+ if (pthread_create(&o, NULL, deboosted_owner, (void *)mode)) {
+ ksft_print_msg("the owner thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (mode == 1) {
+ if (sem_wait_for(&owner_ready, 5000)) {
+ /* The owner may get there late. Let it end. */
+ sem_post(&owner_go);
+ join(o);
+ return KSFT_FAIL; /* the owner said why */
+ }
+ sleep_ns(3 * PERIOD);
+ err = proc_state_of(owner_tid) == 'S' ? set_other(owner_tid) : -EAGAIN;
+ sem_post(&owner_go);
+ if (err) {
+ ksft_print_msg("the owner could not leave the class asleep: %s\n",
+ strerror(-err));
+ join(o);
+ return KSFT_FAIL;
+ }
+ }
+ if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the owner never took the lock\n");
+ join(o);
+ return KSFT_FAIL;
+ }
+ if (pthread_create(&w, NULL, donor_waiter, NULL)) {
+ ksft_print_msg("the waiter thread could not start\n");
+ join(o);
+ return KSFT_FAIL;
+ }
+ err = pthread_join(w, &w_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ } else if (w_ret) {
+ ret = KSFT_FAIL; /* the waiter said why */
+ }
+ err = pthread_join(o, &o_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ } else if (o_ret == (void *)2) {
+ ksft_print_msg("the waiter never boosted the owner\n");
+ ret = KSFT_FAIL;
+ } else if (owner_first_period_ms < 0) {
+ ret = KSFT_FAIL;
+ } else if (owner_first_period_ms > 20) {
+ ksft_print_msg("the owner ran %d ms of its first 100: the donor's runtime\n",
+ (int)owner_first_period_ms);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Run deboost_case() with a plain SCHED_OTHER owner. */
+static int deboost_resets_period(void)
+{
+ return deboost_case(0);
+}
+
+/*
+ * Run deboost_case() with an owner that left the deadline class asleep.
+ * Its bandwidth was released then.
+ */
+static int deboost_released_resets_period(void)
+{
+ return deboost_case(1);
+}
+
+/* Run deboost_case() with an owner that enters the class while boosted. */
+static int deboost_in_class_resets_period(void)
+{
+ return deboost_case(2);
+}
+
+/*
+ * Return the runtime that the deadline task @tid has left, in ns, from
+ * dl.runtime in /proc/<tid>/sched. Return -1 if it cannot be read.
+ */
+static long long dl_runtime_left(pid_t tid)
+{
+ char path[64], line[256];
+ long long left = -1;
+ FILE *f;
+
+ snprintf(path, sizeof(path), "/proc/%d/sched", tid);
+ f = fopen(path, "r");
+ if (!f)
+ return -1;
+ while (fgets(line, sizeof(line), f)) {
+ char *val = strchr(line, ':');
+
+ if (!strncmp(line, "dl.runtime", 10) && val) {
+ left = atoll(val + 1);
+ break;
+ }
+ }
+ fclose(f);
+ return left;
+}
+
+#define REFILL_OWN_RUNTIME (5 * NSEC_PER_MSEC)
+#define REFILL_DONOR_RUNTIME (2 * NSEC_PER_MSEC)
+#define REFILL_DONOR_PERIOD (10 * NSEC_PER_MSEC)
+
+static sem_t refill_donor_ready;
+static atomic_int refill_donor_locking;
+static _Atomic long long refill_owner_left;
+
+/*
+ * Run as the donor D of refill_case(). Become a deadline task of 2ms
+ * every 10ms and sleep until the owner holds pi_lock. Then block on
+ * pi_lock. Return NULL once it got the lock, or 1 on an error.
+ */
+static void *refill_donor(void *arg)
+{
+ int err;
+
+ donor_tid = gettid_();
+ err = set_deadline(0, REFILL_DONOR_RUNTIME, REFILL_DONOR_PERIOD);
+ if (err) {
+ ksft_print_msg("the donor refused as a deadline task: %s\n", strerror(-err));
+ donor_tid = -1;
+ sem_post(&refill_donor_ready);
+ return (void *)1;
+ }
+ sem_post(&refill_donor_ready);
+ sem_wait_sig(&owner_go);
+ refill_donor_locking = 1;
+ err = take_pi_lock();
+ if (!err)
+ drop_pi_lock();
+ if (set_other(0))
+ ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
+ return (void *)(long)err;
+}
+
+/*
+ * Run as the owner O of refill_case(). @arg is the CPU time, in ms, that
+ * O spins with the lock once it is boosted. Store the runtime O has left
+ * after the unlock in refill_owner_left. Return NULL, 1 on an error, or 2
+ * if the donor did not block on the lock within 1s.
+ */
+static void *refill_owner(void *arg)
+{
+ uint64_t t0;
+ int err;
+
+ owner_tid = gettid_();
+ err = set_deadline(0, REFILL_OWN_RUNTIME, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ spin_ns(2 * NSEC_PER_MSEC);
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ sem_post(&owner_go);
+ t0 = now_ns();
+ while (!refill_donor_locking || proc_state_of(donor_tid) != 'S') {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ drop_pi_lock();
+ set_other(0);
+ return (void *)2;
+ }
+ }
+ spin_ns((long)arg * NSEC_PER_MSEC);
+ drop_pi_lock();
+ refill_owner_left = dl_runtime_left(owner_tid);
+ if (set_other(0))
+ ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+ return NULL;
+}
+
+/*
+ * Run a case on the runtime that a deadline owner has after its boost.
+ * O and D share the CPU of P.
+ *
+ * owner O: 5ms every 100ms donor D: 2ms every 10ms
+ *
+ * sleeps
+ * starts its period, runs 2ms
+ * takes pi_lock
+ * wakes D up
+ * preempts O: its deadline is
+ * earlier
+ * blocks on pi_lock, boosts O
+ * runs @spin_ms more
+ * unlocks: the boost ends
+ * takes pi_lock, unlocks
+ * reads its dl.runtime
+ *
+ * A boosted task that runs out of runtime is not throttled. Its runtime
+ * is refilled from the parameters of its donor: 2ms here.
+ *
+ * - @spin_ms is 8: O runs out of its own 5ms, and is refilled from D at
+ * least once. At the unlock it holds what the last refill left, 2ms at
+ * most. That is not the runtime of its own reservation. O must start a
+ * new period at the unlock, with 5ms. The case fails if O has 3.5ms or
+ * less.
+ * - @spin_ms is 1: O has used about 3ms of its own 5ms at the unlock, and
+ * was never refilled. It must keep what it has left. A new period would
+ * hand it a full runtime at every contended unlock. The case fails if O
+ * has 4ms or more.
+ */
+static int refill_case(long spin_ms)
+{
+ void *o_ret = NULL, *d_ret = NULL;
+ int err, ret = KSFT_PASS;
+ pthread_t o, d;
+ long long left;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (sem_init(&refill_donor_ready, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ refill_donor_locking = 0;
+ refill_owner_left = -1;
+ donor_tid = 0;
+ if (pthread_create(&d, NULL, refill_donor, NULL)) {
+ ksft_print_msg("the donor thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (sem_wait_for(&refill_donor_ready, 5000) || donor_tid <= 0) {
+ ksft_print_msg("the donor is not ready\n");
+ sem_post(&owner_go);
+ join(d);
+ return KSFT_FAIL;
+ }
+ if (pthread_create(&o, NULL, refill_owner, (void *)spin_ms)) {
+ ksft_print_msg("the owner thread could not start\n");
+ sem_post(&owner_go);
+ join(d);
+ return KSFT_FAIL;
+ }
+ err = pthread_join(o, &o_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ }
+ if (o_ret == (void *)1)
+ sem_post(&owner_go); /* the owner never woke the donor up */
+ err = pthread_join(d, &d_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ }
+ left = refill_owner_left;
+ if (o_ret == (void *)2) {
+ ksft_print_msg("the donor never blocked on the lock\n");
+ ret = KSFT_FAIL;
+ } else if (o_ret || d_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (left < 0) {
+ ksft_print_msg("the runtime of the owner cannot be read\n");
+ ret = KSFT_SKIP;
+ } else if (spin_ms == 8 && left <= 3500 * NSEC_PER_USEC) {
+ ksft_print_msg("the owner has %lld us after the unlock: what the donor's refill left\n",
+ left / NSEC_PER_USEC);
+ ret = KSFT_FAIL;
+ } else if (spin_ms == 1 && left >= 4 * NSEC_PER_MSEC) {
+ ksft_print_msg("the owner has %lld us after the unlock: a new runtime, with no refill from the donor\n",
+ left / NSEC_PER_USEC);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Run refill_case() with an owner that is refilled from its donor. */
+static int deboost_after_donor_refill(void)
+{
+ return refill_case(8);
+}
+
+/* Run refill_case() with an owner that stays on its own runtime. */
+static int deboost_keeps_own_runtime(void)
+{
+ return refill_case(1);
+}
+
+static sem_t ret_ready, ret_go;
+static _Atomic pid_t ret_tid;
+static atomic_int ret_first_period_ms;
+
+/*
+ * Run as the thread of the return cases. Become a deadline task of 50ms
+ * every 100ms and run 20ms. 30ms of runtime are left, and the 0-lag time
+ * is about 20ms ahead. Then change to new parameters, as @arg says:
+ * - 0: the thread switches itself to SCHED_OTHER and back to
+ * SCHED_DEADLINE, while it runs.
+ * - 1: the thread sleeps. The main thread switches it to SCHED_OTHER and
+ * back.
+ * - 2: the thread changes its own parameters while it runs. It stays in
+ * SCHED_DEADLINE.
+ * - 3: as 2, while a donor boosts the thread. The thread takes pi_lock and
+ * wakes the donor up. The donor blocks on pi_lock. The thread then
+ * changes its parameters, and unlocks.
+ * - 4: as 2, to a constrained deadline: 8ms in 20ms, every 100ms.
+ * - 5: the thread sleeps. The main thread changes it to 8ms in 20ms,
+ * every 100ms.
+ * The new parameters are 10ms every 100ms, except in modes 4 and 5.
+ * Then spin for 100ms, and store the CPU time used, in ms, in
+ * ret_first_period_ms. Return NULL, or 1 on an error.
+ */
+static void *returner(void *arg)
+{
+ long mode = (long)arg;
+ uint64_t t0, cpu0;
+ int err = 0;
+
+ ret_tid = gettid_();
+ err = set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the task refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ spin_ns(20 * NSEC_PER_MSEC);
+ if (mode == 1 || mode == 5) {
+ sem_post(&ret_ready);
+ sem_wait_sig(&ret_go);
+ } else if (mode == 4) {
+ err = set_constrained(0, 8 * NSEC_PER_MSEC, 20 * NSEC_PER_MSEC, PERIOD);
+ } else if (mode == 3) {
+ if (take_pi_lock())
+ return (void *)1;
+ sem_post(&owner_go);
+ t0 = now_ns();
+ while (!refill_donor_locking || proc_state_of(donor_tid) != 'S') {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ ksft_print_msg("the donor did not block on the lock\n");
+ drop_pi_lock();
+ return (void *)1;
+ }
+ }
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ drop_pi_lock();
+ } else {
+ err = mode ? 0 : set_other(0);
+ if (!err)
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ }
+ if (err) {
+ ksft_print_msg("the task could not change: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ t0 = now_ns();
+ cpu0 = thread_time_ns();
+ while (now_ns() - t0 < PERIOD)
+ ;
+ ret_first_period_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+ return NULL;
+}
+
+/*
+ * Run a return case: returner() with @mode as its argument. The thread
+ * must get about 10ms of CPU time in the 100ms after the change, 8ms in
+ * modes 4 and 5. That is its new runtime, not the 30ms left from the old
+ * one. The case fails above 20ms.
+ */
+static int return_case(long mode)
+{
+ void *t_ret, *d_ret = NULL;
+ int err, ret = KSFT_PASS;
+ pthread_t t, d;
+
+ if (sem_init(&ret_ready, 0, 0) || sem_init(&ret_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ ret_first_period_ms = -1;
+ if (mode == 3) {
+ /* The donor: a deadline task of 2ms every 10ms */
+ if (pi_init() || sem_init(&refill_donor_ready, 0, 0))
+ return KSFT_SKIP;
+ refill_donor_locking = 0;
+ donor_tid = 0;
+ if (pthread_create(&d, NULL, refill_donor, NULL)) {
+ ksft_print_msg("the donor thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (sem_wait_for(&refill_donor_ready, 5000) || donor_tid <= 0) {
+ ksft_print_msg("the donor is not ready\n");
+ sem_post(&owner_go);
+ join(d);
+ return KSFT_FAIL;
+ }
+ }
+ if (pthread_create(&t, NULL, returner, (void *)mode)) {
+ ksft_print_msg("the task thread could not start\n");
+ if (mode == 3) {
+ sem_post(&owner_go);
+ join(d);
+ }
+ return KSFT_FAIL;
+ }
+ if (mode == 1 || mode == 5) {
+ if (sem_wait_for(&ret_ready, 5000)) {
+ /* The thread may get there late. Let it end. */
+ sem_post(&ret_go);
+ join(t);
+ return KSFT_FAIL; /* the task said why */
+ }
+ /* Before its 0-lag time, about 20ms ahead */
+ err = wait_for(10, proc_state_of(ret_tid) == 'S') ? -EAGAIN : 0;
+ if (!err && mode == 1)
+ err = set_other(ret_tid);
+ if (!err && mode == 1)
+ err = set_deadline(ret_tid, 10 * NSEC_PER_MSEC, PERIOD);
+ if (!err && mode == 5)
+ err = set_constrained(ret_tid, 8 * NSEC_PER_MSEC, 20 * NSEC_PER_MSEC,
+ PERIOD);
+ sem_post(&ret_go);
+ if (err) {
+ ksft_print_msg("the task could not change asleep: %s\n",
+ strerror(-err));
+ join(t);
+ return KSFT_FAIL;
+ }
+ }
+ err = pthread_join(t, &t_ret);
+ if (mode == 3) {
+ if (t_ret)
+ sem_post(&owner_go); /* the task never woke the donor up */
+ if (pthread_join(d, &d_ret) || d_ret)
+ ret = KSFT_FAIL;
+ pi_fini();
+ sem_destroy(&refill_donor_ready);
+ }
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ } else if (t_ret || ret_first_period_ms < 0) {
+ ret = KSFT_FAIL; /* the task said why */
+ } else if (ret_first_period_ms > 20) {
+ ksft_print_msg("the task ran %d ms of its first 100: the old runtime\n",
+ (int)ret_first_period_ms);
+ ret = KSFT_FAIL;
+ }
+ sem_destroy(&ret_ready);
+ sem_destroy(&ret_go);
+ return ret;
+}
+
+/*
+ * A running task leaves SCHED_DEADLINE and comes back at once with a
+ * smaller runtime. Its 0-lag time is ahead, so it keeps its deadline and
+ * the runtime it has left. It must not run on more than the new runtime.
+ */
+static int return_running_new_params(void)
+{
+ return return_case(0);
+}
+
+/* The same, for a task that leaves and comes back while it sleeps. */
+static int return_asleep_new_params(void)
+{
+ return return_case(1);
+}
+
+/* The same, for a task that changes its parameters within the class. */
+static int change_running_new_params(void)
+{
+ return return_case(2);
+}
+
+/*
+ * The same, for a task that changes its parameters while a donor boosts
+ * it. The donor is a deadline task of 2ms every 10ms. Its deadline is the
+ * earlier one, so the task runs on its own runtime while boosted. The
+ * unlock ends the boost. The runtime left must then be checked against
+ * the new parameters.
+ */
+static int change_boosted_new_params(void)
+{
+ return return_case(3);
+}
+
+/*
+ * A running task of 50ms every 100ms has 30ms left. It changes itself to
+ * 8ms in 20ms, every 100ms. Its deadline stays 80ms ahead, further than
+ * the new 20ms. The check of the runtime left uses the new density over
+ * those 80ms. That allows 32ms, so the check keeps the 30ms. The task
+ * must keep no more than its new 8ms.
+ */
+static int change_constrained_new_params(void)
+{
+ return return_case(4);
+}
+
+/*
+ * The same, for a task that the main thread changes while it sleeps. The
+ * check is at its wakeup.
+ */
+static int wake_constrained_new_params(void)
+{
+ return return_case(5);
+}
+
+static pthread_mutex_t pi_lock2;
+static sem_t holder_go;
+static atomic_int holder_holds, chain_waits, donor_done, x_waits, l2_takes;
+static atomic_int chain_rank, x_rank;
+static _Atomic pid_t x_tid, fair_tid;
+
+/* The holder thread: take pi_lock2, and hold it until holder_go. */
+static void *l2_holder(void *arg)
+{
+ int err = pthread_mutex_lock(&pi_lock2);
+
+ if (err) {
+ ksft_print_msg("the holder cannot take pi_lock2: %s\n", strerror(err));
+ holder_holds = -1;
+ return (void *)1;
+ }
+ holder_holds = 1;
+ sem_wait_sig(&holder_go);
+ pthread_mutex_unlock(&pi_lock2);
+ return NULL;
+}
+
+/*
+ * Take pi_lock2. Store in @rank how many threads took it before the
+ * caller. Then unlock it. Return 0, or -1 if the lock cannot be taken.
+ */
+static int take_l2_ranked(atomic_int *rank)
+{
+ int err = pthread_mutex_lock(&pi_lock2);
+
+ if (err) {
+ ksft_print_msg("a waiter cannot take pi_lock2: %s\n", strerror(err));
+ return -1;
+ }
+ *rank = l2_takes++;
+ pthread_mutex_unlock(&pi_lock2);
+ return 0;
+}
+
+/*
+ * Run as the owner thread, a deadline task of 10ms every 200ms. Take
+ * pi_lock, and sleep until owner_go is posted. The donor boosts the owner
+ * during that sleep. Then spin 20ms. The owner's 10ms run out, and the
+ * boost refills its runtime from the donor's 40ms. Then wait on pi_lock2
+ * while holding pi_lock. The donor's timeout deboosts the owner during
+ * that wait. The owner then has more runtime left than its own 10ms.
+ * Return NULL, 1 on an error, or 2 if the spin took over 50ms: the owner
+ * was throttled, so it was not boosted.
+ */
+static void *chain_owner(void *arg)
+{
+ uint64_t t0;
+ int err;
+
+ owner_tid = gettid_();
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, 2 * PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+ owner_holds = -1;
+ return (void *)1;
+ }
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ sem_wait_sig(&owner_go);
+ t0 = now_ns();
+ spin_ns(20 * NSEC_PER_MSEC);
+ if (now_ns() - t0 > PERIOD / 2) {
+ drop_pi_lock();
+ return (void *)2; /* throttled: never boosted */
+ }
+ chain_waits = 1;
+ err = take_l2_ranked(&chain_rank);
+ drop_pi_lock();
+ return err ? (void *)1 : NULL;
+}
+
+/*
+ * The donor thread: a deadline task of 40ms every 100ms. Wait 150ms for
+ * pi_lock. Its timeout deboosts the owner, which waits on pi_lock2 then.
+ */
+static void *timed_donor(void *arg)
+{
+ struct timespec ts;
+ int err;
+
+ donor_tid = gettid_();
+ err = set_deadline(0, 40 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the donor refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += 150 * 1000000L;
+ ts.tv_sec += ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ err = pthread_mutex_timedlock(&pi_lock, &ts);
+ donor_done = 1;
+ if (!err)
+ pthread_mutex_unlock(&pi_lock);
+ if (set_other(0))
+ ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
+ if (err != ETIMEDOUT) {
+ ksft_print_msg("the donor: %s, not a timeout\n",
+ err ? strerror(err) : "got pi_lock");
+ return (void *)1;
+ }
+ return NULL;
+}
+
+/*
+ * The other waiter: a deadline task of 5ms every 50ms. Its deadline is
+ * earlier than the one the owner gets at the deboost. Wait on pi_lock2.
+ */
+static void *early_waiter(void *arg)
+{
+ int err;
+
+ x_tid = gettid_();
+ err = set_deadline(0, 5 * NSEC_PER_MSEC, PERIOD / 2);
+ if (err) {
+ ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ x_waits = 1;
+ err = take_l2_ranked(&x_rank);
+ if (set_other(0))
+ ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+ return err ? (void *)1 : NULL;
+}
+
+/* Set up pi_lock2 and holder_go. Return 0, or -1. */
+static int pi_init_lock2(void)
+{
+ pthread_mutexattr_t a;
+ int err;
+
+ err = pthread_mutexattr_init(&a);
+ if (!err) {
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutex_init(&pi_lock2, &a);
+ pthread_mutexattr_destroy(&a);
+ }
+ if (err) {
+ ksft_print_msg("no second priority inheritance mutex: %s\n", strerror(err));
+ return -1;
+ }
+ if (sem_init(&holder_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ pthread_mutex_destroy(&pi_lock2);
+ return -1;
+ }
+ holder_holds = 0;
+ chain_waits = 0;
+ donor_done = 0;
+ x_waits = 0;
+ l2_takes = 0;
+ chain_rank = -1;
+ x_rank = -1;
+ donor_tid = 0;
+ x_tid = 0;
+ fair_tid = 0;
+ return 0;
+}
+
+/*
+ * Check the order of the waiters of pi_lock2 after a deboost. The threads
+ * are:
+ * - the holder: SCHED_OTHER. It has pi_lock2.
+ * - the owner: deadline, 10ms every 200ms. It has pi_lock.
+ * - the donor: deadline, 40ms every 100ms.
+ * - the other waiter: deadline, 5ms every 50ms.
+ *
+ * 1. The donor waits on pi_lock, and boosts the owner.
+ * 2. The owner spins on the donor's runtime, then waits on pi_lock2.
+ * 3. The other waiter waits on pi_lock2.
+ * 4. The donor times out after 150ms. The deboost starts a new period
+ * for the owner, which has more runtime left than its own 10ms.
+ * 5. The holder drops pi_lock2.
+ *
+ * The other waiter must take pi_lock2 first: its deadline is earlier
+ * than the owner's new one. At step 4 the owner is not queued, and the
+ * chain walk copies its deadline into its waiter node on pi_lock2. A
+ * deadline of 0 in the node would put the owner first.
+ */
+static int deboost_blocked_keeps_order(void)
+{
+ void *o_ret = NULL, *d_ret = NULL, *x_ret = NULL;
+ int ret = KSFT_FAIL, have_o = 0, have_d = 0, have_x = 0;
+ pthread_t h, o, d, x;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pi_init_lock2()) {
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ if (pthread_create(&h, NULL, l2_holder, NULL)) {
+ ksft_print_msg("the holder thread could not start\n");
+ goto out_fini;
+ }
+ if (wait_for(5000, holder_holds) || holder_holds != 1) {
+ ksft_print_msg("the holder never took pi_lock2\n");
+ goto out;
+ }
+ have_o = !pthread_create(&o, NULL, chain_owner, NULL);
+ if (!have_o || wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the owner never took pi_lock\n");
+ goto out;
+ }
+ have_d = !pthread_create(&d, NULL, timed_donor, NULL);
+ if (!have_d || wait_for(1000, donor_tid && proc_state_of(donor_tid) == 'S')) {
+ ksft_print_msg("the donor never waited on pi_lock\n");
+ goto out;
+ }
+ sem_post(&owner_go);
+ if (wait_for(1000, chain_waits && proc_state_of(owner_tid) == 'S')) {
+ ksft_print_msg("the owner never waited on pi_lock2\n");
+ goto out;
+ }
+ if (donor_done) {
+ ksft_print_msg("the donor timed out before the owner waited on pi_lock2\n");
+ goto out;
+ }
+ have_x = !pthread_create(&x, NULL, early_waiter, NULL);
+ if (!have_x || wait_for(1000, x_waits && proc_state_of(x_tid) == 'S')) {
+ ksft_print_msg("the other waiter never waited on pi_lock2\n");
+ goto out;
+ }
+ /* The donor's timeout deboosts the owner, which waits on pi_lock2. */
+ have_d = 0;
+ if (!pthread_join(d, &d_ret) && !d_ret)
+ ret = KSFT_PASS;
+out:
+ /* The owner may get there late. Let it end. */
+ sem_post(&owner_go);
+ sem_post(&holder_go);
+ if (have_d)
+ join(d);
+ if (have_x && pthread_join(x, &x_ret))
+ ret = KSFT_FAIL;
+ if (have_o && pthread_join(o, &o_ret))
+ ret = KSFT_FAIL;
+ join(h);
+ if (o_ret == (void *)2) {
+ ksft_print_msg("the donor never boosted the owner\n");
+ ret = KSFT_FAIL;
+ } else if (o_ret || x_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (ret == KSFT_PASS && chain_rank < x_rank) {
+ ksft_print_msg("the deboosted owner took pi_lock2 before the earlier waiter\n");
+ ret = KSFT_FAIL;
+ }
+out_fini:
+ pthread_mutex_destroy(&pi_lock2);
+ sem_destroy(&holder_go);
+ pi_fini();
+ return ret;
+}
+
+/*
+ * The fair waiter: wait on pi_lock2 as a fair task. With @arg set, hold
+ * pi_lock meanwhile, for a donor to boost it. Otherwise the main thread
+ * makes it a deadline task while it waits. It leaves the class at the end.
+ */
+static void *fair_waiter(void *arg)
+{
+ int err;
+
+ fair_tid = gettid_();
+ if (arg && take_pi_lock())
+ return (void *)1;
+ chain_waits = 1;
+ err = take_l2_ranked(&chain_rank);
+ if (arg)
+ drop_pi_lock();
+ else if (set_other(0))
+ ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+ return err ? (void *)1 : NULL;
+}
+
+/*
+ * Check the order of the waiters of pi_lock2 when one of them enters the
+ * class while it waits. The holder has pi_lock2:
+ * 1. The fair waiter waits on pi_lock2. With @boost, it holds pi_lock.
+ * 2. The other waiter, a deadline task, waits on pi_lock2.
+ * 3. The fair waiter enters the class:
+ * - Without @boost, the main thread makes it a deadline task of 10ms
+ * every 200ms.
+ * - With @boost, a donor of 40ms every 100ms waits on pi_lock, and
+ * boosts it.
+ * 4. The holder drops pi_lock2.
+ *
+ * The other waiter must take pi_lock2 first: it started its period
+ * earlier, with a shorter deadline. At step 3 the fair waiter is not
+ * queued, and the chain walk copies its deadline into its waiter node on
+ * pi_lock2. A deadline of 0 in the node would put the fair waiter first.
+ */
+static int blocked_keeps_order(long boost)
+{
+ void *f_ret = NULL, *x_ret = NULL, *w_ret = NULL;
+ int ret = KSFT_FAIL, have_f = 0, have_x = 0, have_w = 0, err;
+ pthread_t h, f, x, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pi_init_lock2()) {
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ if (pthread_create(&h, NULL, l2_holder, NULL)) {
+ ksft_print_msg("the holder thread could not start\n");
+ goto out_fini;
+ }
+ if (wait_for(5000, holder_holds) || holder_holds != 1) {
+ ksft_print_msg("the holder never took pi_lock2\n");
+ goto out;
+ }
+ have_f = !pthread_create(&f, NULL, fair_waiter, (void *)boost);
+ if (!have_f || wait_for(1000, chain_waits && proc_state_of(fair_tid) == 'S')) {
+ ksft_print_msg("the fair waiter never waited on pi_lock2\n");
+ goto out;
+ }
+ have_x = !pthread_create(&x, NULL, early_waiter, NULL);
+ if (!have_x || wait_for(1000, x_waits && proc_state_of(x_tid) == 'S')) {
+ ksft_print_msg("the other waiter never waited on pi_lock2\n");
+ goto out;
+ }
+ if (boost) {
+ have_w = !pthread_create(&w, NULL, donor_waiter, NULL);
+ if (!have_w || wait_for(1000, donor_tid && proc_state_of(donor_tid) == 'S')) {
+ ksft_print_msg("the donor never waited on pi_lock\n");
+ goto out;
+ }
+ if (prio_of(fair_tid) != DL_TASK_PRIO) {
+ ksft_print_msg("the donor never boosted the fair waiter\n");
+ goto out;
+ }
+ } else {
+ err = set_deadline(fair_tid, 10 * NSEC_PER_MSEC, 2 * PERIOD);
+ if (err) {
+ ksft_print_msg("the fair waiter refused as a deadline task: %s\n",
+ strerror(-err));
+ if (err == -EBUSY)
+ ret = KSFT_SKIP; /* no room in P */
+ goto out;
+ }
+ }
+ ret = KSFT_PASS;
+out:
+ sem_post(&holder_go); /* pi_lock2 goes to its top waiter */
+ if (have_w && pthread_join(w, &w_ret))
+ ret = KSFT_FAIL;
+ if (have_x && pthread_join(x, &x_ret))
+ ret = KSFT_FAIL;
+ if (have_f && pthread_join(f, &f_ret))
+ ret = KSFT_FAIL;
+ join(h);
+ if (f_ret || x_ret || w_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (ret == KSFT_PASS && chain_rank < x_rank) {
+ ksft_print_msg("the fair waiter took pi_lock2 before the earlier waiter\n");
+ ret = KSFT_FAIL;
+ }
+out_fini:
+ pthread_mutex_destroy(&pi_lock2);
+ sem_destroy(&holder_go);
+ pi_fini();
+ return ret;
+}
+
+/* sched_setattr() puts the fair waiter in the class. */
+static int setattr_blocked_keeps_order(void)
+{
+ return blocked_keeps_order(0);
+}
+
+/* A boost puts the fair waiter in the class. */
+static int boost_blocked_keeps_order(void)
+{
+ return blocked_keeps_order(1);
+}
+
+/*
+ * The threads of pi_chain_donor_leaves(): O, M, W, Z and Y. leave_tid[]
+ * holds their tids. leave_y_ns is the CPU time Y gets in O's window, or
+ * -1.
+ */
+enum { LV_O, LV_M, LV_W, LV_Z, LV_Y, LV_N };
+static _Atomic pid_t leave_tid[LV_N];
+static atomic_int leave_err, leave_m_locking, leave_w_locking, leave_y_stop;
+static _Atomic long long leave_y_ns;
+static sem_t leave_o_go, leave_m_go;
+
+/*
+ * O: take pi_lock2 (L2), and sleep until leave_o_go. Then spin 100ms.
+ * Measure the CPU time of Y (@arg) over the next 200ms, and unlock L2.
+ */
+static void *leave_o(void *arg)
+{
+ pthread_t y = *(pthread_t *)arg;
+ uint64_t t0, y0;
+
+ if (pthread_mutex_lock(&pi_lock2)) {
+ leave_err = 1;
+ return arg;
+ }
+ leave_tid[LV_O] = gettid_(); /* O holds L2 */
+ sem_wait_sig(&leave_o_go);
+ t0 = now_ns();
+ while (now_ns() - t0 < PERIOD)
+ ;
+ y0 = cpu_time_of(y);
+ t0 = now_ns();
+ while (now_ns() - t0 < 2 * PERIOD)
+ ;
+ leave_y_ns = cpu_time_of(y) - y0;
+ pthread_mutex_unlock(&pi_lock2);
+ return NULL;
+}
+
+/*
+ * M: become a deadline task of 10ms every 1s, and take pi_lock (L1). At
+ * leave_m_go, block on L2, which O holds.
+ */
+static void *leave_m(void *arg)
+{
+ if (set_deadline(0, 10 * NSEC_PER_MSEC, NSEC_PER_SEC) ||
+ pthread_mutex_lock(&pi_lock)) {
+ leave_err = 1;
+ return arg;
+ }
+ leave_tid[LV_M] = gettid_(); /* M holds L1 */
+ sem_wait_sig(&leave_m_go);
+ leave_m_locking = 1;
+ if (pthread_mutex_lock(&pi_lock2))
+ leave_err = 1;
+ else
+ pthread_mutex_unlock(&pi_lock2);
+ pthread_mutex_unlock(&pi_lock);
+ if (set_other(0))
+ leave_err = 1;
+ return arg;
+}
+
+/*
+ * W and Z: become a deadline task of @arg ms every 40 times that, and
+ * block on L1. W is the one of 5ms. The main thread switches it to
+ * SCHED_OTHER while it waits.
+ */
+static void *leave_waiter(void *arg)
+{
+ uint64_t runtime = (long)arg * NSEC_PER_MSEC;
+ int w = (long)arg == 5;
+
+ if (set_deadline(0, runtime, 40 * runtime)) {
+ leave_err = 1;
+ return arg;
+ }
+ leave_tid[w ? LV_W : LV_Z] = gettid_();
+ if (w)
+ leave_w_locking = 1;
+ if (pthread_mutex_lock(&pi_lock))
+ leave_err = 1;
+ else
+ pthread_mutex_unlock(&pi_lock);
+ if (set_other(0))
+ leave_err = 1;
+ return arg;
+}
+
+/* Y: become a deadline task of 30ms every 100ms, and spin. */
+static void *leave_y(void *arg)
+{
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ leave_err = 1;
+ return arg;
+ }
+ leave_tid[LV_Y] = gettid_();
+ while (!leave_y_stop)
+ ;
+ return arg;
+}
+
+/*
+ * A chain walk must pass a new donor on to the end of the chain. This
+ * holds even when the sort keys of the task in the middle do not change.
+ *
+ * L1 is pi_lock and L2 is pi_lock2. All threads run on the CPU of P.
+ * Each deadline thread calls sched_setattr() on itself when it starts.
+ * The threads are:
+ * - O: SCHED_OTHER.
+ * - M: a deadline task of 10ms every 1s.
+ * - W: a deadline task of 5ms every 200ms.
+ * - Z: a deadline task of 2ms every 80ms.
+ * - Y: a deadline task of 30ms every 100ms. It spins all along.
+ *
+ * 1. O takes L2, and sleeps.
+ * 2. M takes L1, and sleeps.
+ * 3. W blocks on L1. W's deadline is earlier than M's, so W boosts M. M
+ * keeps its own deadline.
+ * 4. M blocks on L2. M passes W on to O, as O's donor. O enters the
+ * deadline class. It sleeps, so it gets no runtime yet.
+ * 5. Z blocks on L1. Z's deadline is earlier than W's, so Z becomes M's
+ * donor. M sleeps, so its deadline does not move. The chain walk must
+ * go on to O and make Z O's donor too.
+ * 6. The main thread switches W to SCHED_OTHER. W sleeps and has run
+ * almost nothing, so its parameters are cleared at once.
+ * 7. 250ms later, the main thread wakes O up. O has no deadline ahead, so
+ * its wakeup starts a new period from its donor. O spins 100ms.
+ * Then it measures the CPU time Y gets over 200ms, and unlocks L2.
+ *
+ * With Z as its donor, O gets 2ms every 80ms. Y's deadline is soon the
+ * earlier one, and Y gets about its 60ms. With W as its donor, O runs on
+ * cleared parameters. Each refill gives it no runtime and a deadline of
+ * about now. O then runs before Y all the time, and Y gets close to
+ * nothing. The case fails if Y gets less than 30ms.
+ *
+ * W does not exit before O unlocks L2. It waits for L1, which M releases
+ * after it gets L2.
+ */
+static int pi_chain_donor_leaves(void)
+{
+ int ret = KSFT_FAIL, i, have[LV_N] = { 0 };
+ pthread_t t[LV_N];
+ long long y_ns;
+ int err;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pi_init_lock2()) {
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ if (sem_init(&leave_o_go, 0, 0) || sem_init(&leave_m_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ goto out_fini;
+ }
+ for (i = 0; i < LV_N; i++)
+ leave_tid[i] = 0;
+ leave_err = 0;
+ leave_m_locking = 0;
+ leave_w_locking = 0;
+ leave_y_stop = 0;
+ leave_y_ns = -1;
+ have[LV_Y] = !pthread_create(&t[LV_Y], NULL, leave_y, NULL);
+ if (!have[LV_Y] || wait_for(1000, leave_tid[LV_Y] || leave_err)) {
+ ksft_print_msg("Y never became a deadline task\n");
+ goto out;
+ }
+ have[LV_O] = !pthread_create(&t[LV_O], NULL, leave_o, &t[LV_Y]);
+ if (!have[LV_O] || wait_for(1000, leave_tid[LV_O] || leave_err)) {
+ ksft_print_msg("O never took L2\n");
+ goto out;
+ }
+ have[LV_M] = !pthread_create(&t[LV_M], NULL, leave_m, NULL);
+ if (!have[LV_M] || wait_for(1000, leave_tid[LV_M] || leave_err)) {
+ ksft_print_msg("M never took L1\n");
+ goto out;
+ }
+ have[LV_W] = !pthread_create(&t[LV_W], NULL, leave_waiter, (void *)5L);
+ if (!have[LV_W] ||
+ wait_for(1000, (leave_w_locking && proc_state_of(leave_tid[LV_W]) == 'S') ||
+ leave_err)) {
+ ksft_print_msg("W never blocked on L1\n");
+ goto out;
+ }
+ sem_post(&leave_m_go);
+ if (wait_for(1000, (leave_m_locking && prio_of(leave_tid[LV_O]) == DL_TASK_PRIO) ||
+ leave_err)) {
+ ksft_print_msg("O was not boosted through M\n");
+ goto out;
+ }
+ have[LV_Z] = !pthread_create(&t[LV_Z], NULL, leave_waiter, (void *)2L);
+ if (!have[LV_Z] ||
+ wait_for(1000, (leave_tid[LV_Z] && proc_state_of(leave_tid[LV_Z]) == 'S') ||
+ leave_err)) {
+ ksft_print_msg("Z never blocked on L1\n");
+ goto out;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC); /* the chain walk of Z is done */
+ err = leave_err ? -EINVAL : set_other(leave_tid[LV_W]);
+ if (err) {
+ ksft_print_msg("W cannot leave SCHED_DEADLINE: %s\n", strerror(-err));
+ goto out;
+ }
+ sleep_ns(250 * NSEC_PER_MSEC);
+ ret = KSFT_PASS;
+out:
+ /* Let every thread end: O unlocks L2, then M unlocks L1. */
+ sem_post(&leave_m_go);
+ sem_post(&leave_o_go);
+ for (i = LV_O; i < LV_Y; i++)
+ if (have[i])
+ join(t[i]);
+ leave_y_stop = 1;
+ if (have[LV_Y])
+ join(t[LV_Y]);
+ y_ns = leave_y_ns;
+ if (leave_err) {
+ ksft_print_msg("a thread of the chain failed\n");
+ ret = KSFT_FAIL;
+ } else if (ret == KSFT_PASS && y_ns < 30 * (long long)NSEC_PER_MSEC) {
+ ksft_print_msg("Y ran %lld ms of 200: O ran on the cleared parameters of W\n",
+ y_ns / (long long)NSEC_PER_MSEC);
+ ret = KSFT_FAIL;
+ }
+ sem_destroy(&leave_o_go);
+ sem_destroy(&leave_m_go);
+out_fini:
+ pthread_mutex_destroy(&pi_lock2);
+ sem_destroy(&holder_go);
+ pi_fini();
+ return ret;
+}
+
+/* cpuset moves, from the main process, outside P */
+
+/* Shared with the processes of a case: atomic in shared memory. */
+static atomic_int *mv_state;
+
+/*
+ * Wait up to 5s for a child to report in mv_state[0]. Return 0 if it is
+ * up (1). Return -1 if it was refused SCHED_DEADLINE (-1), could not
+ * start its threads (-2) or did not report. @fast polls every 100us.
+ */
+static int child_up(const char *what, int fast)
+{
+ if (wait_for_step(5000, fast ? 100 * NSEC_PER_USEC : NSEC_PER_MSEC, mv_state[0])) {
+ ksft_print_msg("%s: the task never started\n", what);
+ return -1;
+ }
+ if (mv_state[0] == -2) {
+ ksft_print_msg("%s: the process cannot start its threads\n", what);
+ return -1;
+ }
+ if (mv_state[0] != 1) {
+ ksft_print_msg("%s: the task was refused SCHED_DEADLINE\n", what);
+ return -1;
+ }
+ return 0;
+}
+
+enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_YIELDED, MV_NON_CONTENDING };
+
+/*
+ * Fork a process in the caller's cgroup. The process becomes a deadline
+ * task of @runtime every 100ms, and does what @mode says:
+ * - MV_ASLEEP: run 1ms, set mv_state[0] to 1, then sleep.
+ * - MV_THROTTLED: set mv_state[0] to 1, then spin and count in
+ * mv_state[1].
+ * - MV_YIELDED: set mv_state[0] to 1. Then spin 0.5ms, count in
+ * mv_state[1] and yield, in a loop.
+ * - MV_NON_CONTENDING: sleep a period, run a quarter of its runtime, set
+ * mv_state[0] to 1, then sleep. Its 0-lag time is still ahead.
+ * If it is refused SCHED_DEADLINE, the process sets mv_state[0] to -1 and
+ * exits. Return its pid, or -1.
+ */
+static pid_t mover(enum mv_mode mode, uint64_t runtime)
+{
+ pid_t pid = fork_child("a deadline task");
+
+ if (!pid) {
+ if (set_deadline(0, runtime, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ if (mode == MV_ASLEEP) {
+ spin_ns(NSEC_PER_MSEC);
+ *mv_state = 1;
+ for (;;)
+ pause();
+ }
+ if (mode == MV_NON_CONTENDING) {
+ sleep_ns(PERIOD);
+ spin_ns(runtime / 4);
+ *mv_state = 1;
+ for (;;)
+ pause();
+ }
+ if (mode == MV_YIELDED) {
+ *mv_state = 1;
+ for (;;) {
+ spin_ns(NSEC_PER_MSEC / 2);
+ mv_state[1]++;
+ sched_yield();
+ }
+ }
+ *mv_state = 1;
+ for (;;)
+ mv_state[1]++;
+ }
+ return pid;
+}
+
+static int wait_spinner(int moving);
+
+/*
+ * Move a deadline task into P and back, from outside P. The task is a
+ * mover() process of 30ms every 100ms, forked in the cgroup of the main
+ * process. Wait until it is in the state of @mode:
+ * - MV_ASLEEP: asleep for two periods, past its 0-lag time.
+ * - MV_NON_CONTENDING: asleep for 1ms, before its 0-lag time.
+ * - MV_THROTTLED, MV_YIELDED: its counter has stopped for 5ms.
+ * Then write its pid to cgroup.procs of P, then of the root cgroup. Both
+ * moves must succeed. A throttled or yielded task must run again within
+ * three periods. Then kill the task. P and the root cgroup must each
+ * admit as many probe tasks as before the case.
+ */
+static int moves(enum mv_mode mode, const char *name)
+{
+ int base_p, base_r, after_p, after_r, ret = KSFT_PASS;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = mover(mode, 30 * NSEC_PER_MSEC);
+ if (pid < 0 || child_up(name, mode == MV_NON_CONTENDING)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (mode == MV_NON_CONTENDING) {
+ sleep_ns(NSEC_PER_MSEC);
+ } else if (mode == MV_ASLEEP) {
+ sleep_ns(2 * PERIOD);
+ } else if (wait_spinner(0)) {
+ /* Throttled, or yielded: its counter stops. */
+ ksft_print_msg("%s: never stopped\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ ret = KSFT_FAIL;
+ }
+ if (mode == MV_THROTTLED || mode == MV_YIELDED) {
+ int last = mv_state[1];
+
+ sleep_ns(3 * PERIOD);
+ if (mv_state[1] == last) {
+ ksft_print_msg("%s: moved, never ran again\n", name);
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+ ksft_print_msg("%s: probe tasks admitted, P %d -> %d, the other CPUs %d -> %d\n",
+ name, base_p, after_p, base_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * A deadline task of 60ms every 100ms sleeps in P. Move a second sleeping
+ * task of 60ms every 100ms into P from outside. The move must fail with
+ * EBUSY. P and the root cgroup must each admit as many probe tasks after
+ * as before.
+ */
+static int move_refused(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, err;
+ pid_t in_p, pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ in_p = fork_child("a deadline task of P");
+ if (!in_p) {
+ if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+ reap(in_p);
+ return KSFT_FAIL;
+ }
+ mv_state[0] = 0;
+ pid = mover(MV_ASLEEP, 60 * NSEC_PER_MSEC);
+ if (pid < 0 || child_up("a task of 60% to move into P", 0)) {
+ reap(pid);
+ reap(in_p);
+ return KSFT_FAIL;
+ }
+ err = enter(cg_p, pid);
+ if (err != -EBUSY) {
+ ksft_print_msg("a move beyond P's room: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ reap(in_p);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("a refused move changed the free bandwidth\n");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * A runtime of 1024ns every 2s gives a bandwidth of 0. A cpuset move
+ * skips such a task: it has no bandwidth to move.
+ *
+ * A deadline task of 60ms every 100ms sleeps in P. A second task sleeps
+ * outside P. The main process makes it a deadline task of 1024ns every
+ * 2s, moves it into P, and wakes it. It then runs on P's CPU. The main
+ * process then raises it to 60ms every 100ms. P has no room for that, so
+ * sched_setattr() must fail with EBUSY. The kernel must check the request
+ * against P's root domain, not against the one the task left.
+ */
+static int move_zero_bw(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, err, zfd[2];
+ pid_t in_p, pid;
+
+ if (pipe(zfd))
+ return KSFT_FAIL;
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ in_p = fork_child("a deadline task of P");
+ if (!in_p) {
+ if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+ reap(in_p);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ mv_state[0] = 0;
+ pid = fork_child("a task with no bandwidth");
+ if (!pid) {
+ char c;
+
+ close(zfd[1]);
+ mv_state[0] = 1;
+ while (read(zfd[0], &c, 1) == 1)
+ ;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("a task to move into P", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S')) {
+ reap(pid);
+ reap(in_p);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ /*
+ * Set the parameters from here, while the task sleeps. With a runtime
+ * of 1024ns, the task could be throttled before it sleeps again, for
+ * up to its period of 2s.
+ */
+ err = set_deadline(pid, 1024, 2 * NSEC_PER_SEC);
+ if (err) {
+ ksft_print_msg("a runtime of 1024ns every 2s: %s\n", strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (enter(cg_p, pid)) {
+ ksft_print_msg("a task with no bandwidth: its move into P refused\n");
+ ret = KSFT_FAIL;
+ } else if (write(zfd[1], "w", 1) != 1 || wait_for(1000, task_cpu_of(pid) == top_cpu(1))) {
+ ksft_print_msg("a task moved into P did not wake on P's CPU\n");
+ ret = KSFT_FAIL;
+ } else {
+ err = set_deadline(pid, 60 * NSEC_PER_MSEC, PERIOD);
+ if (err != -EBUSY) {
+ ksft_print_msg("0%% to 60%% beside a task of 60%% in P: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ }
+ }
+ /* Throttled, it would wait up to 2s to run and die. */
+ set_other(pid);
+ reap(pid);
+ reap(in_p);
+out:
+ close(zfd[0]);
+ close(zfd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("a refused change changed the free bandwidth\n");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+static int move_asleep(void)
+{
+ return moves(MV_ASLEEP, "move_asleep");
+}
+
+static int move_throttled(void)
+{
+ return moves(MV_THROTTLED, "move_throttled");
+}
+
+static int move_non_contending(void)
+{
+ return moves(MV_NON_CONTENDING, "move_non_contending");
+}
+
+/*
+ * Fork a process that reads sched_rt_runtime_us and writes the same value
+ * back @n times, 5ms apart. Then it exits. Each write rebuilds the sched
+ * domains. Return its pid, or -1.
+ */
+static pid_t rebuilder(int n)
+{
+ pid_t pid = fork_child("a sysctl writer");
+
+ if (!pid) {
+ char buf[64];
+
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)))
+ _exit(1);
+ while (n-- > 0) {
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", buf);
+ sleep_ns(5 * NSEC_PER_MSEC);
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
+static void *pauser(void *arg)
+{
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Run as the deadline thread of wide_mover(): a deadline task of 30% that
+ * runs 1ms, sets mv_state[0] to 1 and sleeps.
+ */
+static void *dl_pauser(void *arg)
+{
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ return arg;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ *mv_state = 1;
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Fork a process of @threads sleeping threads, then one deadline thread,
+ * dl_pauser(). A cgroup move of the process moves its threads one by one,
+ * between cpuset_can_attach() and cpuset_attach(). The deadline thread
+ * moves last, so that a rebuild in that window can still find it in the
+ * cgroup it leaves. Return the pid, or -1.
+ */
+static pid_t wide_mover(int threads)
+{
+ pid_t pid = fork_child("a wide deadline process");
+
+ if (!pid) {
+ pthread_attr_t a;
+ pthread_t t;
+ int i;
+
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ goto no_threads;
+ for (i = 0; i < threads; i++) {
+ if (pthread_create(&t, &a, pauser, NULL))
+ goto no_threads;
+ }
+ if (pthread_create(&t, NULL, dl_pauser, NULL))
+ goto no_threads;
+ for (;;)
+ pause();
+no_threads:
+ mv_state[0] = -2;
+ _exit(1);
+ }
+ return pid;
+}
+
+/*
+ * Move a process in and out of P while rebuilder() rebuilds the sched
+ * domains. The process is wide_mover(): 2048 sleeping threads, then one
+ * sleeping deadline thread of 30%. rebuilder() writes the
+ * sched_rt_runtime_us sysctl 20 times, 5ms apart. Each write is a
+ * rebuild. Without the 5ms gaps, the rebuilds would be over before the
+ * first move reaches the window below.
+ *
+ * A rebuild between cpuset_can_attach() and cpuset_attach() would drop
+ * the bandwidth that cpuset_can_attach() reserved in the destination.
+ * cpuset_attach() would then move the bandwidth of the thread all the
+ * same. So a rebuild must wait for the end of a move in progress.
+ *
+ * A later rebuild corrects the accounting. So move until rebuilder()
+ * exits: no rebuild follows the last move. The free bandwidth of P and
+ * of the other CPUs must then be what it was with the process asleep in
+ * the root cgroup. Run six rounds.
+ */
+static int move_vs_rebuild(void)
+{
+ int base_p, base_r, with_p, with_r, after_p, after_r, round, ret = KSFT_PASS;
+ const char *name = "move_vs_rebuild";
+ pid_t pid, writer;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ pid = wide_mover(2048);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference for the rounds: the task asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ for (round = 0; round < 6 && ret == KSFT_PASS; round++) {
+ int status, err = 0;
+
+ writer = rebuilder(20);
+ if (writer < 0) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ while (!err && waitpid(writer, &status, WNOHANG) == 0) {
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ }
+ if (err) {
+ ksft_print_msg("%s: a move refused: %s\n", name, strerror(-err));
+ reap(writer);
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (after_p != with_p || after_r != with_r) {
+ ksft_print_msg("%s: round %d: P %d -> %d probe tasks, the rest %d -> %d\n",
+ name, round, with_p, after_p, with_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+ ksft_print_msg("%s: task gone: P %d -> %d probe tasks, the rest %d -> %d\n",
+ name, base_p, after_p, base_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* The runtime every 100ms of the deadline threads of late_mover() */
+static uint64_t late_thread_runtime;
+
+/*
+ * Run as a deadline thread of late_mover(): a deadline task of
+ * late_thread_runtime that counts itself in mv_state[1] and sleeps.
+ */
+static void *dl_sleeper(void *arg)
+{
+ if (set_deadline(0, late_thread_runtime, PERIOD)) {
+ mv_state[0] = -1;
+ return arg;
+ }
+ mv_state[1]++;
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Fork a process of two deadline threads, dl_sleeper(), then @threads
+ * sleeping threads. Its main thread, the thread group leader, sleeps and
+ * is not a deadline task. A cgroup move of the process handles the
+ * leader first. The sleeping threads make the move long. Return the pid,
+ * or -1.
+ */
+static pid_t late_mover(int threads)
+{
+ pid_t pid = fork_child("a wide process");
+
+ if (!pid) {
+ pthread_attr_t a;
+ pthread_t t;
+ int i;
+
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ goto no_threads;
+ for (i = 0; i < 2; i++) {
+ if (pthread_create(&t, &a, dl_sleeper, NULL))
+ goto no_threads;
+ }
+ for (i = 0; i < threads; i++) {
+ if (pthread_create(&t, &a, pauser, NULL))
+ goto no_threads;
+ }
+ if (wait_for_step(5000, NSEC_PER_MSEC, mv_state[1] == 2 || mv_state[0]))
+ goto no_threads;
+ if (!mv_state[0])
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+no_threads:
+ mv_state[0] = -2;
+ _exit(1);
+ }
+ return pid;
+}
+
+/*
+ * What late_setter(), late_pinner() and late_killer() do, and the result.
+ * Each acts on thread @pid, @delay ns after @go is set. It stores its
+ * result in @ret and the time it returned in @done. late_setter() uses
+ * @runtime. late_killer() uses @tgid and @kill.
+ */
+static struct {
+ pid_t pid;
+ pid_t tgid;
+ int kill;
+ uint64_t runtime;
+ uint64_t delay;
+ atomic_int go;
+ int ret;
+ uint64_t done;
+} late;
+
+/*
+ * Make late.pid a deadline task of late.runtime every 100ms, late.delay
+ * ns after late.go. Store the result in late.ret and the time it returned
+ * in late.done.
+ */
+static void *late_setter(void *arg)
+{
+ uint64_t start;
+
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ late.ret = set_deadline(late.pid, late.runtime, PERIOD);
+ late.done = now_ns();
+ return arg;
+}
+
+/*
+ * Make the main thread of a process a deadline task while the process
+ * moves from the root cgroup into P.
+ *
+ * The process is late_mover(). Its main thread sleeps and is not a
+ * deadline task. Two of its threads are sleeping deadline tasks of
+ * @thread percent each. 2048 more sleeping threads make the move long.
+ * A thread of the test, late_setter(), calls sched_setattr() on the main
+ * thread to make it a deadline task of @leader percent.
+ *
+ * cpuset_can_attach() reserves room in P for the two deadline threads.
+ * If the main thread became a deadline task after that, no room would be
+ * reserved for it. cpuset_attach() handles the main thread first.
+ *
+ * sched_setattr() waits for the end of a move in progress. So it runs
+ * before the move or after it:
+ * - Before: the move counts three deadline tasks. With room in P for
+ * the three, the move must succeed. P must then count the three, and
+ * the root domain the process left must count none. Without room,
+ * the move must be refused with EBUSY.
+ * - After: sched_setattr() is refused with EPERM. The main thread is
+ * then allowed on P's CPU only. It is still asleep on a CPU of the
+ * root domain it left.
+ * Without room in P for the three, the case fails if the move and
+ * sched_setattr() both succeed.
+ *
+ * Try 30 delays of sched_setattr(), spread over the time a move takes.
+ */
+static int became_deadline(int thread, int leader, const char *name)
+{
+ int base_p, base_r, ret = KSFT_PASS, i, raced = 0;
+ int need = (2 * thread + leader) / 10, fits;
+ uint64_t start, len;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ fits = base_p >= need;
+ if (base_p < 2 * thread / 10) {
+ ksft_print_msg("%s: P admits %d probe tasks, %d needed\n", name, base_p,
+ 2 * thread / 10);
+ return KSFT_SKIP;
+ }
+ late_thread_runtime = thread * NSEC_PER_MSEC;
+ late.runtime = leader * NSEC_PER_MSEC;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = late_mover(2048);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* Time one move, to spread the delays over it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ sleep_ns(2 * PERIOD);
+
+ late.pid = pid;
+ for (i = 0; i < 30 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ int err;
+
+ late.go = 0;
+ late.delay = len * i / 30;
+ if (pthread_create(&t, NULL, late_setter, NULL)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err && (fits || err != -EBUSY || late.ret)) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!err && !late.ret && !fits) {
+ ksft_print_msg("%s: the move and the change both succeeded, with no room in P\n",
+ name);
+ ret = KSFT_FAIL;
+ } else if (!err && !late.ret) {
+ raced++;
+ if (admits_in(cg_p, base_p - need, "P, with the three tasks") ||
+ admits_in(root, base_r, "the root domain the tasks left"))
+ ret = KSFT_FAIL;
+ } else if (late.ret && late.ret != -EPERM) {
+ ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_FAIL)
+ ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ if ((!late.ret && set_other(pid)) || enter(root, pid)) {
+ ksft_print_msg("%s: the process cannot go back\n", name);
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ }
+ if (ret == KSFT_PASS && fits && !raced)
+ ksft_print_msg("%s: no change succeeded with a move\n", name);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("%s: the free bandwidth changed\n", name);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Run it with room in P: two threads of 20% and a main thread of 30%. */
+static int move_became_deadline(void)
+{
+ return became_deadline(20, 30, "move_became_deadline");
+}
+
+/* Run it without room in P: two threads of 30% and a main thread of 40%. */
+static int move_became_deadline_no_room(void)
+{
+ return became_deadline(30, 40, "move_became_deadline_no_room");
+}
+
+static void no_op(int sig)
+{
+}
+
+/* The CPU that dl_sleeper_tid() first runs on, or -1 for any CPU */
+static int sleeper_cpu = -1;
+
+/*
+ * Run as the deadline thread of pinned_mover(): a deadline task of 30%.
+ * Store the tid in mv_state[3], set mv_state[0] to 1 and sleep. A signal
+ * wakes the thread up. It then sleeps again. If mv_state[4] is set, this
+ * thread exits instead, and the process stays.
+ *
+ * With sleeper_cpu set, first run 1ms pinned on that CPU, then restore
+ * the previous affinity. Do this before sched_setattr(): the affinity of
+ * a deadline task must include every CPU of its root domain.
+ */
+static void *dl_sleeper_tid(void *arg)
+{
+ if (sleeper_cpu >= 0) {
+ cpu_set_t all, one;
+
+ CPU_ZERO(&one);
+ CPU_SET(sleeper_cpu, &one);
+ if (sched_getaffinity(0, sizeof(all), &all) ||
+ sched_setaffinity(0, sizeof(one), &one)) {
+ mv_state[0] = -2;
+ return arg;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ if (sched_setaffinity(0, sizeof(all), &all)) {
+ mv_state[0] = -2;
+ return arg;
+ }
+ }
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ return arg;
+ }
+ mv_state[3] = gettid_();
+ mv_state[0] = 1;
+ while (!mv_state[4])
+ pause();
+ syscall(SYS_exit, 0);
+ return arg;
+}
+
+/*
+ * Run as the last thread of pinned_mover() with @fifo. Become a
+ * SCHED_FIFO thread, set mv_state[5] to 1, or to -1 if SCHED_FIFO is
+ * refused, and sleep.
+ */
+static void *fifo_sleeper(void *arg)
+{
+ struct sched_param sp = { .sched_priority = 1 };
+
+ mv_state[5] = sched_setscheduler(0, SCHED_FIFO, &sp) ? -1 : 1;
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Fork a process of @threads sleeping threads, then one deadline thread,
+ * dl_sleeper_tid(). A cgroup move of the process handles the deadline
+ * thread after the sleeping ones, which make the move long. With @fifo,
+ * add a SCHED_FIFO thread, fifo_sleeper(), after it. Return the pid, or
+ * -1.
+ */
+static pid_t pinned_mover(int threads, int fifo)
+{
+ pid_t pid = fork_child("a wide deadline process");
+
+ if (!pid) {
+ pthread_attr_t a;
+ pthread_t t;
+ int i;
+
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ goto no_threads;
+ for (i = 0; i < threads; i++) {
+ if (pthread_create(&t, &a, pauser, NULL))
+ goto no_threads;
+ }
+ signal(SIGUSR1, no_op);
+ if (pthread_create(&t, NULL, dl_sleeper_tid, NULL))
+ goto no_threads;
+ if (fifo && (wait_for(5000, mv_state[0]) ||
+ pthread_create(&t, NULL, fifo_sleeper, NULL)))
+ goto no_threads;
+ for (;;)
+ pause();
+no_threads:
+ mv_state[0] = -2;
+ _exit(1);
+ }
+ return pid;
+}
+
+/*
+ * Restrict late.pid to the CPU of P and the CPU in @arg, late.delay ns
+ * after late.go.
+ */
+static void *late_pinner(void *arg)
+{
+ uint64_t start;
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(top_cpu(1), &set);
+ CPU_SET((long)arg, &set);
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ late.ret = sched_setaffinity(late.pid, sizeof(set), &set) ? -errno : 0;
+ late.done = now_ns();
+ return NULL;
+}
+
+/*
+ * Change the affinity of a sleeping deadline thread while its process
+ * moves from P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then a deadline
+ * thread of 30%. That thread went to sleep on a CPU of the root cgroup,
+ * and was moved into P asleep. So P counts it, and it is still on that
+ * CPU. A thread of the test, late_pinner(), calls sched_setaffinity()
+ * on it.
+ *
+ * cpuset_can_attach() reserves the bandwidth of the thread on one CPU of
+ * the root cgroup. The move records that CPU. It is the CPU the thread
+ * is on, if its affinity allows that CPU. If not, it is the first CPU of
+ * the root cgroup that its affinity allows.
+ *
+ * sched_setaffinity() restricts the thread to two CPUs: the CPU of P,
+ * and a CPU of the root cgroup that the thread is not on. The affinity
+ * of a deadline task must include every CPU of the root domain that
+ * counts it, so the mask needs the CPU of P. The move then records the
+ * second CPU.
+ *
+ * sched_setaffinity() waits for the end of a move in progress. So it
+ * runs before the move or after it:
+ * - Before: the bandwidth is reserved on the CPU the move records. The
+ * root domain of that CPU must count the thread once the move is
+ * done, and P must no longer count it.
+ * - After: the thread is counted in the root domain of the root
+ * cgroup. The mask does not include every CPU of that root domain.
+ * sched_setaffinity() is refused with EBUSY.
+ *
+ * Try 20 delays of sched_setaffinity(), spread over the time a move
+ * takes.
+ */
+static int move_affinity_changed(void)
+{
+ int with_p, with_r, ret = KSFT_PASS, i, raced = 0;
+ const char *name = "move_affinity_changed";
+ uint64_t start, len;
+ cpu_set_t all;
+ pid_t pid;
+
+ CPU_ZERO(&all);
+ for (i = 0; i < ncpus; i++)
+ CPU_SET(online[i], &all);
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference: the thread asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ /* Time one move, to spread the delays over it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+
+ late.pid = mv_state[3];
+ for (i = 0; i < 20 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ long cpu;
+ int err;
+
+ if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid)) {
+ ksft_print_msg("%s: the process cannot go into P\n", name);
+ ret = KSFT_FAIL;
+ break;
+ }
+ /* Not the CPU the thread is on: the move would record that one. */
+ cpu = task_cpu_of(late.pid) == online[1] ? online[0] : online[1];
+ late.go = 0;
+ late.delay = len * i / 20;
+ if (pthread_create(&t, NULL, late_pinner, (void *)cpu)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!late.ret) {
+ raced++;
+ sleep_ns(2 * PERIOD);
+ if (admits_in(cg_p, with_p, "P, which the thread left") ||
+ admits_in(root, with_r, "the root domain of the thread")) {
+ ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ } else if (late.ret && late.ret != -EBUSY) {
+ ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ if (ret == KSFT_PASS && !raced)
+ ksft_print_msg("%s: no change succeeded\n", name);
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Make a thread a deadline task while sched_setaffinity() runs on it, and
+ * move its process from P to the root cgroup meanwhile.
+ *
+ * sched_setaffinity() waits for the end of a move in progress when the
+ * task is a deadline task. It tests the policy first, with no lock. Here
+ * the thread T is SCHED_OTHER at that test:
+ *
+ * sched_setaffinity(T) sched_setattr(T) echo PID > cgroup.procs
+ *
+ * T is SCHED_OTHER: no wait
+ * T is a deadline task
+ * cpuset_can_attach()
+ * reserves room for T
+ * changes the affinity of T
+ * cpuset_attach()
+ * moves T
+ *
+ * The new affinity is the CPU of P and one CPU of the root cgroup, which
+ * is not the CPU the room was reserved on. Once the move is done, the
+ * root domain of the root cgroup must count T, and P must not.
+ *
+ * Delay injection: the window after the test of the policy is a few
+ * instructions wide. The case only reaches it on a kernel that can make
+ * sched_setaffinity() sleep there. It passes without reaching it
+ * otherwise.
+ */
+static int move_affinity_became_deadline(void)
+{
+ const char *name = "move_affinity_became_deadline";
+ int with_p, with_r, ret = KSFT_PASS, i;
+ uint64_t start, len, delay_ms;
+ cpu_set_t all;
+ pid_t pid;
+
+ CPU_ZERO(&all);
+ for (i = 0; i < ncpus; i++)
+ CPU_SET(online[i], &all);
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference: the thread a deadline task, asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ /* Time one move, to place the change inside it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ delay_ms = 3 * len / NSEC_PER_MSEC + 30;
+
+ late.pid = mv_state[3];
+ for (i = 0; i < 10 && ret == KSFT_PASS; i++) {
+ uint64_t t0, at;
+ pthread_t t;
+ int err;
+
+ if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid) ||
+ set_other(late.pid)) {
+ ksft_print_msg("%s: the thread cannot go into P as SCHED_OTHER\n", name);
+ ret = KSFT_FAIL;
+ break;
+ }
+ /*
+ * Wake the thread once, so that it runs on the CPU of P. A
+ * thread moved asleep is refused SCHED_DEADLINE with EPERM.
+ */
+ if (syscall(SYS_tgkill, pid, late.pid, SIGUSR1)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(5 * NSEC_PER_MSEC);
+ /*
+ * Delay injection: sched_setaffinity() would be asked here to
+ * sleep delay_ms after its test of the policy.
+ */
+ late.go = 0;
+ late.delay = 0;
+ if (pthread_create(&t, NULL, late_pinner, (void *)(long)online[1])) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ t0 = now_ns();
+ late.go = 1;
+ /* With the delay, the change has tested the policy and sleeps. */
+ sleep_ns(5 * NSEC_PER_MSEC);
+ err = set_deadline(late.pid, 30 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ /* Start the move so that the change wakes up inside it. */
+ at = t0 + delay_ms * NSEC_PER_MSEC - len * (i + 1) / 11;
+ while (now_ns() < at)
+ ;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!late.ret) {
+ sleep_ns(2 * PERIOD);
+ if (admits_in(cg_p, with_p, "P, which the thread left") ||
+ admits_in(root, with_r, "the root domain of the thread")) {
+ ksft_print_msg("%s: try %d, the change landed in a move of %llu us\n",
+ name, i, (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ } else if (late.ret && late.ret != -EBUSY) {
+ ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Wake thread late.pid of process late.tgid up with a signal, late.delay
+ * ns after late.go. With late.kill, the thread then exits.
+ */
+static void *late_killer(void *arg)
+{
+ uint64_t start;
+
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ mv_state[4] = late.kill;
+ late.ret = syscall(SYS_tgkill, late.tgid, late.pid, SIGUSR1) ? -errno : 0;
+ late.done = now_ns();
+ return arg;
+}
+
+/*
+ * Kill a sleeping deadline thread of 30% while its process moves into
+ * P. A thread that is part of a move cannot exit before the move ends:
+ * its exit waits for it. So a thread killed during the move dies right
+ * after it, and must release its bandwidth where the move left it. Once the process
+ * is gone, the free bandwidth of P and of the root domain the process
+ * left must be what it was.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_died(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, i;
+ const char *name = "move_died";
+ uint64_t len = 0;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ pid_t pid;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ if (!i) {
+ /* Time one move, to spread the delays over it. */
+ uint64_t start = now_ns();
+
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ }
+ late.pid = mv_state[3];
+ late.tgid = pid;
+ late.kill = 1;
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_killer, NULL)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after the thread died") ||
+ admits_in(root, base_r, "the root domain the process left"))) {
+ ksft_print_msg("%s: try %d, the death %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ }
+ return ret;
+}
+
+static int late_sysrq_fd;
+
+/*
+ * Write 'n' to /proc/sysrq-trigger, late.delay ns after late.go. sysrq-n
+ * makes every real-time and deadline task SCHED_OTHER.
+ */
+static void *late_sysrq(void *arg)
+{
+ uint64_t start;
+
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ late.ret = write(late_sysrq_fd, "n", 1) == 1 ? 0 : -errno;
+ late.done = now_ns();
+ return arg;
+}
+
+/*
+ * Make sysrq-n take a sleeping deadline thread of 30% out of the class
+ * while its process moves into P. cpuset_can_attach() reserves room in P
+ * for the thread. sysrq-n does not wait for the move. If it comes after
+ * the reservation, the thread moves no bandwidth, and cpuset_attach()
+ * must give the room back. Once the process is gone, the free bandwidth of P and of
+ * the root domain the process left must be what it was.
+ *
+ * sysrq-n changes the whole system. So the case runs last of the cases
+ * with P, and only with DEADLINE_TEST_SYSRQ set.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_normalized(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, i;
+ const char *name = "move_normalized";
+ uint64_t len = 0;
+
+ if (!getenv("DEADLINE_TEST_SYSRQ")) {
+ ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+ return KSFT_SKIP;
+ }
+ late_sysrq_fd = open("/proc/sysrq-trigger", O_WRONLY);
+ if (late_sysrq_fd < 0) {
+ ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ pid_t pid;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ close(late_sysrq_fd);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ if (!i) {
+ /* Time one move, to spread the delays over it. */
+ uint64_t start = now_ns();
+
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ close(late_sysrq_fd);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ }
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_sysrq, NULL)) {
+ reap(pid);
+ close(late_sysrq_fd);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (late.ret) {
+ ksft_print_msg("%s: sysrq-n: %s\n", name, strerror(-late.ret));
+ ret = KSFT_SKIP;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after sysrq-n") ||
+ admits_in(root, base_r, "the root domain the process left"))) {
+ ksft_print_msg("%s: try %d, sysrq-n %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ }
+ close(late_sysrq_fd);
+ return ret;
+}
+
+static int hold(int n);
+static void release_held(void);
+
+/*
+ * Wake a sleeping deadline thread of 30% up while its process moves from
+ * P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then the deadline
+ * thread. That thread first ran on the second CPU of the root cgroup,
+ * went to sleep there, and was moved into P asleep. So P counts it, and
+ * it is still on that CPU. A thread of the test, late_killer(), wakes it
+ * up with a signal.
+ *
+ * cpuset_can_attach() reserves room for the thread on the CPU that the
+ * move back records: the CPU the thread is on. If the thread wakes up
+ * after that, it runs on the CPU of P. cpuset_attach() must still move
+ * the bandwidth to the CPU recorded. Working the CPU out again would
+ * give the first CPU of the root cgroup, where nothing is reserved.
+ *
+ * Both CPUs are in the same root domain. So that root domain must count
+ * the thread once the move is done, and P must no longer count it.
+ *
+ * With @full, hold() first fills that root domain with deadline threads
+ * of the test, until 40% is free. The thread of 30% then fits once, and
+ * not twice.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int woken(int full, const char *name)
+{
+ int base_p, base_r, free_r, ret = KSFT_PASS, i;
+ uint64_t len = 0;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ if (base_r < 5) {
+ ksft_print_msg("%s: no room for the thread\n", name);
+ return KSFT_SKIP;
+ }
+ free_r = base_r;
+ if (full) {
+ if (hold(base_r - 4)) {
+ release_held();
+ return KSFT_FAIL;
+ }
+ free_r = 4;
+ }
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pid_t pid, tid;
+ pthread_t t;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ /* Not the first CPU: the move falls back to that one. */
+ sleeper_cpu = online[1];
+ pid = pinned_mover(2048, 0);
+ sleeper_cpu = -1;
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ release_held();
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ tid = mv_state[3];
+ sleep_ns(PERIOD);
+ if (!i) {
+ /*
+ * Time the move into P, to spread the delays over
+ * a move.
+ */
+ uint64_t start = now_ns();
+
+ err = enter(cg_p, pid);
+ if (!err) {
+ len = now_ns() - start;
+ err = enter(root, pid);
+ }
+ if (err) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ }
+ if (enter(cg_p, pid)) {
+ ksft_print_msg("%s: the process cannot go into P\n", name);
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ if (task_cpu_of(tid) != online[1]) {
+ ksft_print_msg("%s: the thread is on CPU %d, not on CPU %d\n", name,
+ task_cpu_of(tid), online[1]);
+ reap(pid);
+ release_held();
+ return KSFT_SKIP;
+ }
+ late.pid = tid;
+ late.tgid = pid;
+ late.kill = 0;
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_killer, NULL)) {
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, which the thread left") ||
+ admits_in(root, free_r - 3, "the root domain of the thread"))) {
+ ksft_print_msg("%s: try %d, the wakeup %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ }
+ release_held();
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, at the end") ||
+ admits_in(root, base_r, "the root domain of the rest, at the end")))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+static int move_woken(void)
+{
+ return woken(0, "move_woken");
+}
+
+static int move_woken_full(void)
+{
+ return woken(1, "move_woken_full");
+}
+
+/*
+ * Have the cpu controller refuse a move after cpuset_can_attach()
+ * reserved room. cpuset_cancel_attach() must then give the room back.
+ *
+ * The process is pinned_mover() with a SCHED_FIFO thread: 2048 sleeping
+ * threads, a sleeping deadline thread of 30%, and the SCHED_FIFO thread.
+ * It moves from the root cgroup into P. cpuset_can_attach() reserves
+ * room in P for the deadline thread. With CONFIG_RT_GROUP_SCHED, the cpu
+ * controller then refuses the SCHED_FIFO thread: P has no real-time
+ * runtime.
+ *
+ * Then a thread of the test, late_setter(), changes the runtime of the
+ * deadline thread during 20 such moves. sched_setattr() waits for the
+ * end of a move in progress. It must return, and succeed, once the move
+ * is cancelled.
+ *
+ * Skip if the cpu controller accepts the move.
+ */
+static int move_cancelled(void)
+{
+ int with_p, with_r, ret = KSFT_PASS, i, err;
+ const char *name = "move_cancelled";
+ uint64_t start, len;
+ pid_t pid;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ mv_state[5] = 0;
+ pid = pinned_mover(2048, 1);
+ if (pid < 0 || child_up(name, 0) || wait_for(5000, mv_state[5])) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ if (mv_state[5] < 0 || write_file(root, "cgroup.subtree_control", "+cpu")) {
+ ksft_print_msg("%s: no SCHED_FIFO thread or no cpu controller\n", name);
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ /* The reference: the deadline thread asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ start = now_ns();
+ err = enter(cg_p, pid);
+ len = now_ns() - start;
+ if (!err) {
+ ksft_print_msg("%s: the cpu controller accepts a SCHED_FIFO thread: needs CONFIG_RT_GROUP_SCHED\n",
+ name);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ for (i = 0; i < 5; i++) {
+ if (!enter(cg_p, pid)) {
+ ksft_print_msg("%s: a move accepted\n", name);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ }
+ sleep_ns(2 * PERIOD);
+ if (admits_in(cg_p, with_p, "P, after cancelled moves") ||
+ admits_in(root, with_r, "the root domain of the process")) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+
+ late.pid = mv_state[3];
+ for (i = 0; i < 20 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+
+ late.go = 0;
+ late.done = 0;
+ late.runtime = (i % 2 ? 30 : 20) * NSEC_PER_MSEC;
+ late.delay = len * i / 20;
+ if (pthread_create(&t, NULL, late_setter, NULL)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ if (!enter(cg_p, pid)) {
+ ksft_print_msg("%s: a move accepted\n", name);
+ ret = KSFT_FAIL;
+ }
+ if (wait_for(5000, late.done)) {
+ ksft_print_msg("%s: sched_setattr() still waits after a cancelled move\n",
+ name);
+ ret = KSFT_FAIL;
+ break; /* the thread cannot be joined */
+ }
+ join(t);
+ if (late.ret) {
+ ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, with_p, "P, after the cancelled moves") ||
+ admits_in(root, with_r, "the root domain of the process")))
+ ret = KSFT_FAIL;
+out:
+ reap(pid);
+ if (write_file(root, "cgroup.subtree_control", "-cpu"))
+ ksft_print_msg("%s: the cpu controller stays enabled\n", name);
+ return ret;
+}
+
+/* Return whether every task of @cg is frozen. */
+static int cg_frozen(const char *cg)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ snprintf(path, sizeof(path), "%s/cgroup.events", cg);
+ return !read_file(path, buf, sizeof(buf)) && strstr(buf, "frozen 1");
+}
+
+/*
+ * Freeze P with a running deadline task of 30% in it. Move the frozen
+ * task to the root cgroup, switch it to SCHED_OTHER, thaw P and kill the
+ * task. Do it three times. The free bandwidth of P and of the root
+ * domain of the other CPUs must then be what it was.
+ *
+ * A frozen task sleeps in its signal handling. The root cgroup is not
+ * frozen, so the move thaws the task. It is woken on P's CPU, before
+ * cpuset_attach() gives it the CPUs of the root cgroup.
+ */
+static int frozen_moved_switched(void)
+{
+ const char *what = "a deadline task to freeze";
+ int base_p, base_r, ret = KSFT_PASS, i;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 3 && ret == KSFT_PASS; i++) {
+ pid_t pid;
+
+ mv_state[0] = 0;
+ pid = fork_child(what);
+ if (!pid) {
+ if (enter(cg_p, getpid()) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ spin_ns(NSEC_PER_MSEC);
+ }
+ if (pid < 0 || child_up(what, 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (write_file(cg_p, "cgroup.freeze", "1")) {
+ ksft_print_msg("P cannot be frozen\n");
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ if (wait_for(2000, cg_frozen(cg_p))) {
+ ksft_print_msg("P does not freeze\n");
+ ret = KSFT_FAIL;
+ } else if (enter(root, pid)) {
+ ksft_print_msg("the frozen task cannot leave P\n");
+ ret = KSFT_FAIL;
+ } else if (set_other(pid)) {
+ ksft_print_msg("the task cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ }
+ if (write_file(cg_p, "cgroup.freeze", "0"))
+ ksft_print_msg("P cannot be thawed\n");
+ reap(pid);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after the frozen tasks") ||
+ admits_in(root, base_r, "the root domain of the rest")))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/* Deaths, in P */
+
+enum die_mode { DIE_RUNNING, DIE_SWITCHED_OUT, DIE_SHRUNK };
+
+/*
+ * Fork a deadline task of 30% that runs 5ms of a fresh period and exits
+ * before its 0-lag time. @mode says what it does before it exits:
+ * - DIE_RUNNING: nothing.
+ * - DIE_SWITCHED_OUT: switch to SCHED_OTHER.
+ * - DIE_SHRUNK: shrink to 10%. Its 0-lag time is then in the past.
+ *
+ * Its bandwidth is released at the 0-lag time, or at the exit for
+ * DIE_SHRUNK.
+ */
+static int die_case(enum die_mode mode, const char *what)
+{
+ pid_t pid = fork_child("a task that dies");
+ int status;
+
+ if (!pid) {
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ sleep_ns(PERIOD);
+ spin_ns(5 * NSEC_PER_MSEC);
+ if (mode == DIE_SWITCHED_OUT && set_other(0))
+ _exit(2);
+ if (mode == DIE_SHRUNK && set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+ _exit(3);
+ _exit(0);
+ }
+ if (pid < 0 || waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+ WEXITSTATUS(status)) {
+ ksft_print_msg("a task that dies %s failed\n", what);
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+static int death_running(void)
+{
+ return die_case(DIE_RUNNING, "running");
+}
+
+static int death_switched_out(void)
+{
+ return die_case(DIE_SWITCHED_OUT, "switched out");
+}
+
+static int death_shrunk(void)
+{
+ return die_case(DIE_SHRUNK, "shrunk");
+}
+
+/*
+ * Stop a running deadline task of 30% with SIGSTOP, switch it to
+ * SCHED_OTHER while it is stopped, and kill it. The task is on no
+ * runqueue when it leaves SCHED_DEADLINE. Do it five times.
+ */
+static int stopped_switched_out(void)
+{
+ const char *what = "a deadline task to stop";
+ int i, err;
+
+ for (i = 0; i < 5; i++) {
+ pid_t pid;
+
+ mv_state[0] = 0;
+ pid = fork_child(what);
+ if (!pid) {
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ spin_ns(NSEC_PER_MSEC);
+ }
+ if (pid < 0 || child_up(what, 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ err = 0;
+ if (kill(pid, SIGSTOP) || wait_for(1000, proc_state_of(pid) == 'T')) {
+ ksft_print_msg("the deadline task did not stop\n");
+ err = 1;
+ } else if (set_other(pid)) {
+ ksft_print_msg("the stopped task cannot leave SCHED_DEADLINE\n");
+ err = 1;
+ }
+ reap(pid);
+ if (err)
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+/*
+ * With @moving, wait until the spinner's count in mv_state[1] moves: it
+ * runs. Without, wait until the count stops for 5ms: it is throttled.
+ * Return 0, or -1 after 1s.
+ */
+static int wait_spinner(int moving)
+{
+ uint64_t end = now_ns() + NSEC_PER_SEC;
+ int last = mv_state[1];
+
+ while (now_ns() < end) {
+ sleep_ns(moving ? NSEC_PER_MSEC : 5 * NSEC_PER_MSEC);
+ if (moving ? mv_state[1] != last : mv_state[1] == last)
+ return 0;
+ last = mv_state[1];
+ }
+ return -1;
+}
+
+enum kill_when { KILL_AFTER, KILL_THROTTLED, KILL_RUNNING };
+
+/*
+ * Start a deadline task of 30% with mover() in @mode, then kill it as
+ * @when says:
+ * - KILL_AFTER: @after ns after it is up.
+ * - KILL_THROTTLED: once it is throttled.
+ * - KILL_RUNNING: while it runs.
+ *
+ * The killer first leaves P. It then watches and kills from another CPU,
+ * and does not take the task's CPU.
+ */
+static int kill_case(enum mv_mode mode, enum kill_when when, uint64_t after, const char *what)
+{
+ int ret = KSFT_PASS;
+ pid_t pid;
+
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = mover(mode, 30 * NSEC_PER_MSEC);
+ if (pid < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (enter(root, getpid())) {
+ ksft_print_msg("killed %s: the killer cannot leave P\n", what);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (child_up(what, 1)) {
+ ret = KSFT_FAIL;
+ } else if (when != KILL_AFTER && wait_spinner(when == KILL_RUNNING)) {
+ ksft_print_msg("killed %s: never %s\n", what,
+ when == KILL_RUNNING ? "ran" : "throttled");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(after);
+ }
+out:
+ reap(pid);
+ return ret;
+}
+
+static int death_killed_asleep(void)
+{
+ return kill_case(MV_ASLEEP, KILL_AFTER, 2 * PERIOD, "asleep");
+}
+
+static int death_killed_non_contending(void)
+{
+ return kill_case(MV_NON_CONTENDING, KILL_AFTER, 0, "before its 0-lag time");
+}
+
+static int death_killed_throttled(void)
+{
+ return kill_case(MV_THROTTLED, KILL_THROTTLED, 0, "throttled");
+}
+
+static int death_killed_running(void)
+{
+ return kill_case(MV_THROTTLED, KILL_RUNNING, 0, "running");
+}
+
+/*
+ * A robust priority inheritance lock shared between processes.
+ * mv_state[0] holds the state of its owner, mv_state[2] that of its
+ * waiter.
+ */
+static pthread_mutex_t *shm_lock;
+
+/* Set up shm_lock. Return 0, or -1 if it cannot be set up. */
+static int shm_lock_init(void)
+{
+ pthread_mutexattr_t a;
+ int err;
+
+ err = pthread_mutexattr_init(&a);
+ if (!err)
+ err = pthread_mutexattr_setpshared(&a, PTHREAD_PROCESS_SHARED);
+ if (!err)
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutexattr_setrobust(&a, PTHREAD_MUTEX_ROBUST);
+ if (!err)
+ err = pthread_mutex_init(shm_lock, &a);
+ if (err) {
+ ksft_print_msg("no shared priority inheritance mutex: %s\n", strerror(err));
+ return -1;
+ }
+ mv_state[0] = 0;
+ mv_state[2] = 0;
+ return 0;
+}
+
+/*
+ * Fork a process that takes shm_lock and sleeps with it, as a deadline
+ * task of 30% if @dl. Return its pid, or -1.
+ */
+static pid_t lock_owner(int dl)
+{
+ pid_t pid = fork_child("the owner of the lock");
+
+ if (!pid) {
+ if ((dl && set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) ||
+ pthread_mutex_lock(shm_lock)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * Fork a deadline task of 5ms every 50ms that waits for shm_lock. It sets
+ * mv_state[2] to 1 before the wait, and to 2 once it got the lock. Return
+ * its pid, or -1.
+ */
+static pid_t lock_waiter(void)
+{
+ pid_t pid = fork_child("a waiter of the lock");
+
+ if (!pid) {
+ int err;
+
+ if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+ mv_state[2] = -1;
+ _exit(1);
+ }
+ mv_state[2] = 1;
+ err = pthread_mutex_lock(shm_lock);
+ if (err == EOWNERDEAD)
+ err = pthread_mutex_consistent(shm_lock);
+ if (!err)
+ pthread_mutex_unlock(shm_lock);
+ mv_state[2] = err ? -2 : 2;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/* Wait up to 1s for @var to reach @want. Return 0, or -1. */
+static int wait_state(atomic_int *var, int want)
+{
+ uint64_t end = now_ns() + NSEC_PER_SEC;
+
+ while (now_ns() < end) {
+ if (*var == want)
+ return 0;
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ return -1;
+}
+
+/*
+ * Kill one side of a priority inheritance boost. lock_owner() sleeps
+ * with shm_lock, as a deadline task of 30% if @dl_owner. lock_waiter(),
+ * a deadline task of 5ms every 50ms, blocks on the lock and boosts the
+ * owner. Kill the owner if @kill_owner: the waiter must then get the
+ * lock. If not, kill the waiter.
+ *
+ * A killed owner does not die boosted. Its exit first releases its
+ * priority inheritance futexes in futex_exit_exec_release(), which
+ * deboosts it.
+ */
+static int pi_death(int dl_owner, int kill_owner, const char *what)
+{
+ int ret = KSFT_PASS;
+ pid_t owner, waiter = -1;
+
+ if (shm_lock_init())
+ return KSFT_SKIP;
+ owner = lock_owner(dl_owner);
+ if (owner < 0)
+ return KSFT_FAIL;
+ if (wait_state(&mv_state[0], 1)) {
+ ksft_print_msg("%s: no owner\n", what);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ waiter = lock_waiter();
+ if (waiter < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_state(&mv_state[2], 1)) {
+ ksft_print_msg("%s: no waiter\n", what);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ sleep_ns(50 * NSEC_PER_MSEC); /* blocked, boosting the owner */
+ if (kill_owner) {
+ reap(owner);
+ owner = -1;
+ if (wait_state(&mv_state[2], 2)) {
+ ksft_print_msg("%s: the waiter did not get the lock\n", what);
+ ret = KSFT_FAIL;
+ }
+ } else {
+ reap(waiter);
+ waiter = -1;
+ sleep_ns(50 * NSEC_PER_MSEC); /* the owner deboosted asleep */
+ }
+out:
+ reap(waiter);
+ reap(owner);
+ return ret;
+}
+
+static int death_pi_waiter(void)
+{
+ return pi_death(0, 0, "a waiter killed");
+}
+
+static int death_pi_owner(void)
+{
+ return pi_death(0, 1, "a boosted owner killed");
+}
+
+static int death_pi_deadline_owner(void)
+{
+ return pi_death(1, 1, "a boosted deadline owner killed");
+}
+
+/*
+ * Run as thread @arg of death_group(), a deadline task with a period of
+ * 100ms. Each thread counts itself in mv_state[0] once it has run 1ms.
+ * - Thread 0, of 10ms: run 1ms, then sleep.
+ * - Thread 1, of 10ms: spin and count in mv_state[1], until throttled.
+ * - Thread 2, of 60ms: wait for the killer to set mv_state[2]. Then
+ * spin and count in mv_state[3].
+ */
+static void *group_thread(void *arg)
+{
+ long kind = (long)arg;
+
+ if (set_deadline(0, (kind == 2 ? 60 : 10) * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1000;
+ return NULL;
+ }
+ if (kind == 2)
+ while (!mv_state[2])
+ sleep_ns(NSEC_PER_MSEC);
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0]++;
+ if (!kind)
+ for (;;)
+ pause();
+ for (;;)
+ mv_state[kind == 2 ? 3 : 1]++;
+ return NULL;
+}
+
+/*
+ * Kill a process of three deadline threads with one SIGKILL, while one
+ * sleeps, one is throttled and one runs. See group_thread().
+ */
+static int death_group(void)
+{
+ int last, ret = KSFT_PASS;
+ pid_t pid;
+
+ mv_state[0] = mv_state[1] = mv_state[2] = mv_state[3] = 0;
+ pid = fork_child("a group of deadline threads");
+ if (pid < 0)
+ return KSFT_FAIL;
+ if (!pid) {
+ pthread_t t;
+ long i;
+
+ for (i = 0; i < 3; i++)
+ if (pthread_create(&t, NULL, group_thread, (void *)i))
+ mv_state[0] = -1000;
+ for (;;)
+ pause();
+ }
+ /*
+ * Kill from outside P. P has one CPU: in P, the killer could not
+ * run while thread 2 runs.
+ */
+ if (enter(root, getpid())) {
+ ksft_print_msg("the killer cannot leave P\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_for(5000, mv_state[0] < 0 || mv_state[0] == 2) || mv_state[0] != 2) {
+ ksft_print_msg("only %d threads of the group started\n", (int)mv_state[0]);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_spinner(0)) {
+ ksft_print_msg("the first spinner never throttled\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ last = mv_state[3];
+ mv_state[2] = 1;
+ if (wait_for(5000, mv_state[0] < 0 || mv_state[3] != last) || mv_state[0] != 3) {
+ ksft_print_msg("the second spinner never ran\n");
+ ret = KSFT_FAIL;
+ }
+out:
+ /* One asleep, one throttled, one running. */
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Two deadline tasks are in P. One, of 30%, sleeps. The other, of 10%,
+ * exits and stays a zombie. Switching the zombie to SCHED_OTHER must be
+ * refused with ESRCH.
+ *
+ * On kernels where each cpuset counts its deadline tasks,
+ * cgroup_task_dead() took the zombie off P's count. The switch would
+ * run switched_from_dl() and take it off a second time. P's count would
+ * then be 0 with the task of 30% still in P. A rebuild of the accounting
+ * would skip P, and lose the bandwidth of that task. So rebuild the
+ * accounting, and check that P still counts the 30%.
+ */
+static int death_zombie_switched(void)
+{
+ int base, ret = KSFT_PASS, err;
+ pid_t in_p, zombie;
+
+ base = probe(cg_p);
+ if (base < 4) {
+ ksft_print_msg("no room in P for a task of 30%% and a probe\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ in_p = fork_child("a deadline task of P");
+ if (!in_p) {
+ if (enter(cg_p, getpid()) || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ if (in_p < 0 || child_up("a task of 30% in P", 0)) {
+ reap(in_p);
+ return KSFT_FAIL;
+ }
+ mv_state[0] = 0;
+ zombie = fork_child("a deadline task that exits");
+ if (!zombie) {
+ if (enter(cg_p, getpid()) || set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ mv_state[0] = 1;
+ _exit(0);
+ }
+ if (zombie < 0 || child_up("a task of 10% in P that exits", 0) ||
+ wait_for(1000, proc_state_of(zombie) == 'Z')) {
+ ksft_print_msg("no zombie deadline task\n");
+ reap(zombie);
+ reap(in_p);
+ return KSFT_FAIL;
+ }
+ /* Its bandwidth is released at its 0-lag time, within a period. */
+ sleep_ns(3 * PERIOD);
+ err = set_other(zombie);
+ if (err != -ESRCH) {
+ ksft_print_msg("SCHED_OTHER on a zombie deadline task: %s, not ESRCH\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ }
+ /*
+ * Rebuild the accounting: make Q beside P, then remove it. Q is a
+ * second partition root, of the CPU below P's.
+ */
+ if (make_partition(cg_q, top_cpu(2))) {
+ ksft_print_msg("no partition root Q beside P\n");
+ ret = KSFT_FAIL;
+ }
+ remove_partition(cg_q);
+ if (admits_in(cg_p, base - 3, "P rebuilt, a task of 30% and a zombie in it"))
+ ret = KSFT_FAIL;
+ reap(zombie);
+ reap(in_p);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && admits_in(cg_p, base, "P, its tasks gone"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/* sched_yield() */
+
+static atomic_int y_rounds; /* the yields of the yielder */
+
+/*
+ * Run as the thread of start_yielder(): a deadline task of 30% that
+ * yields after 0.5ms of every period. sched_yield() throttles it until
+ * its next period, with runtime left. So it is throttled most of the
+ * time. t_state is 2 while it is in the yield.
+ */
+static void *yielder(void *arg)
+{
+ t_tid = gettid_();
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ t_state = -1;
+ return (void *)1;
+ }
+ t_state = 1;
+ while (!spin_stop) {
+ spin_ns(NSEC_PER_MSEC / 2);
+ y_rounds++;
+ t_state = 2;
+ sched_yield();
+ t_state = 1;
+ }
+ return NULL;
+}
+
+static int start_yielder(pthread_t *t)
+{
+ t_state = 0;
+ spin_stop = 0;
+ y_rounds = 0;
+ if (pthread_create(t, NULL, yielder, NULL)) {
+ ksft_print_msg("the yielder thread could not start\n");
+ return -1;
+ }
+ if (!wait_for(5000, t_state) && t_state > 0)
+ return 0;
+ spin_stop = 1; /* in case it gets there late */
+ join(*t);
+ return -1;
+}
+
+/*
+ * Switch a deadline task to SCHED_OTHER while sched_yield() throttles it.
+ * Its bandwidth must come back, and it must run again.
+ */
+static int yield_switched_out(void)
+{
+ int base, ret = KSFT_PASS;
+ pthread_t t;
+
+ base = probe(NULL);
+ if (base < 3) {
+ ksft_print_msg("no room for a task of 30%%\n");
+ return KSFT_SKIP;
+ }
+ if (start_yielder(&t))
+ return KSFT_FAIL;
+ if (wait_throttled(t) || t_state != 2) {
+ ksft_print_msg("the task never yielded\n");
+ ret = KSFT_FAIL;
+ } else if (set_other(t_tid)) {
+ ret = KSFT_FAIL;
+ } else {
+ int last = y_rounds;
+
+ sleep_ns(2 * PERIOD);
+ if (y_rounds == last) {
+ ksft_print_msg("yielded, then switched out: never ran again\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ stop_spinner(t);
+ sleep_ns(3 * PERIOD); /* its 0-lag time */
+ if (ret == KSFT_PASS && admits(base, "yielded, then switched out"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+static int yield_moved(void)
+{
+ return moves(MV_YIELDED, "yield_moved");
+}
+
+/* Kill a deadline task while sched_yield() throttles it. */
+static int yield_killed(void)
+{
+ pid_t pid;
+
+ mv_state[0] = 0;
+ pid = fork_child("a task that yields");
+ if (!pid) {
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ spin_ns(NSEC_PER_MSEC / 2);
+ sched_yield();
+ }
+ }
+ if (pid < 0)
+ return KSFT_FAIL;
+ if (child_up("a task that yields", 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ reap(pid);
+ return KSFT_PASS;
+}
+
+/* SCHED_FLAG_DL_OVERRUN */
+
+static atomic_int got_xcpu;
+
+static void catch_xcpu(int sig)
+{
+ got_xcpu = 1;
+}
+
+/*
+ * Run as the thread of overrun_caught(): a deadline task of 10% with
+ * SCHED_FLAG_DL_OVERRUN. Run 15ms, past the runtime, then sleep until
+ * spin_stop is set.
+ */
+static void *overruns(void *arg)
+{
+ t_tid = gettid_();
+ if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD, SCHED_FLAG_DL_OVERRUN)) {
+ t_state = -1;
+ return (void *)1;
+ }
+ t_state = 1;
+ spin_ns(15 * NSEC_PER_MSEC);
+ while (!spin_stop)
+ sleep_ns(10 * NSEC_PER_MSEC);
+ return NULL;
+}
+
+/*
+ * When a task with SCHED_FLAG_DL_OVERRUN overruns its runtime, its thread
+ * group gets SIGXCPU. Catch it here. The task's bandwidth must stay
+ * reserved after the overrun, and come back when the task leaves
+ * SCHED_DEADLINE.
+ */
+static int overrun_caught(void)
+{
+ int base, ret = KSFT_PASS;
+ uint64_t end;
+ pthread_t t;
+
+ if (signal(SIGXCPU, catch_xcpu) == SIG_ERR)
+ return KSFT_FAIL;
+ base = probe(NULL);
+ if (base < 1) {
+ ksft_print_msg("no room for a task of 10%%\n");
+ return KSFT_SKIP;
+ }
+ got_xcpu = 0;
+ t_state = 0;
+ spin_stop = 0;
+ if (pthread_create(&t, NULL, overruns, NULL))
+ return KSFT_FAIL;
+ while (!t_state)
+ sleep_ns(NSEC_PER_MSEC);
+ end = now_ns() + 2 * NSEC_PER_SEC;
+ while (!got_xcpu && now_ns() < end)
+ sleep_ns(5 * NSEC_PER_MSEC);
+ if (t_state != 1) {
+ ret = KSFT_FAIL;
+ } else if (!got_xcpu) {
+ ksft_print_msg("it overran its runtime, no SIGXCPU in 2s\n");
+ ret = KSFT_FAIL;
+ } else if (admits(base - 1, "an overrun told")) {
+ ret = KSFT_FAIL; /* the overrun gave its 10% back */
+ }
+ stop_spinner(t);
+ sleep_ns(3 * PERIOD); /* its 0-lag time */
+ if (ret == KSFT_PASS && admits(base, "after an overrun was told"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/*
+ * Run as thread @arg of overrun_group_death(), a deadline task. Thread 0
+ * overruns its runtime. Threads 1 and 2 count themselves in mv_state[0],
+ * and sleep.
+ */
+static void *og_thread(void *arg)
+{
+ switch ((int)(long)arg) {
+ case 0: /* overruns its runtime: SIGXCPU to the group */
+ if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+ SCHED_FLAG_DL_OVERRUN))
+ _exit(1);
+ /*
+ * Wait for the other two first. On P's one CPU, this
+ * thread has the earliest deadline of the three. It would
+ * overrun, and kill them, before they reach the state the
+ * signal must find them in.
+ */
+ if (wait_for(5000, mv_state[0] >= 2))
+ _exit(2);
+ spin_ns(30 * NSEC_PER_MSEC);
+ break;
+ case 1: /* asleep, its 0-lag time past */
+ if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0]++;
+ break;
+ case 2: /* asleep, mostly before its 0-lag time */
+ if (set_deadline(0, 15 * NSEC_PER_MSEC, PERIOD))
+ _exit(1);
+ mv_state[0]++;
+ for (;;) {
+ spin_ns(3 * NSEC_PER_MSEC);
+ sleep_ns(22 * NSEC_PER_MSEC);
+ }
+ }
+ for (;;)
+ pause();
+}
+
+/*
+ * Let the SIGXCPU of an overrun kill the whole thread group. The
+ * bandwidth of its three deadline threads must come back.
+ */
+static int overrun_group_death(void)
+{
+ pid_t pid;
+ int i, status;
+
+ mv_state[0] = 0;
+ pid = fork_child("a group of deadline threads");
+ if (!pid) {
+ pthread_t t;
+
+ for (i = 0; i < 3; i++)
+ if (pthread_create(&t, NULL, og_thread, (void *)(long)i))
+ _exit(3);
+ for (;;)
+ pause();
+ }
+ if (pid < 0)
+ return KSFT_FAIL;
+ for (i = 0; i < 300; i++) {
+ if (waitpid(pid, &status, WNOHANG) == pid)
+ break;
+ sleep_ns(10 * NSEC_PER_MSEC);
+ }
+ if (i == 300) {
+ reap(pid);
+ ksft_print_msg("an overrun of 10ms every 100ms: no SIGXCPU in 3s\n");
+ return KSFT_FAIL;
+ }
+ if (WIFEXITED(status)) {
+ switch (WEXITSTATUS(status)) {
+ case 1:
+ ksft_print_msg("a thread of the group was refused SCHED_DEADLINE\n");
+ return KSFT_SKIP;
+ case 2:
+ ksft_print_msg("the other two threads of the group were not ready in 5s\n");
+ break;
+ default:
+ ksft_print_msg("a thread of the group could not start\n");
+ }
+ return KSFT_FAIL;
+ }
+ if (!WIFSIGNALED(status) || WTERMSIG(status) != SIGXCPU) {
+ ksft_print_msg("the group of an overrun died of signal %d, not SIGXCPU\n",
+ WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+/* SCHED_FLAG_RECLAIM */
+
+#define RECLAIM_MIN 40 /* least % for a reclaiming task of 10% */
+#define PLAIN_MAX 25 /* most % for one without the flag */
+#define SHARE_SLACK 15 /* % two measures of a share may differ */
+#define SHARE_NS (400 * NSEC_PER_MSEC)
+
+/*
+ * Measure the CPU share of a spinning deadline task of 10%, on P's CPU,
+ * without SCHED_FLAG_RECLAIM and then with it. With the flag, the task
+ * runs past its runtime: it takes the bandwidth of its root domain that
+ * no deadline task uses. It must get at most PLAIN_MAX percent without
+ * the flag, and at least RECLAIM_MIN percent with it.
+ */
+static int reclaim_runs(void)
+{
+ int plain, grub, ret = KSFT_PASS;
+ pthread_t t;
+
+ if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0)) {
+ ksft_print_msg("no room for a task of 10%%\n");
+ return KSFT_SKIP;
+ }
+ plain = share_of(t, SHARE_NS);
+ stop_spinner(t);
+ if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM))
+ return KSFT_FAIL;
+ grub = share_of(t, SHARE_NS);
+ stop_spinner(t);
+ if (plain > PLAIN_MAX || grub < RECLAIM_MIN) {
+ ksft_print_msg("of a CPU, a task of 10%% got %d%%, with SCHED_FLAG_RECLAIM %d%%\n",
+ plain, grub);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Switch a reclaiming task out of SCHED_DEADLINE before its 0-lag time,
+ * and back. Its bandwidth stays until the inactive timer releases it.
+ * Afterwards, the task must reclaim as much as before.
+ */
+static int reclaim_switched(void)
+{
+ int before, after = 0, r, ret = KSFT_PASS;
+ pthread_t t;
+
+ if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+ ksft_print_msg("no room for a task of 10%%\n");
+ return KSFT_SKIP;
+ }
+ before = share_of(t, SHARE_NS);
+ for (r = 0; r < 4; r++) {
+ if (set_other(t_tid)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ if (r & 1)
+ sleep_ns(5 * NSEC_PER_MSEC);
+ if (set_deadline_flags(t_tid, 10 * NSEC_PER_MSEC, PERIOD,
+ SCHED_FLAG_RECLAIM)) {
+ ksft_print_msg("round %d of 4: back to SCHED_DEADLINE refused\n",
+ r + 1);
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(PERIOD);
+ }
+ if (ret == KSFT_PASS)
+ after = share_of(t, SHARE_NS);
+ stop_spinner(t);
+ if (ret == KSFT_PASS &&
+ (after < before - SHARE_SLACK || after > before + SHARE_SLACK)) {
+ ksft_print_msg("reclaiming: %d%% of a CPU, %d%% after 4 round trips\n",
+ before, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Fork a reclaiming deadline task of 10%. It spins, and publishes in
+ * mv_state[1] the CPU time it has used, in milliseconds. It idles while
+ * mv_state[2] is set, and the moves happen then. A cpuset move waits for
+ * an RCU grace period. Where RCU runs in kthreads (PREEMPT_RT), a task
+ * that reclaims its whole CPU keeps that grace period from ending. Return
+ * the pid, or -1.
+ */
+static pid_t reclaim_mover(void)
+{
+ pid_t pid = fork_child("a reclaiming task");
+
+ if (!pid) {
+ if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+ SCHED_FLAG_RECLAIM)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ if (mv_state[2]) {
+ sleep_ns(10 * NSEC_PER_MSEC);
+ continue;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[1] = (int)(thread_time_ns() / NSEC_PER_MSEC);
+ }
+ }
+ return pid;
+}
+
+/* Idle the reclaiming task, then move it to @cg. Return 0, or -errno. */
+static int move_idle(const char *cg, pid_t pid)
+{
+ int err;
+
+ mv_state[2] = 1;
+ sleep_ns(20 * NSEC_PER_MSEC); /* it stops spinning */
+ err = enter(cg, pid);
+ mv_state[2] = 0;
+ return err;
+}
+
+/* Return the CPU share, in percent, of the reclaiming task over @ns. */
+static int mover_share(uint64_t ns)
+{
+ uint64_t wall = now_ns();
+ int cpu = mv_state[1];
+
+ sleep_ns(ns);
+ cpu = mv_state[1] - cpu;
+ wall = now_ns() - wall;
+ return (int)(100 * (uint64_t)cpu * NSEC_PER_MSEC / wall);
+}
+
+/*
+ * Move a reclaiming task into P, back out, and into P again. It must
+ * reclaim as much of P's CPU as it did the first time. Once it is gone,
+ * the free bandwidth of both root domains must be what it was.
+ */
+static int reclaim_moved(void)
+{
+ int base_p, base_r, in_p, again = 0, ret = KSFT_PASS;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ mv_state[2] = 1; /* idle until it is in P */
+ pid = reclaim_mover();
+ if (pid < 0)
+ return KSFT_FAIL;
+ if (child_up("a reclaiming task", 0) || move_idle(cg_p, pid)) {
+ mv_state[2] = 1;
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ in_p = mover_share(SHARE_NS);
+ if (move_idle(root, pid) || move_idle(cg_p, pid)) {
+ ksft_print_msg("a move of a reclaiming task refused\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(2 * PERIOD);
+ again = mover_share(SHARE_NS);
+ if (in_p < RECLAIM_MIN || again < in_p - SHARE_SLACK ||
+ again > in_p + SHARE_SLACK) {
+ ksft_print_msg("reclaiming: %d%% of P's CPU, %d%% after a move\n",
+ in_p, again);
+ ret = KSFT_FAIL;
+ }
+ }
+ mv_state[2] = 1;
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("a reclaiming task moved: the free bandwidth changed\n");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Deadline servers */
+
+#define SERVER_RUNTIME (100 * NSEC_PER_MSEC) /* the cases start at 10% */
+#define SERVER_PERIOD (1000 * NSEC_PER_MSEC)
+
+static const char server_dir[] = "/sys/kernel/debug/sched/fair_server";
+static uint64_t *server_found; /* each fair server's runtime and period */
+
+static int server_get(int cpu, const char *name, uint64_t *val)
+{
+ char path[96], buf[32];
+
+ snprintf(path, sizeof(path), "%s/cpu%d/%s", server_dir, cpu, name);
+ if (read_file(path, buf, sizeof(buf)))
+ return -1;
+ *val = strtoull(buf, NULL, 10);
+ return 0;
+}
+
+static int server_put(int cpu, const char *name, uint64_t val)
+{
+ char dir[96], buf[32];
+
+ snprintf(dir, sizeof(dir), "%s/cpu%d", server_dir, cpu);
+ snprintf(buf, sizeof(buf), "%llu", (unsigned long long)val);
+ return write_file(dir, name, buf);
+}
+
+/*
+ * Set the fair server of @cpu to @runtime every @period.
+ * Return 0, or -errno.
+ */
+static int server_set(int cpu, uint64_t runtime, uint64_t period)
+{
+ uint64_t cur;
+ int err;
+
+ if (!server_found || server_get(cpu, "period", &cur))
+ return -ENOENT;
+ /* Each write keeps the runtime within the period. */
+ if (runtime <= cur) {
+ err = server_put(cpu, "runtime", runtime);
+ return err ? err : server_put(cpu, "period", period);
+ }
+ err = server_put(cpu, "period", period);
+ return err ? err : server_put(cpu, "runtime", runtime);
+}
+
+/*
+ * Record the runtime and period of each fair server. Without debugfs,
+ * record none: the server cases are then skipped.
+ */
+static void server_init(void)
+{
+ int i;
+
+ server_found = calloc(2 * ncpus, sizeof(*server_found));
+ for (i = 0; server_found && i < ncpus; i++) {
+ if (server_get(online[i], "runtime", &server_found[2 * i]) ||
+ server_get(online[i], "period", &server_found[2 * i + 1])) {
+ free(server_found);
+ server_found = NULL;
+ ksft_print_msg("no fair server in debugfs: server cases skipped\n");
+ }
+ }
+}
+
+/* Put the fair servers back as the test found them. */
+static void server_restore(void)
+{
+ uint64_t runtime, period;
+ int i;
+
+ for (i = 0; server_found && i < ncpus; i++) {
+ int cpu = online[i], err;
+
+ if (server_get(cpu, "runtime", &runtime) || server_get(cpu, "period", &period))
+ continue;
+ if (runtime == server_found[2 * i] && period == server_found[2 * i + 1])
+ continue;
+ /*
+ * Retry for up to 1s. While departing tasks still hold
+ * their bandwidth, a root domain can refuse any change of a
+ * server, a decrease too. Their 0-lag times are a period or
+ * two away.
+ */
+ err = wait_for(1000, !server_set(cpu, server_found[2 * i],
+ server_found[2 * i + 1]));
+ if (err)
+ ksft_print_msg("the fair server of CPU %d stays changed\n", cpu);
+ }
+}
+
+/* Deadline threads of 10% every 100ms that sleep until released. */
+static pthread_t *held;
+static int nheld, held_err;
+static sem_t held_up, held_go;
+
+static void *holder(void *arg)
+{
+ int err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+
+ held_err = err;
+ if (sem_post(&held_up))
+ ksft_print_msg("sem_post: %s\n", strerror(errno));
+ if (!err)
+ sem_wait_sig(&held_go);
+ return NULL;
+}
+
+/*
+ * Start @n more holders. Return 0 once they are all admitted, or -1. The
+ * caller must call release_held(), on a failure too.
+ */
+static int hold(int n)
+{
+ int max = 16 * ncpus + 16, err;
+ pthread_attr_t a;
+
+ if (!held) {
+ held = calloc(max, sizeof(*held));
+ if (!held || sem_init(&held_up, 0, 0) || sem_init(&held_go, 0, 0)) {
+ ksft_print_msg("the holders cannot be set up\n");
+ return -1;
+ }
+ }
+ err = pthread_attr_init(&a);
+ if (!err)
+ err = pthread_attr_setstacksize(&a, THREAD_STACK);
+ if (err) {
+ ksft_print_msg("pthread_attr: %s\n", strerror(err));
+ return -1;
+ }
+ for (; n > 0; n--) {
+ if (nheld == max || pthread_create(&held[nheld], &a, holder, NULL)) {
+ ksft_print_msg("a holder thread could not start\n");
+ return -1;
+ }
+ if (sem_wait_for(&held_up, 10000)) {
+ ksft_print_msg("a holder did not answer\n");
+ nheld++; /* release_held() joins it */
+ return -1;
+ }
+ if (held_err) {
+ join(held[nheld]);
+ return -1;
+ }
+ nheld++;
+ }
+ return 0;
+}
+
+static void release_held(void)
+{
+ int i;
+
+ for (i = 0; i < nheld; i++)
+ if (sem_post(&held_go))
+ ksft_print_msg("sem_post: %s\n", strerror(errno));
+ for (i = 0; i < nheld; i++)
+ join(held[i]);
+ /* Drain the semaphore counts a late or refused holder left. */
+ while (held && (!sem_trywait(&held_up) || !sem_trywait(&held_go)))
+ ;
+ nheld = 0;
+ sleep_ns(3 * PERIOD); /* their 0-lag times */
+}
+
+/*
+ * Change the runtime and the period of the fair server of P's CPU. Its
+ * bandwidth comes out of what P's root domain has for deadline tasks.
+ * From 100ms every second (10%), P must admit two probe tasks fewer at
+ * 30%, and one fewer at 100ms every 500ms. With the server off, it must
+ * admit one more. A change that P has no room for must be refused with
+ * EBUSY, and leave the server as it was.
+ */
+static int server_params(void)
+{
+ static const struct {
+ uint64_t runtime, period;
+ int more;
+ const char *what;
+ } step[] = {
+ { 3 * SERVER_RUNTIME, SERVER_PERIOD, -2, "a fair server of 30%" },
+ { SERVER_RUNTIME, SERVER_PERIOD / 2, -1, "a fair server of 100ms every 500ms" },
+ { 0, SERVER_PERIOD, 1, "the fair server off" },
+ { SERVER_RUNTIME, SERVER_PERIOD, 0, "the fair server back to 10%" },
+ };
+ int cpu = top_cpu(1), base, err, i, ret = KSFT_PASS;
+ uint64_t runtime;
+
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+ return KSFT_SKIP;
+ }
+ base = probe(NULL);
+ if (base < 2) {
+ ksft_print_msg("no room in P for two probe tasks\n");
+ return KSFT_SKIP;
+ }
+ for (i = 0; i < (int)ARRAY_SIZE(step); i++) {
+ err = server_set(cpu, step[i].runtime, step[i].period);
+ if (err) {
+ ksft_print_msg("%s: %s\n", step[i].what, strerror(-err));
+ return KSFT_FAIL;
+ }
+ if (admits(base + step[i].more, step[i].what))
+ return KSFT_FAIL;
+ }
+ /*
+ * With base - 1 probe tasks held, P has room for 10% more, and not
+ * for 20%. So the server may grow to 20%, and not to 30%.
+ */
+ if (hold(base - 1)) {
+ release_held();
+ return KSFT_FAIL;
+ }
+ err = server_put(cpu, "runtime", 2 * SERVER_RUNTIME);
+ if (err) {
+ ksft_print_msg("a fair server of 20%% with room for it: %s\n", strerror(-err));
+ ret = KSFT_FAIL;
+ } else {
+ err = server_put(cpu, "runtime", 3 * SERVER_RUNTIME);
+ if (err != -EBUSY) {
+ ksft_print_msg("a fair server of 30%% with no room for it: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ ret = KSFT_FAIL;
+ } else if (server_get(cpu, "runtime", &runtime) ||
+ runtime != 2 * SERVER_RUNTIME) {
+ ksft_print_msg("a refused change changed the fair server\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+ ret = KSFT_FAIL;
+ release_held();
+ if (ret == KSFT_PASS && admits(base, "after a refused change of the fair server"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/*
+ * A rebuild of the root domains' accounting must count each fair server
+ * with its current runtime, not its runtime at boot. Check it for the
+ * server of P's CPU, set to 30%:
+ * - In P, as Q is made and unmade beside P.
+ * - In the root domain of the other CPUs, once P is unmade.
+ * - In P again, once P is made anew.
+ */
+static int server_rebuilds(void)
+{
+ int cpu = top_cpu(1), base, n10, n30;
+
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+ return KSFT_SKIP;
+ }
+ base = probe(cg_p);
+ if (base < 2) {
+ ksft_print_msg("no room in P for two probe tasks\n");
+ return KSFT_SKIP;
+ }
+ if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+ admits_in(cg_p, base - 2, "P, a fair server of 30%"))
+ return KSFT_FAIL;
+ if (make_partition(cg_q, top_cpu(2))) {
+ remove_partition(cg_q);
+ ksft_print_msg("no partition root Q beside P\n");
+ return KSFT_FAIL;
+ }
+ if (admits_in(cg_p, base - 2, "P, Q made beside it")) {
+ remove_partition(cg_q);
+ return KSFT_FAIL;
+ }
+ remove_partition(cg_q);
+ if (admits_in(cg_p, base - 2, "P, Q unmade beside it"))
+ return KSFT_FAIL;
+ /* Unmake P: its CPU and its server join the other CPUs. */
+ remove_p();
+ n30 = probe(root);
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+ return KSFT_FAIL;
+ n10 = probe(root);
+ if (n10 != n30 + 2) {
+ ksft_print_msg("P unmade: %d probe tasks, its CPU's server at 30%%, %d at 10%%\n",
+ n30, n10);
+ return KSFT_FAIL;
+ }
+ if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) || make_p()) {
+ ksft_print_msg("P made anew, a fair server of 30%%: failed\n");
+ return KSFT_FAIL;
+ }
+ if (admits_in(cg_p, base - 2, "P made anew, a fair server of 30%") ||
+ server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+ admits_in(cg_p, base, "P, the fair server back to 10%"))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/* Wait for the futex @f, in shared memory, to be set. */
+static void wait_set(atomic_int *f)
+{
+ while (!*f)
+ syscall(SYS_futex, f, FUTEX_WAIT, 0, NULL, NULL, 0);
+}
+
+static void set_flag(atomic_int *f, int val)
+{
+ *f = val;
+ syscall(SYS_futex, f, FUTEX_WAKE, INT_MAX, NULL, NULL, 0);
+}
+
+/*
+ * Fork a process of P that spins while mv_state[4 + @rt] is set, and
+ * sleeps otherwise. It is a fair task, or a real-time one if @rt. The
+ * real-time one sleeps 1ms every 50ms, to let the real-time kthreads of
+ * P's CPU run: PREEMPT_RT runs RCU and the timers there. Return the pid,
+ * or -1.
+ */
+static pid_t server_client(int rt)
+{
+ pid_t pid = fork_child(rt ? "a real-time task of P" : "a fair task of P");
+
+ if (!pid) {
+ struct sched_param sp = { .sched_priority = 1 };
+
+ if (enter(cg_p, getpid()) ||
+ (rt && sched_setscheduler(0, SCHED_FIFO, &sp)))
+ _exit(1);
+ mv_state[0]++;
+ for (;;) {
+ wait_set(&mv_state[4 + rt]);
+ spin_ns(rt ? 50 * NSEC_PER_MSEC : NSEC_PER_MSEC);
+ if (rt)
+ sleep_ns(NSEC_PER_MSEC);
+ }
+ }
+ return pid;
+}
+
+/* Return the CPU share of process @pid over @ns, in percent, or -1. */
+static int proc_share(pid_t pid, uint64_t ns)
+{
+ struct timespec t0, t1;
+ uint64_t wall = now_ns();
+ clockid_t c;
+
+ if (clock_getcpuclockid(pid, &c) || clock_gettime(c, &t0))
+ return -1;
+ sleep_ns(ns);
+ if (clock_gettime(c, &t1))
+ return -1;
+ wall = now_ns() - wall;
+ return (int)(100 * ((t1.tv_sec - t0.tv_sec) * NSEC_PER_SEC +
+ t1.tv_nsec - t0.tv_nsec) / wall);
+}
+
+/* % that the share of a starved task may differ from its server's bandwidth */
+#define STARVED_SLACK 8
+
+/*
+ * Change the fair server of P's CPU, from outside P, in each state a
+ * change can find it in:
+ * - stopped
+ * - deferred while a fair task of P runs
+ * - deferred while a real-time task starves that fair task
+ * - serving it
+ * - deferred again after serving it
+ * - deferred while the CPU idles
+ *
+ * A fair task and a SCHED_FIFO task of P, see server_client(), put the
+ * server in these states.
+ *
+ * Each change starts a period of the server. The server defers serving a
+ * starved task to its zero-laxity time. That is 0.7s into the period at
+ * 30%, and 0.9s at 10%. Afterwards, the starved task must get the
+ * server's bandwidth, at 30% then at 10%. P's free bandwidth must then be
+ * what it was.
+ *
+ * A change cannot find the server in two more states. Throttled after
+ * serving, the server is deferred anew at once: the last of the served
+ * task's runtime is charged to it as background running. And on a CPU
+ * with no tick in idle, its idle wait begins and ends in one timer
+ * callback.
+ */
+static int server_states(void)
+{
+ static const struct {
+ int fair, rt, ms;
+ } step[] = {
+ { 0, 0, 1500 }, /* stopped: nothing of P ran for a period */
+ { 1, 0, 300 }, /* deferred while the fair task runs */
+ { 1, 1, 300 }, /* deferred, fair task starved: before 0.9s */
+ { 1, 1, 800 }, /* serving it: after 0.7s */
+ { 1, 1, 1500 }, /* deferred again: served from 0.9s to 1s */
+ { 0, 0, 600 }, /* deferred while the CPU idles */
+ };
+ int cpu = top_cpu(1), base, i, s30 = -1, s10 = -1, ret = KSFT_PASS;
+ pid_t fair, rt;
+
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+ return KSFT_SKIP;
+ }
+ base = probe(cg_p);
+ if (base <= 0) {
+ ksft_print_msg("no room in P to probe\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ set_flag(&mv_state[4], 0);
+ set_flag(&mv_state[5], 0);
+ fair = server_client(0);
+ rt = server_client(1);
+ if (fair < 0 || rt < 0 || wait_for(5000, mv_state[0] >= 2)) {
+ if (fair >= 0 && rt >= 0)
+ ksft_print_msg("a fair or real-time task of P could not start\n");
+ ret = KSFT_FAIL;
+ }
+ /*
+ * Even steps set the server's runtime to 30%, odd ones back to 10%.
+ * The period stays.
+ */
+ for (i = 0; i < (int)ARRAY_SIZE(step) && ret == KSFT_PASS; i++) {
+ set_flag(&mv_state[4], step[i].fair);
+ set_flag(&mv_state[5], step[i].rt);
+ sleep_ns(step[i].ms * NSEC_PER_MSEC);
+ if (server_put(cpu, "runtime", (i & 1 ? 1 : 3) * SERVER_RUNTIME)) {
+ ksft_print_msg("change %d of the fair server refused\n", i + 1);
+ ret = KSFT_FAIL;
+ }
+ }
+ if (ret == KSFT_PASS) {
+ set_flag(&mv_state[4], 1);
+ set_flag(&mv_state[5], 1);
+ if (server_put(cpu, "runtime", 3 * SERVER_RUNTIME))
+ ret = KSFT_FAIL;
+ s30 = proc_share(fair, 2 * SERVER_PERIOD);
+ if (server_put(cpu, "runtime", SERVER_RUNTIME))
+ ret = KSFT_FAIL;
+ s10 = proc_share(fair, 2 * SERVER_PERIOD);
+ }
+ set_flag(&mv_state[5], 0);
+ set_flag(&mv_state[4], 0);
+ reap(rt);
+ reap(fair);
+ if (ret == KSFT_PASS &&
+ (s30 < 30 - STARVED_SLACK || s30 > 30 + STARVED_SLACK ||
+ s10 < 10 - STARVED_SLACK || s10 > 10 + STARVED_SLACK)) {
+ ksft_print_msg("a starved fair task got %d%% of a server of 30%%, %d%% of 10%%\n",
+ s30, s10);
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && admits_in(cg_p, base, "after the fair server's changes"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/*
+ * A reclaiming deadline task of P gets the share of P's CPU that its root
+ * domain leaves free. P's fair server takes from that share. The task
+ * must get:
+ * - less once the server grows from 10% to 30%.
+ * - as much after a rebuild of the accounting (Q made and unmade).
+ * - what it had at first once the server is back at 10%.
+ *
+ * Then put the server back as the test found it, so that P's free
+ * bandwidth is what it was.
+ */
+static int server_reclaim(void)
+{
+ int cpu = top_cpu(1), s10, s30, rebuilt, back, ret = KSFT_PASS;
+ pthread_t t;
+
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+ return KSFT_SKIP;
+ }
+ if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+ ksft_print_msg("no room for a task of 10%%\n");
+ return KSFT_SKIP;
+ }
+ s10 = share_of(t, SHARE_NS);
+ if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD))
+ ret = KSFT_FAIL;
+ s30 = share_of(t, SHARE_NS);
+ if (make_partition(cg_q, top_cpu(2)))
+ ret = KSFT_FAIL;
+ remove_partition(cg_q);
+ rebuilt = share_of(t, SHARE_NS);
+ if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+ ret = KSFT_FAIL;
+ back = share_of(t, SHARE_NS);
+ stop_spinner(t);
+ server_restore();
+ if (ret == KSFT_PASS &&
+ (s30 > s10 - SHARE_SLACK ||
+ rebuilt < s30 - SHARE_SLACK || rebuilt > s30 + SHARE_SLACK ||
+ back < s10 - SHARE_SLACK || back > s10 + SHARE_SLACK)) {
+ ksft_print_msg("reclaiming: %d%%, %d%% at 30%%, %d%% rebuilt, %d%% back at 10%%\n",
+ s10, s30, rebuilt, back);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Parameter changes while a task wakes on another CPU */
+
+#define WAKE_NS (5 * NSEC_PER_SEC) /* how long the changes go on */
+#define WAKE_BURST 2000 /* pairs of changes per clock reading */
+
+/* Read this_bw and running_bw of @cpu from sched_debug. Return 0, or -1. */
+static int rq_bw_of(int cpu, long *this_bw, long *running_bw)
+{
+ char line[256], want[32];
+ FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+ int in = 0, got = 0;
+
+ if (!f)
+ return -1;
+ snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+ while (got < 2 && fgets(line, sizeof(line), f)) {
+ char *val = strchr(line, ':');
+
+ if (!strncmp(line, want, strlen(want)))
+ in = 1;
+ else if (in && !strncmp(line, "dl_rq[", 6))
+ break; /* the next CPU's */
+ else if (!in || !val)
+ continue;
+ else if (strstr(line, ".running_bw") && sscanf(val + 1, "%ld", running_bw) == 1)
+ got++;
+ else if (strstr(line, ".this_bw") && sscanf(val + 1, "%ld", this_bw) == 1)
+ got++;
+ }
+ fclose(f);
+ return got == 2 ? 0 : -1;
+}
+
+static atomic_int *wk_state; /* 0: tasks up, 1: wakeups, 2: stop */
+
+/*
+ * Fork a deadline task of @runtime every @period in Q. It counts itself
+ * in wk_state[0], then spins. With @sleeps, it instead runs 20us and
+ * sleeps 80us in a loop, and counts its wakeups in wk_state[1]. It exits
+ * once wk_state[2] is set. Return the pid, or -1.
+ */
+static pid_t waking_task(uint64_t runtime, uint64_t period, int sleeps)
+{
+ pid_t pid = fork_child("a waking deadline task");
+
+ if (!pid) {
+ if (enter(cg_q, getpid()) || set_deadline(0, runtime, period)) {
+ wk_state[0] = -1;
+ _exit(1);
+ }
+ wk_state[0]++;
+ for (;;) {
+ if (wk_state[2])
+ _exit(0);
+ spin_ns(sleeps ? 20 * NSEC_PER_USEC : NSEC_PER_MSEC);
+ if (sleeps) {
+ sleep_ns(80 * NSEC_PER_USEC);
+ wk_state[1]++;
+ }
+ }
+ }
+ return pid;
+}
+
+/*
+ * Fork a process that changes the parameters of @pid over and over, until
+ * wk_state[2] is set. Return its pid, or -1.
+ */
+static pid_t changer_proc(pid_t pid)
+{
+ pid_t self = fork_child("a changer of the parameters");
+
+ if (!self) {
+ while (!wk_state[2]) {
+ if (set_deadline(pid, 20 * NSEC_PER_MSEC, PERIOD) ||
+ set_deadline(pid, 10 * NSEC_PER_MSEC, PERIOD))
+ break;
+ }
+ _exit(0);
+ }
+ return self;
+}
+
+/*
+ * Change the parameters of a deadline task while its wakeups move it
+ * between two CPUs.
+ *
+ * Q is a partition root of two CPUs. Three deadline tasks run in Q, see
+ * waking_task(). Two of 20ms every 50ms spin, and hold both CPUs with
+ * earlier deadlines. The third is a task of 10ms every 100ms. It runs
+ * 20us and sleeps 80us in a loop. It wakes on whichever of the two CPUs
+ * has the later deadline. So a wakeup can move it to the other CPU.
+ *
+ * Two processes outside Q change the runtime of the third task for 5s:
+ * the test and changer_proc(). Each switches it between 20ms and 10ms in
+ * a loop.
+ *
+ * A wakeup that moves a task takes its bandwidth off the runqueue it
+ * slept on. The enqueue on the new runqueue adds it there, and may be
+ * left to the CPU the task wakes on. Between the two, no runqueue counts
+ * the bandwidth. A change in that window must not change this_bw of
+ * either runqueue. Once the tasks of the case are gone, this_bw of both
+ * CPUs must be what it was before.
+ *
+ * Check this_bw only then. sched_debug prints running_bw and this_bw of
+ * a CPU one after the other, with no lock held in between. Every
+ * parameter change takes the bandwidth of a task off its runqueue and
+ * puts it back. So a reading taken during the changes can straddle one
+ * of them.
+ */
+static int params_waking(void)
+{
+ int cpu_a = top_cpu(3), cpu_b = top_cpu(2), i, ret = KSFT_PASS, changes = 0;
+ long ta, ra, tb, rb, ta2, ra2, tb2, rb2;
+ pid_t pid[3] = { -1, -1, -1 }, changer = -1;
+ uint64_t end;
+ char cpus[32];
+
+ if (ncpus < 4) {
+ ksft_print_msg("needs 4 non-isolated CPUs\n");
+ return KSFT_SKIP;
+ }
+ snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+ if (make_partition_cpus(cg_q, cpus)) {
+ remove_partition(cg_q);
+ ksft_print_msg("no partition root of two CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (rq_bw_of(cpu_a, &ta, &ra) || rq_bw_of(cpu_b, &tb, &rb)) {
+ remove_partition(cg_q);
+ ksft_print_msg("sched_debug tells of neither CPU's this_bw\n");
+ return KSFT_SKIP;
+ }
+ wk_state = mv_state + 8;
+ wk_state[0] = wk_state[1] = wk_state[2] = 0;
+ /* Two of 40% hold both CPUs with an earlier deadline than the third. */
+ pid[0] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+ pid[1] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+ pid[2] = waking_task(10 * NSEC_PER_MSEC, PERIOD, 1);
+ if (pid[0] < 0 || pid[1] < 0 || pid[2] < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ wait_for(5000, wk_state[0] >= 3 || wk_state[0] < 0);
+ if (wk_state[0] != 3) {
+ ksft_print_msg("no room for the deadline tasks of a partition of two CPUs\n");
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ changer = changer_proc(pid[2]);
+ if (changer < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ end = now_ns() + WAKE_NS;
+ while (now_ns() < end) {
+ for (i = 0; i < WAKE_BURST; i++, changes += 2) {
+ if (set_deadline(pid[2], 20 * NSEC_PER_MSEC, PERIOD) ||
+ set_deadline(pid[2], 10 * NSEC_PER_MSEC, PERIOD)) {
+ ksft_print_msg("a change of the waking task refused\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ }
+ }
+ if (!wk_state[1]) {
+ ksft_print_msg("the waking task never woke\n");
+ ret = KSFT_FAIL;
+ }
+out:
+ wk_state[2] = 1;
+ reap(changer);
+ for (i = 0; i < 3; i++)
+ reap(pid[i]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS) {
+ if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2)) {
+ ret = KSFT_FAIL;
+ } else if (ta2 != ta || tb2 != tb) {
+ ksft_print_msg("%d changes: this_bw %ld and %ld, was %ld and %ld\n",
+ changes, ta2, tb2, ta, tb);
+ ret = KSFT_FAIL;
+ }
+ }
+ remove_partition(cg_q);
+ return ret;
+}
+
+/* A random mix of the transitions above */
+
+#define MIX_WORKERS 6
+#define MIX_NS (3 * NSEC_PER_SEC)
+#define MIX_SEED 20261103 /* fixed: a rerun starts with the same draws */
+
+/*
+ * random_mix: six workers change their own state at random for 3s, and
+ * the case changes them from outside too. Q is a partition root of two
+ * CPUs, next to P. The case needs 4 housekeeping CPUs.
+ *
+ * Each worker does one of these, over and over:
+ * - become a deadline task, with random parameters and flags
+ * - switch to SCHED_OTHER
+ * - sleep
+ * - spin, at times past its runtime
+ * - call sched_yield()
+ * - wait up to 5ms for shm_lock and hold it 1ms. It is the mutex of the
+ * priority inheritance cases. A deadline waiter boosts its owner.
+ * - exit
+ *
+ * The case does one of these, over and over:
+ * - set the deadline parameters of a random worker
+ * - switch it to SCHED_OTHER
+ * - change its affinity
+ * - move it into P, into Q or into the root cgroup
+ * - kill it
+ * - change the fair server of P's CPU or of one CPU of Q
+ * It starts anew every worker it finds dead.
+ *
+ * The case does not check what a single transition does: the other cases
+ * do that. It checks the accounting afterwards. Once every worker is gone
+ * and its 0-lag time is past:
+ * - P, Q and the housekeeping root domain must each admit what they
+ * admitted before;
+ * - this_bw of P's CPU and of the two CPUs of Q must be what it was.
+ * An over- or underflow of running_bw or this_bw also shows as a warning
+ * in the kernel log, which the test does not read.
+ */
+static atomic_int *mix_state; /* 0: stop, 1 + i: worker i's rounds */
+
+static pid_t mix_worker(int idx, unsigned int seed)
+{
+ static const uint64_t runtime[] = { 5, 10, 20, 30 };
+ static const uint64_t period[] = { 50, 100 };
+ static const uint64_t flag[] = { 0, SCHED_FLAG_RECLAIM, SCHED_FLAG_DL_OVERRUN };
+ pid_t pid = fork_child("a worker of the mix");
+
+ if (!pid) {
+ struct sigaction sa = { .sa_handler = catch_xcpu };
+ struct timespec ts;
+ int err;
+
+ if (sigaction(SIGXCPU, &sa, NULL))
+ _exit(1);
+ seed += idx;
+ /*
+ * Ignore the errors. Each call below is made in whatever
+ * state the worker is in, and the kernel may refuse it.
+ */
+ while (!mix_state[0]) {
+ unsigned int r = rand_r(&seed), v = r / 8;
+
+ switch (r % 8) {
+ case 0:
+ set_deadline_flags(0, runtime[v % 4] * NSEC_PER_MSEC,
+ period[(v / 4) % 2] * NSEC_PER_MSEC,
+ flag[(v / 8) % 3]);
+ break;
+ case 1:
+ set_other(0);
+ break;
+ case 2:
+ sleep_ns((v % 20 + 1) * NSEC_PER_MSEC);
+ break;
+ case 3:
+ spin_ns((v % 15 + 1) * NSEC_PER_MSEC);
+ break;
+ case 4:
+ sched_yield();
+ break;
+ case 5:
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += 5 * NSEC_PER_MSEC;
+ if (ts.tv_nsec >= NSEC_PER_SEC) {
+ ts.tv_nsec -= NSEC_PER_SEC;
+ ts.tv_sec++;
+ }
+ err = pthread_mutex_timedlock(shm_lock, &ts);
+ if (err == EOWNERDEAD) /* its owner died */
+ err = pthread_mutex_consistent(shm_lock);
+ if (!err) {
+ spin_ns(NSEC_PER_MSEC);
+ if (pthread_mutex_unlock(shm_lock))
+ _exit(1);
+ }
+ break;
+ case 6:
+ _exit(0); /* the case starts another */
+ default:
+ sleep_ns(NSEC_PER_MSEC / 2);
+ break;
+ }
+ mix_state[1 + idx]++;
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
+static cpu_set_t mix_mask[4];
+
+static int random_mix(void)
+{
+ static const uint64_t runtime[] = { 5, 10, 20, 30 };
+ int cpu_a = top_cpu(3), cpu_b = top_cpu(2), cpu_p = top_cpu(1);
+ int base_p, base_q, base_r, i, ret = KSFT_PASS, rounds = 0, sums;
+ long ta = 0, ra, tb = 0, rb, tp = 0, rp, ta2, ra2, tb2, rb2, tp2, rp2;
+ pid_t pid[MIX_WORKERS] = { -1, -1, -1, -1, -1, -1 };
+ unsigned int seed = MIX_SEED;
+ uint64_t end;
+ char cpus[32];
+
+ if (ncpus < 4) {
+ ksft_print_msg("needs 4 non-isolated CPUs\n");
+ return KSFT_SKIP;
+ }
+ snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+ if (make_partition_cpus(cg_q, cpus)) {
+ remove_partition(cg_q);
+ ksft_print_msg("no partition root of two CPUs\n");
+ return KSFT_SKIP;
+ }
+ /*
+ * The affinities of the mix: every CPU of the test, those of Q and
+ * P, those of P and the rest, and all but the first.
+ */
+ for (i = 0; i < 4; i++)
+ CPU_ZERO(&mix_mask[i]);
+ for (i = 0; i < ncpus; i++) {
+ int c = online[i];
+
+ CPU_SET(c, &mix_mask[0]);
+ if (c == cpu_a || c == cpu_b || c == cpu_p)
+ CPU_SET(c, &mix_mask[1]);
+ if (c != cpu_a && c != cpu_b)
+ CPU_SET(c, &mix_mask[2]);
+ if (i)
+ CPU_SET(c, &mix_mask[3]);
+ }
+ if (shm_lock_init()) {
+ remove_partition(cg_q);
+ return KSFT_SKIP; /* no shared mutex for the workers */
+ }
+ /* Without this_bw in sched_debug, check the admissions only. */
+ sums = !rq_bw_of(cpu_a, &ta, &ra) && !rq_bw_of(cpu_b, &tb, &rb) &&
+ !rq_bw_of(cpu_p, &tp, &rp);
+ base_p = probe(cg_p);
+ base_q = probe(cg_q);
+ base_r = probe(root);
+ if (base_p <= 0 || base_q <= 0 || base_r <= 0) {
+ remove_partition(cg_q);
+ ksft_print_msg("no room to probe P, Q or the rest\n");
+ return KSFT_SKIP;
+ }
+ mix_state = mv_state + 16;
+ for (i = 0; i <= MIX_WORKERS; i++)
+ mix_state[i] = 0;
+ for (i = 0; i < MIX_WORKERS; i++) {
+ pid[i] = mix_worker(i, seed);
+ if (pid[i] < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ }
+
+ end = now_ns() + MIX_NS;
+ while (now_ns() < end) {
+ unsigned int r = rand_r(&seed);
+ int w = r % MIX_WORKERS;
+
+ switch ((r / MIX_WORKERS) % 8) {
+ case 0:
+ set_deadline(pid[w], runtime[r % 4] * NSEC_PER_MSEC, PERIOD);
+ break;
+ case 7:
+ /*
+ * Refused with EBUSY if the mask leaves out a CPU
+ * of the root domain that counts a deadline task.
+ */
+ sched_setaffinity(pid[w], sizeof(mix_mask[0]), &mix_mask[r % 4]);
+ break;
+ case 1:
+ set_other(pid[w]);
+ break;
+ case 2:
+ enter(cg_p, pid[w]);
+ break;
+ case 3:
+ enter(cg_q, pid[w]);
+ break;
+ case 4:
+ enter(root, pid[w]);
+ break;
+ case 5:
+ /* An exited worker is a zombie: the kill succeeds. */
+ if (kill(pid[w], SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid[w], strerror(errno));
+ break;
+ default:
+ server_set(r % 2 ? cpu_p : cpu_a,
+ (r % 3 + 1) * SERVER_RUNTIME, SERVER_PERIOD);
+ break;
+ }
+ sleep_ns((r % 2000 + 100) * NSEC_PER_USEC);
+ /* Start anew each worker that exited or was killed. */
+ for (i = 0; i < MIX_WORKERS; i++) {
+ pid_t done = waitpid(pid[i], NULL, WNOHANG);
+
+ if (done < 0 && errno != EINTR) {
+ ksft_print_msg("waitpid %d: %s\n", pid[i], strerror(errno));
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (done == pid[i]) {
+ pid[i] = mix_worker(i, seed + rounds);
+ if (pid[i] < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ rounds++;
+ }
+ }
+ }
+out:
+ mix_state[0] = 1;
+ for (i = 0; i < MIX_WORKERS; i++)
+ reap(pid[i]);
+ /*
+ * Wait for the 0-lag times of the workers. Putting the fair servers
+ * back needs the room they held.
+ */
+ sleep_ns(5 * PERIOD);
+ server_restore();
+ if (ret != KSFT_PASS) {
+ remove_partition(cg_q);
+ return ret;
+ }
+ for (i = 0, rounds = 0; i < MIX_WORKERS; i++)
+ rounds += mix_state[1 + i];
+ if (!rounds) {
+ ksft_print_msg("the workers of the mix took no transition\n");
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS && sums) {
+ if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2) ||
+ rq_bw_of(cpu_p, &tp2, &rp2)) {
+ ret = KSFT_FAIL;
+ } else if (ta2 != ta || tb2 != tb || tp2 != tp) {
+ ksft_print_msg("this_bw %ld %ld %ld, was %ld %ld %ld\n",
+ ta2, tb2, tp2, ta, tb, tp);
+ ret = KSFT_FAIL;
+ }
+ }
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P after the mix") ||
+ admits_in(cg_q, base_q, "Q after the mix") ||
+ admits_in(root, base_r, "the root domain of the rest after the mix")))
+ ret = KSFT_FAIL;
+ if (ret != KSFT_PASS)
+ ksft_print_msg("seed %u, %d transitions\n", MIX_SEED, rounds);
+ remove_partition(cg_q);
+ return ret;
+}
+
+/*
+ * CPU hotplug, from the main process, with no P. Two cases below run with
+ * P instead: death_zombie_hotplug and pi_departed_moved.
+ */
+
+static int hp_cpu = -1;
+
+/* Return whether the online file of @cpu can be written. */
+static int cpu_hotpluggable(int cpu)
+{
+ char path[64];
+
+ snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+ return !access(path, W_OK);
+}
+
+static int cpu_online(int cpu, int on)
+{
+ char dir[64];
+
+ snprintf(dir, sizeof(dir), "/sys/devices/system/cpu/cpu%d", cpu);
+ return write_file(dir, "online", on ? "1" : "0");
+}
+
+/* Bring @cpu back online: every case that follows needs it. */
+static void cpu_up(int cpu)
+{
+ int err = cpu_online(cpu, 1);
+
+ if (err)
+ ksft_print_msg("CPU %d cannot come back online: %s\n", cpu, strerror(-err));
+}
+
+/*
+ * Four deadline tasks of 30% sleep. Take hp_cpu offline and back. Once
+ * the sleepers are gone, the housekeeping root domain must admit what it
+ * did before. Skip unless 22 probe tasks fit: the sleepers take the room
+ * of 12, and the offline CPU that of about 10.
+ */
+static int hotplug_sleepers(void)
+{
+ int base = probe(NULL), after, i, ret = KSFT_PASS;
+ pid_t pid[4];
+
+ if (base < 22) {
+ ksft_print_msg("needs room for 22 probe tasks\n");
+ return KSFT_SKIP;
+ }
+ for (i = 0; i < 4; i++) {
+ mv_state[0] = 0;
+ pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+ if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+ while (i >= 0)
+ reap(pid[i--]);
+ return KSFT_FAIL;
+ }
+ }
+ if (cpu_online(hp_cpu, 0) || cpu_online(hp_cpu, 1)) {
+ ksft_print_msg("offline and online of CPU %d with deadline sleepers failed\n",
+ hp_cpu);
+ ret = KSFT_FAIL;
+ cpu_up(hp_cpu);
+ }
+ for (i = 0; i < 4; i++)
+ reap(pid[i]);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("hotplug with sleepers: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Fork a task that runs on @cpu, then restores its affinity, becomes a
+ * deadline task of 30% and sleeps. A deadline task cannot have an
+ * affinity of one CPU, so the task goes to @cpu first. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t sleeper_on(int cpu)
+{
+ pid_t pid = fork_child("a deadline sleeper");
+
+ if (!pid) {
+ cpu_set_t all, one;
+
+ CPU_ZERO(&one);
+ CPU_SET(cpu, &one);
+ if (sched_getaffinity(0, sizeof(all), &all) ||
+ sched_setaffinity(0, sizeof(one), &one))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ if (sched_setaffinity(0, sizeof(all), &all) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * A deadline task of 30% sleeps on hp_cpu. Take hp_cpu offline, then
+ * switch the task to SCHED_OTHER. hp_cpu has left the housekeeping root
+ * domain by then, but that root domain still counts the task. The switch
+ * must release the bandwidth there.
+ *
+ * Check this with hp_cpu still offline: bringing it back rebuilds the
+ * accounting, which would hide a bandwidth left behind. The reference is
+ * what the housekeeping root domain admits with hp_cpu offline and no
+ * task. Once the CPU is back and the task is gone, that root domain must
+ * also admit what it did at first.
+ */
+static int hotplug_sleeper_switched(void)
+{
+ int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+ pid_t pid;
+
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline\n", hp_cpu);
+ cpu_up(hp_cpu);
+ return KSFT_SKIP;
+ }
+ base_off = probe(NULL);
+ cpu_up(hp_cpu);
+ mv_state[0] = 0;
+ pid = sleeper_on(hp_cpu);
+ if (pid < 0 || child_up("a deadline sleeper", 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (cpu != hp_cpu) {
+ ksft_print_msg("the sleeper is on CPU %d, not on CPU %d\n", cpu, hp_cpu);
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline under a deadline sleeper\n", hp_cpu);
+ ret = KSFT_FAIL;
+ } else if (set_other(pid)) {
+ ksft_print_msg("the sleeper cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (after != base_off) {
+ ksft_print_msg("a sleeper switched out, its CPU still offline: %d probe tasks admitted, %d expected\n",
+ after, base_off);
+ ret = KSFT_FAIL;
+ }
+ }
+ cpu_up(hp_cpu);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("a sleeper switched out on an offline CPU: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Turn admission control off (sched_rt_runtime_us = -1). A task pins
+ * itself to hp_cpu, becomes a deadline task of 50% and spins. Without
+ * admission control, a deadline task can have an affinity of one CPU.
+ * Take hp_cpu offline: the mask of the task holds no active CPU. The
+ * task must go on running on another CPU. Once the CPU is back, the task
+ * is gone and admission control is on again, the housekeeping root domain
+ * must admit what it did before.
+ */
+static int hotplug_no_admission_pinned(void)
+{
+ int base = probe(NULL), after, ret = KSFT_PASS, last;
+ char saved[64];
+ pid_t pid;
+
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+ ksft_print_msg("admission control cannot be turned off\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a pinned deadline task");
+ if (!pid) {
+ cpu_set_t one;
+
+ CPU_ZERO(&one);
+ CPU_SET(hp_cpu, &one);
+ if (sched_setaffinity(0, sizeof(one), &one) ||
+ set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ mv_state[1]++;
+ }
+ if (pid < 0 || child_up("a pinned deadline task", 0)) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ sleep_ns(PERIOD);
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline under a pinned deadline task\n", hp_cpu);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ last = mv_state[1];
+ sleep_ns(3 * PERIOD);
+ if (mv_state[1] == last) {
+ ksft_print_msg("the pinned task does not run after its CPU went offline\n");
+ ret = KSFT_FAIL;
+ }
+out:
+ cpu_up(hp_cpu);
+ reap(pid);
+ if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+ ksft_print_msg("admission control cannot be turned on again\n");
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("a pinned task without admission control: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Deadline sleepers take most of the free bandwidth. Taking hp_cpu
+ * offline must then fail with EBUSY. Once the sleepers are gone, the
+ * housekeeping root domain must admit what it did before.
+ */
+static int hotplug_refused(void)
+{
+ int base = probe(NULL), n, i, ret = KSFT_PASS, err;
+ pid_t *pid;
+
+ if (base < 12) {
+ ksft_print_msg("needs room for 12 probe tasks\n");
+ return KSFT_SKIP;
+ }
+ /*
+ * The sleepers of 30% leave 30% to 60% of a CPU free. A CPU that
+ * goes offline takes more than that, so the offline must fail.
+ */
+ n = base / 3 - 1;
+ pid = calloc(n, sizeof(*pid));
+ if (!pid) {
+ ksft_print_msg("calloc: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ for (i = 0; i < n; i++) {
+ mv_state[0] = 0;
+ pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+ if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+ while (i >= 0)
+ reap(pid[i--]);
+ free(pid);
+ return KSFT_FAIL;
+ }
+ }
+ err = cpu_online(hp_cpu, 0);
+ if (!err) {
+ ksft_print_msg("CPU %d went offline with its deadline bandwidth in use\n", hp_cpu);
+ cpu_up(hp_cpu);
+ ret = KSFT_FAIL;
+ } else if (err != -EBUSY) {
+ ksft_print_msg("offline refused with %s, not EBUSY\n", strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ for (i = 0; i < n; i++)
+ reap(pid[i]);
+ free(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && probe(NULL) != base) {
+ ksft_print_msg("a refused offline changed the free bandwidth\n");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * A deadline task of 40% spins. The case switches it to SCHED_OTHER
+ * before its 0-lag time. Then it takes the CPU that the task is on
+ * offline, before that time too. Past the 0-lag time, with the CPU still
+ * offline, the housekeeping root domain must admit what it admits with a
+ * CPU offline and no task. Once the CPU is back and the task is gone, it
+ * must admit what it did at the start.
+ */
+static int hotplug_departed(void)
+{
+ int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+ pid_t pid;
+
+ /*
+ * Take the reference with hp_cpu offline and no task of ours. The
+ * task may be on another CPU: the case assumes that any one CPU
+ * offline leaves the same room. The case compares while that CPU
+ * is still offline. Bringing it back rebuilds the accounting,
+ * which would hide an error.
+ */
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+ return KSFT_FAIL;
+ }
+ base_off = probe(NULL);
+ cpu_up(hp_cpu);
+ if (base <= 0 || base_off <= 0) {
+ ksft_print_msg("no room to probe with the CPU online or offline\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = mover(MV_THROTTLED, 40 * NSEC_PER_MSEC);
+ if (pid < 0 || child_up("a departing deadline task", 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(PERIOD / 2);
+ /* Not asleep: its bandwidth stays until its 0-lag time. */
+ if (set_other(pid))
+ ret = KSFT_FAIL;
+ cpu = task_cpu_of(pid);
+ if (cpu < 0 || !cpu_hotpluggable(cpu)) {
+ cpu_set_t set;
+
+ /* Out of the class now, it may be pinned to hp_cpu. */
+ cpu = hp_cpu;
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ if (sched_setaffinity(pid, sizeof(set), &set))
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS && cpu_online(cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", cpu);
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD); /* its 0-lag time, the CPU offline */
+ if (ret == KSFT_PASS) {
+ after = probe(NULL);
+ if (after != base_off) {
+ ksft_print_msg("departed, CPU %d offline: %d admitted, %d expected\n",
+ cpu, after, base_off);
+ ret = KSFT_FAIL;
+ }
+ }
+ cpu_up(cpu);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("departed, CPU back online: %d admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Read dl_bw->total_bw of @cpu from sched_debug: the bandwidth that the
+ * root domain of @cpu counts. Return it, or -1.
+ */
+static long long total_bw_of(int cpu)
+{
+ char line[256], want[32];
+ FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+ long long bw = -1;
+ int in = 0;
+
+ if (!f)
+ return -1;
+ snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+ while (fgets(line, sizeof(line), f)) {
+ char *val = strchr(line, ':');
+
+ if (!strncmp(line, want, strlen(want)))
+ in = 1;
+ else if (in && !strncmp(line, "dl_rq[", 6))
+ break; /* the next CPU's */
+ else if (in && val && strstr(line, ".dl_bw->total_bw") &&
+ sscanf(val + 1, "%lld", &bw) == 1)
+ break;
+ }
+ fclose(f);
+ return bw;
+}
+
+#define ZOMBIE_RUNTIME (200 * NSEC_PER_MSEC)
+#define ZOMBIE_PERIOD (2 * NSEC_PER_SEC)
+
+/*
+ * Q is a partition root of two CPUs. The kernel records, with the
+ * bandwidth of a task, a CPU of the root domain that counts it. The steps:
+ * 1. A task enters Q and becomes a deadline task of 200ms every 2s.
+ * 2. It runs 190ms and exits, its 0-lag time about 1.7s ahead. The case
+ * does not reap it yet. A rebuild of the accounting still finds a
+ * zombie in the task list.
+ * 3. total_bw of Q must count the zombie.
+ * 4. The case takes each CPU of Q offline and back, one after the other.
+ * One of them is the CPU recorded with the bandwidth of the zombie.
+ * The rebuild at its offline drops that bandwidth.
+ * 5. Before the 0-lag time, total_bw of Q must be what it was without
+ * the task. No later rebuild may count the zombie again: the release
+ * at the 0-lag time removes nothing by then.
+ * 6. The case reaps the zombie. Past the 0-lag time, total_bw of Q must
+ * still be that value.
+ *
+ * The case runs with the cases in P, from the main process: Q needs the
+ * cpuset controller. Q has two CPUs because the last CPU of a root domain
+ * that holds deadline bandwidth cannot go offline.
+ */
+static int death_zombie_hotplug(void)
+{
+ int cpu[2] = { top_cpu(3), top_cpu(2) }, i, ret = KSFT_PASS;
+ long long base, bw, held, after;
+ char cpus[32];
+ uint64_t t;
+ pid_t pid;
+
+ if (ncpus < 4 || !cpu_hotpluggable(cpu[0]) || !cpu_hotpluggable(cpu[1])) {
+ ksft_print_msg("needs 4 non-isolated CPUs, 2 hotpluggable beside P\n");
+ return KSFT_SKIP;
+ }
+ snprintf(cpus, sizeof(cpus), "%d,%d", cpu[0], cpu[1]);
+ if (make_partition_cpus(cg_q, cpus)) {
+ ksft_print_msg("no partition root Q of CPUs %s\n", cpus);
+ remove_partition(cg_q);
+ return KSFT_SKIP;
+ }
+ base = total_bw_of(cpu[0]);
+ if (base < 0) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ remove_partition(cg_q);
+ return KSFT_SKIP;
+ }
+ bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD; /* as to_ratio() */
+ mv_state[0] = 0;
+ pid = fork_child("a deadline task that exits");
+ if (!pid) {
+ if (enter(cg_q, getpid()) || set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD))
+ _exit(1);
+ /* 10ms left: the 0-lag time is 100ms before the deadline. */
+ spin_ns(ZOMBIE_RUNTIME - 10 * NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("a task of Q that exits", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'Z')) {
+ ksft_print_msg("no zombie deadline task\n");
+ reap(pid);
+ remove_partition(cg_q);
+ return KSFT_FAIL;
+ }
+ /* Its 0-lag time is about 1.7s ahead. Check total_bw within 1s. */
+ t = now_ns();
+ held = total_bw_of(cpu[0]);
+ if (held != base + bw) {
+ ksft_print_msg("total_bw of Q: %lld with the zombie, %lld expected\n",
+ held, base + bw);
+ ret = KSFT_FAIL;
+ }
+ for (i = 0; i < 2 && ret == KSFT_PASS; i++) {
+ if (cpu_online(cpu[i], 0) || cpu_online(cpu[i], 1)) {
+ ksft_print_msg("offline and online of CPU %d failed\n", cpu[i]);
+ cpu_up(cpu[i]);
+ ret = KSFT_FAIL;
+ }
+ }
+ after = total_bw_of(cpu[0]);
+ if (ret == KSFT_PASS && now_ns() - t > NSEC_PER_SEC) {
+ ksft_print_msg("the hotplugs took over 1s: the 0-lag time may be past\n");
+ ret = KSFT_SKIP;
+ } else if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("total_bw of Q after the hotplugs: %lld, %lld expected%s\n",
+ after, base,
+ after == base + bw ? ": the zombie counted again" : "");
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(ZOMBIE_PERIOD); /* past its 0-lag time */
+ after = total_bw_of(cpu[0]);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("total_bw of Q past the 0-lag time: %lld, %lld expected\n",
+ after, base);
+ ret = KSFT_FAIL;
+ }
+ remove_partition(cg_q);
+ return ret;
+}
+
+/*
+ * Fork the owner of pi_departed_moved(). It takes shm_lock, becomes a
+ * deadline task and runs most of its runtime. It sets mv_state[0] to 1
+ * and sleeps 300ms. Then it sets mv_state[1] to 1 and sleeps with the
+ * lock. Return its pid, or -1.
+ */
+static pid_t departed_owner(void)
+{
+ pid_t pid = fork_child("the owner of the lock");
+
+ if (!pid) {
+ if (pthread_mutex_lock(shm_lock) ||
+ set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ /* 20ms left: the 0-lag time is 200ms before the deadline. */
+ spin_ns(ZOMBIE_RUNTIME - 20 * NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ sleep_ns(300 * NSEC_PER_MSEC);
+ mv_state[1] = 1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * The owner is a deadline task of 200ms every 2s, outside P. It holds
+ * shm_lock, runs 180ms and sleeps, its 0-lag time about 1.6s ahead. The
+ * steps:
+ * 1. The case switches the sleeping owner to SCHED_OTHER. The
+ * housekeeping root domain keeps its bandwidth until the 0-lag time.
+ * 2. The case moves the owner into P. The owner wakes up and runs there.
+ * 3. A deadline waiter of 5ms every 50ms, outside P, blocks on shm_lock
+ * and boosts the owner.
+ * 4. The case kills the waiter, which ends the boost.
+ *
+ * After step 4, total_bw of the housekeeping root domain must still count
+ * the bandwidth of the owner. total_bw of P never counted it and must be
+ * unchanged: the end of the boost must not subtract it there. Past the
+ * 0-lag time, both must be what they were at the start.
+ */
+static int pi_departed_moved(void)
+{
+ int p_cpu = top_cpu(1), r_cpu = online[0], ret = KSFT_PASS;
+ long long base_p, base_r, bw, in_p, in_r;
+ pid_t owner, waiter = -1;
+
+ base_p = total_bw_of(p_cpu);
+ base_r = total_bw_of(r_cpu);
+ if (base_p < 0 || base_r < 0) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return KSFT_SKIP;
+ }
+ if (shm_lock_init())
+ return KSFT_SKIP;
+ bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD; /* as to_ratio() */
+ mv_state[1] = 0;
+ owner = departed_owner();
+ if (owner < 0)
+ return KSFT_FAIL;
+ if (child_up("the owner of the lock", 0)) {
+ reap(owner);
+ return KSFT_FAIL;
+ }
+ /* The owner is asleep, about 1.6s before its 0-lag time. */
+ sleep_ns(20 * NSEC_PER_MSEC);
+ if (set_other(owner) || enter(cg_p, owner)) {
+ ksft_print_msg("the owner cannot leave the class and enter P\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_state(&mv_state[1], 1) || task_cpu_of(owner) != p_cpu) {
+ ksft_print_msg("the owner did not run in P\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ waiter = lock_waiter();
+ if (waiter < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_state(&mv_state[2], 1)) {
+ ksft_print_msg("no waiter\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ sleep_ns(50 * NSEC_PER_MSEC); /* blocked, boosting the owner */
+ reap(waiter);
+ waiter = -1;
+ sleep_ns(150 * NSEC_PER_MSEC); /* the waiter is past its 0-lag time */
+ in_p = total_bw_of(p_cpu);
+ in_r = total_bw_of(r_cpu);
+ if (in_p != base_p || in_r != base_r + bw) {
+ ksft_print_msg("after the boost: total_bw of P %lld, %lld expected\n",
+ in_p, base_p);
+ ksft_print_msg("after the boost: total_bw outside %lld, %lld expected\n",
+ in_r, base_r + bw);
+ ret = KSFT_FAIL;
+ }
+out:
+ reap(waiter);
+ reap(owner);
+ sleep_ns(ZOMBIE_PERIOD); /* past the owner's 0-lag time */
+ in_p = total_bw_of(p_cpu);
+ in_r = total_bw_of(r_cpu);
+ if (ret == KSFT_PASS && (in_p != base_p || in_r != base_r)) {
+ ksft_print_msg("past the 0-lag time: total_bw of P %lld, %lld expected\n",
+ in_p, base_p);
+ ksft_print_msg("past the 0-lag time: total_bw outside %lld, %lld expected\n",
+ in_r, base_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * The bandwidth of a fair server must leave the root domain with its CPU
+ * and come back with it, whatever its size. Probe the housekeeping root
+ * domain with:
+ * - the server of hp_cpu at 10%, the CPU online: the base;
+ * - the server at 10%, the CPU offline;
+ * - the server at 30%, the CPU online: two probe tasks fewer than the
+ * base;
+ * - the server at 30%, the CPU offline: as many as at 10%, offline.
+ * With the CPU back and its server at 30%, the root domain must admit two
+ * fewer than the base again. With the server back at 10%, it must admit
+ * the base.
+ */
+static int hotplug_server(void)
+{
+ int base, off10, off30;
+
+ if (server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+ return KSFT_SKIP;
+ }
+ base = probe(NULL);
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+ return KSFT_FAIL;
+ }
+ off10 = probe(NULL);
+ cpu_up(hp_cpu);
+ if (server_set(hp_cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+ admits(base - 2, "a fair server of 30%"))
+ return KSFT_FAIL;
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused, its fair server at 30%%\n", hp_cpu);
+ return KSFT_FAIL;
+ }
+ off30 = probe(NULL);
+ cpu_up(hp_cpu);
+ if (off30 != off10) {
+ ksft_print_msg("CPU %d offline: %d probe tasks, its server at 10%%, %d at 30%%\n",
+ hp_cpu, off10, off30);
+ return KSFT_FAIL;
+ }
+ if (admits(base - 2, "a fair server of 30%, its CPU back online") ||
+ server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+ admits(base, "the fair server back to 10%"))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/*
+ * A CPU that goes offline takes its capacity and the bandwidth of its
+ * fair server out of the root domain. Set the server of hp_cpu to the
+ * admission limit less 10%. The offline then costs the root domain the
+ * room of one probe task. Hold every probe task that the root domain
+ * admits but one: the CPU can still go offline. Hold one more: the
+ * offline must fail with EBUSY.
+ */
+static int hotplug_server_refused(void)
+{
+ uint64_t rt_runtime, rt_period, runtime;
+ int n, err, ret = KSFT_PASS;
+ char buf[32];
+
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) ||
+ read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf))) {
+ ksft_print_msg("no sched_rt_runtime_us or sched_rt_period_us\n");
+ return KSFT_SKIP;
+ }
+ read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf));
+ rt_runtime = atoll(buf);
+ read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf));
+ rt_period = atoll(buf);
+ if (!rt_period || 5 * rt_runtime < rt_period) {
+ ksft_print_msg("the admission limit is below a fifth of a CPU\n");
+ return KSFT_SKIP;
+ }
+ runtime = SERVER_PERIOD * rt_runtime / rt_period - SERVER_PERIOD / 10;
+ if (server_set(hp_cpu, runtime, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+ return KSFT_SKIP;
+ }
+ n = probe(NULL);
+ if (n < 2) {
+ ksft_print_msg("no room for two probe tasks\n");
+ return KSFT_SKIP;
+ }
+ if (hold(n - 1)) {
+ release_held();
+ return KSFT_FAIL;
+ }
+ err = cpu_online(hp_cpu, 0);
+ if (err) {
+ ksft_print_msg("CPU %d, room for one probe task left: offline refused (%s)\n",
+ hp_cpu, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (cpu_online(hp_cpu, 1) || hold(1)) {
+ cpu_up(hp_cpu);
+ ret = KSFT_FAIL;
+ } else {
+ err = cpu_online(hp_cpu, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("CPU %d, no room left: offline %s, not EBUSY\n",
+ hp_cpu, err ? strerror(-err) : "accepted");
+ if (!err)
+ cpu_up(hp_cpu);
+ ret = KSFT_FAIL;
+ }
+ }
+ release_held();
+ return ret;
+}
+
+/* Return whether @cpu is in @list, a CPU list as sysfs prints it. */
+static int in_cpu_list(const char *list, int cpu)
+{
+ const char *p = list;
+
+ while (*p >= '0' && *p <= '9') {
+ char *end;
+ long lo = strtol(p, &end, 10), hi = lo;
+
+ if (*end == '-')
+ hi = strtol(end + 1, &end, 10);
+ if (cpu >= lo && cpu <= hi)
+ return 1;
+ p = *end == ',' ? end + 1 : end;
+ }
+ return 0;
+}
+
+/*
+ * Wake @pid, a task that sleeps in read() on a pipe: write a byte to @fd.
+ * The task counts its wakeups in mv_state[1]. Wait until it has run and
+ * sleeps again. Return 0, or -1.
+ */
+static int fallback_wake(int fd, pid_t pid)
+{
+ int n = mv_state[1];
+
+ if (write(fd, "w", 1) != 1) {
+ ksft_print_msg("write to the deadline task: %s\n", strerror(errno));
+ return -1;
+ }
+ if (wait_for(1000, mv_state[1] != n) ||
+ wait_for(1000, proc_state_of(pid) == 'S')) {
+ ksft_print_msg("the deadline task did not wake up and sleep again\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * The CPUs of fallback_case() and no_admission_affinity_moves(). I1 and I2
+ * are isolated by isolcpus="domain". H and K are housekeeping CPUs: CPUs
+ * that isolcpus does not isolate. H is above I2. K is another one.
+ *
+ * H is above I2 because of select_fallback_rq(). It gives a task that
+ * must leave its CPU the first CPU of its mask. A task allowed on I2 and
+ * H then goes to I2, unless the kernel keeps it in its root domain.
+ *
+ * fb_i1_hp says whether I1 can go offline.
+ */
+static int fb_i1 = -1, fb_i2 = -1, fb_h = -1, fb_k = -1, fb_i1_hp;
+
+/*
+ * Read total_bw of the root domain of the isolated CPUs into @def, and of
+ * the housekeeping root domain into @hk. Return 0, or -1.
+ */
+static int fallback_totals(long long *def, long long *hk)
+{
+ *def = total_bw_of(fb_i2);
+ *hk = total_bw_of(fb_h);
+ if (*def == -1 || *hk == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Find I1, I2, H and K. Take a hotpluggable CPU as I1 if there is one.
+ * With @offline, I1 must be hotpluggable. Return 0, or -1 if the CPUs do
+ * not allow the case. fb_k is set only once all four are found.
+ */
+static int fallback_cpus(int offline)
+{
+ int i;
+
+ if (fb_k >= 0)
+ return offline && !fb_i1_hp ? -1 : 0;
+ /* A failed scan can leave I1, I2 or H set. Scan from scratch. */
+ fb_i1 = -1;
+ fb_i2 = -1;
+ fb_h = -1;
+ for (i = 0; i < nisol; i++) {
+ if (fb_i1 < 0 && cpu_hotpluggable(isol[i]))
+ fb_i1 = isol[i];
+ else if (fb_i2 < 0)
+ fb_i2 = isol[i];
+ }
+ fb_i1_hp = fb_i1 >= 0;
+ if (fb_i1 < 0 && nisol >= 2) {
+ fb_i1 = isol[0];
+ fb_i2 = isol[1];
+ }
+ for (i = 0; fb_i2 >= 0 && i < ncpus && fb_h < 0; i++) {
+ if (online[i] > fb_i2)
+ fb_h = online[i];
+ }
+ for (i = 0; fb_h >= 0 && i < ncpus && fb_k < 0; i++) {
+ if (online[i] != fb_h)
+ fb_k = online[i];
+ }
+ if (fb_i1 >= 0 && fb_i2 >= 0 && fb_k >= 0)
+ return offline && !fb_i1_hp ? -1 : 0;
+ fb_k = -1;
+ return -1;
+}
+
+enum fb_mode { FB_ON_I1, FB_ON_K, FB_ON_H, FB_MASK_H, FB_LAST_H, FB_PART_H, FB_STAY_K };
+
+/*
+ * B is a cpuset of I1, I2 and H. C is a cpuset of I1, in B. Neither is a
+ * partition. A deadline task of 10% is allowed on all CPUs. It sleeps on
+ * K or on H, and the housekeeping root domain counts it. The case moves
+ * it, asleep, into C or into B.
+ *
+ * In C, the task has I1 only. The move asks the root domain of the
+ * isolated CPUs for room, and that root domain counts the task. In B, the
+ * task still has H, a CPU of the housekeeping root domain. That root
+ * domain goes on counting it.
+ *
+ * The case then does one thing, and wakes the task. The table gives the
+ * cpuset the task is in, the CPU it sleeps on, what the case does, and
+ * the CPU the task must wake up on:
+ *
+ * case in asleep on then wakes on
+ * hotplug_fallback C I1 I1 goes offline I2
+ * hotplug_fallback_moved C K I1 goes offline I2
+ * hotplug_fallback_in_mask C H I1 goes offline I2
+ * mask_out_refused C H C is given H only: I1
+ * EBUSY
+ * hotplug_last_refused B H H offline: EBUSY H
+ * hotplug_keeps_root_domain B H H goes offline K
+ * partition_takes_refused B H Q takes H: invalid H
+ *
+ * hotplug_fallback wakes the task once after the move, so that it sleeps
+ * on I1. hotplug_keeps_root_domain gives B the CPU K as well. In
+ * partition_takes_refused, Q is a cpuset of H next to B. The case makes
+ * it a partition root, which must be invalid.
+ *
+ * With I1 offline, C has no CPU left, and the task gets the CPUs of B:
+ * I2 and H. It must wake up on I2, in the root domain that counts it. On
+ * H, it would run in a root domain that did not admit it. The three
+ * refusals have one reason: each change would leave the task no CPU in
+ * the root domain that counts it.
+ *
+ * In every case, the root domain of the CPU the task wakes on must count
+ * it, and the other one must not. The case checks the root domain of that
+ * CPU, not its number. The reference values of total_bw are taken with no
+ * task: with the CPU of the case offline if the case takes one offline,
+ * else with every CPU online. Once the task is gone, total_bw must be
+ * back to them. The messages give total_bw of the root domain of the
+ * isolated CPUs first, then of the housekeeping one.
+ */
+static int fallback_case(enum fb_mode mode)
+{
+ int in_b = mode == FB_LAST_H || mode == FB_PART_H || mode == FB_STAY_K;
+ int off_cpu;
+ int on_h = mode == FB_ON_H || mode == FB_MASK_H || in_b;
+ int no_offline = mode == FB_MASK_H || mode == FB_LAST_H || mode == FB_PART_H;
+ char cg_q[PATH_MAX];
+ int on_i1 = mode == FB_ON_I1;
+ char cg_b[PATH_MAX], cg_c[PATH_MAX], path[PATH_MAX + 32], buf[256], one[16];
+ long long def0, hk0, def_off = 0, hk_off = 0, def_ref, hk_ref, def, hk, bw;
+ int fd[2], ret = KSFT_SKIP, added = 0, made = 0, w, at, err;
+ pid_t pid = -1;
+
+ /* hotplug_keeps_root_domain takes H offline, not I1. */
+ if (!root[0] || fallback_cpus(!no_offline && mode != FB_STAY_K)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs%s\n",
+ no_offline ? "" : ", CPU hotplug");
+ return KSFT_SKIP;
+ }
+ off_cpu = mode == FB_STAY_K ? fb_h : fb_i1;
+ if ((mode == FB_LAST_H || mode == FB_STAY_K) && !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("CPU %d cannot go offline\n", fb_h);
+ return KSFT_SKIP;
+ }
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ return KSFT_SKIP;
+ if (!strstr(buf, "cpuset")) {
+ if (write_file(root, "cgroup.subtree_control", "+cpuset"))
+ return KSFT_SKIP;
+ added = 1;
+ }
+ if (snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) >= (int)sizeof(cg_q) ||
+ snprintf(cg_b, sizeof(cg_b), "%s/deadline_bw_b", root) >= (int)sizeof(cg_b) ||
+ snprintf(cg_c, sizeof(cg_c), "%s/c", cg_b) >= (int)sizeof(cg_c))
+ goto out;
+ if (mkdir(cg_b, 0755) && errno != EEXIST) {
+ ksft_print_msg("mkdir %s: %s\n", cg_b, strerror(errno));
+ goto out;
+ }
+ made = 1;
+ if (mode == FB_STAY_K)
+ snprintf(buf, sizeof(buf), "%d,%d,%d,%d", fb_i1, fb_i2, fb_h, fb_k);
+ else
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ snprintf(one, sizeof(one), "%d", fb_i1);
+ if (write_file(cg_b, "cpuset.cpus", buf) ||
+ write_file(cg_b, "cgroup.subtree_control", "+cpuset") ||
+ (mkdir(cg_c, 0755) && errno != EEXIST) ||
+ write_file(cg_c, "cpuset.cpus", one)) {
+ ksft_print_msg("cpusets B and C cannot be made\n");
+ goto out;
+ }
+ if (fallback_totals(&def0, &hk0))
+ goto out;
+ ret = KSFT_FAIL;
+ if (!no_offline) {
+ if (cpu_online(off_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", off_cpu);
+ goto out;
+ }
+ /* H may be the offline CPU: read the housekeeping total_bw on K. */
+ def_off = total_bw_of(fb_i2);
+ hk_off = total_bw_of(fb_k);
+ if (def_off == -1 || hk_off == -1)
+ ret = KSFT_SKIP;
+ cpu_up(off_cpu);
+ if (ret == KSFT_SKIP)
+ goto out;
+ }
+
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ goto out;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ int i, err;
+ char c;
+
+ /* Run on K or H, then allow every CPU again, and stay there. */
+ close(fd[1]);
+ CPU_ZERO(&set);
+ CPU_SET(on_h ? fb_h : fb_k, &set);
+ err = sched_setaffinity(0, sizeof(set), &set);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (err || sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ close(fd[0]);
+ if (pid < 0 || child_up("hotplug_fallback", 0))
+ goto out_fd;
+ if (wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out_fd;
+ bw = def + hk - def0 - hk0;
+ if (bw <= 0) {
+ ksft_print_msg("the deadline task adds %lld to total_bw\n", bw);
+ goto out_fd;
+ }
+
+ /*
+ * Move it asleep into C, or into B. For hotplug_fallback, wake it
+ * once so that it runs on I1.
+ */
+ if (enter(in_b ? cg_b : cg_c, pid) ||
+ (on_i1 && fallback_wake(fd[1], pid)))
+ goto out_fd;
+ at = task_cpu_of(pid);
+ if (on_i1 ? at != fb_i1 : on_h ? at != fb_h : at == fb_i1) {
+ ksft_print_msg("the deadline task sleeps on CPU %d\n", at);
+ ret = on_h ? KSFT_SKIP : KSFT_FAIL;
+ goto out_fd;
+ }
+ /* With no CPU offline, the reference is total_bw with no task. */
+ def_ref = no_offline ? def0 : def_off;
+ hk_ref = no_offline ? hk0 : hk_off;
+ if (mode == FB_MASK_H) {
+ snprintf(one, sizeof(one), "%d", fb_h);
+ err = write_file(cg_c, "cpuset.cpus", one);
+ if (err != -EBUSY) {
+ ksft_print_msg("C given CPU %d only: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ goto out_fd;
+ }
+ } else if (mode == FB_LAST_H) {
+ err = cpu_online(fb_h, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ cpu_up(fb_h);
+ goto out_fd;
+ }
+ } else if (mode == FB_PART_H) {
+ snprintf(one, sizeof(one), "%d", fb_h);
+ if ((mkdir(cg_q, 0755) && errno != EEXIST) ||
+ write_file(cg_q, "cpuset.cpus", one) ||
+ write_file(cg_q, "cpuset.cpus.partition", "root")) {
+ ksft_print_msg("cpuset Q cannot be made\n");
+ goto out_fd;
+ }
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg_q);
+ if (read_file(path, buf, sizeof(buf)) || !strstr(buf, "Deadline task")) {
+ ksft_print_msg("Q takes CPU %d from B: %s", fb_h, buf);
+ goto out_fd;
+ }
+ } else if (cpu_online(off_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d with the task asleep refused\n", off_cpu);
+ goto out_fd;
+ }
+ if (fallback_wake(fd[1], pid))
+ goto out_up;
+ def = total_bw_of(fb_i2);
+ hk = total_bw_of(fb_k);
+ if (def == -1 || hk == -1)
+ goto out_up;
+ w = task_cpu_of(pid);
+ ret = KSFT_PASS;
+ if (in_cpu_list(isolated, w) ? def != def_ref + bw || hk != hk_ref :
+ hk != hk_ref + bw || def != def_ref) {
+ ksft_print_msg("woken on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ w, def, hk,
+ in_cpu_list(isolated, w) ? def_ref + bw : def_ref,
+ in_cpu_list(isolated, w) ? hk_ref : hk_ref + bw);
+ ret = KSFT_FAIL;
+ } else if (in_cpu_list(isolated, w) == in_b) {
+ ksft_print_msg("woken on CPU %d: out of the root domain that counted it\n",
+ w);
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ def = total_bw_of(fb_i2);
+ hk = total_bw_of(fb_k);
+ if (def != def_ref || hk != hk_ref) {
+ ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+ def, hk, def_ref, hk_ref);
+ ret = KSFT_FAIL;
+ }
+out_up:
+ reap(pid);
+ if (!no_offline)
+ cpu_up(off_cpu);
+ pid = -1;
+out_fd:
+ reap(pid);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+ ksft_print_msg("task gone, CPU %d online: total_bw %lld, %lld, not %lld, %lld\n",
+ off_cpu, def, hk, def0, hk0);
+ ret = KSFT_FAIL;
+ }
+out:
+ if (made) {
+ if (rmdir(cg_q) && errno != ENOENT)
+ ksft_print_msg("rmdir %s: %s\n", cg_q, strerror(errno));
+ if (wait_for(1000, !rmdir(cg_c) || errno == ENOENT) ||
+ wait_for(1000, !rmdir(cg_b) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cg_b, strerror(errno));
+ }
+ if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+ return ret;
+}
+
+static int hotplug_fallback(void)
+{
+ return fallback_case(FB_ON_I1);
+}
+
+static int hotplug_fallback_moved(void)
+{
+ return fallback_case(FB_ON_K);
+}
+
+static int hotplug_fallback_in_mask(void)
+{
+ return fallback_case(FB_ON_H);
+}
+
+static int mask_out_refused(void)
+{
+ return fallback_case(FB_MASK_H);
+}
+
+static int hotplug_last_refused(void)
+{
+ return fallback_case(FB_LAST_H);
+}
+
+static int hotplug_keeps_root_domain(void)
+{
+ return fallback_case(FB_STAY_K);
+}
+
+static int partition_takes_refused(void)
+{
+ return fallback_case(FB_PART_H);
+}
+
+/*
+ * Turn admission control off. A task pins itself to H, becomes a deadline
+ * task of 10% and spins. The housekeeping root domain counts it. The case
+ * then sets its affinity to I2 with sched_setaffinity(). Nothing reserves
+ * room for it: its bandwidth must follow it to the root domain of the
+ * isolated CPUs. Once the task is gone and admission control is on again,
+ * total_bw must be what it was.
+ */
+static int no_admission_affinity_moves(void)
+{
+ long long def0, hk0, def, hk, bw;
+ int ret = KSFT_FAIL, at;
+ char saved[64];
+ cpu_set_t set;
+ pid_t pid;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (fallback_totals(&def0, &hk0))
+ return KSFT_SKIP;
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+ ksft_print_msg("admission control cannot be turned off\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ CPU_ZERO(&set);
+ CPU_SET(fb_h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ spin_ns(NSEC_PER_MSEC);
+ }
+ if (pid < 0 || child_up("no_admission_affinity_moves", 0))
+ goto out;
+ sleep_ns(PERIOD);
+ if (fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ if (bw <= 0 || def != def0) {
+ ksft_print_msg("the deadline task adds %lld to total_bw of its root domain\n", bw);
+ goto out;
+ }
+ CPU_ZERO(&set);
+ CPU_SET(fb_i2, &set);
+ if (sched_setaffinity(pid, sizeof(set), &set)) {
+ ksft_print_msg("sched_setaffinity to CPU %d: %s\n", fb_i2, strerror(errno));
+ goto out;
+ }
+ sleep_ns(2 * PERIOD);
+ at = task_cpu_of(pid);
+ if (at != fb_i2 || fallback_totals(&def, &hk))
+ goto out;
+ ret = KSFT_PASS;
+ if (def != def0 + bw || hk != hk0) {
+ ksft_print_msg("on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ at, def, hk, def0 + bw, hk0);
+ ret = KSFT_FAIL;
+ }
+out:
+ reap(pid);
+ if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+ ksft_print_msg("admission control cannot be turned on again\n");
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+ ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+ def, hk, def0, hk0);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * The cpusets of the cases below: B and Q in the root cgroup, C in B. A
+ * case makes those it needs. cs_setup() names them and enables the cpuset
+ * controller of the root cgroup if it is off. cs_cleanup() removes them,
+ * and turns the controller off again if cs_setup() turned it on.
+ */
+static char cs_b[PATH_MAX], cs_c[PATH_MAX], cs_q[PATH_MAX];
+static int cs_added;
+
+static int cs_setup(void)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ cs_added = 0;
+ if (!root[0] ||
+ snprintf(cs_b, sizeof(cs_b), "%s/deadline_bw_b", root) >= (int)sizeof(cs_b) ||
+ snprintf(cs_c, sizeof(cs_c), "%s/c", cs_b) >= (int)sizeof(cs_c) ||
+ snprintf(cs_q, sizeof(cs_q), "%s/deadline_bw_q", root) >= (int)sizeof(cs_q))
+ return -1;
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ return -1;
+ if (!strstr(buf, "cpuset")) {
+ if (write_file(root, "cgroup.subtree_control", "+cpuset"))
+ return -1;
+ cs_added = 1;
+ }
+ return 0;
+}
+
+/* Make the cpuset @cg, and give it the CPUs of @cpus. Return 0, or -1. */
+static int cs_make(const char *cg, const char *cpus)
+{
+ int err = 0;
+
+ if (mkdir(cg, 0755) && errno != EEXIST)
+ err = -errno;
+ if (!err)
+ err = write_file(cg, "cpuset.cpus", cpus);
+ if (err)
+ ksft_print_msg("cpuset %s of CPUs %s: %s\n", cg, cpus, strerror(-err));
+ return err ? -1 : 0;
+}
+
+static void cs_cleanup(void)
+{
+ remove_partition(cs_c);
+ remove_partition(cs_b);
+ remove_partition(cs_q);
+ if (cs_added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+/*
+ * Return whether the partition file of @cg holds @what. If not, and
+ * @tell, print what it holds.
+ */
+static int partition_has(const char *cg, const char *what, int tell)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+ if (read_file(path, buf, sizeof(buf)))
+ return 0;
+ if (strstr(buf, what))
+ return 1;
+ if (tell)
+ ksft_print_msg("%s/cpuset.cpus.partition: %s", cg, buf);
+ return 0;
+}
+
+/*
+ * Read total_bw of the root domain of @d into @def, and of the root
+ * domain of @h into @hk. Return 0, or -1.
+ */
+static int totals_of(int d, int h, long long *def, long long *hk)
+{
+ *def = total_bw_of(d);
+ *hk = total_bw_of(h);
+ if (*def == -1 || *hk == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Return KSFT_PASS if total_bw of the root domains of @d and @h is
+ * @def and @hk, else KSFT_FAIL. @what names the step in the message.
+ */
+static int totals_check(const char *what, int d, int h, long long def, long long hk)
+{
+ long long now_def, now_hk;
+
+ if (totals_of(d, h, &now_def, &now_hk))
+ return KSFT_FAIL;
+ if (now_def == def && now_hk == hk)
+ return KSFT_PASS;
+ ksft_print_msg("%s: total_bw %lld, %lld, not %lld, %lld\n",
+ what, now_def, now_hk, def, hk);
+ return KSFT_FAIL;
+}
+
+/*
+ * Fork a task that runs on @cpu. Then it allows every CPU if @all. If
+ * not, it allows the housekeeping CPUs that were online at the start,
+ * even those offline now. It becomes a deadline task of @runtime every
+ * @period. If @fd is -1, it spins and counts in mv_state[1]. Else it
+ * sleeps in read() on @fd and counts in mv_state[1] each byte that wakes
+ * it. On the byte 'x', it spins 90ms and exits. It sets mv_state[0] as
+ * mover() does. Return its pid, or -1.
+ */
+static pid_t task_from(int cpu, int all, uint64_t runtime, uint64_t period, int fd)
+{
+ pid_t pid = fork_child("a deadline task");
+
+ if (!pid) {
+ cpu_set_t set, one;
+ int i;
+ char c;
+
+ CPU_ZERO(&one);
+ CPU_SET(cpu, &one);
+ if (sched_setaffinity(0, sizeof(one), &one))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ CPU_ZERO(&set);
+ for (i = 0; i < (all ? CPU_SETSIZE : ncpus); i++)
+ CPU_SET(all ? i : online[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, runtime, period)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (fd < 0)
+ mv_state[1]++;
+ while (read(fd, &c, 1) == 1) {
+ if (c == 'x') {
+ spin_ns(90 * NSEC_PER_MSEC);
+ _exit(0);
+ }
+ mv_state[1]++;
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
+/*
+ * The default root domain spans every CPU that is offline. It also spans
+ * the CPU of an isolated partition. The steps:
+ * 1. Take hp_cpu offline. Make Q an isolated partition of Y, another
+ * housekeeping CPU.
+ * 2. A task in the root cgroup becomes a deadline task of 10% and sleeps.
+ * The housekeeping root domain counts it. Its affinity holds the
+ * housekeeping CPUs, hp_cpu too.
+ * 3. Move it asleep into Q. Wake it: it runs on Y. The default root
+ * domain counts it.
+ * 4. Move it asleep back to the root cgroup. Its mask holds hp_cpu, but
+ * no active CPU of the default root domain.
+ * 5. Wake it. It runs on a housekeeping CPU.
+ * Check after step 4 and again after step 5. The housekeeping root
+ * domain must count the task. The default root domain must not. An
+ * offline CPU of the default root domain is no CPU where the task can
+ * stay.
+ *
+ * The references of total_bw are taken after step 1. Once the task is
+ * gone, total_bw must be back to them.
+ */
+static int move_back_cpu_offline(void)
+{
+ int y = hp_cpu == top_cpu(1) ? top_cpu(2) : top_cpu(1), h = online[0];
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[16];
+ pid_t pid = -1;
+
+ if (cs_setup())
+ return KSFT_SKIP;
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline\n", hp_cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", y);
+ if (cs_make(cs_q, buf) || write_file(cs_q, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_q, "isolated", 1) || partition_has(cs_q, "invalid", 0)) {
+ ksft_print_msg("no isolated partition Q of CPU %d\n", y);
+ goto out;
+ }
+ if (totals_of(y, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(h, 0, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("move_back_cpu_offline", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(y, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ if (bw <= 0 || def != def0) {
+ ksft_print_msg("the deadline task adds %lld, %lld to total_bw\n",
+ def - def0, bw);
+ goto out;
+ }
+ err = enter(cs_q, pid);
+ if (err) {
+ ksft_print_msg("a move into Q refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid) ||
+ totals_check("in Q", y, h, def0 + bw, hk0))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != y) {
+ ksft_print_msg("woken in Q on CPU %d, not %d\n", cpu, y);
+ goto out;
+ }
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the move back refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (totals_check("moved back asleep", y, h, def0, hk0 + bw) ||
+ fallback_wake(fd[1], pid) ||
+ totals_check("woken", y, h, def0, hk0 + bw))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu == y || in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("woken on CPU %d\n", cpu);
+ goto out;
+ }
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ ret = totals_check("task gone", y, h, def0, hk0);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ cs_cleanup();
+ cpu_up(hp_cpu);
+ return ret;
+}
+
+enum sp_mode { SP_HK_10, SP_HK_WAKING, SP_ISO_WAKING };
+
+/*
+ * Return a CPU other than @not: an isolated one if @side, else a
+ * housekeeping one. Return -1 if there is none.
+ */
+static int other_cpu(int side, int not)
+{
+ int n = side ? nisol : ncpus, *cpus = side ? isol : online, i;
+
+ for (i = n - 1; i >= 0; i--) {
+ if (cpus[i] != not)
+ return cpus[i];
+ }
+ return -1;
+}
+
+/*
+ * A deadline task loses its CPU to an offline. It is in the root cgroup,
+ * and allowed on every CPU: its mask holds CPUs of the two root domains.
+ * It must stay in the root domain that counts it. The steps:
+ * 1. A task runs on the last housekeeping CPU that can go offline. Then
+ * it allows every CPU and becomes a deadline task, of 10% for
+ * SP_HK_10, of 90% for SP_HK_WAKING. The housekeeping root domain
+ * counts it.
+ * For SP_ISO_WAKING, the task runs on the last isolated CPU that can
+ * go offline. Then it allows the isolated CPUs only, and becomes a
+ * deadline task of 90%. The root domain of the isolated CPUs counts
+ * it. Then it allows every CPU.
+ * The task of SP_HK_10 spins. The others spin 50us and sleep 50us, in
+ * a loop.
+ * 2. Wait a period. X is the CPU the task runs on.
+ * 3. Take X offline.
+ * 4. Wait two periods. The task must run on a CPU of its root domain.
+ * That root domain must still count it.
+ * The task of SP_HK_10 is throttled most of the time. Its replenishment
+ * timer then finds its CPU offline, and picks a CPU for it. The other
+ * tasks wake up while the offline goes on, and the wakeup picks a CPU
+ * for them.
+ *
+ * A kernel that does not prefer the root domain of the task at a wakeup
+ * gives it the first CPU of its mask. The waking cases fail on such a
+ * kernel only if that CPU is of the other kind. SP_HK_WAKING needs an
+ * isolated CPU first, SP_ISO_WAKING a housekeeping CPU first.
+ *
+ * The kernel records with the task the CPU where it became a deadline
+ * task. The waking cases need X to be that CPU: a wakeup in the offline
+ * looks at the root domain of that CPU. SP_HK_10 takes any housekeeping
+ * CPU as X.
+ *
+ * Do it in rounds, each with a new task. Stop once 3 rounds check
+ * something, or after 10 rounds. A round checks nothing if X cannot go
+ * offline, or for a waking case, if X is not the CPU recorded.
+ *
+ * The references of total_bw are taken with no task: with X offline
+ * after the task is gone, and with every CPU online. The messages give
+ * total_bw of the root domain of the isolated CPUs first, then of the
+ * housekeeping one.
+ */
+static int spin_offline(enum sp_mode mode)
+{
+ uint64_t runtime = (mode == SP_HK_10 ? 10 : 90) * NSEC_PER_MSEC;
+ int iso = mode == SP_ISO_WAKING, start = -1, rounds = 0, round;
+ long long def0, hk0, def, hk, ref_def, ref_hk, bw;
+ int x, d, h, cpu, i, down, ret = KSFT_SKIP;
+ pid_t pid;
+
+ for (i = 0; i < (iso ? nisol : ncpus); i++) {
+ if (cpu_hotpluggable(iso ? isol[i] : online[i]))
+ start = iso ? isol[i] : online[i];
+ }
+ if (!nisol || (iso && nisol < 2) || start < 0) {
+ ksft_print_msg("needs isolcpus=domain: %d isolated CPUs, one hotpluggable\n",
+ iso ? 2 : 1);
+ return KSFT_SKIP;
+ }
+ if (totals_of(isol[0], online[0], &def0, &hk0))
+ return KSFT_SKIP;
+ ret = KSFT_PASS;
+ for (round = 0; round < 10 && rounds < 3 && ret == KSFT_PASS; round++) {
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a spinning deadline task");
+ if (!pid) {
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(start, &set);
+ if (sched_setaffinity(0, sizeof(set), &set))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ for (i = 0; i < (iso ? nisol : CPU_SETSIZE); i++)
+ CPU_SET(iso ? isol[i] : i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, runtime, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ mv_state[1]++;
+ if (mode == SP_HK_10)
+ continue;
+ spin_ns(50 * NSEC_PER_USEC);
+ sleep_ns(50 * NSEC_PER_USEC);
+ }
+ }
+ ret = KSFT_FAIL;
+ down = 0;
+ if (pid < 0 || child_up("spin_offline", 0))
+ goto next;
+ sleep_ns(PERIOD);
+ x = task_cpu_of(pid);
+ if (mode == SP_HK_10 ? in_cpu_list(isolated, x) || !cpu_hotpluggable(x) :
+ x != start) {
+ ret = KSFT_PASS; /* check nothing */
+ goto next;
+ }
+ d = other_cpu(1, x);
+ h = other_cpu(0, x);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ bw = iso ? def - def0 : hk - hk0;
+ if (bw <= 0 || (iso ? hk != hk0 : def != def0)) {
+ ksft_print_msg("on CPU %d, the deadline task adds %lld, %lld to total_bw\n",
+ task_cpu_of(pid), def - def0, hk - hk0);
+ goto next;
+ }
+ if (task_cpu_of(pid) != x) {
+ ret = KSFT_PASS; /* moved off X: check nothing */
+ goto next;
+ }
+ rounds++;
+ if (cpu_online(x, 0)) {
+ ksft_print_msg("offline of CPU %d with the task on it refused\n", x);
+ goto next;
+ }
+ down = 1;
+ sleep_ns(2 * PERIOD);
+ cpu = task_cpu_of(pid);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ if (totals_of(d, h, &ref_def, &ref_hk))
+ goto next;
+ if (in_cpu_list(isolated, cpu) != iso) {
+ ksft_print_msg("round %d: CPU %d offline, the task runs on CPU %d\n",
+ round, x, cpu);
+ goto next;
+ }
+ if (iso ? def != ref_def + bw || hk != ref_hk :
+ hk != ref_hk + bw || def != ref_def) {
+ ksft_print_msg("round %d: on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ round, cpu, def, hk, iso ? ref_def + bw : ref_def,
+ iso ? ref_hk : ref_hk + bw);
+ goto next;
+ }
+ ret = KSFT_PASS;
+next:
+ reap(pid);
+ if (down)
+ cpu_up(x);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", isol[0], online[0], def0, hk0);
+ }
+ if (ret == KSFT_PASS && !rounds) {
+ ksft_print_msg("the task never ran on a CPU that can go offline\n");
+ ret = KSFT_SKIP;
+ }
+ return ret;
+}
+
+static int hotplug_spinner_keeps(void)
+{
+ return spin_offline(SP_HK_10);
+}
+
+static int hotplug_waking_keeps(void)
+{
+ return spin_offline(SP_HK_WAKING);
+}
+
+static int hotplug_isolated_keeps(void)
+{
+ return spin_offline(SP_ISO_WAKING);
+}
+
+/*
+ * A rebuild of the sched domains must keep a deadline task in the root
+ * domain that counts it. The task wakes up often, so that a wakeup can
+ * land inside a rebuild. The steps:
+ * 1. A task runs on H, the highest housekeeping CPU. Then it allows every
+ * CPU and becomes a deadline task of 90%. The housekeeping root domain
+ * counts it. It spins 50us and sleeps 50us, in a loop.
+ * 2. Wait a period. The task must run on a housekeeping CPU.
+ * 3. Make Q a partition root of another housekeeping CPU, and remove it.
+ * Each of the two steps rebuilds the sched domains.
+ * 4. The task must run on a housekeeping CPU. The housekeeping root
+ * domain must still count it, and the other one must not.
+ * A kernel that gets this wrong gives the task the first CPU of the root
+ * domain of the isolated CPUs. That root domain then counts it, with no
+ * admission test. This shows only if that first CPU is isolated, as with
+ * isolcpus=domain,0,1. The case skips otherwise.
+ *
+ * Do it in 20 rounds, each with a new task. The first rebuild of a new
+ * task is the one most likely to go wrong. The case is not deterministic.
+ * The messages give total_bw of the root domain of the isolated CPUs
+ * first, then of the housekeeping one.
+ */
+static int rebuild_waking_keeps(void)
+{
+ int h = top_cpu(1), q = other_cpu(0, top_cpu(1)), d = nisol ? isol[0] : -1;
+ long long def0, hk0, def, hk;
+ int round, cpu, ret;
+ pid_t pid;
+
+ if (d < 0 || d > online[0] || q < 0) {
+ ksft_print_msg("needs isolcpus=domain with the first CPU isolated, as with 0,1\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ ret = KSFT_SKIP;
+ if (totals_of(d, h, &def0, &hk0))
+ goto out;
+ ret = KSFT_PASS;
+ for (round = 0; round < 20 && ret == KSFT_PASS; round++) {
+ mv_state[0] = 0;
+ pid = fork_child("a waking deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ int i;
+
+ CPU_ZERO(&set);
+ CPU_SET(h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 90 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ spin_ns(50 * NSEC_PER_USEC);
+ sleep_ns(50 * NSEC_PER_USEC);
+ }
+ }
+ ret = KSFT_FAIL;
+ if (pid < 0 || child_up("rebuild_waking_keeps", 0))
+ goto next;
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ if (in_cpu_list(isolated, cpu) || def != def0 || hk <= hk0) {
+ ksft_print_msg("round %d: on CPU %d before the rebuild: total_bw %lld, %lld\n",
+ round, cpu, def, hk);
+ goto next;
+ }
+ if (make_partition(cs_q, q)) {
+ ksft_print_msg("no partition root Q of CPU %d\n", q);
+ goto next;
+ }
+ remove_partition(cs_q);
+ cpu = task_cpu_of(pid);
+ if (totals_check("after the rebuild", d, h, def, hk) != KSFT_PASS)
+ ksft_print_msg("round %d: the task runs on CPU %d\n", round, cpu);
+ else if (in_cpu_list(isolated, cpu))
+ ksft_print_msg("round %d: after the rebuild, the task runs on CPU %d\n",
+ round, cpu);
+ else
+ ret = KSFT_PASS;
+next:
+ remove_partition(cs_q);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", d, h, def0, hk0);
+ }
+out:
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A user affinity can keep a task out of the root domain that counts
+ * it. The CPUs are those of fallback_case(). B is a cpuset of I1, I2, H
+ * and K. C, in B, is a cpuset of H. Neither is a partition. The steps:
+ * 1. A task in the root cgroup allows I1 and I2 only. It becomes a
+ * deadline task of 10%. The root domain of the isolated CPUs counts
+ * it.
+ * 2. It sets its affinity to I1, I2 and H. That holds the root domain
+ * that counts it, so it is accepted. It sleeps.
+ * 3. Move it asleep into C. It has H only. The housekeeping root domain
+ * counts it now. Wake it: it runs on H.
+ * 4. Take H offline. C has no CPU left, and the task would get the CPUs
+ * of B: I1, I2 and K. Its affinity keeps I1 and I2 of them. That holds
+ * no CPU of the root domain that counts it.
+ * The offline must fail with EBUSY. The task must still sleep on H, and
+ * total_bw must not change. Wake it: it must run on H.
+ */
+static int hotplug_user_mask_refused(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0) || !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs, hotplug\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d,%d", fb_i1, fb_i2, fb_h, fb_k);
+ if (cs_make(cs_b, buf) || write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_c, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+
+ CPU_ZERO(&set);
+ CPU_SET(fb_i1, &set);
+ CPU_SET(fb_i2, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ CPU_SET(fb_h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("hotplug_user_mask_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = def - def0;
+ if (bw <= 0 || hk != hk0) {
+ ksft_print_msg("the deadline task adds %lld, %lld to total_bw\n", bw, hk - hk0);
+ goto out;
+ }
+ err = enter(cs_c, pid);
+ if (err) {
+ ksft_print_msg("a move into C refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid) ||
+ totals_check("in C", fb_i2, fb_h, def0, hk0 + bw))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken in C on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ err = cpu_online(fb_h, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ if (!err) {
+ if (!fallback_wake(fd[1], pid))
+ ksft_print_msg("woken on CPU %d\n", task_cpu_of(pid));
+ cpu_up(fb_h);
+ }
+ goto out;
+ }
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h || proc_state_of(pid) != 'S') {
+ ksft_print_msg("offline refused: the task is on CPU %d, state %c\n",
+ cpu, proc_state_of(pid));
+ goto out;
+ }
+ if (totals_check("offline refused", fb_i2, fb_h, def0, hk0 + bw) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A dead task runs no more, so it cannot leave its root domain. The CPUs
+ * are those of fallback_case(). B is a cpuset of I1, I2 and H, not a
+ * partition. The steps:
+ * 1. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 100ms every 1s. It sleeps.
+ * 2. Move it asleep into B. It keeps H. The housekeeping root domain
+ * counts it.
+ * 3. Wake it. It spins 90ms and exits. The case does not reap it. Its
+ * 0-lag time is about 800ms ahead.
+ * 4. Take H offline. Once H is offline, the dead task would have no CPU
+ * in the root domain that counts it. That does not matter: it never
+ * runs again.
+ * The offline must be accepted. The case then reaps the task and brings
+ * H back. Past the 0-lag time, total_bw must be what it was.
+ *
+ * The case depends on time: it skips if the task takes more than 500ms
+ * to show as a zombie.
+ */
+static int hotplug_dead_accepted(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk;
+ uint64_t t;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0) || !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs, hotplug\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ if (cs_make(cs_b, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 100 * NSEC_PER_MSEC, NSEC_PER_SEC, fd[0]);
+ if (pid < 0 || child_up("hotplug_dead_accepted", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S'))
+ goto out;
+ err = enter(cs_b, pid);
+ if (err) {
+ ksft_print_msg("a move into B refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_totals(&def, &hk))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h || def != def0 || hk <= hk0) {
+ ksft_print_msg("in B on CPU %d: total_bw %lld, %lld\n", cpu, def, hk);
+ goto out;
+ }
+ if (write(fd[1], "x", 1) != 1) {
+ ksft_print_msg("write to the deadline task: %s\n", strerror(errno));
+ goto out;
+ }
+ t = now_ns();
+ if (wait_for(1000, proc_state_of(pid) == 'Z'))
+ goto out;
+ if (now_ns() - t > 500 * NSEC_PER_MSEC) {
+ ksft_print_msg("the task took over 500ms to exit\n");
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC); /* past do_task_dead() */
+ err = cpu_online(fb_h, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d under a dead task: %s\n", fb_h,
+ strerror(-err));
+ goto out;
+ }
+ reap(pid);
+ pid = -1;
+ cpu_up(fb_h);
+ sleep_ns(NSEC_PER_SEC); /* past the 0-lag time */
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+out:
+ reap(pid);
+ cpu_up(fb_h);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A cpuset that loses all its CPUs takes them from its parent, through
+ * its own cpuset.cpus. The CPUs are those of fallback_case(). The steps:
+ * 1. B, in the root cgroup, has cpuset.cpus H. C, in B, has cpuset.cpus
+ * I2. I2 is not in B, so C gets H.
+ * 2. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 3. Move it asleep into C. It keeps H.
+ * 4. Q, in the root cgroup, has cpuset.cpus H. Make it a partition root.
+ * B would then get the CPUs of the root cgroup, and C would get I2
+ * of them. The task would have no CPU in the root domain that counts
+ * it.
+ * Q must be an invalid partition root, for the deadline task. Wake the
+ * task: it must run on H, and the housekeeping root domain must count it.
+ */
+static int partition_takes_fallback_refused(void)
+{
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_b, buf) || write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_i2);
+ if (cs_make(cs_c, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_takes_fallback_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_c, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != fb_h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in C: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_q, buf))
+ goto out;
+ write_file(cs_q, "cpuset.cpus.partition", "root");
+ if (!partition_has(cs_q, "Deadline task", 1) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A mask that grows can still take a task out of its root domain. That
+ * happens if the user affinity of the task holds only the new CPUs.
+ * The CPUs: I1 is isolated by isolcpus="domain", H is the first
+ * housekeeping CPU. The steps:
+ * 1. Q is a partition root of the other housekeeping CPUs. The
+ * housekeeping root domain of the root cgroup then has H only.
+ * 2. B, in the root cgroup, is a cpuset of H and I1. A task enters B. It
+ * sets its affinity to I1, and runs there. It is not a deadline task.
+ * 3. Give B the CPU H only. The task gets H: its affinity holds no CPU of
+ * B. Wake it: it runs on H.
+ * 4. The task becomes a deadline task of 10%. Its mask holds the whole
+ * housekeeping root domain of the root cgroup, H. That root domain
+ * counts it. It sleeps.
+ * 5. Give B the CPUs H and I1 again. The task would then get I1 only, its
+ * affinity, in another root domain.
+ * The write of step 5 must fail with EBUSY. Wake the task: it must run on
+ * H, and the housekeeping root domain must count it.
+ */
+static int mask_grows_refused(void)
+{
+ int h = online[0], i1 = nisol ? isol[0] : -1, fd[2] = { -1, -1 };
+ int i, n = 0, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[4096];
+ pid_t pid = -1;
+
+ if (i1 < 0) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ for (i = 1; i < ncpus && n < (int)sizeof(buf); i++)
+ n += snprintf(buf + n, sizeof(buf) - n, "%s%d", i > 1 ? "," : "", online[i]);
+ if (n >= (int)sizeof(buf) || make_partition_cpus(cs_q, buf)) {
+ ksft_print_msg("no partition root Q of CPUs %s\n", buf);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ if (cs_make(cs_b, buf) || totals_of(i1, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a task with an affinity");
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+
+ CPU_ZERO(&set);
+ CPU_SET(i1, &set);
+ if (enter(cs_b, getpid()) || sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[1] = 1;
+ /* Wait for B of H only. Run there, and become a deadline task. */
+ if (read(fd[0], &c, 1) != 1)
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ if (set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ if (pid < 0 || wait_for(1000, mv_state[1]) || mv_state[1] != 1 ||
+ wait_for(1000, proc_state_of(pid) == 'S')) {
+ ksft_print_msg("the task cannot take CPU %d\n", i1);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", h);
+ err = write_file(cs_b, "cpuset.cpus", buf);
+ if (err) {
+ ksft_print_msg("B given CPU %d only: %s\n", h, strerror(-err));
+ goto out;
+ }
+ mv_state[1] = 0;
+ if (write(fd[1], "d", 1) != 1 || child_up("mask_grows_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(i1, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || cpu != h) {
+ ksft_print_msg("a deadline task on CPU %d adds %lld, %lld to total_bw\n",
+ cpu, def - def0, bw);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ err = write_file(cs_b, "cpuset.cpus", buf);
+ if (err != -EBUSY) {
+ ksft_print_msg("B given CPUs %s: %s, not EBUSY\n", buf,
+ err ? strerror(-err) : "accepted");
+ if (!err && !fallback_wake(fd[1], pid))
+ ksft_print_msg("woken on CPU %d\n", task_cpu_of(pid));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, h);
+ goto out;
+ }
+ ret = totals_check("woken", i1, h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", i1, h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * One write to cpuset.cpus can make a partition root take CPUs and give
+ * others back. The CPUs: I1 is isolated by isolcpus="domain". H is a
+ * housekeeping CPU. The steps:
+ * 1. Q is an isolated partition root of I1.
+ * 2. B, in the root cgroup, has cpuset.cpus H and I1. Q holds I1, so B
+ * gets H.
+ * 3. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 4. Move it asleep into B. It keeps H.
+ * 5. Write H to the cpuset.cpus of Q. Q would take H and give I1 back. B
+ * would then get I1 only, in another root domain.
+ * Q must become an invalid partition root, for the deadline task. Wake
+ * the task: it must run on H, and the housekeeping root domain must count
+ * it.
+ */
+static int partition_swap_refused(void)
+{
+ int h = online[ncpus - 1], i1 = nisol ? isol[0] : -1, fd[2] = { -1, -1 };
+ int cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (i1 < 0) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d", i1);
+ if (cs_make(cs_q, buf) || write_file(cs_q, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_q, "isolated", 1) || partition_has(cs_q, "invalid", 0)) {
+ ksft_print_msg("no isolated partition Q of CPU %d\n", i1);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ if (cs_make(cs_b, buf) || totals_of(i1, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_swap_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(i1, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_b, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in B: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", h);
+ write_file(cs_q, "cpuset.cpus", buf);
+ if (!partition_has(cs_q, "Deadline task", 1) || fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, h);
+ goto out;
+ }
+ ret = totals_check("woken", i1, h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", i1, h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/* Return the taint flags of the kernel, or -1. */
+static long taint_flags(void)
+{
+ char buf[64];
+
+ if (read_file("/proc/sys/kernel/tainted", buf, sizeof(buf)))
+ return -1;
+ return atol(buf);
+}
+
+#define TAINT_WARN (1L << 9)
+
+/*
+ * An invalid partition root holds no CPU of its parent. A refusal must
+ * not give any back. The CPUs are those of fallback_case(). B is a cpuset
+ * of I1, I2 and H, not a partition. The steps:
+ * 1. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 2. Move it asleep into B. It keeps H.
+ * 3. Q, in the root cgroup, has cpuset.cpus H and K. Its
+ * cpuset.cpus.exclusive is H. Make it a partition root. It would take H,
+ * the only housekeeping CPU of B. Q must be invalid, for the deadline
+ * task.
+ * 4. Write H to the cpuset.cpus of Q. The same check refuses again.
+ * The kernel must not warn: the taint flag W must stay clear. Q must
+ * still be invalid for the deadline task. The case skips if the flag
+ * was set before.
+ */
+static int partition_refused_no_warn(void)
+{
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (taint_flags() < 0 || (taint_flags() & TAINT_WARN)) {
+ ksft_print_msg("the kernel has warned already\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ if (cs_make(cs_b, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_refused_no_warn", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_b, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != fb_h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in B: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", fb_h, fb_k);
+ if (cs_make(cs_q, buf))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (write_file(cs_q, "cpuset.cpus.exclusive", buf)) {
+ ksft_print_msg("Q cannot have CPU %d exclusive\n", fb_h);
+ goto out;
+ }
+ write_file(cs_q, "cpuset.cpus.partition", "root");
+ if (!partition_has(cs_q, "Deadline task", 1))
+ goto out;
+ write_file(cs_q, "cpuset.cpus", buf);
+ if (taint_flags() & TAINT_WARN) {
+ ksft_print_msg("Q given CPU %d: the kernel warned\n", fb_h);
+ goto out;
+ }
+ if (!partition_has(cs_q, "Deadline task", 1) || fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/* Moves between two root domains, from the main process, with P */
+
+/*
+ * Fork a deadline task of 30ms every 100ms, pinned to the @n CPUs of
+ * @cpus. It runs 1ms, then sleeps. If @fd is not -1, each byte read from
+ * @fd wakes it up, and it counts the wakeups in mv_state[1]. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t pinned_sleeper(const int *cpus, int n, int fd)
+{
+ pid_t pid = fork_child("a pinned deadline task");
+
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+ int i;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < n; i++)
+ CPU_SET(cpus[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ while (fd >= 0 && read(fd, &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * Read total_bw of three root domains into @bw: the root domain of the
+ * isolated CPUs, the housekeeping root domain, and P. Return 0, or -1.
+ */
+static int split_totals(long long bw[3])
+{
+ bw[0] = total_bw_of(isol[0]);
+ bw[1] = total_bw_of(online[0]);
+ bw[2] = total_bw_of(top_cpu(1));
+ if (bw[0] == -1 || bw[1] == -1 || bw[2] == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Return KSFT_PASS if total_bw is now @want, else KSFT_FAIL. The message
+ * gives total_bw of the root domain of the isolated CPUs, then of the
+ * housekeeping root domain, then of P.
+ */
+static int split_check(const char *what, const long long want[3])
+{
+ long long bw[3];
+
+ if (split_totals(bw))
+ return KSFT_FAIL;
+ if (bw[0] == want[0] && bw[1] == want[1] && bw[2] == want[2])
+ return KSFT_PASS;
+ ksft_print_msg("%s: total_bw %lld, %lld, %lld, not %lld, %lld, %lld\n",
+ what, bw[0], bw[1], bw[2], want[0], want[1], want[2]);
+ return KSFT_FAIL;
+}
+
+/*
+ * Read total_bw into @base, for the cases below. Return 0, or nonzero if
+ * the case must skip. It needs CPUs isolated by isolcpus="domain".
+ */
+static int split_skip(long long base[3])
+{
+ if (!nisol) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return 1;
+ }
+ return split_totals(base);
+}
+
+/*
+ * The root cpuset holds the CPUs that isolcpus="domain" isolates, so it
+ * spans two root domains. The affinity of a task can keep it in either.
+ * A move into the root cpuset must reserve the bandwidth of the task in
+ * the root domain that the task gets.
+ *
+ * Run twice, with a deadline sleeper of 30% pinned to the isolated CPUs,
+ * then with another one pinned to the housekeeping CPUs except P's. Move
+ * the sleeper into P and back to the root cgroup, then kill it. Once it
+ * is gone, total_bw of the three root domains must be what it was.
+ */
+static int move_isolated(void)
+{
+ long long base[3];
+ int round, err;
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ for (round = 0; round < 2; round++) {
+ const char *what = round ? "pinned to the other CPUs" :
+ "pinned to the isolated CPUs";
+
+ mv_state[0] = 0;
+ pid = round ? pinned_sleeper(online, ncpus - 1, -1) :
+ pinned_sleeper(isol, nisol, -1);
+ if (pid < 0 || child_up(what, 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ reap(pid);
+ if (err) {
+ ksft_print_msg("%s: a move refused: %s\n", what, strerror(-err));
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ if (split_check(what, base))
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+/*
+ * Fork a process pinned to the isolated CPUs. Its threads become deadline
+ * tasks of 10ms every 100ms, as in probe(), until one is refused. Then it
+ * sets mv_state[1] to 1, or to -1 on an error. The threads sleep until the
+ * process is killed.
+ */
+static pid_t fill_isolated(void)
+{
+ pid_t pid = fork_child("the tasks that fill the isolated CPUs");
+
+ if (!pid) {
+ int i, max = 16 * nisol + 16;
+ struct probe_slot *s = calloc(max, sizeof(*s));
+ pthread_attr_t a;
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &set);
+ if (!s || sched_setaffinity(0, sizeof(set), &set) ||
+ sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+ pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ for (i = 0; i < max; i++) {
+ if (pthread_create(&s[i].t, &a, probe_thread, &s[i]) ||
+ sem_wait_for(&probe_done, 10000)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ if (s[i].err)
+ break;
+ }
+ mv_state[1] = i < max && s[i].err == -EBUSY ? 1 : -1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * Move a deadline sleeper pinned to the isolated CPUs into P. Fill the
+ * isolated CPUs with deadline tasks of 10%, until one is refused. Moving
+ * the sleeper back must then be refused with EBUSY.
+ */
+static int move_isolated_refused(void)
+{
+ pid_t pid, fill = -1;
+ long long base[3];
+ int err, ret = KSFT_FAIL;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ mv_state[0] = 0;
+ pid = pinned_sleeper(isol, nisol, -1);
+ if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0))
+ goto out;
+ err = enter(cg_p, pid);
+ if (err) {
+ ksft_print_msg("a move into P refused: %s\n", strerror(-err));
+ goto out;
+ }
+ mv_state[1] = 0;
+ fill = fill_isolated();
+ if (fill < 0 || wait_for(10000, mv_state[1]) || mv_state[1] != 1) {
+ ksft_print_msg("the isolated CPUs cannot be filled\n");
+ goto out;
+ }
+ err = enter(root, pid);
+ if (err == -EBUSY) {
+ ret = KSFT_PASS;
+ } else {
+ ksft_print_msg("a move back to the full isolated CPUs: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ }
+out:
+ reap(fill);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("the isolated CPUs filled", base);
+ return ret;
+}
+
+/*
+ * A deadline sleeper is pinned to the isolated CPUs. The case allows it
+ * all CPUs, then moves it into P and back, without waking it. So it stays
+ * on an isolated CPU, which its mask holds again. The root domain of the
+ * isolated CPUs must still count its bandwidth. Then wake it: it must run
+ * on an isolated CPU, and that root domain must still count it.
+ */
+static int move_isolated_asleep(void)
+{
+ long long base[3], want[3];
+ int fd[2], err, cpu, i, ret = KSFT_FAIL;
+ cpu_set_t set;
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = pinned_sleeper(isol, nisol, fd[0]);
+ if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || split_totals(want))
+ goto out;
+ CPU_ZERO(&set);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+ if (!err)
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the affinity of all CPUs, or a move: %s\n", strerror(-err));
+ goto out;
+ }
+ for (i = 0; i < 2; i++) {
+ const char *what = i ? "woken" : "moved back asleep";
+
+ cpu = task_cpu_of(pid);
+ if (!in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("%s on CPU %d, not an isolated one\n", what, cpu);
+ goto out;
+ }
+ if (split_check(what, want))
+ goto out;
+ if (!i && fallback_wake(fd[1], pid))
+ goto out;
+ }
+ ret = KSFT_PASS;
+out:
+ reap(pid);
+ close(fd[0]);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ return ret;
+}
+
+/* The CPUs of move_spanning(): two root domains, an isolated CPU first */
+static cpu_set_t span_set;
+
+/*
+ * Fork a task that pins itself to the isolated CPUs, becomes a deadline
+ * task of 30ms every 100ms, then takes the CPUs of span_set. If @fd is
+ * -1, it spins and counts in mv_state[1]. Else it sleeps, each byte read
+ * from @fd wakes it up, and it counts the wakeups there. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t spanning_task(int fd)
+{
+ pid_t pid = fork_child("a deadline task on two root domains");
+
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+ int i;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD) ||
+ sched_setaffinity(0, sizeof(span_set), &span_set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (fd < 0)
+ mv_state[1]++;
+ while (read(fd, &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * A task becomes a deadline task on the isolated CPUs, where it is
+ * counted. Then it takes a wider affinity: the isolated CPUs, and the
+ * housekeeping CPUs above the first isolated one except P's. In the root
+ * cpuset, its CPUs span two root domains, and the lowest one is isolated.
+ * The case moves it into P, where it runs, and back to the root cgroup.
+ * The task must then run on a housekeeping CPU, and the housekeeping root
+ * domain must count its bandwidth. The root domain of the isolated CPUs
+ * must count it no more. Run twice: with a running task, then with a
+ * sleeper woken in P and again after the move back.
+ *
+ * The kernel counts a task that has a housekeeping CPU in the
+ * housekeeping root domain. The move reserves there, and a rebuild of the
+ * accounting makes the same choice. The lowest CPU of the task is
+ * isolated, so a move that takes the first CPU of the mask reserves in
+ * the wrong root domain.
+ */
+static int move_spanning(void)
+{
+ int fd[2] = { -1, -1 }, round, cpu, err, i, n = 0, ret = KSFT_PASS;
+ long long base[3], want[3];
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ CPU_ZERO(&span_set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &span_set);
+ for (i = 0; i < ncpus - 1; i++) {
+ if (online[i] > isol[0]) {
+ CPU_SET(online[i], &span_set);
+ n++;
+ }
+ }
+ if (!n) {
+ ksft_print_msg("needs a CPU above the first isolated one, besides P\n");
+ return KSFT_SKIP;
+ }
+ for (round = 0; round < 2 && ret == KSFT_PASS; round++) {
+ const char *what = round ? "a sleeper" : "a running task";
+
+ ret = KSFT_FAIL;
+ if (round && pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ break;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = spanning_task(fd[0]);
+ if (pid < 0 || child_up(what, 0) || split_totals(want))
+ goto next;
+ /*
+ * Moved back, its bandwidth goes to the housekeeping root
+ * domain, not to the root domain of the isolated CPUs.
+ */
+ want[1] += want[0] - base[0];
+ want[0] = base[0];
+ err = enter(cg_p, pid);
+ if (err) {
+ ksft_print_msg("%s: a move into P refused: %s\n", what, strerror(-err));
+ goto next;
+ }
+ if ((round && fallback_wake(fd[1], pid)) ||
+ wait_for(1000, task_cpu_of(pid) == top_cpu(1))) {
+ ksft_print_msg("%s: not on P's CPU\n", what);
+ goto next;
+ }
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("%s: the move back refused: %s\n", what, strerror(-err));
+ goto next;
+ }
+ if (round ? fallback_wake(fd[1], pid) :
+ wait_for(1000, task_cpu_of(pid) != top_cpu(1)))
+ goto next;
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("%s: moved back to isolated CPU %d\n", what, cpu);
+ goto next;
+ }
+ ret = split_check(what, want);
+next:
+ reap(pid);
+ if (round) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ }
+ return ret;
+}
+
+/*
+ * A deadline task moved asleep into another cpuset still sleeps on a CPU
+ * of the root domain it left. The root domain it moved to counts its
+ * bandwidth. sched_setaffinity() must check a new mask against the root
+ * domain that counts the task, not the one it sleeps in.
+ *
+ * H is a cpuset here, not the CPU of fallback_case(). It holds the
+ * housekeeping CPUs except P's. The steps:
+ * 1. Pin a fair task to the isolated CPUs.
+ * 2. Move it into H. Its affinity has no CPU of H, so it gets all the
+ * CPUs of H.
+ * 3. Make it a deadline task there, and let it sleep.
+ * 4. Move it asleep into the root cgroup. Its affinity gives it the
+ * isolated CPUs again, and their root domain counts it.
+ * 5. Set its affinity to the CPUs of H. That would take it out of the
+ * root domain that counts it, so it must fail with EBUSY.
+ *
+ * If the change is accepted, wake the task and report where it runs.
+ */
+static int affinity_moved_asleep(void)
+{
+ char cg_h[PATH_MAX], cpus[4096], what[64];
+ int fd[2], i, n = 0, cpu, err, made = 0, ret = KSFT_SKIP;
+ long long base[3], want[3], bw;
+ cpu_set_t set;
+ pid_t pid = -1;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ /*
+ * H holds the housekeeping CPUs except P's. A task there can become
+ * a deadline task: its mask holds its whole root domain.
+ */
+ CPU_ZERO(&set);
+ for (i = 0; i < ncpus - 1 && n < (int)sizeof(cpus); i++) {
+ n += snprintf(cpus + n, sizeof(cpus) - n, "%s%d", i ? "," : "", online[i]);
+ CPU_SET(online[i], &set);
+ }
+ if (snprintf(cg_h, sizeof(cg_h), "%s/deadline_bw_h", root) >= (int)sizeof(cg_h))
+ goto out;
+ if (mkdir(cg_h, 0755) && errno != EEXIST) {
+ ksft_print_msg("mkdir %s: %s\n", cg_h, strerror(errno));
+ goto out;
+ }
+ made = 1;
+ if (n >= (int)sizeof(cpus) || write_file(cg_h, "cpuset.cpus", cpus)) {
+ ksft_print_msg("cpuset H cannot be made\n");
+ goto out;
+ }
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a task moved asleep");
+ if (!pid) {
+ cpu_set_t iso;
+ char c;
+
+ CPU_ZERO(&iso);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &iso);
+ if (sched_setaffinity(0, sizeof(iso), &iso)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ mv_state[1] = 1;
+ /* Wait for the move into H. Become a deadline task there. */
+ if (read(fd[0], &c, 1) != 1 || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ if (pid < 0)
+ goto out;
+ if (wait_for(1000, mv_state[1]) || mv_state[1] != 1) {
+ ksft_print_msg("the task cannot take the isolated CPUs\n");
+ goto out;
+ }
+ err = enter(cg_h, pid);
+ if (err) {
+ ksft_print_msg("a move into H refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (write(fd[1], "d", 1) != 1) {
+ ksft_print_msg("write to the task: %s\n", strerror(errno));
+ goto out;
+ }
+ if (child_up("a task moved into H", 0) || wait_for(1000, proc_state_of(pid) == 'S'))
+ goto out;
+ bw = (30 * NSEC_PER_MSEC << 20) / PERIOD; /* as to_ratio() */
+ memcpy(want, base, sizeof(want));
+ want[1] += bw;
+ if (split_check("a deadline task in H", want))
+ goto out;
+
+ /*
+ * Moved asleep, it stays on its CPU. The root domain of the
+ * isolated CPUs counts it.
+ */
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the move into the root cgroup refused: %s\n", strerror(-err));
+ goto out;
+ }
+ cpu = task_cpu_of(pid);
+ if (in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("moved asleep on CPU %d, an isolated one\n", cpu);
+ goto out;
+ }
+ want[0] += bw;
+ want[1] -= bw;
+ if (split_check("moved asleep", want))
+ goto out;
+
+ /* The CPUs of H leave out the root domain that counts the task. */
+ err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+ if (err == -EBUSY) {
+ ret = KSFT_PASS;
+ goto out;
+ }
+ ksft_print_msg("the affinity of the CPUs of H: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ if (err || fallback_wake(fd[1], pid))
+ goto out;
+ /* Report its CPU, and whether the root domain of that CPU counts it. */
+ cpu = task_cpu_of(pid);
+ memcpy(want, base, sizeof(want));
+ want[in_cpu_list(isolated, cpu) ? 0 : 1] += bw;
+ snprintf(what, sizeof(what), "woken on CPU %d", cpu);
+ if (split_check(what, want) == KSFT_PASS)
+ ksft_print_msg("%s: its root domain counts it\n", what);
+out:
+ reap(pid);
+ close(fd[0]);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ if (made && wait_for(1000, !rmdir(cg_h) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cg_h, strerror(errno));
+ return ret;
+}
+
+/* Setup */
+
+static int find_cgroup2(void)
+{
+ char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+ FILE *f = fopen("/proc/self/mounts", "r");
+ int found = 0;
+
+ while (f && !found && fgets(line, sizeof(line), f)) {
+ if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+ !strcmp(type, "cgroup2")) {
+ snprintf(root, sizeof(root), "%s", dir);
+ found = 1;
+ }
+ }
+ if (f)
+ fclose(f);
+ return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+ if (ret == KSFT_PASS)
+ ksft_test_result_pass("%s\n", name);
+ else if (ret == KSFT_SKIP)
+ ksft_test_result_skip("%s\n", name);
+ else
+ ksft_test_result_fail("%s\n", name);
+}
+
+/* sched_setaffinity() against sched_setattr(), with no P */
+
+/*
+ * The state that affinity_racer() shares with the case. pid and one are
+ * its target and its mask. go and done count the rounds. ret is the
+ * result of its last call.
+ */
+static struct {
+ pid_t pid;
+ cpu_set_t one;
+ atomic_int go;
+ atomic_int done;
+ atomic_int stop;
+ int ret;
+} aff;
+
+/*
+ * Give aff.pid the affinity aff.one, once for each new value of aff.go.
+ * Return when aff.stop is set.
+ */
+static void *affinity_racer(void *arg)
+{
+ int round;
+
+ for (round = 1; ; round++) {
+ while (aff.go < round && !aff.stop)
+ ;
+ if (aff.stop)
+ break;
+ aff.ret = sched_setaffinity(aff.pid, sizeof(aff.one), &aff.one) ? -errno : 0;
+ aff.done = round;
+ }
+ return arg;
+}
+
+/*
+ * Race sched_setaffinity() against sched_setattr() on a sleeping task.
+ * One thread gives the task an affinity of one CPU. Another one makes it
+ * a deadline task. A deadline task must be allowed on its whole root
+ * domain, so at most one of the two calls may succeed:
+ * - sched_setattr() fails with EPERM if the affinity came first;
+ * - sched_setaffinity() fails with EBUSY if the policy came first.
+ *
+ * Do it 1000 times. Each round starts with the task back in SCHED_OTHER
+ * with its full affinity. Once the task is gone, the free bandwidth must
+ * be what it was. The case needs no cpuset: it runs without P, with the
+ * hotplug cases.
+ */
+static int affinity_vs_setattr(void)
+{
+ const char *name = "affinity_vs_setattr";
+ int base, i, cpu, both = 0, ret = KSFT_PASS;
+ cpu_set_t all;
+ pthread_t t;
+ pid_t pid;
+
+ base = probe(NULL);
+ pid = fork_child(name);
+ if (!pid) {
+ for (;;)
+ pause();
+ }
+ if (pid < 0)
+ return KSFT_FAIL;
+ /*
+ * Wait until the task sleeps in pause(). A task that still runs
+ * when it becomes a deadline task can overrun its 2us. Its
+ * bandwidth then stays counted until its 0-lag time, seconds after
+ * it is gone, and the next cases would see it.
+ */
+ if (wait_for(1000, proc_state_of(pid) == 'S') ||
+ sched_getaffinity(pid, sizeof(all), &all)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ CPU_ZERO(&aff.one);
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (CPU_ISSET(cpu, &all)) {
+ CPU_SET(cpu, &aff.one);
+ break;
+ }
+ }
+ aff.pid = pid;
+ aff.go = 0;
+ aff.done = 0;
+ aff.stop = 0;
+ if (pthread_create(&t, NULL, affinity_racer, NULL)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+
+ for (i = 1; i <= 1000 && ret == KSFT_PASS; i++) {
+ int err;
+
+ if (set_other(pid) || sched_setaffinity(pid, sizeof(all), &all)) {
+ ksft_print_msg("%s: the task cannot be reset: %s\n", name, strerror(errno));
+ ret = KSFT_FAIL;
+ break;
+ }
+ aff.go = i;
+ err = set_deadline(pid, 2 * NSEC_PER_USEC, PERIOD);
+ while (aff.done < i)
+ ;
+ if (!err && !aff.ret) {
+ both++;
+ } else if ((err && err != -EPERM) || (aff.ret && aff.ret != -EBUSY)) {
+ ksft_print_msg("%s: round %d: sched_setattr: %s, sched_setaffinity: %s\n",
+ name, i, strerror(-err), strerror(-aff.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ aff.stop = 1;
+ join(t);
+ if (both) {
+ ksft_print_msg("%s: %d rounds of %d: a deadline task with an affinity of one CPU\n",
+ name, both, i - 1);
+ ret = KSFT_FAIL;
+ }
+ /* A deadline task of 2us every 100ms would take seconds to exit. */
+ if (set_other(pid))
+ ksft_print_msg("%s: the task stays a deadline task\n", name);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && admits(base, "after the task is gone"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+/* The tables of the cases, and main() */
+
+enum where {
+ MEASURED_IN_P, /* in a child in P, probed before and after */
+ IN_P, /* in a child in P, which measures itself */
+ FROM_OUTSIDE, /* in the main process, outside P */
+};
+
+static const struct p_case {
+ const char *name;
+ int (*fn)(void);
+ enum where where;
+} p_cases[] = {
+ { "pi_deadline_owner", pi_deadline_owner, MEASURED_IN_P },
+ { "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
+ { "pi_former_owner_asleep", pi_former_owner_asleep, MEASURED_IN_P },
+ { "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
+ { "pi_chain_two_owners", pi_chain_two_owners, MEASURED_IN_P },
+ { "pi_departed_moved", pi_departed_moved, FROM_OUTSIDE },
+ { "params_asleep", params_asleep, IN_P },
+ { "params_non_contending", params_non_contending, MEASURED_IN_P },
+ { "params_throttled", params_throttled, MEASURED_IN_P },
+ { "params_refused", params_refused, IN_P },
+ { "params_keep_class_change", params_keep_class_change, MEASURED_IN_P },
+ { "params_keep_boosted", params_keep_boosted, MEASURED_IN_P },
+ { "boosted_owner_policy", boosted_owner_policy, MEASURED_IN_P },
+ { "deboost_resets_period", deboost_resets_period, MEASURED_IN_P },
+ { "deboost_released_resets_period", deboost_released_resets_period, MEASURED_IN_P },
+ { "deboost_in_class_resets_period", deboost_in_class_resets_period, MEASURED_IN_P },
+ { "deboost_after_donor_refill", deboost_after_donor_refill, MEASURED_IN_P },
+ { "deboost_keeps_own_runtime", deboost_keeps_own_runtime, MEASURED_IN_P },
+ { "return_running_new_params", return_running_new_params, MEASURED_IN_P },
+ { "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
+ { "change_running_new_params", change_running_new_params, MEASURED_IN_P },
+ { "change_boosted_new_params", change_boosted_new_params, MEASURED_IN_P },
+ { "change_constrained_new_params", change_constrained_new_params, MEASURED_IN_P },
+ { "wake_constrained_new_params", wake_constrained_new_params, MEASURED_IN_P },
+ { "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
+ { "setattr_blocked_keeps_order", setattr_blocked_keeps_order, MEASURED_IN_P },
+ { "boost_blocked_keeps_order", boost_blocked_keeps_order, MEASURED_IN_P },
+ { "pi_chain_donor_leaves", pi_chain_donor_leaves, MEASURED_IN_P },
+ { "move_asleep", move_asleep, FROM_OUTSIDE },
+ { "move_throttled", move_throttled, FROM_OUTSIDE },
+ { "move_non_contending", move_non_contending, FROM_OUTSIDE },
+ { "move_refused", move_refused, FROM_OUTSIDE },
+ { "move_zero_bw", move_zero_bw, FROM_OUTSIDE },
+ { "move_vs_rebuild", move_vs_rebuild, FROM_OUTSIDE },
+ { "move_became_deadline", move_became_deadline, FROM_OUTSIDE },
+ { "move_became_deadline_no_room", move_became_deadline_no_room, FROM_OUTSIDE },
+ { "move_affinity_changed", move_affinity_changed, FROM_OUTSIDE },
+ { "move_affinity_became_deadline", move_affinity_became_deadline, FROM_OUTSIDE },
+ { "move_died", move_died, FROM_OUTSIDE },
+ { "move_woken", move_woken, FROM_OUTSIDE },
+ { "move_woken_full", move_woken_full, FROM_OUTSIDE },
+ { "move_cancelled", move_cancelled, FROM_OUTSIDE },
+ { "frozen_moved_switched", frozen_moved_switched, FROM_OUTSIDE },
+ { "move_isolated", move_isolated, FROM_OUTSIDE },
+ { "move_isolated_refused", move_isolated_refused, FROM_OUTSIDE },
+ { "move_isolated_asleep", move_isolated_asleep, FROM_OUTSIDE },
+ { "move_spanning", move_spanning, FROM_OUTSIDE },
+ { "affinity_moved_asleep", affinity_moved_asleep, FROM_OUTSIDE },
+ { "death_running", death_running, MEASURED_IN_P },
+ { "death_switched_out", death_switched_out, MEASURED_IN_P },
+ { "death_shrunk", death_shrunk, MEASURED_IN_P },
+ { "stopped_switched_out", stopped_switched_out, MEASURED_IN_P },
+ { "death_killed_asleep", death_killed_asleep, MEASURED_IN_P },
+ { "death_killed_non_contending", death_killed_non_contending, MEASURED_IN_P },
+ { "death_killed_throttled", death_killed_throttled, MEASURED_IN_P },
+ { "death_killed_running", death_killed_running, MEASURED_IN_P },
+ { "death_pi_waiter", death_pi_waiter, MEASURED_IN_P },
+ { "death_pi_owner", death_pi_owner, MEASURED_IN_P },
+ { "death_pi_deadline_owner", death_pi_deadline_owner, MEASURED_IN_P },
+ { "death_group", death_group, MEASURED_IN_P },
+ { "death_zombie_switched", death_zombie_switched, FROM_OUTSIDE },
+ { "death_zombie_hotplug", death_zombie_hotplug, FROM_OUTSIDE },
+ { "yield_switched_out", yield_switched_out, IN_P },
+ { "yield_moved", yield_moved, FROM_OUTSIDE },
+ { "yield_killed", yield_killed, MEASURED_IN_P },
+ { "overrun_caught", overrun_caught, IN_P },
+ { "overrun_group_death", overrun_group_death, MEASURED_IN_P },
+ { "reclaim_runs", reclaim_runs, MEASURED_IN_P },
+ { "reclaim_switched", reclaim_switched, MEASURED_IN_P },
+ { "reclaim_moved", reclaim_moved, FROM_OUTSIDE },
+ { "server_params", server_params, IN_P },
+ { "server_rebuilds", server_rebuilds, FROM_OUTSIDE },
+ { "server_states", server_states, FROM_OUTSIDE },
+ { "server_reclaim", server_reclaim, MEASURED_IN_P },
+ { "params_waking", params_waking, FROM_OUTSIDE },
+ { "random_mix", random_mix, FROM_OUTSIDE },
+ { "move_normalized", move_normalized, FROM_OUTSIDE },
+};
+
+#define N_P_CASES ((int)ARRAY_SIZE(p_cases))
+
+static const struct hp_case {
+ const char *name;
+ int (*fn)(void);
+ int no_hp; /* takes no CPU offline */
+} hp_cases[] = {
+ { "hotplug_sleepers", hotplug_sleepers },
+ { "hotplug_sleeper_switched", hotplug_sleeper_switched },
+ { "hotplug_no_admission_pinned", hotplug_no_admission_pinned },
+ { "hotplug_refused", hotplug_refused },
+ { "hotplug_departed", hotplug_departed },
+ { "hotplug_server", hotplug_server },
+ { "hotplug_server_refused", hotplug_server_refused },
+ { "hotplug_fallback", hotplug_fallback },
+ { "hotplug_fallback_moved", hotplug_fallback_moved },
+ { "hotplug_fallback_in_mask", hotplug_fallback_in_mask },
+ { "mask_out_refused", mask_out_refused, 1 },
+ { "hotplug_last_refused", hotplug_last_refused },
+ { "hotplug_keeps_root_domain", hotplug_keeps_root_domain },
+ { "partition_takes_refused", partition_takes_refused, 1 },
+ { "no_admission_affinity_moves", no_admission_affinity_moves, 1 },
+ { "affinity_vs_setattr", affinity_vs_setattr, 1 },
+ { "move_back_cpu_offline", move_back_cpu_offline },
+ { "hotplug_spinner_keeps", hotplug_spinner_keeps },
+ { "hotplug_waking_keeps", hotplug_waking_keeps },
+ { "hotplug_isolated_keeps", hotplug_isolated_keeps },
+ { "rebuild_waking_keeps", rebuild_waking_keeps, 1 },
+ { "hotplug_user_mask_refused", hotplug_user_mask_refused },
+ { "hotplug_dead_accepted", hotplug_dead_accepted },
+ { "partition_takes_fallback_refused", partition_takes_fallback_refused, 1 },
+ { "mask_grows_refused", mask_grows_refused, 1 },
+ { "partition_swap_refused", partition_swap_refused, 1 },
+ { "partition_refused_no_warn", partition_refused_no_warn, 1 },
+};
+
+#define N_HP_CASES ((int)ARRAY_SIZE(hp_cases))
+
+/* Run the cases of p_cases[], each with a P made anew. */
+static void run_p_cases(void)
+{
+ char path[PATH_MAX + 32], buf[256];
+ int i, ret, added = 0, told = 0, ok;
+
+ ok = !find_cgroup2() &&
+ snprintf(cg_p, sizeof(cg_p), "%s/deadline_bw_p", root) < (int)sizeof(cg_p) &&
+ snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) < (int)sizeof(cg_q);
+ if (ok) {
+ /*
+ * Enable the cpuset controller for the children of the
+ * root while the cases run.
+ */
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ ok = !read_file(path, buf, sizeof(buf));
+ added = ok && !strstr(buf, "cpuset");
+ if (added && write_file(root, "cgroup.subtree_control", "+cpuset")) {
+ ok = 0;
+ added = 0;
+ }
+ }
+ for (i = 0; i < N_P_CASES; i++) {
+ const struct p_case *c = &p_cases[i];
+
+ if (ok && make_p()) {
+ remove_p();
+ ok = 0;
+ }
+ if (!ok) {
+ if (!told++)
+ ksft_print_msg("no one-CPU partition root: cases in P skipped\n");
+ ksft_test_result_skip("%s\n", c->name);
+ continue;
+ }
+ if (c->where == MEASURED_IN_P)
+ ret = measured(c->name, cg_p, c->fn);
+ else if (c->where == IN_P)
+ ret = run_in(cg_p, c->fn);
+ else
+ ret = c->fn();
+ remove_p();
+ server_restore();
+ report(c->name, ret);
+ }
+ if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+/*
+ * Read the online CPUs into *@cpus, in ascending order. Read those that
+ * isolcpus="domain" isolates if @iso, the others if not. Return their
+ * count, or -1.
+ */
+static int read_online(int **cpus, int iso)
+{
+ static char buf[65536]; /* a sysfs list fits in a page */
+ char *p = buf;
+ int n = 0, max = 0;
+
+ if (read_file("/sys/devices/system/cpu/online", buf, sizeof(buf)) ||
+ strlen(buf) == sizeof(buf) - 1)
+ return -1;
+ while (*p >= '0' && *p <= '9') {
+ long lo = strtol(p, &p, 10), hi = lo;
+
+ if (*p == '-')
+ hi = strtol(p + 1, &p, 10);
+ for (; lo <= hi; lo++) {
+ if (in_cpu_list(isolated, lo) != iso)
+ continue;
+ if (n == max) {
+ int *more = realloc(*cpus, (max = 2 * max + 64) * sizeof(**cpus));
+
+ if (!more)
+ return -1;
+ *cpus = more;
+ }
+ (*cpus)[n++] = lo;
+ }
+ if (*p == ',')
+ p++;
+ }
+ return n;
+}
+
+int main(void)
+{
+ char buf[64];
+ cpu_set_t set;
+ int i;
+
+ ksft_print_header();
+ if (geteuid())
+ ksft_exit_skip("needs root\n");
+ /* No such file: nothing isolated. */
+ if (read_file("/sys/devices/system/cpu/isolated", isolated, sizeof(isolated)))
+ isolated[0] = 0;
+ ncpus = read_online(&online, 0);
+ nisol = read_online(&isol, 1);
+ if (ncpus < 0 || nisol < 0)
+ ksft_exit_fail_msg("the online CPUs cannot be read\n");
+ if (ncpus < 3)
+ ksft_exit_skip("needs 3 CPUs not isolated by isolcpus=domain\n");
+ /*
+ * Keep the tasks of the test off the isolated CPUs: a deadline task
+ * there is admitted in the root domain of the isolated CPUs. Use
+ * sched_setaffinity() for that. A change of the partitions gives the
+ * tasks of the root cpuset all its CPUs again, the isolated CPUs
+ * too. But the kernel keeps them within an affinity set by
+ * sched_setaffinity(). The children inherit this affinity.
+ */
+ CPU_ZERO(&set);
+ for (i = 0; i < ncpus; i++)
+ CPU_SET(online[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set))
+ ksft_exit_fail_msg("sched_setaffinity: %s\n", strerror(errno));
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+ ksft_exit_skip("no deadline admission control\n");
+ mv_state = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (mv_state == MAP_FAILED)
+ ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+ shm_lock = (pthread_mutex_t *)((char *)mv_state + 256);
+
+ ksft_set_plan(N_P_CASES + N_HP_CASES);
+ server_init();
+ run_p_cases();
+
+ for (i = 1; i < ncpus && hp_cpu < 0; i++) {
+ if (cpu_hotpluggable(top_cpu(i)))
+ hp_cpu = top_cpu(i);
+ }
+ if (hp_cpu < 0)
+ ksft_print_msg("no CPU hotplug: hotplug cases skipped\n");
+ for (i = 0; i < N_HP_CASES; i++) {
+ if (hp_cpu < 0 && !hp_cases[i].no_hp) {
+ ksft_test_result_skip("%s\n", hp_cases[i].name);
+ continue;
+ }
+ report(hp_cases[i].name, hp_cases[i].fn());
+ server_restore();
+ }
+ ksft_finished();
+}
diff --git a/tools/testing/selftests/sched/deadline/settings b/tools/testing/selftests/sched/deadline/settings
new file mode 100644
index 000000000000..a62d2fa1275c
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/settings
@@ -0,0 +1 @@
+timeout=600
--
2.47.3
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline
2026-11-17 14:10 ` [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
@ 2026-11-17 14:38 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 14:38 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/debug: sched_server_write_common() misses CPU
deactivation window
--
commit 6c51bc6b398f0bfbd0028d2f5672f277e0ab2db8
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Refuse dl_server debugfs writes while the rq is offline
This commit replaces the cpu_online() check in sched_server_write_common()
with an !rq->online check to reject dl_server debugfs writes while a
runqueue is offline during CPU hotplug transitions.
> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> index e6a3b516c703b..e0d6cd3bd607d 100644
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -410,7 +410,15 @@ static ssize_t sched_server_write_common(struct file *filp, const char __user *u
> return -EINVAL;
> }
>
> - if (!cpu_online(cpu_of(rq)))
> + /*
> + * Refuse the write while the runqueue is offline. This
> + * includes the windows where the CPU goes offline or comes
> + * online. The root domain then does not count the server's
> + * bandwidth, and may have no active CPU. rq->online is set
> + * under this lock when the CPU joins a root domain that
> + * counts it. cpu_active() is set before that.
> + */
> + if (!rq->online)
> return -EBUSY;
>
> update_rq_clock(rq);
[Severity: Medium]
Does checking only !rq->online still leave a window during CPU
deactivation where cpu_active(cpu) is already false while rq->online is
still set?
In sched_cpu_deactivate(), set_cpu_active(cpu, false) clears the CPU from
cpu_active_mask before synchronize_rcu(), and sched_set_rq_offline() sets
rq->online = 0 only after synchronize_rcu() completes:
CPU0 CPU1
sched_cpu_deactivate(cpu)
set_cpu_active(cpu, false)
synchronize_rcu()
sched_server_write_common()
guard(rq_lock_irqsave)(rq)
if (!rq->online) // still 1
dl_server_apply_params()
cpus = dl_bw_cpus(cpu);
dl_server_start()
sched_set_rq_offline(rq, cpu)
rq->online = 0
Because dl_bw_cpus(cpu) and dl_bw_capacity(cpu) intersect rq->rd->span
with cpu_active_mask, they already exclude the deactivating CPU before
cpuset_cpu_inactive() rebuilds the root domain. If the CPU was the only
active CPU in its root domain (for example, a single-CPU cpuset
partition), dl_bw_cpus(cpu) is 0 while rq->online is still 1. Even when
N - 1 active CPUs remain in the root domain, dl_b->total_bw still counts
all N servers while __dl_overflow(), __dl_sub(), and __dl_add() account
against N - 1 CPUs, and dl_server_start() re-enqueues the server on the
deactivating runqueue after balance_push_set(cpu, true).
Checking (!rq->online || !cpu_active(cpu_of(rq))) under rq_lock_irqsave
would close both the activation and deactivation windows.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=5
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes
2026-11-17 14:10 ` [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
@ 2026-11-17 14:50 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 14:50 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] rtmutex: premature chain walk stop in
rt_mutex_adjust_prio_chain()
--
commit 9c99e3811a419d96fcdd59dd7a4617c1edc67678
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
rtmutex: Continue the chain walk when a waiter's deadline donor changes
This commit records the task's deadline donor in struct rt_mutex_waiter
and checks it in rt_waiter_up_to_date() so that priority inheritance chain
walks continue when a blocked task's deadline donor changes without
changing its sort keys.
> diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
> index 5a9534c715b84..c78dfe9c93fc5 100644
> --- a/kernel/locking/rtmutex.c
> +++ b/kernel/locking/rtmutex.c
[ ... ]
> @@ -824,12 +841,13 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
>
> /*
> * If the waiter priority is the same as the task priority
> - * then there is no further priority adjustment necessary. If
> + * then there is no further priority adjustment necessary. The
> + * task must also pass on the same deadline donor. If
> * deadlock detection is off, we stop the chain walk. If its
> * enabled we continue, but stop the requeueing in the chain
> * walk.
> */
> - if (rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
> + if (rt_waiter_up_to_date(waiter, task)) {
> if (!detect_deadlock)
> goto out_unlock_pi;
> else
[Severity: High]
Can rt_mutex_adjust_prio_chain() still stop one lock too early at the
preceding top_waiter != task_top_pi_waiter(task) check when a task holds
two locks and its top PI waiter switches from one lock to the other?
Consider the commit message setup where O (SCHED_OTHER) holds L2, M
(SCHED_OTHER) holds L1 and L1' and blocks on L2, W (SCHED_DEADLINE,
earlier deadline) blocks on L1', and Z (SCHED_DEADLINE, later deadline)
blocks on L1, so both M and O have dl.pi_se = &W->dl. If W switches to
SCHED_OTHER via sched_setscheduler() -> rt_mutex_adjust_pi(W), the first
iteration of rt_mutex_adjust_prio_chain() demotes W's waiter on L1' in
M->pi_waiters, making Z's waiter on L1 the new task_top_pi_waiter(M) and
updating M->dl.pi_se = &Z->dl, while step [12] records L1's top waiter:
kernel/locking/rtmutex.c:rt_mutex_adjust_prio_chain() {
...
if (top_waiter) {
if (!task_has_pi_waiters(task))
goto out_unlock_pi;
if (top_waiter != task_top_pi_waiter(task)) {
if (!detect_deadlock)
goto out_unlock_pi;
...
top_waiter = rt_mutex_top_waiter(lock);
}
On the next iteration at M, top_waiter is still W's waiter on L1' while
task_top_pi_waiter(M) is Z's waiter on L1, so the walk exits at
goto out_unlock_pi before reaching rt_waiter_up_to_date(waiter, task).
O->dl.pi_se remains &W->dl, and once W unblocks and exits, O's next
replenish_dl_entity() reads W's freed task_struct (use-after-free).
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=9
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep
2026-11-17 14:10 ` [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
@ 2026-11-17 14:55 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 14:55 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: dl_task_timer() refills sleeping boosted task
from donor
--
commit 2a5083dbbf5a615c669ceffa5d4cba59162262f3
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Replenish a throttled task boosted while asleep
This commit changes dl_task_timer() to return early for a boosted task only
when it is queued on the runqueue, allowing a sleeping boosted task to fall
through and be replenished when its deadline timer fires.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 989e7e7d3f8f1..2b5a2a43addfd 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -1319,10 +1319,13 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
> }
>
> /*
> - * The task might have been boosted by someone else and might be in the
> - * boosting/deboosting path, its not throttled.
> + * A boost overrides the throttle. enqueue_task_dl() lifts it when the
> + * boost enqueues the task, or when the task wakes up boosted. So a
> + * queued task needs nothing here. A boost of a sleeping task does not
> + * enqueue it, and it can end before the wakeup. Nothing would lift the
> + * throttle then. So let a sleeping task reach the !queued case below.
> */
> - if (is_dl_boosted(dl_se))
> + if (is_dl_boosted(dl_se) && task_on_rq_queued(p))
> goto unlock;
>
> /*
[Severity: Medium]
Does letting a sleeping boosted task fall through to replenish_dl_entity()
corrupt its CBS accounting by refilling its pre-boost deficit from the
donor's reservation parameters?
A task only sets dl_throttled and arms dl_timer while unboosted, so when
dl_task_timer() fires while p is still asleep, p has never executed under
the boost and dl_se->runtime and dl_se->deadline still reflect p's own
pre-sleep reservation. However, replenish_dl_entity() reads runtime,
deadline, and period from pi_of(dl_se):
kernel/sched/deadline.c:replenish_dl_entity() {
...
if (dl_se->runtime <= 0)
dl_note_donor_refill(dl_se);
while (dl_se->runtime <= 0) {
dl_se->deadline += period;
dl_se->runtime += runtime;
}
...
}
Because dl_note_donor_refill() sets dl_se->dl_donor_refill, if the donor
times out and deboosts p before wakeup, dl_deboost() resets dl_se->deadline
and dl_se->runtime to a full new period, forgiving any runtime overrun p
owed before going to sleep.
If p instead wakes up while still boosted after overrunning its own budget,
the while loop above pays off p's overrun in multiples of the donor's
smaller runtime and period, which can push p's boosted deadline several
donor periods into the future. For a constrained-deadline task whose
dl_timer fires at dl_next_period(dl_se), adding a donor's shorter dl_period
to p's earlier dl_se->deadline can leave dl_se->deadline behind rq_clock(rq)
and falsely trigger the "sched: DL replenish lagged too much" warning. If
the donor left SCHED_DEADLINE and had its parameters cleared before
dl_task_timer() runs, the !runtime branch in replenish_dl_entity() skips
dl_note_donor_refill() and the deadline update because dl_se->dl_deadline is
non-zero, leaving p's expired deadline in the past after dl_deboost().
Finally, if p changed policy while sleeping and boosted, refilling from the
donor before 0-lag time leaves donor-derived runtime and deadline on p when
rt_mutex_setprio() drops the boost via dl_drop_boost().
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=12
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class
2026-11-17 14:10 ` [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
@ 2026-11-17 14:55 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 14:55 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: rt_mutex_setprio() skips dl_deboost() on
donor change
- [Medium] sched/deadline: donor runtime kept when leaving
dl_sched_class
- [Low] sched/deadline: missing dl_note_donor_refill() on cleared donor
in replenish_dl_entity()
--
commit bb738227d35bdf5cba0afaf4e41cb6707e38d0c4
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Reset the donor's runtime at a deboost in the class
This commit adds a dl_donor_refill flag to record when a boosted task has its
runtime replenished from a donor's reservation, and starts a new period from
the task's own parameters when the boost ends in the deadline class.
[ ... ]
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 5ed660f099a55..a22df8b43a1a7 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7834,16 +7834,26 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
> p->dl.pi_se = pi_task->dl.pi_se;
> scope->flags |= ENQUEUE_REPLENISH;
> } else {
> - p->dl.pi_se = &p->dl;
> + dl_deboost(&p->dl);
> + /*
> + * dl_deboost() can start a new period. Its
> + * deadline can then be after the top waiter's.
> + * Boost the task again then.
> + */
> + if (pi_task && dl_prio(pi_task->prio) &&
> + dl_entity_preempt(&pi_task->dl, &p->dl)) {
> + p->dl.pi_se = pi_task->dl.pi_se;
> + scope->flags |= ENQUEUE_REPLENISH;
> + }
> }
[Severity: Medium]
Can a SCHED_DEADLINE task skip dl_deboost(&p->dl) here when switching from
one deadline donor to another?
In rt_mutex_setprio(), the outer if statement tests
dl_entity_preempt(&pi_task->dl, &p->dl) before calling dl_deboost(&p->dl).
When p was refilled from a previous donor W1, p->dl.deadline holds W1's
postponed deadline rather than p's own reservation deadline. If p unlocks
W1's lock while another deadline waiter W2 is queued on a lock held by p, and
W2->dl.deadline is earlier than W1's postponed deadline, the outer check
takes the first branch and skips dl_deboost(&p->dl).
Because p->dl.dl_deadline is non-zero and p->dl.runtime is still positive
from W1's refill, ENQUEUE_REPLENISH in replenish_dl_entity() does not reset
p->dl.runtime or p->dl.deadline either. p then keeps W1's leftover runtime
and postponed deadline, even if p's own new-period deadline would have been
earlier than W2->dl.deadline.
Calling dl_deboost(&p->dl) unconditionally when dl_prio(p->normal_prio)
before testing whether pi_task preempts &p->dl would compare pi_task against
p's own reservation.
> } else if (rt_prio(prio)) {
> if (dl_prio(oldprio))
> - p->dl.pi_se = &p->dl;
> + dl_drop_boost(&p->dl);
> if (oldprio < prio)
> scope->flags |= ENQUEUE_HEAD;
> } else {
> if (dl_prio(oldprio))
> - p->dl.pi_se = &p->dl;
> + dl_drop_boost(&p->dl);
> if (rt_prio(oldprio))
> p->rt.timeout = 0;
> }
[Severity: Medium]
Does a task that still has SCHED_DEADLINE parameters (p->dl.dl_runtime != 0)
keep the donor's refilled runtime and deadline when leaving the deadline
class at deboost?
A task can have p->dl.dl_runtime != 0 and p->dl.dl_donor_refill == 1 while
leaving dl_sched_class (for example, if it changed its policy away from
SCHED_DEADLINE while boosted and then unlocks, or if it left SCHED_DEADLINE
before its 0-lag time and was boosted and refilled before inactive_timer
fired). When it leaves dl_sched_class, sched_change_begin() runs
switched_from_dl() before dl_drop_boost(&p->dl) clears dl_donor_refill:
kernel/sched/deadline.c:switched_from_dl() {
if (task_on_rq_queued(p) && p->dl.dl_runtime)
task_non_contending(&p->dl, false);
...
if (!p->dl.dl_runtime) {
p->dl.deadline = 0;
p->dl.runtime = 0;
}
}
Because p->dl.dl_runtime is non-zero, switched_from_dl() passes the donor's
refilled runtime and deadline to task_non_contending(), which mixes them with
p's own dl_period and dl_runtime to compute zerolag_time. If the donor's
remaining runtime makes zerolag_time negative, task_non_contending() clears
p's parameters immediately and lets p re-enter SCHED_DEADLINE with a fresh
budget in the same period; otherwise, p retains the donor's runtime and
deadline if it returns to SCHED_DEADLINE before inactive_timer fires.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index d6d2657dcddd6..989e7e7d3f8f1 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -832,6 +869,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
> if (!dl_se->dl_deadline) {
> dl_se->deadline = rq_clock(rq);
> dl_se->runtime = 0;
> + dl_note_donor_refill(dl_se);
> }
> if (dl_se->runtime < 0 || dl_se->dl_yielded)
> dl_se->runtime = 0;
[Severity: Low]
In this cleared-donor fallback (!runtime || !deadline || !period), should
dl_note_donor_refill(dl_se) also be called when dl_se->dl_deadline is
non-zero?
If a boosted SCHED_DEADLINE task hits this path when its runtime is exhausted
or after sched_yield(), replenish_dl_entity() zeroes dl_se->runtime and
clears dl_se->dl_throttled without setting dl_donor_refill. A subsequent
dl_deboost() then leaves the task in the class with runtime == 0,
dl_throttled == 0, and its old deadline.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=11
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters
2026-11-17 14:10 ` [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
@ 2026-11-17 14:55 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 14:55 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: donor runtime kept in switched_from_dl() when
dl_runtime != 0
--
commit c6cd5264704d8a0de90183dbd3576cbeaa15e19c
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Reset the deadline and the runtime with the parameters
This commit resets a deadline entity's dynamic deadline and runtime in
__dl_clear_params() and in switched_from_dl() when a boosted task with no
parameters of its own leaves SCHED_DEADLINE, so that its next entry into
the class starts a new period.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 71b1c155c2424..d6d2657dcddd6 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3569,6 +3569,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
> if (task_on_rq_queued(p) && p->dl.dl_runtime)
> task_non_contending(&p->dl, false);
>
> + /*
> + * A task with no parameters of its own was boosted into the class.
> + * Its deadline and runtime are the donor's. Reset them, so that its
> + * next entry into the class starts a new period.
> + */
> + if (!p->dl.dl_runtime) {
> + p->dl.deadline = 0;
> + p->dl.runtime = 0;
> + }
[Severity: Medium]
Can a task deboosting via switched_from_dl() still retain a donor's
deadline and runtime when p->dl.dl_runtime is non-zero?
If a SCHED_DEADLINE task holding an rt_mutex contended by a deadline
waiter changes policy to SCHED_OTHER, __setscheduler_dl_pi() keeps it
boosted in dl_sched_class with its non-zero p->dl.dl_runtime while setting
p->dl.pi_se to the donor. When its runtime is replenished while boosted,
replenish_dl_new_period() refills p->dl.deadline and p->dl.runtime from
the donor's reservation.
At unlock, rt_mutex_setprio() invokes switched_from_dl() before calling
dl_drop_boost(&p->dl). Because p->dl.dl_runtime is non-zero,
switched_from_dl() skips resetting p->dl.deadline and p->dl.runtime, and
task_non_contending() computes zerolag_time by mixing the donor's dynamic
deadline and runtime with p's static dl_period and dl_runtime. If the
donor's runtime makes zerolag_time negative, task_non_contending() clears
p's bandwidth and parameters before its true 0-lag time; otherwise, p
keeps the donor's runtime and can consume it on top of its own reservation
if it re-enters SCHED_DEADLINE before the inactive timer fires.
Commits "sched/deadline: Reset the donor's runtime at a deboost in the
class" and "sched/deadline: Check the runtime of a task whose parameters
change" later in this series handle deboosts that stay in dl_sched_class,
but do not cover leaving dl_sched_class via switched_from_dl() with
non-zero p->dl.dl_runtime.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=10
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth
2026-11-17 14:10 ` [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
@ 2026-11-17 15:14 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:14 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: __sched_setscheduler() checks task_rq(p) root
domain
--
commit 889f3e0f3403117f479bd558c00411593ac35a7e
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Record the root domain holding a task's bandwidth
This commit replaces dl_departed_cpu with dl_bw_cpu in struct sched_dl_entity
to record a CPU of the root domain that accounts for a deadline task's
bandwidth across admissions, moves, rebuilds, and releases.
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 8e8c62c574d55..01fbbd02cf8d0 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -4213,6 +4280,23 @@ int sched_dl_overflow(struct task_struct *p, int policy,
> if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
> return 0;
>
> + /*
> + * Change the bandwidth of a deadline task where it is counted: in
> + * the root domain of the CPU recorded with it. A task moved asleep
> + * has not run there yet, so the CPU it is on can be elsewhere. No
> + * bandwidth is counted if a rebuild dropped it, or if the task
> + * enters the class. A runtime below about a millionth of the period
> + * gives a bandwidth of 0. It counts for nothing, and no move updates
> + * its record. In these three cases, the root domain of the CPU the
> + * task is on takes the new bandwidth. The caller holds cpuset_mutex,
> + * or dl_accounting_mutex without cpusets, so no rebuild runs
> + * meanwhile.
> + */
> + counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
> + p->dl.dl_bw_gen == dl_bw_gen;
> + cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
> + dl_b = dl_bw_of(cpu);
[ ... ]
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 4c013caa24713..306cd8d32173b 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -1185,6 +1185,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>
> int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
> {
> + unsigned int cpu;
> +
> /*
> * If the task isn't a deadline task or admission control is
> * disabled then we don't care about affinity changes.
> @@ -1204,9 +1206,16 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
> * if admission test is enabled, we only admit -deadline
> * tasks allowed to run on all the CPUs in the task's
> * root_domain.
> + *
> + * That is the root domain that counts the bandwidth of the task:
> + * the one of the CPU recorded with it. A task moved asleep to
> + * another cpuset is still on a CPU of the root domain it left. A
> + * bandwidth of 0 is counted nowhere, and no move updates its
> + * record: use the CPU the task is on.
> */
> + cpu = READ_ONCE(p->dl.dl_bw) ? dl_task_root_domain_cpu(p) : task_cpu(p);
> guard(rcu)();
> - if (!cpumask_subset(task_rq(p)->rd->span, mask))
> + if (!cpumask_subset(cpu_rq(cpu)->rd->span, mask))
> return -EBUSY;
>
> return 0;
[Severity: Medium]
Should __sched_setscheduler() also use the root domain recorded in
p->dl.dl_bw_cpu rather than rq->rd when validating a deadline task's
root domain span and dl_bw.bw?
kernel/sched/syscalls.c:__sched_setscheduler() {
...
if (dl_bandwidth_enabled() && dl_policy(policy) &&
!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
cpumask_t *span = rq->rd->span;
if (!cpumask_subset(span, &p->cpus_mask) ||
rq->rd->dl_bw.bw == 0) {
retval = -EPERM;
goto unlock;
}
}
...
}
When a sleeping SCHED_DEADLINE task is moved to another exclusive cpuset
partition (or its CPU goes offline into the default root domain),
p->cpus_mask and p->dl.dl_bw_cpu reflect the root domain holding its
bandwidth, while rq (task_rq(p)) remains in the old root domain. Because
rq->rd->span is not a subset of p->cpus_mask, __sched_setscheduler()
rejects sched_setattr() parameter updates on that task with -EPERM before
reaching the moved-sleeper handling in sched_dl_overflow().
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=18
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach
2026-11-17 14:10 ` [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
@ 2026-11-17 15:18 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:18 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched: stale cpumask before cpuset_lock_no_attach() in
__sched_setaffinity()
--
commit b29e127976b751a796fee47f2183b160f8e83a4d
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched: Make deadline policy and affinity changes wait for a cpuset attach
This commit introduces cpuset_lock_no_attach() and uses it in
__sched_setscheduler() and sched_setaffinity() so that deadline policy and
affinity changes wait until any in-progress cpuset task attach finishes.
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 4f47b17151ed9..4aa3ccbff320f 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
[ ... ]
> @@ -1271,9 +1276,21 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
> /*
> * The cpuset mask must win here, as it does when
> * cpuset_update_tasks_cpumask() sets it on a deadline task.
> + *
> + * A sched_setattr() may have made @p a deadline task since
> + * the first call. SCA_DL_NOWAIT would then skip this second
> + * call. Wait for a cpuset attach in progress instead, as
> + * sched_setaffinity() does for a deadline task.
> */
> ctx->flags &= ~SCA_DL_CHECK;
> - __set_cpus_allowed_ptr(p, ctx);
> + if (ctx->flags & SCA_DL_NOWAIT) {
> + ctx->flags &= ~SCA_DL_NOWAIT;
> + cpuset_lock_no_attach();
> + __set_cpus_allowed_ptr(p, ctx);
> + cpuset_unlock();
> + } else {
> + __set_cpus_allowed_ptr(p, ctx);
> + }
> retval = -EINVAL;
> }
[Severity: Medium]
Can this second __set_cpus_allowed_ptr() call apply a stale new_mask after
cpuset_lock_no_attach() sleeps?
In __sched_setaffinity(), when the concurrent cpuset update recovery path
(!cpumask_subset(new_mask, cpus_allowed)) is entered with SCA_DL_NOWAIT set,
cpuset_cpus_allowed(p, cpus_allowed) and new_mask are computed before
cpuset_lock_no_attach() is called:
kernel/sched/syscalls.c:__sched_setaffinity() {
...
cpuset_cpus_allowed(p, cpus_allowed);
if (!cpumask_subset(new_mask, cpus_allowed)) {
cpumask_copy(new_mask, cpus_allowed);
if (unlikely((ctx->flags & SCA_USER) && ctx->user_mask)) {
bool empty = !cpumask_and(new_mask, new_mask,
ctx->user_mask);
if (empty)
cpumask_copy(new_mask, cpus_allowed);
}
...
}
If p becomes a SCHED_DEADLINE task after the first __set_cpus_allowed_ptr()
call and a cpuset attach or update runs before cpuset_lock_no_attach()
acquires cpuset_mutex, new_mask is not recomputed under cpuset_mutex:
CPU0 CPU1
sched_setaffinity(p)
!task_has_dl_policy(p)
__sched_setaffinity(p, &ac)
__set_cpus_allowed_ptr()
cpuset_cpus_allowed()
cpumask_copy(new_mask, ...)
sched_setattr(p, SCHED_DEADLINE)
cgroup_procs_write() -> cpuset_attach()
moves p and DL bw to new partition P
cpuset_lock_no_attach()
__set_cpus_allowed_ptr()
sets p->cpus_mask to stale new_mask <-- wrong CPUs for p's DL bw
Because SCA_DL_CHECK is cleared before the second __set_cpus_allowed_ptr()
call, p's affinity is overwritten with the pre-attach cpuset mask while its
deadline bandwidth remains accounted in P's root domain.
In addition, if css_set_move_task() moves p to the destination cpuset before
cpuset_cpus_allowed() is read while attach_ctx.in_progress is still set,
waiting in cpuset_lock_no_attach() delays restoring p->user_cpus_ptr from
ctx->user_mask until after cpuset_attach_task() has already run using the
un-restored p->user_cpus_ptr from the first __set_cpus_allowed_ptr() call,
and then applies new_mask intersected with the old ctx->user_mask without
SCA_DL_CHECK, narrowing the deadline task's affinity below its root domain
span.
Recomputing cpus_allowed and new_mask after acquiring cpuset_lock_no_attach()
(or holding cpuset_lock_no_attach() before reading cpus_allowed on the
recovery path) would ensure the mask matches the post-attach cpuset state.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=23
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips
2026-11-17 14:10 ` [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
@ 2026-11-17 15:23 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:23 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] selftests/sched: 64 KiB FILLER_STACK fails
pthread_attr_setstacksize()
- [Low] selftests/sched: missing sem_destroy() and pipe leak in
deadline_switch_test
--
commit 84b080b4abc565be69cf0d196565fbf11ac9aa91
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
selftests/sched: Add a test of SCHED_DEADLINE policy round trips
This commit adds a new sched/deadline selftest covering SCHED_DEADLINE
policy round trips to SCHED_OTHER and back, as well as sysrq-n
normalization with deadline tasks and priority-inheritance lock owners.
> diff --git a/tools/testing/selftests/sched/deadline/deadline_switch_test.c b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
> new file mode 100644
> index 0000000000000..daf10f81cb968
> --- /dev/null
> +++ b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
[ ... ]
> +static int test_sysrq_boosted_owner(int normalized)
> +{
[ ... ]
> + if (sem_init(&owner_locked, 0, 0) || sem_init(&owner_go, 0, 0)) {
> + ksft_print_msg("sem_init: %s\n", strerror(errno));
> + ret = KSFT_FAIL;
> + goto out;
> + }
[ ... ]
> +out:
> + close(fd);
> + pthread_mutex_destroy(&pi_lock);
> + return ret;
> +}
> +
> +#define FILLERS 2048
> +#define FILLER_STACK (64 * 1024)
[ ... ]
> +static int test_sysrq_running_owner(int normalized)
> +{
[ ... ]
> + fill = calloc(FILLERS, sizeof(*fill));
> + if (!fill || pipe(filler_pipe) || sem_init(&owner_locked, 0, 0) ||
> + sem_init(&filler_ready, 0, 0) || sem_init(&waiter_go, 0, 0) ||
> + pthread_attr_init(&fa) || pthread_attr_setstacksize(&fa, FILLER_STACK)) {
> + ksft_print_msg("setup: %s\n", strerror(errno));
> + free(fill);
> + close(fd);
> + pthread_mutex_destroy(&pi_lock);
> + return KSFT_FAIL;
> + }
[Severity: Medium]
Does pthread_attr_setstacksize(&fa, FILLER_STACK) fail with EINVAL on
architectures where PTHREAD_STACK_MIN is larger than 64 KiB, such as arm64
and powerpc64 (128 KiB)? That causes test_sysrq_running_owner() to fail
during setup on those architectures.
Also, pthread_attr_init() and pthread_attr_setstacksize() return an error
number directly without setting errno, so strerror(errno) reports a stale
errno (typically "Success"), and this error path returns without closing
filler_pipe or destroying fa.
[ ... ]
> +out:
> + close(filler_pipe[1]);
> + for (i = 0; i < started; i++)
> + if (join(fill[i]))
> + ret = KSFT_FAIL;
> + close(filler_pipe[0]);
> + sched_setaffinity(0, sizeof(old), &old);
> + pthread_attr_destroy(&fa);
> + free(fill);
> + close(fd);
> + pthread_mutex_destroy(&pi_lock);
> + return ret;
> +}
[Severity: Low]
Should test_sysrq_boosted_owner() and test_sysrq_running_owner() call
sem_destroy() on owner_locked, owner_go, filler_ready, and waiter_go during
cleanup? Without sem_destroy(&owner_locked) in test_sysrq_boosted_owner(),
test_sysrq_running_owner() calls sem_init() on an already initialized
owner_locked semaphore, which is undefined behavior in POSIX.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=38
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to the root domain it runs in
2026-11-17 14:10 ` [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
@ 2026-11-17 15:29 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:29 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: dl_bw_rebuilding() drops bw moves during
rebuild task walk
- [Medium] sched/deadline: migrate_task_rq_dl() drops bw move for
mutex-blocked tasks
- [Medium] sched/deadline: dl_task_move_bw() moves departed bw of
PI-boosted tasks
--
commit 00a2592b3e4a7ff7a78ebc503a680dacf59e846b
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Move a task's bandwidth to the root domain it runs in
This commit moves a deadline task's bandwidth to the root domain of its
new CPU in migrate_task_rq_dl() and task_woken_dl() when it differs from
the root domain that currently counts it, while suppressing moves during
sched domain rebuilds and proxy execution migrations.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 9ff79141c6306..6c6a4d9cd0548 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -2930,11 +2968,86 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
> return cpu;
> }
>
> -static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
> +/*
> + * Return true if the root domain of @cpu may not count the bandwidth of
> + * @p. Either another root domain counts it, or a rebuild left @p out.
> + * Test without dl_departed_lock, so that a move inside a root domain
> + * takes no lock. dl_task_move_bw() tests again under it.
> + *
> + * Return false if @cpu is not active. A migration-disabled task can wake
> + * up on a CPU that goes offline, after it joined the default root domain.
> + * Keep the bandwidth where it is counted. balance_push() moves the task
> + * off that CPU once it enables migration, or its next wakeup does once
> + * the CPU is offline. That migration moves the bandwidth if needed.
> + *
> + * Return false while the sched domains are rebuilt. The CPUs then move
> + * to their new root domains one by one. That alone can make the two root
> + * domains differ. The rebuild of the accounting
> + * that follows counts @p.
> + */
> +static bool dl_task_bw_elsewhere(struct task_struct *p, int cpu)
> +{
> + int bw_cpu = READ_ONCE(p->dl.dl_bw_cpu);
> +
> + if (bw_cpu < 0 || !p->dl.dl_bw || !cpu_active(cpu) || dl_bw_rebuilding())
> + return false;
> + return READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen) ||
> + cpu_rq(bw_cpu)->rd != cpu_rq(cpu)->rd;
> +}
> +
> +/*
> + * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
> + * also if a rebuild left @p out. Add it even without room, since @p runs
> + * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
> + */
> +static void dl_task_move_bw(struct task_struct *p, int new_cpu)
> +{
> + struct dl_bw *dl_b;
> + int cpu;
> +
> + if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
> + return;
> +
> + guard(raw_spinlock)(&dl_departed_lock);
> + cpu = p->dl.dl_bw_cpu;
> + if (cpu < 0 || dl_bw_rebuilding())
> + return;
> + if (p->dl.dl_bw_gen == dl_bw_gen) {
> + if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
> + return;
> + dl_b = dl_bw_of(cpu);
> + raw_spin_lock(&dl_b->lock);
> + __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
> + raw_spin_unlock(&dl_b->lock);
> + }
> + dl_b = dl_bw_of(new_cpu);
> + raw_spin_lock(&dl_b->lock);
> + __dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
> + raw_spin_unlock(&dl_b->lock);
> + p->dl.dl_bw_cpu = new_cpu;
> + p->dl.dl_bw_gen = dl_bw_gen;
> +}
[Severity: Medium]
Can dl_task_move_bw() move the departing bandwidth of a non-deadline task
that was PI-boosted back into the deadline class?
When a task leaves SCHED_DEADLINE before its 0-lag time, switched_from_dl()
arms inactive_timer and leaves p->dl.dl_bw and p->dl.dl_bw_cpu set in the
original root domain. As a non-deadline task (!task_has_dl_policy(p)), it
can move to another root domain without admission control, and if it is
then PI-boosted by a deadline waiter before its 0-lag time,
switched_to_dl() cancels the inactive timer and takes it off
dl_departed_list while keeping the original dl_bw_cpu recorded.
Because neither dl_task_bw_elsewhere() nor dl_task_move_bw() checks
task_has_dl_policy(p), waking or migrating the boosted task in the new root
domain subtracts p->dl.dl_bw from the original root domain before its
0-lag time and adds that unreserved bandwidth to the new root domain.
> +
> +static void migrate_task_rq_dl(struct task_struct *p, int new_cpu)
> {
> struct rq_flags rf;
> struct rq *rq;
>
> + /*
> + * The new CPU can be in another root domain than the one that counts
> + * the bandwidth. A mask can hold CPUs of two root domains, and
> + * select_fallback_rq() can widen it. Move the bandwidth there, as
> + * dl_task_offline_migration() does for a throttled task.
> + *
> + * Not for a task that is blocked on a mutex and only queued for
> + * proxy execution. It does not run on the new CPU, so its bandwidth
> + * stays in its root domain. If its wake-up then picks a CPU of another
> + * root domain, the bandwidth moves at that time.
> + */
> + if (unlikely(dl_task_bw_elsewhere(p, new_cpu)) &&
> + !(p->is_blocked && task_on_rq_migrating(p)))
> + dl_task_move_bw(p, new_cpu);
[Severity: Medium]
Does skipping dl_task_move_bw() when p->is_blocked && task_on_rq_migrating(p)
also drop the bandwidth move for normal queued migrations of mutex-blocked
tasks under CONFIG_SCHED_PROXY_EXEC?
When a mutex-blocked task (p->is_blocked == 1) is migrated across root
domains by move_queued_task() or move_queued_task_locked() (for example
from __balance_push_cpu_stop() during CPU hotplug or push_dl_task()),
deactivate_task() sets p->on_rq = TASK_ON_RQ_MIGRATING and set_task_cpu()
updates both p->cpu and p->wake_cpu to new_cpu:
kernel/sched/core.c:move_queued_task() {
...
deactivate_task(rq, p, DEQUEUE_NOCLOCK);
set_task_cpu(p, new_cpu);
...
activate_task(rq, p, 0);
}
When the mutex is unlocked, try_to_wake_up() sees p->on_rq == TASK_ON_RQ_QUEUED
and takes ttwu_runnable():
kernel/sched/core.c:ttwu_runnable() {
...
if (p->is_blocked) {
if (p->se.sched_delayed)
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
if (proxy_needs_return(rq, p))
return 0;
}
...
ttwu_do_wakeup(p);
return 1;
}
Because task_cpu(p) == p->wake_cpu, proxy_needs_return() returns false and
ttwu_runnable() finishes the wakeup in place without calling
migrate_task_rq_dl() or task_woken_dl(), leaving p's bandwidth counted in
the old root domain.
[ ... ]
> @@ -4068,8 +4192,10 @@ void dl_rebuild_rd_accounting(void)
> * a resume run one for each CPU. A task gets a bandwidth under the
> * locks this rebuild holds, so the count cannot leave 0 meanwhile.
> */
> - if (!atomic_read(&dl_nr_bw_tasks))
> + if (!atomic_read(&dl_nr_bw_tasks)) {
> + dl_bw_rebuild_end();
> return;
> + }
>
> read_lock(&tasklist_lock);
> for_each_process_thread(g, p) {
> @@ -4083,6 +4209,7 @@ void dl_rebuild_rd_accounting(void)
> dl_add_task_root_domain(p);
> }
> read_unlock(&tasklist_lock);
> + dl_bw_rebuild_end();
> }
[Severity: Medium]
Can keeping dl_bw_rebuilding() true until the end of the tasklist walk
drop bandwidth moves for tasks that dl_add_task_root_domain() has already
visited?
During CPU deactivation, tasks are woken via select_fallback_rq() or pushed
by __balance_push_cpu_stop() while dl_rebuild_rd_accounting() walks
for_each_process_thread() under read_lock(&tasklist_lock):
CPU0 CPU1
partition_sched_domains_locked()
dl_bw_rebuild_begin()
...
dl_rebuild_rd_accounting()
dl_clear_root_domains()
read_lock(&tasklist_lock)
for_each_process_thread(g, p)
dl_add_task_root_domain(p)
/* counts p in RD_A (or skips
* p if cpu >= nr_cpu_ids) */
try_to_wake_up(p)
select_fallback_rq() -> cpu in RD_B
set_task_cpu(p, new_cpu)
migrate_task_rq_dl(p, new_cpu)
dl_task_bw_elsewhere()
dl_bw_rebuilding() == true
--> returns false (bw not moved)
/* still visiting later tasks */
read_unlock(&tasklist_lock)
dl_bw_rebuild_end()
Once dl_add_task_root_domain(p) releases p->pi_lock, p can wake up or
migrate to an active CPU in another root domain before the walk finishes.
Because dl_bw_rebuilding() is still true, migrate_task_rq_dl() and
task_woken_dl() ignore the move, leaving p's bandwidth counted in RD_A (or
uncounted if dl_task_rebuild_cpu(p) returned nr_cpu_ids).
In addition, when dl_task_bw_elsewhere() runs under rq->__lock without
p->pi_lock, it reads bw_cpu = READ_ONCE(p->dl.dl_bw_cpu) before checking
dl_bw_rebuilding() and READ_ONCE(dl_bw_gen), whereas
dl_add_task_root_domain() updates p->dl.dl_bw_cpu and p->dl.dl_bw_gen under
p->pi_lock and dl_departed_lock, so dl_task_bw_elsewhere() can compare
cpu_rq(bw_cpu)->rd using a stale bw_cpu read from before the rebuild ended.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=27
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in
2026-11-17 14:10 ` [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
@ 2026-11-17 15:31 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:31 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: dl_task_rebuild_cpu() uses recorded CPU
outside task cpuset
- [Medium] sched/deadline: dl_task_rebuild_cpu() fallback reads stale
HK_TYPE_DOMAIN
--
commit 5f53c9545c6d42e90c32f229213e080ced9acb3f
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Make a rebuild count a task in the root domain it runs in
This commit introduces dl_task_rebuild_cpu() to select an active CPU in
the root domain where a deadline task runs during root domain bandwidth
rebuilds, and leaves out tasks with no active CPU until offline migration
or a later rebuild places them.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index a9b8bd019d4ba..9ff79141c6306 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -3820,6 +3826,60 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
> cpuset_cpus_allowed_locked(p, cpus);
> }
>
> +/*
> + * Return a CPU of the root domain where a rebuild of the accounting should
> + * count the bandwidth of @p, or nr_cpu_ids if there is none.
> + *
> + * Prefer the CPU @p is on, if it is active and in the mask of @p. The
> + * task runs, or wakes up, in the root domain of that CPU. A mask can hold
> + * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
> + * isolcpus="domain". When every CPU of a mask goes offline,
> + * select_fallback_rq() can widen it to both. It moves no bandwidth.
> + * Prefer that CPU also if @p is queued there, out of its mask, and the
> + * CPU is the one recorded. Without admission control,
> + * dl_task_offline_migration() can put a throttled task and its bandwidth
> + * on any active CPU. The task runs there until it blocks.
> + *
> + * Else prefer the CPU recorded with the bandwidth, if it is active and in
> + * the mask. It is in the root domain that admitted the bandwidth, or that
> + * its last move put it in. A task moved asleep is still on a CPU of the
> + * root domain it left, out of its mask. Else take the first active CPU of
> + * the mask, a housekeeping one if there is one. A task on isolated CPUs
> + * that all went offline can have no active CPU at all.
> + */
> +static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
> +{
> + struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
> + int recorded = p->dl.dl_bw_cpu;
> + unsigned int cpu;
> +
> + /*
> + * Look at the mask of the task first. A change of the cpusets sets
> + * the mask of each task before its rebuild. The housekeeping mask
> + * that dl_get_task_effective_cpus() reads is updated after it. If no
> + * CPU of the mask is active, take the CPU recorded if it is active.
> + * dl_task_offline_migration() moved the task there, or an earlier
> + * rebuild counted it there. Else ask the cpuset.
> + */
> + cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
> + cpu = task_cpu(p);
> + if (cpumask_test_cpu(cpu, msk) ||
> + (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
> + return cpu;
> + if (cpumask_empty(msk)) {
> + if (recorded >= 0 && cpu_active(recorded))
> + return recorded;
[Severity: Medium]
Can returning recorded here without checking the task's cpuset charge the
bandwidth to a newly created partition that no longer contains the task?
Suppose a sleeping task in top_cpuset has user_cpus_ptr restricted via
sched_setaffinity() to CPUs 0 and 1, and both go offline while CPUs 2 and
3 remain online. The hotplug rebuild sees an empty msk, falls back to
dl_get_task_effective_cpus(), and records CPU 2 in p->dl.dl_bw_cpu.
If CPU 2 is then moved into a new cpuset partition root,
cpuset_update_tasks_cpumask() updates top_cpuset tasks with
possible_mask & ~subpartitions_cpus. Because __set_cpus_allowed_ptr()
intersects that mask with p->user_cpus_ptr ({0, 1}), p->cpus_mask stays
{0, 1} and msk remains empty. This check then sees CPU 2 is still active
and returns it, counting the top_cpuset task's bandwidth in the new
partition on CPU 2 instead of top_cpuset on CPU 3.
> + dl_get_task_effective_cpus(p, msk);
> + cpumask_and(msk, msk, cpu_active_mask);
> + } else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
> + return recorded;
> + }
> + cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> + if (cpu < nr_cpu_ids)
> + return cpu;
> + return cpumask_first(msk);
> +}
[Severity: Medium]
Does calling dl_get_task_effective_cpus() here read a stale
HK_TYPE_DOMAIN mask during cpuset partition transitions?
dl_get_task_effective_cpus() checks housekeeping_cpumask(HK_TYPE_DOMAIN):
kernel/sched/deadline.c:dl_get_task_effective_cpus() {
hk_msk = housekeeping_cpumask(HK_TYPE_DOMAIN);
if (housekeeping_enabled(HK_TYPE_DOMAIN)) {
if (!cpumask_intersects(p->cpus_ptr, hk_msk)) {
cpumask_andnot(cpus, cpu_active_mask, hk_msk);
return;
}
}
}
HK_TYPE_DOMAIN excludes both boot-time isolcpus=domain CPUs and runtime
isolated cpuset partitions, and cpuset_update_sd_hk_unlock() rebuilds
root domain accounting before housekeeping_update() refreshes
HK_TYPE_DOMAIN. If a task is confined to an offline isolcpus=domain CPU
(so msk is empty and recorded is inactive) when a runtime isolated
partition on another CPU transitions to root or member,
cpu_active_mask & ~hk_msk still includes that partition's CPU.
dl_add_task_root_domain() will then charge the task's bandwidth to that
root domain and record the CPU in p->dl.dl_bw_cpu.
Should dl_get_task_effective_cpus() use HK_TYPE_DOMAIN_BOOT to match
dl_task_rebuild_cpu()?
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=26
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth
2026-11-17 14:10 ` [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
@ 2026-11-17 15:32 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:32 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: missing wake_cpu update in
__set_cpus_allowed_ptr_locked()
- [Medium] sched/deadline: wrong root domain in
dl_task_offline_migration() on rebuild
- [Medium] sched/deadline: missing READ_ONCE() on dl_bw_cpu in
dl_task_rebuild_cpu()
--
commit fecf231094845bbb7da45c72106b5c90fffb123d
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Keep a task in the root domain that counts its bandwidth
This commit adds dl_task_bw_span() and updates CPU selection paths for
SCHED_DEADLINE tasks to prefer CPUs in the root domain that accounts for
the task's bandwidth.
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index e9f80407f75be..dc270a23179d5 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3264,6 +3264,23 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
> dest_cpu = ctx->dl_bw_cpu;
> if (!task_on_rq_queued(p))
> p->wake_cpu = dest_cpu;
> + } else {
> + const struct cpumask *span = dl_task_bw_span(p);
> + unsigned int cpu = nr_cpu_ids;
> +
> + /*
> + * No room was made for @p in another root domain. Keep @p
> + * in the root domain that counts its bandwidth, if the new
> + * mask holds one of its CPUs.
> + */
> + if (span && !cpumask_test_cpu(dest_cpu, span))
> + cpu = cpumask_first_and_and(cpu_valid_mask,
> + ctx->new_mask, span);
> + if (cpu < nr_cpu_ids) {
> + dest_cpu = cpu;
> + if (!task_on_rq_queued(p))
> + p->wake_cpu = dest_cpu;
> + }
> }
[Severity: Medium]
Does this leave p->wake_cpu stale when cpumask_any_and_distribute()
already picked a dest_cpu inside span?
When cpumask_test_cpu(dest_cpu, span) is true, cpu stays at nr_cpu_ids and
p->wake_cpu is not updated for a sleeping task. Because affine_move_task()
does not move sleeping tasks, p->wake_cpu remains on the old CPU (such as
an offlining housekeeping CPU during cpuset_hotplug_update_tasks()). If p
then wakes up during the subsequent sched domain rebuild while
dl_task_bw_span(p) returns NULL, select_task_rq() and select_fallback_rq()
can place p on an isolated CPU in def_root_domain and move its bandwidth
without admission control.
Updating p->wake_cpu whenever span is non-NULL and dest_cpu is in span
(and !task_on_rq_queued(p)) would avoid this.
[ ... ]
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 6c6a4d9cd0548..e5f77d3c5fa03 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -901,18 +903,36 @@ static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *rq);
>
> static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p)
> {
> + const struct cpumask *span;
> struct rq *later_rq = NULL;
> struct dl_bw *dl_b;
>
> later_rq = find_lock_later_rq(p, rq);
> + /*
> + * find_lock_later_rq() looks in the root domain of @rq. Once @rq is
> + * offline, a rebuild attaches it to def_root_domain. That root domain
> + * may not count @p. Do not take @p out of the root domain that counts
> + * it while @p has an active CPU there.
> + */
> + span = dl_task_bw_span(p);
> + if (later_rq && span && !cpumask_test_cpu(later_rq->cpu, span) &&
> + cpumask_first_and_and(cpu_active_mask, p->cpus_ptr, span) < nr_cpu_ids) {
> + double_unlock_balance(rq, later_rq);
> + later_rq = NULL;
> + }
> if (!later_rq) {
> int cpu;
>
> /*
> * If we cannot preempt any rq, fall back to pick any
> - * online CPU:
> + * online CPU. Try the root domain that counts @p first.
> */
> - cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
> + cpu = nr_cpu_ids;
> + if (span)
> + cpu = cpumask_first_and_and(cpu_active_mask,
> + p->cpus_ptr, span);
> + if (cpu >= nr_cpu_ids)
> + cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
[Severity: Medium]
Can dl_task_offline_migration() still move a throttled task and its
bandwidth into def_root_domain when dl_task_timer() fires during a sched
domain rebuild?
CPU0 CPU1 (<hardirq>)
sched_cpu_deactivate(H)
sched_set_rq_offline(rq_H)
cpuset_cpu_inactive(H)
partition_sched_domains_locked()
dl_bw_rebuild_begin()
detach_destroy_domains()
cpu_attach_domain(def_rd, H)
dl_task_timer(p)
dl_task_offline_migration(rq_H, p)
find_lock_later_rq() -> iso_rq
dl_task_bw_span(p) -> NULL
p->dl.dl_bw_cpu = iso_rq->cpu
dl_rebuild_rd_accounting()
dl_add_task_root_domain(p)
dl_task_rebuild_cpu(p) -> iso_rq->cpu
While dl_bw_rebuild_seq is odd, rq_H is already attached to
def_root_domain, so find_lock_later_rq() searches def_root_domain (or
cpumask_any_and() picks the first active CPU in p->cpus_ptr, which can be
an isolated CPU). Because dl_task_bw_span(p) returns NULL during the
rebuild, the check above does not reject the isolated later_rq.
Unlike migrate_task_rq_dl() and task_woken_dl(), which skip moving
bandwidth while dl_bw_rebuilding() is true, dl_task_offline_migration()
then unconditionally subtracts p->dl.dl_bw from src_rq->rd->dl_bw (which
can underflow total_bw before dl_clear_root_domains() runs), adds it to
def_root_domain, and overwrites p->dl.dl_bw_cpu with the isolated CPU.
Both the rebuild and later wakeups then treat def_root_domain as the root
domain that counts p.
[ ... ]
> @@ -3987,17 +4075,37 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
> */
> cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
> cpu = task_cpu(p);
> - if (cpumask_test_cpu(cpu, msk) ||
> - (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
> + if (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p))
> return cpu;
> if (cpumask_empty(msk)) {
> if (recorded >= 0 && cpu_active(recorded))
> return recorded;
> dl_get_task_effective_cpus(p, msk);
> cpumask_and(msk, msk, cpu_active_mask);
> - } else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
> - return recorded;
> }
> + /*
> + * Keep the bandwidth where it was counted: with the housekeeping
> + * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
> + * domains are new, and the CPU recorded can be offline. So prefer a
> + * CPU of the mask of the same kind as the CPU recorded: housekeeping
> + * or not.
> + */
> + if (recorded >= 0) {
> + const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
> + bool rec_hk = cpumask_test_cpu(recorded, hk);
> + unsigned int same;
> +
> + if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
> + return cpu;
> + if (cpumask_test_cpu(recorded, msk))
> + return recorded;
> + for_each_cpu(same, msk) {
> + if (cpumask_test_cpu(same, hk) == rec_hk)
> + return same;
> + }
> + }
[Severity: Medium]
Should p->dl.dl_bw_cpu be read with READ_ONCE() when initializing recorded
in dl_task_rebuild_cpu()?
dl_add_task_root_domain() calls dl_task_rebuild_cpu(p) while holding only
p->pi_lock, before acquiring dl_departed_lock. Concurrently, a dying
deadline task in its final __schedule() holds rq->lock and
dl_departed_lock (not p->pi_lock) and can write p->dl.dl_bw_cpu = -1 via
task_non_contending() -> dl_task_release_bw(). Because recorded is loaded
with a plain read and used after calling housekeeping_cpumask(), the
compiler is free to reload p->dl.dl_bw_cpu after the recorded >= 0 check
and pass -1 to cpumask_test_cpu().
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=28
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain
2026-11-17 14:10 ` [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-17 15:41 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:41 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] cgroup/cpuset: missing descendant cpus_allowed filter in
cpuset_cpus_allowed_without()
- [Medium] cgroup/cpuset: filter by task_cpu_possible_mask() in
cpuset_cpus_allowed_without()
--
commit 119af82ab355761f9fade868df0ef32c60b9344b
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Keep the last CPU a task has in its root domain
This commit refuses to take a CPU offline with -EBUSY when doing so would
leave a SCHED_DEADLINE task with runnable CPUs only outside the root
domain that accounts for its bandwidth.
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 7bdb9e21a7edb..3ccd61a37d302 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -4312,6 +4312,32 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)
> spin_unlock_irqrestore(&callback_lock, flags);
> }
>
> +/*
> + * Set @pmask to the active CPUs that the cpuset of @tsk gives it once @cpu
> + * is offline: those of the nearest ancestor that keeps one. Leave it empty
> + * if there is none. As cpuset_update_tasks_cpumask() does, keep the CPUs
> + * possible for @tsk. As __set_cpus_allowed_ptr() does, keep those of the
> + * affinity the user set, if one is left. The caller holds the CPU hotplug
> + * lock for writing, which keeps the cpusets and their tasks as they are.
> + * It holds the pi_lock of @tsk, which keeps its user affinity.
> + */
> +void cpuset_cpus_allowed_without(struct task_struct *tsk, int cpu,
> + struct cpumask *pmask)
> +{
> + struct cpuset *cs;
> +
> + guard(rcu)();
> + for (cs = task_cs(tsk); cs; cs = parent_cs(cs)) {
> + cpumask_and(pmask, cs->effective_cpus, cpu_active_mask);
> + cpumask_clear_cpu(cpu, pmask);
> + if (!cpumask_empty(pmask))
> + break;
> + }
[Severity: Medium]
Does cpuset_cpus_allowed_without() return an overly wide mask when it
walks past an intermediate ancestor to find remaining active CPUs?
During cgroup v2 hotplug, compute_effective_cpumask() propagates
effective_cpus top-down, intersecting each descendant's configured
cpus_allowed with its parent's new effective_cpus and only inheriting the
parent's mask when that intersection is empty:
kernel/cgroup/cpuset.c:compute_effective_cpumask() {
has_cpus = cpumask_and(new_cpus, cs->cpus_allowed, parent->effective_cpus);
if (!has_cpus && is_in_v2_mode())
cpumask_copy(new_cpus, parent->effective_cpus);
}
Consider a system with isolcpus=domain,0 and housekeeping CPUs 1-2, where
parent P has cpus_allowed = {1} and child C has cpus_allowed = {0, 1}
(giving C effective_cpus = {1}). When CPU 1 is taken offline, walking up
from C skips P (which becomes empty) and returns top_cpuset's remaining
active CPUs {0, 2}. dl_task_leaves_span() sees CPU 2 in the housekeeping
root domain span {1, 2} and allows the offline. However, once P inherits
{0, 2} from top_cpuset, C computes cpus_allowed & P->effective_cpus =
{0, 1} & {0, 2} = {0}, moving the deadline task and its bandwidth onto
isolated CPU 0 in def_root_domain without admission control.
> + cpumask_and(pmask, pmask, task_cpu_possible_mask(tsk));
[Severity: Medium]
Can intersecting pmask with task_cpu_possible_mask(tsk) after the
ancestor loop leave pmask empty when an ancestor has possible CPUs for
tsk?
On asymmetric arm64 systems where a 32-bit deadline task can only run on
a subset of CPUs, task_cs(tsk)->effective_cpus may retain active
64-bit-only CPUs after the outgoing 32-bit CPU is cleared. The loop then
stops at task_cs(tsk), and intersecting with task_cpu_possible_mask(tsk)
afterward produces an empty pmask. dl_task_leaves_span() treats an empty
msk as false and allows the last 32-bit CPU in the task's root domain to
go offline, after which guarantee_active_cpus() (which filters by
task_cpu_possible_mask(tsk) before testing each ancestor) walks up to an
ancestor with a 32-bit CPU in another root domain and moves the task's
bandwidth there without admission control.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=29
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change
2026-11-17 14:10 ` [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
@ 2026-11-17 15:45 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:45 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: boosted task with !dl_donor_refill runs on
old runtime
- [Medium] sched/deadline: dl_new_params deferred past dl_deboost()
breaks PI re-boost
- [Medium] sched/deadline: update_dl_entity() on dl_new_params mixes old
and new params
--
commit 707d1303811b0425863d63e123b53368955b72a6
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Check the runtime of a task whose parameters change
This commit adds a dl_new_params flag to struct sched_dl_entity so that
when sched_setattr() modifies a deadline task's runtime, deadline, or
period, the next non-boosted enqueue calls update_dl_entity() to check the
remaining runtime against the new parameters.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index b650c37ae5615..f1d259df8f2a0 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -2785,7 +2785,16 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
> static void
> enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
> {
> + bool new_params = dl_se->dl_new_params;
> +
> WARN_ON_ONCE(on_dl_rq(dl_se));
> + /*
> + * A boosted task runs on the parameters of its donor. Keep the mark
> + * until an enqueue checks the runtime against its own: the one at
> + * the deboost, or at a wakeup.
> + */
> + if (!is_dl_boosted(dl_se))
> + dl_se->dl_new_params = 0;
[Severity: Medium]
Can a boosted SCHED_DEADLINE task with !dl_se->dl_donor_refill keep
running on its old reservation's remaining runtime after sched_setattr()
shrinks its parameters?
When a SCHED_DEADLINE task with dl_se->runtime > 0 is boosted by an
earlier-deadline waiter, replenish_dl_entity() does not refill it from the
donor and leaves dl_se->dl_donor_refill at 0:
kernel/sched/deadline.c:replenish_dl_entity() {
...
if (dl_se->runtime <= 0)
dl_note_donor_refill(dl_se);
while (dl_se->runtime <= 0) {
dl_se->deadline += period;
dl_se->runtime += runtime;
}
...
}
Because the task is still consuming its own remaining runtime rather than
the donor's, deferring dl_new_params whenever is_dl_boosted(dl_se) is true
lets the task consume the old larger runtime while holding the lock, even
though sched_dl_overflow() has already released the old bandwidth in the
root domain for other deadline tasks to use.
> @@ -2834,12 +2843,13 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
> update_dl_entity(dl_se);
> } else if (flags & ENQUEUE_REPLENISH) {
> replenish_dl_entity(dl_se);
> - } else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
> + } else if (((flags & ENQUEUE_CLASS) || new_params) &&
> + !is_dl_boosted(dl_se)) {
> /*
> - * The task enters the class runnable. It may come back before
> - * its deadline, with the runtime left from older parameters.
> - * Check that runtime against the new parameters, as a wakeup
> - * does.
> + * The task enters the class, or its parameters change, while
> + * it is runnable. It may have runtime left from older
> + * parameters. Check that runtime against the new parameters,
> + * as a wakeup does.
> */
> update_dl_entity(dl_se);
> } else if ((flags & ENQUEUE_MOVE) &&
[Severity: Medium]
When a boosted task with dl_new_params set and !dl_se->dl_donor_refill is
deboosted in rt_mutex_setprio(), does deferring update_dl_entity() until
enqueue_dl_entity() miss the post-deboost re-boost check?
In rt_mutex_setprio(), dl_deboost(&p->dl) only updates p->dl.deadline when
dl_se->dl_donor_refill is set, and immediately checks whether the task
must be boosted again by the remaining top waiter:
kernel/sched/core.c:rt_mutex_setprio() {
...
dl_deboost(&p->dl);
/*
* dl_deboost() can start a new period. Its
* deadline can then be after the top waiter's.
* Boost the task again then.
*/
if (pi_task && dl_prio(pi_task->prio) &&
dl_entity_preempt(&pi_task->dl, &p->dl)) {
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
}
...
}
Because update_dl_entity() only runs later in sched_change_end() ->
enqueue_task_dl() -> enqueue_dl_entity(), dl_entity_preempt() compares
pi_task->dl against p->dl.deadline before update_dl_entity() advances
p->dl.deadline to a new period, leaving p unboosted with a deadline later
than its top waiter. If p is blocked on another rt_mutex (!scope->queued)
when deboosted, enqueue_dl_entity() does not run at all, so the priority
inheritance chain walk propagates p's stale deadline.
[Severity: Medium]
Can calling update_dl_entity() on dl_new_params mix the previous period's
dynamic state with the newly written static parameters when switching
between constrained and implicit deadlines?
If a running task started its current period with a constrained deadline
(dl_deadline < dl_period) and calls sched_setattr() to switch to an
implicit deadline (dl_deadline == dl_period), __setparam_dl() overwrites
dl_se->dl_deadline and dl_se->dl_period before enqueue_dl_entity() runs.
When dl_entity_overflow() then triggers in update_dl_entity(),
!dl_is_implicit(dl_se) is now false:
kernel/sched/deadline.c:update_dl_entity() {
...
if (unlikely((!dl_is_implicit(dl_se) ||
(dl_se->dl_defer && dl_se->dl_defer_running)) &&
!dl_time_before(dl_se->deadline, rq_clock(rq)) &&
!is_dl_boosted(dl_se))) {
update_dl_revised_wakeup(dl_se, rq);
return;
}
dl_se->dl_defer_running = 0;
replenish_dl_new_period(dl_se, rq);
...
}
Instead of capping the runtime via update_dl_revised_wakeup(),
update_dl_entity() calls replenish_dl_new_period() and grants a full fresh
runtime before the constrained reservation's period or 0-lag time has
elapsed. Conversely, if a runnable task whose old deadline has already
passed switches to a constrained deadline, dl_check_constrained_dl() at
the top of enqueue_dl_entity() computes dl_next_period() from the old
dl_se->deadline and the new dl_deadline/dl_period, throttles the task, and
returns before update_dl_entity() can start a new period.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=36
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked
2026-11-17 14:10 ` [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
@ 2026-11-17 15:46 ` sashiko-bot
0 siblings, 0 replies; 55+ messages in thread
From: sashiko-bot @ 2026-11-17 15:46 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: unstarted period when sleeping task has
!dl_task_pi_blocked()
- [Medium] sched/deadline: sched_setattr() on PI-boosted blocked task
skips period update
--
commit 5f0fe1e32222b5c48a70b881608ff3bc4a90abf3
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
sched/deadline: Start a period for a task that enters the class blocked
This commit starts a deadline period in switched_to_dl() and
dl_boost_blocked() when a task blocked on an rt_mutex enters the deadline
class or receives a new donor so its waiter does not sort with a deadline
of 0.
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 0b30059483136..90b17969554bf 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -125,6 +125,12 @@ void dl_deboost(struct sched_dl_entity *dl_se)
> }
> dl_drop_boost(dl_se);
> }
> +
> +/* Return true if @p waits on an rt_mutex. The caller holds its pi_lock. */
> +static inline bool dl_task_pi_blocked(struct task_struct *p)
> +{
> + return p->pi_blocked_on;
> +}
[Severity: Medium]
Can gating replenish_dl_new_period() on dl_task_pi_blocked(p) still leave
p->dl.deadline at 0 when a sleeping task participates in rt_mutex priority
inheritance before waking up?
A non-deadline task sleeping in futex_wait_requeue_pi() on uaddr1 has
!task_on_rq_queued(p) and p->pi_blocked_on == NULL. If it enters
dl_sched_class while sleeping on uaddr1 (via sched_setattr() or a PI boost
on another rt_mutex it owns), dl_task_pi_blocked(p) is false, so both
switched_to_dl() and dl_boost_blocked() leave p->dl.deadline at 0. A
subsequent FUTEX_CMP_REQUEUE_PI then calls rt_mutex_start_proxy_lock() ->
task_blocks_on_rt_mutex() without waking p, queueing its waiter on uaddr2
with a deadline of 0 ahead of all real deadline waiters.
Similarly, if a task holding an rt_mutex sleeps in a non-rt_mutex syscall
(p->pi_blocked_on == NULL) and sched_setattr() makes it SCHED_DEADLINE,
p->dl.deadline remains 0 while dl_prio(p->normal_prio) becomes true. If an
earlier-deadline waiter then blocks on that rt_mutex before p wakes up,
rt_mutex_setprio() evaluates dl_entity_preempt(&pi_task->dl, &p->dl)
against deadline 0, which returns false and fails to boost p.
[ ... ]
> @@ -4483,6 +4514,17 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
> if (!task_on_rq_queued(p)) {
> add_rq_bw(&p->dl, &rq->dl);
>
> + /*
> + * A task that waits on an rt_mutex cannot wait for its wakeup.
> + * The chain walk sorts its waiter by its deadline first. A task
> + * new to the class has a deadline of 0, which goes before every
> + * real one. Start a new period now, as setup_new_dl_entity()
> + * does at an enqueue.
> + */
> + if (dl_task_pi_blocked(p) && !p->dl.dl_throttled &&
> + dl_time_before(p->dl.deadline, rq_clock(rq)))
> + replenish_dl_new_period(&p->dl, rq);
> +
> return;
> }
[Severity: Medium]
What happens when sched_setattr() changes the policy of a task blocked on
an rt_mutex that is already in dl_sched_class due to a PI boost?
Because prev_class and next_class are both &dl_sched_class,
__sched_setscheduler() does not set DEQUEUE_CLASS, so switched_to_dl() does
not run:
kernel/sched/syscalls.c:__setscheduler_dl_pi() {
if (dl_prio(newprio) && !dl_policy(policy)) {
struct task_struct *pi_task = rt_mutex_get_top_task(p);
if (pi_task) {
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
}
}
...
}
When sched_setattr() gives such a blocked task its own SCHED_DEADLINE
reservation with an earlier deadline than its donor's,
__setscheduler_dl_pi() does nothing because dl_policy(policy) is true, and
prio_changed_dl() returns early because !task_on_rq_queued(p). Neither
p->dl.deadline nor p->dl.pi_se is updated, so rt_mutex_adjust_pi() sees
rt_waiter_up_to_date(waiter, p) and stops without re-sorting p's waiter or
propagating the earlier deadline down the lock chain.
Conversely, when a blocked task leaves SCHED_DEADLINE (!dl_policy(policy))
while remaining boosted in dl_sched_class (dl_prio(newprio)), setting
ENQUEUE_REPLENISH in __setscheduler_dl_pi() has no effect while
!scope->queued because dl_boost_blocked() is not called.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794922146.git.mathieu.desnoyers@efficios.com?part=33
^ permalink raw reply [flat|nested] 55+ messages in thread
end of thread, other threads:[~2026-11-17 15:46 UTC | newest]
Thread overview: 55+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 14:10 [PATCH v7 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 01/40] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 02/40] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 03/40] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 04/40] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 05/40] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
2026-11-17 14:38 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 06/40] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 07/40] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 08/40] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 09/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
2026-11-17 14:50 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 10/40] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 11/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 12/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
2026-11-17 14:55 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 13/40] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 14/40] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 15/40] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 16/40] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 17/40] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 18/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-17 15:14 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 19/40] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 20/40] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 21/40] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 22/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 23/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-17 15:18 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 24/40] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 25/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 26/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-17 15:31 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 27/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-17 15:29 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 28/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-17 15:32 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 29/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-17 15:41 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 30/40] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 31/40] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 32/40] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 33/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-17 15:46 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 34/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 35/40] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 36/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-17 15:45 ` sashiko-bot
2026-11-17 14:10 ` [PATCH v7 37/40] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
2026-11-17 14:10 ` [PATCH v7 38/40] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
2026-11-17 15:23 ` sashiko-bot
2026-11-17 14:11 ` [PATCH v7 39/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-17 14:11 ` [PATCH v7 40/40] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®