mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests
@ 2026-11-16 19:05 Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
                   ` (36 more replies)
  0 siblings, 37 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel

Hi Ingo, Juri,

This series fixes 33 bugs of SCHED_DEADLINE. Most of them are in its
bandwidth accounting. Then it shows the deadline bandwidth of each
runqueue in sched_debug, and adds three selftests. The fixes come first,
as you asked, so that stable can take patches 1 to 33.

Three of the bugs take the kernel down (patches 4, 8 and 24). One leaves
cpuset_mutex locked (patch 6). Most of the others leave total_bw of a
root domain, or this_bw of a runqueue, wrong. The root domain then
admits deadline tasks beyond its capacity, or refuses every one.

Patch 18 is Andrea Parri's. I carry it unchanged, with his agreement.
It is reviewed by Sebastian. It adds an affinity flag and a check next
to the code that patch 19 changes. Take it by whichever route you
prefer.

Patch 1 of v5, also Andrea's, is upstream. It is
commit f7141257dd67 ("sched/core: Fix cpuset_mutex double unlock in
__sched_setscheduler()"), and is dropped here.

Juri, Ingo dropped your four fixes from sched/urgent after the first
one crashed Marek Szyprowski's hotplug test. They are here:
- 1/4 is patch 16. It does not use the task's mask, which can have no
  active CPU during hotplug.
- 2/4 is patch 11, without its double release. It does not carry your
  Signed-off-by, which is yours to add back.
- 3/4 is patch 1.
- 4/4 is patch 12, extended.
Marek found v2 fixed on the board that crashed.

The fixes
=========

 1  Refuse SCHED_FLAG_KEEP_PARAMS requests that would change the policy
    or the class.
 2  Skip admission for parameters that SCHED_FLAG_KEEP_PARAMS keeps.
 3  Let a task boosted by a deadline waiter change its policy.
 4  Do not divide by zero active CPUs of a root domain.
 5  Refuse a dl_server debugfs write while the runqueue is offline.
 6  Take cpuset_mutex for a deadline policy change only with pi, so
    that sysrq-n does not leave it locked.
 7  Replenish a throttled task switched out and back, so that it runs
    again.
 8  Do not refill a boost from a runtime of 0, so that the owner's CPU
    no longer locks up.
 9  Reset the deadline and the runtime with the parameters.
 10 Start a new period at a deboost in the class, if the boost
    refilled the task's runtime from its donor.
 11 Release the bandwidth of a task that leaves the class asleep.
 12 Reject SCHED_DEADLINE on exited tasks (Juri Lelli).
 13 Keep the bandwidth of departing tasks across a root domain rebuild.
 14 Keep this_bw right when a task changes during a remote wakeup.
 15 Rebuild the root domains' accounting from the task list.
 16 Record the root domain that holds a task's bandwidth.
 17 Leave out of a rebuild a dying task that released its bandwidth.
 18 Check the affinity of a deadline task under the runqueue lock, so
    that sched_setaffinity() and sched_setattr() cannot both succeed
    (Andrea Parri).
 19 Move a task's bandwidth only where an attach made room for it.
 20 Wait for an attach before rebuilding the sched domains.
 21 Make a policy or affinity change of a deadline task wait for a
    cpuset attach in progress.
 22 Give back the bandwidth an attach did not move.
 23 Reserve an attach's bandwidth in the root domain each task moves to.
 24 Make a rebuild count a task in the root domain it runs in.
 25 Move a task's bandwidth to the root domain it runs in.
 26 Pick a CPU for a task in the root domain that counts its
    bandwidth, when that root domain has one.
 27 Refuse to take offline the last CPU a task has in the root domain
    that counts its bandwidth.
 28 Refuse a cpuset change that leaves a deadline task no CPU in the
    root domain that counts its bandwidth.
 29 Cancel the replenishment timer with the parameters.
 30 Count in this_bw the bandwidth a boosted task is given.
 31 Give a deadline to a task that enters the class while it waits on
    an rt_mutex.
 32 Check the runtime left of a task that comes back to the class
    runnable, as a wakeup does.
 33 Check the runtime left of a runnable task whose parameters change.

Patches 16, 17 and 19 to 28 go together. 16 records the root domain that
counts a task's bandwidth. 17 and 19 to 25 fix the rebuild, attach and
migration paths that the record exposes. 26 to 28 keep a task in the
root domain that counts it, as long as that root domain has a CPU for
it. Patch 30 completes patch 14. Patch 34
shows this_bw in sched_debug. Patches 35 to 37 add 96 cases in
tools/testing/selftests/sched/deadline, as Juri suggested.

Peter, patch 23 also touches __set_cpus_allowed_ptr_locked(). A cpuset
attach can reserve a deadline task's bandwidth in one root domain. If
the CPU picked for the task is in the other one, the patch steers the
task to the reserved CPU. Patch 21 touches __sched_setscheduler() and
sched_setaffinity(). A change of a deadline task waits there for a
cpuset attach in progress. Patch 26 touches select_fallback_rq() and
__set_cpus_allowed_ptr_locked() for deadline tasks.

Waiman, patches 27 and 28 add refusals that only isolcpus="domain" can
trigger. They apply when a change would leave a deadline task no CPU in
the root domain that counts it. A CPU offline or a write to cpuset.cpus
then fails with EBUSY. cpuset makes a partition root invalid, with a
new reason. Nothing is refused with admission control off
(sched_rt_runtime_us set to -1).

Known limits
============

The fixes leave these cases open. The message of each patch named
states its own.

- Patch 13: a task that left the class keeps its bandwidth until its
  0-lag time. A rebuild still drops it earlier if the CPU of the task
  is inactive, or if the task dies during the rebuild.
- Patch 25: a task can run in a root domain that does not count its
  bandwidth. The bandwidth then moves there without a check for room,
  as dl_task_offline_migration() does upstream.
- Patches 26 to 28: a task with no CPU left in the root domain that
  counts it still moves, and its bandwidth with it. With admission
  control on, patches 27 and 28 refuse the CPU offline and the cpuset
  changes that I found to lead there. Nothing is refused during a
  suspend, or with admission control off.
- A cpuset that holds isolated and housekeeping CPUs still admits a
  deadline task. Juri pointed at the XXX in dl_bw_capacity() for this
  in the v5 thread. It is left for later.

Testing
=======

The 96 cases pass wherever they run, with lockdep and no kernel
warning, in 19 QEMU guests:
- 5 guests of 8 vCPUs, one with proxy execution off, and 1 of 4 vCPUs.
- 3 guests with PREEMPT_RT, of 8, 8 and 4 vCPUs.
- 3 guests without CONFIG_CPUSETS, of 4, 4 and 5 vCPUs.
- 4 guests of 5 vCPUs booted with isolcpus=domain,1,2 or
  isolcpus=domain,0,1, 2 of them with PREEMPT_RT.
- 1 guest of 8 vCPUs booted with isolcpus=domain,6,7.
- 2 guests with CONFIG_RT_GROUP_SCHED, of 8 and 4 vCPUs.
The thirteen isolcpus cases run in the fourth group. There, three
cases that need 4 CPUs that are not isolated skip. Four of the thirteen
also run in the fifth group. move_cancelled runs only in the last
group.

On v7.4-rc1, 32 cases of deadline_bw_test fail in 8 guests of 8 vCPUs,
as patch 37 lists. overrun_caught and hotplug_sleeper_switched fail in
7 guests of 8, the others in every guest. The three cases of
deadline_rebuild_test fail there too, and so do throttled_round_trip
and asleep_round_trip of deadline_switch_test. The sysrq cases were not
run on v7.4-rc1: sysrq-n leaves cpuset_mutex locked there, which patch
6 fixes.

Each case added in v5 or v6 for a fix fails without that fix. Each fix
names the cases that test it, and patch 37 lists the rest. There are
three exceptions:
- deboost_blocked_keeps_order passes on v7.4-rc1, and fails on v5.
- return_asleep_new_params passes everywhere. It covers the wakeup
  path that patches 32 and 33 copy.
- deboost_keeps_own_runtime passes with and without the flag of patch
  10. It checks that a task keeps the runtime of its own reservation.

Two of the new cases pass on v7.4-rc1 and add coverage only:
frozen_moved_switched and pi_chain_two_owners. So do the affinity
changes in random_mix.

The three sysrq cases only run with DEADLINE_TEST_SYSRQ=1, which the
19 guests set. move_normalized is one of them.

For patches 11, 13 and 20, a kernel built with the series up to that
patch confirms the claim of its message. Each of the 37 patches builds
alone. The whole series also builds without CONFIG_CPUSETS, with
PREEMPT_RT and with CONFIG_RT_GROUP_SCHED.

Not tested: cgroup v1 cpusets, threaded cgroups (every move is a write
to cgroup.procs), the sched_ext server, and CPUs of unequal capacity.

Patch 8 has no selftest. A 300ms debug delay before
rt_mutex_adjust_pi() locks up the series without patch 8, but not with
it. A stress test switches the waiter in and out of the class. The
series runs it for 600 seconds, with and without PREEMPT_RT.

How this was developed
======================

An LLM coding assistant wrote these patches, their messages and the
tests, under my direction and review. I know the concepts behind
SCHED_DEADLINE, but not its implementation well. So I ask those who
know this code for a careful reading of the fixes. A reproducer that
stops failing shows that the symptom goes away. It does not show that
the fix is the one you would have written.

The bugs came from:
- flaky cgroup selftests of a feature I am developing;
- the review of Juri's four fixes, and his review of v5;
- building deadline_bw_test, and reading the code;
- the Sashiko AI reviews of v1 to v5. Each finding was checked against
  the code first.

Changes since v5
================

- Rebased on tip sched/urgent ec6f6731538e. Patch 1 of v5 is upstream,
  so it is dropped and the patches are renumbered.
- Patch 8 is the smaller fix Juri suggested. replenish_dl_entity()
  reads the inherited parameters once, and does not refill from a
  runtime of 0. The owner runs on until the chain walk deboosts it. The
  series no longer touches the rt_mutex code. So it no longer needs v2
  of "locking/rtmutex: priority inheritance fixes and tests", and
  applies on tip sched/urgent alone. Patches 10 and 11 are rebased on
  the new patch 8.
- Patch 9: the comment of __dl_clear_params() says that it also resets
  the deadline and the runtime.
- Patch 11 calls task_non_contending() for a task that leaves the
  class asleep, as Juri suggested. The code is shorter and does the
  same.
- Patch 10: in v5 the deboost tested the runtime against dl_runtime.
  A task that its donor refilled could pass that test with a runtime
  and a deadline that were not its own. The refill is now recorded in
  a flag, dl_donor_refill. The deboost starts a new period when the
  flag is set, and keeps the runtime and the deadline otherwise. Two
  new cases test this: deboost_after_donor_refill and
  deboost_keeps_own_runtime.
- The Fixes: tags of patch 24 also name 318e18ed22e8. Its
  subject says what the rebuild does: it counts a task in the root
  domain it runs in.
- Patch 16: the second case of its message needs no pin and no change
  of admission control. hotplug_sleeper_switched now checks the
  accounting while the CPU is still offline, and fails without the
  patch. Bringing the CPU back rebuilt the accounting, and hid the
  bandwidth left behind.
- Fixes from the Sashiko review of v5:
  - 10: the deboost starts the new period at once. A task blocked on
    another rt_mutex kept a deadline of 0. Its waiter then went first.
    If the new period ends after the top waiter's deadline, the task is
    boosted again.
  - 16: sched_setaffinity() checks the root domain that counts the
    task. A departing task records the CPU it departs from.
  - 19: an attach that reserved the bandwidth moves it, even if the
    mask does not change.
  - 24: a simpler test of the recorded CPU. The behaviour is the same.
  - 25: the bandwidth also moves at a wakeup with no migration. The test
    compares root domains, not the CPU the task leaves. New subject.
- Patch 13 (13 in v5), from Juri's review:
  - A departed task keeps the CPU recorded when priority inheritance
    boosts it back into the class. In v5, the end of the boost took the
    bandwidth from the root domain the task ran in. Patch 16 hid this
    at the end of the series. The new case pi_departed_moved tests it.
  - Two comments are Juri's: what dl_bw_gen dates, and why a rebuild
    drops the bandwidth of a departed task whose CPU is inactive.
- Patches 15 to 17 (15 to 17 in v5), from Juri's review:
  - 15: the rebuild does not walk the task list when no task holds a
    deadline bandwidth. A new count, dl_nr_bw_tasks, tells it. A suspend
    and a resume run one rebuild for each CPU. In a 64-vCPU guest with
    10000 threads and no deadline task, a rebuild took 0.7 to 1.2 ms
    without the count. It takes 0.03 ms with it. Patch 27 uses the
    count too.
  - 16: set_cpus_allowed_dl() tests dl_bw_gen under dl_departed_lock.
  - 17: the message says that the patch completes patch 15 and needs
    patch 16.
- New patch 18, by Andrea Parri, posted on its own on 2026-11-05. It
  moves the affinity check of a deadline task under the runqueue lock.
  It adds SCA_DL_CHECK as 0x10, so SCA_DL_BW_RESERVED of patch 19 stays
  0x20. Patch 19 is rebased on it. The new case affinity_vs_setattr
  tests it.
- New patch 21, and patches 22 and 23 (20 and 21 in v5), from Juri's
  review. In v5, a task could become a deadline task between
  cpuset_can_attach() and cpuset_attach(). Nothing was reserved for it.
  It could use the room reserved for the other tasks, and their
  bandwidth stayed behind.
  - New patch 21: sched_setattr() and sched_setaffinity() of a deadline
    task wait for an attach in progress. The change then comes before
    the attach, and is part of its reservation, or after it.
    sched_setaffinity() tests the policy with no lock before it decides
    to wait. So the policy is tested again under the runqueue lock. The
    call starts over if a sched_setattr() made the task a deadline task
    meanwhile. The wait is not interruptible.
  - Patch 22 comes after that wait. Only sysrq-n can then take a task
    out of the class between the two halves of an attach. The patch
    gives back the part of the reservation that no task used. The new
    case move_normalized tests it.
  - Patch 23: cpuset_can_attach() records, with each task it counts,
    the CPU the room was reserved on. Juri suggested to track the
    counted tasks. cpuset_attach_task() moves the bandwidth of a task
    only if the task has that record. It moves it to that CPU, and does
    not compute the CPU again. A task that wakes up during the attach
    and runs on another CPU no longer misses its reservation. A wakeup
    needs no wait. During the attach, the root domain the task leaves
    still counts its bandwidth. The destination has room reserved for
    it.
  - Seven new cases test this: move_became_deadline,
    move_became_deadline_no_room, move_affinity_changed,
    move_affinity_became_deadline, move_woken, move_woken_full and
    move_cancelled.
- Six more new cases:
  - move_died and frozen_moved_switched: a thread killed during a
    move, and a task frozen by the cgroup freezer.
  - Four follow reproducers of Juri's sched-deadline-tests:
    stopped_switched_out, hotplug_sleeper_switched,
    hotplug_no_admission_pinned and pi_chain_two_owners.
- random_mix also changes the affinity of its tasks.
- New patches 26 to 28. With isolcpus="domain", a cpuset can hold CPUs
  of two root domains. A deadline task could go from one to the other
  with no cpuset attach. A CPU offline or a cpuset that lost CPUs could
  do it. Patch 25 then moved its bandwidth with no check for room.
  - Patch 26: a CPU of the root domain that counts the task is tried
    first, wherever a CPU is picked for it.
  - Patch 27: a CPU offline fails with EBUSY if it takes the last CPU a
    deadline task has in that root domain.
  - Patch 28: a cpuset change that leaves a deadline task no CPU in
    that root domain is refused. A write to cpuset.cpus fails with
    EBUSY. cpuset makes a partition root invalid.
  - Four new cases test this: hotplug_keeps_root_domain,
    hotplug_last_refused, mask_out_refused (was mask_back_asleep) and
    partition_takes_refused. The three hotplug_fallback cases now fail
    if the task wakes up in the other root domain.
    no_admission_affinity_moves tests patch 25 with admission control
    off.
- Patches 2, 3, 6, 14, 17, 20, 29 and 30 have Juri's Reviewed-by.
  Patches 7 and 9 have his Acked-by.
- Not in this series: the single list of accounted tasks that Juri
  suggested in his review of patch 13 of v5. I tried it. It removes
  dl_bw_gen and the walk of the task list. But the rebuild then runs
  with interrupts disabled for its whole length. In a 64-vCPU guest,
  that was 0.15ms with 1000 deadline tasks and 1.2ms with 10000. Today
  it is less than 0.1ms at a time. I left it out for that reason.
- New patch 31: a task that enters the class while it waits on an
  rt_mutex gets a deadline. Before, its waiter was sorted with a
  deadline of 0, ahead of every real one. Upstream has this bug. The
  triage of the Sashiko review of v5 found it.
- Six new selftest cases test these fixes. They fail on v5.
- Patch 2's comment names the use of SCHED_FLAG_KEEP_PARAMS on a
  deadline task: a change of its utilization clamps (Juri Lelli).
- Patch 5 of v5 is split in two, as Juri asked. 4 removes the division
  by zero. 5 refuses the debugfs write in the hotplug windows. Each has its
  own Fixes: tag.
- Patch 6 has clearer comments on its two policy checks, and an
  unlikely() (Juri Lelli).
- New patches 32 and 33, for the case Juri pointed out in v5. A task
  kept the runtime left from its old parameters when it came back to
  the class runnable. A runnable task that changed its parameters kept
  it too. Both now get the check a wakeup does. Upstream has these
  bugs. Three new cases test them.
- Patch 23's message says what an attach without a reservation does.
- Patch 37 (29 in v5) said that 13 cases fail on v7.4-rc1. That count
  left out two cases added in v5. A new run of the 89 cases gives 32,
  and the message lists them. The message is a third of its length: it
  no longer repeats what each fix says about its own cases.
- v5 reported three upstream problems found with isolcpus=domain. Waiman
  Long answered them; see the v5 cover. The tests avoid them.

Not changed:
- A departed task that priority inheritance boosts back into the class
  is counted once across a rebuild. The rebuild counts it where it
  runs. If the deboost races the rebuild, its bandwidth is dropped
  until its 0-lag time. Nothing is counted twice.
- At a deboost, the 0-lag time of a task comes from the deadline and
  runtime it ran on during the boost, as upstream.
- The EPERM of a sleeping task moved to another partition is the
  baseline's affinity check.
- An attach whose mask keeps a CPU of the task's root domain reserves
  nothing. The task then stays in that root domain.

Based on tip sched/urgent, ec6f6731538e.

v5: https://lore.kernel.org/all/20261108185406.1549770-1-mathieu.desnoyers@efficios.com/
v4: https://lore.kernel.org/all/20261107014451.1487825-1-mathieu.desnoyers@efficios.com/


Andrea Parri (1):
  sched/deadline: Check SCHED_DEADLINE affinity under the rq lock

Juri Lelli (1):
  sched/deadline: Reject SCHED_DEADLINE on exited tasks

Mathieu Desnoyers (35):
  sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or
    the class
  sched/deadline: Do not admit bandwidth for parameters that are kept
  sched: Let a task boosted by a deadline waiter change its policy
  sched/deadline: Do not divide by zero active CPUs of a root domain
  sched/deadline: Refuse dl_server debugfs writes while the rq is
    offline
  sched: Take cpuset_mutex for a deadline policy change only with pi
  sched/deadline: Do not lose a throttled task switched out and back
  sched/deadline: Do not replenish a boost from cleared parameters
  sched/deadline: Reset the deadline and the runtime with the parameters
  sched/deadline: Reset the donor's runtime at a deboost in the class
  sched/deadline: Release the bandwidth of a task that leaves the class
    asleep
  sched/deadline: Keep departing bandwidth across a root domain rebuild
  sched/deadline: Fix this_bw when a task is changed during a remote
    wakeup
  sched/deadline: Rebuild the root domains' accounting from the task
    list
  sched/deadline: Record the root domain holding a task's bandwidth
  sched/deadline: Leave a dying task out of a rebuild of the accounting
  sched/deadline: Move a task's bandwidth only where room was made
  cgroup/cpuset: Wait for an attach before rebuilding the sched domains
  sched: Make deadline policy and affinity changes wait for a cpuset
    attach
  cgroup/cpuset: Give back the bandwidth an attach did not move
  cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
    to
  sched/deadline: Make a rebuild count a task in the root domain it runs
    in
  sched/deadline: Move a task's bandwidth to the root domain it runs in
  sched/deadline: Keep a task in the root domain that counts its
    bandwidth
  sched/deadline: Keep the last CPU a task has in its root domain
  cgroup/cpuset: Refuse to take a deadline task out of its root domain
  sched/deadline: Cancel the replenishment timer with the parameters
  sched/deadline: Count the bandwidth a boosted task is given
  sched/deadline: Start a period for a task that enters the class
    blocked
  sched/deadline: Check the runtime of a task that comes back runnable
  sched/deadline: Check the runtime of a task whose parameters change
  sched/debug: Show the deadline bandwidth of each runqueue
  selftests/sched: Add a test of SCHED_DEADLINE policy round trips
  selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain
    rebuild
  selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

 include/linux/cpuset.h                        |   20 +-
 include/linux/sched.h                         |   43 +
 include/linux/sched/deadline.h                |   38 +-
 kernel/cgroup/cgroup.c                        |    3 -
 kernel/cgroup/cpuset-internal.h               |   15 +-
 kernel/cgroup/cpuset.c                        |  496 +-
 kernel/sched/core.c                           |  124 +-
 kernel/sched/deadline.c                       | 1157 ++-
 kernel/sched/debug.c                          |   12 +-
 kernel/sched/sched.h                          |   20 +
 kernel/sched/syscalls.c                       |  163 +-
 kernel/sched/topology.c                       |    8 +
 tools/testing/selftests/Makefile              |    1 +
 .../selftests/sched/deadline/.gitignore       |    3 +
 .../testing/selftests/sched/deadline/Makefile |    8 +
 tools/testing/selftests/sched/deadline/config |    5 +
 .../sched/deadline/deadline_bw_test.c         | 8101 +++++++++++++++++
 .../sched/deadline/deadline_rebuild_test.c    |  497 +
 .../sched/deadline/deadline_switch_test.c     |  808 ++
 .../testing/selftests/sched/deadline/settings |    1 +
 20 files changed, 11218 insertions(+), 305 deletions(-)
 create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
 create mode 100644 tools/testing/selftests/sched/deadline/Makefile
 create mode 100644 tools/testing/selftests/sched/deadline/config
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/settings


base-commit: ec6f6731538e0d45872eb279fdeb477bed152d6f
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 02/37] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
                   ` (35 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A sched_setattr() request with SCHED_FLAG_KEEP_PARAMS can ask for
another policy. __sched_setscheduler() then applies only half of the
change, and returns 0:
- SCHED_OTHER to SCHED_BATCH leaves the policy as it was.
- SCHED_DEADLINE to SCHED_OTHER sets DEQUEUE_CLASS. switched_from_dl()
  then runs on a task that stays in the class. It would release
  bandwidth the task still uses.
- SCHED_DEADLINE on a task of another policy reserves bandwidth in
  sched_dl_overflow(). The task keeps its policy and a dl_bw of 0, so
  nothing releases that bandwidth. For a task boosted into the deadline
  class, it is not even a class change.

SCHED_FLAG_KEEP_PARAMS came with utilization clamping. With
SCHED_FLAG_KEEP_POLICY, it lets sched_setattr() change only the clamps
of a task. So __sched_setscheduler() skips __setscheduler_params(),
which sets p->policy, and the sched_class assignment. Nothing checks
that the request keeps the policy.

In testing on v7.4-rc1, a deadline waiter boosted a SCHED_OTHER mutex
owner with a 40ms slice. One such call on the owner left 0.4 CPU in
total_bw, until the next rebuild of the root domains.

Fix this by refusing a request for another policy or class with
EINVAL. Move sched_dl_overflow() after it, as the last check. Nothing
gives back the bandwidth it reserves. A refused request also leaves
sched_reset_on_fork alone.

With this patch, that call is refused and total_bw does not change. The
params_keep_boosted case of deadline_bw_test, added later in this
series, tests it.

Juri Lelli found the class change, from a report by Tejun Heo. The
Sashiko AI review of the first posting of this series found the policy
change and the boosted case.

Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-3-dd761fe005b4@redhat.com/
---
 kernel/sched/syscalls.c | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 0ec401af80d1..89b0d5787f45 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -647,17 +647,6 @@ int __sched_setscheduler(struct task_struct *p,
 		goto recheck;
 	}
 
-	/*
-	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
-	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
-	 * is available.
-	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
-		retval = -EBUSY;
-		goto unlock;
-	}
-
-	p->sched_reset_on_fork = reset_on_fork;
 	oldprio = p->prio;
 
 	newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
@@ -677,9 +666,38 @@ int __sched_setscheduler(struct task_struct *p,
 	prev_class = p->sched_class;
 	next_class = __setscheduler_class(policy, newprio);
 
+	/*
+	 * With SCHED_FLAG_KEEP_PARAMS, the change below skips
+	 * __setscheduler_params() and the sched_class assignment. The
+	 * policy and the class stay as they are. Refuse a request for
+	 * another policy or class, rather than carry out half of it.
+	 *
+	 * The class depends on the priority, which priority inheritance
+	 * can raise. It also depends on whether a BPF scheduler is in
+	 * charge. So test the class too, not only the policy.
+	 */
+	if ((attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (policy != p->policy || prev_class != next_class)) {
+		retval = -EINVAL;
+		goto unlock;
+	}
+
 	if (prev_class != next_class)
 		queue_flags |= DEQUEUE_CLASS;
 
+	/*
+	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
+	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
+	 * is available. Keep this check last. If a later check failed,
+	 * nothing would give back the bandwidth it reserves.
+	 */
+	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+		retval = -EBUSY;
+		goto unlock;
+	}
+
+	p->sched_reset_on_fork = reset_on_fork;
+
 	scx_prepare_setscheduler(p, policy);
 
 	scoped_guard (sched_change, p, queue_flags) {
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 02/37] sched/deadline: Do not admit bandwidth for parameters that are kept
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 03/37] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
                   ` (34 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

__sched_setscheduler() runs the deadline admission test on a request
with SCHED_FLAG_KEEP_PARAMS. But the task keeps its parameters: the
call skips __setscheduler_params().

Commit f4dddf90d58d ("sched: Skip priority checks with
SCHED_FLAG_KEEP_PARAMS") made sys_sched_setattr() copy the task's own
parameters into such a request. Then the priority checks pass.
get_params() copies them without a lock.

Another call can change the task's parameters before
__sched_setscheduler() takes the runqueue lock. The request is then
stale, and dl_param_changed() takes it for a change. sched_dl_overflow()
would put the stale bandwidth in place of the task's own in total_bw,
and in this_bw for a sleeping task. But the task keeps its own dl_bw.
Its release at the 0-lag time takes that dl_bw, and the difference
stays behind.

Fix this by skipping sched_dl_overflow() for SCHED_FLAG_KEEP_PARAMS.
There is nothing to admit for parameters that are kept.

Found by reading, after the Sashiko AI review of the first posting of
this series raised the boosted case of the previous patch. Not
reproduced: the parameter change must land inside that window.

Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/syscalls.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 89b0d5787f45..6f8ce5fd94a1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -690,8 +690,16 @@ int __sched_setscheduler(struct task_struct *p,
 	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
 	 * is available. Keep this check last. If a later check failed,
 	 * nothing would give back the bandwidth it reserves.
+	 *
+	 * With SCHED_FLAG_KEEP_PARAMS, the task keeps its parameters, so
+	 * there is nothing to admit. Such a call changes other attributes,
+	 * such as the utilization clamps. A deadline task does not use its
+	 * clamps, but keeps them for when it leaves the class. The
+	 * parameters in attr may also be stale: sys_sched_setattr() read
+	 * them from the task without a lock.
 	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
 		retval = -EBUSY;
 		goto unlock;
 	}
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 03/37] sched: Let a task boosted by a deadline waiter change its policy
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 02/37] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 04/37] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
                   ` (33 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A SCHED_OTHER or SCHED_FIFO task that holds an rt_mutex can be boosted
into the deadline class by a deadline waiter. Then a sched_setattr() or
sched_setscheduler() call on it fails with EBUSY, unless it asks for
SCHED_DEADLINE. sysrq-n would not reset it either.

The admission test runs "if setscheduling to SCHED_DEADLINE (or
changing the parameters of a SCHED_DEADLINE task)", as its comment
says. sched_dl_overflow() then handles three cases, by the task's
policy: the task enters, leaves or stays in SCHED_DEADLINE. But the
call site tests dl_task(p). That tests the priority, so a boosted task
passes it. For such a task and another policy, sched_dl_overflow()
matches no case and returns -1.

Fix this by testing the policy, with task_has_dl_policy(). A boosted
task has no bandwidth of its own to admit or release.

Now sysrq-n can take a boosted task out of the class.
normalize_rt_tasks() calls __sched_setscheduler() without pi, so the
task gets its normal priority. Nothing then drops the boost. At the
unlock, rt_mutex_setprio() sees no change of priority. The pi_se of the
task would keep pointing to the waiter, which can exit. So also drop the
boost in __setscheduler_dl_pi() when the task leaves the class, as
rt_mutex_setprio() does.

The boosted_owner_policy case of deadline_bw_test, added later in this
series, fails with EBUSY on v7.4-rc1. It passes with this series. The
sysrq_boosted_owner case of deadline_switch_test tests the sysrq-n path.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/syscalls.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 6f8ce5fd94a1..cb115aa9b0b3 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -304,6 +304,13 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
 			scope->flags |= ENQUEUE_REPLENISH;
 		}
 	}
+
+	/*
+	 * Without pi, as from normalize_rt_tasks(), a boosted task can leave
+	 * the class. Drop its boost then, as rt_mutex_setprio() does.
+	 */
+	if (!dl_prio(newprio))
+		p->dl.pi_se = &p->dl;
 }
 #else /* !CONFIG_RT_MUTEXES */
 static inline void __setscheduler_dl_pi(int newprio, int policy,
@@ -691,6 +698,10 @@ int __sched_setscheduler(struct task_struct *p,
 	 * is available. Keep this check last. If a later check failed,
 	 * nothing would give back the bandwidth it reserves.
 	 *
+	 * Test the policy, not the priority. A task boosted into the class
+	 * by priority inheritance has no bandwidth of its own to admit or
+	 * release.
+	 *
 	 * With SCHED_FLAG_KEEP_PARAMS, the task keeps its parameters, so
 	 * there is nothing to admit. Such a call changes other attributes,
 	 * such as the utilization clamps. A deadline task does not use its
@@ -699,7 +710,8 @@ int __sched_setscheduler(struct task_struct *p,
 	 * them from the task without a lock.
 	 */
 	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
-	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+	    (dl_policy(policy) || task_has_dl_policy(p)) &&
+	    sched_dl_overflow(p, policy, attr)) {
 		retval = -EBUSY;
 		goto unlock;
 	}
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 04/37] sched/deadline: Do not divide by zero active CPUs of a root domain
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (2 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 03/37] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
                   ` (32 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

__dl_sub() and __dl_add() divide by their cpus argument. Most callers
pass dl_bw_cpus(), the number of active CPUs of a root domain. That
number can be 0, and the division then traps.

Commit daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl
tasks") added extra_bw, the bandwidth that deadline tasks can reclaim
on a runqueue. A change of a root domain's total_bw is spread over the
extra_bw of its active CPUs.

During hotplug, a CPU sits in the default root domain, which can then
have no active CPU. This lasts from the rebuild in
sched_cpu_deactivate() to the offline, and from the online to
sched_cpu_activate(). A debugfs write to the runtime or period of its
fair or ext server divides there. A write of the current value passes
admission in the window. In 8-CPU VMs of v7.4-rc1, a loop of such writes
oopses at the first offline and online:

  Oops: divide error: 0000 [#1] SMP NOPTI
  RIP: 0010:dl_server_apply_params+0x335/0x380
  Call Trace:
   sched_server_write_common.isra.0+0x1c0/0x260
   full_proxy_write+0x58/0x90
   vfs_write+0xcb/0x550

By reading, two other paths would divide by zero:
- inactive_task_timer(), for a task that left SCHED_DEADLINE or died,
  if its CPU went offline.
- sched_setattr(SCHED_DEADLINE) on a task asleep on an offline CPU,
  with admission control off.

Fix this by changing total_bw alone when the root domain has no active
CPU.

With this patch, the reproducer survives 20 rounds with no warning.

Fixes: daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index dfae08625342..b614d5fc1a1a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
 	}
 }
 
+/*
+ * Spread a change of total_bw over the extra_bw of the active CPUs of the
+ * root domain. @cpus is their number. With no active CPU, as can happen
+ * during CPU hotplug, change total_bw alone.
+ */
 static inline
 void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw -= tsk_bw;
-	__dl_update(dl_b, (s32)tsk_bw / cpus);
+	if (cpus)
+		__dl_update(dl_b, (s32)tsk_bw / cpus);
 }
 
 static inline
 void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw += tsk_bw;
-	__dl_update(dl_b, -((s32)tsk_bw / cpus));
+	if (cpus)
+		__dl_update(dl_b, -((s32)tsk_bw / cpus));
 }
 
 static inline bool
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (3 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 04/37] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:30   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 06/37] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
                   ` (31 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit 4043f5498416 ("sched/deadline: Reject debugfs dl_server writes
for offline CPUs") refuses a write to a dl_server's runtime or period
on an offline CPU. Its message says that such a write has no meaningful
semantics.

It tests cpu_online(). But during hotplug the CPU is still online while
it sits in the default root domain. This lasts from the rebuild in
sched_cpu_deactivate() to the offline, and from the online to
sched_cpu_activate(). That root domain does not count the server's
bandwidth. A write in the window still goes through admission. With
admission control off, it changes total_bw there. On v7.4-rc1, a write
of the current value divides by zero in that window. The previous patch
removes the division, not the write.

Fix this by testing rq->online instead. It is set under the runqueue
lock when the CPU joins a root domain that counts it, after
cpu_active() is set.

A write now fails with EBUSY while the CPU goes offline or comes
online, as a write on an offline CPU already does. Writes at any other
time are unchanged.

Applied without the previous patch, this patch alone also stops the
oops. The reproducer survives 20 rounds with no warning, in 3 VMs of 3.

Fixes: d741f297bcea ("sched/fair: Fair server interface")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/debug.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..e0d6cd3bd607 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -410,7 +410,15 @@ static ssize_t sched_server_write_common(struct file *filp, const char __user *u
 			return  -EINVAL;
 		}
 
-		if (!cpu_online(cpu_of(rq)))
+		/*
+		 * Refuse the write while the runqueue is offline. This
+		 * includes the windows where the CPU goes offline or comes
+		 * online. The root domain then does not count the server's
+		 * bandwidth, and may have no active CPU. rq->online is set
+		 * under this lock when the CPU joins a root domain that
+		 * counts it. cpu_active() is set before that.
+		 */
+		if (!rq->online)
 			return -EBUSY;
 
 		update_rq_clock(rq);
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 06/37] sched: Take cpuset_mutex for a deadline policy change only with pi
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (4 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 07/37] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
                   ` (30 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. On success, it releases it only when pi is set.
normalize_rt_tasks(), for sysrq-n, is the one caller with pi unset. It
runs under read_lock(&tasklist_lock). So sysrq-n takes the mutex in
atomic context for a deadline task, and never releases it. Every later
change that involves SCHED_DEADLINE then blocks forever.

Commit 710da3c8ea7d ("sched/core: Prevent race condition between cpuset
and __sched_setscheduler()") took the cpuset lock only when pi is set.
normalize_rt_tasks() cannot take it under tasklist_lock. Its changelog
accepts this: after sysrq-n, the deadline guarantees are gone anyway.
Commit 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex") then took
the mutex only for deadline tasks. It replaced the pi test by a policy
test, and left the release under pi.

Fix this by taking the mutex for a deadline change only when pi is
set.

The policy test reads p->policy before the runqueue lock. It is checked
again under the lock only when the caller gave no policy. Otherwise,
the task can enter SCHED_DEADLINE in between. The call then takes it
out of the class without the mutex. A rebuild of the root domain
accounting could run at the same time. So check again under the
runqueue lock, and retry with the mutex.

sysrq-n on v7.4-rc1, with a deadline task and lockdep on:

  sysrq: Nice All RT Tasks
  BUG: sleeping function called from invalid context at kernel/locking/mutex.c:623
  in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 4356, name: deadline_switch
  locks held by deadline_switch/4356: 4, last CPU#3:
   ...
   #3: ffffffff83006098 (tasklist_lock){.+.+}-{3:3}, at: normalize_rt_tasks+0x34/0x150

A sched_setattr(SCHED_DEADLINE) after it never returned, in 2 VMs of 2.
The sysrq_normalize case of deadline_switch_test, added later in this
series, tests this with DEADLINE_TEST_SYSRQ=1. It passes with this
series.

Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/syscalls.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index cb115aa9b0b3..9b04e77297ae 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -561,9 +561,11 @@ int __sched_setscheduler(struct task_struct *p,
 
 	/*
 	 * SCHED_DEADLINE bandwidth accounting relies on stable cpusets
-	 * information.
+	 * information. Do not take cpuset_mutex when pi is unset: the
+	 * caller is normalize_rt_tasks(), which holds tasklist_lock and
+	 * cannot sleep.
 	 */
-	if (dl_policy(policy) || dl_policy(p->policy)) {
+	if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
 		cpuset_locked = true;
 		cpuset_lock();
 	}
@@ -643,7 +645,13 @@ int __sched_setscheduler(struct task_struct *p,
 		}
 	}
 
-	/* Re-check policy now with rq lock held: */
+	/*
+	 * The caller gave no policy (SETPARAM_POLICY): the request was
+	 * derived from an unlocked read of p->policy above. If the task
+	 * changed policy since, re-derive the request from scratch, and
+	 * release cpuset_mutex: the re-derived request re-decides whether
+	 * it is needed.
+	 */
 	if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
 		policy = oldpolicy = -1;
 		task_rq_unlock(rq, p, &rf);
@@ -654,6 +662,18 @@ int __sched_setscheduler(struct task_struct *p,
 		goto recheck;
 	}
 
+	/*
+	 * The caller gave an explicit policy, so the request needs no
+	 * re-derivation (oldpolicy is -1 above). But the decision to
+	 * take cpuset_mutex used the same unlocked read: the task
+	 * entered SCHED_DEADLINE since, and the change below would take
+	 * it out of the class. Retry to take the mutex.
+	 */
+	if (unlikely(pi && !cpuset_locked && task_has_dl_policy(p))) {
+		task_rq_unlock(rq, p, &rf);
+		goto recheck;
+	}
+
 	oldprio = p->prio;
 
 	newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 07/37] sched/deadline: Do not lose a throttled task switched out and back
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (5 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 06/37] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
                   ` (29 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A SCHED_DEADLINE task can be throttled, switched to another policy, and
switched back before its 0-lag time. It may then never run again, not
even to die of a fatal signal.

A task that leaves SCHED_DEADLINE keeps its parameters until its 0-lag
time, so that leaving and coming back gains nothing. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
their reset from the replenishment timer to the inactive timer, which
switched_to_dl() cancels.

This misses a task that was throttled when it left. An overrun can put
its 0-lag time after its replenishment timer. That timer fires while
the task is out of the class, and does nothing. On the return of the
task, enqueue_dl_entity() leaves it to that timer, which has fired. The
task is then on no dl_rq, with no timer pending, until it leaves
SCHED_DEADLINE again.

Fix this by replenishing the task when the timer finds it out of the
class, as for a throttled task that is not queued. Its return then
enqueues it.

Clearing dl_throttled alone would not do. It would forgive the overrun.
A yield left set, or the old deadline of a constrained task, would
throttle the task again. The Sashiko AI review of the first posting
found this in the first version of this patch.

A cgroup selftest under development hung on this bug about once in
1350 runs. The throttled_round_trip case of deadline_switch_test, added
later in this series, fails 8 times out of 8 on v7.3 and v7.4-rc1. It
passes with this patch.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b614d5fc1a1a..e4d7cbf4861d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1233,9 +1233,18 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	/*
 	 * The task might have changed its scheduling policy to something
 	 * different than SCHED_DEADLINE (through switched_from_dl()).
+	 * It keeps its parameters until its 0-lag time. If it comes back
+	 * before then, nothing else would lift its throttle. So replenish
+	 * it now, as the !queued case below does. Its return enqueues it.
 	 */
-	if (!dl_task(p))
+	if (!dl_task(p)) {
+		if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
+			sched_clock_tick();
+			update_rq_clock(rq);
+			replenish_dl_entity(dl_se);
+		}
 		goto unlock;
+	}
 
 	/*
 	 * The task might have been boosted by someone else and might be in the
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (6 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 07/37] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:42   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 09/37] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
                   ` (28 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

When a SCHED_DEADLINE waiter boosts the owner of an rt_mutex, the owner
runs on the waiter's parameters, through dl.pi_se. Commit 2d3d891d3344
("sched/deadline: Add SCHED_DEADLINE inheritance logic") calls this a
temporary solution. The owner is never throttled. When its runtime runs
out, replenish_dl_entity() refills it at once from the waiter's
dl_runtime.

The waiter can leave SCHED_DEADLINE before the chain walk deboosts the
owner. Its parameters can be cleared in between, at its 0-lag time.
replenish_dl_entity() then loops forever on a dl_runtime of 0. It
holds the rq lock with interrupts off, so the owner's CPU locks up.

O is a SCHED_OTHER task that holds the rt_mutex L. W is a SCHED_DEADLINE
task. T is any other task:

  CPU 0 (O)                         CPU 1 (W)                     CPU 2 (T)

  locks L
                                    blocks on L
                                      rt_mutex_setprio(O, W)
                                        O->dl.pi_se = &W->dl
                                    sleeps
                                      task_non_contending()
                                        arms W's inactive timer
                                                                  sched_setscheduler(W)
                                                                    W leaves SCHED_DEADLINE
                                                                    task_rq_unlock(W)
                                    <W's inactive timer>
                                    inactive_task_timer()
                                      __dl_clear_params()
                                        W->dl.dl_runtime = 0
  <tick>
  update_curr_dl_se()
    enqueue_task_dl()
      replenish_dl_entity()
        runtime += W->dl.dl_runtime
        loops forever
                                                                    rt_mutex_adjust_pi(W)
                                                                      rt_mutex_setprio(O, W)
                                                                        spins on O's rq lock

Fix this by reading the inherited parameters once, and by not refilling
from a runtime of 0. The owner stays runnable. Its deadline stays where
it was. It runs until the chain walk deboosts it, or until it releases
the lock. That also covers sysrq-n: normalize_rt_tasks() changes the
waiter without a chain walk. The warning stays for an entity that is not
boosted.

A 300ms debug delay before rt_mutex_adjust_pi() locks up v7.4-rc1 in
8-CPU VMs, but not a kernel with this patch.

The Sashiko AI review of the third posting found the lockup. Juri Lelli
suggested this fix in place of a larger one.

Fixes: a5e7be3b28a2 ("sched/deadline: Clear dl_entity params when setscheduling to different class")
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/avL8oOtjKqVNR7-l@jlelli-thinkpadt14gen4.remote.csb/
---
 kernel/sched/deadline.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index e4d7cbf4861d..c4ea51059941 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -807,8 +807,26 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 {
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
 	struct rq *rq = rq_of_dl_rq(dl_rq);
+	struct sched_dl_entity *pi_se = pi_of(dl_se);
+	u64 runtime = READ_ONCE(pi_se->dl_runtime);
+	u64 deadline = READ_ONCE(pi_se->dl_deadline);
+	u64 period = READ_ONCE(pi_se->dl_period);
 
-	WARN_ON_ONCE(pi_of(dl_se)->dl_runtime <= 0);
+	/*
+	 * A boosted entity runs on the parameters of its donor, the top
+	 * waiter of an rt_mutex. The donor can leave the class and have them
+	 * cleared before the chain walk deboosts this entity. They are
+	 * cleared under the donor's rq lock, which can be another one. So
+	 * read each of them once. Refilling from a runtime of 0 would loop
+	 * forever. Leave the entity runnable instead: the walk deboosts it,
+	 * or its task releases the lock.
+	 */
+	if (unlikely(!runtime)) {
+		WARN_ON_ONCE(!is_dl_boosted(dl_se));
+		dl_se->dl_yielded = 0;
+		dl_se->dl_throttled = 0;
+		return;
+	}
 
 	/*
 	 * This could be the case for a !-dl task that is boosted.
@@ -822,8 +840,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 */
 	if (dl_se->dl_deadline == 0 ||
 	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
-		dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
-		dl_se->runtime = pi_of(dl_se)->dl_runtime;
+		dl_se->deadline = rq_clock(rq) + deadline;
+		dl_se->runtime = runtime;
 	}
 
 	if (dl_se->dl_yielded && dl_se->runtime > 0)
@@ -836,8 +854,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 * arbitrary large.
 	 */
 	while (dl_se->runtime <= 0) {
-		dl_se->deadline += pi_of(dl_se)->dl_period;
-		dl_se->runtime += pi_of(dl_se)->dl_runtime;
+		dl_se->deadline += period;
+		dl_se->runtime += runtime;
 	}
 
 	/*
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 09/37] sched/deadline: Reset the deadline and the runtime with the parameters
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (7 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
                   ` (27 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A task that leaves SCHED_DEADLINE keeps its parameters until its 0-lag
time, so that leaving and coming back gains nothing. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
their reset to that time. Before it, the replenishment timer did the
reset at the deadline. So a task back after the reset found its
deadline passed, and started a new period. But __dl_clear_params()
resets neither the deadline nor the runtime.

Now a task back after its 0-lag time can be enqueued on its old
deadline and runtime. With runtime left, its 0-lag time comes before
its deadline. Back before that deadline, it would first spend the runtime
of the reservation it gave back. Back after it, a constrained task
would be throttled until its old period ends. "sched/deadline: Do not
lose a throttled task switched out and back" moves the deadline of a
throttled task on. This widens the window before the deadline to about
a period.

A task with no parameters of its own, boosted by a deadline waiter,
keeps the donor's deadline and runtime after its deboost. If it then
enters SCHED_DEADLINE before that deadline, it runs on the donor's
runtime. On v7.4-rc1, a task of 10ms per 100ms ran 40ms of its first
period.

Fix this by resetting the deadline and the runtime in
__dl_clear_params(), and in switched_from_dl() at the deboost of a task
with no parameters. The next enqueue then starts a new period.

The Sashiko AI review found the first case in the second posting of
this series, and the deboost in the third. The deboost_resets_period
case of deadline_bw_test, added later in this series, tests the
deboost.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c4ea51059941..dd48f74412b8 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3557,6 +3557,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	if (task_on_rq_queued(p) && p->dl.dl_runtime)
 		task_non_contending(&p->dl, false);
 
+	/*
+	 * A task with no parameters of its own was boosted into the class.
+	 * Its deadline and runtime are the donor's. Reset them, so that its
+	 * next entry into the class starts a new period.
+	 */
+	if (!p->dl.dl_runtime) {
+		p->dl.deadline = 0;
+		p->dl.runtime = 0;
+	}
+
 	/*
 	 * In case a task is setscheduled out from SCHED_DEADLINE we need to
 	 * keep track of that on its cpuset (for correct bandwidth tracking).
@@ -3972,7 +3982,8 @@ bool __checkparam_dl(const struct sched_attr *attr)
 }
 
 /*
- * This function clears the sched_dl_entity static params.
+ * Clear the static parameters of @dl_se. Also reset its deadline and its
+ * runtime, so that its next enqueue starts a new period.
  */
 static void __dl_clear_params(struct sched_dl_entity *dl_se)
 {
@@ -3983,6 +3994,9 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_bw			= 0;
 	dl_se->dl_density		= 0;
 
+	dl_se->deadline			= 0;
+	dl_se->runtime			= 0;
+
 	dl_se->dl_throttled		= 0;
 	dl_se->dl_yielded		= 0;
 	dl_se->dl_non_contending	= 0;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (8 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 09/37] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:38   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 11/37] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
                   ` (26 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A task boosted by a deadline waiter, its donor, runs on the donor's
reservation. Commit 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE
inheritance logic") uses the waiter's parameters "during the whole
period it holds the pi-lock".

A boosted task that runs out of runtime is not throttled.
replenish_dl_entity() refills it from the waiter's parameters: it adds
the waiter's runtime, and moves the deadline by the waiter's period.
From then on, the runtime and the deadline of the task are not those of
its own reservation.

A task that leaves the class at its deboost does not keep them.
"sched/deadline: Reset the deadline and the runtime with the parameters"
resets them for a task with no parameters. For a task with parameters,
they are cleared with the parameters at its 0-lag time.

But the boosted task can be a SCHED_DEADLINE task itself, when the
waiter has an earlier deadline. It stays in the class at its deboost.
rt_mutex_setprio() only resets dl.pi_se. enqueue_dl_entity() keeps a
deadline that is ahead. So a task that was refilled keeps the waiter's
runtime and deadline after the unlock. A task that sched_setattr() makes
SCHED_DEADLINE while boosted ends its boost the same way.

In one period, the task can run what the waiter left on top of its own
runtime. Other deadline tasks can miss deadlines.

Fix this by recording that refill in a new flag, dl_donor_refill. If
the boost ends in the class with the flag set, start a new period at
the deboost. The next enqueue is too late for a task blocked on another
rt_mutex: the chain walk copies its deadline into its waiter first.

The task can still hold a lock that another deadline task waits on. The
new deadline can be later than that waiter's. rt_mutex_setprio() then
boosts the task again.

A task that was not refilled keeps its runtime and its deadline. They
are those of its own reservation. A new period there would give it a
full runtime at every contended unlock.

The Sashiko AI review of the fourth posting found the sched_setattr()
case. Four cases of deadline_bw_test, added later in this series, test
this fix: deboost_in_class_resets_period, deboost_blocked_keeps_order,
deboost_after_donor_refill and deboost_keeps_own_runtime.

In deboost_after_donor_refill, an owner of 5ms every 100ms runs 8ms with
the lock. Its donor is a task of 2ms every 10ms. A deboost that tests
the runtime against dl_runtime, in place of the flag, misses this
refill: it leaves less than dl_runtime. The owner then held less than
1ms after the unlock, in 3 guests of 3. With the flag, it holds its own
5ms.

Fixes: 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance logic")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h   |  6 ++++++
 kernel/sched/core.c     | 16 +++++++++++++---
 kernel/sched/deadline.c | 37 +++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h    | 10 ++++++++++
 kernel/sched/syscalls.c |  2 +-
 5 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..96b498f79484 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -683,6 +683,11 @@ struct sched_dl_entity {
 	 * @dl_overrun tells if the task asked to be informed about runtime
 	 * overruns.
 	 *
+	 * @dl_donor_refill tells if a boosted task had its runtime refilled
+	 * from the parameters of its donor. Its runtime and deadline are
+	 * then not those of its own reservation. The end of the boost
+	 * clears it.
+	 *
 	 * @dl_server tells if this is a server entity.
 	 *
 	 * @dl_server_active tells if the dlserver is active(started).
@@ -711,6 +716,7 @@ struct sched_dl_entity {
 	unsigned int			dl_yielded        : 1;
 	unsigned int			dl_non_contending : 1;
 	unsigned int			dl_overrun	  : 1;
+	unsigned int			dl_donor_refill   : 1;
 	unsigned int			dl_server         : 1;
 	unsigned int			dl_server_active  : 1;
 	unsigned int			dl_defer	  : 1;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5ed660f099a5..a22df8b43a1a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7834,16 +7834,26 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 				p->dl.pi_se = pi_task->dl.pi_se;
 				scope->flags |= ENQUEUE_REPLENISH;
 			} else {
-				p->dl.pi_se = &p->dl;
+				dl_deboost(&p->dl);
+				/*
+				 * dl_deboost() can start a new period. Its
+				 * deadline can then be after the top waiter's.
+				 * Boost the task again then.
+				 */
+				if (pi_task && dl_prio(pi_task->prio) &&
+				    dl_entity_preempt(&pi_task->dl, &p->dl)) {
+					p->dl.pi_se = pi_task->dl.pi_se;
+					scope->flags |= ENQUEUE_REPLENISH;
+				}
 			}
 		} else if (rt_prio(prio)) {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				dl_drop_boost(&p->dl);
 			if (oldprio < prio)
 				scope->flags |= ENQUEUE_HEAD;
 		} else {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				dl_drop_boost(&p->dl);
 			if (rt_prio(oldprio))
 				p->rt.timeout = 0;
 		}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index dd48f74412b8..1f3da0e0a698 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -93,6 +93,38 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
 {
 	return pi_of(dl_se) != dl_se;
 }
+
+/*
+ * Record that @dl_se had its runtime refilled from the parameters of its
+ * donor, if it is boosted.
+ */
+static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
+{
+	if (is_dl_boosted(dl_se))
+		dl_se->dl_donor_refill = 1;
+}
+
+/*
+ * End the boost of @dl_se, whose task stays in the class. If the boost
+ * refilled its runtime from the donor's parameters, start a new period
+ * now. The next enqueue is too late for a task blocked on another
+ * rt_mutex: the chain walk copies its deadline into its waiter node
+ * first. Without a refill, the task keeps the runtime and the deadline
+ * of its own reservation.
+ *
+ * The caller holds the rq lock and the pi_lock of the task, and has
+ * updated the rq clock.
+ */
+void dl_deboost(struct sched_dl_entity *dl_se)
+{
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se->dl_donor_refill) {
+		dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
+		dl_se->runtime = dl_se->dl_runtime;
+	}
+	dl_drop_boost(dl_se);
+}
 #else /* !CONFIG_RT_MUTEXES: */
 static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
 {
@@ -734,6 +766,7 @@ static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
 	/* for non-boosted task, pi_of(dl_se) == dl_se */
 	dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
 	dl_se->runtime = pi_of(dl_se)->dl_runtime;
+	dl_note_donor_refill(dl_se);
 
 	/*
 	 * If it is a deferred reservation, and the server
@@ -842,11 +875,15 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
 		dl_se->deadline = rq_clock(rq) + deadline;
 		dl_se->runtime = runtime;
+		dl_note_donor_refill(dl_se);
 	}
 
 	if (dl_se->dl_yielded && dl_se->runtime > 0)
 		dl_se->runtime = 0;
 
+	if (dl_se->runtime <= 0)
+		dl_note_donor_refill(dl_se);
+
 	/*
 	 * We keep moving the deadline away until we get some
 	 * available runtime for the entity. This ensures correct
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 92134ce326a4..d87296abec4c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -356,6 +356,16 @@ extern int  sched_dl_global_validate(void);
 extern void sched_dl_do_global(void);
 extern int  sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
 extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+#ifdef CONFIG_RT_MUTEXES
+extern void dl_deboost(struct sched_dl_entity *dl_se);
+
+/* End the boost of @dl_se. The caller holds the rq lock of its task. */
+static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
+{
+	dl_se->pi_se = dl_se;
+	dl_se->dl_donor_refill = 0;
+}
+#endif
 extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
 extern bool __checkparam_dl(const struct sched_attr *attr);
 extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 9b04e77297ae..f346176f1ba1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -310,7 +310,7 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
 	 * the class. Drop its boost then, as rt_mutex_setprio() does.
 	 */
 	if (!dl_prio(newprio))
-		p->dl.pi_se = &p->dl;
+		dl_drop_boost(&p->dl);
 }
 #else /* !CONFIG_RT_MUTEXES */
 static inline void __setscheduler_dl_pi(int newprio, int policy,
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 11/37] sched/deadline: Release the bandwidth of a task that leaves the class asleep
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (9 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 12/37] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
                   ` (25 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A task that leaves SCHED_DEADLINE while it is not queued can keep its
bandwidth in total_bw until the next rebuild of the root domains.
Repeated, this fills total_bw up to the admission limit. Then
sched_setattr(SCHED_DEADLINE) and CPU offlining fail with EBUSY.

A task that leaves the class keeps its bandwidth until its 0-lag time.
Releasing it earlier would allow temporary overloads. For a queued task,
switched_from_dl() calls task_non_contending(). It releases the
bandwidth if the 0-lag time is past. Otherwise it arms the inactive
timer, which releases it then. For a task that is not queued,
switched_from_dl() relies on the timer armed when the task blocked.

But the timer is not always pending when the task leaves the class:
- The task blocked after its 0-lag time. The timer was not armed.
- The timer fired while the task was still in the class. It only
  removed running_bw.
- The task was woken up on another CPU and is not enqueued there yet.
  The wakeup cancelled the timer.
- The task came back to the class asleep. switched_to_dl() cancelled
  the timer.

Then nothing releases total_bw.

Fix this in switched_from_dl(). For a task that is not queued and has
no timer pending, call task_non_contending(), as for a queued task. It
arms the timer if the 0-lag time is ahead. Otherwise it releases the
bandwidth now and clears the parameters, as inactive_task_timer() does.

Clearing the parameters keeps later paths from releasing the bandwidth
again:
- A timer handler that already started finds a dl_bw of 0.
- The deboost of a later priority inheritance boost finds no
  parameters.

An owner that the task boosts as a waiter can find the parameters
cleared too. "sched/deadline: Do not replenish a boost from cleared
parameters" handles this, so a backport needs it first. Without it, the
owner's CPU can lock up as soon as the waiter leaves the class.

Juri Lelli's first fix, since dropped from tip, released the bandwidth
here without clearing dl_bw. The bandwidth of a boosted task was then
released twice.

Juri Lelli suggested the call to task_non_contending() in review.

The Sashiko AI review of the third posting found the two cases with the
0-lag time ahead. deadline_bw_test, added later in this series, tests
this in pi_former_owner_asleep and deboost_released_resets_period.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/
---
 kernel/sched/deadline.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 1f3da0e0a698..09188a28b3a1 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3620,6 +3620,14 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 		if (p->dl.dl_non_contending)
 			sub_running_bw(&p->dl, &rq->dl);
 		sub_rq_bw(&p->dl, &rq->dl);
+		/*
+		 * Without an armed timer, nothing releases total_bw. Do as for
+		 * a queued task: arm the timer if the 0-lag time is ahead, or
+		 * release the bandwidth and clear the parameters now. Call
+		 * task_non_contending() after sub_rq_bw(), which needs dl_bw.
+		 */
+		if (!p->dl.dl_non_contending)
+			task_non_contending(&p->dl, false);
 	}
 
 	/*
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 12/37] sched/deadline: Reject SCHED_DEADLINE on exited tasks
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (10 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 11/37] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 13/37] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
                   ` (24 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

From: Juri Lelli <juri.lelli@redhat.com>

sched_setattr(SCHED_DEADLINE) on a zombie task succeeds and charges
total_bw via sched_dl_overflow(), but the task has already done its
final schedule() and will never run again. Nothing releases the reserved
bandwidth. Repeating this fills total_bw to the admission cap, after
which sched_setattr(SCHED_DEADLINE) and CPU offline fail with -EBUSY.

Reject sched_setattr(SCHED_DEADLINE) on exited tasks with -ESRCH.

A task of the class that exited, switched to another policy, runs
switched_from_dl(), which decrements the count of deadline tasks of its
cpuset that cgroup_task_dead() decremented already: at 0 with a deadline
task left in it, the cpuset is skipped by the next rebuild of the root
domains' accounting, and that task's bandwidth with it. Reject that as
well. The deadline_bw_test selftest added later in this series has the
case, death_zombie_switched: on v7.4-rc1 the switch is accepted, and
the one-CPU partition root the zombie ran in admits 9 tasks of 10% after
a rebuild, with a task of 30% still in it, where 6 are due.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/3f6e6c87-645b-45fd-8fc6-8da5bc5b82d8@gmail.com/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-4-dd761fe005b4@redhat.com/
Cc: stable@vger.kernel.org
Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
[mathieu: reject a task of the class leaving it too, found by the
 Sashiko AI review of the patch]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index f346176f1ba1..8c72e751ff4e 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -592,6 +592,19 @@ int __sched_setscheduler(struct task_struct *p,
 	if (retval)
 		goto unlock;
 
+	/*
+	 * Reject SCHED_DEADLINE on exited tasks. A zombie will never be
+	 * scheduled again, so sched_dl_overflow() would charge total_bw
+	 * that nothing will ever release. And reject a change of policy of
+	 * one in the class: cgroup_task_dead() took it off the count of
+	 * deadline tasks of its cpuset, which switched_from_dl() would
+	 * decrement a second time.
+	 */
+	if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
+		retval = -ESRCH;
+		goto unlock;
+	}
+
 	/*
 	 * If not changing anything there's no need to proceed further,
 	 * but store a possible modification of reset_on_fork.
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 13/37] sched/deadline: Keep departing bandwidth across a root domain rebuild
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (11 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 12/37] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 14/37] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
                   ` (23 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth in
total_bw until its 0-lag time. This keeps admission control on the safe
side. Then inactive_task_timer() removes it from the root domain of the
task's CPU. Two things break this:

 - A rebuild of the accounting clears every root domain, then adds back
   the deadline tasks of the cpusets. It misses a task that left
   SCHED_DEADLINE, or a dead one. Their bandwidth is dropped, and the
   timer removes it again.
 - A task that left SCHED_DEADLINE can also move to another root
   domain. The timer then takes total_bw from that one, which never
   had it.

Either way, a root domain admits deadline tasks beyond its capacity
until the next rebuild.

Fix this with dl_departed_list. A departing task goes on it with the
CPU it is on. A rebuild puts its bandwidth back in that CPU's root
domain. The timer removes it from there, and so does
sched_dl_overflow() for a task that returns to SCHED_DEADLINE.

Priority inheritance can boost a departed task back into the class,
after it moved to another root domain. The boost cancels the timer and
takes the task off the list. Keep the CPU recorded then. When the boost
ends, the bandwidth is taken from the root domain that counts it, not
from the one the task runs in. The pi_departed_moved case of
deadline_bw_test, added later in this series, tests this.

A rebuild still drops the bandwidth of a task whose CPU is inactive, or
that dies during the rebuild. Each task records a count of the rebuilds,
dl_bw_gen, with its bandwidth. The timer removes nothing if a rebuild
dropped the bandwidth since.

Make dl_cookie and dl_bw_visited() static: only deadline.c uses them.

deadline_rebuild_test, added later in this series, fails every time on
v7.3 and v7.4-rc1 kernels. A one-CPU partition root admits two tasks of
600ms every second there, where it has room for one. It passes with
this patch.

Fixes: 387e31300b57 ("sched/deadline: Fix the update of the total -deadline utilization")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h          |  16 +++
 include/linux/sched/deadline.h |   5 +-
 kernel/cgroup/cpuset.c         |  11 +-
 kernel/sched/deadline.c        | 254 +++++++++++++++++++++++++++++----
 4 files changed, 249 insertions(+), 37 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 96b498f79484..063e49690658 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -740,6 +740,22 @@ struct sched_dl_entity {
 	 */
 	struct hrtimer			inactive_timer;
 
+	/*
+	 * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
+	 * time" keeps its bandwidth in its root domain until the inactive
+	 * timer removes it. Meanwhile, @dl_departed_node links the task on
+	 * the list that a rebuild of the root domains' accounting reads.
+	 * @dl_departed_cpu is a CPU of the root domain that counts the
+	 * bandwidth. It stays recorded while priority inheritance boosts the
+	 * task back into the class. Otherwise @dl_departed_cpu is -1.
+	 *
+	 * @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
+	 * counted. If a rebuild dropped it since, there is nothing to remove.
+	 */
+	struct list_head		dl_departed_node;
+	int				dl_departed_cpu;
+	unsigned int			dl_bw_gen;
+
 	/*
 	 * Bits for DL-server functionality. Also see the comment near
 	 * dl_server_update().
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..bdf39ac1d43d 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -32,7 +32,7 @@ static inline bool dl_time_before(u64 a, u64 b)
 struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
-extern void dl_clear_root_domain_cpu(int cpu);
+extern void dl_clear_root_domains(void);
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
@@ -43,9 +43,6 @@ extern void dl_clear_root_domain_cpu(int cpu);
 extern bool dl_task_needs_bw_move(struct task_struct *p,
 				  const struct cpumask *new_mask);
 
-extern u64 dl_cookie;
-extern bool dl_bw_visited(int cpu, u64 cookie);
-
 static inline bool dl_server(struct sched_dl_entity *dl_se)
 {
 	return dl_se->dl_server;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..4159abfbbbb5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -985,21 +985,14 @@ void dl_rebuild_rd_accounting(void)
 {
 	struct cpuset *cs = NULL;
 	struct cgroup_subsys_state *pos_css;
-	int cpu;
-	u64 cookie = ++dl_cookie;
 
 	lockdep_assert_cpuset_lock_held();
 	lockdep_assert_cpus_held();
 	lockdep_assert_held(&sched_domains_mutex);
 
-	rcu_read_lock();
-
-	for_each_possible_cpu(cpu) {
-		if (dl_bw_visited(cpu, cookie))
-			continue;
+	dl_clear_root_domains();
 
-		dl_clear_root_domain_cpu(cpu);
-	}
+	rcu_read_lock();
 
 	cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
 
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 09188a28b3a1..cbdfa0793703 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -195,7 +195,13 @@ static inline unsigned long dl_bw_capacity(int i)
 	}
 }
 
-bool dl_bw_visited(int cpu, u64 cookie)
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+static u64 dl_cookie;
+
+static bool dl_bw_visited(int cpu, u64 cookie)
 {
 	struct root_domain *rd = cpu_rq(cpu)->rd;
 
@@ -249,6 +255,112 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
 	       cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
 }
 
+/*
+ * A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
+ * its bandwidth in total_bw until that time. Then the inactive timer
+ * removes it. Meanwhile the task may run anywhere, under another policy.
+ * The timer holds a reference to it.
+ *
+ * A rebuild of the root domains' accounting (dl_clear_root_domains() and
+ * dl_add_task_root_domain()) does not find such a task among the deadline
+ * tasks of the cpusets. So dl_departed_list holds these tasks, from their
+ * departure to their timer. The rebuild puts their bandwidth back in the
+ * root domain of the CPU each one was on when it left.
+ *
+ * dl_bw_gen numbers the rebuilds of the accounting: how many times the
+ * root domains were wiped and reconstructed (dl_clear_root_domains()
+ * increments it). A task records the number of its last definite
+ * state, counted or settled. That number dates the state:
+ *
+ *  - Current: the bandwidth is counted, or a release settled it since
+ *    the rebuild began. A release subtracts; the walk of the cpusets
+ *    skips.
+ *  - One behind: the clear that just ran wiped it. The rebuild puts
+ *    it back.
+ *  - Older: an earlier rebuild dropped it. A task on the departed list
+ *    stays out, and its release removes nothing. A task that is in the
+ *    class is counted again.
+ *
+ * Only equalities are compared, so the wrap of the counter is
+ * harmless.
+ *
+ * A rebuild drops the bandwidth when the CPU recorded is no longer
+ * active. It also drops the bandwidth of a task that dies while the
+ * rebuild runs. The cpuset walk skips that task, and it was not dead yet
+ * when the list was put back.
+ *
+ * dl_departed_lock protects the list and both counts. Take it with
+ * interrupts disabled, under the task's rq lock or pi_lock or in the
+ * rebuild, and before any dl_bw lock.
+ */
+static LIST_HEAD(dl_departed_list);
+static DEFINE_RAW_SPINLOCK(dl_departed_lock);
+static unsigned int dl_bw_gen;
+
+/*
+ * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
+ * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
+ * recorded if @p has one: priority inheritance boosted @p back into the
+ * class and its boost ends. The caller holds the rq lock of @p.
+ */
+static void dl_task_depart(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	if (!list_empty(&dl_se->dl_departed_node))
+		return;
+	if (dl_se->dl_departed_cpu < 0)
+		dl_se->dl_departed_cpu = cpu;
+	list_add(&dl_se->dl_departed_node, &dl_departed_list);
+}
+
+/*
+ * Take @p off dl_departed_list. Priority inheritance boosts @p back into
+ * the deadline class. Keep the CPU recorded: its root domain still counts
+ * the bandwidth, wherever @p runs now. The caller holds the rq lock of
+ * @p.
+ */
+static void dl_task_undepart(struct task_struct *p)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	list_del_init(&dl_se->dl_departed_node);
+}
+
+/*
+ * Remove the bandwidth of @p from the root domain that counts it. That is
+ * the root domain of the CPU recorded when @p departed, or else of @cpu.
+ * Clear the record. Take @p off dl_departed_list. Remove nothing if a
+ * rebuild dropped the bandwidth since it was counted. Then record the
+ * current dl_bw_gen in @p, so that a rebuild in progress does not count it
+ * again. The caller holds the rq lock of @p.
+ */
+static void dl_task_release_bw(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct dl_bw *dl_b;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	if (dl_se->dl_departed_cpu >= 0) {
+		cpu = dl_se->dl_departed_cpu;
+		dl_se->dl_departed_cpu = -1;
+		list_del_init(&dl_se->dl_departed_node);
+	}
+	if (dl_se->dl_bw_gen == dl_bw_gen) {
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+	}
+	dl_se->dl_bw_gen = dl_bw_gen;
+}
+
 static inline
 void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
@@ -481,13 +593,9 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				sub_running_bw(dl_se, dl_rq);
 
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
-				struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				raw_spin_lock(&dl_b->lock);
-				__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
-				raw_spin_unlock(&dl_b->lock);
+				dl_task_release_bw(p, task_cpu(p));
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -496,8 +604,18 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 	}
 
 	dl_se->dl_non_contending = 1;
-	if (!dl_server(dl_se))
-		get_task_struct(dl_task_of(dl_se));
+	if (!dl_server(dl_se)) {
+		struct task_struct *p = dl_task_of(dl_se);
+
+		get_task_struct(p);
+		/*
+		 * A dying task keeps its bandwidth in the root domain of the
+		 * CPU it runs on. Record that CPU. switched_from_dl() records
+		 * a task that leaves SCHED_DEADLINE.
+		 */
+		if (READ_ONCE(p->__state) == TASK_DEAD)
+			dl_task_depart(p, task_cpu(p));
+	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
 }
@@ -2242,17 +2360,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		goto no_task;
 
 	if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
-		struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 		if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
 			sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
 			sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
 			dl_se->dl_non_contending = 0;
 		}
 
-		raw_spin_lock(&dl_b->lock);
-		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-		raw_spin_unlock(&dl_b->lock);
+		dl_task_release_bw(p, task_cpu(p));
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3513,12 +3627,33 @@ void dl_add_task_root_domain(struct task_struct *p)
 	dl_get_task_effective_cpus(p, msk);
 	cpu = cpumask_first_and(cpu_active_mask, msk);
 	BUG_ON(cpu >= nr_cpu_ids);
+
+	raw_spin_lock(&dl_departed_lock);
+	/*
+	 * Skip a task whose dl_bw_gen is current. Either
+	 * dl_clear_root_domains() put it back as a departed task, and
+	 * priority inheritance boosted it into the class since. Or it died
+	 * since the rebuild started, and dl_task_release_bw() released its
+	 * bandwidth. A task that died since with its 0-lag time ahead is on
+	 * the departed list. Count it in the root domain of the CPU
+	 * recorded, if that CPU is active.
+	 */
+	if (p->dl.dl_bw_gen == dl_bw_gen)
+		goto unlock;
+	if (p->dl.dl_departed_cpu >= 0) {
+		cpu = p->dl.dl_departed_cpu;
+		if (!cpu_active(cpu))
+			goto unlock;
+	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
 
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_gen = dl_bw_gen;
+unlock:
+	raw_spin_unlock(&dl_departed_lock);
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 }
 
@@ -3576,9 +3711,51 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
-void dl_clear_root_domain_cpu(int cpu)
+/*
+ * Clear every root domain before a rebuild of their accounting, and count
+ * the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
+ * back the bandwidth of the tasks on dl_departed_list, which no cpuset
+ * holds as deadline tasks. Put each in the root domain of the CPU
+ * recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ */
+void dl_clear_root_domains(void)
 {
-	dl_clear_root_domain(cpu_rq(cpu)->rd);
+	struct sched_dl_entity *dl_se;
+	u64 cookie = ++dl_cookie;
+	struct dl_bw *dl_b;
+	int cpu;
+
+	guard(raw_spinlock_irqsave)(&dl_departed_lock);
+	dl_bw_gen++;
+	for_each_possible_cpu(cpu) {
+		if (!dl_bw_visited(cpu, cookie))
+			dl_clear_root_domain(cpu_rq(cpu)->rd);
+	}
+	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
+		cpu = dl_se->dl_departed_cpu;
+		/*
+		 * Skip bandwidth that an earlier rebuild dropped. It is not
+		 * counted now, and its release removes nothing
+		 * (dl_task_release_bw()).
+		 */
+		if (dl_se->dl_bw_gen != dl_bw_gen - 1)
+			continue;
+		/*
+		 * Drop the bandwidth of a task whose CPU is inactive. A CPU
+		 * that goes offline is in the default root domain by then.
+		 * The root domain that counted the task cannot be found
+		 * from the recorded number. The release then removes
+		 * nothing, and no later rebuild restores it.
+		 */
+		if (!cpu_active(cpu))
+			continue;
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_add(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+		dl_se->dl_bw_gen = dl_bw_gen;
+	}
 }
 
 static void switched_from_dl(struct rq *rq, struct task_struct *p)
@@ -3634,9 +3811,20 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 * We cannot use inactive_task_timer() to invoke sub_running_bw()
 	 * at the 0-lag time, because the task could have been migrated
 	 * while SCHED_OTHER in the meanwhile.
+	 *
+	 * For the same reason, the timer cannot take the task's bandwidth
+	 * from the root domain of the CPU the task is on by then. And a
+	 * rebuild of the root domains meanwhile would not find the task
+	 * among the deadline tasks. So record a CPU of the root domain that
+	 * keeps the bandwidth until the timer removes it (dl_task_depart()).
+	 * Test dl_non_contending: it tells that the timer is armed and its
+	 * handler has not run. hrtimer_active() stays true for a while after
+	 * the handler returns.
 	 */
-	if (p->dl.dl_non_contending)
+	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
+		dl_task_depart(p, task_cpu(p));
+	}
 
 	/*
 	 * Since this might be the only -deadline task on the rq,
@@ -3656,6 +3844,17 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 static void switched_to_dl(struct rq *rq, struct task_struct *p)
 {
 	cancel_inactive_timer(&p->dl);
+	/*
+	 * A task on the departed list gets here when priority inheritance
+	 * boosts it back into the class before its 0-lag time. (A policy change
+	 * took it off the list in sched_dl_overflow().) The inactive timer no
+	 * longer removes its bandwidth: the task does when it leaves the class
+	 * again. So take it off the list. The task keeps the CPU recorded. It
+	 * takes the bandwidth from the root domain of that CPU, even if it
+	 * moved to another root domain meanwhile.
+	 */
+	if (!list_empty(&p->dl.dl_departed_node))
+		dl_task_undepart(p);
 
 	/*
 	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -3770,12 +3969,6 @@ DEFINE_SCHED_CLASS(dl) = {
 #endif
 };
 
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
 int sched_dl_global_validate(void)
 {
 	u64 runtime = global_rt_runtime();
@@ -3896,8 +4089,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 
 	if (dl_policy(policy) && !task_has_dl_policy(p) &&
 	    !__dl_overflow(dl_b, cap, 0, new_bw)) {
-		if (hrtimer_active(&p->dl.inactive_timer))
-			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
@@ -3923,6 +4114,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	}
 	raw_spin_unlock(&dl_b->lock);
 
+	/*
+	 * A task that comes back before its 0-lag time still has the
+	 * bandwidth it left with. The new bandwidth replaces it, so remove the
+	 * old one from the root domain that keeps it (dl_task_release_bw()).
+	 * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+	 */
+	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
+		lockdep_assert_cpuset_lock_held();
+		if (p->dl.dl_departed_cpu >= 0)
+			dl_task_release_bw(p, -1);
+		p->dl.dl_bw_gen = dl_bw_gen;
+	}
+
 	return err;
 }
 
@@ -4062,6 +4266,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
 	__dl_clear_params(dl_se);
+	INIT_LIST_HEAD(&dl_se->dl_departed_node);
+	dl_se->dl_departed_cpu = -1;
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 14/37] sched/deadline: Fix this_bw when a task is changed during a remote wakeup
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (12 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 13/37] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 15/37] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
                   ` (22 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

The this_bw of a runqueue is the bandwidth of the deadline tasks
assigned to it, running or not. GRUB uses it to compute the inactive
utilization of the runqueue. When a task wakes up on another CPU,
migrate_task_rq_dl() takes its bandwidth off the old runqueue. The
enqueue adds it to the new one.

Between these two steps, no runqueue counts the bandwidth.
try_to_wake_up() can leave the enqueue to the wake list of the new CPU,
and return before it runs. task_cpu() already points at the new CPU. A
change of the task in that window gets this_bw wrong.

In this example, T is a deadline task. S is another task. It changes
the bandwidth of T from "old bw" to "new bw":

  CPU 0                               CPU 1                       CPU 2
  T blocks
  this_bw of CPU 0 counts T
  <timer> try_to_wake_up(T)
    set_task_cpu(T, 1)
      migrate_task_rq_dl()
        sub_rq_bw(): CPU 0 drops T
    T on the wake list of CPU 1
    return
                                                                  S: sched_setattr(T)
                                                                    task_rq_lock(T): CPU 1
                                                                    T is not queued
                                                                    dl_change_utilization()
                                                                      this_bw of CPU 1:
                                                                      - old bw + new bw
                                      sched_ttwu_pending()
                                        enqueue_task_dl(T)
                                          task_contending()
                                            add_rq_bw():
                                            this_bw of CPU 1:
                                            + new bw

CPU 1 then counts the new bandwidth twice, less the old one. Repeated,
this makes this_bw drift until it underflows. If T left the class in
that window, switched_from_dl() would take its bandwidth off CPU 1 the
same way. A task that enters the class in that window would be counted
twice, by switched_to_dl() and by the enqueue.

This needs a wakeup that moves the task, so a root domain of more than
one CPU, with busy CPUs. CONFIG_SCHED_PROXY_EXEC=y makes such wakeups
more common. In testing, a task sleeps and wakes in a busy two-CPU
partition root, and another CPU changes its runtime. The kernel warns
within a second in each of 8 VMs:

  WARNING: kernel/sched/deadline.c:352 at dl_rq_change_utilization+0x71/0x2e0, CPU#0: wake_race/109
   __sub_rq_bw(): this_bw underflow
  WARNING: kernel/sched/deadline.c:316 at task_contending+0x244/0x260, CPU#6: swapper/6/0
   __add_running_bw(): running_bw overflow

Fix this with a new bit, dl_bw_oncpu, that tells whether a runqueue
counts the bandwidth of the task. add_rq_bw() adds the bandwidth only if
no runqueue counts it. sub_rq_bw() takes it off only if one does.
dl_change_utilization() does nothing while no runqueue counts the
bandwidth. The enqueue adds the new one.

With this patch, none of the 8 VMs warns. deadline_bw_test, added later
in this series, tests this in params_waking and random_mix. The Sashiko
AI review of the first posting found the double count, and a bit that
the first version of this patch left set.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h   |  5 +++++
 kernel/sched/deadline.c | 39 +++++++++++++++++++++++++++++++++++----
 2 files changed, 40 insertions(+), 4 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 063e49690658..33e2d9492aa2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -711,6 +711,10 @@ struct sched_dl_entity {
 	 * contributes to the root domain's total_bw. Only meaningful for server
 	 * entities (@dl_server == 1). Allows toggling the reservation on/off
 	 * without losing the configured @dl_runtime/@dl_period.
+	 *
+	 * @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
+	 * @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
+	 * runqueue's lock. Not for servers, which keep @dl_bw_attached.
 	 */
 	unsigned int			dl_throttled      : 1;
 	unsigned int			dl_yielded        : 1;
@@ -724,6 +728,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer_running  : 1;
 	unsigned int			dl_defer_idle     : 1;
 	unsigned int			dl_bw_attached    : 1;
+	unsigned int			dl_bw_oncpu       : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index cbdfa0793703..6444c4550936 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -411,18 +411,35 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
 	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
 }
 
+/*
+ * @dl_se->dl_bw_oncpu tells whether the this_bw of a runqueue counts the
+ * bandwidth of @dl_se. add_rq_bw() adds the bandwidth only if no runqueue
+ * counts it. sub_rq_bw() subtracts it only if one does.
+ *
+ * A wakeup on another CPU takes the bandwidth off the old runqueue. The
+ * enqueue on the new CPU adds it back later. A change of the task in
+ * between must not count the bandwidth twice, nor take it off a runqueue
+ * that does not count it.
+ *
+ * The servers track their bandwidth with dl_bw_attached. They call
+ * __add_rq_bw() and __sub_rq_bw() directly.
+ */
 static inline
 void add_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__add_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || dl_se->dl_bw_oncpu)
+		return;
+	__add_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 1;
 }
 
 static inline
 void sub_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || !dl_se->dl_bw_oncpu)
+		return;
+	__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 0;
 }
 
 static inline
@@ -491,6 +508,14 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 	if (task_on_rq_queued(p))
 		return;
 
+	/*
+	 * The task is waking up on another CPU, and no runqueue counts its
+	 * bandwidth. The enqueue adds it, with the new parameters that
+	 * __setparam_dl() sets right after this.
+	 */
+	if (!p->dl.dl_bw_oncpu)
+		return;
+
 	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
 }
 
@@ -2865,6 +2890,11 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
 		 */
 		cancel_inactive_timer(&p->dl);
 	}
+	/*
+	 * set_task_cpu() points task_cpu() at the new CPU next. The runqueue
+	 * of that CPU counts this bandwidth only once the enqueue there adds
+	 * it. Until then, no runqueue counts it, as dl_bw_oncpu now says.
+	 */
 	sub_rq_bw(&p->dl, &rq->dl);
 	rq_unlock(rq, &rf);
 }
@@ -4254,6 +4284,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer			= 0;
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
+	dl_se->dl_bw_oncpu		= 0;
 
 #ifdef CONFIG_RT_MUTEXES
 	dl_se->pi_se			= dl_se;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 15/37] sched/deadline: Rebuild the root domains' accounting from the task list
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (13 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 14/37] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
                   ` (21 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A rebuild of the root domains, as on CPU hotplug, clears their deadline
accounting. dl_rebuild_rd_accounting() then adds back the deadline tasks
of each cpuset. Without CONFIG_CPUSETS, it is a stub. So the first
hotplug drops every deadline task's bandwidth, and their releases wrap
total_bw around. In testing, a root domain admitted 9 tasks of 10ms
every 100ms before a hotplug with deadline sleepers, and none after.

With cpusets, the rebuild skips a cpuset whose nr_deadline_tasks is 0.
This count spares suspend and resume a costly walk of all tasks. An
attach moves it by a tally taken in cpuset_can_attach(). A task that
leaves the class between the tally and the move is taken off the old
cpuset twice. Its count can then reach 0 with a deadline task in it.
Every rebuild would then drop that task's bandwidth.

Fix both by walking the task list, with or without cpusets. The walk
finds every deadline task, so the count and its hooks go. It holds
tasklist_lock for reading, so fork and exit wait meanwhile. It takes
the pi_lock of deadline tasks only. The walk that the count avoided
took the pi_lock of every task. It leaves out an exited task with a
dl_bw of 0. It also leaves out a zombie whose bandwidth an earlier
rebuild dropped, as dl_clear_root_domains() does.

The walk visits every thread, and a suspend or a resume runs one
rebuild for each CPU. So keep one count for the whole system,
dl_nr_bw_tasks: the number of tasks whose dl_bw is not 0.
__setparam_dl() and __dl_clear_params() are the only places that change
the dl_bw of a task, and they update the count. With a count of 0, the
rebuild does not walk the task list, as Juri Lelli suggested.

In testing, a 64-vCPU guest held 10000 threads and no deadline task. A
rebuild took 0.7 to 1.2 ms without the count, and 0.03 ms with it.

Without cpusets, a new dl_accounting_mutex keeps admissions out of a
rebuild, as cpuset_mutex does with them.

Three hotplug cases of deadline_bw_test, added later in this series,
fail without cpusets. With this patch, its five hotplug cases pass, in 4
VMs. Its death_zombie_hotplug case tests the zombie. The Sashiko AI
reviews of v3, and of v4 before posting, found the wrong count and the
exited tasks. Its review of v4 found the zombie.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/cpuset.h          |   9 --
 include/linux/sched/deadline.h  |  16 ++++
 kernel/cgroup/cgroup.c          |   3 -
 kernel/cgroup/cpuset-internal.h |   6 +-
 kernel/cgroup/cpuset.c          |  67 +-------------
 kernel/sched/deadline.c         | 154 ++++++++++++++++++++++++++------
 kernel/sched/syscalls.c         |  14 +--
 kernel/sched/topology.c         |   8 ++
 8 files changed, 163 insertions(+), 114 deletions(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..db1cc2fc0324 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -72,8 +72,6 @@ extern int cpuset_init(void);
 extern void cpuset_init_smp(void);
 extern void cpuset_force_rebuild(void);
 extern void cpuset_update_active_cpus(void);
-extern void inc_dl_tasks_cs(struct task_struct *task);
-extern void dec_dl_tasks_cs(struct task_struct *task);
 extern void cpuset_lock(void);
 extern void cpuset_unlock(void);
 extern void lockdep_assert_cpuset_lock_held(void);
@@ -129,7 +127,6 @@ static inline int cpuset_do_page_mem_spread(void)
 
 extern bool current_cpuset_is_being_rebound(void);
 
-extern void dl_rebuild_rd_accounting(void);
 extern void rebuild_sched_domains(void);
 
 extern void cpuset_print_current_mems_allowed(void);
@@ -194,8 +191,6 @@ static inline void cpuset_update_active_cpus(void)
 	partition_sched_domains(1, NULL, NULL);
 }
 
-static inline void inc_dl_tasks_cs(struct task_struct *task) { }
-static inline void dec_dl_tasks_cs(struct task_struct *task) { }
 static inline void cpuset_lock(void) { }
 static inline void cpuset_unlock(void) { }
 static inline void lockdep_assert_cpuset_lock_held(void) { }
@@ -273,10 +268,6 @@ static inline bool current_cpuset_is_being_rebound(void)
 	return false;
 }
 
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
-
 static inline void rebuild_sched_domains(void)
 {
 	guard(cpus_read_lock)();
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdf39ac1d43d..60be226699bb 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,22 @@ struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
 extern void dl_clear_root_domains(void);
+extern void dl_rebuild_rd_accounting(void);
+/*
+ * Keep a rebuild of the root domains' accounting and an admission of
+ * bandwidth from running at the same time. With cpusets, cpuset_mutex
+ * does it, since every rebuild comes from the cpuset code. Without
+ * cpusets, dl_accounting_lock() takes a mutex of its own.
+ */
+#ifdef CONFIG_CPUSETS
+static inline void dl_accounting_lock(void) { }
+static inline void dl_accounting_unlock(void) { }
+static inline void lockdep_assert_dl_accounting_held(void) { }
+#else
+extern void dl_accounting_lock(void);
+extern void dl_accounting_unlock(void);
+extern void lockdep_assert_dl_accounting_held(void);
+#endif
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..2cfa770c2dcb 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -7240,9 +7240,6 @@ static void do_cgroup_task_dead(struct task_struct *tsk)
 	if (thread_group_leader(tsk) && atomic_read(&tsk->signal->live))
 		list_add_tail(&tsk->cg_list, &cset->dying_tasks);
 
-	if (dl_task(tsk))
-		dec_dl_tasks_cs(tsk);
-
 	WARN_ON_ONCE(cgroup_task_frozen(tsk));
 	if (unlikely(!(tsk->flags & PF_KTHREAD) &&
 		     test_bit(CGRP_FREEZE, &task_dfl_cgroup(tsk)->flags)))
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 1ba94ffaff9d..bf847953c78d 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,11 +166,7 @@ struct cpuset {
 	 */
 	bool remote_partition_updating;
 
-	/*
-	 * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
-	 * know when to rebuild associated root domain bandwidth information.
-	 */
-	atomic_t nr_deadline_tasks;
+	/* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
 	int nr_migrate_dl_tasks;
 	/* DL bandwidth that needs destination reservation for this attach. */
 	u64 sum_migrate_dl_bw;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 4159abfbbbb5..7628cf77bdfd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -225,20 +225,6 @@ struct tmpmasks {
 	cpumask_var_t new_cpus;		/* For update_cpumasks_hier() */
 };
 
-void inc_dl_tasks_cs(struct task_struct *p)
-{
-	struct cpuset *cs = task_cs(p);
-
-	atomic_inc(&cs->nr_deadline_tasks);
-}
-
-void dec_dl_tasks_cs(struct task_struct *p)
-{
-	struct cpuset *cs = task_cs(p);
-
-	atomic_dec(&cs->nr_deadline_tasks);
-}
-
 static inline bool is_partition_valid(const struct cpuset *cs)
 {
 	return cs->partition_root_state > 0;
@@ -965,54 +951,6 @@ static int generate_sched_domains(cpumask_var_t **domains,
 	return ndoms;
 }
 
-static void dl_update_tasks_root_domain(struct cpuset *cs)
-{
-	struct css_task_iter it;
-	struct task_struct *task;
-
-	if (atomic_read(&cs->nr_deadline_tasks) == 0)
-		return;
-
-	css_task_iter_start(&cs->css, 0, &it);
-
-	while ((task = css_task_iter_next(&it)))
-		dl_add_task_root_domain(task);
-
-	css_task_iter_end(&it);
-}
-
-void dl_rebuild_rd_accounting(void)
-{
-	struct cpuset *cs = NULL;
-	struct cgroup_subsys_state *pos_css;
-
-	lockdep_assert_cpuset_lock_held();
-	lockdep_assert_cpus_held();
-	lockdep_assert_held(&sched_domains_mutex);
-
-	dl_clear_root_domains();
-
-	rcu_read_lock();
-
-	cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
-
-		if (cpumask_empty(cs->effective_cpus)) {
-			pos_css = css_rightmost_descendant(pos_css);
-			continue;
-		}
-
-		css_get(&cs->css);
-
-		rcu_read_unlock();
-
-		dl_update_tasks_root_domain(cs);
-
-		rcu_read_lock();
-		css_put(&cs->css);
-	}
-	rcu_read_unlock();
-}
-
 /*
  * Rebuild scheduler domains.
  *
@@ -3181,9 +3119,7 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
 	llist_for_each_entry_safe(cs, next, lnode, attach_node) {
 		init_llist_node(&cs->attach_node);
 		if (cs->nr_migrate_dl_tasks) {
-			if (!cancel)
-				atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
-			else if (cs->dl_bw_cpu >= 0) /* && cancel */
+			if (cancel && cs->dl_bw_cpu >= 0)
 				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
 			cs->nr_migrate_dl_tasks = 0;
 			cs->sum_migrate_dl_bw = 0;
@@ -3272,7 +3208,6 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 			 * contribute to sum_migrate_dl_bw.
 			 */
 			cs->nr_migrate_dl_tasks++;
-			oldcs->nr_migrate_dl_tasks--;
 			if (dl_task_needs_bw_move(task, cs->effective_cpus))
 				cs->sum_migrate_dl_bw += task->dl.dl_bw;
 		}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 6444c4550936..ccd6426811b9 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -262,10 +262,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * The timer holds a reference to it.
  *
  * A rebuild of the root domains' accounting (dl_clear_root_domains() and
- * dl_add_task_root_domain()) does not find such a task among the deadline
- * tasks of the cpusets. So dl_departed_list holds these tasks, from their
- * departure to their timer. The rebuild puts their bandwidth back in the
- * root domain of the CPU each one was on when it left.
+ * dl_add_task_root_domain()) does not count such a task as a deadline
+ * task. So dl_departed_list holds these tasks, from their departure to
+ * their timer. The rebuild puts their bandwidth back in the root domain
+ * of the CPU each one was on when it left.
  *
  * dl_bw_gen numbers the rebuilds of the accounting: how many times the
  * root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -273,7 +273,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * state, counted or settled. That number dates the state:
  *
  *  - Current: the bandwidth is counted, or a release settled it since
- *    the rebuild began. A release subtracts; the walk of the cpusets
+ *    the rebuild began. A release subtracts; the walk of the task list
  *    skips.
  *  - One behind: the clear that just ran wiped it. The rebuild puts
  *    it back.
@@ -286,8 +286,8 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  *
  * A rebuild drops the bandwidth when the CPU recorded is no longer
  * active. It also drops the bandwidth of a task that dies while the
- * rebuild runs. The cpuset walk skips that task, and it was not dead yet
- * when the list was put back.
+ * rebuild runs. The walk of the task list does not find that task, and
+ * it was not dead yet when the list was put back.
  *
  * dl_departed_lock protects the list and both counts. Take it with
  * interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -297,6 +297,22 @@ static LIST_HEAD(dl_departed_list);
 static DEFINE_RAW_SPINLOCK(dl_departed_lock);
 static unsigned int dl_bw_gen;
 
+/*
+ * The number of tasks with a bandwidth: those whose dl_bw is not 0. A
+ * special entity and a server are not counted. __setparam_dl() and
+ * __dl_clear_params() are the only places that change the dl_bw of a
+ * task, and they update the count. With a count of 0, a rebuild of the
+ * accounting has no task to count, and does not walk the task list.
+ */
+static atomic_t dl_nr_bw_tasks;
+
+/* Return true if @dl_se is a task that holds a bandwidth. */
+static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
+{
+	return dl_se->dl_bw && !dl_se->dl_server &&
+	       !(dl_se->flags & SCHED_FLAG_SUGOV);
+}
+
 /*
  * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
  * keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
@@ -3638,7 +3654,7 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	cpuset_cpus_allowed_locked(p, cpus);
 }
 
-/* The caller should hold cpuset_mutex */
+/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
 void dl_add_task_root_domain(struct task_struct *p)
 {
 	struct rq_flags rf;
@@ -3648,7 +3664,13 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct cpumask *msk;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
-	if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
+	/*
+	 * Count nothing for a special entity, or for a task with no
+	 * bandwidth. The walk of the task list also finds exited tasks. An
+	 * exited task whose bandwidth went with its parameters has a dl_bw
+	 * of 0.
+	 */
+	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 		return;
 	}
@@ -3671,6 +3693,14 @@ void dl_add_task_root_domain(struct task_struct *p)
 	if (p->dl.dl_bw_gen == dl_bw_gen)
 		goto unlock;
 	if (p->dl.dl_departed_cpu >= 0) {
+		/*
+		 * The walk also finds a dead task that departed before an
+		 * earlier rebuild, if it is not reaped yet. If that rebuild
+		 * dropped its bandwidth, dl_clear_root_domains() skipped it,
+		 * and its release removes nothing. Skip it here too.
+		 */
+		if (p->dl.dl_bw_gen != dl_bw_gen - 1)
+			goto unlock;
 		cpu = p->dl.dl_departed_cpu;
 		if (!cpu_active(cpu))
 			goto unlock;
@@ -3741,13 +3771,80 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
+#ifndef CONFIG_CPUSETS
+/*
+ * With cpusets, cpuset_mutex keeps an admission of bandwidth from running
+ * against a rebuild of the accounting. __sched_setscheduler() takes it for
+ * every change of or into SCHED_DEADLINE. The callers of
+ * partition_sched_domains() hold it from the attachment of the new root
+ * domains to the rebuild. Without cpusets, cpuset_lock() is a stub, and
+ * this mutex takes its place in both.
+ */
+static DEFINE_MUTEX(dl_accounting_mutex);
+
+void dl_accounting_lock(void)
+{
+	mutex_lock(&dl_accounting_mutex);
+}
+
+void dl_accounting_unlock(void)
+{
+	mutex_unlock(&dl_accounting_mutex);
+}
+
+void lockdep_assert_dl_accounting_held(void)
+{
+	lockdep_assert_held(&dl_accounting_mutex);
+}
+#endif /* !CONFIG_CPUSETS */
+
+/*
+ * Rebuild the root domains' accounting. Clear it, then walk all tasks and
+ * count each deadline task again. The caller holds the locks that this
+ * function asserts.
+ */
+void dl_rebuild_rd_accounting(void)
+{
+	struct task_struct *g, *p;
+
+	lockdep_assert_cpus_held();
+	lockdep_assert_held(&sched_domains_mutex);
+	lockdep_assert_cpuset_lock_held();
+	lockdep_assert_dl_accounting_held();
+
+	dl_clear_root_domains();
+
+	/*
+	 * Without a task that holds a bandwidth, there is nothing to count.
+	 * The walk would still visit every thread of the system. A CPU
+	 * that goes offline or comes back runs a rebuild, so a suspend and
+	 * a resume run one for each CPU. A task gets a bandwidth under the
+	 * locks this rebuild holds, so the count cannot leave 0 meanwhile.
+	 */
+	if (!atomic_read(&dl_nr_bw_tasks))
+		return;
+
+	read_lock(&tasklist_lock);
+	for_each_process_thread(g, p) {
+		/*
+		 * Test the class without the task's locks, and take them only
+		 * for a deadline task. A task enters the class under the locks
+		 * this rebuild holds, or by a boost. A boost brings no
+		 * bandwidth of its own to count.
+		 */
+		if (dl_task(p))
+			dl_add_task_root_domain(p);
+	}
+	read_unlock(&tasklist_lock);
+}
+
 /*
  * Clear every root domain before a rebuild of their accounting, and count
  * the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
- * back the bandwidth of the tasks on dl_departed_list, which no cpuset
- * holds as deadline tasks. Put each in the root domain of the CPU
- * recorded for it, or drop it if that CPU is not active.
- * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ * back the bandwidth of the tasks on dl_departed_list, which the walk of
+ * the deadline tasks does not count. Put each in the root domain of the
+ * CPU recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks.
  */
 void dl_clear_root_domains(void)
 {
@@ -3811,12 +3908,6 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 		p->dl.runtime = 0;
 	}
 
-	/*
-	 * In case a task is setscheduled out from SCHED_DEADLINE we need to
-	 * keep track of that on its cpuset (for correct bandwidth tracking).
-	 */
-	dec_dl_tasks_cs(p);
-
 	if (!task_on_rq_queued(p)) {
 		/*
 		 * Inactive timer is armed. However, p is leaving DEADLINE and
@@ -3886,12 +3977,6 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	if (!list_empty(&p->dl.dl_departed_node))
 		dl_task_undepart(p);
 
-	/*
-	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
-	 * track of that on its cpuset (for correct bandwidth tracking).
-	 */
-	inc_dl_tasks_cs(p);
-
 	/* If p is not queued we will update its parameters at next wakeup. */
 	if (!task_on_rq_queued(p)) {
 		add_rq_bw(&p->dl, &rq->dl);
@@ -4148,10 +4233,12 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 * A task that comes back before its 0-lag time still has the
 	 * bandwidth it left with. The new bandwidth replaces it, so remove the
 	 * old one from the root domain that keeps it (dl_task_release_bw()).
-	 * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+	 * No rebuild can run meanwhile: the caller holds cpuset_mutex, or
+	 * dl_accounting_mutex without cpusets.
 	 */
 	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
 		lockdep_assert_cpuset_lock_held();
+		lockdep_assert_dl_accounting_held();
 		if (p->dl.dl_departed_cpu >= 0)
 			dl_task_release_bw(p, -1);
 		p->dl.dl_bw_gen = dl_bw_gen;
@@ -4171,12 +4258,18 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	bool held;
 
+	held = dl_se_holds_bw(dl_se);
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
 	dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
 	dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
 	dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+	if (dl_se_holds_bw(dl_se) && !held)
+		atomic_inc(&dl_nr_bw_tasks);
+	else if (!dl_se_holds_bw(dl_se) && held)
+		atomic_dec(&dl_nr_bw_tasks);
 	dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
 }
 
@@ -4266,6 +4359,9 @@ bool __checkparam_dl(const struct sched_attr *attr)
  */
 static void __dl_clear_params(struct sched_dl_entity *dl_se)
 {
+	if (dl_se_holds_bw(dl_se))
+		WARN_ON_ONCE(atomic_dec_return(&dl_nr_bw_tasks) < 0);
+
 	dl_se->dl_runtime		= 0;
 	dl_se->dl_deadline		= 0;
 	dl_se->dl_period		= 0;
@@ -4296,6 +4392,12 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	RB_CLEAR_NODE(&dl_se->rb_node);
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
+	/*
+	 * A new task has a copy of the entity of its parent. That copy
+	 * holds no bandwidth, and dl_nr_bw_tasks does not count it. Clear
+	 * dl_bw first, so that __dl_clear_params() leaves the count alone.
+	 */
+	dl_se->dl_bw = 0;
 	__dl_clear_params(dl_se);
 	INIT_LIST_HEAD(&dl_se->dl_departed_node);
 	dl_se->dl_departed_cpu = -1;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 8c72e751ff4e..4c013caa2471 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -568,6 +568,7 @@ int __sched_setscheduler(struct task_struct *p,
 	if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
 		cpuset_locked = true;
 		cpuset_lock();
+		dl_accounting_lock();
 	}
 
 	/*
@@ -596,9 +597,7 @@ int __sched_setscheduler(struct task_struct *p,
 	 * Reject SCHED_DEADLINE on exited tasks. A zombie will never be
 	 * scheduled again, so sched_dl_overflow() would charge total_bw
 	 * that nothing will ever release. And reject a change of policy of
-	 * one in the class: cgroup_task_dead() took it off the count of
-	 * deadline tasks of its cpuset, which switched_from_dl() would
-	 * decrement a second time.
+	 * one in the class: its bandwidth went at its last schedule.
 	 */
 	if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
 		retval = -ESRCH;
@@ -669,6 +668,7 @@ int __sched_setscheduler(struct task_struct *p,
 		policy = oldpolicy = -1;
 		task_rq_unlock(rq, p, &rf);
 		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
 			cpuset_locked = false;
 		}
@@ -779,8 +779,10 @@ int __sched_setscheduler(struct task_struct *p,
 	task_rq_unlock(rq, p, &rf);
 
 	if (pi) {
-		if (cpuset_locked)
+		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
+		}
 		rt_mutex_adjust_pi(p);
 	}
 
@@ -792,8 +794,10 @@ int __sched_setscheduler(struct task_struct *p,
 
 unlock:
 	task_rq_unlock(rq, p, &rf);
-	if (cpuset_locked)
+	if (cpuset_locked) {
+		dl_accounting_unlock();
 		cpuset_unlock();
+	}
 	return retval;
 }
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 9d3a1b8dd393..35e56a45c238 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3528,7 +3528,15 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
 void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[],
 			     struct sched_domain_attr *dattr_new)
 {
+	/*
+	 * The root domains attached here count no task's bandwidth until
+	 * dl_rebuild_rd_accounting() at the end. No admission of bandwidth may
+	 * run meanwhile. With cpusets, the callers hold cpuset_mutex, which
+	 * keeps admissions out. Without cpusets, dl_accounting_lock() does it.
+	 */
+	dl_accounting_lock();
 	sched_domains_mutex_lock();
 	partition_sched_domains_locked(ndoms_new, doms_new, dattr_new);
 	sched_domains_mutex_unlock();
+	dl_accounting_unlock();
 }
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (14 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 15/37] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:44   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 17/37] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
                   ` (20 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Admission, moves and releases of a deadline task's bandwidth find the
root domain that counts it from the CPU the task is on. That CPU can be
in another root domain. Two cases show it.

Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the root domain of the
task's runqueue. Since commit 5dd74441cbf4 ("cgroup/cpuset: Reserve DL
bandwidth only for root-domain moves"), both halves move nothing when
that root domain overlaps the destination. But a sleeping task stays on
the CPU it last ran on. So a sleeping task moved to a partition and back
leaves its bandwidth in the partition:

  admin                                       task T

  makes P a partition root of CPU 3
    CPU 3 is now in root domain B
    CPUs 0-2 are in root domain A
                                              sched_setattr(SCHED_DEADLINE)
                                                A: total_bw += bw of T
                                              runs on CPU 0, then sleeps
  echo T > P/cgroup.procs
    cpuset_can_attach()
      dl_task_needs_bw_move(): true,
      A does not overlap P
      B: total_bw += bw of T
    set_cpus_allowed_dl()
      dl_task_needs_bw_move(): true
      A: total_bw -= bw of T
                                              still asleep on CPU 0
  echo T > cgroup.procs of the root cgroup
    cpuset_can_attach()
      dl_task_needs_bw_move(): false,
      A overlaps the root cpuset
    set_cpus_allowed_dl()
      dl_task_needs_bw_move(): false
  kill -9 T
                                              wakes up on a CPU of A
                                              exits
                                                A: total_bw -= bw of T

B keeps the bandwidth of T. A does not count T while T is in A, so A
admits beyond its capacity. After T exits, the total_bw of A is short
by the bandwidth of T, and can underflow.

The second case needs no move:

 1. A deadline task T sleeps on a CPU of root domain A.
 2. That CPU goes offline, and joins the default root domain.
 3. The rebuild counts T in A, the root domain of its cpuset.
 4. sched_setattr() takes the sleeping T out of SCHED_DEADLINE.
 5. The release takes the bandwidth from the root domain of the CPU T
    is on: the default one.

A keeps the bandwidth of T until the next rebuild, and refuses deadline
tasks that would fit.

The first case was reproduced on v7.4-rc1. The second needs the earlier
patches of this series: without "sched/deadline: Release the bandwidth
of a task that leaves the class asleep", nothing releases the bandwidth
of that sleeping task. The hotplug_sleeper_switched case of
deadline_bw_test, added later in this series, tests it. Without this
patch and with the CPU still offline, A admitted 63 tasks of 10% where
it had room for 66, in 3 guests of 3.

Fix this by recording, in dl_bw_cpu, a CPU of the root domain that
counts the bandwidth. The record replaces dl_departed_cpu. It is used
as follows:
- Each site that counts the bandwidth sets the record.
- Each release or move reads it. The release clears it.
- If dl_bw_gen shows that a rebuild dropped the bandwidth, nothing is
  taken back. set_cpus_allowed_dl() makes that test under
  dl_departed_lock, so that a rebuild cannot change dl_bw_gen
  meanwhile.
- A move skips a bandwidth of 0, and does not update its record. So
  sched_dl_overflow() uses the CPU the task is on for it. The
  move_zero_bw case of deadline_bw_test, added later in this series,
  tests this.
- A rebuild drops departed bandwidth whose CPU is inactive. So a
  departing task records the CPU it departs from, if that CPU is
  active and in the same root domain.

The affinity check of sched_setaffinity() reads the record too. A task
moved asleep is still on a CPU of the root domain it left. Testing that
root domain let a new mask take the task out of the one that counts it.
The affinity_moved_asleep case tests this.

A move now leaves out a sugov kthread and a task boosted into the class
with no bandwidth of its own. The rebuild leaves them out too: no root
domain counts theirs. A move would take a sugov kthread's bandwidth
from a root domain that never held it.

Weiheng Qiu reported the moved sleeper, and Juri Lelli analysed it. His
fix, linked below, took the first active CPU of the task's mask. A mask
can have none during CPU hotplug. Tip dropped the fix after a sugov
kthread hit its BUG_ON() there. The Sashiko AI reviews of this series
found the other cases.

Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/14508cb2-805f-41ac-9f6b-bfd7c3d7cb1c@gmail.com/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/
---
 include/linux/sched.h   |  10 +-
 kernel/sched/deadline.c | 247 +++++++++++++++++++++++++++-------------
 kernel/sched/syscalls.c |  11 +-
 3 files changed, 187 insertions(+), 81 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 33e2d9492aa2..4a1cb869facc 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -750,15 +750,17 @@ struct sched_dl_entity {
 	 * time" keeps its bandwidth in its root domain until the inactive
 	 * timer removes it. Meanwhile, @dl_departed_node links the task on
 	 * the list that a rebuild of the root domains' accounting reads.
-	 * @dl_departed_cpu is a CPU of the root domain that counts the
-	 * bandwidth. It stays recorded while priority inheritance boosts the
-	 * task back into the class. Otherwise @dl_departed_cpu is -1.
+	 * Otherwise that node is empty.
+	 *
+	 * @dl_bw_cpu is a CPU of the root domain that counts the bandwidth,
+	 * or -1. Each site that counts the bandwidth sets it. Each site that
+	 * removes or moves the bandwidth reads it.
 	 *
 	 * @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
 	 * counted. If a rebuild dropped it since, there is nothing to remove.
 	 */
 	struct list_head		dl_departed_node;
-	int				dl_departed_cpu;
+	int				dl_bw_cpu;
 	unsigned int			dl_bw_gen;
 
 	/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ccd6426811b9..44db625bffc3 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -265,7 +265,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * dl_add_task_root_domain()) does not count such a task as a deadline
  * task. So dl_departed_list holds these tasks, from their departure to
  * their timer. The rebuild puts their bandwidth back in the root domain
- * of the CPU each one was on when it left.
+ * of the CPU recorded with each task's bandwidth.
  *
  * dl_bw_gen numbers the rebuilds of the accounting: how many times the
  * root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -287,7 +287,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * A rebuild drops the bandwidth when the CPU recorded is no longer
  * active. It also drops the bandwidth of a task that dies while the
  * rebuild runs. The walk of the task list does not find that task, and
- * it was not dead yet when the list was put back.
+ * it was not dead yet when the list was put back. The same happens to a
+ * task that priority inheritance deboosts while the rebuild runs. It was
+ * in the class when the list was put back, and out of it when the walk
+ * reaches it.
  *
  * dl_departed_lock protects the list and both counts. Take it with
  * interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -314,66 +317,75 @@ static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
 }
 
 /*
- * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * Put @p on dl_departed_list. The root domain of the CPU recorded with it
  * keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
- * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
- * recorded if @p has one: priority inheritance boosted @p back into the
- * class and its boost ends. The caller holds the rq lock of @p.
+ * 0-lag time. Do nothing if @p is on the list already. The caller holds
+ * the rq lock of @p.
  */
-static void dl_task_depart(struct task_struct *p, int cpu)
+static void dl_task_depart(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	int cpu = task_cpu(p);
 
 	lockdep_assert_irqs_disabled();
 	guard(raw_spinlock)(&dl_departed_lock);
-	if (!list_empty(&dl_se->dl_departed_node))
+	if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
 		return;
-	if (dl_se->dl_departed_cpu < 0)
-		dl_se->dl_departed_cpu = cpu;
+	/*
+	 * A rebuild drops the bandwidth if the CPU recorded is not active.
+	 * That CPU can be one that @p left long ago. Record the CPU @p
+	 * departs from instead, if it is active and the same root domain
+	 * counts the bandwidth.
+	 */
+	if (dl_se->dl_bw_gen == dl_bw_gen && cpu_active(cpu) &&
+	    cpu_rq(cpu)->rd == cpu_rq(dl_se->dl_bw_cpu)->rd)
+		dl_se->dl_bw_cpu = cpu;
 	list_add(&dl_se->dl_departed_node, &dl_departed_list);
 }
 
 /*
- * Take @p off dl_departed_list. Priority inheritance boosts @p back into
- * the deadline class. Keep the CPU recorded: its root domain still counts
- * the bandwidth, wherever @p runs now. The caller holds the rq lock of
- * @p.
+ * Take @p off dl_departed_list. @p comes back to the deadline class, and
+ * its bandwidth is that of a deadline task again. The caller holds the rq
+ * lock of @p.
  */
 static void dl_task_undepart(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 
 	lockdep_assert_irqs_disabled();
+	if (list_empty(&dl_se->dl_departed_node))
+		return;
 	guard(raw_spinlock)(&dl_departed_lock);
 	list_del_init(&dl_se->dl_departed_node);
 }
 
 /*
  * Remove the bandwidth of @p from the root domain that counts it. That is
- * the root domain of the CPU recorded when @p departed, or else of @cpu.
- * Clear the record. Take @p off dl_departed_list. Remove nothing if a
- * rebuild dropped the bandwidth since it was counted. Then record the
- * current dl_bw_gen in @p, so that a rebuild in progress does not count it
- * again. The caller holds the rq lock of @p.
+ * the root domain of the CPU recorded with @p. Clear the record.
+ * Take @p off dl_departed_list. Remove nothing if a rebuild dropped the
+ * bandwidth since it was counted. Then record the current dl_bw_gen in
+ * @p, so that a rebuild in progress does not count it again. The caller
+ * holds the rq lock of @p.
  */
-static void dl_task_release_bw(struct task_struct *p, int cpu)
+static void dl_task_release_bw(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 	struct dl_bw *dl_b;
+	int cpu;
 
 	lockdep_assert_irqs_disabled();
 	guard(raw_spinlock)(&dl_departed_lock);
-	if (dl_se->dl_departed_cpu >= 0) {
-		cpu = dl_se->dl_departed_cpu;
-		dl_se->dl_departed_cpu = -1;
+	if (!list_empty(&dl_se->dl_departed_node))
 		list_del_init(&dl_se->dl_departed_node);
-	}
-	if (dl_se->dl_bw_gen == dl_bw_gen) {
+	cpu = dl_se->dl_bw_cpu;
+	WARN_ON_ONCE(dl_se->dl_bw && cpu < 0);
+	if (cpu >= 0 && dl_se->dl_bw_gen == dl_bw_gen) {
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
 		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
 		raw_spin_unlock(&dl_b->lock);
 	}
+	dl_se->dl_bw_cpu = -1;
 	dl_se->dl_bw_gen = dl_bw_gen;
 }
 
@@ -536,6 +548,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p);
 
 /*
  * The utilization of a task cannot be immediately removed from
@@ -636,7 +649,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				dl_task_release_bw(p, task_cpu(p));
+				dl_task_release_bw(p);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -650,12 +663,13 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 
 		get_task_struct(p);
 		/*
-		 * A dying task keeps its bandwidth in the root domain of the
-		 * CPU it runs on. Record that CPU. switched_from_dl() records
-		 * a task that leaves SCHED_DEADLINE.
+		 * A dying task keeps its bandwidth until its 0-lag time. Put
+		 * it on the departed list. The CPU recorded with its bandwidth
+		 * tells the root domain that counts it. switched_from_dl() does
+		 * the same for a task that leaves SCHED_DEADLINE.
 		 */
 		if (READ_ONCE(p->__state) == TASK_DEAD)
-			dl_task_depart(p, task_cpu(p));
+			dl_task_depart(p);
 	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
@@ -893,19 +907,31 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 	}
 
 	/*
-	 * And we finally need to fix up root_domain(s) bandwidth accounting,
-	 * since p is still hanging out in the old (now moved to default) root
-	 * domain.
+	 * And we finally need to fix up root_domain(s) bandwidth accounting.
+	 * The bandwidth of @p is counted in the root domain of the CPU
+	 * recorded with it. That is not the root domain of @rq if @p was
+	 * moved to another cpuset while it was not running. If a rebuild
+	 * dropped the bandwidth, there is nothing to move: the next rebuild
+	 * counts it.
 	 */
-	dl_b = &rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_sub(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
-
-	dl_b = &later_rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
+	scoped_guard (raw_spinlock, &dl_departed_lock) {
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
+
+			dl_b = &src_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw,
+				 cpumask_weight(src_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+
+			dl_b = &later_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_add(dl_b, p->dl.dl_bw,
+				 cpumask_weight(later_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+		}
+		p->dl.dl_bw_cpu = later_rq->cpu;
+	}
 
 	set_task_cpu(p, later_rq->cpu);
 	double_unlock_balance(later_rq, rq);
@@ -2407,7 +2433,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 			dl_se->dl_non_contending = 0;
 		}
 
-		dl_task_release_bw(p, task_cpu(p));
+		dl_task_release_bw(p);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3555,14 +3581,26 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
 	}
 }
 
+/*
+ * Return a CPU of the root domain that counts the bandwidth of @p: the
+ * CPU recorded with it. Without a record, no root domain counts that
+ * bandwidth. Return the CPU @p is on then. Read the record once:
+ * dl_task_check_affinity() holds no lock.
+ */
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
+{
+	int cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+	if (cpu < 0)
+		return task_cpu(p);
+	return cpu;
+}
+
 static void set_cpus_allowed_dl(struct task_struct *p,
 				struct affinity_context *ctx)
 {
-	struct rq *rq;
-
 	WARN_ON_ONCE(!dl_task(p));
 
-	rq = task_rq(p);
 	/*
 	 * Migrating a SCHED_DEADLINE task between exclusive
 	 * cpusets (different root_domains) entails a bandwidth
@@ -3570,17 +3608,31 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * domain (see cpuset_can_attach()).
 	 */
 	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
-		struct dl_bw *src_dl_b;
+		unsigned int src_cpu = dl_task_root_domain_cpu(p);
+		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
+		unsigned int cpu;
 
-		src_dl_b = dl_bw_of(cpu_of(rq));
 		/*
 		 * We now free resources of the root_domain we are migrating
 		 * off. In the worst case, sched_setattr() may temporary fail
-		 * until we complete the update.
+		 * until we complete the update. A rebuild that dropped the
+		 * bandwidth of @p leaves nothing to free. dl_departed_lock
+		 * keeps a rebuild from changing dl_bw_gen meanwhile.
 		 */
-		raw_spin_lock(&src_dl_b->lock);
-		__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-		raw_spin_unlock(&src_dl_b->lock);
+		guard(raw_spinlock)(&dl_departed_lock);
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			raw_spin_lock(&src_dl_b->lock);
+			__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+			raw_spin_unlock(&src_dl_b->lock);
+		}
+
+		/*
+		 * cpuset_can_attach() made room in the root domain of the new
+		 * mask. That root domain counts the bandwidth from now on.
+		 */
+		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
+		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
+		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
 	set_cpus_allowed_common(p, ctx);
@@ -3589,10 +3641,18 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 bool dl_task_needs_bw_move(struct task_struct *p,
 			   const struct cpumask *new_mask)
 {
-	if (!dl_task(p))
+	unsigned int cpu;
+
+	/*
+	 * Nothing to move for a special entity, or for a task boosted into
+	 * the class with no bandwidth of its own. No root domain counts
+	 * their bandwidth: dl_add_task_root_domain() leaves them out.
+	 */
+	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
 		return false;
 
-	return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
+	cpu = dl_task_root_domain_cpu(p);
+	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
 }
 
 /* Assumes rq->lock is held */
@@ -3668,7 +3728,8 @@ void dl_add_task_root_domain(struct task_struct *p)
 	 * Count nothing for a special entity, or for a task with no
 	 * bandwidth. The walk of the task list also finds exited tasks. An
 	 * exited task whose bandwidth went with its parameters has a dl_bw
-	 * of 0.
+	 * of 0. So has a task boosted into the class with no bandwidth of
+	 * its own, which gets no record.
 	 */
 	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
@@ -3692,7 +3753,9 @@ void dl_add_task_root_domain(struct task_struct *p)
 	 */
 	if (p->dl.dl_bw_gen == dl_bw_gen)
 		goto unlock;
-	if (p->dl.dl_departed_cpu >= 0) {
+	if (!list_empty(&p->dl.dl_departed_node)) {
+		int recorded = p->dl.dl_bw_cpu;
+
 		/*
 		 * The walk also finds a dead task that departed before an
 		 * earlier rebuild, if it is not reaped yet. If that rebuild
@@ -3701,9 +3764,9 @@ void dl_add_task_root_domain(struct task_struct *p)
 		 */
 		if (p->dl.dl_bw_gen != dl_bw_gen - 1)
 			goto unlock;
-		cpu = p->dl.dl_departed_cpu;
-		if (!cpu_active(cpu))
+		if (recorded < 0 || !cpu_active(recorded))
 			goto unlock;
+		cpu = recorded;
 	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
@@ -3711,6 +3774,7 @@ void dl_add_task_root_domain(struct task_struct *p)
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_cpu = cpu;
 	p->dl.dl_bw_gen = dl_bw_gen;
 unlock:
 	raw_spin_unlock(&dl_departed_lock);
@@ -3860,7 +3924,7 @@ void dl_clear_root_domains(void)
 			dl_clear_root_domain(cpu_rq(cpu)->rd);
 	}
 	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
-		cpu = dl_se->dl_departed_cpu;
+		cpu = dl_se->dl_bw_cpu;
 		/*
 		 * Skip bandwidth that an earlier rebuild dropped. It is not
 		 * counted now, and its release removes nothing
@@ -3875,7 +3939,7 @@ void dl_clear_root_domains(void)
 		 * from the recorded number. The release then removes
 		 * nothing, and no later rebuild restores it.
 		 */
-		if (!cpu_active(cpu))
+		if (cpu < 0 || !cpu_active(cpu))
 			continue;
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
@@ -3936,15 +4000,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 * For the same reason, the timer cannot take the task's bandwidth
 	 * from the root domain of the CPU the task is on by then. And a
 	 * rebuild of the root domains meanwhile would not find the task
-	 * among the deadline tasks. So record a CPU of the root domain that
-	 * keeps the bandwidth until the timer removes it (dl_task_depart()).
+	 * among the deadline tasks. So put the task on the departed list
+	 * until the timer removes the bandwidth (dl_task_depart()). The CPU
+	 * recorded with the bandwidth tells the root domain that keeps it.
 	 * Test dl_non_contending: it tells that the timer is armed and its
 	 * handler has not run. hrtimer_active() stays true for a while after
 	 * the handler returns.
 	 */
 	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
-		dl_task_depart(p, task_cpu(p));
+		dl_task_depart(p);
 	}
 
 	/*
@@ -3972,10 +4037,11 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	 * longer removes its bandwidth: the task does when it leaves the class
 	 * again. So take it off the list. The task keeps the CPU recorded. It
 	 * takes the bandwidth from the root domain of that CPU, even if it
-	 * moved to another root domain meanwhile.
+	 * moved to another root domain meanwhile. A rebuild while it is
+	 * boosted counts it like any deadline task, and records the CPU it
+	 * picks.
 	 */
-	if (!list_empty(&p->dl.dl_departed_node))
-		dl_task_undepart(p);
+	dl_task_undepart(p);
 
 	/* If p is not queued we will update its parameters at next wakeup. */
 	if (!task_on_rq_queued(p)) {
@@ -4182,9 +4248,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	u64 period = attr->sched_period ?: attr->sched_deadline;
 	u64 runtime = attr->sched_runtime;
 	u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
-	int cpus, err = -1, cpu = task_cpu(p);
-	struct dl_bw *dl_b = dl_bw_of(cpu);
+	int cpus, err = -1, cpu;
+	struct dl_bw *dl_b;
 	unsigned long cap;
+	bool counted;
 
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
@@ -4193,6 +4260,23 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
 		return 0;
 
+	/*
+	 * Change the bandwidth of a deadline task where it is counted: in
+	 * the root domain of the CPU recorded with it. A task moved asleep
+	 * has not run there yet, so the CPU it is on can be elsewhere. No
+	 * bandwidth is counted if a rebuild dropped it, or if the task
+	 * enters the class. A runtime below about a millionth of the period
+	 * gives a bandwidth of 0. It counts for nothing, and no move updates
+	 * its record. In these three cases, the root domain of the CPU the
+	 * task is on takes the new bandwidth. The caller holds cpuset_mutex,
+	 * or dl_accounting_mutex without cpusets, so no rebuild runs
+	 * meanwhile.
+	 */
+	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
+		  p->dl.dl_bw_gen == dl_bw_gen;
+	cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
+	dl_b = dl_bw_of(cpu);
+
 	/*
 	 * Either if a task, enters, leave, or stays -deadline but changes
 	 * its parameters, we may need to update accordingly the total
@@ -4207,7 +4291,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
-		   !__dl_overflow(dl_b, cap, p->dl.dl_bw, new_bw)) {
+		   !__dl_overflow(dl_b, cap, counted ? p->dl.dl_bw : 0,
+				  new_bw)) {
 		/*
 		 * XXX this is slightly incorrect: when the task
 		 * utilization decreases, we should delay the total
@@ -4215,7 +4300,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		 * But this would require to set the task's "inactive
 		 * timer" when the task is not inactive.
 		 */
-		__dl_sub(dl_b, p->dl.dl_bw, cpus);
+		if (counted)
+			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		dl_change_utilization(p, new_bw);
 		err = 0;
@@ -4236,11 +4322,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 * No rebuild can run meanwhile: the caller holds cpuset_mutex, or
 	 * dl_accounting_mutex without cpusets.
 	 */
-	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
-		lockdep_assert_cpuset_lock_held();
-		lockdep_assert_dl_accounting_held();
-		if (p->dl.dl_departed_cpu >= 0)
-			dl_task_release_bw(p, -1);
+	if (!err && dl_policy(policy)) {
+		if (!task_has_dl_policy(p)) {
+			lockdep_assert_cpuset_lock_held();
+			lockdep_assert_dl_accounting_held();
+			/*
+			 * The new bandwidth replaces any the task still has
+			 * counted. That one may be on the departed list, or
+			 * taken off it by a boost into the class
+			 * (switched_to_dl()). The record says where it is.
+			 */
+			if (p->dl.dl_bw_cpu >= 0)
+				dl_task_release_bw(p);
+		}
+		p->dl.dl_bw_cpu = cpu;
 		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
@@ -4399,8 +4494,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	 */
 	dl_se->dl_bw = 0;
 	__dl_clear_params(dl_se);
+	dl_se->dl_bw_cpu = -1;
 	INIT_LIST_HEAD(&dl_se->dl_departed_node);
-	dl_se->dl_departed_cpu = -1;
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4c013caa2471..306cd8d32173 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1185,6 +1185,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 
 int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
 {
+	unsigned int cpu;
+
 	/*
 	 * If the task isn't a deadline task or admission control is
 	 * disabled then we don't care about affinity changes.
@@ -1204,9 +1206,16 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
 	 * if admission test is enabled, we only admit -deadline
 	 * tasks allowed to run on all the CPUs in the task's
 	 * root_domain.
+	 *
+	 * That is the root domain that counts the bandwidth of the task:
+	 * the one of the CPU recorded with it. A task moved asleep to
+	 * another cpuset is still on a CPU of the root domain it left. A
+	 * bandwidth of 0 is counted nowhere, and no move updates its
+	 * record: use the CPU the task is on.
 	 */
+	cpu = READ_ONCE(p->dl.dl_bw) ? dl_task_root_domain_cpu(p) : task_cpu(p);
 	guard(rcu)();
-	if (!cpumask_subset(task_rq(p)->rd->span, mask))
+	if (!cpumask_subset(cpu_rq(cpu)->rd->span, mask))
 		return -EBUSY;
 
 	return 0;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 17/37] sched/deadline: Leave a dying task out of a rebuild of the accounting
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (15 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 18/37] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
                   ` (19 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A rebuild of the root domains' accounting takes the pi_lock of each
deadline task, not its rq lock. A dying task past its 0-lag time
releases its bandwidth at its last schedule. It holds its rq lock there,
not its pi_lock. Then it clears dl_bw. The rebuild can find the task
between the two steps.

In this diagram, T is a deadline task that exits on CPU 0 past its 0-lag
time. bw is its dl_bw. CPU 1 rebuilds the accounting:

  CPU 0 (T)                                  CPU 1 (rebuild)

  do_task_dead(): TASK_DEAD, under pi_lock
  __schedule(): T's rq lock taken
    task_non_contending(T)
      dl_task_release_bw(T)
        total_bw -= bw
        dl_bw_gen recorded
        record cleared
                                             dl_rebuild_rd_accounting()
                                               dl_clear_root_domains()
                                                 dl_bw_gen++
                                                 total_bw reset
                                               dl_add_task_root_domain(T)
                                                 T's pi_lock taken
                                                 dl_bw != 0, dl_bw_gen old
                                                 total_bw += bw
      __dl_clear_params(T): dl_bw = 0

dl_bw_gen does not help, since the release came before the rebuild. The
rebuild would count a bandwidth that nothing releases again. The root
domain would then refuse deadline tasks it has room for, until the next
rebuild.

Fix this with the record of the bandwidth, dl_bw_cpu. The release
clears it under dl_departed_lock, which the walk holds. So leave out a
dead task with no record. A dead task with a record has not reached its
last schedule. The walk counts it as a live task. That schedule then
releases the bandwidth, or puts the task on the departed list. A dead
task on the departed list keeps its bandwidth until its timer, and the
rebuild counts it as before.

The Sashiko AI review of v2 of this series found this window. Its review
of v4 found a gap in the first version of this fix. That version also
left out a task between TASK_DEAD and its last schedule, with its 0-lag
time ahead.

This patch completes "sched/deadline: Rebuild the root domains'
accounting from the task list". The walk of that patch finds exiting
tasks, which the cpuset iterator skipped. The fix needs the record
that "sched/deadline: Record the root domain holding a task's
bandwidth" adds. The three go to stable together.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 44db625bffc3..75fc9314da67 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3767,6 +3767,17 @@ void dl_add_task_root_domain(struct task_struct *p)
 		if (recorded < 0 || !cpu_active(recorded))
 			goto unlock;
 		cpu = recorded;
+	} else if (READ_ONCE(p->__state) == TASK_DEAD && p->dl.dl_bw_cpu < 0) {
+		/*
+		 * Count nothing for a dead task with no record. Its last
+		 * schedule released the bandwidth under its rq lock, and has
+		 * not cleared dl_bw yet. The release clears the record under
+		 * dl_departed_lock, which this walk holds. A dead task with a
+		 * record has not reached its last schedule. Count it as a
+		 * live task. That schedule then releases what the walk
+		 * counted, or puts the task on the departed list.
+		 */
+		goto unlock;
 	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 18/37] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (16 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 17/37] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 19/37] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
                   ` (18 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

From: Andrea Parri <parri.andrea@gmail.com>

A task can end up SCHED_DEADLINE with an affinity narrower than its root
domain when sched_setaffinity() and sched_setattr() race on it. Both
calls succeed, although each fails when made after the other:
sched_setattr() with -EPERM, sched_setaffinity() with -EBUSY.

__sched_setaffinity() calls dl_task_check_affinity() before
__set_cpus_allowed_ptr() takes task_rq_lock(), whereas
__sched_setscheduler() runs its admission test and switches the policy
under a single task_rq_lock(). So both calls can pass their checks:

  CPU 1: sched_setaffinity(p, cpu0)    CPU 2: sched_setattr(p, DEADLINE)

  dl_task_check_affinity()
    !task_has_dl_policy(p): return 0
                                       task_rq_lock()
                                       admission test passes
                                       p->policy = SCHED_DEADLINE
                                       task_rq_unlock()
  __set_cpus_allowed_ptr()
    task_rq_lock()
    p->cpus_mask = cpu0

The window includes CPU 1 spinning on task_rq_lock() while CPU 2 holds
it. With one thread narrowing a sleeping task to CPU 0 while another
makes it SCHED_DEADLINE, 923514, 959547 and 983997 of 1000000 rounds
ended with both calls succeeding on three boots of a 4-CPU guest.

Move the check into __set_cpus_allowed_ptr_locked(), next to the
PF_NO_SETAFFINITY re-check, under the same lock as the admission test.
Whichever call takes task_rq_lock() second then sees the other's
result: sched_setaffinity() fails with -EBUSY if p is already
SCHED_DEADLINE, and sched_setattr() fails with -EPERM if p's affinity
is already cpu0.

A new SCA_DL_CHECK flag selects the check, and only
__sched_setaffinity() sets it, so sched_setaffinity() and
relax_compatible_cpus_allowed_ptr() keep the check and other callers
stay unchecked. restrict_cpus_allowed_ptr() already does its deadline
check this way.

__sched_setaffinity() clears SCA_DL_CHECK before the second
__set_cpus_allowed_ptr() call it makes after racing with a cpuset
update, so the cpuset mask still wins there, as it does when
cpuset_update_tasks_cpumask() narrows a deadline task. SCA_CHECK stays
set for the PF_NO_SETAFFINITY re-check.

With this applied, the same test ends with both calls succeeding in 0
of 1000000 rounds on each of three boots, and sched_setaffinity()
fails with -EBUSY whenever it loses.

The race was reported by Sebastian. The fix was written with the help
of an LLM.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Closes: https://lore.kernel.org/20261103081149.DuyWNhVe@linutronix.de
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/core.c     | 11 +++++++++++
 kernel/sched/sched.h    |  1 +
 kernel/sched/syscalls.c | 12 +++++++-----
 3 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index a22df8b43a1a..c77cae5d1be1 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,17 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 		goto out;
 	}
 
+	/*
+	 * Check under the same lock that __sched_setscheduler() holds for
+	 * its admission test, so that p cannot become SCHED_DEADLINE
+	 * between this check and the mask update below.
+	 */
+	if (ctx->flags & SCA_DL_CHECK) {
+		ret = dl_task_check_affinity(p, ctx->new_mask);
+		if (ret)
+			goto out;
+	}
+
 	if (!(ctx->flags & SCA_MIGRATE_ENABLE)) {
 		if (cpumask_equal(&p->cpus_mask, ctx->new_mask)) {
 			if (ctx->flags & SCA_USER)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d87296abec4c..fd6d36b6b1e2 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2982,6 +2982,7 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 #define SCA_MIGRATE_DISABLE	0x02
 #define SCA_MIGRATE_ENABLE	0x04
 #define SCA_USER		0x08
+#define SCA_DL_CHECK		0x10
 
 extern void update_group_capacity(struct sched_domain *sd, int cpu);
 
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 306cd8d32173..4f47b17151ed 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1238,11 +1238,7 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
 	cpumask_and(new_mask, ctx->new_mask, cpus_allowed);
 
 	ctx->new_mask = new_mask;
-	ctx->flags |= SCA_CHECK;
-
-	retval = dl_task_check_affinity(p, new_mask);
-	if (retval)
-		goto out_free_new_mask;
+	ctx->flags |= SCA_CHECK | SCA_DL_CHECK;
 
 	retval = __set_cpus_allowed_ptr(p, ctx);
 	if (retval)
@@ -1271,6 +1267,12 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
 			if (empty)
 				cpumask_copy(new_mask, cpus_allowed);
 		}
+
+		/*
+		 * The cpuset mask must win here, as it does when
+		 * cpuset_update_tasks_cpumask() sets it on a deadline task.
+		 */
+		ctx->flags &= ~SCA_DL_CHECK;
 		__set_cpus_allowed_ptr(p, ctx);
 		retval = -EINVAL;
 	}
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 19/37] sched/deadline: Move a task's bandwidth only where room was made
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (17 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 18/37] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 20/37] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
                   ` (17 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination root domain. set_cpus_allowed_dl() then frees it in the
source, when the task's affinity changes. Its comment relies on the
first half: "We already made space for us in the destination domain".

But set_cpus_allowed_dl() runs on every change of affinity. It frees the
bandwidth whenever the new mask leaves the task's root domain, with or
without a reservation. sched_setaffinity() gets there too, if the task's
cpuset spans two root domains. With isolcpus="domain", any cpuset that
is not a partition root can hold isolated CPUs and other CPUs, which are
in two root domains. With admission control on, __sched_setaffinity()
refuses such a mask. With sched_rt_runtime_us at -1, nothing refuses it.
The bandwidth is then freed in the source and added nowhere: no root
domain counts it any more. The destination would underflow when the task
leaves.

Fix this by flagging the affinity changes that come with a reservation,
as SCA_DL_BW_RESERVED. cpuset_attach_task() sets it, through the new
set_cpus_allowed_bw_reserved(). set_cpus_allowed_dl() moves the
bandwidth only with the flag. Without it, the bandwidth stays where it
is counted, until the next rebuild of the accounting counts it anew.
The flag also skips the early exit of __set_cpus_allowed_ptr_locked()
for a mask that does not change. The task may already have that mask
from a change that moved no bandwidth, such as a sched_setaffinity()
without admission control.

The attach still flags a task that sched_setattr() made a deadline task
after cpuset_can_attach(), with nothing reserved for it. "sched: Make
deadline policy and affinity changes wait for a cpuset attach" fixes
that.

No case of deadline_bw_test fails before this patch and passes with it.
no_admission_affinity_moves, added later in this series, covers this
path together with "sched/deadline: Move a task's bandwidth to the root
domain it runs in".

Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h   |  2 ++
 kernel/cgroup/cpuset.c  | 13 ++++++++++++-
 kernel/sched/core.c     | 26 +++++++++++++++++++++++++-
 kernel/sched/deadline.c | 14 +++++++++-----
 kernel/sched/sched.h    |  2 ++
 5 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4a1cb869facc..dee4d4df624c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1972,6 +1972,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  * Return: zero if successful, or a negative error code
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
+extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
+					const struct cpumask *new_mask);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7628cf77bdfd..194a438e25c3 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3261,6 +3261,7 @@ static cpumask_var_t cpus_attach;
 
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
+	bool reserved = false;
 	struct mm_struct *mm;
 
 	lockdep_assert_cpuset_lock_held();
@@ -3270,11 +3271,21 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	else
 		cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
 			       subpartitions_cpus);
+	/*
+	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
+	 * deadline tasks that cpuset_can_attach() counted. Flag the move, so
+	 * that the bandwidth of this task moves with it.
+	 */
+	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+		reserved = true;
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	if (reserved)
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
+	else
+		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c77cae5d1be1..2a964286a7cd 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3207,7 +3207,15 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 	}
 
 	if (!(ctx->flags & SCA_MIGRATE_ENABLE)) {
-		if (cpumask_equal(&p->cpus_mask, ctx->new_mask)) {
+		/*
+		 * A move with SCA_DL_BW_RESERVED must reach
+		 * set_cpus_allowed_dl(), even if the mask does not change.
+		 * @p may have this mask from a change that moved no
+		 * bandwidth, such as sched_setaffinity() without admission
+		 * control.
+		 */
+		if (cpumask_equal(&p->cpus_mask, ctx->new_mask) &&
+		    !(ctx->flags & SCA_DL_BW_RESERVED)) {
 			if (ctx->flags & SCA_USER)
 				swap(p->user_cpus_ptr, ctx->user_mask);
 			goto out;
@@ -3280,6 +3288,22 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
+/*
+ * Like set_cpus_allowed_ptr(), for a caller that has made room for the
+ * deadline bandwidth of @p in the root domain of @new_mask. Move that
+ * bandwidth there with @p. Return 0 on success, or a negative error
+ * code.
+ */
+int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+{
+	struct affinity_context ac = {
+		.new_mask  = new_mask,
+		.flags     = SCA_DL_BW_RESERVED,
+	};
+
+	return __set_cpus_allowed_ptr(p, &ac);
+}
+
 /*
  * Change a given task's CPU affinity to the intersection of its current
  * affinity mask and @subset_mask, writing the resulting mask to @new_mask.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 75fc9314da67..1825ca22d422 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3602,12 +3602,16 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	WARN_ON_ONCE(!dl_task(p));
 
 	/*
-	 * Migrating a SCHED_DEADLINE task between exclusive
-	 * cpusets (different root_domains) entails a bandwidth
-	 * update. We already made space for us in the destination
-	 * domain (see cpuset_can_attach()).
+	 * Migrating a SCHED_DEADLINE task between exclusive cpusets
+	 * (different root_domains) entails a bandwidth update. Move the
+	 * bandwidth only if the caller made room for it in the destination,
+	 * as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
+	 * Otherwise, the bandwidth stays where it is counted, until the next
+	 * rebuild of the accounting. Moving it without room would leave no
+	 * root domain counting it.
 	 */
-	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
+	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+	    dl_task_needs_bw_move(p, ctx->new_mask)) {
 		unsigned int src_cpu = dl_task_root_domain_cpu(p);
 		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
 		unsigned int cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index fd6d36b6b1e2..9e3ca69abc36 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2983,6 +2983,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 #define SCA_MIGRATE_ENABLE	0x04
 #define SCA_USER		0x08
 #define SCA_DL_CHECK		0x10
+/* The caller made room for the deadline bandwidth in the new root domain. */
+#define SCA_DL_BW_RESERVED	0x20
 
 extern void update_group_capacity(struct sched_domain *sd, int cpu);
 
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 20/37] cgroup/cpuset: Wait for an attach before rebuilding the sched domains
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (18 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 19/37] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
                   ` (16 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

cpuset_can_attach() reserves the bandwidth of the deadline tasks it
moves in the destination root domain. cpuset_attach() moves their
bandwidth later, on the strength of that reservation.

Commit 75f7a25ec6bb ("cgroup/cpuset: Prevent race between task attach
and cpuset state change") lets cpuset_attach() rely on what
cpuset_can_attach() set up. The writes of cpus, mems and cpus.partition
wait for an attach in progress, in wait_attach_done_lock(). CPU hotplug
cannot run in between either: it takes cpus_write_lock, and the attach
holds cpus_read_lock throughout.

But a rebuild of the sched domains does not wait. A write of
sched_rt_runtime_us and the energy model rebuild them through
rebuild_sched_domains(). The cgroup v1 writes of sched_load_balance and
sched_relax_domain_level, and hk_sd_workfn(), use cpuset_full_lock().

In this example, a rebuild drops the reservation. T is a deadline task
asleep in root domain R. Thread A moves T to a partition root P, which
has a root domain of its own. Thread B writes sched_rt_runtime_us:

  A                                      B

  cpuset_can_attach()
    takes cpuset_mutex
    cpuset_reserve_dl_bw()
      P's total_bw += T's bandwidth
    drops cpuset_mutex
                                         rebuild_sched_domains()
                                           takes cpuset_mutex
                                           dl_rebuild_rd_accounting()
                                             clears R's and P's total_bw
                                             counts T in R (not moved yet)
                                           drops cpuset_mutex
  cpuset_attach()
    takes cpuset_mutex
    cpuset_attach_task(T)
      set_cpus_allowed_dl(T)
        R's total_bw -= T's bandwidth
        records a CPU of P

P now lacks the bandwidth of T, and admits deadline tasks beyond its
capacity. When T leaves P, P's total_bw underflows. In testing, the
race left a destination admitting 3 tasks of 10% too many, or P
admitting no task at all.

Fix this by waiting for an attach in progress in
rebuild_sched_domains_cpuslocked() and in cpuset_full_lock(). The
latter then takes its locks in the order of the writes of cpus and
cpus.partition. A rebuild from CPU hotplug finds no attach in progress.
The online, offline and kill of a cpuset never wait: they hold
cgroup_mutex, which the attach holds throughout.

The move_vs_rebuild case of deadline_bw_test, added later in this
series, tests this race. Without this patch, it fails in 5 of 8 VMs with
8 CPUs. With it, the case passes in 8 of 8. The next patch needs this
wait too: without it, it would give back a reservation that a rebuild
dropped already.

Found by the Sashiko AI reviews of v2 (the race) and v3 (the cgroup v1
writes) of this series.

Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/cgroup/cpuset.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 194a438e25c3..51b6be460794 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -296,17 +296,20 @@ void lockdep_assert_cpuset_lock_held(void)
 	lockdep_assert_held(&cpuset_mutex);
 }
 
+static void wait_attach_done_lock(void) __acquires(&cpuset_mutex);
+
 /**
  * cpuset_full_lock - Acquire full protection for cpuset modification
  *
  * Takes both CPU hotplug read lock (cpus_read_lock()) and cpuset mutex
- * to safely modify cpuset data.
+ * to safely modify cpuset data. Waits for an attach in progress before
+ * it takes cpuset mutex.
  */
 void cpuset_full_lock(void)
 {
 	mutex_lock(&cpuset_top_mutex);
 	cpus_read_lock();
-	mutex_lock(&cpuset_mutex);
+	wait_attach_done_lock();
 }
 
 void cpuset_full_unlock(void)
@@ -1005,7 +1008,14 @@ void rebuild_sched_domains_locked(void)
 
 static void rebuild_sched_domains_cpuslocked(void)
 {
-	mutex_lock(&cpuset_mutex);
+	/*
+	 * Wait for an attach in progress. cpuset_can_attach() has reserved
+	 * the deadline bandwidth of the moving tasks in their new root
+	 * domain, and cpuset_attach() moves it on the strength of that
+	 * reservation. A rebuild of the accounting in between would drop
+	 * the reservation.
+	 */
+	wait_attach_done_lock();
 	rebuild_sched_domains_locked();
 	mutex_unlock(&cpuset_mutex);
 }
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (19 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 20/37] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:31   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 22/37] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
                   ` (15 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. Commit 710da3c8ea7d ("sched/core: Prevent race condition
between cpuset and __sched_setscheduler()") added that lock. It keeps
the cpuset code from changing the root domains while the deadline
admission test runs.

But a cpuset attach drops cpuset_mutex between cpuset_can_attach() and
cpuset_attach(). A sched_setattr() can run in between. Take a process
in the root cgroup. It has deadline threads, and a thread T that is not
a deadline task. The process is moved into a partition root P:

  echo PID > P/cgroup.procs              sched_setattr(T, SCHED_DEADLINE)

  cpuset_can_attach()
    lock cpuset_mutex
    reserve room in P for the deadline
      threads: T is not one of them
    unlock cpuset_mutex
                                         lock cpuset_mutex
                                         admit T in the root domain it
                                           is still in
                                         unlock cpuset_mutex
  cpuset_attach()
    lock cpuset_mutex
    move T to P, where nothing is
      reserved for it
    unlock cpuset_mutex

The attach still moves the bandwidth of T. It is freed in the root
domain T left, and P has no room reserved to count it. No root domain
counts it any more, and P can admit deadline tasks beyond its capacity.

In testing, P was a one-CPU partition root, which admits 95%. The
process had two sleeping deadline threads of 30% each, and
sched_setattr() made T a deadline task of 40%. The move and the
sched_setattr() both succeeded, with no room in P for the three. With
threads of 20% and a T of 30%, P then admitted five more tasks of 10%,
where it had room for two.

Fix this by taking cpuset_mutex in __sched_setscheduler() only when no
attach is in progress. cpuset_partition_write() and
cpuset_hotplug_update_tasks() already wait for an attach that way. A
deadline policy change then comes before the attach, and is part of its
reservation, or after it.

With this patch, and no room in P for the three, the two never both
succeed. If the sched_setattr() comes first, the move is refused with
EBUSY. If it comes after, T has its new CPUs and has not run on them.
The call is then refused with EPERM, as for any sleeping task moved to
another partition.

sched_setaffinity() on a deadline task can run in between as well. A
later patch of this series reserves the bandwidth of the task on one CPU
of its new affinity, and records that CPU with the task. It is
"cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to". A new affinity in between could leave that CPU out. cpuset_attach()
would then move no bandwidth, and the task would stay counted in the
root domain it left. So sched_setaffinity() waits the same way when the
task is a deadline task.

sched_setaffinity() tests the policy with no lock, before it decides to
wait. A sched_setattr() can make the task a deadline task right after
that test. So test the policy again under the runqueue lock, where the
mask changes. If the task is a deadline task there and the caller did
not wait, the call starts over, and waits. The new SCA_DL_NOWAIT flag
asks for that test.

For user-space, these calls now wait until a cpuset attach in progress
ends:

 - sched_setattr() and sched_setscheduler() on a deadline task
 - sched_setattr() that makes a task a deadline task
 - sched_setaffinity() on a deadline task

The attach can be the move of any process. The wait is not
interruptible: a signal does not end it. Outside that window, the
results of these calls are unchanged.

Three cases of deadline_bw_test, added later in this series, test this.
move_became_deadline and move_became_deadline_no_room fail without this
patch. move_affinity_changed fails once the bandwidth is reserved on
one CPU, if sched_setaffinity() does not wait.

A fourth case, move_affinity_became_deadline, covers the second test of
the policy. Its window is a few instructions wide. A delay added after
the first test, for testing only, widens it. The case then fails without
the second test in 3 guests of 3. It passes with it.

Fixes: 710da3c8ea7d ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/cpuset.h  |  2 ++
 kernel/cgroup/cpuset.c  | 11 +++++++++++
 kernel/sched/core.c     | 10 ++++++++++
 kernel/sched/sched.h    |  2 ++
 kernel/sched/syscalls.c | 40 ++++++++++++++++++++++++++++++++--------
 5 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index db1cc2fc0324..5d0c01a25354 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -73,6 +73,7 @@ extern void cpuset_init_smp(void);
 extern void cpuset_force_rebuild(void);
 extern void cpuset_update_active_cpus(void);
 extern void cpuset_lock(void);
+extern void cpuset_lock_no_attach(void);
 extern void cpuset_unlock(void);
 extern void lockdep_assert_cpuset_lock_held(void);
 extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
@@ -192,6 +193,7 @@ static inline void cpuset_update_active_cpus(void)
 }
 
 static inline void cpuset_lock(void) { }
+static inline void cpuset_lock_no_attach(void) { }
 static inline void cpuset_unlock(void) { }
 static inline void lockdep_assert_cpuset_lock_held(void) { }
 
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 51b6be460794..ce9c57617f49 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -378,6 +378,17 @@ static void wait_attach_done_lock(void)
 	}
 }
 
+/**
+ * cpuset_lock_no_attach - Acquire cpuset_mutex with no task attach in progress
+ *
+ * A task attach drops cpuset_mutex between cpuset_can_attach() and
+ * cpuset_attach(). Use this to keep a change out of that window.
+ */
+void cpuset_lock_no_attach(void)
+{
+	wait_attach_done_lock();
+}
+
 static inline void check_insane_mems_config(nodemask_t *nodes)
 {
 	if (!cpusets_insane_config() &&
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 2a964286a7cd..52c291de384e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,16 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 		goto out;
 	}
 
+	/*
+	 * sched_setaffinity() tested the policy with no lock, and did not
+	 * wait for a cpuset attach in progress. @p became a deadline task
+	 * since: let the caller wait and try again.
+	 */
+	if ((ctx->flags & SCA_DL_NOWAIT) && task_has_dl_policy(p)) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
 	/*
 	 * Check under the same lock that __sched_setscheduler() holds for
 	 * its admission test, so that p cannot become SCHED_DEADLINE
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9e3ca69abc36..2087ad70c56e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2985,6 +2985,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 #define SCA_DL_CHECK		0x10
 /* The caller made room for the deadline bandwidth in the new root domain. */
 #define SCA_DL_BW_RESERVED	0x20
+/* The caller did not wait for a cpuset attach: -EAGAIN for a deadline task. */
+#define SCA_DL_NOWAIT		0x40
 
 extern void update_group_capacity(struct sched_domain *sd, int cpu);
 
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4f47b17151ed..70b364b4f82d 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -564,10 +564,15 @@ int __sched_setscheduler(struct task_struct *p,
 	 * information. Do not take cpuset_mutex when pi is unset: the
 	 * caller is normalize_rt_tasks(), which holds tasklist_lock and
 	 * cannot sleep.
+	 *
+	 * Wait for a task attach in progress. cpuset_can_attach() reserved
+	 * the bandwidth of the deadline tasks it saw, and cpuset_attach()
+	 * moves them. A task that enters or leaves the class in between
+	 * would not match that reservation.
 	 */
 	if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
 		cpuset_locked = true;
-		cpuset_lock();
+		cpuset_lock_no_attach();
 		dl_accounting_lock();
 	}
 
@@ -1289,6 +1294,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
 	struct affinity_context ac;
 	struct cpumask *user_mask;
 	int retval;
+	bool dl;
 
 	CLASS(find_get_task, p)(pid);
 	if (!p)
@@ -1318,13 +1324,31 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
 		return -ENOMEM;
 	}
 
-	ac = (struct affinity_context){
-		.new_mask  = in_mask,
-		.user_mask = user_mask,
-		.flags     = SCA_USER,
-	};
-
-	retval = __sched_setaffinity(p, &ac);
+	/*
+	 * Wait for a cpuset attach in progress before changing the affinity
+	 * of a deadline task. cpuset_can_attach() reserved the bandwidth of
+	 * the task for the CPUs it moves to, and cpuset_attach() moves it
+	 * there. A new affinity in between could change those CPUs.
+	 *
+	 * The policy is tested here with no lock. A task that becomes a
+	 * deadline task after this test is found under its runqueue lock,
+	 * with SCA_DL_NOWAIT. Wait and try again then.
+	 */
+	for (;;) {
+		dl = task_has_dl_policy(p);
+		ac = (struct affinity_context){
+			.new_mask  = in_mask,
+			.user_mask = user_mask,
+			.flags     = SCA_USER | (dl ? 0 : SCA_DL_NOWAIT),
+		};
+		if (dl)
+			cpuset_lock_no_attach();
+		retval = __sched_setaffinity(p, &ac);
+		if (dl)
+			cpuset_unlock();
+		if (dl || retval != -EAGAIN)
+			break;
+	}
 	kfree(ac.user_mask);
 
 	return retval;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 22/37] cgroup/cpuset: Give back the bandwidth an attach did not move
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (20 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
                   ` (14 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach()
fails") reserves the bandwidth of an attach as one sum.
cpuset_can_attach() adds up the bandwidth of the deadline tasks it
moves. Once all tasks pass their checks, it reserves the sum in the
destination root domain. If another controller refuses the attach,
cpuset_cancel_attach() frees the sum. If the attach goes through, the
move of each task uses its part of the sum.

But a task can leave SCHED_DEADLINE between cpuset_can_attach() and
cpuset_attach(). The previous patch makes sched_setattr() wait for the
attach. sysrq-n does not wait: it cannot sleep. The task then moves no
bandwidth. Its part of the reservation stays in the destination's
total_bw, until the next rebuild of the accounting. The destination can
then refuse deadline tasks that would fit.

Fix this by keeping the tally of what the attach moves, in
sum_moved_dl_bw. Once the attach is through, give back what it did not
move.

set_cpus_allowed_dl() decides the move under the runqueue lock.
cpuset_attach_task() does not hold it. So the tally counts what
set_cpus_allowed_dl() reports in the affinity context.

The random_mix case of deadline_bw_test, added later in this series,
found the bug. It could no longer lower a fair server in a partition
root with no room left.

The move_normalized case, added with that test, fires sysrq-n while a
process with a sleeping deadline thread of 30% moves into a one-CPU
partition root. Without this patch, the partition kept the 30% in 3
guests of 3. It admitted 6 tasks of 10% where it had room for 9.

Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h           |  3 ++-
 kernel/cgroup/cpuset-internal.h |  2 ++
 kernel/cgroup/cpuset.c          | 30 +++++++++++++++++++++++++-----
 kernel/sched/core.c             | 12 ++++++++----
 kernel/sched/deadline.c         |  4 ++++
 kernel/sched/sched.h            |  2 ++
 6 files changed, 43 insertions(+), 10 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index dee4d4df624c..471cd27894ff 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1973,7 +1973,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
-					const struct cpumask *new_mask);
+					const struct cpumask *new_mask,
+					u64 *moved);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index bf847953c78d..93aa46d76b04 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -170,6 +170,8 @@ struct cpuset {
 	int nr_migrate_dl_tasks;
 	/* DL bandwidth that needs destination reservation for this attach. */
 	u64 sum_migrate_dl_bw;
+	/* The part of sum_migrate_dl_bw moved. The rest is given back. */
+	u64 sum_moved_dl_bw;
 	/*
 	 * CPU used for temporary DL bandwidth allocation during attach;
 	 * -1 if no DL bandwidth was allocated in the current attach.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ce9c57617f49..2f39c3d12550 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3140,10 +3140,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
 	llist_for_each_entry_safe(cs, next, lnode, attach_node) {
 		init_llist_node(&cs->attach_node);
 		if (cs->nr_migrate_dl_tasks) {
-			if (cancel && cs->dl_bw_cpu >= 0)
-				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+			/*
+			 * Give back the bandwidth the attach did not move.
+			 * If the attach is cancelled, that is all of the
+			 * reservation. Otherwise, it is what no move used,
+			 * such as the part for a task that left
+			 * SCHED_DEADLINE. The class it left releases its own
+			 * bandwidth, from the root domain that counts it.
+			 */
+			if (cs->dl_bw_cpu >= 0) {
+				u64 bw = cancel ? cs->sum_migrate_dl_bw :
+					 cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+
+				if (bw)
+					dl_bw_free(cs->dl_bw_cpu, bw);
+			}
 			cs->nr_migrate_dl_tasks = 0;
 			cs->sum_migrate_dl_bw = 0;
+			cs->sum_moved_dl_bw = 0;
 			cs->dl_bw_cpu = -1;
 		}
 	}
@@ -3303,10 +3317,16 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	if (reserved)
-		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
-	else
+	if (reserved) {
+		u64 moved;
+
+		/* Count what the move reports it moved. */
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
+							  &moved));
+		cs->sum_moved_dl_bw += moved;
+	} else {
 		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	}
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 52c291de384e..229b7bfa9634 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3301,17 +3301,21 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 /*
  * Like set_cpus_allowed_ptr(), for a caller that has made room for the
  * deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p. Return 0 on success, or a negative error
- * code.
+ * bandwidth there with @p, and set @moved to the bandwidth moved, 0
+ * if none. Return 0 on success, or a negative error code.
  */
-int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+int set_cpus_allowed_bw_reserved(struct task_struct *p,
+				 const struct cpumask *new_mask, u64 *moved)
 {
 	struct affinity_context ac = {
 		.new_mask  = new_mask,
 		.flags     = SCA_DL_BW_RESERVED,
 	};
+	int ret;
 
-	return __set_cpus_allowed_ptr(p, &ac);
+	ret = __set_cpus_allowed_ptr(p, &ac);
+	*moved = ac.dl_bw_moved;
+	return ret;
 }
 
 /*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 1825ca22d422..5503820c5b26 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3609,6 +3609,9 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * Otherwise, the bandwidth stays where it is counted, until the next
 	 * rebuild of the accounting. Moving it without room would leave no
 	 * root domain counting it.
+	 *
+	 * Report in @ctx the bandwidth moved. The caller made its own test
+	 * without the runqueue lock, so it must count only what this reports.
 	 */
 	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
 	    dl_task_needs_bw_move(p, ctx->new_mask)) {
@@ -3637,6 +3640,7 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
 		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
 		p->dl.dl_bw_gen = dl_bw_gen;
+		ctx->dl_bw_moved = p->dl.dl_bw;
 	}
 
 	set_cpus_allowed_common(p, ctx);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 2087ad70c56e..661da2b5c278 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2688,6 +2688,8 @@ extern const u32		sched_prio_to_wmult[40];
 struct affinity_context {
 	const struct cpumask	*new_mask;
 	struct cpumask		*user_mask;
+	/* Out: the deadline bandwidth moved to the new root domain. */
+	u64			dl_bw_moved;
 	unsigned int		flags;
 };
 
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (21 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 22/37] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:56   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
                   ` (13 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the source when the
task's affinity changes. The reservation goes to the root domain of the
first active CPU of the destination cpuset. So cpuset_can_attach()
treats the cpuset as one root domain.

But a cpuset can span more than one root domain. With isolcpus=domain,
the root cpuset holds the isolated CPUs, which are in the default root
domain. A task also keeps the affinity set by sched_setaffinity(). Its
new CPUs can then all be in a root domain that got no reservation. The
task runs there, and its bandwidth is reserved in the other root domain.

Take a machine with CPUs 0-4, booted with isolcpus=domain,1,2. P is a
partition root of CPU 4. A deadline task T, pinned to CPUs 1-2, is moved
into P and back to the root cgroup:

                                       total_bw of the root domain of
  admin, task T                        CPUs 0, 3   CPUs 1, 2   CPU 4 (P)

  sched_setaffinity(T, CPUs 1-2)
  sched_setattr(T, SCHED_DEADLINE)                 + bw
  echo T > P/cgroup.procs
    cpuset_can_attach()
      reserves on CPU 4                                        + bw
    set_cpus_allowed_dl()
      frees where recorded, CPU 1                  - bw
      records CPU 4
  T runs on CPU 4
  echo T > cgroup.procs of the root
    cpuset_can_attach()
      reserves on CPU 0                + bw
    __set_cpus_allowed_ptr()
      keeps T's affinity: CPUs 1-2
    set_cpus_allowed_dl()
      frees where recorded, CPU 4                              - bw
      records CPU 1
  T runs on CPU 1
  T exits
    releases where recorded, CPU 1                 - bw
                                       ---------   ---------   ---------
  left after T exits                   + bw        - bw        0

T ran in the root domain of CPUs 1 and 2 without being counted there.
In testing, the total_bw of that root domain went below zero when T
died. It then refused every deadline task, until the next rebuild of
the accounting.

Fix this by reserving the bandwidth on the CPU that the move records
with the task. The new dl_task_bw_reserve_cpu() works out that CPU and
records it with the task. It is the CPU the task is on, if the task can
stay there. Else it is the first active CPU of the new mask, a
housekeeping one if any. The rebuild of the accounting makes the same
choice, later in this series. cpuset_can_attach() reserves the bandwidth
per such CPU, and refuses with EBUSY as before. cpuset_attach_task()
moves the bandwidth to the CPU recorded, and does not work it out again:
the task may have run on another CPU since. cpuset_attach() gives back
the rest.

The record is a new field of the task, dl_bw_resv_cpu.
cpuset_attach_task() moves the bandwidth of a task only if the task has
that record. Juri Lelli suggested to track the tasks that
cpuset_can_attach() counts. A task it did not count cannot use the room
made for another task.

A new mask can also span both root domains. A running task would then go
to the CPU that cpumask_any_and_distribute() picks. A sleeping one would
wake up where select_fallback_rq() picks. Either CPU can be in the root
domain without the reservation. So __set_cpus_allowed_ptr_locked() takes
the CPU with the reservation, if cpumask_any_and_distribute() picked a
CPU of the other root domain. A sleeping task gets the CPU chosen as its
wakeup CPU.

A new mask that still holds a CPU of the root domain counting the task
moves no bandwidth at the attach. The task can still land in the other
root domain, if it must leave its CPU. A later patch of this series
moves the bandwidth there, without a check for room. It is
"sched/deadline: Move a task's bandwidth to the root domain it runs in".

The sums are kept per CPU, not per cpuset: one cpuset can need room in
two root domains. One set of sums is enough, since cgroup_mutex lets
one attach run at a time. guarantee_active_cpus() now takes the cpuset
to start from.

Later in this series, deadline_bw_test gets four cases for this:
move_isolated, move_isolated_refused, move_isolated_asleep and
move_spanning. Without this patch, all four fail with
isolcpus=domain,1,2. With isolcpus=domain,0,1, move_isolated,
move_spanning and six other move cases fail.

Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h           |   8 +-
 include/linux/sched/deadline.h  |  15 ++-
 kernel/cgroup/cpuset-internal.h |  12 --
 kernel/cgroup/cpuset.c          | 209 +++++++++++++++++++-------------
 kernel/sched/core.c             |  27 ++++-
 kernel/sched/deadline.c         | 128 ++++++++++++++-----
 kernel/sched/sched.h            |   2 +
 kernel/sched/syscalls.c         |   5 +-
 8 files changed, 270 insertions(+), 136 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 471cd27894ff..2f2f59402f09 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -758,10 +758,16 @@ struct sched_dl_entity {
 	 *
 	 * @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
 	 * counted. If a rebuild dropped it since, there is nothing to remove.
+	 *
+	 * @dl_bw_resv_cpu is a CPU of the root domain where a cpuset attach
+	 * in progress reserved room for the bandwidth, or -1.
+	 * cpuset_can_attach() sets it, and cpuset_attach() moves the
+	 * bandwidth there.
 	 */
 	struct list_head		dl_departed_node;
 	int				dl_bw_cpu;
 	unsigned int			dl_bw_gen;
+	int				dl_bw_resv_cpu;
 
 	/*
 	 * Bits for DL-server functionality. Also see the comment near
@@ -1974,7 +1980,7 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
 					const struct cpumask *new_mask,
-					u64 *moved);
+					int cpu, u64 *moved);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 60be226699bb..cbb7cd71db58 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -50,14 +50,17 @@ extern void dl_accounting_unlock(void);
 extern void lockdep_assert_dl_accounting_held(void);
 #endif
 /*
- * Return whether moving DL task @p to @new_mask requires moving DL
- * bandwidth accounting between root domains. This helper is specific to
- * DL bandwidth move accounting semantics and is shared by
- * cpuset_can_attach() and set_cpus_allowed_dl() so both paths use the
- * same source root-domain test.
+ * Return the CPU that a move of DL task @p to the CPUs of @new_mask
+ * records with its DL bandwidth. Return -1 if the bandwidth stays in the
+ * root domain that counts it. Like the move, keep the CPUs within the
+ * affinity set by the user. cpuset_can_attach() reserves the bandwidth in
+ * the root domain of that CPU. cpuset_attach_task() then moves it there,
+ * and set_cpus_allowed_dl() makes the same test.
  */
-extern bool dl_task_needs_bw_move(struct task_struct *p,
+extern int dl_task_bw_reserve_cpu(struct task_struct *p,
 				  const struct cpumask *new_mask);
+extern int dl_task_bw_reserved_cpu(struct task_struct *p);
+extern void dl_task_bw_unreserve(struct task_struct *p);
 
 static inline bool dl_server(struct sched_dl_entity *dl_se)
 {
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 93aa46d76b04..a83443dfb788 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,18 +166,6 @@ struct cpuset {
 	 */
 	bool remote_partition_updating;
 
-	/* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
-	int nr_migrate_dl_tasks;
-	/* DL bandwidth that needs destination reservation for this attach. */
-	u64 sum_migrate_dl_bw;
-	/* The part of sum_migrate_dl_bw moved. The rest is given back. */
-	u64 sum_moved_dl_bw;
-	/*
-	 * CPU used for temporary DL bandwidth allocation during attach;
-	 * -1 if no DL bandwidth was allocated in the current attach.
-	 */
-	int dl_bw_cpu;
-
 	/* Invalid partition error code, not lock protected */
 	enum prs_errcode prs_err;
 
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 2f39c3d12550..0d55f092d9cb 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -272,7 +272,6 @@ struct cpuset top_cpuset = {
 	.flags = BIT(CS_CPU_EXCLUSIVE) |
 		 BIT(CS_MEM_EXCLUSIVE) | BIT(CS_SCHED_LOAD_BALANCE),
 	.partition_root_state = PRS_ROOT,
-	.dl_bw_cpu = -1,
 };
 
 /**
@@ -487,28 +486,24 @@ static inline bool partition_is_populated(struct cpuset *cs,
 }
 
 /*
- * Return in pmask the portion of a task's cpusets's cpus_allowed that
- * are online and are capable of running the task.  If none are found,
- * walk up the cpuset hierarchy until we find one that does have some
- * appropriate cpus.
+ * Return in pmask the portion of the cpus_allowed of @cs that are online
+ * and are capable of running @tsk.  If none are found, walk up the cpuset
+ * hierarchy until we find one that does have some appropriate cpus.
  *
  * One way or another, we guarantee to return some non-empty subset
  * of cpu_active_mask.
  *
  * Call with callback_lock or cpuset_mutex held.
  */
-static void guarantee_active_cpus(struct task_struct *tsk,
+static void guarantee_active_cpus(struct cpuset *cs, struct task_struct *tsk,
 				  struct cpumask *pmask)
 {
 	const struct cpumask *possible_mask = task_cpu_possible_mask(tsk);
-	struct cpuset *cs;
 
 	if (WARN_ON(!cpumask_and(pmask, possible_mask, cpu_active_mask)))
 		cpumask_copy(pmask, cpu_active_mask);
 
 	rcu_read_lock();
-	cs = task_cs(tsk);
-
 	while (!cpumask_intersects(cs->effective_cpus, pmask)) {
 		cs = parent_cs(cs);
 		if (unlikely(!cs)) {
@@ -630,7 +625,6 @@ struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)
 	if (!trial)
 		return NULL;
 
-	trial->dl_bw_cpu = -1;
 	init_llist_node(&trial->attach_node);
 
 	/* Setup cpumask pointer array */
@@ -3106,61 +3100,89 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
 	return 0;
 }
 
+/*
+ * The deadline bandwidth of the attach in progress, per CPU. When the
+ * bandwidth of a deadline task moves to another root domain,
+ * dl_task_bw_reserve_cpu() returns the CPU that the move records with it.
+ * It also records that CPU with the task. cpuset_can_attach() sums the
+ * bandwidth of these tasks per such CPU. It reserves each sum in the root
+ * domain of its CPU. cpuset_attach_task() moves the bandwidth of each task
+ * to the CPU recorded with it. It counts what it moves, and cpuset_attach()
+ * gives back the rest.
+ * dl_attach_cpus holds the CPUs with a sum. cgroup_mutex lets one attach
+ * run at a time, and cpuset_mutex protects these.
+ */
+struct cpuset_dl_bw {
+	u64	sum;
+	u64	moved;		/* the part of sum moved */
+	bool	reserved;	/* dl_bw_alloc() took sum */
+};
+
+static DEFINE_PER_CPU(struct cpuset_dl_bw, cpuset_dl_bw);
+static cpumask_var_t dl_attach_cpus;
+
 static int cpuset_reserve_dl_bw(void)
 {
-	struct cpuset *cs;
 	int cpu, ret;
 
-	llist_for_each_entry(cs, dst_cs_head.first, attach_node) {
-		if (!cs->sum_migrate_dl_bw)
-			continue;
-
-		cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
-		if (unlikely(cpu >= nr_cpu_ids))
-			return -EINVAL;
+	for_each_cpu(cpu, dl_attach_cpus) {
+		struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
 
-		ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+		ret = dl_bw_alloc(cpu, b->sum);
 		if (ret)
 			return ret;
-
-		cs->dl_bw_cpu = cpu;
+		b->reserved = true;
 	}
 	return 0;
 }
 
 /*
- * Clear and optionally apply (@cancel is false) the attach related data in the
- * source or destination cpuset.
+ * Give back the bandwidth the attach reserved and did not move. If the
+ * attach is cancelled, that is all of the reservation. Otherwise, it is
+ * what no move used, such as the part for a task that left
+ * SCHED_DEADLINE. The class it left releases its own bandwidth, from the
+ * root domain that counts it.
  */
-static void clear_attach_data(struct llist_head *head, bool cancel)
+static void cpuset_free_dl_bw(void)
+{
+	int cpu;
+
+	for_each_cpu(cpu, dl_attach_cpus) {
+		struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
+
+		if (b->reserved && b->sum != b->moved)
+			dl_bw_free(cpu, b->sum - b->moved);
+		*b = (struct cpuset_dl_bw){ };
+	}
+	cpumask_clear(dl_attach_cpus);
+}
+
+/* Empty @head, the list of the source or destination cpusets. */
+static void clear_attach_data(struct llist_head *head)
 {
 	struct cpuset *cs, *next;
 	struct llist_node *lnode = __llist_del_all(head);
 
-	llist_for_each_entry_safe(cs, next, lnode, attach_node) {
+	llist_for_each_entry_safe(cs, next, lnode, attach_node)
 		init_llist_node(&cs->attach_node);
-		if (cs->nr_migrate_dl_tasks) {
-			/*
-			 * Give back the bandwidth the attach did not move.
-			 * If the attach is cancelled, that is all of the
-			 * reservation. Otherwise, it is what no move used,
-			 * such as the part for a task that left
-			 * SCHED_DEADLINE. The class it left releases its own
-			 * bandwidth, from the root domain that counts it.
-			 */
-			if (cs->dl_bw_cpu >= 0) {
-				u64 bw = cancel ? cs->sum_migrate_dl_bw :
-					 cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+}
 
-				if (bw)
-					dl_bw_free(cs->dl_bw_cpu, bw);
-			}
-			cs->nr_migrate_dl_tasks = 0;
-			cs->sum_migrate_dl_bw = 0;
-			cs->sum_moved_dl_bw = 0;
-			cs->dl_bw_cpu = -1;
-		}
-	}
+/*
+ * Protected by cpuset_mutex. cpus_attach is used by cpuset_can_attach() and
+ * cpuset_attach_task(), but we can't allocate it dynamically there. Define
+ * it global and allocate from cpuset_init().
+ */
+static cpumask_var_t cpus_attach;
+
+/* Set @mask to the CPUs that an attach to @cs gives @task. */
+static void cpuset_attach_cpus(struct cpuset *cs, struct task_struct *task,
+			       struct cpumask *mask)
+{
+	if (cs != &top_cpuset)
+		guarantee_active_cpus(cs, task, mask);
+	else
+		cpumask_andnot(mask, task_cpu_possible_mask(task),
+			       subpartitions_cpus);
 }
 
 /* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
@@ -3170,7 +3192,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	struct cpuset *cs, *oldcs;
 	struct task_struct *task;
 	bool setsched_check;
-	int ret;
+	int cpu, ret;
 
 	cs = oldcs = NULL;
 	mutex_lock(&cpuset_mutex);
@@ -3236,15 +3258,20 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 				goto out_unlock;
 		}
 
-		if (dl_task(task)) {
-			/*
-			 * Count all migrating DL tasks for cpuset task accounting.
-			 * Only tasks that need a root-domain bandwidth move
-			 * contribute to sum_migrate_dl_bw.
-			 */
-			cs->nr_migrate_dl_tasks++;
-			if (dl_task_needs_bw_move(task, cs->effective_cpus))
-				cs->sum_migrate_dl_bw += task->dl.dl_bw;
+		if (!dl_task(task))
+			continue;
+		/*
+		 * Sum the bandwidth of the deadline tasks that move to another
+		 * root domain, per CPU that the move records. That CPU depends
+		 * on the CPUs the task gets. A cpuset can span more than one
+		 * root domain, and the affinity of a task can keep it in any
+		 * of them.
+		 */
+		cpuset_attach_cpus(cs, task, cpus_attach);
+		cpu = dl_task_bw_reserve_cpu(task, cpus_attach);
+		if (cpu >= 0) {
+			per_cpu_ptr(&cpuset_dl_bw, cpu)->sum += task->dl.dl_bw;
+			cpumask_set_cpu(cpu, dl_attach_cpus);
 		}
 	}
 
@@ -3268,8 +3295,11 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 
 out_unlock:
 	if (ret) {
-		clear_attach_data(&src_cs_head, true);
-		clear_attach_data(&dst_cs_head, true);
+		clear_attach_data(&src_cs_head);
+		clear_attach_data(&dst_cs_head);
+		cpuset_free_dl_bw();
+		cgroup_taskset_for_each(task, css, tset)
+			dl_task_bw_unreserve(task);
 	} else {
 		attach_ctx.in_progress++;
 	}
@@ -3280,50 +3310,59 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 
 static void cpuset_cancel_attach(struct cgroup_taskset *tset)
 {
+	struct cgroup_subsys_state *css;
+	struct task_struct *task;
+
 	mutex_lock(&cpuset_mutex);
 	dec_attach_in_progress_locked();
-	clear_attach_data(&src_cs_head, true);
-	clear_attach_data(&dst_cs_head, true);
+	clear_attach_data(&src_cs_head);
+	clear_attach_data(&dst_cs_head);
+	cpuset_free_dl_bw();
+	cgroup_taskset_for_each(task, css, tset)
+		dl_task_bw_unreserve(task);
 	mutex_unlock(&cpuset_mutex);
 }
 
-/*
- * Protected by cpuset_mutex. cpus_attach is used only by cpuset_attach_task()
- * but we can't allocate it dynamically there.  Define it global and
- * allocate from cpuset_init().
- */
-static cpumask_var_t cpus_attach;
-
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
-	bool reserved = false;
+	struct cpuset_dl_bw *b = NULL;
 	struct mm_struct *mm;
+	int cpu = -1;
 
 	lockdep_assert_cpuset_lock_held();
 
-	if (cs != &top_cpuset)
-		guarantee_active_cpus(task, cpus_attach);
-	else
-		cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
-			       subpartitions_cpus);
+	cpuset_attach_cpus(cs, task, cpus_attach);
 	/*
-	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
-	 * deadline tasks that cpuset_can_attach() counted. Flag the move, so
-	 * that the bandwidth of this task moves with it.
+	 * cpuset_can_attach() made room for the bandwidth of each deadline
+	 * task it moves, in the root domain of the CPU it recorded with the
+	 * task. Flag the move with that CPU, so that the bandwidth goes
+	 * there. The task may have run on another CPU since: the CPU is not
+	 * worked out again.
+	 *
+	 * A task with no CPU recorded needs no move, or was not a deadline
+	 * task in cpuset_can_attach(). Its move is not flagged, so that it
+	 * cannot use the room made for another task. Its bandwidth stays
+	 * where it is counted.
 	 */
-	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
-		reserved = true;
+	cpu = dl_task_bw_reserved_cpu(task);
+	if (cpu >= 0)
+		b = per_cpu_ptr(&cpuset_dl_bw, cpu);
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	if (reserved) {
+	if (b) {
 		u64 moved;
 
 		/* Count what the move reports it moved. */
 		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
-							  &moved));
-		cs->sum_moved_dl_bw += moved;
+							  cpu, &moved));
+		b->moved += moved;
+		/*
+		 * set_cpus_allowed_dl() cleared the record, unless the task
+		 * left SCHED_DEADLINE. Clear it then too.
+		 */
+		dl_task_bw_unreserve(task);
 	} else {
 		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 	}
@@ -3407,8 +3446,9 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 		schedule_flush_migrate_mm();
 	cs->old_mems_allowed = attach_ctx.nodemask_to;
 out:
-	clear_attach_data(&src_cs_head, false);
-	clear_attach_data(&dst_cs_head, false);
+	clear_attach_data(&src_cs_head);
+	clear_attach_data(&dst_cs_head);
+	cpuset_free_dl_bw();
 	dec_attach_in_progress_locked();
 
 	mutex_unlock(&cpuset_mutex);
@@ -3913,6 +3953,7 @@ int __init cpuset_init(void)
 	cpuset1_init(&top_cpuset);
 
 	BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
+	BUG_ON(!zalloc_cpumask_var(&dl_attach_cpus, GFP_KERNEL));
 
 	if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT))
 		cpumask_andnot(isolated_cpus, cpu_possible_mask,
@@ -4211,7 +4252,7 @@ static void __cpuset_cpus_allowed_locked(struct task_struct *tsk, struct cpumask
 
 	cs = task_cs(tsk);
 	if (cs != &top_cpuset)
-		guarantee_active_cpus(tsk, pmask);
+		guarantee_active_cpus(cs, tsk, pmask);
 	/*
 	 * Tasks in the top cpuset won't get update to their cpumasks
 	 * when a hotplug online/offline event happens. So we include all
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 229b7bfa9634..e9f80407f75b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3250,6 +3250,22 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 		goto out;
 	}
 
+	/*
+	 * The caller made room for the deadline bandwidth of @p in the root
+	 * domain of ctx->dl_bw_cpu. The new mask can span two root domains:
+	 * keep @p in that one. Also wake a sleeping @p up on dest_cpu, since
+	 * select_fallback_rq() could pick a CPU of the other one.
+	 */
+	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+	    cpumask_test_cpu(ctx->dl_bw_cpu, cpu_valid_mask) &&
+	    cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask)) {
+		if (!cpumask_test_cpu(dest_cpu,
+				      cpu_rq(ctx->dl_bw_cpu)->rd->span))
+			dest_cpu = ctx->dl_bw_cpu;
+		if (!task_on_rq_queued(p))
+			p->wake_cpu = dest_cpu;
+	}
+
 	do_set_cpus_allowed(p, ctx);
 
 	return affine_move_task(rq, p, rf, dest_cpu, ctx->flags);
@@ -3300,15 +3316,18 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
 /*
  * Like set_cpus_allowed_ptr(), for a caller that has made room for the
- * deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p, and set @moved to the bandwidth moved, 0
- * if none. Return 0 on success, or a negative error code.
+ * deadline bandwidth of @p in the root domain of @cpu. If the move records
+ * @cpu with that bandwidth, move the bandwidth there with @p. Set @moved
+ * to the bandwidth moved, 0 if none. Return 0 on success, or a negative
+ * error code.
  */
 int set_cpus_allowed_bw_reserved(struct task_struct *p,
-				 const struct cpumask *new_mask, u64 *moved)
+				 const struct cpumask *new_mask, int cpu,
+				 u64 *moved)
 {
 	struct affinity_context ac = {
 		.new_mask  = new_mask,
+		.dl_bw_cpu = cpu,
 		.flags     = SCA_DL_BW_RESERVED,
 	};
 	int ret;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 5503820c5b26..b43106a59025 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3610,44 +3610,63 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * rebuild of the accounting. Moving it without room would leave no
 	 * root domain counting it.
 	 *
-	 * Report in @ctx the bandwidth moved. The caller made its own test
-	 * without the runqueue lock, so it must count only what this reports.
+	 * cpuset_can_attach() made room for @p in the root domain of
+	 * ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
+	 * there. Do not work out the CPU again: @p may have run on another
+	 * CPU since. __set_cpus_allowed_ptr_locked() keeps @p in that root
+	 * domain.
+	 *
+	 * Report in @ctx the bandwidth moved. The caller gives back the room
+	 * that no move used.
 	 */
 	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
-	    dl_task_needs_bw_move(p, ctx->new_mask)) {
+	    p->dl.dl_bw_resv_cpu == ctx->dl_bw_cpu) {
 		unsigned int src_cpu = dl_task_root_domain_cpu(p);
 		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
-		unsigned int cpu;
 
+		p->dl.dl_bw_resv_cpu = -1;
 		/*
-		 * We now free resources of the root_domain we are migrating
-		 * off. In the worst case, sched_setattr() may temporary fail
-		 * until we complete the update. A rebuild that dropped the
-		 * bandwidth of @p leaves nothing to free. dl_departed_lock
-		 * keeps a rebuild from changing dl_bw_gen meanwhile.
+		 * Nothing moves if @p left the CPUs it was to move to, or if
+		 * its root domain already counts it: @p ran there since.
 		 */
-		guard(raw_spinlock)(&dl_departed_lock);
-		if (p->dl.dl_bw_gen == dl_bw_gen) {
-			raw_spin_lock(&src_dl_b->lock);
-			__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
-			raw_spin_unlock(&src_dl_b->lock);
-		}
+		if (p->dl.dl_bw && !dl_entity_is_special(&p->dl) &&
+		    cpu_active(ctx->dl_bw_cpu) &&
+		    cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask) &&
+		    cpu_rq(src_cpu)->rd != cpu_rq(ctx->dl_bw_cpu)->rd) {
+			/*
+			 * We now free resources of the root_domain we are
+			 * migrating off. In the worst case, sched_setattr()
+			 * may temporary fail until we complete the update. A
+			 * rebuild that dropped the bandwidth of @p leaves
+			 * nothing to free. dl_departed_lock keeps a rebuild
+			 * from changing dl_bw_gen meanwhile.
+			 */
+			guard(raw_spinlock)(&dl_departed_lock);
+			if (p->dl.dl_bw_gen == dl_bw_gen) {
+				raw_spin_lock(&src_dl_b->lock);
+				__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+				raw_spin_unlock(&src_dl_b->lock);
+			}
 
-		/*
-		 * cpuset_can_attach() made room in the root domain of the new
-		 * mask. That root domain counts the bandwidth from now on.
-		 */
-		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
-		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
-		p->dl.dl_bw_gen = dl_bw_gen;
-		ctx->dl_bw_moved = p->dl.dl_bw;
+			/* Record the CPU the caller made room on. */
+			p->dl.dl_bw_cpu = ctx->dl_bw_cpu;
+			p->dl.dl_bw_gen = dl_bw_gen;
+			ctx->dl_bw_moved = p->dl.dl_bw;
+		}
 	}
 
 	set_cpus_allowed_common(p, ctx);
 }
 
-bool dl_task_needs_bw_move(struct task_struct *p,
-			   const struct cpumask *new_mask)
+/*
+ * Return the CPU that a move of @p to the CPUs of @new_mask records with
+ * its bandwidth. That is the CPU @p is on, if @p can stay there. Else it
+ * is the first active CPU of @new_mask, a housekeeping one if there is
+ * one. Return -1 if the bandwidth stays in the root domain that counts
+ * it. The caller holds the rq lock of @p.
+ */
+static int __dl_task_bw_move_cpu(struct task_struct *p,
+				 const struct cpumask *new_mask)
 {
 	unsigned int cpu;
 
@@ -3657,10 +3676,64 @@ bool dl_task_needs_bw_move(struct task_struct *p,
 	 * their bandwidth: dl_add_task_root_domain() leaves them out.
 	 */
 	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
-		return false;
+		return -1;
 
 	cpu = dl_task_root_domain_cpu(p);
-	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
+	if (cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask))
+		return -1;
+	/*
+	 * A task moved asleep is still on a CPU of the root domain it left.
+	 * If the new mask holds that CPU, the task wakes up there.
+	 */
+	cpu = task_cpu(p);
+	if (cpu_active(cpu) && cpumask_test_cpu(cpu, new_mask))
+		return cpu;
+	cpu = cpumask_first_and_and(new_mask, cpu_active_mask,
+				    housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+	if (cpu >= nr_cpu_ids)
+		cpu = cpumask_any_and(new_mask, cpu_active_mask);
+	return cpu < nr_cpu_ids ? (int)cpu : -1;
+}
+
+/*
+ * Work out the CPU that a move of @p to the CPUs of @new_mask records
+ * with its bandwidth. Record it with @p: a cpuset attach reserves room
+ * on it. Return it, or -1 if the bandwidth stays where it is.
+ */
+int dl_task_bw_reserve_cpu(struct task_struct *p, const struct cpumask *new_mask)
+{
+	struct rq_flags rf;
+	struct rq *rq;
+	int cpu;
+
+	/* Like __set_cpus_allowed_ptr(), keep within the user affinity. */
+	rq = task_rq_lock(p, &rf);
+	if (p->user_cpus_ptr &&
+	    cpumask_and(rq->scratch_mask, new_mask, p->user_cpus_ptr))
+		new_mask = rq->scratch_mask;
+	cpu = __dl_task_bw_move_cpu(p, new_mask);
+	p->dl.dl_bw_resv_cpu = cpu;
+	task_rq_unlock(rq, p, &rf);
+	return cpu;
+}
+
+/* Return the CPU that dl_task_bw_reserve_cpu() recorded with @p, or -1. */
+int dl_task_bw_reserved_cpu(struct task_struct *p)
+{
+	return READ_ONCE(p->dl.dl_bw_resv_cpu);
+}
+
+/* Forget the CPU that dl_task_bw_reserve_cpu() recorded with @p. */
+void dl_task_bw_unreserve(struct task_struct *p)
+{
+	struct rq_flags rf;
+	struct rq *rq;
+
+	if (READ_ONCE(p->dl.dl_bw_resv_cpu) < 0)
+		return;
+	rq = task_rq_lock(p, &rf);
+	p->dl.dl_bw_resv_cpu = -1;
+	task_rq_unlock(rq, p, &rf);
 }
 
 /* Assumes rq->lock is held */
@@ -4514,6 +4587,7 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	dl_se->dl_bw = 0;
 	__dl_clear_params(dl_se);
 	dl_se->dl_bw_cpu = -1;
+	dl_se->dl_bw_resv_cpu = -1;
 	INIT_LIST_HEAD(&dl_se->dl_departed_node);
 }
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 661da2b5c278..3cc287785b91 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2690,6 +2690,8 @@ struct affinity_context {
 	struct cpumask		*user_mask;
 	/* Out: the deadline bandwidth moved to the new root domain. */
 	u64			dl_bw_moved;
+	/* In, with SCA_DL_BW_RESERVED: the CPU the caller made room on. */
+	int			dl_bw_cpu;
 	unsigned int		flags;
 };
 
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 70b364b4f82d..59d78ae6b9fd 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1327,8 +1327,9 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
 	/*
 	 * Wait for a cpuset attach in progress before changing the affinity
 	 * of a deadline task. cpuset_can_attach() reserved the bandwidth of
-	 * the task for the CPUs it moves to, and cpuset_attach() moves it
-	 * there. A new affinity in between could change those CPUs.
+	 * the task on one CPU, and recorded that CPU with the task.
+	 * cpuset_attach() moves the bandwidth there. A new affinity in
+	 * between could leave that CPU out, and the bandwidth would not move.
 	 *
 	 * The policy is tested here with no lock. A task that becomes a
 	 * deadline task after this test is found under its runqueue lock,
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (22 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 20:01   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
                   ` (12 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A rebuild of the accounting counts each deadline task on the first
active CPU of its cpuset. Commit 318e18ed22e8 ("sched/deadline: Walk up
cpuset hierarchy to decide root domain when hot-unplug") made this
choice. It relies on the bandwidth check to keep such a CPU active.
This guess would go wrong in three ways:

 - A cpuset, like the root one, can hold CPUs isolated by
   isolcpus="domain" beside housekeeping CPUs. If an isolated CPU comes
   first, the task would be counted in the default root domain. Its own
   root domain would then be short of that bandwidth.
 - A task confined to isolated CPUs can see them all go offline. The
   rebuild would then find no active CPU, and hit its BUG_ON().
 - dl_task_offline_migration() moves a throttled task, with its
   bandwidth, off CPUs that went offline. The rebuild would count it in
   its cpuset, not where it moved.

In the first and third cases, a root domain would admit deadline tasks
beyond its capacity.

Fix this by choosing the CPU in this order:

 1. The CPU the task is on, if it is active and in the task's mask. A
    mask can hold CPUs of two root domains, for instance after
    select_fallback_rq() widened it.
 2. dl_bw_cpu, the CPU recorded with the bandwidth, if it is active and
    in the mask. A task moved asleep is not on its new CPUs yet.
 3. The first active CPU of the mask, a housekeeping one if any.
 4. With no active CPU in the mask: dl_bw_cpu if it is active, else an
    active CPU of the cpuset.

The mask of the task comes before its cpuset. A partition change sets
the task masks before its rebuild, and the housekeeping mask after.

Leave out a task with no active CPU, instead of the BUG_ON(). dl_bw_gen
then keeps its release from removing anything.
dl_task_offline_migration() counts it again when it moves it.

A task that select_fallback_rq() moves to another root domain still runs
where its bandwidth is not counted. The next patch moves the bandwidth
with it.

The Sashiko AI reviews of v1 to v4 found these cases.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Fixes: 318e18ed22e8 ("sched/deadline: Walk up cpuset hierarchy to decide root domain when hot-unplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 83 +++++++++++++++++++++++++++++++++++------
 1 file changed, 72 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b43106a59025..c765a403014a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -923,14 +923,20 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 			__dl_sub(dl_b, p->dl.dl_bw,
 				 cpumask_weight(src_rq->rd->span));
 			raw_spin_unlock(&dl_b->lock);
-
-			dl_b = &later_rq->rd->dl_bw;
-			raw_spin_lock(&dl_b->lock);
-			__dl_add(dl_b, p->dl.dl_bw,
-				 cpumask_weight(later_rq->rd->span));
-			raw_spin_unlock(&dl_b->lock);
 		}
+		/*
+		 * Count the bandwidth in the root domain the task moves to.
+		 * Do it also when a rebuild left the task out, for want of an
+		 * active CPU. A rebuild that reaches the task after this finds
+		 * its dl_bw_gen current, and leaves it alone.
+		 */
+		dl_b = &later_rq->rd->dl_bw;
+		raw_spin_lock(&dl_b->lock);
+		__dl_add(dl_b, p->dl.dl_bw,
+			 cpumask_weight(later_rq->rd->span));
+		raw_spin_unlock(&dl_b->lock);
 		p->dl.dl_bw_cpu = later_rq->cpu;
+		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
 	set_task_cpu(p, later_rq->cpu);
@@ -3795,6 +3801,55 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	cpuset_cpus_allowed_locked(p, cpus);
 }
 
+/*
+ * Return a CPU of the root domain where a rebuild of the accounting should
+ * count the bandwidth of @p, or nr_cpu_ids if there is none.
+ *
+ * Prefer the CPU @p is on, if it is active and in the mask of @p. The
+ * task runs, or wakes up, in the root domain of that CPU. A mask can hold
+ * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
+ * isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ *
+ * Else prefer the CPU recorded with the bandwidth, if it is active and in
+ * the mask. It is in the root domain that admitted the bandwidth, or that
+ * its last move put it in. A task moved asleep is still on a CPU of the
+ * root domain it left, out of its mask. Else take the first active CPU of
+ * the mask, a housekeeping one if there is one. A task on isolated CPUs
+ * that all went offline can have no active CPU at all.
+ */
+static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
+{
+	struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	int recorded = p->dl.dl_bw_cpu;
+	unsigned int cpu;
+
+	/*
+	 * Look at the mask of the task first. A change of the cpusets sets
+	 * the mask of each task before its rebuild. The housekeeping mask
+	 * that dl_get_task_effective_cpus() reads is updated after it. If no
+	 * CPU of the mask is active, take the CPU recorded if it is active.
+	 * dl_task_offline_migration() moved the task there, or an earlier
+	 * rebuild counted it there. Else ask the cpuset.
+	 */
+	cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+	cpu = task_cpu(p);
+	if (cpumask_test_cpu(cpu, msk))
+		return cpu;
+	if (cpumask_empty(msk)) {
+		if (recorded >= 0 && cpu_active(recorded))
+			return recorded;
+		dl_get_task_effective_cpus(p, msk);
+		cpumask_and(msk, msk, cpu_active_mask);
+	} else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
+		return recorded;
+	}
+	cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+	if (cpu < nr_cpu_ids)
+		return cpu;
+	return cpumask_first(msk);
+}
+
 /* From dl_rebuild_rd_accounting(), under the locks it asserts. */
 void dl_add_task_root_domain(struct task_struct *p)
 {
@@ -3802,7 +3857,6 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct rq *rq;
 	struct dl_bw *dl_b;
 	unsigned int cpu;
-	struct cpumask *msk;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
 	/*
@@ -3817,10 +3871,17 @@ void dl_add_task_root_domain(struct task_struct *p)
 		return;
 	}
 
-	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
-	dl_get_task_effective_cpus(p, msk);
-	cpu = cpumask_first_and(cpu_active_mask, msk);
-	BUG_ON(cpu >= nr_cpu_ids);
+	cpu = dl_task_rebuild_cpu(p);
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * No active CPU can count the bandwidth of this task. Leave it
+		 * out of this rebuild. dl_bw_gen then keeps its release from
+		 * removing anything. A later rebuild, or
+		 * dl_task_offline_migration(), counts it again.
+		 */
+		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+		return;
+	}
 
 	raw_spin_lock(&dl_departed_lock);
 	/*
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to the root domain it runs in
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (23 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:47   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
                   ` (11 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

When a CPU goes offline, select_fallback_rq() picks a new CPU for a
deadline task asleep on it. balance_push() moves a queued task the same
way. The new CPU can be in another root domain. With isolcpus="domain",
the isolated CPUs are in the default root domain. A cpuset that is not
a partition can hold CPUs of both root domains.

Commit 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all
levels after offline migration") fixed this for a throttled task.
dl_task_offline_migration() moves its bandwidth with it. A task that
select_fallback_rq() places still moves no bandwidth. So it runs in a
root domain that does not count its bandwidth, until the next rebuild.
That root domain can admit deadline tasks beyond its capacity.

Fix this in migrate_task_rq_dl(). The CPU recorded with the bandwidth
gives the root domain that counts it. If the new CPU is in another root
domain, move the bandwidth there. Record the new CPU. As
dl_task_offline_migration() does, add the bandwidth even without room,
since the task runs there. Add it also if a rebuild left the task out.

Compare the root domains, not the CPU the task leaves. That CPU can
still be in the mask when select_fallback_rq() picks a CPU of the other
root domain. A task can also wake up on its CPU with no migration while
another root domain counts its bandwidth. A cpuset move or a rebuild can
move the bandwidth while the task sleeps. A change of the mask can then
give that CPU back. So task_woken_dl() makes the same check.

Both checks read the record without dl_departed_lock, so a move inside
a root domain takes no lock. dl_task_move_bw() checks again under
dl_departed_lock. Its caller holds the pi_lock or the rq lock of the
task. Together, the two locks keep the record stable.

These cases of deadline_bw_test, added later in this series, fail
without this patch:
 - hotplug_fallback, hotplug_fallback_moved and
   hotplug_fallback_in_mask: a task woken on an isolated CPU stays
   counted in the housekeeping root domain.
 - no_admission_affinity_moves: without admission control, a task
   given an affinity of one isolated CPU stays counted in the
   housekeeping root domain.

This follows the Sashiko AI reviews of v4 and v5.

Fixes: 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all levels after offline migration")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 92 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 81 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c765a403014a..35f40185953a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2910,11 +2910,69 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
 	return cpu;
 }
 
-static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
+/*
+ * Return true if the root domain of @cpu may not count the bandwidth of
+ * @p. Either another root domain counts it, or a rebuild left @p out.
+ * Test without dl_departed_lock, so that a move inside a root domain
+ * takes no lock. dl_task_move_bw() tests again under it.
+ */
+static bool dl_task_bw_elsewhere(struct task_struct *p, int cpu)
+{
+	int bw_cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+	if (bw_cpu < 0 || !p->dl.dl_bw)
+		return false;
+	return READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen) ||
+	       cpu_rq(bw_cpu)->rd != cpu_rq(cpu)->rd;
+}
+
+/*
+ * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
+ * also if a rebuild left @p out. Add it even without room, since @p runs
+ * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
+ */
+static void dl_task_move_bw(struct task_struct *p, int new_cpu)
+{
+	struct dl_bw *dl_b;
+	int cpu;
+
+	if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
+		return;
+
+	guard(raw_spinlock)(&dl_departed_lock);
+	cpu = p->dl.dl_bw_cpu;
+	if (cpu < 0)
+		return;
+	if (p->dl.dl_bw_gen == dl_bw_gen) {
+		if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
+			return;
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+	}
+	dl_b = dl_bw_of(new_cpu);
+	raw_spin_lock(&dl_b->lock);
+	__dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
+	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_cpu = new_cpu;
+	p->dl.dl_bw_gen = dl_bw_gen;
+}
+
+static void migrate_task_rq_dl(struct task_struct *p, int new_cpu)
 {
 	struct rq_flags rf;
 	struct rq *rq;
 
+	/*
+	 * The new CPU can be in another root domain than the one that counts
+	 * the bandwidth. A mask can hold CPUs of two root domains, and
+	 * select_fallback_rq() can widen it. Move the bandwidth there, as
+	 * dl_task_offline_migration() does for a throttled task.
+	 */
+	if (unlikely(dl_task_bw_elsewhere(p, new_cpu)))
+		dl_task_move_bw(p, new_cpu);
+
 	if (READ_ONCE(p->__state) != TASK_WAKING)
 		return;
 
@@ -3577,6 +3635,15 @@ static void pull_dl_task(struct rq *this_rq)
  */
 static void task_woken_dl(struct rq *rq, struct task_struct *p)
 {
+	/*
+	 * A task can wake up on the CPU it slept on, with no migration,
+	 * while another root domain counts its bandwidth. The bandwidth
+	 * moved while the task slept out of its mask, and the mask then got
+	 * that CPU back. Move the bandwidth to this root domain.
+	 */
+	if (unlikely(dl_task_bw_elsewhere(p, cpu_of(rq))))
+		dl_task_move_bw(p, cpu_of(rq));
+
 	if (!task_on_cpu(rq, p) &&
 	    !test_tsk_need_resched(rq->curr) &&
 	    p->nr_cpus_allowed > 1 &&
@@ -3612,9 +3679,10 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * (different root_domains) entails a bandwidth update. Move the
 	 * bandwidth only if the caller made room for it in the destination,
 	 * as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
-	 * Otherwise, the bandwidth stays where it is counted, until the next
-	 * rebuild of the accounting. Moving it without room would leave no
-	 * root domain counting it.
+	 * Otherwise, the bandwidth stays where it is counted, until the task
+	 * runs in another root domain (migrate_task_rq_dl(), task_woken_dl()),
+	 * or the next rebuild of the accounting. Freeing it here without room
+	 * would leave no root domain counting it.
 	 *
 	 * cpuset_can_attach() made room for @p in the root domain of
 	 * ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
@@ -3806,10 +3874,11 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
  * count the bandwidth of @p, or nr_cpu_ids if there is none.
  *
  * Prefer the CPU @p is on, if it is active and in the mask of @p. The
- * task runs, or wakes up, in the root domain of that CPU. A mask can hold
- * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
- * isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * task runs, or most often wakes up, in the root domain of that CPU. A
+ * mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
+ * isolated by isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. A task that then runs in
+ * the other root domain takes its bandwidth along.
  *
  * Else prefer the CPU recorded with the bandwidth, if it is active and in
  * the mask. It is in the root domain that admitted the bandwidth, or that
@@ -3876,8 +3945,8 @@ void dl_add_task_root_domain(struct task_struct *p)
 		/*
 		 * No active CPU can count the bandwidth of this task. Leave it
 		 * out of this rebuild. dl_bw_gen then keeps its release from
-		 * removing anything. A later rebuild, or
-		 * dl_task_offline_migration(), counts it again.
+		 * removing anything. A later rebuild counts it again. So does
+		 * the move of the task to an active CPU.
 		 */
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 		return;
@@ -4071,7 +4140,8 @@ void dl_clear_root_domains(void)
 	int cpu;
 
 	guard(raw_spinlock_irqsave)(&dl_departed_lock);
-	dl_bw_gen++;
+	/* dl_task_bw_elsewhere() reads it without the lock. */
+	WRITE_ONCE(dl_bw_gen, dl_bw_gen + 1);
 	for_each_possible_cpu(cpu) {
 		if (!dl_bw_visited(cpu, cookie))
 			dl_clear_root_domain(cpu_rq(cpu)->rd);
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (24 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:50   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
                   ` (10 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

The root domain that admits a deadline task counts its bandwidth. The
mask of the task can hold CPUs of another root domain. With
isolcpus="domain", the isolated CPUs are in the default root domain. A
cpuset that is not a partition root can hold isolated and housekeeping
CPUs.

The code that picks a CPU for a task does not look at the root domain
that counts it:

 - select_fallback_rq() and dl_task_offline_migration() pick a CPU when
   the CPU of the task goes offline.
 - __set_cpus_allowed_ptr_locked() picks one when the mask changes.
 - select_task_rq_dl() can leave a task on the CPU it slept on, which
   can be in the other root domain.
 - dl_task_rebuild_cpu() picks the CPU that a rebuild counts a task
   with. It prefers the CPU the task is on, then a housekeeping CPU.

Each can take the task, or its bandwidth, to the other root domain
while its own root domain still has a CPU for it. The bandwidth then
moves with no admission test, and the root domain it arrives in can
exceed its limit.

Add dl_task_bw_span(), which returns the CPUs of the root domain that
counts a task. Make each of these places try one of those CPUs first.
After a rebuild the root domains are new, and the CPU recorded with the
bandwidth can be offline. So dl_task_rebuild_cpu() cannot use the root
domain. It looks at the kind of the CPU recorded, housekeeping or
isolated. It prefers a CPU of the mask of the same kind.

A task with no CPU left in its root domain still moves, and its
bandwidth with it. The next two patches refuse the changes that lead
to that.

The case hotplug_keeps_root_domain of deadline_bw_test, added later in
this series, tests this. A cpuset holds two isolated CPUs and two
housekeeping CPUs, H and K. A deadline task counted in the housekeeping
root domain sleeps on H. Then H goes offline. The task must wake up on
K. Without this patch it woke up on an isolated CPU: CPU 0, in a guest
with isolcpus=domain,0,1.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/core.c     | 38 +++++++++++++++++++++++
 kernel/sched/deadline.c | 69 +++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h    |  1 +
 3 files changed, 106 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e9f80407f75b..dc270a23179d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3264,6 +3264,23 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 			dest_cpu = ctx->dl_bw_cpu;
 		if (!task_on_rq_queued(p))
 			p->wake_cpu = dest_cpu;
+	} else {
+		const struct cpumask *span = dl_task_bw_span(p);
+		unsigned int cpu = nr_cpu_ids;
+
+		/*
+		 * No room was made for @p in another root domain. Keep @p
+		 * in the root domain that counts its bandwidth, if the new
+		 * mask holds one of its CPUs.
+		 */
+		if (span && !cpumask_test_cpu(dest_cpu, span))
+			cpu = cpumask_first_and_and(cpu_valid_mask,
+						    ctx->new_mask, span);
+		if (cpu < nr_cpu_ids) {
+			dest_cpu = cpu;
+			if (!task_on_rq_queued(p))
+				p->wake_cpu = dest_cpu;
+		}
 	}
 
 	do_set_cpus_allowed(p, ctx);
@@ -3668,8 +3685,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
 	int nid = cpu_to_node(cpu);
 	const struct cpumask *nodemask = NULL;
 	enum { cpuset, possible, fail } state = cpuset;
+	const struct cpumask *span;
 	int dest_cpu;
 
+	/*
+	 * Keep a deadline task in the root domain that counts its
+	 * bandwidth, if one of its CPUs is allowed and online.
+	 */
+	span = dl_task_bw_span(p);
+	if (span) {
+		for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+			if (is_cpu_allowed(p, dest_cpu))
+				return dest_cpu;
+		}
+	}
+
 	/*
 	 * If the node that the CPU is on has been offlined, cpu_to_node()
 	 * will return -1. There is no CPU on the node, and we should
@@ -3686,6 +3716,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
 	}
 
 	for (;;) {
+		/* The mask is wider now: look again in that root domain. */
+		if (span && state != cpuset) {
+			for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+				if (is_cpu_allowed(p, dest_cpu))
+					goto out;
+			}
+		}
+
 		/* Any allowed, online CPU? */
 		for_each_cpu(dest_cpu, p->cpus_ptr) {
 			if (!is_cpu_allowed(p, dest_cpu))
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 35f40185953a..5294546786f1 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -864,13 +864,19 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 
 	later_rq = find_lock_later_rq(p, rq);
 	if (!later_rq) {
+		const struct cpumask *span = dl_task_bw_span(p);
 		int cpu;
 
 		/*
 		 * If we cannot preempt any rq, fall back to pick any
-		 * online CPU:
+		 * online CPU. Try the root domain that counts @p first.
 		 */
-		cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
+		cpu = nr_cpu_ids;
+		if (span)
+			cpu = cpumask_first_and_and(cpu_active_mask,
+						    p->cpus_ptr, span);
+		if (cpu >= nr_cpu_ids)
+			cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
 		if (cpu >= nr_cpu_ids) {
 			/*
 			 * Failed to find any suitable CPU.
@@ -2865,6 +2871,7 @@ static int
 select_task_rq_dl(struct task_struct *p, int cpu, int flags)
 {
 	struct task_struct *curr, *donor;
+	const struct cpumask *span;
 	bool select_rq;
 	struct rq *rq;
 
@@ -2905,6 +2912,20 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
 		    dl_task_is_earliest_deadline(p, cpu_rq(target)))
 			cpu = target;
 	}
+
+	/*
+	 * Keep @p in the root domain that counts its bandwidth. Its mask
+	 * can hold CPUs of another root domain, with isolcpus="domain",
+	 * and @p can sleep on one of them.
+	 */
+	span = dl_task_bw_span(p);
+	if (span && !cpumask_test_cpu(cpu, span)) {
+		unsigned int in_span;
+
+		in_span = cpumask_first_and_and(span, p->cpus_ptr, cpu_active_mask);
+		if (in_span < nr_cpu_ids)
+			cpu = in_span;
+	}
 	rcu_read_unlock();
 
 	return cpu;
@@ -3669,6 +3690,29 @@ static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
 	return cpu;
 }
 
+/*
+ * Return the CPUs of the root domain that counts the bandwidth of @p, or
+ * NULL if none does. The mask of @p can hold CPUs of another root domain,
+ * with isolcpus="domain". Where a CPU is picked for @p, a CPU of this
+ * root domain comes first: no other root domain admitted @p.
+ *
+ * The caller holds the pi_lock or the rq lock of @p. With interrupts
+ * off, the root domain is not freed.
+ */
+const struct cpumask *dl_task_bw_span(struct task_struct *p)
+{
+	int cpu;
+
+	if (!task_has_dl_policy(p) || dl_entity_is_special(&p->dl) ||
+	    !p->dl.dl_bw)
+		return NULL;
+	cpu = READ_ONCE(p->dl.dl_bw_cpu);
+	if (cpu < 0 || !cpu_active(cpu) ||
+	    READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen))
+		return NULL;
+	return cpu_rq(cpu)->rd->span;
+}
+
 static void set_cpus_allowed_dl(struct task_struct *p,
 				struct affinity_context *ctx)
 {
@@ -3903,6 +3947,27 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
 	 */
 	cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
 	cpu = task_cpu(p);
+	/*
+	 * Keep the bandwidth where it was counted: with the housekeeping
+	 * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
+	 * domains are new, and the CPU recorded can be offline. So prefer a
+	 * CPU of the mask of the same kind as the CPU recorded: housekeeping
+	 * or not.
+	 */
+	if (recorded >= 0) {
+		const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+		bool rec_hk = cpumask_test_cpu(recorded, hk);
+		unsigned int same;
+
+		if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
+			return cpu;
+		if (cpumask_test_cpu(recorded, msk))
+			return recorded;
+		for_each_cpu(same, msk) {
+			if (cpumask_test_cpu(same, hk) == rec_hk)
+				return same;
+		}
+	}
 	if (cpumask_test_cpu(cpu, msk))
 		return cpu;
 	if (cpumask_empty(msk)) {
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3cc287785b91..f80ea5541b3f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -371,6 +371,7 @@ extern bool __checkparam_dl(const struct sched_attr *attr);
 extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
 extern int  dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
 extern int  dl_bw_deactivate(int cpu);
+extern const struct cpumask *dl_task_bw_span(struct task_struct *p);
 extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec);
 /*
  * SCHED_DEADLINE supports servers (nested scheduling) with the following
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (25 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:51   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
                   ` (9 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

dl_bw_deactivate() refuses to take a CPU offline when the rest of its
root domain cannot hold the deadline bandwidth counted there. It also
keeps the last CPU of a root domain that counts a deadline task. That
test comes from commit d4742f6ed7ea ("sched/deadline: Correctly account
for allocated bandwidth during hotplug").

That test looks at the root domain, not at the mask of each task. With
isolcpus="domain", a cpuset that is not a partition root can hold one
housekeeping CPU and some isolated CPUs. A deadline task in it can be
counted in the housekeeping root domain. When that housekeeping CPU
goes offline, the root domain keeps its other CPUs, so the offline is
accepted. But the task has only isolated CPUs left. It runs in the
default root domain, which did not admit it. Its bandwidth follows with
no admission test, and the default root domain can exceed its limit.

Refuse the offline with EBUSY in that case. Walk the deadline tasks
counted in the root domain of the CPU. Refuse if one of them has CPUs
left to run on, but none in that root domain. Skip the walk if no task
holds a bandwidth. A task whose mask holds no other CPU gets the CPUs of
an ancestor of its cpuset: add cpuset_cpus_allowed_without() to tell
which.

Do not refuse during a suspend. It takes the CPUs offline with the
tasks frozen, and the resume puts them back. Do not refuse without
admission control either: there is no limit to exceed.

An administrator who needs the CPU offline can first give the cpuset of
the task another CPU of that root domain. The offline is then accepted,
and the task wakes up on that CPU: the hotplug_keeps_root_domain case
tests this. The administrator can also turn admission control off, by
writing -1 to /proc/sys/kernel/sched_rt_runtime_us.

The walk cannot miss a task. sched_setattr() only makes a deadline task
of a task whose mask holds its whole root domain. sched_setaffinity()
only gives a deadline task such a mask. A mask also gets smaller by a
cpuset change or a cpuset move, and both exclude CPU hotplug.

For user-space, the refused offline is a write of 0 to
/sys/devices/system/cpu/cpuN/online that fails with EBUSY. An offline
that leaves each deadline task a CPU in its root domain is accepted as
before.

The case hotplug_last_refused of deadline_bw_test, added later in this
series, tests this. Without this patch the offline is accepted.

Fixes: d4742f6ed7ea ("sched/deadline: Correctly account for allocated bandwidth during hotplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/cpuset.h  |  9 +++++++
 kernel/cgroup/cpuset.c  | 20 +++++++++++++++
 kernel/sched/deadline.c | 56 ++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 84 insertions(+), 1 deletion(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 5d0c01a25354..68849539cc8b 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -79,6 +79,8 @@ extern void lockdep_assert_cpuset_lock_held(void);
 extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
 extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
 extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern void cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+					struct cpumask *mask);
 extern int cpuset_num_cpus(struct cgroup *cgroup);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
@@ -214,6 +216,13 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)
 	return false;
 }
 
+static inline void cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+					       struct cpumask *mask)
+{
+	cpumask_and(mask, task_cpu_possible_mask(p), cpu_active_mask);
+	cpumask_clear_cpu(cpu, mask);
+}
+
 static inline int cpuset_num_cpus(struct cgroup *cgroup)
 {
 	return num_online_cpus();
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0d55f092d9cb..ef5ad8e38f06 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4305,6 +4305,26 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)
 	spin_unlock_irqrestore(&callback_lock, flags);
 }
 
+/*
+ * Set @pmask to the active CPUs that the cpuset of @tsk gives it once @cpu
+ * is offline: those of the nearest ancestor that keeps one. Leave it empty
+ * if there is none. The caller holds the CPU hotplug lock for writing,
+ * which keeps the cpusets and their tasks as they are.
+ */
+void cpuset_cpus_allowed_without(struct task_struct *tsk, int cpu,
+				 struct cpumask *pmask)
+{
+	struct cpuset *cs;
+
+	guard(rcu)();
+	for (cs = task_cs(tsk); cs; cs = parent_cs(cs)) {
+		cpumask_and(pmask, cs->effective_cpus, cpu_active_mask);
+		cpumask_clear_cpu(cpu, pmask);
+		if (!cpumask_empty(pmask))
+			return;
+	}
+}
+
 /**
  * cpuset_cpus_allowed_fallback - final fallback before complete catastrophe.
  * @tsk: pointer to task_struct with which the scheduler is struggling
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 5294546786f1..096c0ec55cdc 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -4896,9 +4896,63 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
 	return overflow ? -EBUSY : 0;
 }
 
+/*
+ * Return true if @p has CPUs to run on once @cpu is offline, but none in
+ * the root domain of @cpu, which counts its bandwidth.
+ */
+static bool dl_task_leaves_span(struct task_struct *p, int cpu,
+				const struct cpumask *span)
+{
+	struct cpumask *msk;
+
+	guard(raw_spinlock_irqsave)(&p->pi_lock);
+	if (dl_task_bw_span(p) != span)
+		return false;
+	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	cpumask_and(msk, p->cpus_ptr, cpu_active_mask);
+	cpumask_clear_cpu(cpu, msk);
+	/* With no CPU left, the cpuset gives @p the CPUs of an ancestor. */
+	if (cpumask_empty(msk))
+		cpuset_cpus_allowed_without(p, cpu, msk);
+	return !cpumask_empty(msk) && !cpumask_intersects(msk, span);
+}
+
+/*
+ * Return true if @cpu is the last CPU that a deadline task can run on in
+ * the root domain that counts its bandwidth. With @cpu offline, the task
+ * would run in another root domain, which did not admit it. The mask of
+ * a task holds CPUs of two root domains only with isolcpus="domain".
+ *
+ * A suspend takes the CPUs offline with the tasks frozen, and the resume
+ * puts them back. Without admission control, there is no limit to exceed.
+ * Return false in both cases.
+ */
+static bool dl_cpu_last_of_task(int cpu)
+{
+	const struct cpumask *span;
+	struct task_struct *g, *p;
+
+	if (cpuhp_tasks_frozen || !dl_bandwidth_enabled())
+		return false;
+	/* No task holds a bandwidth: do not walk the task list. */
+	if (!atomic_read(&dl_nr_bw_tasks))
+		return false;
+	guard(rcu)();
+	span = cpu_rq(cpu)->rd->span;
+	for_each_process_thread(g, p) {
+		if (task_has_dl_policy(p) && dl_task_leaves_span(p, cpu, span))
+			return true;
+	}
+	return false;
+}
+
 int dl_bw_deactivate(int cpu)
 {
-	return dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+	int ret = dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+
+	if (!ret && dl_cpu_last_of_task(cpu))
+		ret = -EBUSY;
+	return ret;
 }
 
 int dl_bw_alloc(int cpu, u64 dl_bw)
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of its root domain
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (26 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 20:08   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 29/37] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
                   ` (8 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A cpuset attach asks the destination root domain for room before it
moves a deadline task there. A change of the CPUs of a cpuset asks
nothing. validate_change() only checks that a partition root keeps
enough capacity for its deadline bandwidth. That check comes from
commit f82f80426f7a ("sched/deadline: Ensure that updates to exclusive
cpusets don't break AC").

With isolcpus="domain", a cpuset that is not a partition root can hold
housekeeping CPUs and isolated CPUs, which are in two root domains. Two
changes can take every CPU of one root domain away from that cpuset:

 - a write to cpuset.cpus, of that cpuset or of an ancestor;
 - a partition root that takes those CPUs, when it is created or when
   it grows.

A deadline task counted in that root domain then runs in the other one,
which did not admit it. Its bandwidth follows with no admission test,
and that root domain can exceed its limit.

Refuse both:

 - A write to cpuset.cpus fails with EBUSY, and changes nothing. The
   cpuset.cpus file of cgroup v1 gets the same check.
 - The write that makes the partition root, or makes it grow, still
   succeeds. cpuset makes the partition root invalid. For a "root"
   partition, cpuset.cpus.partition then reads "root invalid (Deadline
   task would leave its root domain)".

A task goes to another root domain by a move to another cpuset, which
asks for room.

A change that leaves each deadline task a CPU in its root domain is
accepted as before.

dl_task_keeps_root_domain() tells whether a new mask keeps a task in
the root domain that counts it. Without admission control it accepts
any mask: there is no limit to exceed. An administrator who needs the
change can turn admission control off, by writing -1 to
/proc/sys/kernel/sched_rt_runtime_us.

The cases mask_out_refused and partition_takes_refused of
deadline_bw_test, added later in this series, test this. Without this
patch the write is accepted and the partition root is valid.

Fixes: f82f80426f7a ("sched/deadline: Ensure that updates to exclusive cpusets don't break AC")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched/deadline.h  |   2 +
 kernel/cgroup/cpuset-internal.h |   1 +
 kernel/cgroup/cpuset.c          | 179 +++++++++++++++++++++++++++++++-
 kernel/sched/deadline.c         |  19 ++++
 4 files changed, 198 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index cbb7cd71db58..d3796d874aa2 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -61,6 +61,8 @@ extern int dl_task_bw_reserve_cpu(struct task_struct *p,
 				  const struct cpumask *new_mask);
 extern int dl_task_bw_reserved_cpu(struct task_struct *p);
 extern void dl_task_bw_unreserve(struct task_struct *p);
+extern bool dl_task_keeps_root_domain(struct task_struct *p,
+				      const struct cpumask *new_mask);
 
 static inline bool dl_server(struct sched_dl_entity *dl_se)
 {
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index a83443dfb788..8fe0225704eb 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -36,6 +36,7 @@ enum prs_errcode {
 	PERR_ACCESS,
 	PERR_REMOTE,
 	PERR_ISOPARENT,
+	PERR_DLTASK,
 };
 
 /* bits in struct cpuset flags field */
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ef5ad8e38f06..5b4f7d4ef3ae 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -61,6 +61,7 @@ static const char * const perr_strings[] = {
 	[PERR_ACCESS]    = "Enable partition not permitted",
 	[PERR_REMOTE]    = "Have remote partition underneath",
 	[PERR_ISOPARENT] = "Root partition is not allowed with isolated parent",
+	[PERR_DLTASK]    = "Deadline task would leave its root domain",
 };
 
 /*
@@ -1126,6 +1127,92 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
 		cpuset_force_rebuild();
 }
 
+/*
+ * Return false if the CPUs of @new_cpus make a deadline task of @cp leave
+ * the root domain that counts its bandwidth. Called under rcu_read_lock(),
+ * which it drops and takes again.
+ */
+static bool dl_tasks_keep_cpus(struct cpuset *cp, const struct cpumask *new_cpus)
+{
+	struct css_task_iter it;
+	struct task_struct *task;
+	bool keep = true;
+
+	if (!css_tryget_online(&cp->css))
+		return true;
+	rcu_read_unlock();
+
+	css_task_iter_start(&cp->css, 0, &it);
+	while (keep && (task = css_task_iter_next(&it))) {
+		/* The top cpuset leaves the mask of these tasks alone. */
+		if (cp == &top_cpuset &&
+		    (task->flags & (PF_KTHREAD | PF_NO_SETAFFINITY)))
+			continue;
+		keep = dl_task_keeps_root_domain(task, new_cpus);
+	}
+	css_task_iter_end(&it);
+
+	rcu_read_lock();
+	css_put(&cp->css);
+	return keep;
+}
+
+/*
+ * Return false if a partition that takes the CPUs of @xcpus makes a
+ * deadline task leave the root domain that counts its bandwidth. A cpuset
+ * holds CPUs of two root domains only with isolcpus="domain". Such a task
+ * would run in a root domain that did not admit it.
+ *
+ * @root is the partition root that gives the CPUs. Its cpusets lose them,
+ * down to the next partition roots. A cpuset left with no CPU gets those
+ * of its nearest ancestor that keeps one. @part is the cpuset that takes
+ * the CPUs. If @enable, it becomes a partition root: its own cpusets keep
+ * only CPUs of @xcpus. @new_cpus is a temporary mask.
+ */
+static bool dl_tasks_keep_without(struct cpuset *root, struct cpuset *part,
+				  const struct cpumask *xcpus, bool enable,
+				  struct cpumask *new_cpus)
+{
+	struct cgroup_subsys_state *pos_css;
+	struct cpuset *cp, *a;
+	bool keep = true;
+
+	rcu_read_lock();
+	cpuset_for_each_descendant_pre(cp, pos_css, root) {
+		if (cp == part || (cp != root && is_partition_valid(cp))) {
+			pos_css = css_rightmost_descendant(pos_css);
+			continue;
+		}
+		if (!cpumask_intersects(cp->effective_cpus, xcpus))
+			continue;
+		for (a = cp; ; a = parent_cs(a)) {
+			if (cpumask_andnot(new_cpus, a->effective_cpus, xcpus) ||
+			    a == root)
+				break;
+		}
+		keep = dl_tasks_keep_cpus(cp, new_cpus);
+		if (!keep)
+			break;
+	}
+	if (keep && enable) {
+		cpuset_for_each_descendant_pre(cp, pos_css, part) {
+			if (cp != part && is_partition_valid(cp)) {
+				pos_css = css_rightmost_descendant(pos_css);
+				continue;
+			}
+			if (cpumask_subset(cp->effective_cpus, xcpus))
+				continue;
+			if (!cpumask_and(new_cpus, cp->effective_cpus, xcpus))
+				cpumask_copy(new_cpus, xcpus);
+			keep = dl_tasks_keep_cpus(cp, new_cpus);
+			if (!keep)
+				break;
+		}
+	}
+	rcu_read_unlock();
+	return keep;
+}
+
 /*
  * tasks_nocpu_error - Return true if tasks will have no effective_cpus
  */
@@ -1676,6 +1763,8 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
 		return PERR_NOCPUS;
 	if (prstate_housekeeping_conflict(new_prs, PRS_ROOT, tmp->new_cpus, NULL))
 		return PERR_HKEEPING;
+	if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->new_cpus, true, tmp->addmask))
+		return PERR_DLTASK;
 
 	spin_lock_irq(&callback_lock);
 	partition_xcpus_add(new_prs, NULL, tmp->new_cpus);
@@ -1779,6 +1868,9 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 		else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
 						       excpus, tmp->delmask))
 			WRITE_ONCE(cs->prs_err, PERR_HKEEPING);
+		else if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->addmask,
+						false, tmp->new_cpus))
+			WRITE_ONCE(cs->prs_err, PERR_DLTASK);
 		if (cs->prs_err)
 			goto invalidate;
 	}
@@ -1926,6 +2018,10 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		if (nocpu)
 			return PERR_NOCPUS;
 
+		if (!dl_tasks_keep_without(parent, cs, tmp->delmask, true,
+					   tmp->new_cpus))
+			return PERR_DLTASK;
+
 		/*
 		 * This function will only be called when all the preliminary
 		 * checks have passed. At this point, the following condition
@@ -2013,6 +2109,15 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 			deleting = false;
 			adding = cpumask_and(tmp->addmask,
 					     cs->effective_xcpus, parent->effective_xcpus);
+		} else if (deleting &&
+			   !dl_tasks_keep_without(parent, cs, tmp->delmask,
+						  is_partition_invalid(cs),
+						  tmp->new_cpus)) {
+			/* As above: the partition becomes invalid. */
+			part_error = PERR_DLTASK;
+			deleting = false;
+			adding = cpumask_and(tmp->addmask,
+					     cs->effective_xcpus, parent->effective_xcpus);
 		}
 	} else {
 		/*
@@ -2084,10 +2189,13 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 				}
 				rcu_read_unlock();
 			}
-			if (exclusive)
-				deleting = !cpumask_empty(tmp->delmask);
-			else
+			if (!exclusive)
 				part_error = PERR_NOTEXCL;
+			else if (!dl_tasks_keep_without(parent, cs, tmp->delmask,
+							true, tmp->new_cpus))
+				part_error = PERR_DLTASK;
+			else
+				deleting = !cpumask_empty(tmp->delmask);
 		}
 		/*
 		 * When both cs and parent remain valid, parent's
@@ -2496,6 +2604,66 @@ static void partition_cpus_change(struct cpuset *cs, struct cpuset *trialcs,
 	}
 }
 
+/*
+ * Set @new_cpus to the effective CPUs that @cp gets if @cs takes the CPUs
+ * of @trialcs. @cp is @cs or one of its descendants.
+ */
+static void trial_effective_cpus(struct cpuset *cp, struct cpuset *cs,
+				 struct cpuset *trialcs, struct cpumask *new_cpus)
+{
+	struct cpuset *parent = parent_cs(cs), *a;
+	int depth = 0, i;
+
+	if (!cpumask_and(new_cpus, trialcs->cpus_allowed, parent->effective_cpus) &&
+	    is_in_v2_mode())
+		cpumask_copy(new_cpus, parent->effective_cpus);
+	for (a = cp; a != cs; a = parent_cs(a))
+		depth++;
+	/* From the child of @cs down to @cp, as compute_effective_cpumask(). */
+	while (depth--) {
+		for (a = cp, i = 0; i < depth; i++)
+			a = parent_cs(a);
+		if (cpumask_intersects(new_cpus, a->cpus_allowed) || !is_in_v2_mode())
+			cpumask_and(new_cpus, new_cpus, a->cpus_allowed);
+	}
+}
+
+/*
+ * Return false if the CPUs of @trialcs make a deadline task of @cs, or of
+ * a descendant, leave the root domain that counts its bandwidth. A cpuset
+ * holds CPUs of two root domains only with isolcpus="domain". The task
+ * would run in a root domain that did not admit it. A move of the task to
+ * another cpuset is the way there: it asks the other root domain for room.
+ *
+ * A partition root takes its CPUs from its parent, which checks the same
+ * for its own tasks: see dl_tasks_keep_without().
+ */
+static bool dl_tasks_keep_root_domain(struct cpuset *cs, struct cpuset *trialcs,
+				      struct cpumask *new_cpus)
+{
+	struct cgroup_subsys_state *pos_css;
+	struct cpuset *cp;
+	bool keep = true;
+
+	if (is_partition_valid(cs))
+		return true;
+	rcu_read_lock();
+	cpuset_for_each_descendant_pre(cp, pos_css, cs) {
+		if (is_partition_valid(cp)) {
+			pos_css = css_rightmost_descendant(pos_css);
+			continue;
+		}
+		trial_effective_cpus(cp, cs, trialcs, new_cpus);
+		if (cpumask_subset(cp->effective_cpus, new_cpus))
+			continue;
+		keep = dl_tasks_keep_cpus(cp, new_cpus);
+		if (!keep)
+			break;
+	}
+	rcu_read_unlock();
+	return keep;
+}
+
 /**
  * update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it
  * @cs: the cpuset to consider
@@ -2528,6 +2696,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,
 	if (alloc_tmpmasks(&tmp))
 		return -ENOMEM;
 
+	if (!dl_tasks_keep_root_domain(cs, trialcs, tmp.new_cpus)) {
+		free_tmpmasks(&tmp);
+		return -EBUSY;
+	}
+
 	/*
 	 * Check all the descendants in update_cpumasks_hier() if
 	 * effective_xcpus is to be changed.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 096c0ec55cdc..61b50266468f 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -4955,6 +4955,25 @@ int dl_bw_deactivate(int cpu)
 	return ret;
 }
 
+/*
+ * Return false if @p has CPUs in the root domain that counts its
+ * bandwidth and @new_mask holds none of them. A cpuset refuses such a
+ * mask: only a move to another cpuset makes room in another root domain.
+ */
+bool dl_task_keeps_root_domain(struct task_struct *p,
+			       const struct cpumask *new_mask)
+{
+	const struct cpumask *span;
+
+	if (!dl_bandwidth_enabled() || !task_has_dl_policy(p))
+		return true;
+	guard(raw_spinlock_irqsave)(&p->pi_lock);
+	guard(rcu)();
+	span = dl_task_bw_span(p);
+	return !span || !cpumask_intersects(span, p->cpus_ptr) ||
+	       cpumask_intersects(span, new_mask);
+}
+
 int dl_bw_alloc(int cpu, u64 dl_bw)
 {
 	return dl_bw_manage(dl_bw_req_alloc, cpu, dl_bw);
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 29/37] sched/deadline: Cancel the replenishment timer with the parameters
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (27 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 30/37] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
                   ` (7 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

A task that leaves SCHED_DEADLINE has its parameters cleared at its
0-lag time. Its replenishment timer can still be armed then. If the
task comes back and throttles before that timer fires, start_dl_timer()
does not restart it, since it is queued. The replenishment would then
come at the period of the reservation the task left. Too early, it would
give the task runtime it has not earned. Too late, it would delay the
task by up to a period.

Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole") armed
the replenishment timer when a task leaves the class. The timer then
reset the task if it was still out. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
that reset to the inactive timer, at the 0-lag time. Nothing cancels
the replenishment timer there.

The timer of a throttled task fires at its next period, which can come
after the 0-lag time:
- A task that yielded is throttled with runtime left. Its 0-lag time
  comes before its deadline.
- A constrained-deadline task that overran by little reaches its 0-lag
  time before the end of its period.

Fix this by cancelling the timer where the parameters are cleared. This
includes task_non_contending(). switched_from_dl() calls it to release
the bandwidth of a task that leaves asleep past its 0-lag time. A
constrained task that overran as it blocked has the timer armed there
too.

The Sashiko AI review of the first posting of this series found this.
It has no reproducer.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 61b50266468f..72bafa46ae41 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -650,6 +650,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
 				dl_task_release_bw(p);
+				cancel_replenish_timer(dl_se);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -2446,6 +2447,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		}
 
 		dl_task_release_bw(p);
+		cancel_replenish_timer(dl_se);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 30/37] sched/deadline: Count the bandwidth a boosted task is given
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (28 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 29/37] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
                   ` (6 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

sched_setattr(SCHED_DEADLINE) on a sleeping task that priority
inheritance boosted into the class leaves the new bandwidth out of
this_bw. switched_to_dl() does not run, since the task is in the class
already. sched_dl_overflow() changes this_bw only for a task of the
deadline policy. When the task wakes up, __add_running_bw() warns:

	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);

When the task leaves the runqueue, this_bw would stay short by that
bandwidth, and skew GRUB reclaiming on that CPU.

Fix this by changing this_bw in __setparam_dl(), which every task that
gains deadline parameters goes through. If no runqueue counts the
bandwidth of the task, dl_change_utilization() now adds the new one. It
now also runs when the bandwidth does not change. It then returns early,
so that a sleeping task keeps its running_bw and its inactive timer.

A sugov kthread given a reservation of its own now has its bandwidth
counted in this_bw. It reaches dl_change_utilization(), so the
WARN_ON_ONCE() for SCHED_FLAG_SUGOV there goes. sched_dl_overflow()
returned early for an unchanged bandwidth, even if no root domain counts
it. That is the case for such a kthread, and for a task that a rebuild
left out. Admit such a bandwidth as new. Without room, that fails with
EBUSY. The Sashiko AI review of the third posting found this early
return.

A reproducer shows the warning, and this_bw unchanged by the change.
With this patch, this_bw takes the new bandwidth, and nothing warns.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 54 ++++++++++++++++++++++++++++++++---------
 1 file changed, 43 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 72bafa46ae41..69a338c1526d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -529,22 +529,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+/*
+ * Make the runqueue of a sleeping task count @new_bw as the bandwidth of
+ * the task. Do nothing for a queued task. The dequeue and the enqueue
+ * around the change of its parameters update this_bw.
+ */
 static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 {
-	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct rq *rq = task_rq(p);
 
 	if (task_on_rq_queued(p))
 		return;
 
 	/*
-	 * The task is waking up on another CPU, and no runqueue counts its
-	 * bandwidth. The enqueue adds it, with the new parameters that
-	 * __setparam_dl() sets right after this.
+	 * No runqueue counts the bandwidth of the task. Add the new one to
+	 * this runqueue, so that add_rq_bw() adds nothing later. A task that
+	 * enters the class asleep comes here, and so does a special entity
+	 * given a reservation of its own. So does a task waking up on
+	 * another CPU: task_cpu() already points at the CPU it wakes on.
 	 */
-	if (!p->dl.dl_bw_oncpu)
+	if (!dl_se->dl_bw_oncpu) {
+		__add_rq_bw(new_bw, &rq->dl);
+		dl_se->dl_bw_oncpu = 1;
+		return;
+	}
+
+	/* Same bandwidth: leave running_bw and the inactive timer alone. */
+	if (new_bw == dl_se->dl_bw)
 		return;
 
-	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+	dl_rq_change_utilization(rq, dl_se, new_bw);
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4546,10 +4561,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
 
-	/* !deadline task may carry old deadline bandwidth */
-	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
-		return 0;
-
 	/*
 	 * Change the bandwidth of a deadline task where it is counted: in
 	 * the root domain of the CPU recorded with it. A task moved asleep
@@ -4564,6 +4575,16 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 */
 	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
 		  p->dl.dl_bw_gen == dl_bw_gen;
+
+	/*
+	 * !deadline task may carry old deadline bandwidth. The same
+	 * bandwidth needs no admission if it is counted. If it is not, admit
+	 * it as new below. A special entity given a reservation of its own
+	 * has no bandwidth counted, nor has a task that a rebuild left out.
+	 */
+	if (new_bw == p->dl.dl_bw && counted)
+		return 0;
+
 	cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
 	dl_b = dl_bw_of(cpu);
 
@@ -4593,7 +4614,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		if (counted)
 			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
-		dl_change_utilization(p, new_bw);
 		err = 0;
 	} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
 		/*
@@ -4643,8 +4663,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	u64 period = attr->sched_period ?: attr->sched_deadline;
 	bool held;
 
+	/*
+	 * Change the bandwidth that the runqueue counts here, with the
+	 * parameters. A task that priority inheritance boosted into the
+	 * class is in it already, so switched_to_dl() does not run for it.
+	 *
+	 * A special entity has no bandwidth on a runqueue. Test the flags
+	 * of the request, since those of the entity are still the old ones.
+	 */
+	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
+
 	held = dl_se_holds_bw(dl_se);
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (29 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 30/37] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 20:12   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
                   ` (5 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit fb00aca47440 ("rtmutex: Turn the plist into an rb-tree") sorts
the waiters of an rt_mutex by priority. For deadline tasks, "the one
with the earliest deadline wins". A waiter takes the deadline of its
task.

A task can enter the class while it waits on an rt_mutex.
sched_setattr() can make it SCHED_DEADLINE. A priority inheritance
boost can also put it in the class, if it owns another rt_mutex.
switched_to_dl() leaves a task that is not queued to its wakeup. But
the chain walk reads the deadline first, and copies it into the waiter.
A task new to the class has a deadline of 0.

So the waiter goes before every other deadline waiter. It takes the
rt_mutex first, whatever their deadlines. Those waiters can then miss
their deadlines. It also boosts a deadline owner whose own deadline is
earlier.

Fix this by starting a new period in switched_to_dl() for a task that
waits on an rt_mutex, if its deadline has passed. The period uses the
reservation the task runs on: its own, or the copy a boost made. This
is what setup_new_dl_entity() does at an enqueue. At the wakeup,
update_dl_entity() then checks that period as for any wakeup.

The setattr_blocked_keeps_order and boost_blocked_keeps_order cases of
deadline_bw_test, added later in this series, test this. Both fail on
v7.4-rc1 and on the series without this patch. The triage of the Sashiko
AI review of the fifth posting found the bug.

Fixes: fb00aca47440 ("rtmutex: Turn the plist into an rb-tree")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 69a338c1526d..ffa8b946e804 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -125,6 +125,12 @@ void dl_deboost(struct sched_dl_entity *dl_se)
 	}
 	dl_drop_boost(dl_se);
 }
+
+/* Return true if @p waits on an rt_mutex. The caller holds its pi_lock. */
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+	return p->pi_blocked_on;
+}
 #else /* !CONFIG_RT_MUTEXES: */
 static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
 {
@@ -135,6 +141,11 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
 {
 	return false;
 }
+
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+	return false;
+}
 #endif /* !CONFIG_RT_MUTEXES */
 
 static inline u8 dl_get_type(struct sched_dl_entity *dl_se, struct rq *rq)
@@ -4352,6 +4363,17 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	if (!task_on_rq_queued(p)) {
 		add_rq_bw(&p->dl, &rq->dl);
 
+		/*
+		 * A task that waits on an rt_mutex cannot wait for its wakeup.
+		 * The chain walk sorts its waiter by its deadline first. A task
+		 * new to the class has a deadline of 0, which goes before every
+		 * real one. Start a new period now, as setup_new_dl_entity()
+		 * does at an enqueue.
+		 */
+		if (dl_task_pi_blocked(p) && !p->dl.dl_throttled &&
+		    dl_time_before(p->dl.deadline, rq_clock(rq)))
+			replenish_dl_new_period(&p->dl, rq);
+
 		return;
 	}
 
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (30 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 20:08   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
                   ` (4 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
keeps the runtime and the deadline of a task that leaves
SCHED_DEADLINE. A task that comes back before its deadline goes on with
them. So leaving the class and coming back gives no new runtime.

But the task can come back with new parameters. If it is asleep, its
wakeup checks the runtime left against them, with dl_entity_overflow().
If it is runnable, enqueue_dl_entity() starts a new period only if the
deadline has passed. The runtime left is not checked.

So a task can come back with a smaller runtime, and still run on the
old one. Admission control counts only the new one. Juri Lelli found
this case. On v7.4-rc1, a task of 50ms every 100ms came back through
SCHED_OTHER with 10ms every 100ms. It ran 39ms of the next 100ms.

Fix this by applying the wakeup rule to a task that enters the class
runnable, with update_dl_entity(). It starts a new period, or reduces
the runtime of a constrained task, if the runtime left does not fit the
new parameters. The task gets no more than a wakeup would give it.

The return_running_new_params case of deadline_bw_test, added later in
this series, tests this. It fails on v7.4-rc1 and passes with this
patch. The return_asleep_new_params case tests the wakeup, and passes
on both.

Fixes: a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
Reported-by: Juri Lelli <juri.lelli@redhat.com>
Closes: https://lore.kernel.org/all/avHvPjz4dL0zNyGb@jlelli-thinkpadt14gen4.remote.csb/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 kernel/sched/deadline.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ffa8b946e804..b47c13c8714d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2723,6 +2723,14 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
 		update_dl_entity(dl_se);
 	} else if (flags & ENQUEUE_REPLENISH) {
 		replenish_dl_entity(dl_se);
+	} else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+		/*
+		 * The task enters the class runnable. It may come back before
+		 * its deadline, with the runtime left from older parameters.
+		 * Check that runtime against the new parameters, as a wakeup
+		 * does.
+		 */
+		update_dl_entity(dl_se);
 	} else if ((flags & ENQUEUE_MOVE) &&
 		   !is_dl_boosted(dl_se) &&
 		   dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (31 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 20:04   ` sashiko-bot
  2026-11-16 19:05 ` [PATCH v6 34/37] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
                   ` (3 subsequent siblings)
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel, stable

Commit 40767b0dc768 ("sched/deadline: Fix deadline parameter
modification handling") lets new parameters take effect at the next
period. So sched_setattr() cannot give a task a fresh runtime.

But admission control counts the new bandwidth at once. When a task
shrinks its reservation, the runtime left from the old one is not
checked against the new one. A comment in sched_dl_overflow() names
the gap: the old bandwidth should stay counted until the 0-lag time.

So a running task can shrink its reservation, and still run on the old
one. Other tasks can meanwhile be admitted into the bandwidth it gave
back. On v7.4-rc1, a task of 50ms every 100ms changed to 10ms every
100ms. It ran 39ms of the next 100ms.

Fix this by applying the wakeup rule after a change of parameters, as
the previous patch does for a task that enters the class.
__setparam_dl() marks a change of the runtime, the deadline or the
period. The next enqueue then calls update_dl_entity(). It starts a new
period, or reduces the runtime of a constrained task, if the runtime
left does not fit. The task gets no more than a wakeup would give it.
A larger reservation still takes effect at the next period.

The enqueue flags cannot tell a change of parameters: a cgroup move
also uses ENQUEUE_MOVE. Without the mark, a task that waited behind
others could fail the check, and get a later deadline for no reason.

The change_running_new_params case of deadline_bw_test, added later in
this series, tests this. It fails on v7.4-rc1 and passes with this
patch.

Fixes: 40767b0dc768 ("sched/deadline: Fix deadline parameter modification handling")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h   |  5 +++++
 kernel/sched/deadline.c | 20 +++++++++++++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2f2f59402f09..f40abe5b9ace 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -715,6 +715,10 @@ struct sched_dl_entity {
 	 * @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
 	 * @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
 	 * runqueue's lock. Not for servers, which keep @dl_bw_attached.
+	 *
+	 * @dl_new_params tells if sched_setattr() changed the runtime, the
+	 * deadline or the period. The next enqueue checks the runtime left
+	 * against them, and clears it.
 	 */
 	unsigned int			dl_throttled      : 1;
 	unsigned int			dl_yielded        : 1;
@@ -729,6 +733,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer_idle     : 1;
 	unsigned int			dl_bw_attached    : 1;
 	unsigned int			dl_bw_oncpu       : 1;
+	unsigned int			dl_new_params     : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b47c13c8714d..3b62ee43c209 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2674,7 +2674,10 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
 static void
 enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
 {
+	bool new_params = dl_se->dl_new_params;
+
 	WARN_ON_ONCE(on_dl_rq(dl_se));
+	dl_se->dl_new_params = 0;
 
 	update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
 
@@ -2723,12 +2726,13 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
 		update_dl_entity(dl_se);
 	} else if (flags & ENQUEUE_REPLENISH) {
 		replenish_dl_entity(dl_se);
-	} else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+	} else if (((flags & ENQUEUE_CLASS) || new_params) &&
+		   !is_dl_boosted(dl_se)) {
 		/*
-		 * The task enters the class runnable. It may come back before
-		 * its deadline, with the runtime left from older parameters.
-		 * Check that runtime against the new parameters, as a wakeup
-		 * does.
+		 * The task enters the class, or its parameters change, while
+		 * it is runnable. It may have runtime left from older
+		 * parameters. Check that runtime against the new parameters,
+		 * as a wakeup does.
 		 */
 		update_dl_entity(dl_se);
 	} else if ((flags & ENQUEUE_MOVE) &&
@@ -4696,6 +4700,11 @@ void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 	u64 period = attr->sched_period ?: attr->sched_deadline;
 	bool held;
 
+	if (attr->sched_runtime != dl_se->dl_runtime ||
+	    attr->sched_deadline != dl_se->dl_deadline ||
+	    period != dl_se->dl_period)
+		dl_se->dl_new_params = 1;
+
 	/*
 	 * Change the bandwidth that the runqueue counts here, with the
 	 * parameters. A task that priority inheritance boosted into the
@@ -4828,6 +4837,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
 	dl_se->dl_bw_oncpu		= 0;
+	dl_se->dl_new_params		= 0;
 
 #ifdef CONFIG_RT_MUTEXES
 	dl_se->pi_se			= dl_se;
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 34/37] sched/debug: Show the deadline bandwidth of each runqueue
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (32 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 35/37] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
                   ` (2 subsequent siblings)
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel

For each CPU, sched_debug prints the deadline bandwidth of its root
domain. It does not print the two sums of its runqueue. this_bw is the
bandwidth of the deadline tasks assigned to the CPU. running_bw is the
part of it that belongs to active tasks.

GRUB reclaiming depends on both sums. grub_reclaim() takes this_bw -
running_bw as the inactive bandwidth of the CPU. The more inactive
bandwidth, the slower a reclaiming task uses up its runtime.

The kernel warns when a sum wraps around, and when running_bw exceeds
this_bw. Bandwidth left behind in this_bw triggers no warning.

Print both sums. The deadline_bw_test selftest added later in this
series reads this_bw in its params_waking and random_mix cases. It
checks that this_bw is back to its first value once their tasks are
gone.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/debug.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e0d6cd3bd607..3dbe1d7d2d77 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1174,6 +1174,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 	SEQ_printf(m, "  .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
 
 	PU(dl_nr_running);
+	PU(running_bw);
+	PU(this_bw);
 	dl_bw = &cpu_rq(cpu)->rd->dl_bw;
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 35/37] selftests/sched: Add a test of SCHED_DEADLINE policy round trips
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (33 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 34/37] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 36/37] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel

No selftest runs a SCHED_DEADLINE task.

Add deadline_switch_test, with four cases:
- throttled_round_trip: a task of 10ms every 2s spins, so it is
  throttled in every period. Right after a throttle, the test switches
  it to SCHED_OTHER. The test switches it back 200us after its
  deadline, between its replenishment timer and its 0-lag time. It must
  then run again.
- asleep_round_trip: a task of 9ms every 10ms runs 0.2ms, then sleeps
  50ms. While it sleeps past its 0-lag time, the test switches it to
  SCHED_OTHER and back. A leak at each round trip fills the root
  domain, and the switch back fails with EBUSY.
- sysrq_normalize: sysrq-n takes a deadline task out of the class.
  Another task must then still enter SCHED_DEADLINE.
- sysrq_boosted_owner: sysrq-n takes a SCHED_OTHER task out of the
  deadline class while a deadline waiter boosts it. After the unlock,
  the task enters SCHED_DEADLINE at 10ms every 100ms. It must not run
  more than its runtime in its first period.

The cases need CAP_SYS_NICE, and skip without it. asleep_round_trip
also skips without admission control, when sched_rt_runtime_us is -1.
sysrq-n changes the policy of every real-time user task. So the two
sysrq cases run only with DEADLINE_TEST_SYSRQ set. They also need root
and CONFIG_MAGIC_SYSRQ. sysrq_boosted_owner runs only if
sysrq_normalize passed.

On a kernel with the bugs, asleep_round_trip and sysrq_normalize keep
new tasks out of SCHED_DEADLINE. So they run last, followed only by
sysrq_boosted_owner.

On v7.4-rc1, in 8-CPU VMs, three of the four cases fail:
- throttled_round_trip. "sched/deadline: Do not lose a throttled task
  switched out and back" fixes it.
- asleep_round_trip. "sched/deadline: Release the bandwidth of a task
  that leaves the class asleep" fixes it.
- sysrq_normalize. "sched: Take cpuset_mutex for a deadline policy
  change only with pi" fixes it.

sysrq_boosted_owner tests "sched: Let a task boosted by a deadline
waiter change its policy". Without the sysrq-n part of that fix, the
owner ran 99-100ms of its first period, in 4 runs of 4. With it, it ran
10-11ms.

With the whole series, the four cases pass, with and without
PREEMPT_RT.

The test goes in a new selftest target, sched/deadline.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/Makefile              |   1 +
 .../selftests/sched/deadline/.gitignore       |   1 +
 .../testing/selftests/sched/deadline/Makefile |   8 +
 tools/testing/selftests/sched/deadline/config |   1 +
 .../sched/deadline/deadline_switch_test.c     | 808 ++++++++++++++++++
 5 files changed, 819 insertions(+)
 create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
 create mode 100644 tools/testing/selftests/sched/deadline/Makefile
 create mode 100644 tools/testing/selftests/sched/deadline/config
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 07c3719f8b04..35b0c5e86ba7 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -120,6 +120,7 @@ TARGETS += rlimits
 TARGETS += rseq
 TARGETS += rtc
 TARGETS += rust
+TARGETS += sched/deadline
 TARGETS += sched_ext
 TARGETS += seccomp
 TARGETS += sgx
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
new file mode 100644
index 000000000000..9f203c455b91
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -0,0 +1 @@
+deadline_switch_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
new file mode 100644
index 000000000000..3f4dc1ec6119
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
+LDLIBS += -lpthread
+
+TEST_GEN_PROGS := deadline_switch_test
+
+include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
new file mode 100644
index 000000000000..3a3a7285a708
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/config
@@ -0,0 +1 @@
+CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_switch_test.c b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
new file mode 100644
index 000000000000..f20839433609
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
@@ -0,0 +1,808 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test SCHED_DEADLINE tasks that switch to SCHED_OTHER and back:
+ *
+ * - throttled_round_trip: the test switches a throttled task to
+ *   SCHED_OTHER. The task's replenishment timer fires while it is out of
+ *   SCHED_DEADLINE. The test switches it back after its deadline, before
+ *   its 0-lag time. The task must run again.
+ * - asleep_round_trip: a task is switched out and back while it sleeps
+ *   past its 0-lag time. Its bandwidth must leave the root domain when it
+ *   leaves SCHED_DEADLINE. Otherwise each switch back adds it again, and
+ *   sched_setattr() soon fails with EBUSY.
+ * - sysrq_normalize: sysrq-n takes a deadline task out of the class.
+ *   Another task must then still enter SCHED_DEADLINE.
+ * - sysrq_boosted_owner: a deadline waiter boosts a sleeping SCHED_OTHER
+ *   owner into the deadline class. sysrq-n must end the boost. When the
+ *   owner later calls sched_setattr() to enter SCHED_DEADLINE, it must be
+ *   throttled at its own runtime.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+/* Written by the child, read by the parent. */
+struct shared {
+	uint64_t stamp;		/* last time the child ran */
+	uint64_t run_start;	/* when it last resumed after a throttle */
+	uint64_t runs;		/* resumptions after a throttle */
+	uint64_t sleeps;	/* sleeps begun */
+};
+
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void sleep_until(uint64_t t)
+{
+	struct timespec ts = { .tv_sec = t / NSEC_PER_SEC, .tv_nsec = t % NSEC_PER_SEC };
+
+	while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL) == EINTR)
+		;
+}
+
+static int set_deadline(pid_t pid, uint64_t runtime, uint64_t period)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+static struct shared *sh;
+
+/*
+ * Fork a child that is killed when the test dies, so that it cannot keep
+ * its bandwidth. Return what fork() returns.
+ */
+static pid_t fork_child(void)
+{
+	pid_t parent = getpid();
+	pid_t pid = fork();
+
+	if (!pid && (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent))
+		_exit(KSFT_FAIL);
+	return pid;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	/*
+	 * A deadline task left throttled for good never runs, even to die.
+	 * Switched to SCHED_OTHER, it runs and dies. Ignore the result: the
+	 * task may be dead already.
+	 */
+	set_other(pid);
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+/*
+ * Report a child that died before the end of its case. Return KSFT_SKIP
+ * if it could not enter SCHED_DEADLINE, and KSFT_FAIL otherwise.
+ */
+static int gone(int status)
+{
+	if (WIFEXITED(status) && WEXITSTATUS(status) == KSFT_SKIP) {
+		ksft_print_msg("SCHED_DEADLINE refused\n");
+		return KSFT_SKIP;
+	}
+	if (WIFSIGNALED(status))
+		ksft_print_msg("the task died of signal %d\n", WTERMSIG(status));
+	else
+		ksft_print_msg("the task exited with %d\n", WEXITSTATUS(status));
+	return KSFT_FAIL;
+}
+
+/*
+ * Spin, so that the task uses up its runtime and is throttled in every
+ * period. Record in sh when it runs, and when it resumes after a
+ * throttle. A throttle stops the task for most of its 2s period. A gap
+ * of 1s or less, such as a preemption, is not a throttle.
+ */
+static void spinner(void)
+{
+	uint64_t last = now_ns();
+
+	for (;;) {
+		uint64_t t = now_ns();
+
+		if (t - last > NSEC_PER_SEC) {
+			__atomic_store_n(&sh->run_start, t, __ATOMIC_RELAXED);
+			__atomic_fetch_add(&sh->runs, 1, __ATOMIC_RELEASE);
+		}
+		__atomic_store_n(&sh->stamp, t, __ATOMIC_RELEASE);
+		last = t;
+	}
+}
+
+/*
+ * Return the runtime @pid has left, from dl.runtime in /proc/<pid>/sched.
+ * It is below zero for a throttled task: the task overran by that much.
+ * Return 1 if it cannot be read.
+ */
+static long long dl_runtime_left(pid_t pid)
+{
+	char path[64], line[256];
+	long long left = 1;
+	FILE *f;
+
+	snprintf(path, sizeof(path), "/proc/%d/sched", pid);
+	f = fopen(path, "r");
+	if (!f)
+		return 1;
+	while (fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, "dl.runtime", 10) && val) {
+			left = atoll(val + 1);
+			break;
+		}
+	}
+	fclose(f);
+	return left;
+}
+
+#define LOST_RUNTIME	(10 * NSEC_PER_MSEC)
+#define LOST_PERIOD	(2 * NSEC_PER_SEC)
+#define LOST_MARGIN	(200 * NSEC_PER_USEC)
+
+/*
+ * The child is a deadline task of 10ms every 2s, and spins. Three times,
+ * the parent does this:
+ * - Wait until the child resumes at a replenishment, at time r, and is
+ *   throttled again.
+ * - Switch it to SCHED_OTHER.
+ * - Sleep until r + LOST_PERIOD + LOST_MARGIN. r + LOST_PERIOD is the
+ *   child's deadline. Its replenishment timer fires there, on a task out
+ *   of SCHED_DEADLINE.
+ * - Switch it back to SCHED_DEADLINE.
+ * - Check that it runs again within 1.5 periods.
+ *
+ * The switch back must come before the 0-lag time. The 0-lag time is after
+ * the deadline by the overrun times period/runtime, which is 200 here.
+ * The overrun is how far the task ran past its runtime before the kernel
+ * throttled it. The parent reads it from /proc. A round whose switch back
+ * comes after the 0-lag time tests nothing, and does not count. The check
+ * keeps half of LOST_MARGIN as a margin. The case is skipped if no round
+ * counts. If the overrun cannot be read, every round counts.
+ */
+static int test_throttled_round_trip(void)
+{
+	int attempt, ret = KSFT_PASS, err, counted = 0;
+	pid_t pid;
+
+	memset(sh, 0, sizeof(*sh));
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+			_exit(KSFT_SKIP);
+		spinner();
+	}
+	for (attempt = 0; attempt < 3 && ret == KSFT_PASS; attempt++) {
+		uint64_t runs, r, s, t;
+		long long left;
+
+		/*
+		 * Wait until the task resumes at a replenishment, at time r.
+		 * Then wait until it stops running: it is throttled.
+		 */
+		runs = __atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE) == runs) {
+			if (now_ns() - t > 3 * LOST_PERIOD) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not resume within 3 periods\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		r = __atomic_load_n(&sh->run_start, __ATOMIC_RELAXED);
+		do {
+			s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+			sleep_ns(2 * NSEC_PER_MSEC);
+		} while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) != s);
+		/*
+		 * The task is throttled until r + LOST_PERIOD. Switch it out
+		 * now, and back LOST_MARGIN after that.
+		 */
+		left = dl_runtime_left(pid);
+		err = set_other(pid);
+		if (!err) {
+			sleep_until(r + LOST_PERIOD + LOST_MARGIN);
+			err = set_deadline(pid, LOST_RUNTIME, LOST_PERIOD);
+			t = now_ns();
+		}
+		if (err) {
+			ksft_print_msg("policy switch: %s\n", strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+		/* The deadline is at most r + LOST_PERIOD: r is read after it. */
+		if (left > 0 ||
+		    t + LOST_MARGIN / 2 <
+		    r + LOST_PERIOD + (uint64_t)-left * (LOST_PERIOD / LOST_RUNTIME))
+			counted++;
+		else
+			ksft_print_msg("an overrun of %lld ns: switched back after the 0-lag time\n",
+				       -left);
+		sleep_ns(10 * NSEC_PER_MSEC);
+		s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s &&
+		       now_ns() - t < LOST_PERIOD + LOST_PERIOD / 2)
+			sleep_ns(NSEC_PER_MSEC);
+		if (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s) {
+			ksft_print_msg("throttled, switched out and back: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	if (ret == KSFT_PASS && !counted)
+		ret = KSFT_SKIP;
+	return ret;
+}
+
+#define LEAK_RUNTIME	(9 * NSEC_PER_MSEC)
+#define LEAK_PERIOD	(10 * NSEC_PER_MSEC)
+
+/* Run 0.2ms, then sleep 50ms, forever. Count the sleeps in sh. */
+static void sleeper(void)
+{
+	for (;;) {
+		uint64_t end = now_ns() + 200 * NSEC_PER_USEC;
+
+		while (now_ns() < end)
+			;
+		__atomic_fetch_add(&sh->sleeps, 1, __ATOMIC_RELEASE);
+		sleep_ns(50 * NSEC_PER_MSEC);
+	}
+}
+
+/*
+ * Switch a sleeping task out of SCHED_DEADLINE and back, over and over.
+ * The task has 90% of a CPU. If each round trip leaks it, the root domain
+ * is full after about as many round trips as it has CPUs.
+ */
+static int test_asleep_round_trip(void)
+{
+	int i, trips = get_nprocs() * 10 / 9 + 3, ret = KSFT_PASS, err;
+	pid_t pid;
+	FILE *f;
+
+	f = fopen("/proc/sys/kernel/sched_rt_runtime_us", "r");
+	if (f) {
+		long rt = 0;
+
+		if (fscanf(f, "%ld", &rt) == 1 && rt < 0) {
+			fclose(f);
+			ksft_print_msg("no admission control: a leak cannot be seen\n");
+			return KSFT_SKIP;
+		}
+		fclose(f);
+	}
+	memset(sh, 0, sizeof(*sh));
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LEAK_RUNTIME, LEAK_PERIOD))
+			_exit(KSFT_SKIP);
+		sleeper();
+	}
+	for (i = 0; i < trips; i++) {
+		uint64_t sleeps = __atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE);
+		uint64_t t = now_ns();
+
+		while (__atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE) == sleeps) {
+			if (now_ns() - t > NSEC_PER_SEC) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not run within 1s\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		/*
+		 * Wait 5ms. The task is still asleep, and past its 0-lag
+		 * time.
+		 */
+		sleep_ns(5 * NSEC_PER_MSEC);
+		err = set_other(pid);
+		if (!err)
+			err = set_deadline(pid, LEAK_RUNTIME, LEAK_PERIOD);
+		if (err) {
+			ksft_print_msg("switch back refused after %d of %d round trips: %s\n",
+				       i, trips, strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+/*
+ * Take a deadline task out of the class with sysrq-n. Then check that
+ * another task can still enter SCHED_DEADLINE.
+ *
+ * sysrq-n calls __sched_setscheduler() on every real-time and deadline
+ * user task, under tasklist_lock, to make it SCHED_OTHER. On a kernel
+ * with the bug, that call takes cpuset_mutex for a deadline task and
+ * never releases it. No task can enter SCHED_DEADLINE after that.
+ *
+ * sysrq-n changes the whole system. So the sysrq cases run last, and only
+ * with DEADLINE_TEST_SYSRQ set.
+ */
+static int test_sysrq_normalize(void)
+{
+	int fd, status, ret = KSFT_PASS;
+	pid_t pid, prober, w;
+	uint64_t t;
+
+	if (!getenv("DEADLINE_TEST_SYSRQ")) {
+		ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+		return KSFT_SKIP;
+	}
+	fd = open("/proc/sysrq-trigger", O_WRONLY);
+	if (fd < 0) {
+		ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		close(fd);
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+			_exit(KSFT_SKIP);
+		for (;;)
+			pause();
+	}
+	t = now_ns();
+	while (sched_getscheduler(pid) != SCHED_DEADLINE) {
+		if (waitpid(pid, &status, WNOHANG) == pid) {
+			close(fd);
+			return gone(status);
+		}
+		if (now_ns() - t > NSEC_PER_SEC) {
+			ksft_print_msg("the task did not enter SCHED_DEADLINE within 1s\n");
+			close(fd);
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	if (write(fd, "n", 1) != 1) {
+		ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+		close(fd);
+		reap(pid);
+		return KSFT_SKIP;
+	}
+	close(fd);
+	t = now_ns();
+	while (sched_getscheduler(pid) != SCHED_OTHER && now_ns() - t < NSEC_PER_SEC)
+		sleep_ns(NSEC_PER_MSEC);
+	if (sched_getscheduler(pid) != SCHED_OTHER) {
+		ksft_print_msg("still SCHED_DEADLINE after sysrq-n\n");
+		ret = KSFT_FAIL;
+	}
+	/* A new task must still enter SCHED_DEADLINE within 5s. */
+	prober = fork_child();
+	if (prober < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (!prober)
+		_exit(set_deadline(0, LOST_RUNTIME, LOST_PERIOD) ? 1 : 0);
+	t = now_ns();
+	while ((w = waitpid(prober, &status, WNOHANG)) == 0) {
+		if (now_ns() - t > 5 * NSEC_PER_SEC) {
+			ksft_print_msg("SCHED_DEADLINE admission blocked after sysrq-n\n");
+			reap(pid);
+			/*
+			 * The prober is blocked on a mutex. Do not wait for
+			 * it.
+			 */
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	if (w != prober) {
+		ksft_print_msg("waitpid %d: %s\n", prober, strerror(errno));
+		ret = KSFT_FAIL;
+	} else if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+		ksft_print_msg("SCHED_DEADLINE refused after sysrq-n\n");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	return ret;
+}
+
+/* The priority /proc shows for a deadline task, or a task boosted by one. */
+#define DL_TASK_PRIO	(-101)
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_locked, owner_go;
+static pid_t owner_tid;
+static int owner_ms;		/* the owner's CPU time in its first period */
+
+static uint64_t thread_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/*
+ * Wait on @s for at most @ms, retrying on signals. Return 0, or -1 on a
+ * timeout or an error.
+ */
+static int sem_wait_for(sem_t *s, int ms)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+	ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	while (sem_timedwait(s, &ts)) {
+		if (errno != EINTR)
+			return -1;
+	}
+	return 0;
+}
+
+/* Return the priority @tid runs at, from /proc, or 0. */
+static int prio_of(pid_t tid)
+{
+	char path[64], buf[1024], *p;
+	ssize_t n;
+	int fd, i;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", tid);
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return 0;
+	n = read(fd, buf, sizeof(buf) - 1);
+	close(fd);
+	if (n <= 0)
+		return 0;
+	buf[n] = 0;
+	/* The priority is field 18, the 16th after the command name. */
+	p = strrchr(buf, ')');
+	for (i = 0; p && i < 16; i++)
+		p = strchr(p + 1, ' ');
+	return p ? atoi(p + 1) : 0;
+}
+
+/* Join @t. Return -1 if the join fails or @t returned an error. */
+static int join(pthread_t t)
+{
+	void *res;
+	int err = pthread_join(t, &res);
+
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+	return err || res ? -1 : 0;
+}
+
+/*
+ * Run as the owner thread. Take pi_lock, post owner_locked, and sleep on
+ * owner_go while holding the lock. A deadline waiter boosts the owner
+ * during that sleep. Then drop the lock, become a deadline task of 10ms
+ * every 100ms, and spin for 100ms. Store the CPU time used, in ms, in
+ * owner_ms. Return NULL, or 1 on an error.
+ */
+static void *sleeping_owner(void *arg)
+{
+	uint64_t t0, cpu0;
+	int err;
+
+	owner_tid = syscall(SYS_gettid);
+	err = pthread_mutex_lock(&pi_lock);
+	if (err) {
+		ksft_print_msg("the owner cannot take the lock: %s\n", strerror(err));
+		return (void *)1;
+	}
+	sem_post(&owner_locked);
+	while (sem_wait(&owner_go) && errno == EINTR)
+		;
+	err = pthread_mutex_unlock(&pi_lock);
+	if (err) {
+		ksft_print_msg("the owner cannot unlock: %s\n", strerror(err));
+		return (void *)1;
+	}
+	err = set_deadline(0, 10 * NSEC_PER_MSEC, 100 * NSEC_PER_MSEC);
+	if (err) {
+		ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	t0 = now_ns();
+	cpu0 = thread_time_ns();
+	while (now_ns() - t0 < 100 * NSEC_PER_MSEC)
+		;
+	owner_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+	set_other(0);
+	return NULL;
+}
+
+/*
+ * The waiter thread: become a deadline task of 5ms every 50ms, and wait
+ * up to 300ms for pi_lock. The owner holds the lock all along. Return
+ * NULL if the wait timed out, or 1.
+ */
+static void *boosting_waiter(void *arg)
+{
+	struct timespec ts;
+	int err;
+
+	err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+	if (err) {
+		ksft_print_msg("the waiter refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += 300 * 1000000L;
+	ts.tv_sec += ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	err = pthread_mutex_timedlock(&pi_lock, &ts);
+	if (!err)
+		pthread_mutex_unlock(&pi_lock);
+	set_other(0);			/* sysrq-n may have done it */
+	if (err != ETIMEDOUT) {
+		ksft_print_msg("the waiter of the lock: %s, not a timeout\n",
+			       err ? strerror(err) : "got it");
+		return (void *)1;
+	}
+	return NULL;
+}
+
+/*
+ * Check that sysrq-n ends the boost of a SCHED_OTHER lock owner.
+ *
+ * M is the main thread, O the owner, W the waiter. All three start as
+ * SCHED_OTHER. pi_lock is a priority inheritance mutex.
+ *
+ *   M                           O                          W
+ *   start O
+ *                               lock pi_lock
+ *                               post owner_locked
+ *                               sleep on owner_go
+ *   start W
+ *                                                          sched_setattr():
+ *                                                            deadline,
+ *                                                            5ms every 50ms
+ *                                                          timedlock pi_lock,
+ *                                                            300ms: boosts O
+ *   see O at deadline priority
+ *   write "n" to sysrq-trigger:
+ *     O and W leave the class
+ *                                                          timedlock times out:
+ *                                                            deboosts O
+ *   join W
+ *   post owner_go
+ *                               unlock pi_lock
+ *                               sched_setattr(): deadline,
+ *                                 10ms every 100ms
+ *                               spin 100ms, measure
+ *                                 CPU time
+ *   join O
+ *
+ * sysrq-n takes O out of the deadline class without a chain walk. The
+ * boost must end there. On a kernel with the bug it does not, and the
+ * deboost at W's timeout sees no change of priority, so it leaves the
+ * boost too. O would then enter SCHED_DEADLINE as a boosted task, and
+ * never be throttled. O must get about 10ms of CPU time in its 100ms. The
+ * case fails above 20ms.
+ *
+ * O sleeps while it is boosted, on purpose. A boosted task runs on its
+ * waiter's parameters. sysrq-n takes W out of the class too, and W's
+ * parameters can then be cleared. On a kernel that refills the runtime
+ * from cleared parameters, a running O would loop in
+ * replenish_dl_entity().
+ *
+ * Run the case only if sysrq_normalize passed. @normalized is its result.
+ * Otherwise sysrq-n may leave cpuset_mutex held, and O would block in
+ * sched_setattr().
+ */
+static int test_sysrq_boosted_owner(int normalized)
+{
+	int fd, err, ret = KSFT_PASS;
+	pthread_mutexattr_t a;
+	pthread_t o, w;
+	uint64_t t;
+
+	if (!getenv("DEADLINE_TEST_SYSRQ")) {
+		ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+		return KSFT_SKIP;
+	}
+	if (normalized != KSFT_PASS) {
+		ksft_print_msg("sysrq_normalize did not pass: sched_setattr() may block\n");
+		return KSFT_SKIP;
+	}
+	fd = open("/proc/sysrq-trigger", O_WRONLY);
+	if (fd < 0) {
+		ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	err = pthread_mutexattr_init(&a);
+	if (!err) {
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+		if (!err)
+			err = pthread_mutex_init(&pi_lock, &a);
+		pthread_mutexattr_destroy(&a);
+	}
+	if (err) {
+		ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+		close(fd);
+		return KSFT_SKIP;
+	}
+	if (sem_init(&owner_locked, 0, 0) || sem_init(&owner_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	owner_ms = -1;
+	if (pthread_create(&o, NULL, sleeping_owner, NULL)) {
+		ksft_print_msg("the owner thread could not start\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (sem_wait_for(&owner_locked, 5000)) {
+		ksft_print_msg("the owner never took the lock\n");
+		sem_post(&owner_go);	/* in case it gets there late */
+		join(o);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		ret = KSFT_FAIL;
+		goto unlock;
+	}
+	t = now_ns();
+	while (prio_of(owner_tid) != DL_TASK_PRIO && now_ns() - t < 200 * NSEC_PER_MSEC)
+		sleep_ns(NSEC_PER_MSEC);
+	if (prio_of(owner_tid) != DL_TASK_PRIO) {
+		ksft_print_msg("the waiter never boosted the owner\n");
+		ret = KSFT_FAIL;
+	} else if (write(fd, "n", 1) != 1) {
+		ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+		ret = KSFT_SKIP;
+	}
+	if (join(w))
+		ret = KSFT_FAIL;	/* the waiter said why */
+unlock:
+	sem_post(&owner_go);
+	if (join(o)) {
+		ret = KSFT_FAIL;	/* the owner said why */
+	} else if (ret == KSFT_PASS && owner_ms > 20) {
+		ksft_print_msg("the owner ran %d ms of its first 100: the boost outlived sysrq-n\n",
+			       owner_ms);
+		ret = KSFT_FAIL;
+	}
+out:
+	close(fd);
+	pthread_mutex_destroy(&pi_lock);
+	return ret;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	int ret;
+
+	sh = mmap(NULL, sizeof(*sh), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (sh == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	ksft_print_header();
+	ksft_set_plan(4);
+	/*
+	 * Keep this order. On a kernel with the bugs, asleep_round_trip fills
+	 * the root domain, and sysrq_normalize blocks every later admission.
+	 * So sysrq_boosted_owner runs only if sysrq_normalize passed.
+	 */
+	report("throttled_round_trip", test_throttled_round_trip());
+	report("asleep_round_trip", test_asleep_round_trip());
+	ret = test_sysrq_normalize();
+	report("sysrq_normalize", ret);
+	report("sysrq_boosted_owner", test_sysrq_boosted_owner(ret));
+	ksft_finished();
+}
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 36/37] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (34 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 35/37] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:05 ` [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
  36 siblings, 0 replies; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel

A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth until
its 0-lag time. No selftest checks what happens to this bandwidth
meanwhile.

Add deadline_rebuild_test, with three cases. In each case, a task of
400ms every second runs 380ms, then leaves SCHED_DEADLINE. Its 0-lag
time is then about 570ms ahead. P is a partition root of one CPU.
Before the 0-lag time:
- dies_before_rebuild: the task exits in P. Then a second partition
  root is made, which rebuilds the accounting of every root domain.
- switched_out_before_rebuild: the same, but the task switches to
  SCHED_OTHER and sleeps in P.
- moved_after_switch_out: the task switches to SCHED_OTHER outside P.
  Then it moves into P and dies there.

After the 0-lag time, P must admit one task of 600ms every second, and
refuse a second one with EBUSY. A kernel with the bug removes the
bandwidth from P's root domain at the 0-lag time. But the rebuild
dropped it already, or P never had it. total_bw then wraps around, and
P admits both tasks.

The test skips without root, 3 CPUs, the cgroup v2 cpuset controller or
deadline admission control. Admission control is off when
sched_rt_runtime_us is -1.

On v7.4-rc1, in 8-CPU VMs, the three cases fail. "sched/deadline: Keep
departing bandwidth across a root domain rebuild" fixes them. With the
whole series, the three cases pass, with and without PREEMPT_RT.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../selftests/sched/deadline/.gitignore       |   1 +
 .../testing/selftests/sched/deadline/Makefile |   2 +-
 tools/testing/selftests/sched/deadline/config |   2 +
 .../sched/deadline/deadline_rebuild_test.c    | 497 ++++++++++++++++++
 4 files changed, 501 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c

diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index 9f203c455b91..bf7dd0da81a9 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1 +1,2 @@
 deadline_switch_test
+deadline_rebuild_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index 3f4dc1ec6119..f3a1d1746d22 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
 CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
 LDLIBS += -lpthread
 
-TEST_GEN_PROGS := deadline_switch_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
 
 include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
index 3a3a7285a708..a711e8af46bf 100644
--- a/tools/testing/selftests/sched/deadline/config
+++ b/tools/testing/selftests/sched/deadline/config
@@ -1 +1,3 @@
+CONFIG_CGROUPS=y
+CONFIG_CPUSETS=y
 CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
new file mode 100644
index 000000000000..714a0ff03f1b
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the bandwidth of a SCHED_DEADLINE task that leaves the policy, or
+ * dies, before its 0-lag time. The bandwidth stays in the task's root
+ * domain until the inactive timer removes it. It must be removed once,
+ * from the root domain that holds it.
+ *
+ * P is a partition root of one CPU. In each case the task is a deadline
+ * task of 400ms every second, which first runs 380ms. The cases are:
+ * - dies_before_rebuild: the task is in P and exits. Before its 0-lag
+ *   time, the test makes a second partition root, Q. This rebuilds the
+ *   accounting of every root domain.
+ * - switched_out_before_rebuild: the same, but the task switches to
+ *   SCHED_OTHER and sleeps instead of exiting.
+ * - moved_after_switch_out: the task is outside P. It switches to
+ *   SCHED_OTHER and sleeps. Before its 0-lag time, the test moves it into
+ *   P and kills it there.
+ *
+ * Once the inactive timer has fired, P must admit one task of 600ms
+ * every second, and refuse a second one with EBUSY. A kernel with the
+ * bug removes the bandwidth from P's root domain after the rebuild
+ * dropped it, or when P never had it. total_bw of that root domain then
+ * wraps around, and P admits both tasks.
+ *
+ * The test needs root, 3 CPUs, the cgroup v2 cpuset controller and
+ * deadline admission control.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+static int set_deadline(pid_t pid, uint64_t runtime)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = NSEC_PER_SEC,
+		.sched_period = NSEC_PER_SEC,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+/*
+ * Return the CLOCK_MONOTONIC time in ns. clock_gettime() cannot fail: the
+ * clock is always supported, and the pointer is valid.
+ */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	/* Ignore the result: the task may be dead already. */
+	set_other(pid);
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* Make @cg a partition root of CPU @cpu alone. Return 0, or -1 on error. */
+static int make_partition(const char *cg, int cpu)
+{
+	char path[PATH_MAX], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	if (write_file(cg, "cpuset.cpus", buf) || write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+	uint64_t end;
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	end = now_ns() + NSEC_PER_SEC;
+	while (rmdir(cg)) {
+		if (errno == ENOENT)
+			return;
+		if (errno != EBUSY || now_ns() > end) {
+			ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+			return;
+		}
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+}
+
+/*
+ * Return the CPU time of the calling thread in ns. clock_gettime() cannot
+ * fail: the clock is always supported, and the pointer is valid.
+ */
+static uint64_t cpu_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+enum depart { DIES, SWITCHED_OUT };
+
+/*
+ * Fork a child that enters @cg, if given, then SCHED_DEADLINE at 400ms
+ * every second. The child runs 380ms of its runtime. Then it dies if @how
+ * is DIES. Otherwise it switches to SCHED_OTHER, writes a byte to @fd and
+ * sleeps. Its 0-lag time is then about 570ms ahead. Return the pid of the
+ * child, or -1. Only the read end of @fd stays open in the caller.
+ */
+static pid_t departing(const char *cg, enum depart how, int fd[2])
+{
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		uint64_t t0;
+
+		close(fd[0]);
+		if (cg && enter(cg, getpid()))
+			_exit(2);
+		if (set_deadline(0, 400 * NSEC_PER_MSEC))
+			_exit(3);
+		t0 = cpu_time_ns();
+		while (cpu_time_ns() - t0 < 380 * NSEC_PER_MSEC)
+			;
+		if (how == DIES)
+			_exit(0);
+		if (set_other(0) || write(fd[1], "x", 1) != 1)
+			_exit(4);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	fd[1] = -1;
+	return pid;
+}
+
+/*
+ * Wait until the child is out of SCHED_DEADLINE: dead or switched out.
+ * Return 0 on success. If the child failed, return its exit status, or
+ * -1. Set *@pid to -1 once the child is reaped.
+ */
+static int departed(pid_t *pid, enum depart how, int fd)
+{
+	int status;
+	char c;
+
+	if (*pid < 0)
+		return -1;
+	/*
+	 * A child that switches out sends a byte. Without the byte, the
+	 * child died or failed, and its exit status tells which.
+	 */
+	if (how == SWITCHED_OUT) {
+		ssize_t n;
+
+		do {
+			n = read(fd, &c, 1);
+		} while (n < 0 && errno == EINTR);
+		if (n == 1)
+			return 0;
+	}
+	if (waitpid(*pid, &status, 0) != *pid)
+		return -1;
+	*pid = -1;
+	return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
+}
+
+/*
+ * Fork a child that enters @cg and sleeps there. The child moves itself,
+ * so that it last ran on a CPU of @cg. Wait until it is there. Return its
+ * pid, or -1.
+ */
+static pid_t sleeper(const char *cg)
+{
+	int fd[2];
+	pid_t pid;
+	char c;
+
+	if (pipe(fd))
+		return -1;
+	pid = fork();
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		close(fd[0]);
+		if (enter(cg, getpid()) || write(fd[1], "x", 1) != 1)
+			_exit(1);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	if (pid > 0 && read(fd[0], &c, 1) != 1) {
+		reap(pid);
+		pid = -1;
+	}
+	close(fd[0]);
+	return pid;
+}
+
+/*
+ * Check that P admits one task of 600ms every second, but not two. The
+ * caller must wait until every inactive timer has fired.
+ */
+static int check_p(void)
+{
+	pid_t a = sleeper(cg_p), b = sleeper(cg_p);
+	int ret = KSFT_FAIL, e1, e2;
+
+	if (a < 0 || b < 0) {
+		ksft_print_msg("no task could enter P\n");
+		goto out;
+	}
+	e1 = set_deadline(a, 600 * NSEC_PER_MSEC);
+	e2 = set_deadline(b, 600 * NSEC_PER_MSEC);
+	if (e1)
+		ksft_print_msg("a first task refused in P: %s\n", strerror(-e1));
+	else if (!e2)
+		ksft_print_msg("1.2 CPUs of deadline tasks admitted in a partition of one CPU\n");
+	else if (e2 != -EBUSY)
+		ksft_print_msg("the second task refused with %s, not EBUSY\n", strerror(-e2));
+	else
+		ret = KSFT_PASS;
+out:
+	reap(a);
+	reap(b);
+	return ret;
+}
+
+/*
+ * Have a task of P die or switch out, as @how says. Make Q before its
+ * 0-lag time, then check P.
+ */
+static int test_rebuild(enum depart how)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(cg_p, how, fd);
+	err = departed(&pid, how, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (make_partition(cg_q, ncpus - 2)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 2);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	/*
+	 * Wait past the 0-lag time. In the SWITCHED_OUT case the task is still
+	 * alive then: it sleeps in P as a SCHED_OTHER task.
+	 */
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_q);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+/*
+ * Have a task outside P switch to SCHED_OTHER and sleep. Before its 0-lag
+ * time, move it into P and kill it. Then check P.
+ */
+static int test_move(void)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(NULL, SWITCHED_OUT, fd);
+	err = departed(&pid, SWITCHED_OUT, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (enter(cg_p, pid)) {
+		ksft_print_msg("the task cannot move into P\n");
+		goto out;
+	}
+	/* The task wakes up and dies on P's CPU, before its 0-lag time. */
+	reap(pid);
+	pid = -1;
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	char buf[256], path[PATH_MAX + 32];
+	int added;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	ncpus = get_nprocs();
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs\n");
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	if (find_cgroup2())
+		ksft_exit_skip("cgroup v2 is not mounted\n");
+	/*
+	 * Enable the cpuset controller for the root's children, if it is
+	 * off. It is turned off again at the end.
+	 */
+	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+	if (read_file(path, buf, sizeof(buf)))
+		ksft_exit_skip("no cgroup.subtree_control\n");
+	added = !strstr(buf, "cpuset");
+	if (added && write_file(root, "cgroup.subtree_control", "+cpuset"))
+		ksft_exit_skip("no cpuset controller\n");
+	if (snprintf(cg_p, sizeof(cg_p), "%s/deadline_rebuild_p", root) >= (int)sizeof(cg_p) ||
+	    snprintf(cg_q, sizeof(cg_q), "%s/deadline_rebuild_q", root) >= (int)sizeof(cg_q))
+		ksft_exit_skip("cgroup v2 mount path too long\n");
+
+	ksft_set_plan(3);
+	report("dies_before_rebuild", test_rebuild(DIES));
+	report("switched_out_before_rebuild", test_rebuild(SWITCHED_OUT));
+	report("moved_after_switch_out", test_move());
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+	ksft_finished();
+}
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting
  2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
                   ` (35 preceding siblings ...)
  2026-11-16 19:05 ` [PATCH v6 36/37] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-16 19:05 ` Mathieu Desnoyers
  2026-11-16 19:49   ` sashiko-bot
  36 siblings, 1 reply; 53+ messages in thread
From: Mathieu Desnoyers @ 2026-11-16 19:05 UTC (permalink / raw)
  To: juri.lelli, peterz, mingo, vincent.guittot
  Cc: linux-kernel, Mathieu Desnoyers, tglx, will, boqun,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, luca.abeni, mathieu.poirier, longman,
	ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
	parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu, cgroups,
	linux-kselftest, linux-rt-devel

Upstream has no selftest of the SCHED_DEADLINE bandwidth accounting. A
leak or a double release does not show at once. Later, the root domain
refuses deadline tasks, or admits more than it can run.

Add deadline_bw_test, with 89 cases. Most cases count the tasks of 10ms
every 100ms that a root domain admits, before and after they run. Once
the tasks of the case are gone and past their 0-lag time, the two counts
must be equal. The cases cover:
- priority inheritance, boost and deboost
- parameter changes, SCHED_FLAG_KEEP_PARAMS among them
- cpuset moves
- what can happen during a cpuset move: a rebuild of the sched domains,
  a policy change, an affinity change, a wakeup, the death of a thread,
  a refusal by another controller
- deaths, sched_yield() and runtime overruns
- a stopped task and a frozen task
- bandwidth reclaiming and fair server changes
- CPU hotplug, also with admission control turned off
- sched_setaffinity() racing with sched_setattr()
- parameter changes while a task wakes on another CPU (params_waking)
- a random mix of all of the above, with affinity changes (random_mix)

The comment above each case gives its scenario. params_waking and
random_mix also compare the this_bw of their CPUs, which "sched/debug:
Show the deadline bandwidth of each runqueue" prints. Four cases follow
reproducers of Juri Lelli's sched-deadline-tests: stopped_switched_out,
hotplug_sleeper_switched, hotplug_no_admission_pinned and
pi_chain_two_owners.

The test needs root, deadline admission control and 3 CPUs. Most cases
need the cgroup v2 cpuset controller, and some need debugfs, CPU hotplug
or 4 CPUs. Thirteen cases need CPUs isolated by isolcpus=domain.
move_cancelled needs CONFIG_RT_GROUP_SCHED. move_normalized fires
sysrq-n, which changes the whole system, so it needs
DEADLINE_TEST_SYSRQ=1. A case skips without what it needs.

The message of each fix in this series names the cases that test it.
These cases are not named there. Each fails up to the patch before its
fix, and passes from the fix:
- overrun_caught and stopped_switched_out: "sched/deadline: Release the
  bandwidth of a task that leaves the class asleep"
- hotplug_departed and hotplug_no_admission_pinned: "sched/deadline:
  Keep departing bandwidth across a root domain rebuild". Before it,
  the second one fails in part of the runs.
- move_asleep, move_non_contending, move_died, move_woken and
  move_woken_full: "sched/deadline: Record the root domain holding a
  task's bandwidth"
- affinity_vs_setattr: "sched/deadline: Check SCHED_DEADLINE affinity
  under the rq lock"
- params_keep_class_change: "sched: Refuse SCHED_FLAG_KEEP_PARAMS that
  would change the policy or the class"

On v7.4-rc1, in 8 VMs of 8 CPUs, 32 cases fail. 30 fail in every VM.
overrun_caught and hotplug_sleeper_switched each failed in 7 VMs of 8.
The 32 cases are:

  affinity_vs_setattr              move_affinity_became_deadline
  boost_blocked_keeps_order        move_affinity_changed
  boosted_owner_policy             move_asleep
  change_running_new_params        move_became_deadline
  death_zombie_hotplug             move_became_deadline_no_room
  death_zombie_switched            move_died
  deboost_after_donor_refill       move_non_contending
  deboost_in_class_resets_period   move_vs_rebuild
  deboost_released_resets_period   move_woken
  deboost_resets_period            move_woken_full
  hotplug_departed                 overrun_caught
  hotplug_no_admission_pinned      params_keep_boosted
  hotplug_sleeper_switched         params_keep_class_change
  pi_departed_moved                pi_former_owner_asleep
  random_mix                       return_running_new_params
  setattr_blocked_keeps_order      stopped_switched_out

18 cases skip there. 15 lack what they need. hotplug_refused and
hotplug_server cannot set up their state. move_normalized was not run:
on v7.4-rc1, sysrq-n leaves cpuset_mutex locked, which "sched: Take
cpuset_mutex for a deadline policy change only with pi" fixes.

With the whole series, every case passes wherever it runs, PREEMPT_RT
included. The thirteen isolcpus cases ran in guests booted with
isolcpus=domain,1,2 or isolcpus=domain,0,1. move_cancelled ran in two
guests built with CONFIG_RT_GROUP_SCHED. No kernel warning is logged.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://github.com/jlelli/sched-deadline-tests
---
 .../selftests/sched/deadline/.gitignore       |    1 +
 .../testing/selftests/sched/deadline/Makefile |    2 +-
 tools/testing/selftests/sched/deadline/config |    2 +
 .../sched/deadline/deadline_bw_test.c         | 8101 +++++++++++++++++
 .../testing/selftests/sched/deadline/settings |    1 +
 5 files changed, 8106 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/settings

diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index bf7dd0da81a9..42f6b2945e96 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1,2 +1,3 @@
 deadline_switch_test
 deadline_rebuild_test
+deadline_bw_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index f3a1d1746d22..15f1e96a21e6 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
 CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
 LDLIBS += -lpthread
 
-TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test deadline_bw_test
 
 include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
index a711e8af46bf..b19db0c962e0 100644
--- a/tools/testing/selftests/sched/deadline/config
+++ b/tools/testing/selftests/sched/deadline/config
@@ -1,3 +1,5 @@
 CONFIG_CGROUPS=y
 CONFIG_CPUSETS=y
+CONFIG_DEBUG_FS=y
+CONFIG_HOTPLUG_CPU=y
 CONFIG_MAGIC_SYSRQ=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
new file mode 100644
index 000000000000..19c61a3a9529
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -0,0 +1,8101 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the bandwidth accounting of SCHED_DEADLINE. The deadline tasks
+ * of the cases go through:
+ *  - priority inheritance
+ *  - parameter changes
+ *  - cpuset moves
+ *  - affinity changes
+ *  - deaths, yields and runtime overruns
+ *  - bandwidth reclaiming and fair server changes
+ *  - CPU hotplug
+ *  - a random mix of most of these
+ *
+ * Most cases measure the free bandwidth of a root domain before and after
+ * they run. probe() counts the tasks of 10ms every 100ms that the root
+ * domain admits. Once the tasks of the case are gone and their 0-lag time
+ * is past, the two counts must be equal. Fewer means leaked bandwidth.
+ * More means bandwidth released twice, which lets the root domain admit
+ * deadline tasks beyond its capacity. A deadline task that a case keeps
+ * running must also make progress.
+ *
+ * The reclaiming cases measure the share of a CPU that a deadline task
+ * with SCHED_FLAG_RECLAIM gets. That share follows extra_bw: the free
+ * bandwidth that the task's runqueue can reclaim. probe() follows the
+ * root domain's total_bw instead.
+ *
+ * The CPUs that isolcpus="domain" isolates are in the default root
+ * domain: the root domain of the isolated CPUs. The other CPUs are the
+ * housekeeping CPUs, in the housekeeping root domain. The test leaves the
+ * isolated CPUs out, and keeps its tasks off them. Only the cases that
+ * need them use them.
+ *
+ * The cases of p_cases[] run with P, a partition root of the last
+ * housekeeping CPU. P is made anew for each case. Making it rebuilds the
+ * accounting, which clears what an earlier case left.
+ *
+ * The cases of hp_cases[] run after them, without P. These are the
+ * hotplug cases, and four other cases: mask_out_refused,
+ * partition_takes_refused, no_admission_affinity_moves and
+ * affinity_vs_setattr. The test skips the hotplug cases if no CPU can go
+ * offline. It still runs these four.
+ * death_zombie_hotplug takes CPUs offline too, but it is in p_cases[]: it
+ * needs the cpuset setup of those cases.
+ *
+ * The server cases change the fair server of a CPU through debugfs, and
+ * the test then puts it back.
+ *
+ * The test needs root, deadline admission control and 3 housekeeping
+ * CPUs. Some cases need 4. Most also need the cgroup v2 cpuset
+ * controller. Some need debugfs or CPU hotplug. Thirteen need
+ * isolcpus="domain" at boot.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+#include <linux/futex.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+#ifndef SCHED_FLAG_RECLAIM
+#define SCHED_FLAG_RECLAIM	0x02
+#endif
+#ifndef SCHED_FLAG_DL_OVERRUN
+#define SCHED_FLAG_DL_OVERRUN	0x04
+#endif
+#ifndef SCHED_FLAG_KEEP_PARAMS
+#define SCHED_FLAG_KEEP_PARAMS	0x10
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+/* The threads of probe() and hold() do nothing: a small stack will do. */
+#define THREAD_STACK	(64 * 1024 > PTHREAD_STACK_MIN ? 64 * 1024 : PTHREAD_STACK_MIN)
+
+#define PROBE_RUNTIME	(10 * NSEC_PER_MSEC)
+#define PROBE_PERIOD	(100 * NSEC_PER_MSEC)
+#define PERIOD		(100 * NSEC_PER_MSEC)
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+/* The online CPUs at the start, ascending, except the isolated ones */
+static int ncpus, *online;
+static int nisol, *isol;	/* the online CPUs that isolcpus="domain" isolates */
+static char isolated[4096];	/* the CPU list of /sys/devices/system/cpu/isolated */
+
+/* Return the @n-th highest online CPU, counting from 1. */
+static int top_cpu(int n)
+{
+	return online[ncpus - n];
+}
+
+/*
+ * clock_gettime() cannot fail in now_ns() and thread_time_ns(). Both
+ * clocks are always supported, and the pointer is valid.
+ */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static uint64_t thread_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void spin_ns(uint64_t ns)
+{
+	uint64_t end = thread_time_ns() + ns;
+
+	while (thread_time_ns() < end)
+		;
+}
+
+/*
+ * Poll @cond every @step ns, for at most @ms. Return 0 once it holds, or
+ * -1 on a timeout. The bound makes a child that never started, or was
+ * refused SCHED_DEADLINE, fail its case instead of hanging it.
+ */
+#define wait_for_step(ms, step, cond)					\
+({									\
+	uint64_t wait_end = now_ns() + (uint64_t)(ms) * NSEC_PER_MSEC;	\
+	int wait_ret = -1;						\
+									\
+	while (1) {							\
+		if (cond) {						\
+			wait_ret = 0;					\
+			break;						\
+		}							\
+		if (now_ns() >= wait_end)				\
+			break;						\
+		sleep_ns(step);						\
+	}								\
+	wait_ret;							\
+})
+
+#define wait_for(ms, cond)	wait_for_step(ms, NSEC_PER_MSEC, cond)
+
+/* Wait on @s. Retry when a signal interrupts the wait. */
+static void sem_wait_sig(sem_t *s)
+{
+	while (sem_wait(s)) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("sem_wait: %s\n", strerror(errno));
+		return;
+	}
+}
+
+/*
+ * Wait on @s for at most @ms, retrying on signals. Return 0, or -1 on a
+ * timeout or an error.
+ */
+static int sem_wait_for(sem_t *s, int ms)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+	ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	while (sem_timedwait(s, &ts)) {
+		if (errno == EINTR)
+			continue;
+		if (errno != ETIMEDOUT)
+			ksft_print_msg("sem_timedwait: %s\n", strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * Fork a child that is killed when the thread that forked it dies, so
+ * that it cannot keep its bandwidth. Every caller forks from the main
+ * thread of its process. Print a message with @what if fork() fails.
+ * Return what fork() returns. The caller must handle -1.
+ */
+static pid_t fork_child(const char *what)
+{
+	pid_t parent = getpid();
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork for %s: %s\n", what, strerror(errno));
+	else if (!pid && (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent))
+		_exit(KSFT_FAIL);
+	return pid;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static void join(pthread_t t)
+{
+	int err = pthread_join(t, NULL);
+
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+}
+
+static pid_t gettid_(void)
+{
+	return syscall(SYS_gettid);
+}
+
+static int set_deadline_flags(pid_t tid, uint64_t runtime, uint64_t period,
+			      uint64_t flags)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = flags,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int set_deadline(pid_t tid, uint64_t runtime, uint64_t period)
+{
+	return set_deadline_flags(tid, runtime, period, 0);
+}
+
+static int set_other(pid_t tid)
+{
+	struct dl_attr attr = { .size = sizeof(attr), .sched_policy = SCHED_OTHER };
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* Make @cg a partition root of the CPUs in @cpus. Return 0, or -1. */
+static int make_partition_cpus(const char *cg, const char *cpus)
+{
+	char path[PATH_MAX + 32], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	if (write_file(cg, "cpuset.cpus", cpus) ||
+	    write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+/* Make @cg a partition root of CPU @cpu. Return 0, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+	char buf[32];
+
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	return make_partition_cpus(cg, buf);
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	if (wait_for(1000, !rmdir(cg) || errno == ENOENT))
+		ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+}
+
+/* Make P a partition root of the last online CPU. Return 0, or -1. */
+static int make_p(void)
+{
+	return make_partition(cg_p, top_cpu(1));
+}
+
+static void remove_p(void)
+{
+	remove_partition(cg_p);
+}
+
+/*
+ * Return field @n of /proc/@pid/stat, or @fail. Fields are numbered as
+ * in proc(5), and @n must be 3 or more: a field after the command name.
+ */
+static int stat_field(pid_t pid, int n, int fail)
+{
+	char path[64], buf[1024], *p;
+	int i;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+	if (read_file(path, buf, sizeof(buf)))
+		return fail;
+	p = strrchr(buf, ')');
+	for (i = 0; p && i < n - 2; i++)
+		p = strchr(p + 1, ' ');
+	return p ? atoi(p + 1) : fail;
+}
+
+/* Return the CPU @pid last ran on, from /proc, or -1. */
+static int task_cpu_of(pid_t pid)
+{
+	return stat_field(pid, 39, -1);
+}
+
+/* The priority /proc shows for a deadline task, or a task boosted by one. */
+#define DL_TASK_PRIO	(-101)
+
+/* Return the priority @pid runs at, from /proc, or 0. */
+static int prio_of(pid_t pid)
+{
+	return stat_field(pid, 18, 0);
+}
+
+/* Return the CPU time @t has used, or 0 if it cannot be read. */
+static uint64_t cpu_time_of(pthread_t t)
+{
+	static int told;
+	struct timespec ts;
+	clockid_t c;
+	int err;
+
+	err = pthread_getcpuclockid(t, &c);
+	if (!err && clock_gettime(c, &ts))
+		err = errno;
+	if (err) {
+		if (!told++)
+			ksft_print_msg("the CPU time of a thread cannot be read: %s\n",
+				       strerror(err));
+		return 0;
+	}
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/* Return the state letter of @pid in /proc, or 0. */
+static char proc_state_of(pid_t pid)
+{
+	char path[64], buf[1024], *p;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+	if (read_file(path, buf, sizeof(buf)))
+		return 0;
+	p = strrchr(buf, ')');
+	if (!p || p[1] != ' ')
+		return 0;
+	return p[2];
+}
+
+/*
+ * Wait a period. Then return the CPU share, in percent, that @t uses
+ * over the next @ns.
+ */
+static int share_of(pthread_t t, uint64_t ns)
+{
+	uint64_t cpu, wall;
+
+	sleep_ns(PERIOD);		/* a period of its own to settle in */
+	cpu = cpu_time_of(t);
+	wall = now_ns();
+	sleep_ns(ns);
+	cpu = cpu_time_of(t) - cpu;
+	wall = now_ns() - wall;
+	return wall ? (int)(100 * cpu / wall) : -1;
+}
+
+struct probe_slot {
+	pthread_t t;
+	int err;
+};
+
+static sem_t probe_done;
+static int probe_hold[2];
+
+static void *probe_thread(void *arg)
+{
+	struct probe_slot *s = arg;
+	char c;
+
+	s->err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+	if (sem_post(&probe_done))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	/* Sleep until the probe closes the write end. */
+	while (!s->err) {
+		ssize_t r = read(probe_hold[0], &c, 1);
+
+		if (r >= 0)
+			break;
+		if (errno != EINTR) {
+			s->err = -errno;
+			break;
+		}
+	}
+	return NULL;
+}
+
+/*
+ * Count the deadline tasks of PROBE_RUNTIME every PROBE_PERIOD that the
+ * root domain of @cg admits. A child process joins @cg, or stays in the
+ * caller's cgroup if @cg is NULL. Its threads become deadline tasks one
+ * at a time, until one is refused. Then they all exit. Return the number
+ * admitted, or -1 on an error.
+ *
+ * The threads barely run, so their bandwidth is released when they exit.
+ * probe() still waits three periods for their 0-lag times.
+ */
+static int probe(const char *cg)
+{
+	int fd[2], n = -1, max = 16 * ncpus + 16;
+	pid_t pid;
+
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		return -1;
+	}
+	pid = fork_child("the admission check");
+	if (!pid) {
+		struct probe_slot *s = calloc(max, sizeof(*s));
+		pthread_attr_t a;
+		int i, k = 0;
+
+		close(fd[0]);
+		if (!s || sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+		    (cg && enter(cg, getpid())))
+			_exit(1);
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			_exit(1);
+		for (i = 0; i < max; i++) {
+			if (pthread_create(&s[i].t, &a, probe_thread, &s[i])) {
+				k = -1;	/* an error, not a refusal */
+				break;
+			}
+			if (sem_wait_for(&probe_done, 10000)) {
+				k = -1;
+				i++;	/* join that thread too */
+				break;
+			}
+			if (s[i].err) {
+				if (s[i].err != -EBUSY)
+					k = -1;
+				join(s[i].t);
+				break;
+			}
+			if (k >= 0)
+				k++;
+		}
+		if (i == max)
+			k = -1;
+		close(probe_hold[1]);
+		while (--i >= 0)
+			join(s[i].t);
+		if (write(fd[1], &k, sizeof(k)) != sizeof(k))
+			_exit(1);
+		_exit(0);
+	}
+	close(fd[1]);
+	if (pid < 0 || read(fd[0], &n, sizeof(n)) != sizeof(n))
+		n = -1;
+	close(fd[0]);
+	reap(pid);
+	sleep_ns(3 * PROBE_PERIOD);
+	return n;
+}
+
+/*
+ * Run @fn in a child process in @cg. Every thread of the case is then in
+ * @cg, and dies with the child. Return KSFT_PASS, KSFT_FAIL or KSFT_SKIP.
+ */
+static int run_in(const char *cg, int (*fn)(void))
+{
+	pid_t pid = fork_child("a case");
+	int status;
+
+	if (!pid) {
+		int err = cg ? enter(cg, getpid()) : 0;
+
+		if (err) {
+			ksft_print_msg("the case cannot enter %s: %s\n", cg, strerror(-err));
+			_exit(KSFT_SKIP);
+		}
+		_exit(fn());
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!WIFEXITED(status)) {
+		ksft_print_msg("the case died of signal %d\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return WEXITSTATUS(status);
+}
+
+/*
+ * Run @fn in @cg with run_in(), and probe the root domain of @cg before
+ * and after. Return KSFT_SKIP if the first probe admits no task. Return
+ * KSFT_FAIL if a probe fails, or if @fn passed and the two probes differ.
+ * Otherwise return the result of @fn.
+ */
+static int measured(const char *name, const char *cg, int (*fn)(void))
+{
+	int before, after, ret;
+
+	before = probe(cg);
+	if (before <= 0) {
+		ksft_print_msg("%s: %s\n", name, before ? "the probe failed" : "no room to probe");
+		return before ? KSFT_FAIL : KSFT_SKIP;
+	}
+	ret = run_in(cg, fn);
+	sleep_ns(3 * PERIOD);
+	after = probe(cg);
+	if (after < 0) {
+		ksft_print_msg("%s: the probe failed after the case\n", name);
+		ret = KSFT_FAIL;
+	} else if (ret == KSFT_PASS && after != before) {
+		ksft_print_msg("%s: %d probe tasks admitted before, %d after: %s\n",
+			       name, before, after,
+			       after < before ? "bandwidth leaked" : "bandwidth released twice");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Priority inheritance */
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_ready, owner_go;
+static atomic_int owner_holds, owner_tid;
+
+/*
+ * Set up pi_lock, a priority inheritance mutex, and the owner's
+ * semaphores. Return 0, or -1 if they cannot be set up.
+ */
+static int pi_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err) {
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+		if (!err)
+			err = pthread_mutex_init(&pi_lock, &a);
+		pthread_mutexattr_destroy(&a);
+	}
+	if (err) {
+		ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	if (sem_init(&owner_ready, 0, 0) || sem_init(&owner_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	owner_holds = 0;
+	return 0;
+}
+
+/* Undo pi_init(). The owner and the waiters must be gone. */
+static void pi_fini(void)
+{
+	pthread_mutex_destroy(&pi_lock);
+	sem_destroy(&owner_ready);
+	sem_destroy(&owner_go);
+}
+
+/*
+ * Make the calling thread a deadline task, and wait up to @ms for
+ * pi_lock. Then switch back to SCHED_OTHER. Return 0 if the wait ends as
+ * @timeout expects: a timeout if it is set, the lock if not. Return -1
+ * otherwise.
+ */
+static int wait_lock_for(int ms, int timeout)
+{
+	struct timespec ts;
+	int err;
+
+	err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return -1;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)ms * 1000000L;
+	ts.tv_sec += ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	err = pthread_mutex_timedlock(&pi_lock, &ts);
+	if (!err) {
+		int e = pthread_mutex_unlock(&pi_lock);
+
+		if (e)
+			ksft_print_msg("a waiter cannot unlock: %s\n", strerror(e));
+	}
+	if (set_other(0)) {
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+		return -1;
+	}
+	if (err != (timeout ? ETIMEDOUT : 0)) {
+		ksft_print_msg("a waiter of the lock: %s, not %s\n", err ? strerror(err) : "got it",
+			       timeout ? "a timeout" : "the lock");
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * Take pi_lock as the owner, and return 0. On a failure, report it, set
+ * owner_holds to -1 and return -1.
+ */
+static int take_pi_lock(void)
+{
+	int err = pthread_mutex_lock(&pi_lock);
+
+	if (err) {
+		ksft_print_msg("an owner cannot take the lock: %s\n", strerror(err));
+		owner_holds = -1;
+	}
+	return err ? -1 : 0;
+}
+
+static void drop_pi_lock(void)
+{
+	int err = pthread_mutex_unlock(&pi_lock);
+
+	if (err)
+		ksft_print_msg("an owner cannot unlock: %s\n", strerror(err));
+}
+
+/*
+ * A chain of two priority inheritance mutexes, M1 and M2. The comment of
+ * pi_chain_two_owners() describes the threads D1, N1 and N2. chain_tid[]
+ * holds their tids, in that order. chain_step says how far the case is:
+ * - 1: N2 holds M2. N2 sets it.
+ * - 2: the main thread has checked the boost of N1. N1 may now block on
+ *   M2.
+ * - 3: the main thread is done. N2 may now unlock M2.
+ */
+static pthread_mutex_t chain_m1, chain_m2;
+static atomic_int chain_tid[3], chain_step, chain_err;
+
+/*
+ * Wait up to 2s for chain_step to reach @step. Return 0 once it does, or
+ * once a thread of the chain has set chain_err. Return -1 on a timeout.
+ */
+static int chain_wait(int step)
+{
+	return wait_for(2000, chain_step >= step || chain_err);
+}
+
+/* N2: take M2, and hold it until step 3. */
+static void *chain_n2(void *arg)
+{
+	chain_tid[2] = gettid_();
+	if (pthread_mutex_lock(&chain_m2)) {
+		chain_err = 1;
+		return arg;
+	}
+	chain_step = 1;
+	if (chain_wait(3))
+		chain_err = 1;
+	pthread_mutex_unlock(&chain_m2);
+	return arg;
+}
+
+/* N1: take M1. At step 2, block on M2, which N2 holds. */
+static void *chain_n1(void *arg)
+{
+	if (pthread_mutex_lock(&chain_m1)) {
+		chain_err = 1;
+		return arg;
+	}
+	chain_tid[1] = gettid_();	/* N1 holds M1 */
+	if (chain_wait(2) || pthread_mutex_lock(&chain_m2)) {
+		chain_err = 1;
+	} else {
+		pthread_mutex_unlock(&chain_m2);
+	}
+	pthread_mutex_unlock(&chain_m1);
+	return arg;
+}
+
+/* D1: become a deadline task and block on M1, which N1 holds. */
+static void *chain_d1(void *arg)
+{
+	struct timespec ts;
+
+	chain_tid[0] = gettid_();
+	if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+		chain_err = 1;
+		return arg;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_sec += 5;
+	if (pthread_mutex_timedlock(&chain_m1, &ts))
+		chain_err = 1;
+	else
+		pthread_mutex_unlock(&chain_m1);
+	if (set_other(0))
+		chain_err = 1;
+	return arg;
+}
+
+/*
+ * Boost two SCHED_OTHER tasks through a chain of two mutexes:
+ *
+ *   D1 (deadline) waits for M1, held by N1
+ *   N1 (boosted)  waits for M2, held by N2
+ *
+ * N1 is boosted when D1 blocks. N2 is boosted when N1 then blocks: the
+ * boost goes through N1, which has no deadline parameters of its own.
+ * Both must be in the deadline class then. Once N2 unlocks, the three
+ * must finish.
+ */
+static int pi_chain_two_owners(void)
+{
+	pthread_mutexattr_t a;
+	int ret = KSFT_PASS, i;
+	pthread_t t[3];
+
+	if (pthread_mutexattr_init(&a) ||
+	    pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT) ||
+	    pthread_mutex_init(&chain_m1, &a) || pthread_mutex_init(&chain_m2, &a)) {
+		ksft_print_msg("no priority inheritance mutex\n");
+		return KSFT_SKIP;
+	}
+	chain_step = 0;
+	chain_err = 0;
+	for (i = 0; i < 3; i++)
+		chain_tid[i] = 0;
+	if (pthread_create(&t[2], NULL, chain_n2, NULL) || chain_wait(1) ||
+	    pthread_create(&t[1], NULL, chain_n1, NULL) ||
+	    wait_for(2000, chain_tid[1]) ||
+	    pthread_create(&t[0], NULL, chain_d1, NULL)) {
+		ksft_print_msg("the threads of the chain could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(2000, prio_of(chain_tid[1]) == DL_TASK_PRIO || chain_err)) {
+		ksft_print_msg("N1 is not boosted by D1\n");
+		ret = KSFT_FAIL;
+	}
+	chain_step = 2;
+	if (ret == KSFT_PASS &&
+	    wait_for(2000, prio_of(chain_tid[2]) == DL_TASK_PRIO || chain_err)) {
+		ksft_print_msg("N2 is not boosted through N1\n");
+		ret = KSFT_FAIL;
+	}
+	chain_step = 3;
+	for (i = 0; i < 3; i++)
+		join(t[i]);
+	if (chain_err) {
+		ksft_print_msg("a thread of the chain failed\n");
+		ret = KSFT_FAIL;
+	}
+	pthread_mutex_destroy(&chain_m1);
+	pthread_mutex_destroy(&chain_m2);
+	return ret;
+}
+
+/*
+ * Run as the owner thread. With @arg (was_dl) set, first become a
+ * deadline task of 30ms every 100ms, run 1ms, and sleep on owner_go. The
+ * main thread switches the owner to SCHED_OTHER during that sleep. Then
+ * take pi_lock, and sleep 400ms while holding it. Return NULL, or 1 on an
+ * error.
+ */
+static void *sleeping_owner(void *arg)
+{
+	int was_dl = (long)arg;
+
+	owner_tid = gettid_();
+	if (was_dl) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			owner_holds = -1;
+			sem_post(&owner_ready);
+			return (void *)1;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		/*
+		 * Sleep. The main thread switches this thread to SCHED_OTHER
+		 * 200ms later, past its 0-lag time.
+		 */
+		sem_post(&owner_ready);
+		sem_wait_sig(&owner_go);
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	sleep_ns(400 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+/*
+ * A SCHED_OTHER owner takes pi_lock and sleeps 400ms while holding it.
+ * The main thread then waits 20ms for the lock, @rounds times. For each
+ * wait it becomes a deadline task of 5ms every 50ms. Each wait boosts the
+ * sleeping owner, and its timeout deboosts it. Every wait must time out.
+ *
+ * With @was_dl, the owner was a deadline task of 30ms every 100ms before.
+ * It ran 1ms and slept. The main thread switched it to SCHED_OTHER 200ms
+ * later, past its 0-lag time.
+ */
+static int pi_asleep(int was_dl, int rounds)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)(long)was_dl)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (was_dl) {
+		if (sem_wait_for(&owner_ready, 5000)) {
+			ksft_print_msg("the owner did not become a deadline task\n");
+			ret = KSFT_FAIL;
+		} else {
+			sleep_ns(200 * NSEC_PER_MSEC);
+			if (set_other(owner_tid)) {
+				ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+				ret = KSFT_FAIL;
+			}
+		}
+		sem_post(&owner_go);
+	}
+	if (ret == KSFT_PASS && (wait_for(5000, owner_holds) || owner_holds != 2)) {
+		ksft_print_msg("the owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS)
+		sleep_ns(20 * NSEC_PER_MSEC);
+	for (r = 0; r < rounds && ret == KSFT_PASS; r++) {
+		if (wait_lock_for(20, 1)) {
+			ksft_print_msg("round %d of %d failed\n", r + 1, rounds);
+			ret = KSFT_FAIL;
+		}
+	}
+	join(o);
+	return ret;
+}
+
+static int pi_other_owner_asleep(void)
+{
+	return pi_asleep(0, 5);
+}
+
+static int pi_former_owner_asleep(void)
+{
+	return pi_asleep(1, 5);
+}
+
+/*
+ * Run as the owner thread. Become a deadline task of 30ms every 100ms,
+ * and take pi_lock. Hold it for 300ms, spinning 2ms and sleeping 5ms in
+ * turn. A waiter boosts the owner when its deadline is the earlier one.
+ */
+static void *dl_owner(void *arg)
+{
+	uint64_t end;
+
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		owner_holds = -1;
+		return (void *)1;
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	end = now_ns() + 300 * NSEC_PER_MSEC;
+	while (now_ns() < end) {
+		spin_ns(2 * NSEC_PER_MSEC);
+		sleep_ns(5 * NSEC_PER_MSEC);
+	}
+	drop_pi_lock();
+	return NULL;
+}
+
+/*
+ * A deadline owner of 30ms every 100ms holds pi_lock for 300ms. The main
+ * thread waits 20ms for the lock, five times. For each wait it becomes a
+ * deadline task of 5ms every 50ms. Every wait must time out, and the
+ * owner must end without an error.
+ */
+static int pi_deadline_owner(void)
+{
+	int r, ret = KSFT_PASS, err;
+	pthread_t o;
+	void *res;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, dl_owner, NULL)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the deadline owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++)
+		if (wait_lock_for(20, 1))
+			ret = KSFT_FAIL;
+	err = pthread_join(o, &res);
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+	return res || err ? KSFT_FAIL : ret;
+}
+
+/*
+ * The owner thread: take pi_lock and spin 50ms with it. The unlock
+ * deboosts it while it runs.
+ */
+static void *running_owner(void *arg)
+{
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	spin_ns(50 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+/*
+ * A SCHED_OTHER owner takes pi_lock and spins 50ms. The main thread waits
+ * for the lock as a deadline task of 5ms every 50ms, which boosts the
+ * running owner. The owner's unlock deboosts it while it runs. The main
+ * thread must get the lock within 1s. Do this five times.
+ */
+static int pi_unlock_running(void)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (pi_init())
+			return KSFT_SKIP;
+		if (pthread_create(&o, NULL, running_owner, NULL)) {
+			ksft_print_msg("the owner thread could not start\n");
+			return KSFT_FAIL;
+		}
+		if (wait_for_step(5000, 100 * NSEC_PER_USEC, owner_holds) ||
+		    owner_holds != 2) {
+			ksft_print_msg("the owner never took the lock\n");
+			ret = KSFT_FAIL;
+		} else if (wait_lock_for(1000, 0)) {
+			ret = KSFT_FAIL;
+		}
+		join(o);
+		pi_fini();
+	}
+	return ret;
+}
+
+/* Parameter changes */
+
+static atomic_int t_state;	/* 1: about to sleep, or spinning */
+static _Atomic pid_t t_tid;
+static sem_t t_exit;
+
+/*
+ * The sleeper thread: become a deadline task of 20ms every 100ms. Run
+ * @arg ns of a fresh period, then sleep until t_exit is posted.
+ */
+static void *runs_then_sleeps(void *arg)
+{
+	uint64_t run = (uint64_t)(long)arg;
+
+	t_tid = gettid_();
+	if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	sleep_ns(PERIOD);	/* a fresh period */
+	spin_ns(run);
+	t_state = 1;
+	sem_wait_sig(&t_exit);
+	return NULL;
+}
+
+/*
+ * Start runs_then_sleeps() in @t, to run @run ns. Return 0 once it is
+ * about to sleep, or -1.
+ */
+static int start_sleeper(pthread_t *t, uint64_t run)
+{
+	t_state = 0;
+	if (sem_init(&t_exit, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	if (pthread_create(t, NULL, runs_then_sleeps, (void *)(long)run)) {
+		ksft_print_msg("the sleeper thread could not start\n");
+		return -1;
+	}
+	if (wait_for_step(5000, 100 * NSEC_PER_USEC, t_state) || t_state != 1) {
+		ksft_print_msg("the sleeper was refused SCHED_DEADLINE\n");
+		sem_post(&t_exit);	/* in case it gets there late */
+		join(*t);
+		return -1;
+	}
+	return 0;
+}
+
+static void stop_sleeper(pthread_t t)
+{
+	if (sem_post(&t_exit))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	join(t);
+}
+
+/* Return 0 if the root domain of @cg admits @want probe tasks now, or -1. */
+static int admits_in(const char *cg, int want, const char *what)
+{
+	int n = probe(cg);
+
+	if (n != want) {
+		ksft_print_msg("%s: %d probe tasks admitted, %d expected\n", what, n, want);
+		return -1;
+	}
+	return 0;
+}
+
+/* Do the same check in the root domain of the caller's cgroup. */
+static int admits(int want, const char *what)
+{
+	return admits_in(NULL, want, what);
+}
+
+/*
+ * A deadline task of 20ms every 100ms sleeps in P, past its 0-lag time.
+ * The main thread changes its runtime to 40ms, 10ms, 30ms and 20ms. After
+ * each change, P must admit one probe task fewer for each 10ms of the new
+ * runtime.
+ */
+static int params_asleep(void)
+{
+	static const uint64_t rt[] = { 40, 10, 30, 20 };
+	int base, i, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	/*
+	 * probe(NULL) probes P: this process is in P. The sleeper holds 20%,
+	 * which is two probe tasks. base is what P admits without it.
+	 */
+	base = probe(NULL) + 2;
+	for (i = 0; i < 4 && ret == KSFT_PASS; i++) {
+		char what[64];
+
+		if (set_deadline(t_tid, rt[i] * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		snprintf(what, sizeof(what), "asleep, changed to %llums every 100ms",
+			 (unsigned long long)rt[i]);
+		if (admits(base - (int)(rt[i] / 10), what))
+			ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+/*
+ * A deadline task of 20ms every 100ms runs 5ms in a new period, then
+ * sleeps. Its 0-lag time is then about 20ms ahead. 1ms later, the main
+ * thread changes its runtime to 40ms, then to 10ms. measured() checks
+ * that P admits as many probe tasks after the case as before.
+ */
+static int params_non_contending(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS;
+
+	if (start_sleeper(&t, 5 * NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(NSEC_PER_MSEC);
+	if (set_deadline(t_tid, 40 * NSEC_PER_MSEC, PERIOD) ||
+	    set_deadline(t_tid, 10 * NSEC_PER_MSEC, PERIOD))
+		ret = KSFT_FAIL;
+	stop_sleeper(t);
+	return ret;
+}
+
+static atomic_int spin_stop;
+/* Set before dl_spinner() starts. */
+static uint64_t spin_runtime, spin_flags;
+
+static void *dl_spinner(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, spin_runtime, PERIOD, spin_flags)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop)
+		;
+	return NULL;
+}
+
+/*
+ * Start dl_spinner() in @t, a deadline task of @runtime every 100ms with
+ * @flags. Return 0, or -1.
+ */
+static int start_spinner(pthread_t *t, uint64_t runtime, uint64_t flags)
+{
+	t_state = 0;
+	spin_stop = 0;
+	spin_runtime = runtime;
+	spin_flags = flags;
+	if (pthread_create(t, NULL, dl_spinner, NULL)) {
+		ksft_print_msg("the spinner thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state == 1)
+		return 0;
+	spin_stop = 1;		/* in case it gets there late */
+	join(*t);
+	return -1;
+}
+
+static void stop_spinner(pthread_t t)
+{
+	spin_stop = 1;
+	/* The thread may have left the class already. That is fine. */
+	set_other(t_tid);
+	join(t);
+}
+
+/*
+ * Wait until @t stops using CPU time: it is throttled. Return 0, or -1
+ * if that takes over 1s or its CPU time cannot be read.
+ */
+static int wait_throttled(pthread_t t)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC, last = cpu_time_of(t);
+
+	while (last && now_ns() < end) {
+		uint64_t cur;
+
+		sleep_ns(3 * NSEC_PER_MSEC);
+		cur = cpu_time_of(t);
+		if (cur == last)
+			return 0;
+		last = cur;
+	}
+	return -1;
+}
+
+/* Return 0 if @t runs 1ms more within @ns, or -1. */
+static int progresses(pthread_t t, uint64_t ns)
+{
+	uint64_t start = cpu_time_of(t), end = now_ns() + ns;
+
+	while (now_ns() < end) {
+		if (cpu_time_of(t) > start + NSEC_PER_MSEC)
+			return 0;
+		sleep_ns(2 * NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+/*
+ * A deadline task of 10ms every 100ms spins. Five times, wait until it is
+ * throttled, then change its runtime from the main thread: to 20ms, then
+ * back to 10ms, in turn. After each change it must run 1ms more within
+ * three periods.
+ */
+static int params_throttled(void)
+{
+	int r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0))
+		return KSFT_FAIL;
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (wait_throttled(t) ||
+		    set_deadline(t_tid, (r & 1 ? 10 : 20) * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (progresses(t, 3 * PERIOD)) {
+			ksft_print_msg("throttled, parameters changed: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	return ret;
+}
+
+/*
+ * A deadline task of 20ms every 100ms sleeps in P. The main thread tries
+ * to raise its runtime by more than P has free. sched_setattr() must fail
+ * with EBUSY, and P must admit as many probe tasks after as before.
+ */
+static int params_refused(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS, base, err;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	base = probe(NULL);
+	/*
+	 * P has room for base probe tasks of 10%, and not for one more. So
+	 * growing the task's runtime by (base + 1) * 10ms must be refused,
+	 * whatever the admission limit.
+	 */
+	if (20 + 10 * (base + 1) > 100) {
+		stop_sleeper(t);
+		ksft_print_msg("P has more room than one of its tasks can take\n");
+		return KSFT_SKIP;
+	}
+	err = set_deadline(t_tid, (20 + 10 * (uint64_t)(base + 1)) * NSEC_PER_MSEC, PERIOD);
+	if (err != -EBUSY) {
+		ksft_print_msg("an increase beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	} else if (admits(base, "after a refused increase")) {
+		ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+/*
+ * A SCHED_OTHER task with a slice of 40ms asks for SCHED_DEADLINE with
+ * SCHED_FLAG_KEEP_PARAMS. That changes the class, so it must be refused
+ * with EINVAL. It must not reserve bandwidth for the parameters it would
+ * keep, which take the slice as their runtime.
+ */
+static int params_keep_class_change(void)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,
+	};
+	int err;
+
+	if (syscall(SYS_sched_setattr, 0, &attr, 0)) {
+		ksft_print_msg("a slice of its own is refused: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	attr.sched_policy = SCHED_DEADLINE;
+	attr.sched_flags = SCHED_FLAG_KEEP_PARAMS;
+	attr.sched_deadline = PERIOD;
+	attr.sched_period = PERIOD;
+	err = syscall(SYS_sched_setattr, 0, &attr, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("to SCHED_DEADLINE with SCHED_FLAG_KEEP_PARAMS: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/*
+ * The waiter thread: wait up to 300ms for pi_lock as a deadline task,
+ * which boosts the owner. Return NULL if the wait timed out.
+ */
+static void *boosting_waiter(void *arg)
+{
+	return (void *)(long)wait_lock_for(300, 1);
+}
+
+/*
+ * Start a boosting_waiter() thread in @w. Wait up to 200ms for it to boost
+ * the owner. Return 0 once the owner is boosted. Otherwise report the
+ * failure, join the waiter and return -1.
+ */
+static int start_boost(pthread_t *w)
+{
+	if (pthread_create(w, NULL, boosting_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(200, prio_of(owner_tid) == DL_TASK_PRIO))
+		return 0;
+	ksft_print_msg("the waiter never boosted the owner\n");
+	join(*w);
+	return -1;
+}
+
+/*
+ * Join the boosting_waiter() thread @w. Return -1 if its wait did not
+ * time out.
+ */
+static int join_waiter(pthread_t w)
+{
+	void *res;
+	int err = pthread_join(w, &res);
+
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+	return err || res ? -1 : 0;
+}
+
+/*
+ * A SCHED_OTHER owner holds pi_lock and sleeps. The main thread gives it
+ * a slice of 40ms with sched_setattr(). A deadline waiter then boosts the
+ * owner into the deadline class. The main thread then calls
+ * sched_setattr() on the owner, with SCHED_DEADLINE and
+ * SCHED_FLAG_KEEP_PARAMS.
+ *
+ * The call must fail with EINVAL. The boost put the owner in the deadline
+ * class, but its policy is still SCHED_OTHER. So the call asks for another
+ * policy, which SCHED_FLAG_KEEP_PARAMS cannot give. If the call were
+ * admitted, it would reserve bandwidth for the 40ms slice, and nothing
+ * would release it.
+ */
+static int params_keep_boosted(void)
+{
+	struct dl_attr slice = {
+		.size = sizeof(slice),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,	/* the slice to keep */
+	};
+	struct dl_attr keep = {
+		.size = sizeof(keep),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = SCHED_FLAG_KEEP_PARAMS,
+		.sched_deadline = PERIOD,
+		.sched_period = PERIOD,
+	};
+	int ret = KSFT_PASS, err;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (syscall(SYS_sched_setattr, owner_tid, &slice, 0)) {
+		ksft_print_msg("the owner cannot take a slice of its own: %s\n",
+			       strerror(errno));
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	if (start_boost(&w)) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	err = syscall(SYS_sched_setattr, owner_tid, &keep, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("KEEP_PARAMS to SCHED_DEADLINE on a boosted owner: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	if (join_waiter(w))
+		ret = KSFT_FAIL;	/* the waiter said why */
+out:
+	join(o);
+	return ret;
+}
+
+/*
+ * A deadline waiter boosts a sleeping SCHED_OTHER owner. The main thread
+ * then calls sched_setattr() on the owner, with SCHED_OTHER. The call
+ * must succeed while the owner is boosted.
+ */
+static int boosted_owner_policy(void)
+{
+	int ret = KSFT_PASS, err;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (start_boost(&w)) {
+		join(o);
+		return KSFT_FAIL;
+	}
+	err = set_other(owner_tid);
+	if (err) {
+		ksft_print_msg("SCHED_OTHER on a boosted owner: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	}
+	if (join_waiter(w))
+		ret = KSFT_FAIL;	/* the waiter said why */
+	join(o);
+	return ret;
+}
+
+static atomic_int owner_first_period_ms;
+static _Atomic pid_t donor_tid;
+
+/*
+ * Run as the owner O of deboost_case(). @arg is its @mode. Store the CPU
+ * time of step 4, in ms, in owner_first_period_ms. Return NULL, 1 on an
+ * error, or 2 if no waiter boosted it within 1s.
+ */
+static void *deboosted_owner(void *arg)
+{
+	uint64_t t0, cpu0;
+	int err;
+
+	owner_tid = gettid_();
+	if (arg == (void *)1L) {
+		err = set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD);
+		if (err) {
+			ksft_print_msg("the owner refused as a deadline task: %s\n",
+				       strerror(-err));
+			return (void *)1;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		sem_post(&owner_ready);
+		sem_wait_sig(&owner_go);
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	t0 = now_ns();
+	while (prio_of(owner_tid) != DL_TASK_PRIO) {
+		if (now_ns() - t0 > NSEC_PER_SEC) {
+			drop_pi_lock();
+			return (void *)2;	/* never boosted */
+		}
+	}
+	if (arg == (void *)2L) {
+		err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+		if (err) {
+			ksft_print_msg("the boosted owner refused as a deadline task: %s\n",
+				       strerror(-err));
+			drop_pi_lock();
+			return (void *)1;
+		}
+	}
+	spin_ns(5 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	err = arg == (void *)2L ? 0 : set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	t0 = now_ns();
+	cpu0 = thread_time_ns();
+	while (now_ns() - t0 < PERIOD)
+		;
+	owner_first_period_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+	return NULL;
+}
+
+/*
+ * The waiter thread: take pi_lock as a deadline task of 40ms every
+ * 100ms. It boosts the owner, which then runs on the waiter's runtime.
+ * Return NULL once it got the lock.
+ */
+static void *donor_waiter(void *arg)
+{
+	int err;
+
+	donor_tid = gettid_();
+	err = set_deadline(0, 40 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	err = take_pi_lock();
+	if (!err)
+		drop_pi_lock();
+	if (set_other(0))
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+	return (void *)(long)err;
+}
+
+/*
+ * Run a deboost case. The owner O starts as SCHED_OTHER. The donor D is
+ * the waiter that boosts it: a deadline task of 40ms every 100ms.
+ * 1. O takes pi_lock and spins.
+ * 2. D blocks on pi_lock. This boosts O, which then runs on D's runtime.
+ * 3. O spins 5ms more and unlocks. The unlock deboosts O.
+ * 4. O becomes a deadline task of 10ms every 100ms. It spins for 100ms
+ *    and measures its CPU time.
+ *
+ * O must get about 10ms in step 4. The case fails above 20ms: O then
+ * still runs on the runtime that the boost left.
+ *
+ * @mode changes one step:
+ * - 0: nothing.
+ * - 1: before step 1, O runs 1ms as a deadline task of 60ms every 100ms,
+ *   and sleeps. The main thread switches it to SCHED_OTHER 300ms later,
+ *   past its 0-lag time.
+ * - 2: O becomes the deadline task of step 4 in step 3 instead, before
+ *   the 5ms. It stays in the deadline class at the deboost.
+ */
+static int deboost_case(long mode)
+{
+	int err, ret = KSFT_PASS;
+	void *o_ret, *w_ret;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	owner_first_period_ms = -1;
+	if (pthread_create(&o, NULL, deboosted_owner, (void *)mode)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (mode == 1) {
+		if (sem_wait_for(&owner_ready, 5000)) {
+			/* The owner may get there late. Let it end. */
+			sem_post(&owner_go);
+			join(o);
+			return KSFT_FAIL;	/* the owner said why */
+		}
+		sleep_ns(3 * PERIOD);
+		err = proc_state_of(owner_tid) == 'S' ? set_other(owner_tid) : -EAGAIN;
+		sem_post(&owner_go);
+		if (err) {
+			ksft_print_msg("the owner could not leave the class asleep: %s\n",
+				       strerror(-err));
+			join(o);
+			return KSFT_FAIL;
+		}
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (pthread_create(&w, NULL, donor_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	err = pthread_join(w, &w_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	} else if (w_ret) {
+		ret = KSFT_FAIL;	/* the waiter said why */
+	}
+	err = pthread_join(o, &o_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	} else if (o_ret == (void *)2) {
+		ksft_print_msg("the waiter never boosted the owner\n");
+		ret = KSFT_FAIL;
+	} else if (owner_first_period_ms < 0) {
+		ret = KSFT_FAIL;
+	} else if (owner_first_period_ms > 20) {
+		ksft_print_msg("the owner ran %d ms of its first 100: the donor's runtime\n",
+			       (int)owner_first_period_ms);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Run deboost_case() with a plain SCHED_OTHER owner. */
+static int deboost_resets_period(void)
+{
+	return deboost_case(0);
+}
+
+/*
+ * Run deboost_case() with an owner that left the deadline class asleep.
+ * Its bandwidth was released then.
+ */
+static int deboost_released_resets_period(void)
+{
+	return deboost_case(1);
+}
+
+/* Run deboost_case() with an owner that enters the class while boosted. */
+static int deboost_in_class_resets_period(void)
+{
+	return deboost_case(2);
+}
+
+/*
+ * Return the runtime that the deadline task @tid has left, in ns, from
+ * dl.runtime in /proc/<tid>/sched. Return -1 if it cannot be read.
+ */
+static long long dl_runtime_left(pid_t tid)
+{
+	char path[64], line[256];
+	long long left = -1;
+	FILE *f;
+
+	snprintf(path, sizeof(path), "/proc/%d/sched", tid);
+	f = fopen(path, "r");
+	if (!f)
+		return -1;
+	while (fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, "dl.runtime", 10) && val) {
+			left = atoll(val + 1);
+			break;
+		}
+	}
+	fclose(f);
+	return left;
+}
+
+#define REFILL_OWN_RUNTIME	(5 * NSEC_PER_MSEC)
+#define REFILL_DONOR_RUNTIME	(2 * NSEC_PER_MSEC)
+#define REFILL_DONOR_PERIOD	(10 * NSEC_PER_MSEC)
+
+static sem_t refill_donor_ready;
+static atomic_int refill_donor_locking;
+static _Atomic long long refill_owner_left;
+
+/*
+ * Run as the donor D of refill_case(). Become a deadline task of 2ms
+ * every 10ms and sleep until the owner holds pi_lock. Then block on
+ * pi_lock. Return NULL once it got the lock, or 1 on an error.
+ */
+static void *refill_donor(void *arg)
+{
+	int err;
+
+	donor_tid = gettid_();
+	err = set_deadline(0, REFILL_DONOR_RUNTIME, REFILL_DONOR_PERIOD);
+	if (err) {
+		ksft_print_msg("the donor refused as a deadline task: %s\n", strerror(-err));
+		donor_tid = -1;
+		sem_post(&refill_donor_ready);
+		return (void *)1;
+	}
+	sem_post(&refill_donor_ready);
+	sem_wait_sig(&owner_go);
+	refill_donor_locking = 1;
+	err = take_pi_lock();
+	if (!err)
+		drop_pi_lock();
+	if (set_other(0))
+		ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
+	return (void *)(long)err;
+}
+
+/*
+ * Run as the owner O of refill_case(). @arg is the CPU time, in ms, that
+ * O spins with the lock once it is boosted. Store the runtime O has left
+ * after the unlock in refill_owner_left. Return NULL, 1 on an error, or 2
+ * if the donor did not block on the lock within 1s.
+ */
+static void *refill_owner(void *arg)
+{
+	uint64_t t0;
+	int err;
+
+	owner_tid = gettid_();
+	err = set_deadline(0, REFILL_OWN_RUNTIME, PERIOD);
+	if (err) {
+		ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	spin_ns(2 * NSEC_PER_MSEC);
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	sem_post(&owner_go);
+	t0 = now_ns();
+	while (!refill_donor_locking || proc_state_of(donor_tid) != 'S') {
+		if (now_ns() - t0 > NSEC_PER_SEC) {
+			drop_pi_lock();
+			set_other(0);
+			return (void *)2;
+		}
+	}
+	spin_ns((long)arg * NSEC_PER_MSEC);
+	drop_pi_lock();
+	refill_owner_left = dl_runtime_left(owner_tid);
+	if (set_other(0))
+		ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+	return NULL;
+}
+
+/*
+ * Run a case on the runtime that a deadline owner has after its boost.
+ * O and D share the CPU of P.
+ *
+ *   owner O: 5ms every 100ms             donor D: 2ms every 10ms
+ *
+ *                                        sleeps
+ *   starts its period, runs 2ms
+ *   takes pi_lock
+ *   wakes D up
+ *                                        preempts O: its deadline is
+ *                                          earlier
+ *                                        blocks on pi_lock, boosts O
+ *   runs @spin_ms more
+ *   unlocks: the boost ends
+ *                                        takes pi_lock, unlocks
+ *   reads its dl.runtime
+ *
+ * A boosted task that runs out of runtime is not throttled. Its runtime
+ * is refilled from the parameters of its donor: 2ms here.
+ *
+ * - @spin_ms is 8: O runs out of its own 5ms, and is refilled from D at
+ *   least once. At the unlock it holds what the last refill left, 2ms at
+ *   most. That is not the runtime of its own reservation. O must start a
+ *   new period at the unlock, with 5ms. The case fails if O has 3.5ms or
+ *   less.
+ * - @spin_ms is 1: O has used about 3ms of its own 5ms at the unlock, and
+ *   was never refilled. It must keep what it has left. A new period would
+ *   hand it a full runtime at every contended unlock. The case fails if O
+ *   has 4ms or more.
+ */
+static int refill_case(long spin_ms)
+{
+	void *o_ret = NULL, *d_ret = NULL;
+	int err, ret = KSFT_PASS;
+	pthread_t o, d;
+	long long left;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (sem_init(&refill_donor_ready, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		pi_fini();
+		return KSFT_SKIP;
+	}
+	refill_donor_locking = 0;
+	refill_owner_left = -1;
+	donor_tid = 0;
+	if (pthread_create(&d, NULL, refill_donor, NULL)) {
+		ksft_print_msg("the donor thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (sem_wait_for(&refill_donor_ready, 5000) || donor_tid <= 0) {
+		ksft_print_msg("the donor is not ready\n");
+		sem_post(&owner_go);
+		join(d);
+		return KSFT_FAIL;
+	}
+	if (pthread_create(&o, NULL, refill_owner, (void *)spin_ms)) {
+		ksft_print_msg("the owner thread could not start\n");
+		sem_post(&owner_go);
+		join(d);
+		return KSFT_FAIL;
+	}
+	err = pthread_join(o, &o_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	}
+	if (o_ret == (void *)1)
+		sem_post(&owner_go);	/* the owner never woke the donor up */
+	err = pthread_join(d, &d_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	}
+	left = refill_owner_left;
+	if (o_ret == (void *)2) {
+		ksft_print_msg("the donor never blocked on the lock\n");
+		ret = KSFT_FAIL;
+	} else if (o_ret || d_ret) {
+		ret = KSFT_FAIL;	/* the thread said why */
+	} else if (left < 0) {
+		ksft_print_msg("the runtime of the owner cannot be read\n");
+		ret = KSFT_SKIP;
+	} else if (spin_ms == 8 && left <= 3500 * NSEC_PER_USEC) {
+		ksft_print_msg("the owner has %lld us after the unlock: what the donor's refill left\n",
+			       left / NSEC_PER_USEC);
+		ret = KSFT_FAIL;
+	} else if (spin_ms == 1 && left >= 4 * NSEC_PER_MSEC) {
+		ksft_print_msg("the owner has %lld us after the unlock: a new runtime, with no refill from the donor\n",
+			       left / NSEC_PER_USEC);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Run refill_case() with an owner that is refilled from its donor. */
+static int deboost_after_donor_refill(void)
+{
+	return refill_case(8);
+}
+
+/* Run refill_case() with an owner that stays on its own runtime. */
+static int deboost_keeps_own_runtime(void)
+{
+	return refill_case(1);
+}
+
+static sem_t ret_ready, ret_go;
+static _Atomic pid_t ret_tid;
+static atomic_int ret_first_period_ms;
+
+/*
+ * Run as the thread of the return cases. Become a deadline task of 50ms
+ * every 100ms and run 20ms. 30ms of runtime are left, and the 0-lag time
+ * is about 20ms ahead. Then change to 10ms every 100ms, as @arg says:
+ * - 0: the thread switches itself to SCHED_OTHER and back to
+ *   SCHED_DEADLINE, while it runs.
+ * - 1: the thread sleeps. The main thread switches it to SCHED_OTHER and
+ *   back.
+ * - 2: the thread changes its own parameters while it runs. It stays in
+ *   SCHED_DEADLINE.
+ * Then spin for 100ms, and store the CPU time used, in ms, in
+ * ret_first_period_ms. Return NULL, or 1 on an error.
+ */
+static void *returner(void *arg)
+{
+	uint64_t t0, cpu0;
+	int err;
+
+	ret_tid = gettid_();
+	err = set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("the task refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	spin_ns(20 * NSEC_PER_MSEC);
+	if (arg == (void *)1L) {
+		sem_post(&ret_ready);
+		sem_wait_sig(&ret_go);
+	} else {
+		err = arg ? 0 : set_other(0);
+		if (!err)
+			err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+		if (err) {
+			ksft_print_msg("the task could not change: %s\n", strerror(-err));
+			return (void *)1;
+		}
+	}
+	t0 = now_ns();
+	cpu0 = thread_time_ns();
+	while (now_ns() - t0 < PERIOD)
+		;
+	ret_first_period_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+	return NULL;
+}
+
+/*
+ * Run a return case: returner() with @mode as its argument. The thread
+ * must get about 10ms of CPU time in the 100ms after the change. That is
+ * its new runtime, not the 30ms left from the old one. The case fails
+ * above 20ms.
+ */
+static int return_case(long mode)
+{
+	int err, ret = KSFT_PASS;
+	void *t_ret;
+	pthread_t t;
+
+	if (sem_init(&ret_ready, 0, 0) || sem_init(&ret_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	ret_first_period_ms = -1;
+	if (pthread_create(&t, NULL, returner, (void *)mode)) {
+		ksft_print_msg("the task thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (mode == 1) {
+		if (sem_wait_for(&ret_ready, 5000)) {
+			/* The thread may get there late. Let it end. */
+			sem_post(&ret_go);
+			join(t);
+			return KSFT_FAIL;	/* the task said why */
+		}
+		/* Before its 0-lag time, about 20ms ahead */
+		err = wait_for(10, proc_state_of(ret_tid) == 'S') ? -EAGAIN : 0;
+		if (!err)
+			err = set_other(ret_tid);
+		if (!err)
+			err = set_deadline(ret_tid, 10 * NSEC_PER_MSEC, PERIOD);
+		sem_post(&ret_go);
+		if (err) {
+			ksft_print_msg("the task could not come back asleep: %s\n",
+				       strerror(-err));
+			join(t);
+			return KSFT_FAIL;
+		}
+	}
+	err = pthread_join(t, &t_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	} else if (t_ret || ret_first_period_ms < 0) {
+		ret = KSFT_FAIL;	/* the task said why */
+	} else if (ret_first_period_ms > 20) {
+		ksft_print_msg("the task ran %d ms of its first 100: the old runtime\n",
+			       (int)ret_first_period_ms);
+		ret = KSFT_FAIL;
+	}
+	sem_destroy(&ret_ready);
+	sem_destroy(&ret_go);
+	return ret;
+}
+
+/*
+ * A running task leaves SCHED_DEADLINE and comes back at once with a
+ * smaller runtime. Its 0-lag time is ahead, so it keeps its deadline and
+ * the runtime it has left. It must not run on more than the new runtime.
+ */
+static int return_running_new_params(void)
+{
+	return return_case(0);
+}
+
+/* The same, for a task that leaves and comes back while it sleeps. */
+static int return_asleep_new_params(void)
+{
+	return return_case(1);
+}
+
+/* The same, for a task that changes its parameters within the class. */
+static int change_running_new_params(void)
+{
+	return return_case(2);
+}
+
+static pthread_mutex_t pi_lock2;
+static sem_t holder_go;
+static atomic_int holder_holds, chain_waits, donor_done, x_waits, l2_takes;
+static atomic_int chain_rank, x_rank;
+static _Atomic pid_t x_tid, fair_tid;
+
+/* The holder thread: take pi_lock2, and hold it until holder_go. */
+static void *l2_holder(void *arg)
+{
+	int err = pthread_mutex_lock(&pi_lock2);
+
+	if (err) {
+		ksft_print_msg("the holder cannot take pi_lock2: %s\n", strerror(err));
+		holder_holds = -1;
+		return (void *)1;
+	}
+	holder_holds = 1;
+	sem_wait_sig(&holder_go);
+	pthread_mutex_unlock(&pi_lock2);
+	return NULL;
+}
+
+/*
+ * Take pi_lock2. Store in @rank how many threads took it before the
+ * caller. Then unlock it. Return 0, or -1 if the lock cannot be taken.
+ */
+static int take_l2_ranked(atomic_int *rank)
+{
+	int err = pthread_mutex_lock(&pi_lock2);
+
+	if (err) {
+		ksft_print_msg("a waiter cannot take pi_lock2: %s\n", strerror(err));
+		return -1;
+	}
+	*rank = l2_takes++;
+	pthread_mutex_unlock(&pi_lock2);
+	return 0;
+}
+
+/*
+ * Run as the owner thread, a deadline task of 10ms every 200ms. Take
+ * pi_lock, and sleep until owner_go is posted. The donor boosts the owner
+ * during that sleep. Then spin 20ms. The owner's 10ms run out, and the
+ * boost refills its runtime from the donor's 40ms. Then wait on pi_lock2
+ * while holding pi_lock. The donor's timeout deboosts the owner during
+ * that wait. The owner then has more runtime left than its own 10ms.
+ * Return NULL, 1 on an error, or 2 if the spin took over 50ms: the owner
+ * was throttled, so it was not boosted.
+ */
+static void *chain_owner(void *arg)
+{
+	uint64_t t0;
+	int err;
+
+	owner_tid = gettid_();
+	err = set_deadline(0, 10 * NSEC_PER_MSEC, 2 * PERIOD);
+	if (err) {
+		ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+		owner_holds = -1;
+		return (void *)1;
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	sem_wait_sig(&owner_go);
+	t0 = now_ns();
+	spin_ns(20 * NSEC_PER_MSEC);
+	if (now_ns() - t0 > PERIOD / 2) {
+		drop_pi_lock();
+		return (void *)2;	/* throttled: never boosted */
+	}
+	chain_waits = 1;
+	err = take_l2_ranked(&chain_rank);
+	drop_pi_lock();
+	return err ? (void *)1 : NULL;
+}
+
+/*
+ * The donor thread: a deadline task of 40ms every 100ms. Wait 150ms for
+ * pi_lock. Its timeout deboosts the owner, which waits on pi_lock2 then.
+ */
+static void *timed_donor(void *arg)
+{
+	struct timespec ts;
+	int err;
+
+	donor_tid = gettid_();
+	err = set_deadline(0, 40 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("the donor refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += 150 * 1000000L;
+	ts.tv_sec += ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	err = pthread_mutex_timedlock(&pi_lock, &ts);
+	donor_done = 1;
+	if (!err)
+		pthread_mutex_unlock(&pi_lock);
+	if (set_other(0))
+		ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
+	if (err != ETIMEDOUT) {
+		ksft_print_msg("the donor: %s, not a timeout\n",
+			       err ? strerror(err) : "got pi_lock");
+		return (void *)1;
+	}
+	return NULL;
+}
+
+/*
+ * The other waiter: a deadline task of 5ms every 50ms. Its deadline is
+ * earlier than the one the owner gets at the deboost. Wait on pi_lock2.
+ */
+static void *early_waiter(void *arg)
+{
+	int err;
+
+	x_tid = gettid_();
+	err = set_deadline(0, 5 * NSEC_PER_MSEC, PERIOD / 2);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	x_waits = 1;
+	err = take_l2_ranked(&x_rank);
+	if (set_other(0))
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+	return err ? (void *)1 : NULL;
+}
+
+/* Set up pi_lock2 and holder_go. Return 0, or -1. */
+static int pi_init_lock2(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err) {
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+		if (!err)
+			err = pthread_mutex_init(&pi_lock2, &a);
+		pthread_mutexattr_destroy(&a);
+	}
+	if (err) {
+		ksft_print_msg("no second priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	if (sem_init(&holder_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		pthread_mutex_destroy(&pi_lock2);
+		return -1;
+	}
+	holder_holds = 0;
+	chain_waits = 0;
+	donor_done = 0;
+	x_waits = 0;
+	l2_takes = 0;
+	chain_rank = -1;
+	x_rank = -1;
+	donor_tid = 0;
+	x_tid = 0;
+	fair_tid = 0;
+	return 0;
+}
+
+/*
+ * Check the order of the waiters of pi_lock2 after a deboost. The threads
+ * are:
+ * - the holder: SCHED_OTHER. It has pi_lock2.
+ * - the owner: deadline, 10ms every 200ms. It has pi_lock.
+ * - the donor: deadline, 40ms every 100ms.
+ * - the other waiter: deadline, 5ms every 50ms.
+ *
+ * 1. The donor waits on pi_lock, and boosts the owner.
+ * 2. The owner spins on the donor's runtime, then waits on pi_lock2.
+ * 3. The other waiter waits on pi_lock2.
+ * 4. The donor times out after 150ms. The deboost starts a new period
+ *    for the owner, which has more runtime left than its own 10ms.
+ * 5. The holder drops pi_lock2.
+ *
+ * The other waiter must take pi_lock2 first: its deadline is earlier
+ * than the owner's new one. At step 4 the owner is not queued, and the
+ * chain walk copies its deadline into its waiter node on pi_lock2. A
+ * deadline of 0 in the node would put the owner first.
+ */
+static int deboost_blocked_keeps_order(void)
+{
+	void *o_ret = NULL, *d_ret = NULL, *x_ret = NULL;
+	int ret = KSFT_FAIL, have_o = 0, have_d = 0, have_x = 0;
+	pthread_t h, o, d, x;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pi_init_lock2()) {
+		pi_fini();
+		return KSFT_SKIP;
+	}
+	if (pthread_create(&h, NULL, l2_holder, NULL)) {
+		ksft_print_msg("the holder thread could not start\n");
+		goto out_fini;
+	}
+	if (wait_for(5000, holder_holds) || holder_holds != 1) {
+		ksft_print_msg("the holder never took pi_lock2\n");
+		goto out;
+	}
+	have_o = !pthread_create(&o, NULL, chain_owner, NULL);
+	if (!have_o || wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took pi_lock\n");
+		goto out;
+	}
+	have_d = !pthread_create(&d, NULL, timed_donor, NULL);
+	if (!have_d || wait_for(1000, donor_tid && proc_state_of(donor_tid) == 'S')) {
+		ksft_print_msg("the donor never waited on pi_lock\n");
+		goto out;
+	}
+	sem_post(&owner_go);
+	if (wait_for(1000, chain_waits && proc_state_of(owner_tid) == 'S')) {
+		ksft_print_msg("the owner never waited on pi_lock2\n");
+		goto out;
+	}
+	if (donor_done) {
+		ksft_print_msg("the donor timed out before the owner waited on pi_lock2\n");
+		goto out;
+	}
+	have_x = !pthread_create(&x, NULL, early_waiter, NULL);
+	if (!have_x || wait_for(1000, x_waits && proc_state_of(x_tid) == 'S')) {
+		ksft_print_msg("the other waiter never waited on pi_lock2\n");
+		goto out;
+	}
+	/* The donor's timeout deboosts the owner, which waits on pi_lock2. */
+	have_d = 0;
+	if (!pthread_join(d, &d_ret) && !d_ret)
+		ret = KSFT_PASS;
+out:
+	/* The owner may get there late. Let it end. */
+	sem_post(&owner_go);
+	sem_post(&holder_go);
+	if (have_d)
+		join(d);
+	if (have_x && pthread_join(x, &x_ret))
+		ret = KSFT_FAIL;
+	if (have_o && pthread_join(o, &o_ret))
+		ret = KSFT_FAIL;
+	join(h);
+	if (o_ret == (void *)2) {
+		ksft_print_msg("the donor never boosted the owner\n");
+		ret = KSFT_FAIL;
+	} else if (o_ret || x_ret) {
+		ret = KSFT_FAIL;	/* the thread said why */
+	} else if (ret == KSFT_PASS && chain_rank < x_rank) {
+		ksft_print_msg("the deboosted owner took pi_lock2 before the earlier waiter\n");
+		ret = KSFT_FAIL;
+	}
+out_fini:
+	pthread_mutex_destroy(&pi_lock2);
+	sem_destroy(&holder_go);
+	pi_fini();
+	return ret;
+}
+
+/*
+ * The fair waiter: wait on pi_lock2 as a fair task. With @arg set, hold
+ * pi_lock meanwhile, for a donor to boost it. Otherwise the main thread
+ * makes it a deadline task while it waits. It leaves the class at the end.
+ */
+static void *fair_waiter(void *arg)
+{
+	int err;
+
+	fair_tid = gettid_();
+	if (arg && take_pi_lock())
+		return (void *)1;
+	chain_waits = 1;
+	err = take_l2_ranked(&chain_rank);
+	if (arg)
+		drop_pi_lock();
+	else if (set_other(0))
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+	return err ? (void *)1 : NULL;
+}
+
+/*
+ * Check the order of the waiters of pi_lock2 when one of them enters the
+ * class while it waits. The holder has pi_lock2:
+ * 1. The fair waiter waits on pi_lock2. With @boost, it holds pi_lock.
+ * 2. The other waiter, a deadline task, waits on pi_lock2.
+ * 3. The fair waiter enters the class:
+ *    - Without @boost, the main thread makes it a deadline task of 10ms
+ *      every 200ms.
+ *    - With @boost, a donor of 40ms every 100ms waits on pi_lock, and
+ *      boosts it.
+ * 4. The holder drops pi_lock2.
+ *
+ * The other waiter must take pi_lock2 first: it started its period
+ * earlier, with a shorter deadline. At step 3 the fair waiter is not
+ * queued, and the chain walk copies its deadline into its waiter node on
+ * pi_lock2. A deadline of 0 in the node would put the fair waiter first.
+ */
+static int blocked_keeps_order(long boost)
+{
+	void *f_ret = NULL, *x_ret = NULL, *w_ret = NULL;
+	int ret = KSFT_FAIL, have_f = 0, have_x = 0, have_w = 0, err;
+	pthread_t h, f, x, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pi_init_lock2()) {
+		pi_fini();
+		return KSFT_SKIP;
+	}
+	if (pthread_create(&h, NULL, l2_holder, NULL)) {
+		ksft_print_msg("the holder thread could not start\n");
+		goto out_fini;
+	}
+	if (wait_for(5000, holder_holds) || holder_holds != 1) {
+		ksft_print_msg("the holder never took pi_lock2\n");
+		goto out;
+	}
+	have_f = !pthread_create(&f, NULL, fair_waiter, (void *)boost);
+	if (!have_f || wait_for(1000, chain_waits && proc_state_of(fair_tid) == 'S')) {
+		ksft_print_msg("the fair waiter never waited on pi_lock2\n");
+		goto out;
+	}
+	have_x = !pthread_create(&x, NULL, early_waiter, NULL);
+	if (!have_x || wait_for(1000, x_waits && proc_state_of(x_tid) == 'S')) {
+		ksft_print_msg("the other waiter never waited on pi_lock2\n");
+		goto out;
+	}
+	if (boost) {
+		have_w = !pthread_create(&w, NULL, donor_waiter, NULL);
+		if (!have_w || wait_for(1000, donor_tid && proc_state_of(donor_tid) == 'S')) {
+			ksft_print_msg("the donor never waited on pi_lock\n");
+			goto out;
+		}
+		if (prio_of(fair_tid) != DL_TASK_PRIO) {
+			ksft_print_msg("the donor never boosted the fair waiter\n");
+			goto out;
+		}
+	} else {
+		err = set_deadline(fair_tid, 10 * NSEC_PER_MSEC, 2 * PERIOD);
+		if (err) {
+			ksft_print_msg("the fair waiter refused as a deadline task: %s\n",
+				       strerror(-err));
+			if (err == -EBUSY)
+				ret = KSFT_SKIP;	/* no room in P */
+			goto out;
+		}
+	}
+	ret = KSFT_PASS;
+out:
+	sem_post(&holder_go);	/* pi_lock2 goes to its top waiter */
+	if (have_w && pthread_join(w, &w_ret))
+		ret = KSFT_FAIL;
+	if (have_x && pthread_join(x, &x_ret))
+		ret = KSFT_FAIL;
+	if (have_f && pthread_join(f, &f_ret))
+		ret = KSFT_FAIL;
+	join(h);
+	if (f_ret || x_ret || w_ret) {
+		ret = KSFT_FAIL;	/* the thread said why */
+	} else if (ret == KSFT_PASS && chain_rank < x_rank) {
+		ksft_print_msg("the fair waiter took pi_lock2 before the earlier waiter\n");
+		ret = KSFT_FAIL;
+	}
+out_fini:
+	pthread_mutex_destroy(&pi_lock2);
+	sem_destroy(&holder_go);
+	pi_fini();
+	return ret;
+}
+
+/* sched_setattr() puts the fair waiter in the class. */
+static int setattr_blocked_keeps_order(void)
+{
+	return blocked_keeps_order(0);
+}
+
+/* A boost puts the fair waiter in the class. */
+static int boost_blocked_keeps_order(void)
+{
+	return blocked_keeps_order(1);
+}
+
+/* cpuset moves, from the main process, outside P */
+
+/* Shared with the processes of a case: atomic in shared memory. */
+static atomic_int *mv_state;
+
+/*
+ * Wait up to 5s for a child to report in mv_state[0]. Return 0 if it is
+ * up (1). Return -1 if it was refused SCHED_DEADLINE (-1), could not
+ * start its threads (-2) or did not report. @fast polls every 100us.
+ */
+static int child_up(const char *what, int fast)
+{
+	if (wait_for_step(5000, fast ? 100 * NSEC_PER_USEC : NSEC_PER_MSEC, mv_state[0])) {
+		ksft_print_msg("%s: the task never started\n", what);
+		return -1;
+	}
+	if (mv_state[0] == -2) {
+		ksft_print_msg("%s: the process cannot start its threads\n", what);
+		return -1;
+	}
+	if (mv_state[0] != 1) {
+		ksft_print_msg("%s: the task was refused SCHED_DEADLINE\n", what);
+		return -1;
+	}
+	return 0;
+}
+
+enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_YIELDED, MV_NON_CONTENDING };
+
+/*
+ * Fork a process in the caller's cgroup. The process becomes a deadline
+ * task of @runtime every 100ms, and does what @mode says:
+ * - MV_ASLEEP: run 1ms, set mv_state[0] to 1, then sleep.
+ * - MV_THROTTLED: set mv_state[0] to 1, then spin and count in
+ *   mv_state[1].
+ * - MV_YIELDED: set mv_state[0] to 1. Then spin 0.5ms, count in
+ *   mv_state[1] and yield, in a loop.
+ * - MV_NON_CONTENDING: sleep a period, run a quarter of its runtime, set
+ *   mv_state[0] to 1, then sleep. Its 0-lag time is still ahead.
+ * If it is refused SCHED_DEADLINE, the process sets mv_state[0] to -1 and
+ * exits. Return its pid, or -1.
+ */
+static pid_t mover(enum mv_mode mode, uint64_t runtime)
+{
+	pid_t pid = fork_child("a deadline task");
+
+	if (!pid) {
+		if (set_deadline(0, runtime, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		if (mode == MV_ASLEEP) {
+			spin_ns(NSEC_PER_MSEC);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_NON_CONTENDING) {
+			sleep_ns(PERIOD);
+			spin_ns(runtime / 4);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_YIELDED) {
+			*mv_state = 1;
+			for (;;) {
+				spin_ns(NSEC_PER_MSEC / 2);
+				mv_state[1]++;
+				sched_yield();
+			}
+		}
+		*mv_state = 1;
+		for (;;)
+			mv_state[1]++;
+	}
+	return pid;
+}
+
+static int wait_spinner(int moving);
+
+/*
+ * Move a deadline task into P and back, from outside P. The task is a
+ * mover() process of 30ms every 100ms, forked in the cgroup of the main
+ * process. Wait until it is in the state of @mode:
+ * - MV_ASLEEP: asleep for two periods, past its 0-lag time.
+ * - MV_NON_CONTENDING: asleep for 1ms, before its 0-lag time.
+ * - MV_THROTTLED, MV_YIELDED: its counter has stopped for 5ms.
+ * Then write its pid to cgroup.procs of P, then of the root cgroup. Both
+ * moves must succeed. A throttled or yielded task must run again within
+ * three periods. Then kill the task. P and the root cgroup must each
+ * admit as many probe tasks as before the case.
+ */
+static int moves(enum mv_mode mode, const char *name)
+{
+	int base_p, base_r, after_p, after_r, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up(name, mode == MV_NON_CONTENDING)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (mode == MV_NON_CONTENDING) {
+		sleep_ns(NSEC_PER_MSEC);
+	} else if (mode == MV_ASLEEP) {
+		sleep_ns(2 * PERIOD);
+	} else if (wait_spinner(0)) {
+		/* Throttled, or yielded: its counter stops. */
+		ksft_print_msg("%s: never stopped\n", name);
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		ret = KSFT_FAIL;
+	}
+	if (mode == MV_THROTTLED || mode == MV_YIELDED) {
+		int last = mv_state[1];
+
+		sleep_ns(3 * PERIOD);
+		if (mv_state[1] == last) {
+			ksft_print_msg("%s: moved, never ran again\n", name);
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after_p = probe(cg_p);
+	after_r = probe(root);
+	if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+		ksft_print_msg("%s: probe tasks admitted, P %d -> %d, the other CPUs %d -> %d\n",
+			       name, base_p, after_p, base_r, after_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A deadline task of 60ms every 100ms sleeps in P. Move a second sleeping
+ * task of 60ms every 100ms into P from outside. The move must fail with
+ * EBUSY. P and the root cgroup must each admit as many probe tasks after
+ * as before.
+ */
+static int move_refused(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, err;
+	pid_t in_p, pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	pid = mover(MV_ASLEEP, 60 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a task of 60% to move into P", 0)) {
+		reap(pid);
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	err = enter(cg_p, pid);
+	if (err != -EBUSY) {
+		ksft_print_msg("a move beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	reap(in_p);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a refused move changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A runtime of 1024ns every 2s gives a bandwidth of 0. A cpuset move
+ * skips such a task: it has no bandwidth to move.
+ *
+ * A deadline task of 60ms every 100ms sleeps in P. A second task sleeps
+ * outside P. The main process makes it a deadline task of 1024ns every
+ * 2s, moves it into P, and wakes it. It then runs on P's CPU. The main
+ * process then raises it to 60ms every 100ms. P has no room for that, so
+ * sched_setattr() must fail with EBUSY. The kernel must check the request
+ * against P's root domain, not against the one the task left.
+ */
+static int move_zero_bw(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, err, zfd[2];
+	pid_t in_p, pid;
+
+	if (pipe(zfd))
+		return KSFT_FAIL;
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+		reap(in_p);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	mv_state[0] = 0;
+	pid = fork_child("a task with no bandwidth");
+	if (!pid) {
+		char c;
+
+		close(zfd[1]);
+		mv_state[0] = 1;
+		while (read(zfd[0], &c, 1) == 1)
+			;
+		_exit(0);
+	}
+	if (pid < 0 || child_up("a task to move into P", 0) ||
+	    wait_for(1000, proc_state_of(pid) == 'S')) {
+		reap(pid);
+		reap(in_p);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	/*
+	 * Set the parameters from here, while the task sleeps. With a runtime
+	 * of 1024ns, the task could be throttled before it sleeps again, for
+	 * up to its period of 2s.
+	 */
+	err = set_deadline(pid, 1024, 2 * NSEC_PER_SEC);
+	if (err) {
+		ksft_print_msg("a runtime of 1024ns every 2s: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	} else if (enter(cg_p, pid)) {
+		ksft_print_msg("a task with no bandwidth: its move into P refused\n");
+		ret = KSFT_FAIL;
+	} else if (write(zfd[1], "w", 1) != 1 || wait_for(1000, task_cpu_of(pid) == top_cpu(1))) {
+		ksft_print_msg("a task moved into P did not wake on P's CPU\n");
+		ret = KSFT_FAIL;
+	} else {
+		err = set_deadline(pid, 60 * NSEC_PER_MSEC, PERIOD);
+		if (err != -EBUSY) {
+			ksft_print_msg("0%% to 60%% beside a task of 60%% in P: %s, not EBUSY\n",
+				       err ? strerror(-err) : "accepted");
+			ret = KSFT_FAIL;
+		}
+	}
+	/* Throttled, it would wait up to 2s to run and die. */
+	set_other(pid);
+	reap(pid);
+	reap(in_p);
+out:
+	close(zfd[0]);
+	close(zfd[1]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a refused change changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int move_asleep(void)
+{
+	return moves(MV_ASLEEP, "move_asleep");
+}
+
+static int move_throttled(void)
+{
+	return moves(MV_THROTTLED, "move_throttled");
+}
+
+static int move_non_contending(void)
+{
+	return moves(MV_NON_CONTENDING, "move_non_contending");
+}
+
+/*
+ * Fork a process that reads sched_rt_runtime_us and writes the same value
+ * back @n times, 5ms apart. Then it exits. Each write rebuilds the sched
+ * domains. Return its pid, or -1.
+ */
+static pid_t rebuilder(int n)
+{
+	pid_t pid = fork_child("a sysctl writer");
+
+	if (!pid) {
+		char buf[64];
+
+		if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)))
+			_exit(1);
+		while (n-- > 0) {
+			write_file("/proc/sys/kernel", "sched_rt_runtime_us", buf);
+			sleep_ns(5 * NSEC_PER_MSEC);
+		}
+		_exit(0);
+	}
+	return pid;
+}
+
+static void *pauser(void *arg)
+{
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * Run as the deadline thread of wide_mover(): a deadline task of 30% that
+ * runs 1ms, sets mv_state[0] to 1 and sleeps.
+ */
+static void *dl_pauser(void *arg)
+{
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1;
+		return arg;
+	}
+	spin_ns(NSEC_PER_MSEC);
+	*mv_state = 1;
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * Fork a process of @threads sleeping threads, then one deadline thread,
+ * dl_pauser(). A cgroup move of the process moves its threads one by one,
+ * between cpuset_can_attach() and cpuset_attach(). The deadline thread
+ * moves last, so that a rebuild in that window can still find it in the
+ * cgroup it leaves. Return the pid, or -1.
+ */
+static pid_t wide_mover(int threads)
+{
+	pid_t pid = fork_child("a wide deadline process");
+
+	if (!pid) {
+		pthread_attr_t a;
+		pthread_t t;
+		int i;
+
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			goto no_threads;
+		for (i = 0; i < threads; i++) {
+			if (pthread_create(&t, &a, pauser, NULL))
+				goto no_threads;
+		}
+		if (pthread_create(&t, NULL, dl_pauser, NULL))
+			goto no_threads;
+		for (;;)
+			pause();
+no_threads:
+		mv_state[0] = -2;
+		_exit(1);
+	}
+	return pid;
+}
+
+/*
+ * Move a process in and out of P while rebuilder() rebuilds the sched
+ * domains. The process is wide_mover(): 2048 sleeping threads, then one
+ * sleeping deadline thread of 30%. rebuilder() writes the
+ * sched_rt_runtime_us sysctl 20 times, 5ms apart. Each write is a
+ * rebuild. Without the 5ms gaps, the rebuilds would be over before the
+ * first move reaches the window below.
+ *
+ * A rebuild between cpuset_can_attach() and cpuset_attach() would drop
+ * the bandwidth that cpuset_can_attach() reserved in the destination.
+ * cpuset_attach() would then move the bandwidth of the thread all the
+ * same. So a rebuild must wait for the end of a move in progress.
+ *
+ * A later rebuild corrects the accounting. So move until rebuilder()
+ * exits: no rebuild follows the last move. The free bandwidth of P and
+ * of the other CPUs must then be what it was with the process asleep in
+ * the root cgroup. Run six rounds.
+ */
+static int move_vs_rebuild(void)
+{
+	int base_p, base_r, with_p, with_r, after_p, after_r, round, ret = KSFT_PASS;
+	const char *name = "move_vs_rebuild";
+	pid_t pid, writer;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	pid = wide_mover(2048);
+	if (pid < 0 || child_up(name, 0)) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	/* The reference for the rounds: the task asleep in the root cgroup. */
+	sleep_ns(2 * PERIOD);
+	with_p = probe(cg_p);
+	with_r = probe(root);
+	for (round = 0; round < 6 && ret == KSFT_PASS; round++) {
+		int status, err = 0;
+
+		writer = rebuilder(20);
+		if (writer < 0) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		while (!err && waitpid(writer, &status, WNOHANG) == 0) {
+			err = enter(cg_p, pid);
+			if (!err)
+				err = enter(root, pid);
+		}
+		if (err) {
+			ksft_print_msg("%s: a move refused: %s\n", name, strerror(-err));
+			reap(writer);
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(3 * PERIOD);
+		after_p = probe(cg_p);
+		after_r = probe(root);
+		if (after_p != with_p || after_r != with_r) {
+			ksft_print_msg("%s: round %d: P %d -> %d probe tasks, the rest %d -> %d\n",
+				       name, round, with_p, after_p, with_r, after_r);
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after_p = probe(cg_p);
+	after_r = probe(root);
+	if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+		ksft_print_msg("%s: task gone: P %d -> %d probe tasks, the rest %d -> %d\n",
+			       name, base_p, after_p, base_r, after_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* The runtime every 100ms of the deadline threads of late_mover() */
+static uint64_t late_thread_runtime;
+
+/*
+ * Run as a deadline thread of late_mover(): a deadline task of
+ * late_thread_runtime that counts itself in mv_state[1] and sleeps.
+ */
+static void *dl_sleeper(void *arg)
+{
+	if (set_deadline(0, late_thread_runtime, PERIOD)) {
+		mv_state[0] = -1;
+		return arg;
+	}
+	mv_state[1]++;
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * Fork a process of two deadline threads, dl_sleeper(), then @threads
+ * sleeping threads. Its main thread, the thread group leader, sleeps and
+ * is not a deadline task. A cgroup move of the process handles the
+ * leader first. The sleeping threads make the move long. Return the pid,
+ * or -1.
+ */
+static pid_t late_mover(int threads)
+{
+	pid_t pid = fork_child("a wide process");
+
+	if (!pid) {
+		pthread_attr_t a;
+		pthread_t t;
+		int i;
+
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			goto no_threads;
+		for (i = 0; i < 2; i++) {
+			if (pthread_create(&t, &a, dl_sleeper, NULL))
+				goto no_threads;
+		}
+		for (i = 0; i < threads; i++) {
+			if (pthread_create(&t, &a, pauser, NULL))
+				goto no_threads;
+		}
+		if (wait_for_step(5000, NSEC_PER_MSEC, mv_state[1] == 2 || mv_state[0]))
+			goto no_threads;
+		if (!mv_state[0])
+			mv_state[0] = 1;
+		for (;;)
+			pause();
+no_threads:
+		mv_state[0] = -2;
+		_exit(1);
+	}
+	return pid;
+}
+
+/*
+ * What late_setter(), late_pinner() and late_killer() do, and the result.
+ * Each acts on thread @pid, @delay ns after @go is set. It stores its
+ * result in @ret and the time it returned in @done. late_setter() uses
+ * @runtime. late_killer() uses @tgid and @kill.
+ */
+static struct {
+	pid_t pid;
+	pid_t tgid;
+	int kill;
+	uint64_t runtime;
+	uint64_t delay;
+	atomic_int go;
+	int ret;
+	uint64_t done;
+} late;
+
+/*
+ * Make late.pid a deadline task of late.runtime every 100ms, late.delay
+ * ns after late.go. Store the result in late.ret and the time it returned
+ * in late.done.
+ */
+static void *late_setter(void *arg)
+{
+	uint64_t start;
+
+	while (!late.go)
+		;
+	start = now_ns();
+	while (now_ns() - start < late.delay)
+		;
+	late.ret = set_deadline(late.pid, late.runtime, PERIOD);
+	late.done = now_ns();
+	return arg;
+}
+
+/*
+ * Make the main thread of a process a deadline task while the process
+ * moves from the root cgroup into P.
+ *
+ * The process is late_mover(). Its main thread sleeps and is not a
+ * deadline task. Two of its threads are sleeping deadline tasks of
+ * @thread percent each. 2048 more sleeping threads make the move long.
+ * A thread of the test, late_setter(), calls sched_setattr() on the main
+ * thread to make it a deadline task of @leader percent.
+ *
+ * cpuset_can_attach() reserves room in P for the two deadline threads.
+ * If the main thread became a deadline task after that, no room would be
+ * reserved for it. cpuset_attach() handles the main thread first.
+ *
+ * sched_setattr() waits for the end of a move in progress. So it runs
+ * before the move or after it:
+ *  - Before: the move counts three deadline tasks. With room in P for
+ *    the three, the move must succeed. P must then count the three, and
+ *    the root domain the process left must count none. Without room,
+ *    the move must be refused with EBUSY.
+ *  - After: sched_setattr() is refused with EPERM. The main thread is
+ *    then allowed on P's CPU only. It is still asleep on a CPU of the
+ *    root domain it left.
+ * Without room in P for the three, the case fails if the move and
+ * sched_setattr() both succeed.
+ *
+ * Try 30 delays of sched_setattr(), spread over the time a move takes.
+ */
+static int became_deadline(int thread, int leader, const char *name)
+{
+	int base_p, base_r, ret = KSFT_PASS, i, raced = 0;
+	int need = (2 * thread + leader) / 10, fits;
+	uint64_t start, len;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	fits = base_p >= need;
+	if (base_p < 2 * thread / 10) {
+		ksft_print_msg("%s: P admits %d probe tasks, %d needed\n", name, base_p,
+			       2 * thread / 10);
+		return KSFT_SKIP;
+	}
+	late_thread_runtime = thread * NSEC_PER_MSEC;
+	late.runtime = leader * NSEC_PER_MSEC;
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = late_mover(2048);
+	if (pid < 0 || child_up(name, 0)) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	/* Time one move, to spread the delays over it. */
+	start = now_ns();
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	len = (now_ns() - start) / 2;
+	sleep_ns(2 * PERIOD);
+
+	late.pid = pid;
+	for (i = 0; i < 30 && ret == KSFT_PASS; i++) {
+		pthread_t t;
+		int err;
+
+		late.go = 0;
+		late.delay = len * i / 30;
+		if (pthread_create(&t, NULL, late_setter, NULL)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		err = enter(cg_p, pid);
+		join(t);
+		if (err && (fits || err != -EBUSY || late.ret)) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		} else if (!err && !late.ret && !fits) {
+			ksft_print_msg("%s: the move and the change both succeeded, with no room in P\n",
+				       name);
+			ret = KSFT_FAIL;
+		} else if (!err && !late.ret) {
+			raced++;
+			if (admits_in(cg_p, base_p - need, "P, with the three tasks") ||
+			    admits_in(root, base_r, "the root domain the tasks left"))
+				ret = KSFT_FAIL;
+		} else if (late.ret && late.ret != -EPERM) {
+			ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-late.ret));
+			ret = KSFT_FAIL;
+		}
+		if (ret == KSFT_FAIL)
+			ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+				       name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+				       (unsigned long long)(len / NSEC_PER_USEC));
+		if ((!late.ret && set_other(pid)) || enter(root, pid)) {
+			ksft_print_msg("%s: the process cannot go back\n", name);
+			ret = KSFT_FAIL;
+		}
+		sleep_ns(2 * PERIOD);
+	}
+	if (ret == KSFT_PASS && fits && !raced)
+		ksft_print_msg("%s: no change succeeded with a move\n", name);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("%s: the free bandwidth changed\n", name);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Run it with room in P: two threads of 20% and a main thread of 30%. */
+static int move_became_deadline(void)
+{
+	return became_deadline(20, 30, "move_became_deadline");
+}
+
+/* Run it without room in P: two threads of 30% and a main thread of 40%. */
+static int move_became_deadline_no_room(void)
+{
+	return became_deadline(30, 40, "move_became_deadline_no_room");
+}
+
+static void no_op(int sig)
+{
+}
+
+/* The CPU that dl_sleeper_tid() first runs on, or -1 for any CPU */
+static int sleeper_cpu = -1;
+
+/*
+ * Run as the deadline thread of pinned_mover(): a deadline task of 30%.
+ * Store the tid in mv_state[3], set mv_state[0] to 1 and sleep. A signal
+ * wakes the thread up. It then sleeps again. If mv_state[4] is set, this
+ * thread exits instead, and the process stays.
+ *
+ * With sleeper_cpu set, first run 1ms pinned on that CPU, then restore
+ * the previous affinity. Do this before sched_setattr(): the affinity of
+ * a deadline task must include every CPU of its root domain.
+ */
+static void *dl_sleeper_tid(void *arg)
+{
+	if (sleeper_cpu >= 0) {
+		cpu_set_t all, one;
+
+		CPU_ZERO(&one);
+		CPU_SET(sleeper_cpu, &one);
+		if (sched_getaffinity(0, sizeof(all), &all) ||
+		    sched_setaffinity(0, sizeof(one), &one)) {
+			mv_state[0] = -2;
+			return arg;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		if (sched_setaffinity(0, sizeof(all), &all)) {
+			mv_state[0] = -2;
+			return arg;
+		}
+	}
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1;
+		return arg;
+	}
+	mv_state[3] = gettid_();
+	mv_state[0] = 1;
+	while (!mv_state[4])
+		pause();
+	syscall(SYS_exit, 0);
+	return arg;
+}
+
+/*
+ * Run as the last thread of pinned_mover() with @fifo. Become a
+ * SCHED_FIFO thread, set mv_state[5] to 1, or to -1 if SCHED_FIFO is
+ * refused, and sleep.
+ */
+static void *fifo_sleeper(void *arg)
+{
+	struct sched_param sp = { .sched_priority = 1 };
+
+	mv_state[5] = sched_setscheduler(0, SCHED_FIFO, &sp) ? -1 : 1;
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * Fork a process of @threads sleeping threads, then one deadline thread,
+ * dl_sleeper_tid(). A cgroup move of the process handles the deadline
+ * thread after the sleeping ones, which make the move long. With @fifo,
+ * add a SCHED_FIFO thread, fifo_sleeper(), after it. Return the pid, or
+ * -1.
+ */
+static pid_t pinned_mover(int threads, int fifo)
+{
+	pid_t pid = fork_child("a wide deadline process");
+
+	if (!pid) {
+		pthread_attr_t a;
+		pthread_t t;
+		int i;
+
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			goto no_threads;
+		for (i = 0; i < threads; i++) {
+			if (pthread_create(&t, &a, pauser, NULL))
+				goto no_threads;
+		}
+		signal(SIGUSR1, no_op);
+		if (pthread_create(&t, NULL, dl_sleeper_tid, NULL))
+			goto no_threads;
+		if (fifo && (wait_for(5000, mv_state[0]) ||
+			     pthread_create(&t, NULL, fifo_sleeper, NULL)))
+			goto no_threads;
+		for (;;)
+			pause();
+no_threads:
+		mv_state[0] = -2;
+		_exit(1);
+	}
+	return pid;
+}
+
+/*
+ * Restrict late.pid to the CPU of P and the CPU in @arg, late.delay ns
+ * after late.go.
+ */
+static void *late_pinner(void *arg)
+{
+	uint64_t start;
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(top_cpu(1), &set);
+	CPU_SET((long)arg, &set);
+	while (!late.go)
+		;
+	start = now_ns();
+	while (now_ns() - start < late.delay)
+		;
+	late.ret = sched_setaffinity(late.pid, sizeof(set), &set) ? -errno : 0;
+	late.done = now_ns();
+	return NULL;
+}
+
+/*
+ * Change the affinity of a sleeping deadline thread while its process
+ * moves from P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then a deadline
+ * thread of 30%. That thread went to sleep on a CPU of the root cgroup,
+ * and was moved into P asleep. So P counts it, and it is still on that
+ * CPU. A thread of the test, late_pinner(), calls sched_setaffinity()
+ * on it.
+ *
+ * cpuset_can_attach() reserves the bandwidth of the thread on one CPU of
+ * the root cgroup. The move records that CPU. It is the CPU the thread
+ * is on, if its affinity allows that CPU. If not, it is the first CPU of
+ * the root cgroup that its affinity allows.
+ *
+ * sched_setaffinity() restricts the thread to two CPUs: the CPU of P,
+ * and a CPU of the root cgroup that the thread is not on. The affinity
+ * of a deadline task must include every CPU of the root domain that
+ * counts it, so the mask needs the CPU of P. The move then records the
+ * second CPU.
+ *
+ * sched_setaffinity() waits for the end of a move in progress. So it
+ * runs before the move or after it:
+ *  - Before: the bandwidth is reserved on the CPU the move records. The
+ *    root domain of that CPU must count the thread once the move is
+ *    done, and P must no longer count it.
+ *  - After: the thread is counted in the root domain of the root
+ *    cgroup. The mask does not include every CPU of that root domain.
+ *    sched_setaffinity() is refused with EBUSY.
+ *
+ * Try 20 delays of sched_setaffinity(), spread over the time a move
+ * takes.
+ */
+static int move_affinity_changed(void)
+{
+	int with_p, with_r, ret = KSFT_PASS, i, raced = 0;
+	const char *name = "move_affinity_changed";
+	uint64_t start, len;
+	cpu_set_t all;
+	pid_t pid;
+
+	CPU_ZERO(&all);
+	for (i = 0; i < ncpus; i++)
+		CPU_SET(online[i], &all);
+	mv_state[0] = 0;
+	mv_state[3] = 0;
+	mv_state[4] = 0;
+	pid = pinned_mover(2048, 0);
+	if (pid < 0 || child_up(name, 0)) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	/* The reference: the thread asleep in the root cgroup. */
+	sleep_ns(2 * PERIOD);
+	with_p = probe(cg_p);
+	with_r = probe(root);
+	/* Time one move, to spread the delays over it. */
+	start = now_ns();
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	len = (now_ns() - start) / 2;
+
+	late.pid = mv_state[3];
+	for (i = 0; i < 20 && ret == KSFT_PASS; i++) {
+		pthread_t t;
+		long cpu;
+		int err;
+
+		if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid)) {
+			ksft_print_msg("%s: the process cannot go into P\n", name);
+			ret = KSFT_FAIL;
+			break;
+		}
+		/* Not the CPU the thread is on: the move would record that one. */
+		cpu = task_cpu_of(late.pid) == online[1] ? online[0] : online[1];
+		late.go = 0;
+		late.delay = len * i / 20;
+		if (pthread_create(&t, NULL, late_pinner, (void *)cpu)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		err = enter(root, pid);
+		join(t);
+		if (err) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		} else if (!late.ret) {
+			raced++;
+			sleep_ns(2 * PERIOD);
+			if (admits_in(cg_p, with_p, "P, which the thread left") ||
+			    admits_in(root, with_r, "the root domain of the thread")) {
+				ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+					       name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+					       (unsigned long long)(len / NSEC_PER_USEC));
+				ret = KSFT_FAIL;
+			}
+		} else if (late.ret && late.ret != -EBUSY) {
+			ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS && !raced)
+		ksft_print_msg("%s: no change succeeded\n", name);
+	reap(pid);
+	return ret;
+}
+
+/*
+ * Make a thread a deadline task while sched_setaffinity() runs on it, and
+ * move its process from P to the root cgroup meanwhile.
+ *
+ * sched_setaffinity() waits for the end of a move in progress when the
+ * task is a deadline task. It tests the policy first, with no lock. Here
+ * the thread T is SCHED_OTHER at that test:
+ *
+ *   sched_setaffinity(T)         sched_setattr(T)        echo PID > cgroup.procs
+ *
+ *   T is SCHED_OTHER: no wait
+ *                                T is a deadline task
+ *                                                        cpuset_can_attach()
+ *                                                          reserves room for T
+ *   changes the affinity of T
+ *                                                        cpuset_attach()
+ *                                                          moves T
+ *
+ * The new affinity is the CPU of P and one CPU of the root cgroup, which
+ * is not the CPU the room was reserved on. Once the move is done, the
+ * root domain of the root cgroup must count T, and P must not.
+ *
+ * Delay injection: the window after the test of the policy is a few
+ * instructions wide. The case only reaches it on a kernel that can make
+ * sched_setaffinity() sleep there. It passes without reaching it
+ * otherwise.
+ */
+static int move_affinity_became_deadline(void)
+{
+	const char *name = "move_affinity_became_deadline";
+	int with_p, with_r, ret = KSFT_PASS, i;
+	uint64_t start, len, delay_ms;
+	cpu_set_t all;
+	pid_t pid;
+
+	CPU_ZERO(&all);
+	for (i = 0; i < ncpus; i++)
+		CPU_SET(online[i], &all);
+	mv_state[0] = 0;
+	mv_state[3] = 0;
+	mv_state[4] = 0;
+	pid = pinned_mover(2048, 0);
+	if (pid < 0 || child_up(name, 0)) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	/* The reference: the thread a deadline task, asleep in the root cgroup. */
+	sleep_ns(2 * PERIOD);
+	with_p = probe(cg_p);
+	with_r = probe(root);
+	/* Time one move, to place the change inside it. */
+	start = now_ns();
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	len = (now_ns() - start) / 2;
+	delay_ms = 3 * len / NSEC_PER_MSEC + 30;
+
+	late.pid = mv_state[3];
+	for (i = 0; i < 10 && ret == KSFT_PASS; i++) {
+		uint64_t t0, at;
+		pthread_t t;
+		int err;
+
+		if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid) ||
+		    set_other(late.pid)) {
+			ksft_print_msg("%s: the thread cannot go into P as SCHED_OTHER\n", name);
+			ret = KSFT_FAIL;
+			break;
+		}
+		/*
+		 * Wake the thread once, so that it runs on the CPU of P. A
+		 * thread moved asleep is refused SCHED_DEADLINE with EPERM.
+		 */
+		if (syscall(SYS_tgkill, pid, late.pid, SIGUSR1)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(5 * NSEC_PER_MSEC);
+		/*
+		 * Delay injection: sched_setaffinity() would be asked here to
+		 * sleep delay_ms after its test of the policy.
+		 */
+		late.go = 0;
+		late.delay = 0;
+		if (pthread_create(&t, NULL, late_pinner, (void *)(long)online[1])) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		t0 = now_ns();
+		late.go = 1;
+		/* With the delay, the change has tested the policy and sleeps. */
+		sleep_ns(5 * NSEC_PER_MSEC);
+		err = set_deadline(late.pid, 30 * NSEC_PER_MSEC, PERIOD);
+		if (err) {
+			ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		}
+		/* Start the move so that the change wakes up inside it. */
+		at = t0 + delay_ms * NSEC_PER_MSEC - len * (i + 1) / 11;
+		while (now_ns() < at)
+			;
+		err = enter(root, pid);
+		join(t);
+		if (err) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		} else if (!late.ret) {
+			sleep_ns(2 * PERIOD);
+			if (admits_in(cg_p, with_p, "P, which the thread left") ||
+			    admits_in(root, with_r, "the root domain of the thread")) {
+				ksft_print_msg("%s: try %d, the change landed in a move of %llu us\n",
+					       name, i, (unsigned long long)(len / NSEC_PER_USEC));
+				ret = KSFT_FAIL;
+			}
+		} else if (late.ret && late.ret != -EBUSY) {
+			ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+/*
+ * Wake thread late.pid of process late.tgid up with a signal, late.delay
+ * ns after late.go. With late.kill, the thread then exits.
+ */
+static void *late_killer(void *arg)
+{
+	uint64_t start;
+
+	while (!late.go)
+		;
+	start = now_ns();
+	while (now_ns() - start < late.delay)
+		;
+	mv_state[4] = late.kill;
+	late.ret = syscall(SYS_tgkill, late.tgid, late.pid, SIGUSR1) ? -errno : 0;
+	late.done = now_ns();
+	return arg;
+}
+
+/*
+ * Kill a sleeping deadline thread of 30% while its process moves into
+ * P. A thread that is part of a move cannot exit before the move ends:
+ * its exit waits for it. So a thread killed during the move dies right
+ * after it, and must release its bandwidth where the move left it. Once the process
+ * is gone, the free bandwidth of P and of the root domain the process
+ * left must be what it was.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_died(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, i;
+	const char *name = "move_died";
+	uint64_t len = 0;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+		pthread_t t;
+		pid_t pid;
+		int err;
+
+		mv_state[0] = 0;
+		mv_state[3] = 0;
+		mv_state[4] = 0;
+		pid = pinned_mover(2048, 0);
+		if (pid < 0 || child_up(name, 0)) {
+			reap(pid);
+			return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+		}
+		if (!i) {
+			/* Time one move, to spread the delays over it. */
+			uint64_t start = now_ns();
+
+			if (enter(cg_p, pid) || enter(root, pid)) {
+				ksft_print_msg("%s: a move refused\n", name);
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			len = (now_ns() - start) / 2;
+		}
+		late.pid = mv_state[3];
+		late.tgid = pid;
+		late.kill = 1;
+		late.go = 0;
+		late.delay = len * i / 12;
+		if (pthread_create(&t, NULL, late_killer, NULL)) {
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		err = enter(cg_p, pid);
+		join(t);
+		if (err) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		}
+		reap(pid);
+		sleep_ns(3 * PERIOD);
+		if (ret == KSFT_PASS &&
+		    (admits_in(cg_p, base_p, "P, after the thread died") ||
+		     admits_in(root, base_r, "the root domain the process left"))) {
+			ksft_print_msg("%s: try %d, the death %llu us into a move of %llu us\n",
+				       name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+				       (unsigned long long)(len / NSEC_PER_USEC));
+			ret = KSFT_FAIL;
+		}
+	}
+	return ret;
+}
+
+static int late_sysrq_fd;
+
+/*
+ * Write 'n' to /proc/sysrq-trigger, late.delay ns after late.go. sysrq-n
+ * makes every real-time and deadline task SCHED_OTHER.
+ */
+static void *late_sysrq(void *arg)
+{
+	uint64_t start;
+
+	while (!late.go)
+		;
+	start = now_ns();
+	while (now_ns() - start < late.delay)
+		;
+	late.ret = write(late_sysrq_fd, "n", 1) == 1 ? 0 : -errno;
+	late.done = now_ns();
+	return arg;
+}
+
+/*
+ * Make sysrq-n take a sleeping deadline thread of 30% out of the class
+ * while its process moves into P. cpuset_can_attach() reserves room in P
+ * for the thread. sysrq-n does not wait for the move. If it comes after
+ * the reservation, the thread moves no bandwidth, and cpuset_attach()
+ * must give the room back. Once the process is gone, the free bandwidth of P and of
+ * the root domain the process left must be what it was.
+ *
+ * sysrq-n changes the whole system. So the case runs last of the cases
+ * with P, and only with DEADLINE_TEST_SYSRQ set.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_normalized(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, i;
+	const char *name = "move_normalized";
+	uint64_t len = 0;
+
+	if (!getenv("DEADLINE_TEST_SYSRQ")) {
+		ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+		return KSFT_SKIP;
+	}
+	late_sysrq_fd = open("/proc/sysrq-trigger", O_WRONLY);
+	if (late_sysrq_fd < 0) {
+		ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+		pthread_t t;
+		pid_t pid;
+		int err;
+
+		mv_state[0] = 0;
+		mv_state[3] = 0;
+		mv_state[4] = 0;
+		pid = pinned_mover(2048, 0);
+		if (pid < 0 || child_up(name, 0)) {
+			reap(pid);
+			close(late_sysrq_fd);
+			return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+		}
+		if (!i) {
+			/* Time one move, to spread the delays over it. */
+			uint64_t start = now_ns();
+
+			if (enter(cg_p, pid) || enter(root, pid)) {
+				ksft_print_msg("%s: a move refused\n", name);
+				reap(pid);
+				close(late_sysrq_fd);
+				return KSFT_FAIL;
+			}
+			len = (now_ns() - start) / 2;
+		}
+		late.go = 0;
+		late.delay = len * i / 12;
+		if (pthread_create(&t, NULL, late_sysrq, NULL)) {
+			reap(pid);
+			close(late_sysrq_fd);
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		err = enter(cg_p, pid);
+		join(t);
+		if (err) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		} else if (late.ret) {
+			ksft_print_msg("%s: sysrq-n: %s\n", name, strerror(-late.ret));
+			ret = KSFT_SKIP;
+		}
+		reap(pid);
+		sleep_ns(3 * PERIOD);
+		if (ret == KSFT_PASS &&
+		    (admits_in(cg_p, base_p, "P, after sysrq-n") ||
+		     admits_in(root, base_r, "the root domain the process left"))) {
+			ksft_print_msg("%s: try %d, sysrq-n %llu us into a move of %llu us\n",
+				       name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+				       (unsigned long long)(len / NSEC_PER_USEC));
+			ret = KSFT_FAIL;
+		}
+	}
+	close(late_sysrq_fd);
+	return ret;
+}
+
+static int hold(int n);
+static void release_held(void);
+
+/*
+ * Wake a sleeping deadline thread of 30% up while its process moves from
+ * P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then the deadline
+ * thread. That thread first ran on the second CPU of the root cgroup,
+ * went to sleep there, and was moved into P asleep. So P counts it, and
+ * it is still on that CPU. A thread of the test, late_killer(), wakes it
+ * up with a signal.
+ *
+ * cpuset_can_attach() reserves room for the thread on the CPU that the
+ * move back records: the CPU the thread is on. If the thread wakes up
+ * after that, it runs on the CPU of P. cpuset_attach() must still move
+ * the bandwidth to the CPU recorded. Working the CPU out again would
+ * give the first CPU of the root cgroup, where nothing is reserved.
+ *
+ * Both CPUs are in the same root domain. So that root domain must count
+ * the thread once the move is done, and P must no longer count it.
+ *
+ * With @full, hold() first fills that root domain with deadline threads
+ * of the test, until 40% is free. The thread of 30% then fits once, and
+ * not twice.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int woken(int full, const char *name)
+{
+	int base_p, base_r, free_r, ret = KSFT_PASS, i;
+	uint64_t len = 0;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	if (base_r < 5) {
+		ksft_print_msg("%s: no room for the thread\n", name);
+		return KSFT_SKIP;
+	}
+	free_r = base_r;
+	if (full) {
+		if (hold(base_r - 4)) {
+			release_held();
+			return KSFT_FAIL;
+		}
+		free_r = 4;
+	}
+	for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+		pid_t pid, tid;
+		pthread_t t;
+		int err;
+
+		mv_state[0] = 0;
+		mv_state[3] = 0;
+		mv_state[4] = 0;
+		/* Not the first CPU: the move falls back to that one. */
+		sleeper_cpu = online[1];
+		pid = pinned_mover(2048, 0);
+		sleeper_cpu = -1;
+		if (pid < 0 || child_up(name, 0)) {
+			reap(pid);
+			release_held();
+			return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+		}
+		tid = mv_state[3];
+		sleep_ns(PERIOD);
+		if (!i) {
+			/*
+			 * Time the move into P, to spread the delays over
+			 * a move.
+			 */
+			uint64_t start = now_ns();
+
+			err = enter(cg_p, pid);
+			if (!err) {
+				len = now_ns() - start;
+				err = enter(root, pid);
+			}
+			if (err) {
+				ksft_print_msg("%s: a move refused\n", name);
+				reap(pid);
+				release_held();
+				return KSFT_FAIL;
+			}
+		}
+		if (enter(cg_p, pid)) {
+			ksft_print_msg("%s: the process cannot go into P\n", name);
+			reap(pid);
+			release_held();
+			return KSFT_FAIL;
+		}
+		if (task_cpu_of(tid) != online[1]) {
+			ksft_print_msg("%s: the thread is on CPU %d, not on CPU %d\n", name,
+				       task_cpu_of(tid), online[1]);
+			reap(pid);
+			release_held();
+			return KSFT_SKIP;
+		}
+		late.pid = tid;
+		late.tgid = pid;
+		late.kill = 0;
+		late.go = 0;
+		late.delay = len * i / 12;
+		if (pthread_create(&t, NULL, late_killer, NULL)) {
+			reap(pid);
+			release_held();
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		err = enter(root, pid);
+		join(t);
+		if (err) {
+			ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+			ret = KSFT_FAIL;
+		}
+		sleep_ns(2 * PERIOD);
+		if (ret == KSFT_PASS &&
+		    (admits_in(cg_p, base_p, "P, which the thread left") ||
+		     admits_in(root, free_r - 3, "the root domain of the thread"))) {
+			ksft_print_msg("%s: try %d, the wakeup %llu us into a move of %llu us\n",
+				       name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+				       (unsigned long long)(len / NSEC_PER_USEC));
+			ret = KSFT_FAIL;
+		}
+		reap(pid);
+		sleep_ns(3 * PERIOD);
+	}
+	release_held();
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, base_p, "P, at the end") ||
+	     admits_in(root, base_r, "the root domain of the rest, at the end")))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+static int move_woken(void)
+{
+	return woken(0, "move_woken");
+}
+
+static int move_woken_full(void)
+{
+	return woken(1, "move_woken_full");
+}
+
+/*
+ * Have the cpu controller refuse a move after cpuset_can_attach()
+ * reserved room. cpuset_cancel_attach() must then give the room back.
+ *
+ * The process is pinned_mover() with a SCHED_FIFO thread: 2048 sleeping
+ * threads, a sleeping deadline thread of 30%, and the SCHED_FIFO thread.
+ * It moves from the root cgroup into P. cpuset_can_attach() reserves
+ * room in P for the deadline thread. With CONFIG_RT_GROUP_SCHED, the cpu
+ * controller then refuses the SCHED_FIFO thread: P has no real-time
+ * runtime.
+ *
+ * Then a thread of the test, late_setter(), changes the runtime of the
+ * deadline thread during 20 such moves. sched_setattr() waits for the
+ * end of a move in progress. It must return, and succeed, once the move
+ * is cancelled.
+ *
+ * Skip if the cpu controller accepts the move.
+ */
+static int move_cancelled(void)
+{
+	int with_p, with_r, ret = KSFT_PASS, i, err;
+	const char *name = "move_cancelled";
+	uint64_t start, len;
+	pid_t pid;
+
+	mv_state[0] = 0;
+	mv_state[3] = 0;
+	mv_state[4] = 0;
+	mv_state[5] = 0;
+	pid = pinned_mover(2048, 1);
+	if (pid < 0 || child_up(name, 0) || wait_for(5000, mv_state[5])) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	if (mv_state[5] < 0 || write_file(root, "cgroup.subtree_control", "+cpu")) {
+		ksft_print_msg("%s: no SCHED_FIFO thread or no cpu controller\n", name);
+		reap(pid);
+		return KSFT_SKIP;
+	}
+	/* The reference: the deadline thread asleep in the root cgroup. */
+	sleep_ns(2 * PERIOD);
+	with_p = probe(cg_p);
+	with_r = probe(root);
+	start = now_ns();
+	err = enter(cg_p, pid);
+	len = now_ns() - start;
+	if (!err) {
+		ksft_print_msg("%s: the cpu controller accepts a SCHED_FIFO thread: needs CONFIG_RT_GROUP_SCHED\n",
+			       name);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	for (i = 0; i < 5; i++) {
+		if (!enter(cg_p, pid)) {
+			ksft_print_msg("%s: a move accepted\n", name);
+			ret = KSFT_FAIL;
+			goto out;
+		}
+	}
+	sleep_ns(2 * PERIOD);
+	if (admits_in(cg_p, with_p, "P, after cancelled moves") ||
+	    admits_in(root, with_r, "the root domain of the process")) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+
+	late.pid = mv_state[3];
+	for (i = 0; i < 20 && ret == KSFT_PASS; i++) {
+		pthread_t t;
+
+		late.go = 0;
+		late.done = 0;
+		late.runtime = (i % 2 ? 30 : 20) * NSEC_PER_MSEC;
+		late.delay = len * i / 20;
+		if (pthread_create(&t, NULL, late_setter, NULL)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+		late.go = 1;
+		if (!enter(cg_p, pid)) {
+			ksft_print_msg("%s: a move accepted\n", name);
+			ret = KSFT_FAIL;
+		}
+		if (wait_for(5000, late.done)) {
+			ksft_print_msg("%s: sched_setattr() still waits after a cancelled move\n",
+				       name);
+			ret = KSFT_FAIL;
+			break;	/* the thread cannot be joined */
+		}
+		join(t);
+		if (late.ret) {
+			ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-late.ret));
+			ret = KSFT_FAIL;
+		}
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, with_p, "P, after the cancelled moves") ||
+	     admits_in(root, with_r, "the root domain of the process")))
+		ret = KSFT_FAIL;
+out:
+	reap(pid);
+	if (write_file(root, "cgroup.subtree_control", "-cpu"))
+		ksft_print_msg("%s: the cpu controller stays enabled\n", name);
+	return ret;
+}
+
+/* Return whether every task of @cg is frozen. */
+static int cg_frozen(const char *cg)
+{
+	char path[PATH_MAX + 32], buf[256];
+
+	snprintf(path, sizeof(path), "%s/cgroup.events", cg);
+	return !read_file(path, buf, sizeof(buf)) && strstr(buf, "frozen 1");
+}
+
+/*
+ * Freeze P with a running deadline task of 30% in it. Move the frozen
+ * task to the root cgroup, switch it to SCHED_OTHER, thaw P and kill the
+ * task. Do it three times. The free bandwidth of P and of the root
+ * domain of the other CPUs must then be what it was.
+ *
+ * A frozen task sleeps in its signal handling. The root cgroup is not
+ * frozen, so the move thaws the task. It is woken on P's CPU, before
+ * cpuset_attach() gives it the CPUs of the root cgroup.
+ */
+static int frozen_moved_switched(void)
+{
+	const char *what = "a deadline task to freeze";
+	int base_p, base_r, ret = KSFT_PASS, i;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	for (i = 0; i < 3 && ret == KSFT_PASS; i++) {
+		pid_t pid;
+
+		mv_state[0] = 0;
+		pid = fork_child(what);
+		if (!pid) {
+			if (enter(cg_p, getpid()) ||
+			    set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+				mv_state[0] = -1;
+				_exit(1);
+			}
+			mv_state[0] = 1;
+			for (;;)
+				spin_ns(NSEC_PER_MSEC);
+		}
+		if (pid < 0 || child_up(what, 0)) {
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		if (write_file(cg_p, "cgroup.freeze", "1")) {
+			ksft_print_msg("P cannot be frozen\n");
+			reap(pid);
+			return KSFT_SKIP;
+		}
+		if (wait_for(2000, cg_frozen(cg_p))) {
+			ksft_print_msg("P does not freeze\n");
+			ret = KSFT_FAIL;
+		} else if (enter(root, pid)) {
+			ksft_print_msg("the frozen task cannot leave P\n");
+			ret = KSFT_FAIL;
+		} else if (set_other(pid)) {
+			ksft_print_msg("the task cannot leave SCHED_DEADLINE\n");
+			ret = KSFT_FAIL;
+		}
+		if (write_file(cg_p, "cgroup.freeze", "0"))
+			ksft_print_msg("P cannot be thawed\n");
+		reap(pid);
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, base_p, "P, after the frozen tasks") ||
+	     admits_in(root, base_r, "the root domain of the rest")))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* Deaths, in P */
+
+enum die_mode { DIE_RUNNING, DIE_SWITCHED_OUT, DIE_SHRUNK };
+
+/*
+ * Fork a deadline task of 30% that runs 5ms of a fresh period and exits
+ * before its 0-lag time. @mode says what it does before it exits:
+ *  - DIE_RUNNING: nothing.
+ *  - DIE_SWITCHED_OUT: switch to SCHED_OTHER.
+ *  - DIE_SHRUNK: shrink to 10%. Its 0-lag time is then in the past.
+ *
+ * Its bandwidth is released at the 0-lag time, or at the exit for
+ * DIE_SHRUNK.
+ */
+static int die_case(enum die_mode mode, const char *what)
+{
+	pid_t pid = fork_child("a task that dies");
+	int status;
+
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		sleep_ns(PERIOD);
+		spin_ns(5 * NSEC_PER_MSEC);
+		if (mode == DIE_SWITCHED_OUT && set_other(0))
+			_exit(2);
+		if (mode == DIE_SHRUNK && set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+			_exit(3);
+		_exit(0);
+	}
+	if (pid < 0 || waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status)) {
+		ksft_print_msg("a task that dies %s failed\n", what);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+static int death_running(void)
+{
+	return die_case(DIE_RUNNING, "running");
+}
+
+static int death_switched_out(void)
+{
+	return die_case(DIE_SWITCHED_OUT, "switched out");
+}
+
+static int death_shrunk(void)
+{
+	return die_case(DIE_SHRUNK, "shrunk");
+}
+
+/*
+ * Stop a running deadline task of 30% with SIGSTOP, switch it to
+ * SCHED_OTHER while it is stopped, and kill it. The task is on no
+ * runqueue when it leaves SCHED_DEADLINE. Do it five times.
+ */
+static int stopped_switched_out(void)
+{
+	const char *what = "a deadline task to stop";
+	int i, err;
+
+	for (i = 0; i < 5; i++) {
+		pid_t pid;
+
+		mv_state[0] = 0;
+		pid = fork_child(what);
+		if (!pid) {
+			if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+				mv_state[0] = -1;
+				_exit(1);
+			}
+			mv_state[0] = 1;
+			for (;;)
+				spin_ns(NSEC_PER_MSEC);
+		}
+		if (pid < 0 || child_up(what, 0)) {
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		err = 0;
+		if (kill(pid, SIGSTOP) || wait_for(1000, proc_state_of(pid) == 'T')) {
+			ksft_print_msg("the deadline task did not stop\n");
+			err = 1;
+		} else if (set_other(pid)) {
+			ksft_print_msg("the stopped task cannot leave SCHED_DEADLINE\n");
+			err = 1;
+		}
+		reap(pid);
+		if (err)
+			return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/*
+ * With @moving, wait until the spinner's count in mv_state[1] moves: it
+ * runs. Without, wait until the count stops for 5ms: it is throttled.
+ * Return 0, or -1 after 1s.
+ */
+static int wait_spinner(int moving)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+	int last = mv_state[1];
+
+	while (now_ns() < end) {
+		sleep_ns(moving ? NSEC_PER_MSEC : 5 * NSEC_PER_MSEC);
+		if (moving ? mv_state[1] != last : mv_state[1] == last)
+			return 0;
+		last = mv_state[1];
+	}
+	return -1;
+}
+
+enum kill_when { KILL_AFTER, KILL_THROTTLED, KILL_RUNNING };
+
+/*
+ * Start a deadline task of 30% with mover() in @mode, then kill it as
+ * @when says:
+ *  - KILL_AFTER: @after ns after it is up.
+ *  - KILL_THROTTLED: once it is throttled.
+ *  - KILL_RUNNING: while it runs.
+ *
+ * The killer first leaves P. It then watches and kills from another CPU,
+ * and does not take the task's CPU.
+ */
+static int kill_case(enum mv_mode mode, enum kill_when when, uint64_t after, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t pid;
+
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (enter(root, getpid())) {
+		ksft_print_msg("killed %s: the killer cannot leave P\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (child_up(what, 1)) {
+		ret = KSFT_FAIL;
+	} else if (when != KILL_AFTER && wait_spinner(when == KILL_RUNNING)) {
+		ksft_print_msg("killed %s: never %s\n", what,
+			       when == KILL_RUNNING ? "ran" : "throttled");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(after);
+	}
+out:
+	reap(pid);
+	return ret;
+}
+
+static int death_killed_asleep(void)
+{
+	return kill_case(MV_ASLEEP, KILL_AFTER, 2 * PERIOD, "asleep");
+}
+
+static int death_killed_non_contending(void)
+{
+	return kill_case(MV_NON_CONTENDING, KILL_AFTER, 0, "before its 0-lag time");
+}
+
+static int death_killed_throttled(void)
+{
+	return kill_case(MV_THROTTLED, KILL_THROTTLED, 0, "throttled");
+}
+
+static int death_killed_running(void)
+{
+	return kill_case(MV_THROTTLED, KILL_RUNNING, 0, "running");
+}
+
+/*
+ * A robust priority inheritance lock shared between processes.
+ * mv_state[0] holds the state of its owner, mv_state[2] that of its
+ * waiter.
+ */
+static pthread_mutex_t *shm_lock;
+
+/* Set up shm_lock. Return 0, or -1 if it cannot be set up. */
+static int shm_lock_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err)
+		err = pthread_mutexattr_setpshared(&a, PTHREAD_PROCESS_SHARED);
+	if (!err)
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+	if (!err)
+		err = pthread_mutexattr_setrobust(&a, PTHREAD_MUTEX_ROBUST);
+	if (!err)
+		err = pthread_mutex_init(shm_lock, &a);
+	if (err) {
+		ksft_print_msg("no shared priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	mv_state[0] = 0;
+	mv_state[2] = 0;
+	return 0;
+}
+
+/*
+ * Fork a process that takes shm_lock and sleeps with it, as a deadline
+ * task of 30% if @dl. Return its pid, or -1.
+ */
+static pid_t lock_owner(int dl)
+{
+	pid_t pid = fork_child("the owner of the lock");
+
+	if (!pid) {
+		if ((dl && set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) ||
+		    pthread_mutex_lock(shm_lock)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * Fork a deadline task of 5ms every 50ms that waits for shm_lock. It sets
+ * mv_state[2] to 1 before the wait, and to 2 once it got the lock. Return
+ * its pid, or -1.
+ */
+static pid_t lock_waiter(void)
+{
+	pid_t pid = fork_child("a waiter of the lock");
+
+	if (!pid) {
+		int err;
+
+		if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+			mv_state[2] = -1;
+			_exit(1);
+		}
+		mv_state[2] = 1;
+		err = pthread_mutex_lock(shm_lock);
+		if (err == EOWNERDEAD)
+			err = pthread_mutex_consistent(shm_lock);
+		if (!err)
+			pthread_mutex_unlock(shm_lock);
+		mv_state[2] = err ? -2 : 2;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/* Wait up to 1s for @var to reach @want. Return 0, or -1. */
+static int wait_state(atomic_int *var, int want)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+
+	while (now_ns() < end) {
+		if (*var == want)
+			return 0;
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+/*
+ * Kill one side of a priority inheritance boost. lock_owner() sleeps
+ * with shm_lock, as a deadline task of 30% if @dl_owner. lock_waiter(),
+ * a deadline task of 5ms every 50ms, blocks on the lock and boosts the
+ * owner. Kill the owner if @kill_owner: the waiter must then get the
+ * lock. If not, kill the waiter.
+ *
+ * A killed owner does not die boosted. Its exit first releases its
+ * priority inheritance futexes in futex_exit_exec_release(), which
+ * deboosts it.
+ */
+static int pi_death(int dl_owner, int kill_owner, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t owner, waiter = -1;
+
+	if (shm_lock_init())
+		return KSFT_SKIP;
+	owner = lock_owner(dl_owner);
+	if (owner < 0)
+		return KSFT_FAIL;
+	if (wait_state(&mv_state[0], 1)) {
+		ksft_print_msg("%s: no owner\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	waiter = lock_waiter();
+	if (waiter < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_state(&mv_state[2], 1)) {
+		ksft_print_msg("%s: no waiter\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);	/* blocked, boosting the owner */
+	if (kill_owner) {
+		reap(owner);
+		owner = -1;
+		if (wait_state(&mv_state[2], 2)) {
+			ksft_print_msg("%s: the waiter did not get the lock\n", what);
+			ret = KSFT_FAIL;
+		}
+	} else {
+		reap(waiter);
+		waiter = -1;
+		sleep_ns(50 * NSEC_PER_MSEC);	/* the owner deboosted asleep */
+	}
+out:
+	reap(waiter);
+	reap(owner);
+	return ret;
+}
+
+static int death_pi_waiter(void)
+{
+	return pi_death(0, 0, "a waiter killed");
+}
+
+static int death_pi_owner(void)
+{
+	return pi_death(0, 1, "a boosted owner killed");
+}
+
+static int death_pi_deadline_owner(void)
+{
+	return pi_death(1, 1, "a boosted deadline owner killed");
+}
+
+/*
+ * Run as thread @arg of death_group(), a deadline task with a period of
+ * 100ms. Each thread counts itself in mv_state[0] once it has run 1ms.
+ *  - Thread 0, of 10ms: run 1ms, then sleep.
+ *  - Thread 1, of 10ms: spin and count in mv_state[1], until throttled.
+ *  - Thread 2, of 60ms: wait for the killer to set mv_state[2]. Then
+ *    spin and count in mv_state[3].
+ */
+static void *group_thread(void *arg)
+{
+	long kind = (long)arg;
+
+	if (set_deadline(0, (kind == 2 ? 60 : 10) * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1000;
+		return NULL;
+	}
+	if (kind == 2)
+		while (!mv_state[2])
+			sleep_ns(NSEC_PER_MSEC);
+	spin_ns(NSEC_PER_MSEC);
+	mv_state[0]++;
+	if (!kind)
+		for (;;)
+			pause();
+	for (;;)
+		mv_state[kind == 2 ? 3 : 1]++;
+	return NULL;
+}
+
+/*
+ * Kill a process of three deadline threads with one SIGKILL, while one
+ * sleeps, one is throttled and one runs. See group_thread().
+ */
+static int death_group(void)
+{
+	int last, ret = KSFT_PASS;
+	pid_t pid;
+
+	mv_state[0] = mv_state[1] = mv_state[2] = mv_state[3] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (!pid) {
+		pthread_t t;
+		long i;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, group_thread, (void *)i))
+				mv_state[0] = -1000;
+		for (;;)
+			pause();
+	}
+	/*
+	 * Kill from outside P. P has one CPU: in P, the killer could not
+	 * run while thread 2 runs.
+	 */
+	if (enter(root, getpid())) {
+		ksft_print_msg("the killer cannot leave P\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_for(5000, mv_state[0] < 0 || mv_state[0] == 2) || mv_state[0] != 2) {
+		ksft_print_msg("only %d threads of the group started\n", (int)mv_state[0]);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_spinner(0)) {
+		ksft_print_msg("the first spinner never throttled\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	last = mv_state[3];
+	mv_state[2] = 1;
+	if (wait_for(5000, mv_state[0] < 0 || mv_state[3] != last) || mv_state[0] != 3) {
+		ksft_print_msg("the second spinner never ran\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	/* One asleep, one throttled, one running. */
+	reap(pid);
+	return ret;
+}
+
+/*
+ * Two deadline tasks are in P. One, of 30%, sleeps. The other, of 10%,
+ * exits and stays a zombie. Switching the zombie to SCHED_OTHER must be
+ * refused with ESRCH.
+ *
+ * On kernels where each cpuset counts its deadline tasks,
+ * cgroup_task_dead() took the zombie off P's count. The switch would
+ * run switched_from_dl() and take it off a second time. P's count would
+ * then be 0 with the task of 30% still in P. A rebuild of the accounting
+ * would skip P, and lose the bandwidth of that task. So rebuild the
+ * accounting, and check that P still counts the 30%.
+ */
+static int death_zombie_switched(void)
+{
+	int base, ret = KSFT_PASS, err;
+	pid_t in_p, zombie;
+
+	base = probe(cg_p);
+	if (base < 4) {
+		ksft_print_msg("no room in P for a task of 30%% and a probe\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 30% in P", 0)) {
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	zombie = fork_child("a deadline task that exits");
+	if (!zombie) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		_exit(0);
+	}
+	if (zombie < 0 || child_up("a task of 10% in P that exits", 0) ||
+	    wait_for(1000, proc_state_of(zombie) == 'Z')) {
+		ksft_print_msg("no zombie deadline task\n");
+		reap(zombie);
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	/* Its bandwidth is released at its 0-lag time, within a period. */
+	sleep_ns(3 * PERIOD);
+	err = set_other(zombie);
+	if (err != -ESRCH) {
+		ksft_print_msg("SCHED_OTHER on a zombie deadline task: %s, not ESRCH\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	/*
+	 * Rebuild the accounting: make Q beside P, then remove it. Q is a
+	 * second partition root, of the CPU below P's.
+	 */
+	if (make_partition(cg_q, top_cpu(2))) {
+		ksft_print_msg("no partition root Q beside P\n");
+		ret = KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	if (admits_in(cg_p, base - 3, "P rebuilt, a task of 30% and a zombie in it"))
+		ret = KSFT_FAIL;
+	reap(zombie);
+	reap(in_p);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits_in(cg_p, base, "P, its tasks gone"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* sched_yield() */
+
+static atomic_int y_rounds;	/* the yields of the yielder */
+
+/*
+ * Run as the thread of start_yielder(): a deadline task of 30% that
+ * yields after 0.5ms of every period. sched_yield() throttles it until
+ * its next period, with runtime left. So it is throttled most of the
+ * time. t_state is 2 while it is in the yield.
+ */
+static void *yielder(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop) {
+		spin_ns(NSEC_PER_MSEC / 2);
+		y_rounds++;
+		t_state = 2;
+		sched_yield();
+		t_state = 1;
+	}
+	return NULL;
+}
+
+static int start_yielder(pthread_t *t)
+{
+	t_state = 0;
+	spin_stop = 0;
+	y_rounds = 0;
+	if (pthread_create(t, NULL, yielder, NULL)) {
+		ksft_print_msg("the yielder thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state > 0)
+		return 0;
+	spin_stop = 1;		/* in case it gets there late */
+	join(*t);
+	return -1;
+}
+
+/*
+ * Switch a deadline task to SCHED_OTHER while sched_yield() throttles it.
+ * Its bandwidth must come back, and it must run again.
+ */
+static int yield_switched_out(void)
+{
+	int base, ret = KSFT_PASS;
+	pthread_t t;
+
+	base = probe(NULL);
+	if (base < 3) {
+		ksft_print_msg("no room for a task of 30%%\n");
+		return KSFT_SKIP;
+	}
+	if (start_yielder(&t))
+		return KSFT_FAIL;
+	if (wait_throttled(t) || t_state != 2) {
+		ksft_print_msg("the task never yielded\n");
+		ret = KSFT_FAIL;
+	} else if (set_other(t_tid)) {
+		ret = KSFT_FAIL;
+	} else {
+		int last = y_rounds;
+
+		sleep_ns(2 * PERIOD);
+		if (y_rounds == last) {
+			ksft_print_msg("yielded, then switched out: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "yielded, then switched out"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+static int yield_moved(void)
+{
+	return moves(MV_YIELDED, "yield_moved");
+}
+
+/* Kill a deadline task while sched_yield() throttles it. */
+static int yield_killed(void)
+{
+	pid_t pid;
+
+	mv_state[0] = 0;
+	pid = fork_child("a task that yields");
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			spin_ns(NSEC_PER_MSEC / 2);
+			sched_yield();
+		}
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a task that yields", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	reap(pid);
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_DL_OVERRUN */
+
+static atomic_int got_xcpu;
+
+static void catch_xcpu(int sig)
+{
+	got_xcpu = 1;
+}
+
+/*
+ * Run as the thread of overrun_caught(): a deadline task of 10% with
+ * SCHED_FLAG_DL_OVERRUN. Run 15ms, past the runtime, then sleep until
+ * spin_stop is set.
+ */
+static void *overruns(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD, SCHED_FLAG_DL_OVERRUN)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	spin_ns(15 * NSEC_PER_MSEC);
+	while (!spin_stop)
+		sleep_ns(10 * NSEC_PER_MSEC);
+	return NULL;
+}
+
+/*
+ * When a task with SCHED_FLAG_DL_OVERRUN overruns its runtime, its thread
+ * group gets SIGXCPU. Catch it here. The task's bandwidth must stay
+ * reserved after the overrun, and come back when the task leaves
+ * SCHED_DEADLINE.
+ */
+static int overrun_caught(void)
+{
+	int base, ret = KSFT_PASS;
+	uint64_t end;
+	pthread_t t;
+
+	if (signal(SIGXCPU, catch_xcpu) == SIG_ERR)
+		return KSFT_FAIL;
+	base = probe(NULL);
+	if (base < 1) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	got_xcpu = 0;
+	t_state = 0;
+	spin_stop = 0;
+	if (pthread_create(&t, NULL, overruns, NULL))
+		return KSFT_FAIL;
+	while (!t_state)
+		sleep_ns(NSEC_PER_MSEC);
+	end = now_ns() + 2 * NSEC_PER_SEC;
+	while (!got_xcpu && now_ns() < end)
+		sleep_ns(5 * NSEC_PER_MSEC);
+	if (t_state != 1) {
+		ret = KSFT_FAIL;
+	} else if (!got_xcpu) {
+		ksft_print_msg("it overran its runtime, no SIGXCPU in 2s\n");
+		ret = KSFT_FAIL;
+	} else if (admits(base - 1, "an overrun told")) {
+		ret = KSFT_FAIL;	/* the overrun gave its 10% back */
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "after an overrun was told"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * Run as thread @arg of overrun_group_death(), a deadline task. Thread 0
+ * overruns its runtime. Threads 1 and 2 count themselves in mv_state[0],
+ * and sleep.
+ */
+static void *og_thread(void *arg)
+{
+	switch ((int)(long)arg) {
+	case 0:		/* overruns its runtime: SIGXCPU to the group */
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_DL_OVERRUN))
+			_exit(1);
+		/*
+		 * Wait for the other two first. On P's one CPU, this
+		 * thread has the earliest deadline of the three. It would
+		 * overrun, and kill them, before they reach the state the
+		 * signal must find them in.
+		 */
+		if (wait_for(5000, mv_state[0] >= 2))
+			_exit(2);
+		spin_ns(30 * NSEC_PER_MSEC);
+		break;
+	case 1:		/* asleep, its 0-lag time past */
+		if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		spin_ns(NSEC_PER_MSEC);
+		mv_state[0]++;
+		break;
+	case 2:		/* asleep, mostly before its 0-lag time */
+		if (set_deadline(0, 15 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			spin_ns(3 * NSEC_PER_MSEC);
+			sleep_ns(22 * NSEC_PER_MSEC);
+		}
+	}
+	for (;;)
+		pause();
+}
+
+/*
+ * Let the SIGXCPU of an overrun kill the whole thread group. The
+ * bandwidth of its three deadline threads must come back.
+ */
+static int overrun_group_death(void)
+{
+	pid_t pid;
+	int i, status;
+
+	mv_state[0] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (!pid) {
+		pthread_t t;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, og_thread, (void *)(long)i))
+				_exit(3);
+		for (;;)
+			pause();
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	for (i = 0; i < 300; i++) {
+		if (waitpid(pid, &status, WNOHANG) == pid)
+			break;
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+	if (i == 300) {
+		reap(pid);
+		ksft_print_msg("an overrun of 10ms every 100ms: no SIGXCPU in 3s\n");
+		return KSFT_FAIL;
+	}
+	if (WIFEXITED(status)) {
+		switch (WEXITSTATUS(status)) {
+		case 1:
+			ksft_print_msg("a thread of the group was refused SCHED_DEADLINE\n");
+			return KSFT_SKIP;
+		case 2:
+			ksft_print_msg("the other two threads of the group were not ready in 5s\n");
+			break;
+		default:
+			ksft_print_msg("a thread of the group could not start\n");
+		}
+		return KSFT_FAIL;
+	}
+	if (!WIFSIGNALED(status) || WTERMSIG(status) != SIGXCPU) {
+		ksft_print_msg("the group of an overrun died of signal %d, not SIGXCPU\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_RECLAIM */
+
+#define RECLAIM_MIN	40	/* least % for a reclaiming task of 10% */
+#define PLAIN_MAX	25	/* most % for one without the flag */
+#define SHARE_SLACK	15	/* % two measures of a share may differ */
+#define SHARE_NS	(400 * NSEC_PER_MSEC)
+
+/*
+ * Measure the CPU share of a spinning deadline task of 10%, on P's CPU,
+ * without SCHED_FLAG_RECLAIM and then with it. With the flag, the task
+ * runs past its runtime: it takes the bandwidth of its root domain that
+ * no deadline task uses. It must get at most PLAIN_MAX percent without
+ * the flag, and at least RECLAIM_MIN percent with it.
+ */
+static int reclaim_runs(void)
+{
+	int plain, grub, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	plain = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM))
+		return KSFT_FAIL;
+	grub = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (plain > PLAIN_MAX || grub < RECLAIM_MIN) {
+		ksft_print_msg("of a CPU, a task of 10%% got %d%%, with SCHED_FLAG_RECLAIM %d%%\n",
+			       plain, grub);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Switch a reclaiming task out of SCHED_DEADLINE before its 0-lag time,
+ * and back. Its bandwidth stays until the inactive timer releases it.
+ * Afterwards, the task must reclaim as much as before.
+ */
+static int reclaim_switched(void)
+{
+	int before, after = 0, r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	before = share_of(t, SHARE_NS);
+	for (r = 0; r < 4; r++) {
+		if (set_other(t_tid)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (r & 1)
+			sleep_ns(5 * NSEC_PER_MSEC);
+		if (set_deadline_flags(t_tid, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			ksft_print_msg("round %d of 4: back to SCHED_DEADLINE refused\n",
+				       r + 1);
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(PERIOD);
+	}
+	if (ret == KSFT_PASS)
+		after = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (ret == KSFT_PASS &&
+	    (after < before - SHARE_SLACK || after > before + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%% of a CPU, %d%% after 4 round trips\n",
+			       before, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Fork a reclaiming deadline task of 10%. It spins, and publishes in
+ * mv_state[1] the CPU time it has used, in milliseconds. It idles while
+ * mv_state[2] is set, and the moves happen then. A cpuset move waits for
+ * an RCU grace period. Where RCU runs in kthreads (PREEMPT_RT), a task
+ * that reclaims its whole CPU keeps that grace period from ending. Return
+ * the pid, or -1.
+ */
+static pid_t reclaim_mover(void)
+{
+	pid_t pid = fork_child("a reclaiming task");
+
+	if (!pid) {
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			if (mv_state[2]) {
+				sleep_ns(10 * NSEC_PER_MSEC);
+				continue;
+			}
+			spin_ns(NSEC_PER_MSEC);
+			mv_state[1] = (int)(thread_time_ns() / NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* Idle the reclaiming task, then move it to @cg. Return 0, or -errno. */
+static int move_idle(const char *cg, pid_t pid)
+{
+	int err;
+
+	mv_state[2] = 1;
+	sleep_ns(20 * NSEC_PER_MSEC);	/* it stops spinning */
+	err = enter(cg, pid);
+	mv_state[2] = 0;
+	return err;
+}
+
+/* Return the CPU share, in percent, of the reclaiming task over @ns. */
+static int mover_share(uint64_t ns)
+{
+	uint64_t wall = now_ns();
+	int cpu = mv_state[1];
+
+	sleep_ns(ns);
+	cpu = mv_state[1] - cpu;
+	wall = now_ns() - wall;
+	return (int)(100 * (uint64_t)cpu * NSEC_PER_MSEC / wall);
+}
+
+/*
+ * Move a reclaiming task into P, back out, and into P again. It must
+ * reclaim as much of P's CPU as it did the first time. Once it is gone,
+ * the free bandwidth of both root domains must be what it was.
+ */
+static int reclaim_moved(void)
+{
+	int base_p, base_r, in_p, again = 0, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	mv_state[2] = 1;		/* idle until it is in P */
+	pid = reclaim_mover();
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a reclaiming task", 0) || move_idle(cg_p, pid)) {
+		mv_state[2] = 1;
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	in_p = mover_share(SHARE_NS);
+	if (move_idle(root, pid) || move_idle(cg_p, pid)) {
+		ksft_print_msg("a move of a reclaiming task refused\n");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(2 * PERIOD);
+		again = mover_share(SHARE_NS);
+		if (in_p < RECLAIM_MIN || again < in_p - SHARE_SLACK ||
+		    again > in_p + SHARE_SLACK) {
+			ksft_print_msg("reclaiming: %d%% of P's CPU, %d%% after a move\n",
+				       in_p, again);
+			ret = KSFT_FAIL;
+		}
+	}
+	mv_state[2] = 1;
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a reclaiming task moved: the free bandwidth changed\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Deadline servers */
+
+#define SERVER_RUNTIME	(100 * NSEC_PER_MSEC)	/* the cases start at 10% */
+#define SERVER_PERIOD	(1000 * NSEC_PER_MSEC)
+
+static const char server_dir[] = "/sys/kernel/debug/sched/fair_server";
+static uint64_t *server_found;	/* each fair server's runtime and period */
+
+static int server_get(int cpu, const char *name, uint64_t *val)
+{
+	char path[96], buf[32];
+
+	snprintf(path, sizeof(path), "%s/cpu%d/%s", server_dir, cpu, name);
+	if (read_file(path, buf, sizeof(buf)))
+		return -1;
+	*val = strtoull(buf, NULL, 10);
+	return 0;
+}
+
+static int server_put(int cpu, const char *name, uint64_t val)
+{
+	char dir[96], buf[32];
+
+	snprintf(dir, sizeof(dir), "%s/cpu%d", server_dir, cpu);
+	snprintf(buf, sizeof(buf), "%llu", (unsigned long long)val);
+	return write_file(dir, name, buf);
+}
+
+/*
+ * Set the fair server of @cpu to @runtime every @period.
+ * Return 0, or -errno.
+ */
+static int server_set(int cpu, uint64_t runtime, uint64_t period)
+{
+	uint64_t cur;
+	int err;
+
+	if (!server_found || server_get(cpu, "period", &cur))
+		return -ENOENT;
+	/* Each write keeps the runtime within the period. */
+	if (runtime <= cur) {
+		err = server_put(cpu, "runtime", runtime);
+		return err ? err : server_put(cpu, "period", period);
+	}
+	err = server_put(cpu, "period", period);
+	return err ? err : server_put(cpu, "runtime", runtime);
+}
+
+/*
+ * Record the runtime and period of each fair server. Without debugfs,
+ * record none: the server cases are then skipped.
+ */
+static void server_init(void)
+{
+	int i;
+
+	server_found = calloc(2 * ncpus, sizeof(*server_found));
+	for (i = 0; server_found && i < ncpus; i++) {
+		if (server_get(online[i], "runtime", &server_found[2 * i]) ||
+		    server_get(online[i], "period", &server_found[2 * i + 1])) {
+			free(server_found);
+			server_found = NULL;
+			ksft_print_msg("no fair server in debugfs: server cases skipped\n");
+		}
+	}
+}
+
+/* Put the fair servers back as the test found them. */
+static void server_restore(void)
+{
+	uint64_t runtime, period;
+	int i;
+
+	for (i = 0; server_found && i < ncpus; i++) {
+		int cpu = online[i], err;
+
+		if (server_get(cpu, "runtime", &runtime) || server_get(cpu, "period", &period))
+			continue;
+		if (runtime == server_found[2 * i] && period == server_found[2 * i + 1])
+			continue;
+		/*
+		 * Retry for up to 1s. While departing tasks still hold
+		 * their bandwidth, a root domain can refuse any change of a
+		 * server, a decrease too. Their 0-lag times are a period or
+		 * two away.
+		 */
+		err = wait_for(1000, !server_set(cpu, server_found[2 * i],
+						 server_found[2 * i + 1]));
+		if (err)
+			ksft_print_msg("the fair server of CPU %d stays changed\n", cpu);
+	}
+}
+
+/* Deadline threads of 10% every 100ms that sleep until released. */
+static pthread_t *held;
+static int nheld, held_err;
+static sem_t held_up, held_go;
+
+static void *holder(void *arg)
+{
+	int err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+
+	held_err = err;
+	if (sem_post(&held_up))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	if (!err)
+		sem_wait_sig(&held_go);
+	return NULL;
+}
+
+/*
+ * Start @n more holders. Return 0 once they are all admitted, or -1. The
+ * caller must call release_held(), on a failure too.
+ */
+static int hold(int n)
+{
+	int max = 16 * ncpus + 16, err;
+	pthread_attr_t a;
+
+	if (!held) {
+		held = calloc(max, sizeof(*held));
+		if (!held || sem_init(&held_up, 0, 0) || sem_init(&held_go, 0, 0)) {
+			ksft_print_msg("the holders cannot be set up\n");
+			return -1;
+		}
+	}
+	err = pthread_attr_init(&a);
+	if (!err)
+		err = pthread_attr_setstacksize(&a, THREAD_STACK);
+	if (err) {
+		ksft_print_msg("pthread_attr: %s\n", strerror(err));
+		return -1;
+	}
+	for (; n > 0; n--) {
+		if (nheld == max || pthread_create(&held[nheld], &a, holder, NULL)) {
+			ksft_print_msg("a holder thread could not start\n");
+			return -1;
+		}
+		if (sem_wait_for(&held_up, 10000)) {
+			ksft_print_msg("a holder did not answer\n");
+			nheld++;	/* release_held() joins it */
+			return -1;
+		}
+		if (held_err) {
+			join(held[nheld]);
+			return -1;
+		}
+		nheld++;
+	}
+	return 0;
+}
+
+static void release_held(void)
+{
+	int i;
+
+	for (i = 0; i < nheld; i++)
+		if (sem_post(&held_go))
+			ksft_print_msg("sem_post: %s\n", strerror(errno));
+	for (i = 0; i < nheld; i++)
+		join(held[i]);
+	/* Drain the semaphore counts a late or refused holder left. */
+	while (held && (!sem_trywait(&held_up) || !sem_trywait(&held_go)))
+		;
+	nheld = 0;
+	sleep_ns(3 * PERIOD);	/* their 0-lag times */
+}
+
+/*
+ * Change the runtime and the period of the fair server of P's CPU. Its
+ * bandwidth comes out of what P's root domain has for deadline tasks.
+ * From 100ms every second (10%), P must admit two probe tasks fewer at
+ * 30%, and one fewer at 100ms every 500ms. With the server off, it must
+ * admit one more. A change that P has no room for must be refused with
+ * EBUSY, and leave the server as it was.
+ */
+static int server_params(void)
+{
+	static const struct {
+		uint64_t runtime, period;
+		int more;
+		const char *what;
+	} step[] = {
+		{ 3 * SERVER_RUNTIME, SERVER_PERIOD, -2, "a fair server of 30%" },
+		{ SERVER_RUNTIME, SERVER_PERIOD / 2, -1, "a fair server of 100ms every 500ms" },
+		{ 0, SERVER_PERIOD, 1, "the fair server off" },
+		{ SERVER_RUNTIME, SERVER_PERIOD, 0, "the fair server back to 10%" },
+	};
+	int cpu = top_cpu(1), base, err, i, ret = KSFT_PASS;
+	uint64_t runtime;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	for (i = 0; i < (int)ARRAY_SIZE(step); i++) {
+		err = server_set(cpu, step[i].runtime, step[i].period);
+		if (err) {
+			ksft_print_msg("%s: %s\n", step[i].what, strerror(-err));
+			return KSFT_FAIL;
+		}
+		if (admits(base + step[i].more, step[i].what))
+			return KSFT_FAIL;
+	}
+	/*
+	 * With base - 1 probe tasks held, P has room for 10% more, and not
+	 * for 20%. So the server may grow to 20%, and not to 30%.
+	 */
+	if (hold(base - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = server_put(cpu, "runtime", 2 * SERVER_RUNTIME);
+	if (err) {
+		ksft_print_msg("a fair server of 20%% with room for it: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	} else {
+		err = server_put(cpu, "runtime", 3 * SERVER_RUNTIME);
+		if (err != -EBUSY) {
+			ksft_print_msg("a fair server of 30%% with no room for it: %s, not EBUSY\n",
+				       err ? strerror(-err) : "accepted");
+			ret = KSFT_FAIL;
+		} else if (server_get(cpu, "runtime", &runtime) ||
+			   runtime != 2 * SERVER_RUNTIME) {
+			ksft_print_msg("a refused change changed the fair server\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	release_held();
+	if (ret == KSFT_PASS && admits(base, "after a refused change of the fair server"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * A rebuild of the root domains' accounting must count each fair server
+ * with its current runtime, not its runtime at boot. Check it for the
+ * server of P's CPU, set to 30%:
+ *  - In P, as Q is made and unmade beside P.
+ *  - In the root domain of the other CPUs, once P is unmade.
+ *  - In P again, once P is made anew.
+ */
+static int server_rebuilds(void)
+{
+	int cpu = top_cpu(1), base, n10, n30;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base - 2, "P, a fair server of 30%"))
+		return KSFT_FAIL;
+	if (make_partition(cg_q, top_cpu(2))) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root Q beside P\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P, Q made beside it")) {
+		remove_partition(cg_q);
+		return KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	if (admits_in(cg_p, base - 2, "P, Q unmade beside it"))
+		return KSFT_FAIL;
+	/* Unmake P: its CPU and its server join the other CPUs. */
+	remove_p();
+	n30 = probe(root);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		return KSFT_FAIL;
+	n10 = probe(root);
+	if (n10 != n30 + 2) {
+		ksft_print_msg("P unmade: %d probe tasks, its CPU's server at 30%%, %d at 10%%\n",
+			       n30, n10);
+		return KSFT_FAIL;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) || make_p()) {
+		ksft_print_msg("P made anew, a fair server of 30%%: failed\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P made anew, a fair server of 30%") ||
+	    server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base, "P, the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/* Wait for the futex @f, in shared memory, to be set. */
+static void wait_set(atomic_int *f)
+{
+	while (!*f)
+		syscall(SYS_futex, f, FUTEX_WAIT, 0, NULL, NULL, 0);
+}
+
+static void set_flag(atomic_int *f, int val)
+{
+	*f = val;
+	syscall(SYS_futex, f, FUTEX_WAKE, INT_MAX, NULL, NULL, 0);
+}
+
+/*
+ * Fork a process of P that spins while mv_state[4 + @rt] is set, and
+ * sleeps otherwise. It is a fair task, or a real-time one if @rt. The
+ * real-time one sleeps 1ms every 50ms, to let the real-time kthreads of
+ * P's CPU run: PREEMPT_RT runs RCU and the timers there. Return the pid,
+ * or -1.
+ */
+static pid_t server_client(int rt)
+{
+	pid_t pid = fork_child(rt ? "a real-time task of P" : "a fair task of P");
+
+	if (!pid) {
+		struct sched_param sp = { .sched_priority = 1 };
+
+		if (enter(cg_p, getpid()) ||
+		    (rt && sched_setscheduler(0, SCHED_FIFO, &sp)))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			wait_set(&mv_state[4 + rt]);
+			spin_ns(rt ? 50 * NSEC_PER_MSEC : NSEC_PER_MSEC);
+			if (rt)
+				sleep_ns(NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* Return the CPU share of process @pid over @ns, in percent, or -1. */
+static int proc_share(pid_t pid, uint64_t ns)
+{
+	struct timespec t0, t1;
+	uint64_t wall = now_ns();
+	clockid_t c;
+
+	if (clock_getcpuclockid(pid, &c) || clock_gettime(c, &t0))
+		return -1;
+	sleep_ns(ns);
+	if (clock_gettime(c, &t1))
+		return -1;
+	wall = now_ns() - wall;
+	return (int)(100 * ((t1.tv_sec - t0.tv_sec) * NSEC_PER_SEC +
+			    t1.tv_nsec - t0.tv_nsec) / wall);
+}
+
+/* % that the share of a starved task may differ from its server's bandwidth */
+#define STARVED_SLACK	8
+
+/*
+ * Change the fair server of P's CPU, from outside P, in each state a
+ * change can find it in:
+ *  - stopped
+ *  - deferred while a fair task of P runs
+ *  - deferred while a real-time task starves that fair task
+ *  - serving it
+ *  - deferred again after serving it
+ *  - deferred while the CPU idles
+ *
+ * A fair task and a SCHED_FIFO task of P, see server_client(), put the
+ * server in these states.
+ *
+ * Each change starts a period of the server. The server defers serving a
+ * starved task to its zero-laxity time. That is 0.7s into the period at
+ * 30%, and 0.9s at 10%. Afterwards, the starved task must get the
+ * server's bandwidth, at 30% then at 10%. P's free bandwidth must then be
+ * what it was.
+ *
+ * A change cannot find the server in two more states. Throttled after
+ * serving, the server is deferred anew at once: the last of the served
+ * task's runtime is charged to it as background running. And on a CPU
+ * with no tick in idle, its idle wait begins and ends in one timer
+ * callback.
+ */
+static int server_states(void)
+{
+	static const struct {
+		int fair, rt, ms;
+	} step[] = {
+		{ 0, 0, 1500 },	/* stopped: nothing of P ran for a period */
+		{ 1, 0, 300 },	/* deferred while the fair task runs */
+		{ 1, 1, 300 },	/* deferred, fair task starved: before 0.9s */
+		{ 1, 1, 800 },	/* serving it: after 0.7s */
+		{ 1, 1, 1500 },	/* deferred again: served from 0.9s to 1s */
+		{ 0, 0, 600 },	/* deferred while the CPU idles */
+	};
+	int cpu = top_cpu(1), base, i, s30 = -1, s10 = -1, ret = KSFT_PASS;
+	pid_t fair, rt;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base <= 0) {
+		ksft_print_msg("no room in P to probe\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	set_flag(&mv_state[4], 0);
+	set_flag(&mv_state[5], 0);
+	fair = server_client(0);
+	rt = server_client(1);
+	if (fair < 0 || rt < 0 || wait_for(5000, mv_state[0] >= 2)) {
+		if (fair >= 0 && rt >= 0)
+			ksft_print_msg("a fair or real-time task of P could not start\n");
+		ret = KSFT_FAIL;
+	}
+	/*
+	 * Even steps set the server's runtime to 30%, odd ones back to 10%.
+	 * The period stays.
+	 */
+	for (i = 0; i < (int)ARRAY_SIZE(step) && ret == KSFT_PASS; i++) {
+		set_flag(&mv_state[4], step[i].fair);
+		set_flag(&mv_state[5], step[i].rt);
+		sleep_ns(step[i].ms * NSEC_PER_MSEC);
+		if (server_put(cpu, "runtime", (i & 1 ? 1 : 3) * SERVER_RUNTIME)) {
+			ksft_print_msg("change %d of the fair server refused\n", i + 1);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS) {
+		set_flag(&mv_state[4], 1);
+		set_flag(&mv_state[5], 1);
+		if (server_put(cpu, "runtime", 3 * SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s30 = proc_share(fair, 2 * SERVER_PERIOD);
+		if (server_put(cpu, "runtime", SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s10 = proc_share(fair, 2 * SERVER_PERIOD);
+	}
+	set_flag(&mv_state[5], 0);
+	set_flag(&mv_state[4], 0);
+	reap(rt);
+	reap(fair);
+	if (ret == KSFT_PASS &&
+	    (s30 < 30 - STARVED_SLACK || s30 > 30 + STARVED_SLACK ||
+	     s10 < 10 - STARVED_SLACK || s10 > 10 + STARVED_SLACK)) {
+		ksft_print_msg("a starved fair task got %d%% of a server of 30%%, %d%% of 10%%\n",
+			       s30, s10);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits_in(cg_p, base, "after the fair server's changes"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * A reclaiming deadline task of P gets the share of P's CPU that its root
+ * domain leaves free. P's fair server takes from that share. The task
+ * must get:
+ *  - less once the server grows from 10% to 30%.
+ *  - as much after a rebuild of the accounting (Q made and unmade).
+ *  - what it had at first once the server is back at 10%.
+ *
+ * Then put the server back as the test found it, so that P's free
+ * bandwidth is what it was.
+ */
+static int server_reclaim(void)
+{
+	int cpu = top_cpu(1), s10, s30, rebuilt, back, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	s10 = share_of(t, SHARE_NS);
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	s30 = share_of(t, SHARE_NS);
+	if (make_partition(cg_q, top_cpu(2)))
+		ret = KSFT_FAIL;
+	remove_partition(cg_q);
+	rebuilt = share_of(t, SHARE_NS);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	back = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	server_restore();
+	if (ret == KSFT_PASS &&
+	    (s30 > s10 - SHARE_SLACK ||
+	     rebuilt < s30 - SHARE_SLACK || rebuilt > s30 + SHARE_SLACK ||
+	     back < s10 - SHARE_SLACK || back > s10 + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%%, %d%% at 30%%, %d%% rebuilt, %d%% back at 10%%\n",
+			       s10, s30, rebuilt, back);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Parameter changes while a task wakes on another CPU */
+
+#define WAKE_NS		(5 * NSEC_PER_SEC)	/* how long the changes go on */
+#define WAKE_BURST	2000			/* pairs of changes per clock reading */
+
+/* Read this_bw and running_bw of @cpu from sched_debug. Return 0, or -1. */
+static int rq_bw_of(int cpu, long *this_bw, long *running_bw)
+{
+	char line[256], want[32];
+	FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+	int in = 0, got = 0;
+
+	if (!f)
+		return -1;
+	snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+	while (got < 2 && fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, want, strlen(want)))
+			in = 1;
+		else if (in && !strncmp(line, "dl_rq[", 6))
+			break;			/* the next CPU's */
+		else if (!in || !val)
+			continue;
+		else if (strstr(line, ".running_bw") && sscanf(val + 1, "%ld", running_bw) == 1)
+			got++;
+		else if (strstr(line, ".this_bw") && sscanf(val + 1, "%ld", this_bw) == 1)
+			got++;
+	}
+	fclose(f);
+	return got == 2 ? 0 : -1;
+}
+
+static atomic_int *wk_state;		/* 0: tasks up, 1: wakeups, 2: stop */
+
+/*
+ * Fork a deadline task of @runtime every @period in Q. It counts itself
+ * in wk_state[0], then spins. With @sleeps, it instead runs 20us and
+ * sleeps 80us in a loop, and counts its wakeups in wk_state[1]. It exits
+ * once wk_state[2] is set. Return the pid, or -1.
+ */
+static pid_t waking_task(uint64_t runtime, uint64_t period, int sleeps)
+{
+	pid_t pid = fork_child("a waking deadline task");
+
+	if (!pid) {
+		if (enter(cg_q, getpid()) || set_deadline(0, runtime, period)) {
+			wk_state[0] = -1;
+			_exit(1);
+		}
+		wk_state[0]++;
+		for (;;) {
+			if (wk_state[2])
+				_exit(0);
+			spin_ns(sleeps ? 20 * NSEC_PER_USEC : NSEC_PER_MSEC);
+			if (sleeps) {
+				sleep_ns(80 * NSEC_PER_USEC);
+				wk_state[1]++;
+			}
+		}
+	}
+	return pid;
+}
+
+/*
+ * Fork a process that changes the parameters of @pid over and over, until
+ * wk_state[2] is set. Return its pid, or -1.
+ */
+static pid_t changer_proc(pid_t pid)
+{
+	pid_t self = fork_child("a changer of the parameters");
+
+	if (!self) {
+		while (!wk_state[2]) {
+			if (set_deadline(pid, 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid, 10 * NSEC_PER_MSEC, PERIOD))
+				break;
+		}
+		_exit(0);
+	}
+	return self;
+}
+
+/*
+ * Change the parameters of a deadline task while its wakeups move it
+ * between two CPUs.
+ *
+ * Q is a partition root of two CPUs. Three deadline tasks run in Q, see
+ * waking_task(). Two of 20ms every 50ms spin, and hold both CPUs with
+ * earlier deadlines. The third is a task of 10ms every 100ms. It runs
+ * 20us and sleeps 80us in a loop. It wakes on whichever of the two CPUs
+ * has the later deadline. So a wakeup can move it to the other CPU.
+ *
+ * Two processes outside Q change the runtime of the third task for 5s:
+ * the test and changer_proc(). Each switches it between 20ms and 10ms in
+ * a loop.
+ *
+ * A wakeup that moves a task takes its bandwidth off the runqueue it
+ * slept on. The enqueue on the new runqueue adds it there, and may be
+ * left to the CPU the task wakes on. Between the two, no runqueue counts
+ * the bandwidth. A change in that window must not change this_bw of
+ * either runqueue. Once the tasks of the case are gone, this_bw of both
+ * CPUs must be what it was before.
+ *
+ * Check this_bw only then. sched_debug prints running_bw and this_bw of
+ * a CPU one after the other, with no lock held in between. Every
+ * parameter change takes the bandwidth of a task off its runqueue and
+ * puts it back. So a reading taken during the changes can straddle one
+ * of them.
+ */
+static int params_waking(void)
+{
+	int cpu_a = top_cpu(3), cpu_b = top_cpu(2), i, ret = KSFT_PASS, changes = 0;
+	long ta, ra, tb, rb, ta2, ra2, tb2, rb2;
+	pid_t pid[3] = { -1, -1, -1 }, changer = -1;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 non-isolated CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (rq_bw_of(cpu_a, &ta, &ra) || rq_bw_of(cpu_b, &tb, &rb)) {
+		remove_partition(cg_q);
+		ksft_print_msg("sched_debug tells of neither CPU's this_bw\n");
+		return KSFT_SKIP;
+	}
+	wk_state = mv_state + 8;
+	wk_state[0] = wk_state[1] = wk_state[2] = 0;
+	/* Two of 40% hold both CPUs with an earlier deadline than the third. */
+	pid[0] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[1] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[2] = waking_task(10 * NSEC_PER_MSEC, PERIOD, 1);
+	if (pid[0] < 0 || pid[1] < 0 || pid[2] < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	wait_for(5000, wk_state[0] >= 3 || wk_state[0] < 0);
+	if (wk_state[0] != 3) {
+		ksft_print_msg("no room for the deadline tasks of a partition of two CPUs\n");
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	changer = changer_proc(pid[2]);
+	if (changer < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	end = now_ns() + WAKE_NS;
+	while (now_ns() < end) {
+		for (i = 0; i < WAKE_BURST; i++, changes += 2) {
+			if (set_deadline(pid[2], 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid[2], 10 * NSEC_PER_MSEC, PERIOD)) {
+				ksft_print_msg("a change of the waking task refused\n");
+				ret = KSFT_FAIL;
+				goto out;
+			}
+		}
+	}
+	if (!wk_state[1]) {
+		ksft_print_msg("the waking task never woke\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	wk_state[2] = 1;
+	reap(changer);
+	for (i = 0; i < 3; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb) {
+			ksft_print_msg("%d changes: this_bw %ld and %ld, was %ld and %ld\n",
+				       changes, ta2, tb2, ta, tb);
+			ret = KSFT_FAIL;
+		}
+	}
+	remove_partition(cg_q);
+	return ret;
+}
+
+/* A random mix of the transitions above */
+
+#define MIX_WORKERS	6
+#define MIX_NS		(3 * NSEC_PER_SEC)
+#define MIX_SEED	20261103	/* fixed: a rerun starts with the same draws */
+
+/*
+ * random_mix: six workers change their own state at random for 3s, and
+ * the case changes them from outside too. Q is a partition root of two
+ * CPUs, next to P. The case needs 4 housekeeping CPUs.
+ *
+ * Each worker does one of these, over and over:
+ *  - become a deadline task, with random parameters and flags
+ *  - switch to SCHED_OTHER
+ *  - sleep
+ *  - spin, at times past its runtime
+ *  - call sched_yield()
+ *  - wait up to 5ms for shm_lock and hold it 1ms. It is the mutex of the
+ *    priority inheritance cases. A deadline waiter boosts its owner.
+ *  - exit
+ *
+ * The case does one of these, over and over:
+ *  - set the deadline parameters of a random worker
+ *  - switch it to SCHED_OTHER
+ *  - change its affinity
+ *  - move it into P, into Q or into the root cgroup
+ *  - kill it
+ *  - change the fair server of P's CPU or of one CPU of Q
+ * It starts anew every worker it finds dead.
+ *
+ * The case does not check what a single transition does: the other cases
+ * do that. It checks the accounting afterwards. Once every worker is gone
+ * and its 0-lag time is past:
+ *  - P, Q and the housekeeping root domain must each admit what they
+ *    admitted before;
+ *  - this_bw of P's CPU and of the two CPUs of Q must be what it was.
+ * An over- or underflow of running_bw or this_bw also shows as a warning
+ * in the kernel log, which the test does not read.
+ */
+static atomic_int *mix_state;		/* 0: stop, 1 + i: worker i's rounds */
+
+static pid_t mix_worker(int idx, unsigned int seed)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	static const uint64_t period[] = { 50, 100 };
+	static const uint64_t flag[] = { 0, SCHED_FLAG_RECLAIM, SCHED_FLAG_DL_OVERRUN };
+	pid_t pid = fork_child("a worker of the mix");
+
+	if (!pid) {
+		struct sigaction sa = { .sa_handler = catch_xcpu };
+		struct timespec ts;
+		int err;
+
+		if (sigaction(SIGXCPU, &sa, NULL))
+			_exit(1);
+		seed += idx;
+		/*
+		 * Ignore the errors. Each call below is made in whatever
+		 * state the worker is in, and the kernel may refuse it.
+		 */
+		while (!mix_state[0]) {
+			unsigned int r = rand_r(&seed), v = r / 8;
+
+			switch (r % 8) {
+			case 0:
+				set_deadline_flags(0, runtime[v % 4] * NSEC_PER_MSEC,
+						   period[(v / 4) % 2] * NSEC_PER_MSEC,
+						   flag[(v / 8) % 3]);
+				break;
+			case 1:
+				set_other(0);
+				break;
+			case 2:
+				sleep_ns((v % 20 + 1) * NSEC_PER_MSEC);
+				break;
+			case 3:
+				spin_ns((v % 15 + 1) * NSEC_PER_MSEC);
+				break;
+			case 4:
+				sched_yield();
+				break;
+			case 5:
+				clock_gettime(CLOCK_REALTIME, &ts);
+				ts.tv_nsec += 5 * NSEC_PER_MSEC;
+				if (ts.tv_nsec >= NSEC_PER_SEC) {
+					ts.tv_nsec -= NSEC_PER_SEC;
+					ts.tv_sec++;
+				}
+				err = pthread_mutex_timedlock(shm_lock, &ts);
+				if (err == EOWNERDEAD)	/* its owner died */
+					err = pthread_mutex_consistent(shm_lock);
+				if (!err) {
+					spin_ns(NSEC_PER_MSEC);
+					if (pthread_mutex_unlock(shm_lock))
+						_exit(1);
+				}
+				break;
+			case 6:
+				_exit(0);	/* the case starts another */
+			default:
+				sleep_ns(NSEC_PER_MSEC / 2);
+				break;
+			}
+			mix_state[1 + idx]++;
+		}
+		_exit(0);
+	}
+	return pid;
+}
+
+static cpu_set_t mix_mask[4];
+
+static int random_mix(void)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	int cpu_a = top_cpu(3), cpu_b = top_cpu(2), cpu_p = top_cpu(1);
+	int base_p, base_q, base_r, i, ret = KSFT_PASS, rounds = 0, sums;
+	long ta = 0, ra, tb = 0, rb, tp = 0, rp, ta2, ra2, tb2, rb2, tp2, rp2;
+	pid_t pid[MIX_WORKERS] = { -1, -1, -1, -1, -1, -1 };
+	unsigned int seed = MIX_SEED;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 non-isolated CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	/*
+	 * The affinities of the mix: every CPU of the test, those of Q and
+	 * P, those of P and the rest, and all but the first.
+	 */
+	for (i = 0; i < 4; i++)
+		CPU_ZERO(&mix_mask[i]);
+	for (i = 0; i < ncpus; i++) {
+		int c = online[i];
+
+		CPU_SET(c, &mix_mask[0]);
+		if (c == cpu_a || c == cpu_b || c == cpu_p)
+			CPU_SET(c, &mix_mask[1]);
+		if (c != cpu_a && c != cpu_b)
+			CPU_SET(c, &mix_mask[2]);
+		if (i)
+			CPU_SET(c, &mix_mask[3]);
+	}
+	if (shm_lock_init()) {
+		remove_partition(cg_q);
+		return KSFT_SKIP;	/* no shared mutex for the workers */
+	}
+	/* Without this_bw in sched_debug, check the admissions only. */
+	sums = !rq_bw_of(cpu_a, &ta, &ra) && !rq_bw_of(cpu_b, &tb, &rb) &&
+	       !rq_bw_of(cpu_p, &tp, &rp);
+	base_p = probe(cg_p);
+	base_q = probe(cg_q);
+	base_r = probe(root);
+	if (base_p <= 0 || base_q <= 0 || base_r <= 0) {
+		remove_partition(cg_q);
+		ksft_print_msg("no room to probe P, Q or the rest\n");
+		return KSFT_SKIP;
+	}
+	mix_state = mv_state + 16;
+	for (i = 0; i <= MIX_WORKERS; i++)
+		mix_state[i] = 0;
+	for (i = 0; i < MIX_WORKERS; i++) {
+		pid[i] = mix_worker(i, seed);
+		if (pid[i] < 0) {
+			ret = KSFT_FAIL;
+			goto out;
+		}
+	}
+
+	end = now_ns() + MIX_NS;
+	while (now_ns() < end) {
+		unsigned int r = rand_r(&seed);
+		int w = r % MIX_WORKERS;
+
+		switch ((r / MIX_WORKERS) % 8) {
+		case 0:
+			set_deadline(pid[w], runtime[r % 4] * NSEC_PER_MSEC, PERIOD);
+			break;
+		case 7:
+			/*
+			 * Refused with EBUSY if the mask leaves out a CPU
+			 * of the root domain that counts a deadline task.
+			 */
+			sched_setaffinity(pid[w], sizeof(mix_mask[0]), &mix_mask[r % 4]);
+			break;
+		case 1:
+			set_other(pid[w]);
+			break;
+		case 2:
+			enter(cg_p, pid[w]);
+			break;
+		case 3:
+			enter(cg_q, pid[w]);
+			break;
+		case 4:
+			enter(root, pid[w]);
+			break;
+		case 5:
+			/* An exited worker is a zombie: the kill succeeds. */
+			if (kill(pid[w], SIGKILL) && errno != ESRCH)
+				ksft_print_msg("kill %d: %s\n", pid[w], strerror(errno));
+			break;
+		default:
+			server_set(r % 2 ? cpu_p : cpu_a,
+				   (r % 3 + 1) * SERVER_RUNTIME, SERVER_PERIOD);
+			break;
+		}
+		sleep_ns((r % 2000 + 100) * NSEC_PER_USEC);
+		/* Start anew each worker that exited or was killed. */
+		for (i = 0; i < MIX_WORKERS; i++) {
+			pid_t done = waitpid(pid[i], NULL, WNOHANG);
+
+			if (done < 0 && errno != EINTR) {
+				ksft_print_msg("waitpid %d: %s\n", pid[i], strerror(errno));
+				ret = KSFT_FAIL;
+				goto out;
+			}
+			if (done == pid[i]) {
+				pid[i] = mix_worker(i, seed + rounds);
+				if (pid[i] < 0) {
+					ret = KSFT_FAIL;
+					goto out;
+				}
+				rounds++;
+			}
+		}
+	}
+out:
+	mix_state[0] = 1;
+	for (i = 0; i < MIX_WORKERS; i++)
+		reap(pid[i]);
+	/*
+	 * Wait for the 0-lag times of the workers. Putting the fair servers
+	 * back needs the room they held.
+	 */
+	sleep_ns(5 * PERIOD);
+	server_restore();
+	if (ret != KSFT_PASS) {
+		remove_partition(cg_q);
+		return ret;
+	}
+	for (i = 0, rounds = 0; i < MIX_WORKERS; i++)
+		rounds += mix_state[1 + i];
+	if (!rounds) {
+		ksft_print_msg("the workers of the mix took no transition\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && sums) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2) ||
+		    rq_bw_of(cpu_p, &tp2, &rp2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb || tp2 != tp) {
+			ksft_print_msg("this_bw %ld %ld %ld, was %ld %ld %ld\n",
+				       ta2, tb2, tp2, ta, tb, tp);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, base_p, "P after the mix") ||
+	     admits_in(cg_q, base_q, "Q after the mix") ||
+	     admits_in(root, base_r, "the root domain of the rest after the mix")))
+		ret = KSFT_FAIL;
+	if (ret != KSFT_PASS)
+		ksft_print_msg("seed %u, %d transitions\n", MIX_SEED, rounds);
+	remove_partition(cg_q);
+	return ret;
+}
+
+/*
+ * CPU hotplug, from the main process, with no P. Two cases below run with
+ * P instead: death_zombie_hotplug and pi_departed_moved.
+ */
+
+static int hp_cpu = -1;
+
+/* Return whether the online file of @cpu can be written. */
+static int cpu_hotpluggable(int cpu)
+{
+	char path[64];
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	return !access(path, W_OK);
+}
+
+static int cpu_online(int cpu, int on)
+{
+	char dir[64];
+
+	snprintf(dir, sizeof(dir), "/sys/devices/system/cpu/cpu%d", cpu);
+	return write_file(dir, "online", on ? "1" : "0");
+}
+
+/* Bring @cpu back online: every case that follows needs it. */
+static void cpu_up(int cpu)
+{
+	int err = cpu_online(cpu, 1);
+
+	if (err)
+		ksft_print_msg("CPU %d cannot come back online: %s\n", cpu, strerror(-err));
+}
+
+/*
+ * Four deadline tasks of 30% sleep. Take hp_cpu offline and back. Once
+ * the sleepers are gone, the housekeeping root domain must admit what it
+ * did before. Skip unless 22 probe tasks fit: the sleepers take the room
+ * of 12, and the offline CPU that of about 10.
+ */
+static int hotplug_sleepers(void)
+{
+	int base = probe(NULL), after, i, ret = KSFT_PASS;
+	pid_t pid[4];
+
+	if (base < 22) {
+		ksft_print_msg("needs room for 22 probe tasks\n");
+		return KSFT_SKIP;
+	}
+	for (i = 0; i < 4; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			return KSFT_FAIL;
+		}
+	}
+	if (cpu_online(hp_cpu, 0) || cpu_online(hp_cpu, 1)) {
+		ksft_print_msg("offline and online of CPU %d with deadline sleepers failed\n",
+			       hp_cpu);
+		ret = KSFT_FAIL;
+		cpu_up(hp_cpu);
+	}
+	for (i = 0; i < 4; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("hotplug with sleepers: %d probe tasks admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Fork a task that runs on @cpu, then restores its affinity, becomes a
+ * deadline task of 30% and sleeps. A deadline task cannot have an
+ * affinity of one CPU, so the task goes to @cpu first. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t sleeper_on(int cpu)
+{
+	pid_t pid = fork_child("a deadline sleeper");
+
+	if (!pid) {
+		cpu_set_t all, one;
+
+		CPU_ZERO(&one);
+		CPU_SET(cpu, &one);
+		if (sched_getaffinity(0, sizeof(all), &all) ||
+		    sched_setaffinity(0, sizeof(one), &one))
+			_exit(1);
+		spin_ns(NSEC_PER_MSEC);
+		if (sched_setaffinity(0, sizeof(all), &all) ||
+		    set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * A deadline task of 30% sleeps on hp_cpu. Take hp_cpu offline, then
+ * switch the task to SCHED_OTHER. hp_cpu has left the housekeeping root
+ * domain by then, but that root domain still counts the task. The switch
+ * must release the bandwidth there.
+ *
+ * Check this with hp_cpu still offline: bringing it back rebuilds the
+ * accounting, which would hide a bandwidth left behind. The reference is
+ * what the housekeeping root domain admits with hp_cpu offline and no
+ * task. Once the CPU is back and the task is gone, that root domain must
+ * also admit what it did at first.
+ */
+static int hotplug_sleeper_switched(void)
+{
+	int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+	pid_t pid;
+
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("CPU %d cannot go offline\n", hp_cpu);
+		cpu_up(hp_cpu);
+		return KSFT_SKIP;
+	}
+	base_off = probe(NULL);
+	cpu_up(hp_cpu);
+	mv_state[0] = 0;
+	pid = sleeper_on(hp_cpu);
+	if (pid < 0 || child_up("a deadline sleeper", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(PERIOD);
+	cpu = task_cpu_of(pid);
+	if (cpu != hp_cpu) {
+		ksft_print_msg("the sleeper is on CPU %d, not on CPU %d\n", cpu, hp_cpu);
+		reap(pid);
+		return KSFT_SKIP;
+	}
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("CPU %d cannot go offline under a deadline sleeper\n", hp_cpu);
+		ret = KSFT_FAIL;
+	} else if (set_other(pid)) {
+		ksft_print_msg("the sleeper cannot leave SCHED_DEADLINE\n");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(3 * PERIOD);
+		after = probe(NULL);
+		if (after != base_off) {
+			ksft_print_msg("a sleeper switched out, its CPU still offline: %d probe tasks admitted, %d expected\n",
+				       after, base_off);
+			ret = KSFT_FAIL;
+		}
+	}
+	cpu_up(hp_cpu);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("a sleeper switched out on an offline CPU: %d probe tasks admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Turn admission control off (sched_rt_runtime_us = -1). A task pins
+ * itself to hp_cpu, becomes a deadline task of 50% and spins. Without
+ * admission control, a deadline task can have an affinity of one CPU.
+ * Take hp_cpu offline: the mask of the task holds no active CPU. The
+ * task must go on running on another CPU. Once the CPU is back, the task
+ * is gone and admission control is on again, the housekeeping root domain
+ * must admit what it did before.
+ */
+static int hotplug_no_admission_pinned(void)
+{
+	int base = probe(NULL), after, ret = KSFT_PASS, last;
+	char saved[64];
+	pid_t pid;
+
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+	    write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+		ksft_print_msg("admission control cannot be turned off\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = fork_child("a pinned deadline task");
+	if (!pid) {
+		cpu_set_t one;
+
+		CPU_ZERO(&one);
+		CPU_SET(hp_cpu, &one);
+		if (sched_setaffinity(0, sizeof(one), &one) ||
+		    set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			mv_state[1]++;
+	}
+	if (pid < 0 || child_up("a pinned deadline task", 0)) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(PERIOD);
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("CPU %d cannot go offline under a pinned deadline task\n", hp_cpu);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	last = mv_state[1];
+	sleep_ns(3 * PERIOD);
+	if (mv_state[1] == last) {
+		ksft_print_msg("the pinned task does not run after its CPU went offline\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	cpu_up(hp_cpu);
+	reap(pid);
+	if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+		ksft_print_msg("admission control cannot be turned on again\n");
+		return KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("a pinned task without admission control: %d probe tasks admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Deadline sleepers take most of the free bandwidth. Taking hp_cpu
+ * offline must then fail with EBUSY. Once the sleepers are gone, the
+ * housekeeping root domain must admit what it did before.
+ */
+static int hotplug_refused(void)
+{
+	int base = probe(NULL), n, i, ret = KSFT_PASS, err;
+	pid_t *pid;
+
+	if (base < 12) {
+		ksft_print_msg("needs room for 12 probe tasks\n");
+		return KSFT_SKIP;
+	}
+	/*
+	 * The sleepers of 30% leave 30% to 60% of a CPU free. A CPU that
+	 * goes offline takes more than that, so the offline must fail.
+	 */
+	n = base / 3 - 1;
+	pid = calloc(n, sizeof(*pid));
+	if (!pid) {
+		ksft_print_msg("calloc: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			free(pid);
+			return KSFT_FAIL;
+		}
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (!err) {
+		ksft_print_msg("CPU %d went offline with its deadline bandwidth in use\n", hp_cpu);
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else if (err != -EBUSY) {
+		ksft_print_msg("offline refused with %s, not EBUSY\n", strerror(-err));
+		ret = KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++)
+		reap(pid[i]);
+	free(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && probe(NULL) != base) {
+		ksft_print_msg("a refused offline changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A deadline task of 40% spins. The case switches it to SCHED_OTHER
+ * before its 0-lag time. Then it takes the CPU that the task is on
+ * offline, before that time too. Past the 0-lag time, with the CPU still
+ * offline, the housekeeping root domain must admit what it admits with a
+ * CPU offline and no task. Once the CPU is back and the task is gone, it
+ * must admit what it did at the start.
+ */
+static int hotplug_departed(void)
+{
+	int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+	pid_t pid;
+
+	/*
+	 * Take the reference with hp_cpu offline and no task of ours. The
+	 * task may be on another CPU: the case assumes that any one CPU
+	 * offline leaves the same room. The case compares while that CPU
+	 * is still offline. Bringing it back rebuilds the accounting,
+	 * which would hide an error.
+	 */
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	base_off = probe(NULL);
+	cpu_up(hp_cpu);
+	if (base <= 0 || base_off <= 0) {
+		ksft_print_msg("no room to probe with the CPU online or offline\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(MV_THROTTLED, 40 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a departing deadline task", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(PERIOD / 2);
+	/* Not asleep: its bandwidth stays until its 0-lag time. */
+	if (set_other(pid))
+		ret = KSFT_FAIL;
+	cpu = task_cpu_of(pid);
+	if (cpu < 0 || !cpu_hotpluggable(cpu)) {
+		cpu_set_t set;
+
+		/* Out of the class now, it may be pinned to hp_cpu. */
+		cpu = hp_cpu;
+		CPU_ZERO(&set);
+		CPU_SET(cpu, &set);
+		if (sched_setaffinity(pid, sizeof(set), &set))
+			ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && cpu_online(cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", cpu);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);		/* its 0-lag time, the CPU offline */
+	if (ret == KSFT_PASS) {
+		after = probe(NULL);
+		if (after != base_off) {
+			ksft_print_msg("departed, CPU %d offline: %d admitted, %d expected\n",
+				       cpu, after, base_off);
+			ret = KSFT_FAIL;
+		}
+	}
+	cpu_up(cpu);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("departed, CPU back online: %d admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * Read dl_bw->total_bw of @cpu from sched_debug: the bandwidth that the
+ * root domain of @cpu counts. Return it, or -1.
+ */
+static long long total_bw_of(int cpu)
+{
+	char line[256], want[32];
+	FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+	long long bw = -1;
+	int in = 0;
+
+	if (!f)
+		return -1;
+	snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+	while (fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, want, strlen(want)))
+			in = 1;
+		else if (in && !strncmp(line, "dl_rq[", 6))
+			break;			/* the next CPU's */
+		else if (in && val && strstr(line, ".dl_bw->total_bw") &&
+			 sscanf(val + 1, "%lld", &bw) == 1)
+			break;
+	}
+	fclose(f);
+	return bw;
+}
+
+#define ZOMBIE_RUNTIME	(200 * NSEC_PER_MSEC)
+#define ZOMBIE_PERIOD	(2 * NSEC_PER_SEC)
+
+/*
+ * Q is a partition root of two CPUs. The kernel records, with the
+ * bandwidth of a task, a CPU of the root domain that counts it. The steps:
+ * 1. A task enters Q and becomes a deadline task of 200ms every 2s.
+ * 2. It runs 190ms and exits, its 0-lag time about 1.7s ahead. The case
+ *    does not reap it yet. A rebuild of the accounting still finds a
+ *    zombie in the task list.
+ * 3. total_bw of Q must count the zombie.
+ * 4. The case takes each CPU of Q offline and back, one after the other.
+ *    One of them is the CPU recorded with the bandwidth of the zombie.
+ *    The rebuild at its offline drops that bandwidth.
+ * 5. Before the 0-lag time, total_bw of Q must be what it was without
+ *    the task. No later rebuild may count the zombie again: the release
+ *    at the 0-lag time removes nothing by then.
+ * 6. The case reaps the zombie. Past the 0-lag time, total_bw of Q must
+ *    still be that value.
+ *
+ * The case runs with the cases in P, from the main process: Q needs the
+ * cpuset controller. Q has two CPUs because the last CPU of a root domain
+ * that holds deadline bandwidth cannot go offline.
+ */
+static int death_zombie_hotplug(void)
+{
+	int cpu[2] = { top_cpu(3), top_cpu(2) }, i, ret = KSFT_PASS;
+	long long base, bw, held, after;
+	char cpus[32];
+	uint64_t t;
+	pid_t pid;
+
+	if (ncpus < 4 || !cpu_hotpluggable(cpu[0]) || !cpu_hotpluggable(cpu[1])) {
+		ksft_print_msg("needs 4 non-isolated CPUs, 2 hotpluggable beside P\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d,%d", cpu[0], cpu[1]);
+	if (make_partition_cpus(cg_q, cpus)) {
+		ksft_print_msg("no partition root Q of CPUs %s\n", cpus);
+		remove_partition(cg_q);
+		return KSFT_SKIP;
+	}
+	base = total_bw_of(cpu[0]);
+	if (base < 0) {
+		ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+		remove_partition(cg_q);
+		return KSFT_SKIP;
+	}
+	bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD;	/* as to_ratio() */
+	mv_state[0] = 0;
+	pid = fork_child("a deadline task that exits");
+	if (!pid) {
+		if (enter(cg_q, getpid()) || set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD))
+			_exit(1);
+		/* 10ms left: the 0-lag time is 100ms before the deadline. */
+		spin_ns(ZOMBIE_RUNTIME - 10 * NSEC_PER_MSEC);
+		mv_state[0] = 1;
+		_exit(0);
+	}
+	if (pid < 0 || child_up("a task of Q that exits", 0) ||
+	    wait_for(1000, proc_state_of(pid) == 'Z')) {
+		ksft_print_msg("no zombie deadline task\n");
+		reap(pid);
+		remove_partition(cg_q);
+		return KSFT_FAIL;
+	}
+	/* Its 0-lag time is about 1.7s ahead. Check total_bw within 1s. */
+	t = now_ns();
+	held = total_bw_of(cpu[0]);
+	if (held != base + bw) {
+		ksft_print_msg("total_bw of Q: %lld with the zombie, %lld expected\n",
+			       held, base + bw);
+		ret = KSFT_FAIL;
+	}
+	for (i = 0; i < 2 && ret == KSFT_PASS; i++) {
+		if (cpu_online(cpu[i], 0) || cpu_online(cpu[i], 1)) {
+			ksft_print_msg("offline and online of CPU %d failed\n", cpu[i]);
+			cpu_up(cpu[i]);
+			ret = KSFT_FAIL;
+		}
+	}
+	after = total_bw_of(cpu[0]);
+	if (ret == KSFT_PASS && now_ns() - t > NSEC_PER_SEC) {
+		ksft_print_msg("the hotplugs took over 1s: the 0-lag time may be past\n");
+		ret = KSFT_SKIP;
+	} else if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("total_bw of Q after the hotplugs: %lld, %lld expected%s\n",
+			       after, base,
+			       after == base + bw ? ": the zombie counted again" : "");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	sleep_ns(ZOMBIE_PERIOD);		/* past its 0-lag time */
+	after = total_bw_of(cpu[0]);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("total_bw of Q past the 0-lag time: %lld, %lld expected\n",
+			       after, base);
+		ret = KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	return ret;
+}
+
+/*
+ * Fork the owner of pi_departed_moved(). It takes shm_lock, becomes a
+ * deadline task and runs most of its runtime. It sets mv_state[0] to 1
+ * and sleeps 300ms. Then it sets mv_state[1] to 1 and sleeps with the
+ * lock. Return its pid, or -1.
+ */
+static pid_t departed_owner(void)
+{
+	pid_t pid = fork_child("the owner of the lock");
+
+	if (!pid) {
+		if (pthread_mutex_lock(shm_lock) ||
+		    set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		/* 20ms left: the 0-lag time is 200ms before the deadline. */
+		spin_ns(ZOMBIE_RUNTIME - 20 * NSEC_PER_MSEC);
+		mv_state[0] = 1;
+		sleep_ns(300 * NSEC_PER_MSEC);
+		mv_state[1] = 1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * The owner is a deadline task of 200ms every 2s, outside P. It holds
+ * shm_lock, runs 180ms and sleeps, its 0-lag time about 1.6s ahead. The
+ * steps:
+ * 1. The case switches the sleeping owner to SCHED_OTHER. The
+ *    housekeeping root domain keeps its bandwidth until the 0-lag time.
+ * 2. The case moves the owner into P. The owner wakes up and runs there.
+ * 3. A deadline waiter of 5ms every 50ms, outside P, blocks on shm_lock
+ *    and boosts the owner.
+ * 4. The case kills the waiter, which ends the boost.
+ *
+ * After step 4, total_bw of the housekeeping root domain must still count
+ * the bandwidth of the owner. total_bw of P never counted it and must be
+ * unchanged: the end of the boost must not subtract it there. Past the
+ * 0-lag time, both must be what they were at the start.
+ */
+static int pi_departed_moved(void)
+{
+	int p_cpu = top_cpu(1), r_cpu = online[0], ret = KSFT_PASS;
+	long long base_p, base_r, bw, in_p, in_r;
+	pid_t owner, waiter = -1;
+
+	base_p = total_bw_of(p_cpu);
+	base_r = total_bw_of(r_cpu);
+	if (base_p < 0 || base_r < 0) {
+		ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+		return KSFT_SKIP;
+	}
+	if (shm_lock_init())
+		return KSFT_SKIP;
+	bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD;	/* as to_ratio() */
+	mv_state[1] = 0;
+	owner = departed_owner();
+	if (owner < 0)
+		return KSFT_FAIL;
+	if (child_up("the owner of the lock", 0)) {
+		reap(owner);
+		return KSFT_FAIL;
+	}
+	/* The owner is asleep, about 1.6s before its 0-lag time. */
+	sleep_ns(20 * NSEC_PER_MSEC);
+	if (set_other(owner) || enter(cg_p, owner)) {
+		ksft_print_msg("the owner cannot leave the class and enter P\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_state(&mv_state[1], 1) || task_cpu_of(owner) != p_cpu) {
+		ksft_print_msg("the owner did not run in P\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	waiter = lock_waiter();
+	if (waiter < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_state(&mv_state[2], 1)) {
+		ksft_print_msg("no waiter\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);	/* blocked, boosting the owner */
+	reap(waiter);
+	waiter = -1;
+	sleep_ns(150 * NSEC_PER_MSEC);	/* the waiter is past its 0-lag time */
+	in_p = total_bw_of(p_cpu);
+	in_r = total_bw_of(r_cpu);
+	if (in_p != base_p || in_r != base_r + bw) {
+		ksft_print_msg("after the boost: total_bw of P %lld, %lld expected\n",
+			       in_p, base_p);
+		ksft_print_msg("after the boost: total_bw outside %lld, %lld expected\n",
+			       in_r, base_r + bw);
+		ret = KSFT_FAIL;
+	}
+out:
+	reap(waiter);
+	reap(owner);
+	sleep_ns(ZOMBIE_PERIOD);	/* past the owner's 0-lag time */
+	in_p = total_bw_of(p_cpu);
+	in_r = total_bw_of(r_cpu);
+	if (ret == KSFT_PASS && (in_p != base_p || in_r != base_r)) {
+		ksft_print_msg("past the 0-lag time: total_bw of P %lld, %lld expected\n",
+			       in_p, base_p);
+		ksft_print_msg("past the 0-lag time: total_bw outside %lld, %lld expected\n",
+			       in_r, base_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * The bandwidth of a fair server must leave the root domain with its CPU
+ * and come back with it, whatever its size. Probe the housekeeping root
+ * domain with:
+ *  - the server of hp_cpu at 10%, the CPU online: the base;
+ *  - the server at 10%, the CPU offline;
+ *  - the server at 30%, the CPU online: two probe tasks fewer than the
+ *    base;
+ *  - the server at 30%, the CPU offline: as many as at 10%, offline.
+ * With the CPU back and its server at 30%, the root domain must admit two
+ * fewer than the base again. With the server back at 10%, it must admit
+ * the base.
+ */
+static int hotplug_server(void)
+{
+	int base, off10, off30;
+
+	if (server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off10 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (server_set(hp_cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base - 2, "a fair server of 30%"))
+		return KSFT_FAIL;
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused, its fair server at 30%%\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off30 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (off30 != off10) {
+		ksft_print_msg("CPU %d offline: %d probe tasks, its server at 10%%, %d at 30%%\n",
+			       hp_cpu, off10, off30);
+		return KSFT_FAIL;
+	}
+	if (admits(base - 2, "a fair server of 30%, its CPU back online") ||
+	    server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base, "the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/*
+ * A CPU that goes offline takes its capacity and the bandwidth of its
+ * fair server out of the root domain. Set the server of hp_cpu to the
+ * admission limit less 10%. The offline then costs the root domain the
+ * room of one probe task. Hold every probe task that the root domain
+ * admits but one: the CPU can still go offline. Hold one more: the
+ * offline must fail with EBUSY.
+ */
+static int hotplug_server_refused(void)
+{
+	uint64_t rt_runtime, rt_period, runtime;
+	int n, err, ret = KSFT_PASS;
+	char buf[32];
+
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) ||
+	    read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf))) {
+		ksft_print_msg("no sched_rt_runtime_us or sched_rt_period_us\n");
+		return KSFT_SKIP;
+	}
+	read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf));
+	rt_runtime = atoll(buf);
+	read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf));
+	rt_period = atoll(buf);
+	if (!rt_period || 5 * rt_runtime < rt_period) {
+		ksft_print_msg("the admission limit is below a fifth of a CPU\n");
+		return KSFT_SKIP;
+	}
+	runtime = SERVER_PERIOD * rt_runtime / rt_period - SERVER_PERIOD / 10;
+	if (server_set(hp_cpu, runtime, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	n = probe(NULL);
+	if (n < 2) {
+		ksft_print_msg("no room for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	if (hold(n - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (err) {
+		ksft_print_msg("CPU %d, room for one probe task left: offline refused (%s)\n",
+			       hp_cpu, strerror(-err));
+		ret = KSFT_FAIL;
+	} else if (cpu_online(hp_cpu, 1) || hold(1)) {
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else {
+		err = cpu_online(hp_cpu, 0);
+		if (err != -EBUSY) {
+			ksft_print_msg("CPU %d, no room left: offline %s, not EBUSY\n",
+				       hp_cpu, err ? strerror(-err) : "accepted");
+			if (!err)
+				cpu_up(hp_cpu);
+			ret = KSFT_FAIL;
+		}
+	}
+	release_held();
+	return ret;
+}
+
+/* Return whether @cpu is in @list, a CPU list as sysfs prints it. */
+static int in_cpu_list(const char *list, int cpu)
+{
+	const char *p = list;
+
+	while (*p >= '0' && *p <= '9') {
+		char *end;
+		long lo = strtol(p, &end, 10), hi = lo;
+
+		if (*end == '-')
+			hi = strtol(end + 1, &end, 10);
+		if (cpu >= lo && cpu <= hi)
+			return 1;
+		p = *end == ',' ? end + 1 : end;
+	}
+	return 0;
+}
+
+/*
+ * Wake @pid, a task that sleeps in read() on a pipe: write a byte to @fd.
+ * The task counts its wakeups in mv_state[1]. Wait until it has run and
+ * sleeps again. Return 0, or -1.
+ */
+static int fallback_wake(int fd, pid_t pid)
+{
+	int n = mv_state[1];
+
+	if (write(fd, "w", 1) != 1) {
+		ksft_print_msg("write to the deadline task: %s\n", strerror(errno));
+		return -1;
+	}
+	if (wait_for(1000, mv_state[1] != n) ||
+	    wait_for(1000, proc_state_of(pid) == 'S')) {
+		ksft_print_msg("the deadline task did not wake up and sleep again\n");
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * The CPUs of fallback_case() and no_admission_affinity_moves(). I1 and I2
+ * are isolated by isolcpus="domain". H and K are housekeeping CPUs: CPUs
+ * that isolcpus does not isolate. H is above I2. K is another one.
+ *
+ * H is above I2 because of select_fallback_rq(). It gives a task that
+ * must leave its CPU the first CPU of its mask. A task allowed on I2 and
+ * H then goes to I2, unless the kernel keeps it in its root domain.
+ *
+ * fb_i1_hp says whether I1 can go offline.
+ */
+static int fb_i1 = -1, fb_i2 = -1, fb_h = -1, fb_k = -1, fb_i1_hp;
+
+/*
+ * Read total_bw of the root domain of the isolated CPUs into @def, and of
+ * the housekeeping root domain into @hk. Return 0, or -1.
+ */
+static int fallback_totals(long long *def, long long *hk)
+{
+	*def = total_bw_of(fb_i2);
+	*hk = total_bw_of(fb_h);
+	if (*def == -1 || *hk == -1) {
+		ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * Find I1, I2, H and K. Take a hotpluggable CPU as I1 if there is one.
+ * With @offline, I1 must be hotpluggable. Return 0, or -1 if the CPUs do
+ * not allow the case. fb_k is set only once all four are found.
+ */
+static int fallback_cpus(int offline)
+{
+	int i;
+
+	if (fb_k >= 0)
+		return offline && !fb_i1_hp ? -1 : 0;
+	/* A failed scan can leave I1, I2 or H set. Scan from scratch. */
+	fb_i1 = -1;
+	fb_i2 = -1;
+	fb_h = -1;
+	for (i = 0; i < nisol; i++) {
+		if (fb_i1 < 0 && cpu_hotpluggable(isol[i]))
+			fb_i1 = isol[i];
+		else if (fb_i2 < 0)
+			fb_i2 = isol[i];
+	}
+	fb_i1_hp = fb_i1 >= 0;
+	if (fb_i1 < 0 && nisol >= 2) {
+		fb_i1 = isol[0];
+		fb_i2 = isol[1];
+	}
+	for (i = 0; fb_i2 >= 0 && i < ncpus && fb_h < 0; i++) {
+		if (online[i] > fb_i2)
+			fb_h = online[i];
+	}
+	for (i = 0; fb_h >= 0 && i < ncpus && fb_k < 0; i++) {
+		if (online[i] != fb_h)
+			fb_k = online[i];
+	}
+	if (fb_i1 >= 0 && fb_i2 >= 0 && fb_k >= 0)
+		return offline && !fb_i1_hp ? -1 : 0;
+	fb_k = -1;
+	return -1;
+}
+
+enum fb_mode { FB_ON_I1, FB_ON_K, FB_ON_H, FB_MASK_H, FB_LAST_H, FB_PART_H, FB_STAY_K };
+
+/*
+ * B is a cpuset of I1, I2 and H. C is a cpuset of I1, in B. Neither is a
+ * partition. A deadline task of 10% is allowed on all CPUs. It sleeps on
+ * K or on H, and the housekeeping root domain counts it. The case moves
+ * it, asleep, into C or into B.
+ *
+ * In C, the task has I1 only. The move asks the root domain of the
+ * isolated CPUs for room, and that root domain counts the task. In B, the
+ * task still has H, a CPU of the housekeeping root domain. That root
+ * domain goes on counting it.
+ *
+ * The case then does one thing, and wakes the task. The table gives the
+ * cpuset the task is in, the CPU it sleeps on, what the case does, and
+ * the CPU the task must wake up on:
+ *
+ *   case                       in  asleep on  then                  wakes on
+ *   hotplug_fallback           C   I1         I1 goes offline       I2
+ *   hotplug_fallback_moved     C   K          I1 goes offline       I2
+ *   hotplug_fallback_in_mask   C   H          I1 goes offline       I2
+ *   mask_out_refused           C   H          C is given H only:    I1
+ *                                             EBUSY
+ *   hotplug_last_refused       B   H          H offline: EBUSY      H
+ *   hotplug_keeps_root_domain  B   H          H goes offline        K
+ *   partition_takes_refused    B   H          Q takes H: invalid    H
+ *
+ * hotplug_fallback wakes the task once after the move, so that it sleeps
+ * on I1. hotplug_keeps_root_domain gives B the CPU K as well. In
+ * partition_takes_refused, Q is a cpuset of H next to B. The case makes
+ * it a partition root, which must be invalid.
+ *
+ * With I1 offline, C has no CPU left, and the task gets the CPUs of B:
+ * I2 and H. It must wake up on I2, in the root domain that counts it. On
+ * H, it would run in a root domain that did not admit it. The three
+ * refusals have one reason: each change would leave the task no CPU in
+ * the root domain that counts it.
+ *
+ * In every case, the root domain of the CPU the task wakes on must count
+ * it, and the other one must not. The case checks the root domain of that
+ * CPU, not its number. The reference values of total_bw are taken with no
+ * task: with the CPU of the case offline if the case takes one offline,
+ * else with every CPU online. Once the task is gone, total_bw must be
+ * back to them. The messages give total_bw of the root domain of the
+ * isolated CPUs first, then of the housekeeping one.
+ */
+static int fallback_case(enum fb_mode mode)
+{
+	int in_b = mode == FB_LAST_H || mode == FB_PART_H || mode == FB_STAY_K;
+	int off_cpu;
+	int on_h = mode == FB_ON_H || mode == FB_MASK_H || in_b;
+	int no_offline = mode == FB_MASK_H || mode == FB_LAST_H || mode == FB_PART_H;
+	char cg_q[PATH_MAX];
+	int on_i1 = mode == FB_ON_I1;
+	char cg_b[PATH_MAX], cg_c[PATH_MAX], path[PATH_MAX + 32], buf[256], one[16];
+	long long def0, hk0, def_off = 0, hk_off = 0, def_ref, hk_ref, def, hk, bw;
+	int fd[2], ret = KSFT_SKIP, added = 0, made = 0, w, at, err;
+	pid_t pid = -1;
+
+	if (!root[0] || fallback_cpus(!no_offline)) {
+		ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs%s\n",
+			       no_offline ? "" : ", CPU hotplug");
+		return KSFT_SKIP;
+	}
+	off_cpu = mode == FB_STAY_K ? fb_h : fb_i1;
+	if ((mode == FB_LAST_H || mode == FB_STAY_K) && !cpu_hotpluggable(fb_h)) {
+		ksft_print_msg("CPU %d cannot go offline\n", fb_h);
+		return KSFT_SKIP;
+	}
+	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+	if (read_file(path, buf, sizeof(buf)))
+		return KSFT_SKIP;
+	if (!strstr(buf, "cpuset")) {
+		if (write_file(root, "cgroup.subtree_control", "+cpuset"))
+			return KSFT_SKIP;
+		added = 1;
+	}
+	if (snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) >= (int)sizeof(cg_q) ||
+	    snprintf(cg_b, sizeof(cg_b), "%s/deadline_bw_b", root) >= (int)sizeof(cg_b) ||
+	    snprintf(cg_c, sizeof(cg_c), "%s/c", cg_b) >= (int)sizeof(cg_c))
+		goto out;
+	if (mkdir(cg_b, 0755) && errno != EEXIST) {
+		ksft_print_msg("mkdir %s: %s\n", cg_b, strerror(errno));
+		goto out;
+	}
+	made = 1;
+	if (mode == FB_STAY_K)
+		snprintf(buf, sizeof(buf), "%d,%d,%d,%d", fb_i1, fb_i2, fb_h, fb_k);
+	else
+		snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+	snprintf(one, sizeof(one), "%d", fb_i1);
+	if (write_file(cg_b, "cpuset.cpus", buf) ||
+	    write_file(cg_b, "cgroup.subtree_control", "+cpuset") ||
+	    (mkdir(cg_c, 0755) && errno != EEXIST) ||
+	    write_file(cg_c, "cpuset.cpus", one)) {
+		ksft_print_msg("cpusets B and C cannot be made\n");
+		goto out;
+	}
+	if (fallback_totals(&def0, &hk0))
+		goto out;
+	ret = KSFT_FAIL;
+	if (!no_offline) {
+		if (cpu_online(off_cpu, 0)) {
+			ksft_print_msg("offline of CPU %d refused\n", off_cpu);
+			goto out;
+		}
+		/* H may be the offline CPU: read the housekeeping total_bw on K. */
+		def_off = total_bw_of(fb_i2);
+		hk_off = total_bw_of(fb_k);
+		if (def_off == -1 || hk_off == -1)
+			ret = KSFT_SKIP;
+		cpu_up(off_cpu);
+		if (ret == KSFT_SKIP)
+			goto out;
+	}
+
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		goto out;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = fork_child("a deadline task");
+	if (!pid) {
+		cpu_set_t set;
+		int i, err;
+		char c;
+
+		/* Run on K or H, then allow every CPU again, and stay there. */
+		close(fd[1]);
+		CPU_ZERO(&set);
+		CPU_SET(on_h ? fb_h : fb_k, &set);
+		err = sched_setaffinity(0, sizeof(set), &set);
+		for (i = 0; i < CPU_SETSIZE; i++)
+			CPU_SET(i, &set);
+		if (err || sched_setaffinity(0, sizeof(set), &set) ||
+		    set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		while (read(fd[0], &c, 1) == 1)
+			mv_state[1]++;
+		_exit(0);
+	}
+	close(fd[0]);
+	if (pid < 0 || child_up("hotplug_fallback", 0))
+		goto out_fd;
+	if (wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+		goto out_fd;
+	bw = def + hk - def0 - hk0;
+	if (bw <= 0) {
+		ksft_print_msg("the deadline task adds %lld to total_bw\n", bw);
+		goto out_fd;
+	}
+
+	/*
+	 * Move it asleep into C, or into B. For hotplug_fallback, wake it
+	 * once so that it runs on I1.
+	 */
+	if (enter(in_b ? cg_b : cg_c, pid) ||
+	    (on_i1 && fallback_wake(fd[1], pid)))
+		goto out_fd;
+	at = task_cpu_of(pid);
+	if (on_i1 ? at != fb_i1 : on_h ? at != fb_h : at == fb_i1) {
+		ksft_print_msg("the deadline task sleeps on CPU %d\n", at);
+		ret = on_h ? KSFT_SKIP : KSFT_FAIL;
+		goto out_fd;
+	}
+	/* With no CPU offline, the reference is total_bw with no task. */
+	def_ref = no_offline ? def0 : def_off;
+	hk_ref = no_offline ? hk0 : hk_off;
+	if (mode == FB_MASK_H) {
+		snprintf(one, sizeof(one), "%d", fb_h);
+		err = write_file(cg_c, "cpuset.cpus", one);
+		if (err != -EBUSY) {
+			ksft_print_msg("C given CPU %d only: %s, not EBUSY\n", fb_h,
+				       err ? strerror(-err) : "accepted");
+			goto out_fd;
+		}
+	} else if (mode == FB_LAST_H) {
+		err = cpu_online(fb_h, 0);
+		if (err != -EBUSY) {
+			ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+				       err ? strerror(-err) : "accepted");
+			cpu_up(fb_h);
+			goto out_fd;
+		}
+	} else if (mode == FB_PART_H) {
+		snprintf(one, sizeof(one), "%d", fb_h);
+		if ((mkdir(cg_q, 0755) && errno != EEXIST) ||
+		    write_file(cg_q, "cpuset.cpus", one) ||
+		    write_file(cg_q, "cpuset.cpus.partition", "root")) {
+			ksft_print_msg("cpuset Q cannot be made\n");
+			goto out_fd;
+		}
+		snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg_q);
+		if (read_file(path, buf, sizeof(buf)) || !strstr(buf, "Deadline task")) {
+			ksft_print_msg("Q takes CPU %d from B: %s", fb_h, buf);
+			goto out_fd;
+		}
+	} else if (cpu_online(off_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d with the task asleep refused\n", off_cpu);
+		goto out_fd;
+	}
+	if (fallback_wake(fd[1], pid))
+		goto out_up;
+	def = total_bw_of(fb_i2);
+	hk = total_bw_of(fb_k);
+	if (def == -1 || hk == -1)
+		goto out_up;
+	w = task_cpu_of(pid);
+	ret = KSFT_PASS;
+	if (in_cpu_list(isolated, w) ? def != def_ref + bw || hk != hk_ref :
+				     hk != hk_ref + bw || def != def_ref) {
+		ksft_print_msg("woken on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+			       w, def, hk,
+			       in_cpu_list(isolated, w) ? def_ref + bw : def_ref,
+			       in_cpu_list(isolated, w) ? hk_ref : hk_ref + bw);
+		ret = KSFT_FAIL;
+	} else if (in_cpu_list(isolated, w) == in_b) {
+		ksft_print_msg("woken on CPU %d: out of the root domain that counted it\n",
+			       w);
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	pid = -1;
+	sleep_ns(3 * PERIOD);
+	def = total_bw_of(fb_i2);
+	hk = total_bw_of(fb_k);
+	if (def != def_ref || hk != hk_ref) {
+		ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+			       def, hk, def_ref, hk_ref);
+		ret = KSFT_FAIL;
+	}
+out_up:
+	reap(pid);
+	cpu_up(off_cpu);
+	pid = -1;
+out_fd:
+	reap(pid);
+	close(fd[1]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+		ksft_print_msg("task gone, CPU %d online: total_bw %lld, %lld, not %lld, %lld\n",
+			       off_cpu, def, hk, def0, hk0);
+		ret = KSFT_FAIL;
+	}
+out:
+	if (made) {
+		if (rmdir(cg_q) && errno != ENOENT)
+			ksft_print_msg("rmdir %s: %s\n", cg_q, strerror(errno));
+		if (wait_for(1000, !rmdir(cg_c) || errno == ENOENT) ||
+		    wait_for(1000, !rmdir(cg_b) || errno == ENOENT))
+			ksft_print_msg("rmdir %s: %s\n", cg_b, strerror(errno));
+	}
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+	return ret;
+}
+
+static int hotplug_fallback(void)
+{
+	return fallback_case(FB_ON_I1);
+}
+
+static int hotplug_fallback_moved(void)
+{
+	return fallback_case(FB_ON_K);
+}
+
+static int hotplug_fallback_in_mask(void)
+{
+	return fallback_case(FB_ON_H);
+}
+
+static int mask_out_refused(void)
+{
+	return fallback_case(FB_MASK_H);
+}
+
+static int hotplug_last_refused(void)
+{
+	return fallback_case(FB_LAST_H);
+}
+
+static int hotplug_keeps_root_domain(void)
+{
+	return fallback_case(FB_STAY_K);
+}
+
+static int partition_takes_refused(void)
+{
+	return fallback_case(FB_PART_H);
+}
+
+/*
+ * Turn admission control off. A task pins itself to H, becomes a deadline
+ * task of 10% and spins. The housekeeping root domain counts it. The case
+ * then sets its affinity to I2 with sched_setaffinity(). Nothing reserves
+ * room for it: its bandwidth must follow it to the root domain of the
+ * isolated CPUs. Once the task is gone and admission control is on again,
+ * total_bw must be what it was.
+ */
+static int no_admission_affinity_moves(void)
+{
+	long long def0, hk0, def, hk, bw;
+	int ret = KSFT_FAIL, at;
+	char saved[64];
+	cpu_set_t set;
+	pid_t pid;
+
+	if (fallback_cpus(0)) {
+		ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (fallback_totals(&def0, &hk0))
+		return KSFT_SKIP;
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+	    write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+		ksft_print_msg("admission control cannot be turned off\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	pid = fork_child("a deadline task");
+	if (!pid) {
+		CPU_ZERO(&set);
+		CPU_SET(fb_h, &set);
+		if (sched_setaffinity(0, sizeof(set), &set) ||
+		    set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			spin_ns(NSEC_PER_MSEC);
+	}
+	if (pid < 0 || child_up("no_admission_affinity_moves", 0))
+		goto out;
+	sleep_ns(PERIOD);
+	if (fallback_totals(&def, &hk))
+		goto out;
+	bw = hk - hk0;
+	if (bw <= 0 || def != def0) {
+		ksft_print_msg("the deadline task adds %lld to total_bw of its root domain\n", bw);
+		goto out;
+	}
+	CPU_ZERO(&set);
+	CPU_SET(fb_i2, &set);
+	if (sched_setaffinity(pid, sizeof(set), &set)) {
+		ksft_print_msg("sched_setaffinity to CPU %d: %s\n", fb_i2, strerror(errno));
+		goto out;
+	}
+	sleep_ns(2 * PERIOD);
+	at = task_cpu_of(pid);
+	if (at != fb_i2 || fallback_totals(&def, &hk))
+		goto out;
+	ret = KSFT_PASS;
+	if (def != def0 + bw || hk != hk0) {
+		ksft_print_msg("on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+			       at, def, hk, def0 + bw, hk0);
+		ret = KSFT_FAIL;
+	}
+out:
+	reap(pid);
+	if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+		ksft_print_msg("admission control cannot be turned on again\n");
+		return KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+		ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+			       def, hk, def0, hk0);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Moves between two root domains, from the main process, with P */
+
+/*
+ * Fork a deadline task of 30ms every 100ms, pinned to the @n CPUs of
+ * @cpus. It runs 1ms, then sleeps. If @fd is not -1, each byte read from
+ * @fd wakes it up, and it counts the wakeups in mv_state[1]. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t pinned_sleeper(const int *cpus, int n, int fd)
+{
+	pid_t pid = fork_child("a pinned deadline task");
+
+	if (!pid) {
+		cpu_set_t set;
+		char c;
+		int i;
+
+		CPU_ZERO(&set);
+		for (i = 0; i < n; i++)
+			CPU_SET(cpus[i], &set);
+		if (sched_setaffinity(0, sizeof(set), &set) ||
+		    set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		spin_ns(NSEC_PER_MSEC);
+		mv_state[0] = 1;
+		while (fd >= 0 && read(fd, &c, 1) == 1)
+			mv_state[1]++;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * Read total_bw of three root domains into @bw: the root domain of the
+ * isolated CPUs, the housekeeping root domain, and P. Return 0, or -1.
+ */
+static int split_totals(long long bw[3])
+{
+	bw[0] = total_bw_of(isol[0]);
+	bw[1] = total_bw_of(online[0]);
+	bw[2] = total_bw_of(top_cpu(1));
+	if (bw[0] == -1 || bw[1] == -1 || bw[2] == -1) {
+		ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * Return KSFT_PASS if total_bw is now @want, else KSFT_FAIL. The message
+ * gives total_bw of the root domain of the isolated CPUs, then of the
+ * housekeeping root domain, then of P.
+ */
+static int split_check(const char *what, const long long want[3])
+{
+	long long bw[3];
+
+	if (split_totals(bw))
+		return KSFT_FAIL;
+	if (bw[0] == want[0] && bw[1] == want[1] && bw[2] == want[2])
+		return KSFT_PASS;
+	ksft_print_msg("%s: total_bw %lld, %lld, %lld, not %lld, %lld, %lld\n",
+		       what, bw[0], bw[1], bw[2], want[0], want[1], want[2]);
+	return KSFT_FAIL;
+}
+
+/*
+ * Read total_bw into @base, for the cases below. Return 0, or nonzero if
+ * the case must skip. It needs CPUs isolated by isolcpus="domain".
+ */
+static int split_skip(long long base[3])
+{
+	if (!nisol) {
+		ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+		return 1;
+	}
+	return split_totals(base);
+}
+
+/*
+ * The root cpuset holds the CPUs that isolcpus="domain" isolates, so it
+ * spans two root domains. The affinity of a task can keep it in either.
+ * A move into the root cpuset must reserve the bandwidth of the task in
+ * the root domain that the task gets.
+ *
+ * Run twice, with a deadline sleeper of 30% pinned to the isolated CPUs,
+ * then with another one pinned to the housekeeping CPUs except P's. Move
+ * the sleeper into P and back to the root cgroup, then kill it. Once it
+ * is gone, total_bw of the three root domains must be what it was.
+ */
+static int move_isolated(void)
+{
+	long long base[3];
+	int round, err;
+	pid_t pid;
+
+	if (split_skip(base))
+		return KSFT_SKIP;
+	for (round = 0; round < 2; round++) {
+		const char *what = round ? "pinned to the other CPUs" :
+				   "pinned to the isolated CPUs";
+
+		mv_state[0] = 0;
+		pid = round ? pinned_sleeper(online, ncpus - 1, -1) :
+			      pinned_sleeper(isol, nisol, -1);
+		if (pid < 0 || child_up(what, 0)) {
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		err = enter(cg_p, pid);
+		if (!err)
+			err = enter(root, pid);
+		reap(pid);
+		if (err) {
+			ksft_print_msg("%s: a move refused: %s\n", what, strerror(-err));
+			return KSFT_FAIL;
+		}
+		sleep_ns(3 * PERIOD);
+		if (split_check(what, base))
+			return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/*
+ * Fork a process pinned to the isolated CPUs. Its threads become deadline
+ * tasks of 10ms every 100ms, as in probe(), until one is refused. Then it
+ * sets mv_state[1] to 1, or to -1 on an error. The threads sleep until the
+ * process is killed.
+ */
+static pid_t fill_isolated(void)
+{
+	pid_t pid = fork_child("the tasks that fill the isolated CPUs");
+
+	if (!pid) {
+		int i, max = 16 * nisol + 16;
+		struct probe_slot *s = calloc(max, sizeof(*s));
+		pthread_attr_t a;
+		cpu_set_t set;
+
+		CPU_ZERO(&set);
+		for (i = 0; i < nisol; i++)
+			CPU_SET(isol[i], &set);
+		if (!s || sched_setaffinity(0, sizeof(set), &set) ||
+		    sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+		    pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK)) {
+			mv_state[1] = -1;
+			_exit(1);
+		}
+		for (i = 0; i < max; i++) {
+			if (pthread_create(&s[i].t, &a, probe_thread, &s[i]) ||
+			    sem_wait_for(&probe_done, 10000)) {
+				mv_state[1] = -1;
+				_exit(1);
+			}
+			if (s[i].err)
+				break;
+		}
+		mv_state[1] = i < max && s[i].err == -EBUSY ? 1 : -1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * Move a deadline sleeper pinned to the isolated CPUs into P. Fill the
+ * isolated CPUs with deadline tasks of 10%, until one is refused. Moving
+ * the sleeper back must then be refused with EBUSY.
+ */
+static int move_isolated_refused(void)
+{
+	pid_t pid, fill = -1;
+	long long base[3];
+	int err, ret = KSFT_FAIL;
+
+	if (split_skip(base))
+		return KSFT_SKIP;
+	mv_state[0] = 0;
+	pid = pinned_sleeper(isol, nisol, -1);
+	if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0))
+		goto out;
+	err = enter(cg_p, pid);
+	if (err) {
+		ksft_print_msg("a move into P refused: %s\n", strerror(-err));
+		goto out;
+	}
+	mv_state[1] = 0;
+	fill = fill_isolated();
+	if (fill < 0 || wait_for(10000, mv_state[1]) || mv_state[1] != 1) {
+		ksft_print_msg("the isolated CPUs cannot be filled\n");
+		goto out;
+	}
+	err = enter(root, pid);
+	if (err == -EBUSY) {
+		ret = KSFT_PASS;
+	} else {
+		ksft_print_msg("a move back to the full isolated CPUs: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+	}
+out:
+	reap(fill);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS)
+		ret = split_check("the isolated CPUs filled", base);
+	return ret;
+}
+
+/*
+ * A deadline sleeper is pinned to the isolated CPUs. The case allows it
+ * all CPUs, then moves it into P and back, without waking it. So it stays
+ * on an isolated CPU, which its mask holds again. The root domain of the
+ * isolated CPUs must still count its bandwidth. Then wake it: it must run
+ * on an isolated CPU, and that root domain must still count it.
+ */
+static int move_isolated_asleep(void)
+{
+	long long base[3], want[3];
+	int fd[2], err, cpu, i, ret = KSFT_FAIL;
+	cpu_set_t set;
+	pid_t pid;
+
+	if (split_skip(base))
+		return KSFT_SKIP;
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = pinned_sleeper(isol, nisol, fd[0]);
+	if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0) ||
+	    wait_for(1000, proc_state_of(pid) == 'S') || split_totals(want))
+		goto out;
+	CPU_ZERO(&set);
+	for (i = 0; i < CPU_SETSIZE; i++)
+		CPU_SET(i, &set);
+	err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+	if (!err)
+		err = enter(cg_p, pid);
+	if (!err)
+		err = enter(root, pid);
+	if (err) {
+		ksft_print_msg("the affinity of all CPUs, or a move: %s\n", strerror(-err));
+		goto out;
+	}
+	for (i = 0; i < 2; i++) {
+		const char *what = i ? "woken" : "moved back asleep";
+
+		cpu = task_cpu_of(pid);
+		if (!in_cpu_list(isolated, cpu)) {
+			ksft_print_msg("%s on CPU %d, not an isolated one\n", what, cpu);
+			goto out;
+		}
+		if (split_check(what, want))
+			goto out;
+		if (!i && fallback_wake(fd[1], pid))
+			goto out;
+	}
+	ret = KSFT_PASS;
+out:
+	reap(pid);
+	close(fd[0]);
+	close(fd[1]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS)
+		ret = split_check("task gone", base);
+	return ret;
+}
+
+/* The CPUs of move_spanning(): two root domains, an isolated CPU first */
+static cpu_set_t span_set;
+
+/*
+ * Fork a task that pins itself to the isolated CPUs, becomes a deadline
+ * task of 30ms every 100ms, then takes the CPUs of span_set. If @fd is
+ * -1, it spins and counts in mv_state[1]. Else it sleeps, each byte read
+ * from @fd wakes it up, and it counts the wakeups there. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t spanning_task(int fd)
+{
+	pid_t pid = fork_child("a deadline task on two root domains");
+
+	if (!pid) {
+		cpu_set_t set;
+		char c;
+		int i;
+
+		CPU_ZERO(&set);
+		for (i = 0; i < nisol; i++)
+			CPU_SET(isol[i], &set);
+		if (sched_setaffinity(0, sizeof(set), &set) ||
+		    set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD) ||
+		    sched_setaffinity(0, sizeof(span_set), &span_set)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		while (fd < 0)
+			mv_state[1]++;
+		while (read(fd, &c, 1) == 1)
+			mv_state[1]++;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/*
+ * A task becomes a deadline task on the isolated CPUs, where it is
+ * counted. Then it takes a wider affinity: the isolated CPUs, and the
+ * housekeeping CPUs above the first isolated one except P's. In the root
+ * cpuset, its CPUs span two root domains, and the lowest one is isolated.
+ * The case moves it into P, where it runs, and back to the root cgroup.
+ * The task must then run on a housekeeping CPU, and the housekeeping root
+ * domain must count its bandwidth. The root domain of the isolated CPUs
+ * must count it no more. Run twice: with a running task, then with a
+ * sleeper woken in P and again after the move back.
+ *
+ * The kernel counts a task that has a housekeeping CPU in the
+ * housekeeping root domain. The move reserves there, and a rebuild of the
+ * accounting makes the same choice. The lowest CPU of the task is
+ * isolated, so a move that takes the first CPU of the mask reserves in
+ * the wrong root domain.
+ */
+static int move_spanning(void)
+{
+	int fd[2] = { -1, -1 }, round, cpu, err, i, n = 0, ret = KSFT_PASS;
+	long long base[3], want[3];
+	pid_t pid;
+
+	if (split_skip(base))
+		return KSFT_SKIP;
+	CPU_ZERO(&span_set);
+	for (i = 0; i < nisol; i++)
+		CPU_SET(isol[i], &span_set);
+	for (i = 0; i < ncpus - 1; i++) {
+		if (online[i] > isol[0]) {
+			CPU_SET(online[i], &span_set);
+			n++;
+		}
+	}
+	if (!n) {
+		ksft_print_msg("needs a CPU above the first isolated one, besides P\n");
+		return KSFT_SKIP;
+	}
+	for (round = 0; round < 2 && ret == KSFT_PASS; round++) {
+		const char *what = round ? "a sleeper" : "a running task";
+
+		ret = KSFT_FAIL;
+		if (round && pipe(fd)) {
+			ksft_print_msg("pipe: %s\n", strerror(errno));
+			break;
+		}
+		mv_state[0] = 0;
+		mv_state[1] = 0;
+		pid = spanning_task(fd[0]);
+		if (pid < 0 || child_up(what, 0) || split_totals(want))
+			goto next;
+		/*
+		 * Moved back, its bandwidth goes to the housekeeping root
+		 * domain, not to the root domain of the isolated CPUs.
+		 */
+		want[1] += want[0] - base[0];
+		want[0] = base[0];
+		err = enter(cg_p, pid);
+		if (err) {
+			ksft_print_msg("%s: a move into P refused: %s\n", what, strerror(-err));
+			goto next;
+		}
+		if ((round && fallback_wake(fd[1], pid)) ||
+		    wait_for(1000, task_cpu_of(pid) == top_cpu(1))) {
+			ksft_print_msg("%s: not on P's CPU\n", what);
+			goto next;
+		}
+		err = enter(root, pid);
+		if (err) {
+			ksft_print_msg("%s: the move back refused: %s\n", what, strerror(-err));
+			goto next;
+		}
+		if (round ? fallback_wake(fd[1], pid) :
+			    wait_for(1000, task_cpu_of(pid) != top_cpu(1)))
+			goto next;
+		sleep_ns(PERIOD);
+		cpu = task_cpu_of(pid);
+		if (in_cpu_list(isolated, cpu)) {
+			ksft_print_msg("%s: moved back to isolated CPU %d\n", what, cpu);
+			goto next;
+		}
+		ret = split_check(what, want);
+next:
+		reap(pid);
+		if (round) {
+			close(fd[0]);
+			close(fd[1]);
+		}
+		sleep_ns(3 * PERIOD);
+		if (ret == KSFT_PASS)
+			ret = split_check("task gone", base);
+	}
+	return ret;
+}
+
+/*
+ * A deadline task moved asleep into another cpuset still sleeps on a CPU
+ * of the root domain it left. The root domain it moved to counts its
+ * bandwidth. sched_setaffinity() must check a new mask against the root
+ * domain that counts the task, not the one it sleeps in.
+ *
+ * H is a cpuset here, not the CPU of fallback_case(). It holds the
+ * housekeeping CPUs except P's. The steps:
+ * 1. Pin a fair task to the isolated CPUs.
+ * 2. Move it into H. Its affinity has no CPU of H, so it gets all the
+ *    CPUs of H.
+ * 3. Make it a deadline task there, and let it sleep.
+ * 4. Move it asleep into the root cgroup. Its affinity gives it the
+ *    isolated CPUs again, and their root domain counts it.
+ * 5. Set its affinity to the CPUs of H. That would take it out of the
+ *    root domain that counts it, so it must fail with EBUSY.
+ *
+ * If the change is accepted, wake the task and report where it runs.
+ */
+static int affinity_moved_asleep(void)
+{
+	char cg_h[PATH_MAX], cpus[4096], what[64];
+	int fd[2], i, n = 0, cpu, err, made = 0, ret = KSFT_SKIP;
+	long long base[3], want[3], bw;
+	cpu_set_t set;
+	pid_t pid = -1;
+
+	if (split_skip(base))
+		return KSFT_SKIP;
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	/*
+	 * H holds the housekeeping CPUs except P's. A task there can become
+	 * a deadline task: its mask holds its whole root domain.
+	 */
+	CPU_ZERO(&set);
+	for (i = 0; i < ncpus - 1 && n < (int)sizeof(cpus); i++) {
+		n += snprintf(cpus + n, sizeof(cpus) - n, "%s%d", i ? "," : "", online[i]);
+		CPU_SET(online[i], &set);
+	}
+	if (snprintf(cg_h, sizeof(cg_h), "%s/deadline_bw_h", root) >= (int)sizeof(cg_h))
+		goto out;
+	if (mkdir(cg_h, 0755) && errno != EEXIST) {
+		ksft_print_msg("mkdir %s: %s\n", cg_h, strerror(errno));
+		goto out;
+	}
+	made = 1;
+	if (n >= (int)sizeof(cpus) || write_file(cg_h, "cpuset.cpus", cpus)) {
+		ksft_print_msg("cpuset H cannot be made\n");
+		goto out;
+	}
+	ret = KSFT_FAIL;
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = fork_child("a task moved asleep");
+	if (!pid) {
+		cpu_set_t iso;
+		char c;
+
+		CPU_ZERO(&iso);
+		for (i = 0; i < nisol; i++)
+			CPU_SET(isol[i], &iso);
+		if (sched_setaffinity(0, sizeof(iso), &iso)) {
+			mv_state[1] = -1;
+			_exit(1);
+		}
+		mv_state[1] = 1;
+		/* Wait for the move into H. Become a deadline task there. */
+		if (read(fd[0], &c, 1) != 1 || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		spin_ns(NSEC_PER_MSEC);
+		mv_state[0] = 1;
+		while (read(fd[0], &c, 1) == 1)
+			mv_state[1]++;
+		for (;;)
+			pause();
+	}
+	if (pid < 0)
+		goto out;
+	if (wait_for(1000, mv_state[1]) || mv_state[1] != 1) {
+		ksft_print_msg("the task cannot take the isolated CPUs\n");
+		goto out;
+	}
+	err = enter(cg_h, pid);
+	if (err) {
+		ksft_print_msg("a move into H refused: %s\n", strerror(-err));
+		goto out;
+	}
+	if (write(fd[1], "d", 1) != 1) {
+		ksft_print_msg("write to the task: %s\n", strerror(errno));
+		goto out;
+	}
+	if (child_up("a task moved into H", 0) || wait_for(1000, proc_state_of(pid) == 'S'))
+		goto out;
+	bw = (30 * NSEC_PER_MSEC << 20) / PERIOD;	/* as to_ratio() */
+	memcpy(want, base, sizeof(want));
+	want[1] += bw;
+	if (split_check("a deadline task in H", want))
+		goto out;
+
+	/*
+	 * Moved asleep, it stays on its CPU. The root domain of the
+	 * isolated CPUs counts it.
+	 */
+	err = enter(root, pid);
+	if (err) {
+		ksft_print_msg("the move into the root cgroup refused: %s\n", strerror(-err));
+		goto out;
+	}
+	cpu = task_cpu_of(pid);
+	if (in_cpu_list(isolated, cpu)) {
+		ksft_print_msg("moved asleep on CPU %d, an isolated one\n", cpu);
+		goto out;
+	}
+	want[0] += bw;
+	want[1] -= bw;
+	if (split_check("moved asleep", want))
+		goto out;
+
+	/* The CPUs of H leave out the root domain that counts the task. */
+	err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+	if (err == -EBUSY) {
+		ret = KSFT_PASS;
+		goto out;
+	}
+	ksft_print_msg("the affinity of the CPUs of H: %s, not EBUSY\n",
+		       err ? strerror(-err) : "accepted");
+	if (err || fallback_wake(fd[1], pid))
+		goto out;
+	/* Report its CPU, and whether the root domain of that CPU counts it. */
+	cpu = task_cpu_of(pid);
+	memcpy(want, base, sizeof(want));
+	want[in_cpu_list(isolated, cpu) ? 0 : 1] += bw;
+	snprintf(what, sizeof(what), "woken on CPU %d", cpu);
+	if (split_check(what, want) == KSFT_PASS)
+		ksft_print_msg("%s: its root domain counts it\n", what);
+out:
+	reap(pid);
+	close(fd[0]);
+	close(fd[1]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS)
+		ret = split_check("task gone", base);
+	if (made && wait_for(1000, !rmdir(cg_h) || errno == ENOENT))
+		ksft_print_msg("rmdir %s: %s\n", cg_h, strerror(errno));
+	return ret;
+}
+
+/* Setup */
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+/* sched_setaffinity() against sched_setattr(), with no P */
+
+/*
+ * The state that affinity_racer() shares with the case. pid and one are
+ * its target and its mask. go and done count the rounds. ret is the
+ * result of its last call.
+ */
+static struct {
+	pid_t pid;
+	cpu_set_t one;
+	atomic_int go;
+	atomic_int done;
+	atomic_int stop;
+	int ret;
+} aff;
+
+/*
+ * Give aff.pid the affinity aff.one, once for each new value of aff.go.
+ * Return when aff.stop is set.
+ */
+static void *affinity_racer(void *arg)
+{
+	int round;
+
+	for (round = 1; ; round++) {
+		while (aff.go < round && !aff.stop)
+			;
+		if (aff.stop)
+			break;
+		aff.ret = sched_setaffinity(aff.pid, sizeof(aff.one), &aff.one) ? -errno : 0;
+		aff.done = round;
+	}
+	return arg;
+}
+
+/*
+ * Race sched_setaffinity() against sched_setattr() on a sleeping task.
+ * One thread gives the task an affinity of one CPU. Another one makes it
+ * a deadline task. A deadline task must be allowed on its whole root
+ * domain, so at most one of the two calls may succeed:
+ * - sched_setattr() fails with EPERM if the affinity came first;
+ * - sched_setaffinity() fails with EBUSY if the policy came first.
+ *
+ * Do it 1000 times. Each round starts with the task back in SCHED_OTHER
+ * with its full affinity. Once the task is gone, the free bandwidth must
+ * be what it was. The case needs no cpuset: it runs without P, with the
+ * hotplug cases.
+ */
+static int affinity_vs_setattr(void)
+{
+	const char *name = "affinity_vs_setattr";
+	int base, i, cpu, both = 0, ret = KSFT_PASS;
+	cpu_set_t all;
+	pthread_t t;
+	pid_t pid;
+
+	base = probe(NULL);
+	pid = fork_child(name);
+	if (!pid) {
+		for (;;)
+			pause();
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (sched_getaffinity(pid, sizeof(all), &all)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	CPU_ZERO(&aff.one);
+	for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+		if (CPU_ISSET(cpu, &all)) {
+			CPU_SET(cpu, &aff.one);
+			break;
+		}
+	}
+	aff.pid = pid;
+	aff.go = 0;
+	aff.done = 0;
+	aff.stop = 0;
+	if (pthread_create(&t, NULL, affinity_racer, NULL)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+
+	for (i = 1; i <= 1000 && ret == KSFT_PASS; i++) {
+		int err;
+
+		if (set_other(pid) || sched_setaffinity(pid, sizeof(all), &all)) {
+			ksft_print_msg("%s: the task cannot be reset: %s\n", name, strerror(errno));
+			ret = KSFT_FAIL;
+			break;
+		}
+		aff.go = i;
+		err = set_deadline(pid, 2 * NSEC_PER_USEC, PERIOD);
+		while (aff.done < i)
+			;
+		if (!err && !aff.ret) {
+			both++;
+		} else if ((err && err != -EPERM) || (aff.ret && aff.ret != -EBUSY)) {
+			ksft_print_msg("%s: round %d: sched_setattr: %s, sched_setaffinity: %s\n",
+				       name, i, strerror(-err), strerror(-aff.ret));
+			ret = KSFT_FAIL;
+		}
+	}
+	aff.stop = 1;
+	join(t);
+	if (both) {
+		ksft_print_msg("%s: %d rounds of %d: a deadline task with an affinity of one CPU\n",
+			       name, both, i - 1);
+		ret = KSFT_FAIL;
+	}
+	/* A deadline task of 2us every 100ms would take seconds to exit. */
+	if (set_other(pid))
+		ksft_print_msg("%s: the task stays a deadline task\n", name);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits(base, "after the task is gone"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* The tables of the cases, and main() */
+
+enum where {
+	MEASURED_IN_P,	/* in a child in P, probed before and after */
+	IN_P,		/* in a child in P, which measures itself */
+	FROM_OUTSIDE,	/* in the main process, outside P */
+};
+
+static const struct p_case {
+	const char *name;
+	int (*fn)(void);
+	enum where where;
+} p_cases[] = {
+	{ "pi_deadline_owner", pi_deadline_owner, MEASURED_IN_P },
+	{ "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
+	{ "pi_former_owner_asleep", pi_former_owner_asleep, MEASURED_IN_P },
+	{ "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
+	{ "pi_chain_two_owners", pi_chain_two_owners, MEASURED_IN_P },
+	{ "pi_departed_moved", pi_departed_moved, FROM_OUTSIDE },
+	{ "params_asleep", params_asleep, IN_P },
+	{ "params_non_contending", params_non_contending, MEASURED_IN_P },
+	{ "params_throttled", params_throttled, MEASURED_IN_P },
+	{ "params_refused", params_refused, IN_P },
+	{ "params_keep_class_change", params_keep_class_change, MEASURED_IN_P },
+	{ "params_keep_boosted", params_keep_boosted, MEASURED_IN_P },
+	{ "boosted_owner_policy", boosted_owner_policy, MEASURED_IN_P },
+	{ "deboost_resets_period", deboost_resets_period, MEASURED_IN_P },
+	{ "deboost_released_resets_period", deboost_released_resets_period, MEASURED_IN_P },
+	{ "deboost_in_class_resets_period", deboost_in_class_resets_period, MEASURED_IN_P },
+	{ "deboost_after_donor_refill", deboost_after_donor_refill, MEASURED_IN_P },
+	{ "deboost_keeps_own_runtime", deboost_keeps_own_runtime, MEASURED_IN_P },
+	{ "return_running_new_params", return_running_new_params, MEASURED_IN_P },
+	{ "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
+	{ "change_running_new_params", change_running_new_params, MEASURED_IN_P },
+	{ "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
+	{ "setattr_blocked_keeps_order", setattr_blocked_keeps_order, MEASURED_IN_P },
+	{ "boost_blocked_keeps_order", boost_blocked_keeps_order, MEASURED_IN_P },
+	{ "move_asleep", move_asleep, FROM_OUTSIDE },
+	{ "move_throttled", move_throttled, FROM_OUTSIDE },
+	{ "move_non_contending", move_non_contending, FROM_OUTSIDE },
+	{ "move_refused", move_refused, FROM_OUTSIDE },
+	{ "move_zero_bw", move_zero_bw, FROM_OUTSIDE },
+	{ "move_vs_rebuild", move_vs_rebuild, FROM_OUTSIDE },
+	{ "move_became_deadline", move_became_deadline, FROM_OUTSIDE },
+	{ "move_became_deadline_no_room", move_became_deadline_no_room, FROM_OUTSIDE },
+	{ "move_affinity_changed", move_affinity_changed, FROM_OUTSIDE },
+	{ "move_affinity_became_deadline", move_affinity_became_deadline, FROM_OUTSIDE },
+	{ "move_died", move_died, FROM_OUTSIDE },
+	{ "move_woken", move_woken, FROM_OUTSIDE },
+	{ "move_woken_full", move_woken_full, FROM_OUTSIDE },
+	{ "move_cancelled", move_cancelled, FROM_OUTSIDE },
+	{ "frozen_moved_switched", frozen_moved_switched, FROM_OUTSIDE },
+	{ "move_isolated", move_isolated, FROM_OUTSIDE },
+	{ "move_isolated_refused", move_isolated_refused, FROM_OUTSIDE },
+	{ "move_isolated_asleep", move_isolated_asleep, FROM_OUTSIDE },
+	{ "move_spanning", move_spanning, FROM_OUTSIDE },
+	{ "affinity_moved_asleep", affinity_moved_asleep, FROM_OUTSIDE },
+	{ "death_running", death_running, MEASURED_IN_P },
+	{ "death_switched_out", death_switched_out, MEASURED_IN_P },
+	{ "death_shrunk", death_shrunk, MEASURED_IN_P },
+	{ "stopped_switched_out", stopped_switched_out, MEASURED_IN_P },
+	{ "death_killed_asleep", death_killed_asleep, MEASURED_IN_P },
+	{ "death_killed_non_contending", death_killed_non_contending, MEASURED_IN_P },
+	{ "death_killed_throttled", death_killed_throttled, MEASURED_IN_P },
+	{ "death_killed_running", death_killed_running, MEASURED_IN_P },
+	{ "death_pi_waiter", death_pi_waiter, MEASURED_IN_P },
+	{ "death_pi_owner", death_pi_owner, MEASURED_IN_P },
+	{ "death_pi_deadline_owner", death_pi_deadline_owner, MEASURED_IN_P },
+	{ "death_group", death_group, MEASURED_IN_P },
+	{ "death_zombie_switched", death_zombie_switched, FROM_OUTSIDE },
+	{ "death_zombie_hotplug", death_zombie_hotplug, FROM_OUTSIDE },
+	{ "yield_switched_out", yield_switched_out, IN_P },
+	{ "yield_moved", yield_moved, FROM_OUTSIDE },
+	{ "yield_killed", yield_killed, MEASURED_IN_P },
+	{ "overrun_caught", overrun_caught, IN_P },
+	{ "overrun_group_death", overrun_group_death, MEASURED_IN_P },
+	{ "reclaim_runs", reclaim_runs, MEASURED_IN_P },
+	{ "reclaim_switched", reclaim_switched, MEASURED_IN_P },
+	{ "reclaim_moved", reclaim_moved, FROM_OUTSIDE },
+	{ "server_params", server_params, IN_P },
+	{ "server_rebuilds", server_rebuilds, FROM_OUTSIDE },
+	{ "server_states", server_states, FROM_OUTSIDE },
+	{ "server_reclaim", server_reclaim, MEASURED_IN_P },
+	{ "params_waking", params_waking, FROM_OUTSIDE },
+	{ "random_mix", random_mix, FROM_OUTSIDE },
+	{ "move_normalized", move_normalized, FROM_OUTSIDE },
+};
+
+#define N_P_CASES	((int)ARRAY_SIZE(p_cases))
+
+static const struct hp_case {
+	const char *name;
+	int (*fn)(void);
+	int no_hp;		/* takes no CPU offline */
+} hp_cases[] = {
+	{ "hotplug_sleepers", hotplug_sleepers },
+	{ "hotplug_sleeper_switched", hotplug_sleeper_switched },
+	{ "hotplug_no_admission_pinned", hotplug_no_admission_pinned },
+	{ "hotplug_refused", hotplug_refused },
+	{ "hotplug_departed", hotplug_departed },
+	{ "hotplug_server", hotplug_server },
+	{ "hotplug_server_refused", hotplug_server_refused },
+	{ "hotplug_fallback", hotplug_fallback },
+	{ "hotplug_fallback_moved", hotplug_fallback_moved },
+	{ "hotplug_fallback_in_mask", hotplug_fallback_in_mask },
+	{ "mask_out_refused", mask_out_refused, 1 },
+	{ "hotplug_last_refused", hotplug_last_refused },
+	{ "hotplug_keeps_root_domain", hotplug_keeps_root_domain },
+	{ "partition_takes_refused", partition_takes_refused, 1 },
+	{ "no_admission_affinity_moves", no_admission_affinity_moves, 1 },
+	{ "affinity_vs_setattr", affinity_vs_setattr, 1 },
+};
+
+#define N_HP_CASES	((int)ARRAY_SIZE(hp_cases))
+
+/* Run the cases of p_cases[], each with a P made anew. */
+static void run_p_cases(void)
+{
+	char path[PATH_MAX + 32], buf[256];
+	int i, ret, added = 0, told = 0, ok;
+
+	ok = !find_cgroup2() &&
+	     snprintf(cg_p, sizeof(cg_p), "%s/deadline_bw_p", root) < (int)sizeof(cg_p) &&
+	     snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) < (int)sizeof(cg_q);
+	if (ok) {
+		/*
+		 * Enable the cpuset controller for the children of the
+		 * root while the cases run.
+		 */
+		snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+		ok = !read_file(path, buf, sizeof(buf));
+		added = ok && !strstr(buf, "cpuset");
+		if (added && write_file(root, "cgroup.subtree_control", "+cpuset")) {
+			ok = 0;
+			added = 0;
+		}
+	}
+	for (i = 0; i < N_P_CASES; i++) {
+		const struct p_case *c = &p_cases[i];
+
+		if (ok && make_p()) {
+			remove_p();
+			ok = 0;
+		}
+		if (!ok) {
+			if (!told++)
+				ksft_print_msg("no one-CPU partition root: cases in P skipped\n");
+			ksft_test_result_skip("%s\n", c->name);
+			continue;
+		}
+		if (c->where == MEASURED_IN_P)
+			ret = measured(c->name, cg_p, c->fn);
+		else if (c->where == IN_P)
+			ret = run_in(cg_p, c->fn);
+		else
+			ret = c->fn();
+		remove_p();
+		server_restore();
+		report(c->name, ret);
+	}
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+/*
+ * Read the online CPUs into *@cpus, in ascending order. Read those that
+ * isolcpus="domain" isolates if @iso, the others if not. Return their
+ * count, or -1.
+ */
+static int read_online(int **cpus, int iso)
+{
+	static char buf[65536];		/* a sysfs list fits in a page */
+	char *p = buf;
+	int n = 0, max = 0;
+
+	if (read_file("/sys/devices/system/cpu/online", buf, sizeof(buf)) ||
+	    strlen(buf) == sizeof(buf) - 1)
+		return -1;
+	while (*p >= '0' && *p <= '9') {
+		long lo = strtol(p, &p, 10), hi = lo;
+
+		if (*p == '-')
+			hi = strtol(p + 1, &p, 10);
+		for (; lo <= hi; lo++) {
+			if (in_cpu_list(isolated, lo) != iso)
+				continue;
+			if (n == max) {
+				int *more = realloc(*cpus, (max = 2 * max + 64) * sizeof(**cpus));
+
+				if (!more)
+					return -1;
+				*cpus = more;
+			}
+			(*cpus)[n++] = lo;
+		}
+		if (*p == ',')
+			p++;
+	}
+	return n;
+}
+
+int main(void)
+{
+	char buf[64];
+	cpu_set_t set;
+	int i;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	/* No such file: nothing isolated. */
+	if (read_file("/sys/devices/system/cpu/isolated", isolated, sizeof(isolated)))
+		isolated[0] = 0;
+	ncpus = read_online(&online, 0);
+	nisol = read_online(&isol, 1);
+	if (ncpus < 0 || nisol < 0)
+		ksft_exit_fail_msg("the online CPUs cannot be read\n");
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs not isolated by isolcpus=domain\n");
+	/*
+	 * Keep the tasks of the test off the isolated CPUs: a deadline task
+	 * there is admitted in the root domain of the isolated CPUs. Use
+	 * sched_setaffinity() for that. A change of the partitions gives the
+	 * tasks of the root cpuset all its CPUs again, the isolated CPUs
+	 * too. But the kernel keeps them within an affinity set by
+	 * sched_setaffinity(). The children inherit this affinity.
+	 */
+	CPU_ZERO(&set);
+	for (i = 0; i < ncpus; i++)
+		CPU_SET(online[i], &set);
+	if (sched_setaffinity(0, sizeof(set), &set))
+		ksft_exit_fail_msg("sched_setaffinity: %s\n", strerror(errno));
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	mv_state = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (mv_state == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	shm_lock = (pthread_mutex_t *)((char *)mv_state + 256);
+
+	ksft_set_plan(N_P_CASES + N_HP_CASES);
+	server_init();
+	run_p_cases();
+
+	for (i = 1; i < ncpus && hp_cpu < 0; i++) {
+		if (cpu_hotpluggable(top_cpu(i)))
+			hp_cpu = top_cpu(i);
+	}
+	if (hp_cpu < 0)
+		ksft_print_msg("no CPU hotplug: hotplug cases skipped\n");
+	for (i = 0; i < N_HP_CASES; i++) {
+		if (hp_cpu < 0 && !hp_cases[i].no_hp) {
+			ksft_test_result_skip("%s\n", hp_cases[i].name);
+			continue;
+		}
+		report(hp_cases[i].name, hp_cases[i].fn());
+		server_restore();
+	}
+	ksft_finished();
+}
diff --git a/tools/testing/selftests/sched/deadline/settings b/tools/testing/selftests/sched/deadline/settings
new file mode 100644
index 000000000000..a62d2fa1275c
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/settings
@@ -0,0 +1 @@
+timeout=600
-- 
2.47.3


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline
  2026-11-16 19:05 ` [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
@ 2026-11-16 19:30   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:30 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/debug: sched_server_write_common() misses CPU
  deactivation window
--

commit 20746e8301600abc34772d08e7fcdc3dbeb36656
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Refuse dl_server debugfs writes while the rq is offline

This commit replaces the cpu_online() check in sched_server_write_common()
with an rq->online check to reject dl_server debugfs writes while a runqueue
is offline during CPU hotplug transitions.

> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> index e6a3b516c703b..e0d6cd3bd607d 100644
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -410,7 +410,15 @@ static ssize_t sched_server_write_common(struct file *filp, const char __user *u
>  			return  -EINVAL;
>  		}
>  
> -		if (!cpu_online(cpu_of(rq)))
> +		/*
> +		 * Refuse the write while the runqueue is offline. This
> +		 * includes the windows where the CPU goes offline or comes
> +		 * online. The root domain then does not count the server's
> +		 * bandwidth, and may have no active CPU. rq->online is set
> +		 * under this lock when the CPU joins a root domain that
> +		 * counts it. cpu_active() is set before that.
> +		 */
> +		if (!rq->online)
>  			return -EBUSY;

[Severity: Medium]
Can a debugfs write still slip through during sched_cpu_deactivate() after
cpu_active(cpu) is cleared?

CPU0                              CPU1
sched_cpu_deactivate(cpu)
  set_cpu_active(cpu, false)
  synchronize_rcu()
                                  sched_server_write_common()
                                    guard(rq_lock_irqsave)(rq)
                                    if (!rq->online) // still 1
                                    dl_server_apply_params()
                                    dl_server_start()
  sched_set_rq_offline(rq, cpu)
    rq->online = 0

During the synchronize_rcu() window, rq->online is still 1 and the root
domain has not yet been rebuilt by cpuset_cpu_inactive(), but cpu_active(cpu)
is already false.  In dl_server_apply_params(), dl_bw_cpus(cpu) and
dl_bw_capacity(cpu) weight rd->span against cpu_active_mask, so they exclude
the deactivating CPU while dl_b->total_bw still includes its server
bandwidth.  If this was the last active CPU in its root domain, cpus and cap
are 0; if other active CPUs remain, __dl_overflow() checks N servers against
N - 1 CPUs' capacity and __dl_sub() / __dl_add() miscalculate extra_bw
across N - 1 CPUs.  dl_server_start() also re-enqueues the server on the
deactivating runqueue after balance_push_set(cpu, true).

Checking (!rq->online || !cpu_active(cpu_of(rq))) under the rq lock would
cover both the activation and deactivation windows.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=5

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach
  2026-11-16 19:05 ` [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
@ 2026-11-16 19:31   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:31 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched: broken cpuset rollback in __sched_setaffinity() with
  SCA_DL_NOWAIT
--

commit 9d195b268d19186fd430b17e56b76c3ac49f0d39
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched: Make deadline policy and affinity changes wait for a cpuset attach

This commit introduces cpuset_lock_no_attach() and SCA_DL_NOWAIT so that
SCHED_DEADLINE policy changes in __sched_setscheduler() and affinity
changes in sched_setaffinity() wait for an in-progress cpuset attach to
finish.

> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 2a964286a7cd0..52c291de384e7 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3195,6 +3195,16 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
>  		goto out;
>  	}
>  
> +	/*
> +	 * sched_setaffinity() tested the policy with no lock, and did not
> +	 * wait for a cpuset attach in progress. @p became a deadline task
> +	 * since: let the caller wait and try again.
> +	 */
> +	if ((ctx->flags & SCA_DL_NOWAIT) && task_has_dl_policy(p)) {
> +		ret = -EAGAIN;
> +		goto out;
> +	}
> +
>  	/*
>  	 * Check under the same lock that __sched_setscheduler() holds for
>  	 * its admission test, so that p cannot become SCHED_DEADLINE
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 4f47b17151ed9..70b364b4f82dd 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c

[ ... ]

> @@ -1318,13 +1324,31 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
>  		return -ENOMEM;
>  	}
>  
> -	ac = (struct affinity_context){
> -		.new_mask  = in_mask,
> -		.user_mask = user_mask,
> -		.flags     = SCA_USER,
> -	};
> -
> -	retval = __sched_setaffinity(p, &ac);
> +	/*
> +	 * Wait for a cpuset attach in progress before changing the affinity
> +	 * of a deadline task. cpuset_can_attach() reserved the bandwidth of
> +	 * the task for the CPUs it moves to, and cpuset_attach() moves it
> +	 * there. A new affinity in between could change those CPUs.
> +	 *
> +	 * The policy is tested here with no lock. A task that becomes a
> +	 * deadline task after this test is found under its runqueue lock,
> +	 * with SCA_DL_NOWAIT. Wait and try again then.
> +	 */
> +	for (;;) {
> +		dl = task_has_dl_policy(p);
> +		ac = (struct affinity_context){
> +			.new_mask  = in_mask,
> +			.user_mask = user_mask,
> +			.flags     = SCA_USER | (dl ? 0 : SCA_DL_NOWAIT),
> +		};
> +		if (dl)
> +			cpuset_lock_no_attach();
> +		retval = __sched_setaffinity(p, &ac);
> +		if (dl)
> +			cpuset_unlock();
> +		if (dl || retval != -EAGAIN)
> +			break;
> +	}
>  	kfree(ac.user_mask);
>  
>  	return retval;

[Severity: Medium]
Can SCA_DL_NOWAIT cause the cpuset race rollback in __sched_setaffinity()
to fail and leave p->cpus_mask and p->user_cpus_ptr un-restored?

When p is initially not a deadline task, sched_setaffinity() passes
SCA_DL_NOWAIT without holding cpuset_mutex.  If the first
__set_cpus_allowed_ptr() succeeds while a concurrent cpuset update narrows
cpus_allowed and another thread changes p to SCHED_DEADLINE, the recovery
path in __sched_setaffinity() clears SCA_DL_CHECK but leaves SCA_DL_NOWAIT
set:

kernel/sched/syscalls.c:__sched_setaffinity() {
    ...
	cpuset_cpus_allowed(p, cpus_allowed);
	if (!cpumask_subset(new_mask, cpus_allowed)) {
    ...
		ctx->flags &= ~SCA_DL_CHECK;
		__set_cpus_allowed_ptr(p, ctx);
		retval = -EINVAL;
	}
    ...
}

CPU0                                  CPU1
sched_setaffinity(p)
  dl = false (SCA_DL_NOWAIT set)
  __sched_setaffinity(p, &ac)
    cpuset_cpus_allowed(p, ...)
    __set_cpus_allowed_ptr(p, ctx)
      sets p->cpus_mask = new_mask
      swaps p->user_cpus_ptr
        with ctx->user_mask
                                      cpuset narrows p's cpus_allowed
                                      sched_setattr(p, SCHED_DEADLINE)
    cpuset_cpus_allowed(p, ...)
    !cpumask_subset(new_mask, ...)
    ctx->flags &= ~SCA_DL_CHECK
    __set_cpus_allowed_ptr(p, ctx)
      SCA_DL_NOWAIT && dl_policy
      returns -EAGAIN (ignored)  <-- rollback skipped
    returns -EINVAL
  kfree(ac.user_mask)

Because the second __set_cpus_allowed_ptr() aborts with -EAGAIN and its
return value is ignored, p->cpus_mask is left containing CPUs outside the
task's cpuset cpus_allowed, and the rejected user_mask remains installed
in p->user_cpus_ptr while the previous user_cpus_ptr is freed.

Clearing SCA_DL_NOWAIT alongside SCA_DL_CHECK before the recovery
__set_cpus_allowed_ptr() call would allow the rollback to complete.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=21

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class
  2026-11-16 19:05 ` [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
@ 2026-11-16 19:38   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:38 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [Medium] sched/deadline: implicit declaration of
  dl_note_donor_refill()
- [Medium] sched/deadline: rt_mutex_setprio() skips dl_deboost() on
  donor change
- [Medium] sched/deadline: donor runtime kept when leaving
  dl_sched_class
- [Medium] sched/deadline: missing dl_note_donor_refill() on cleared
  donor
- [Low] sched/deadline: __dl_clear_params() fails to clear
  dl_donor_refill
--

commit 25ef3546a7248ed2f828d8c39dc34ecf7474dcc4
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Reset the donor's runtime at a deboost in the class

This commit adds a dl_donor_refill flag to struct sched_dl_entity to track
when a priority-inherited SCHED_DEADLINE task has been replenished from its
donor's reservation, resetting its period on in-class deboost.

> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 5ed660f099a55..a22df8b43a1a7 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7834,16 +7834,26 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
>  				p->dl.pi_se = pi_task->dl.pi_se;
>  				scope->flags |= ENQUEUE_REPLENISH;
>  			} else {
> -				p->dl.pi_se = &p->dl;
> +				dl_deboost(&p->dl);
> +				/*
> +				 * dl_deboost() can start a new period. Its
> +				 * deadline can then be after the top waiter's.
> +				 * Boost the task again then.
> +				 */
> +				if (pi_task && dl_prio(pi_task->prio) &&
> +				    dl_entity_preempt(&pi_task->dl, &p->dl)) {
> +					p->dl.pi_se = pi_task->dl.pi_se;
> +					scope->flags |= ENQUEUE_REPLENISH;
> +				}
>  			}

[Severity: Medium]
When a SCHED_DEADLINE task was refilled from an earlier donor and now switches
to a new top waiter pi_task, doesn't the outer dl_entity_preempt(&pi_task->dl,
&p->dl) check compare pi_task against the old donor's postponed deadline
before dl_deboost() can reset p->dl to its own reservation?

If the first donor's refill pushed p->dl.deadline past pi_task->dl.deadline,
rt_mutex_setprio() takes the first branch and skips dl_deboost(). Because
p->dl.dl_deadline is non-zero and p->dl.runtime > 0, replenish_dl_entity()
leaves the first donor's remaining runtime and postponed deadline in place
after the unlock. Calling dl_deboost(&p->dl) unconditionally when
dl_prio(p->normal_prio) before testing dl_entity_preempt(&pi_task->dl, &p->dl)
would reset the period first and then boost from the new waiter if needed.

>  		} else if (rt_prio(prio)) {
>  			if (dl_prio(oldprio))
> -				p->dl.pi_se = &p->dl;
> +				dl_drop_boost(&p->dl);
>  			if (oldprio < prio)
>  				scope->flags |= ENQUEUE_HEAD;
>  		} else {
>  			if (dl_prio(oldprio))
> -				p->dl.pi_se = &p->dl;
> +				dl_drop_boost(&p->dl);
>  			if (rt_prio(oldprio))
>  				p->rt.timeout = 0;
>  		}

[Severity: Medium]
What happens if a task with its own deadline parameters (p->dl.dl_runtime != 0)
is refilled from a donor and then leaves dl_sched_class at deboost, for
instance after changing its policy via sched_setattr() while boosted (or being
boosted again before its 0-lag timer fires)?

sched_change_begin() runs switched_from_dl() before dl_drop_boost() clears
dl_donor_refill, and switched_from_dl() only resets deadline and runtime when
!p->dl.dl_runtime:

kernel/sched/deadline.c:switched_from_dl() {
	if (task_on_rq_queued(p) && p->dl.dl_runtime)
		task_non_contending(&p->dl, false);
...
	if (!p->dl.dl_runtime) {
		p->dl.deadline = 0;
		p->dl.runtime = 0;
	}
}

task_non_contending() then computes zerolag_time from the donor's refilled
deadline and runtime combined with p's own dl_period and dl_runtime. If the
donor's remaining runtime makes zerolag_time negative, task_non_contending()
immediately clears p's parameters even though p already exhausted its own
budget in this period; otherwise p keeps the donor's runtime and deadline
with dl_donor_refill cleared if it returns to SCHED_DEADLINE before the
inactive timer fires.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index dd48f74412b87..1f3da0e0a698c 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -93,6 +93,38 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
>  {
>  	return pi_of(dl_se) != dl_se;
>  }
> +
> +/*
> + * Record that @dl_se had its runtime refilled from the parameters of its
> + * donor, if it is boosted.
> + */
> +static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
> +{
> +	if (is_dl_boosted(dl_se))
> +		dl_se->dl_donor_refill = 1;
> +}
> +
> +/*
> + * End the boost of @dl_se, whose task stays in the class. If the boost
> + * refilled its runtime from the donor's parameters, start a new period
> + * now. The next enqueue is too late for a task blocked on another
> + * rt_mutex: the chain walk copies its deadline into its waiter node
> + * first. Without a refill, the task keeps the runtime and the deadline
> + * of its own reservation.
> + *
> + * The caller holds the rq lock and the pi_lock of the task, and has
> + * updated the rq clock.
> + */
> +void dl_deboost(struct sched_dl_entity *dl_se)
> +{
> +	struct rq *rq = rq_of_dl_se(dl_se);
> +
> +	if (dl_se->dl_donor_refill) {
> +		dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
> +		dl_se->runtime = dl_se->dl_runtime;
> +	}
> +	dl_drop_boost(dl_se);
> +}
>  #else /* !CONFIG_RT_MUTEXES: */
>  static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
>  {

[Severity: Medium]
Does this break the build when CONFIG_RT_MUTEXES is disabled?
dl_note_donor_refill() is defined inside #ifdef CONFIG_RT_MUTEXES with no
stub in the #else branch, but replenish_dl_new_period() and
replenish_dl_entity() call it unconditionally.

[ ... ]

> @@ -842,11 +875,15 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
>  	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
>  		dl_se->deadline = rq_clock(rq) + deadline;
>  		dl_se->runtime = runtime;
> +		dl_note_donor_refill(dl_se);
>  	}
>  
>  	if (dl_se->dl_yielded && dl_se->runtime > 0)
>  		dl_se->runtime = 0;
>  
> +	if (dl_se->runtime <= 0)
> +		dl_note_donor_refill(dl_se);
> +

[Severity: Medium]
Should the early return when !runtime earlier in replenish_dl_entity() also
call dl_note_donor_refill(dl_se)?

kernel/sched/deadline.c:replenish_dl_entity() {
	if (unlikely(!runtime)) {
		WARN_ON_ONCE(!is_dl_boosted(dl_se));
		dl_se->dl_yielded = 0;
		dl_se->dl_throttled = 0;
		return;
	}
}

When a donor leaves the class and its parameters are cleared before the lock
owner is deboosted, this path keeps the owner runnable after it exhausted its
runtime, and update_curr_dl_se() continues driving dl_se->runtime negative.
Without dl_donor_refill set, dl_deboost() will not start a new period and the
boosted execution is charged against the owner's own reservation (or leaves
deadline at 0 if the owner entered SCHED_DEADLINE while boosted).

> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 92134ce326a49..d87296abec4c2 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -356,6 +356,16 @@ extern int  sched_dl_global_validate(void);
>  extern void sched_dl_do_global(void);
>  extern int  sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
>  extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
> +#ifdef CONFIG_RT_MUTEXES
> +extern void dl_deboost(struct sched_dl_entity *dl_se);
> +
> +/* End the boost of @dl_se. The caller holds the rq lock of its task. */
> +static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
> +{
> +	dl_se->pi_se = dl_se;
> +	dl_se->dl_donor_refill = 0;
> +}
> +#endif

[Severity: Low]
Should __dl_clear_params() also clear dl_donor_refill (or call
dl_drop_boost()) instead of only setting dl_se->pi_se = dl_se?

kernel/sched/deadline.c:__dl_clear_params() {
...
	dl_se->dl_defer_running		= 0;
	dl_se->dl_defer_armed		= 0;

#ifdef CONFIG_RT_MUTEXES
	dl_se->pi_se			= dl_se;
#endif
}

If a task forks while boosted and refilled by a donor, dup_task_struct()
copies dl_donor_refill = 1 into the child and __sched_fork() ->
init_dl_entity() -> __dl_clear_params() leaves it set, which would cause an
unintended period reset if the child later becomes SCHED_DEADLINE and passes
through dl_deboost().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=10

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters
  2026-11-16 19:05 ` [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
@ 2026-11-16 19:42   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:42 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: torn donor parameter reads in
  replenish_dl_entity()
- [Medium] sched/deadline: unreset negative runtime in
  replenish_dl_entity()
- [Medium] sched/deadline: zero deadline in replenish_dl_entity() on
  initial boost
--

commit f55f8a9d26011d4eb967a7bc04c3939d81815727
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Do not replenish a boost from cleared parameters

This commit avoids an infinite loop in replenish_dl_entity() when a
boosted entity's donor has left SCHED_DEADLINE and had its parameters
cleared before the priority inheritance chain walk deboosts the owner.

Link: https://lore.kernel.org/all/avL8oOtjKqVNR7-l@jlelli-thinkpadt14gen4.remote.csb/

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index e4d7cbf4861de..c4ea510599415 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -807,8 +807,26 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
>  {
>  	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
>  	struct rq *rq = rq_of_dl_rq(dl_rq);
> +	struct sched_dl_entity *pi_se = pi_of(dl_se);
> +	u64 runtime = READ_ONCE(pi_se->dl_runtime);
> +	u64 deadline = READ_ONCE(pi_se->dl_deadline);
> +	u64 period = READ_ONCE(pi_se->dl_period);

[Severity: Medium]
Can replenish_dl_entity() observe a non-zero runtime alongside a zeroed
deadline or period when racing with __dl_clear_params()?

__dl_clear_params() zeroes dl_runtime, dl_deadline and dl_period in that
same order under the donor's rq lock without ordering barriers:

CPU0 (boosted owner)              CPU1 (donor)
replenish_dl_entity()
  runtime = READ_ONCE(pi_se->dl_runtime)
                                  inactive_task_timer()
                                    __dl_clear_params()
                                      dl_se->dl_runtime = 0
                                      dl_se->dl_deadline = 0
                                      dl_se->dl_period = 0
  deadline = READ_ONCE(pi_se->dl_deadline)  <-- reads 0
  period = READ_ONCE(pi_se->dl_period)      <-- reads 0

Because only !runtime is checked below, replenish_dl_entity() proceeds
with runtime > 0 and deadline == 0 or period == 0.  A boosted non-deadline
task gets dl_se->deadline set to rq_clock(rq) + 0, while a boosted
SCHED_DEADLINE task has runtime added in the while (dl_se->runtime <= 0)
loop without advancing dl_se->deadline.  If dl_se->deadline is then in the
past, the lag check emits a false "sched: DL replenish lagged too much"
warning and calls replenish_dl_new_period(), which re-reads
pi_of(dl_se)->dl_deadline and pi_of(dl_se)->dl_runtime directly and sets
dl_se->runtime to 0.

> -	WARN_ON_ONCE(pi_of(dl_se)->dl_runtime <= 0);
> +	/*
> +	 * A boosted entity runs on the parameters of its donor, the top
> +	 * waiter of an rt_mutex. The donor can leave the class and have them
> +	 * cleared before the chain walk deboosts this entity. They are
> +	 * cleared under the donor's rq lock, which can be another one. So
> +	 * read each of them once. Refilling from a runtime of 0 would loop
> +	 * forever. Leave the entity runnable instead: the walk deboosts it,
> +	 * or its task releases the lock.
> +	 */
> +	if (unlikely(!runtime)) {
> +		WARN_ON_ONCE(!is_dl_boosted(dl_se));
> +		dl_se->dl_yielded = 0;
> +		dl_se->dl_throttled = 0;
> +		return;
> +	}

[Severity: Medium]
Does returning here without resetting dl_se->runtime leave the entity with
an accumulating negative runtime?

When a boosted task exhausts its runtime after the donor's parameters are
cleared, dl_se->runtime remains <= 0 across this early return.  Every
subsequent tick in update_curr_dl_se() subtracts delta_exec from
dl_se->runtime, sees dl_runtime_exceeded(dl_se), sets dl_se->dl_overrun
(sending repeated SIGXCPU signals if SCHED_FLAG_DL_OVERRUN is enabled),
and re-enters enqueue_task_dl() -> replenish_dl_entity().

With the follow-up commit "sched/deadline: Reset the donor's runtime at a
deboost in the class", this return also happens before
dl_note_donor_refill(dl_se).  If a SCHED_DEADLINE owner had not been
refilled from the donor before the donor's parameters were cleared,
dl_donor_refill stays 0 and dl_deboost() does not reset dl_se->runtime;
when dl_se->deadline is still in the future at deboost, the negative
runtime is charged against the owner's own periods on its next replenish,
or pushes zerolag_time past dl_se->deadline in task_non_contending() if
the owner blocks first.

[Severity: Medium]
Can this early return also leave dl_se->deadline at 0 when a runnable
non-deadline task is first boosted into SCHED_DEADLINE?

When rt_mutex_setprio() boosts a runnable non-deadline task whose
dl_se->deadline is 0, it sets ENQUEUE_REPLENISH and calls
enqueue_task_dl() -> enqueue_dl_entity() -> replenish_dl_entity().  If the
donor's parameters were already cleared (for example after
normalize_rt_tasks(), or before a multi-hop PI chain walk finishes
deboosting an intermediate lock owner), replenish_dl_entity() returns here
before the dl_se->dl_deadline == 0 branch initializes dl_se->deadline.

__enqueue_dl_entity() then calls inc_dl_deadline() with deadline == 0:

kernel/sched/deadline.c:inc_dl_deadline() {
	if (dl_rq->earliest_dl.curr == 0 ||
	    dl_time_before(deadline, dl_rq->earliest_dl.curr)) {
		if (dl_rq->earliest_dl.curr == 0)
			cpupri_set(&rq->rd->cpupri, rq->cpu, CPUPRI_HIGHER);
		dl_rq->earliest_dl.curr = deadline;
		cpudl_set(&rq->rd->cpudl, rq->cpu, deadline);
	}
}

Because dl_rq->earliest_dl.curr == 0 is the sentinel for an empty dl_rq,
enqueuing a subsequent deadline task with a later deadline overwrites
dl_rq->earliest_dl.curr and cpudl, hiding earlier deadlines on this rq
from cross-CPU deadline balancing.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=8

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth
  2026-11-16 19:05 ` [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
@ 2026-11-16 19:44   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:44 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: __sched_setscheduler() checks task_rq(p) root
  domain
--

commit 685fb9b4f51b7675ac0c8a518174590afd51eb14
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Record the root domain holding a task's bandwidth

This commit records in dl_bw_cpu a CPU of the root domain that accounts for
a deadline task's bandwidth, and updates bandwidth moves, releases,
parameter changes, and affinity checks to consult that recorded root domain
instead of task_rq(p)->rd.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index ccd6426811b9e..44db625bffc32 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -4193,6 +4260,23 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
>  		return 0;
>  
> +	/*
> +	 * Change the bandwidth of a deadline task where it is counted: in
> +	 * the root domain of the CPU recorded with it. A task moved asleep
> +	 * has not run there yet, so the CPU it is on can be elsewhere. No
> +	 * bandwidth is counted if a rebuild dropped it, or if the task
> +	 * enters the class. A runtime below about a millionth of the period
> +	 * gives a bandwidth of 0. It counts for nothing, and no move updates
> +	 * its record. In these three cases, the root domain of the CPU the
> +	 * task is on takes the new bandwidth. The caller holds cpuset_mutex,
> +	 * or dl_accounting_mutex without cpusets, so no rebuild runs
> +	 * meanwhile.
> +	 */
> +	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
> +		  p->dl.dl_bw_gen == dl_bw_gen;
> +	cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
> +	dl_b = dl_bw_of(cpu);
[ ... ]
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 4c013caa24713..306cd8d32173b 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -1185,6 +1185,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>  
>  int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
>  {
> +	unsigned int cpu;
> +
>  	/*
>  	 * If the task isn't a deadline task or admission control is
>  	 * disabled then we don't care about affinity changes.
> @@ -1204,9 +1206,16 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
>  	 * if admission test is enabled, we only admit -deadline
>  	 * tasks allowed to run on all the CPUs in the task's
>  	 * root_domain.
> +	 *
> +	 * That is the root domain that counts the bandwidth of the task:
> +	 * the one of the CPU recorded with it. A task moved asleep to
> +	 * another cpuset is still on a CPU of the root domain it left. A
> +	 * bandwidth of 0 is counted nowhere, and no move updates its
> +	 * record: use the CPU the task is on.
>  	 */
> +	cpu = READ_ONCE(p->dl.dl_bw) ? dl_task_root_domain_cpu(p) : task_cpu(p);
>  	guard(rcu)();
> -	if (!cpumask_subset(task_rq(p)->rd->span, mask))
> +	if (!cpumask_subset(cpu_rq(cpu)->rd->span, mask))
>  		return -EBUSY;
>  
>  	return 0;

[Severity: Medium]
Should __sched_setscheduler() also check the root domain of the recorded
dl_bw_cpu instead of rq->rd?

While sched_dl_overflow() and dl_task_check_affinity() are updated to look
up the root domain via dl_bw_cpu for a task moved while asleep,
__sched_setscheduler() still checks rq = task_rq(p) before calling
sched_dl_overflow():

kernel/sched/syscalls.c:__sched_setscheduler() {
	...
		if (dl_bandwidth_enabled() && dl_policy(policy) &&
				!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
			cpumask_t *span = rq->rd->span;

			if (!cpumask_subset(span, &p->cpus_mask) ||
			    rq->rd->dl_bw.bw == 0) {
				retval = -EPERM;
				goto unlock;
			}
		}
	...
}

When a sleeping SCHED_DEADLINE task is moved to another exclusive cpuset,
p->cpus_mask is updated to the destination cpuset while task_rq(p) stays in
the old root domain.  Because the old rq->rd->span is disjoint from
p->cpus_mask, sched_setattr() fails here with -EPERM before reaching
sched_dl_overflow().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=16

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to the root domain it runs in
  2026-11-16 19:05 ` [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
@ 2026-11-16 19:47   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:47 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: missing cpu_active() check in
  dl_task_move_bw()
--

commit 1138754ab6e05269e8ea3d708fa33a8aa8f05601
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Move a task's bandwidth to the root domain it runs in

This commit moves a deadline task's bandwidth to the root domain of its
target CPU during migration and wakeup if its bandwidth is currently
counted in another root domain or was left out of a rebuild.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index c765a403014a2..35f40185953a3 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -2910,11 +2910,69 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
>  	return cpu;
>  }
>  
> -static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
> +/*
> + * Return true if the root domain of @cpu may not count the bandwidth of
> + * @p. Either another root domain counts it, or a rebuild left @p out.
> + * Test without dl_departed_lock, so that a move inside a root domain
> + * takes no lock. dl_task_move_bw() tests again under it.
> + */
> +static bool dl_task_bw_elsewhere(struct task_struct *p, int cpu)
> +{
> +	int bw_cpu = READ_ONCE(p->dl.dl_bw_cpu);
> +
> +	if (bw_cpu < 0 || !p->dl.dl_bw)
> +		return false;
> +	return READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen) ||
> +	       cpu_rq(bw_cpu)->rd != cpu_rq(cpu)->rd;
> +}
> +
> +/*
> + * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
> + * also if a rebuild left @p out. Add it even without room, since @p runs
> + * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
> + */
> +static void dl_task_move_bw(struct task_struct *p, int new_cpu)
> +{
> +	struct dl_bw *dl_b;
> +	int cpu;
> +
> +	if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
> +		return;
> +
> +	guard(raw_spinlock)(&dl_departed_lock);
> +	cpu = p->dl.dl_bw_cpu;
> +	if (cpu < 0)
> +		return;
> +	if (p->dl.dl_bw_gen == dl_bw_gen) {
> +		if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
> +			return;
> +		dl_b = dl_bw_of(cpu);
> +		raw_spin_lock(&dl_b->lock);
> +		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
> +		raw_spin_unlock(&dl_b->lock);
> +	}
> +	dl_b = dl_bw_of(new_cpu);
> +	raw_spin_lock(&dl_b->lock);
> +	__dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
> +	raw_spin_unlock(&dl_b->lock);
> +	p->dl.dl_bw_cpu = new_cpu;
> +	p->dl.dl_bw_gen = dl_bw_gen;
> +}

[Severity: Medium]
Can dl_task_move_bw() move p->dl.dl_bw to def_root_domain and record an
inactive CPU in p->dl.dl_bw_cpu when new_cpu is not active?

During CPU hot-unplug, sched_cpu_deactivate() clears the outgoing CPU from
cpu_active_mask, attaches it to def_root_domain, and rebuilds root-domain
accounting so a sleeping migration-disabled deadline task is counted on an
active CPU in p->cpus_mask.  If that task wakes up before
sched_cpu_wait_empty() finishes, is_cpu_allowed() still permits the
online outgoing CPU, so ttwu_do_activate() -> task_woken_dl() calls
dl_task_move_bw() on it.

Because neither dl_task_bw_elsewhere() nor dl_task_move_bw() checks
cpu_active(), this regression subtracts p->dl.dl_bw from the active root
domain and moves it to def_root_domain, allowing the active root domain to
admit deadline tasks beyond its capacity.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=25

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting
  2026-11-16 19:05 ` [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
@ 2026-11-16 19:49   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:49 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] selftests/sched: unconditional cpu_up() and hotplug check in
  fallback_case()
--

commit a082e176a7ad72cb248a92c19ae446e62c656620
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

This commit adds deadline_bw_test to test SCHED_DEADLINE bandwidth
accounting across priority inheritance, parameter changes, cpuset moves,
deaths, yields, runtime overruns, reclaiming, fair servers, and CPU
hotplug.

Link: https://github.com/jlelli/sched-deadline-tests

> diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
> new file mode 100644
> index 0000000000000..19c61a3a95296
> --- /dev/null
> +++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c

[ ... ]

> +static int fallback_case(enum fb_mode mode)
> +{
> +	int in_b = mode == FB_LAST_H || mode == FB_PART_H || mode == FB_STAY_K;
> +	int off_cpu;
> +	int on_h = mode == FB_ON_H || mode == FB_MASK_H || in_b;
> +	int no_offline = mode == FB_MASK_H || mode == FB_LAST_H || mode == FB_PART_H;
> +	char cg_q[PATH_MAX];
> +	int on_i1 = mode == FB_ON_I1;
> +	char cg_b[PATH_MAX], cg_c[PATH_MAX], path[PATH_MAX + 32], buf[256], one[16];
> +	long long def0, hk0, def_off = 0, hk_off = 0, def_ref, hk_ref, def, hk, bw;
> +	int fd[2], ret = KSFT_SKIP, added = 0, made = 0, w, at, err;
> +	pid_t pid = -1;
> +
> +	if (!root[0] || fallback_cpus(!no_offline)) {
> +		ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs%s\n",
> +			       no_offline ? "" : ", CPU hotplug");
> +		return KSFT_SKIP;
> +	}
> +	off_cpu = mode == FB_STAY_K ? fb_h : fb_i1;
> +	if ((mode == FB_LAST_H || mode == FB_STAY_K) && !cpu_hotpluggable(fb_h)) {
> +		ksft_print_msg("CPU %d cannot go offline\n", fb_h);
> +		return KSFT_SKIP;
> +	}

[ ... ]

> +	reap(pid);
> +	pid = -1;
> +	sleep_ns(3 * PERIOD);
> +	def = total_bw_of(fb_i2);
> +	hk = total_bw_of(fb_k);
> +	if (def != def_ref || hk != hk_ref) {
> +		ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
> +			       def, hk, def_ref, hk_ref);
> +		ret = KSFT_FAIL;
> +	}
> +out_up:
> +	reap(pid);
> +	cpu_up(off_cpu);
> +	pid = -1;

[Severity: Low]
Should cpu_up(off_cpu) at out_up be guarded by !no_offline?

For FB_MASK_H, FB_LAST_H, and FB_PART_H, off_cpu is never taken offline
(and mask_out_refused and partition_takes_refused have no_hp = 1), so
falling through to out_up attempts to write to the sysfs online file of
fb_i1 and logs an error when fb_i1 is not hotpluggable.

Also, for FB_STAY_K, off_cpu is fb_h rather than fb_i1, but passing
!no_offline to fallback_cpus() requires an isolated CPU to be
hotpluggable and skips hotplug_keeps_root_domain if neither isolated CPU
supports hotplug.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=37

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth
  2026-11-16 19:05 ` [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
@ 2026-11-16 19:50   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:50 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: dl_task_offline_migration() bypasses
  dl_task_bw_span()
- [Medium] sched/deadline: dl_task_bw_span() returns NULL for
  deactivating dl_bw_cpu
- [Low] sched/deadline: dead branch and fallback order in
  dl_task_rebuild_cpu()
--

commit a54933b747047e1dd4434be5e5299159be0de9fe
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Keep a task in the root domain that counts its bandwidth

This commit adds dl_task_bw_span() and updates CPU selection paths and
dl_task_rebuild_cpu() to prefer CPUs in the root domain that counts a
deadline task's bandwidth.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 35f40185953a3..5294546786f12 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -864,13 +864,19 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
>  
>  	later_rq = find_lock_later_rq(p, rq);
>  	if (!later_rq) {
> +		const struct cpumask *span = dl_task_bw_span(p);
>  		int cpu;
>  
>  		/*
>  		 * If we cannot preempt any rq, fall back to pick any
> -		 * online CPU:
> +		 * online CPU. Try the root domain that counts @p first.
>  		 */
> -		cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
> +		cpu = nr_cpu_ids;
> +		if (span)
> +			cpu = cpumask_first_and_and(cpu_active_mask,
> +						    p->cpus_ptr, span);
> +		if (cpu >= nr_cpu_ids)
> +			cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);

[Severity: Medium]
Can find_lock_later_rq(p, rq) bypass the dl_task_bw_span(p) check and
migrate p onto an isolated CPU in def_root_domain?

When a housekeeping CPU goes offline, detach_destroy_domains() attaches
its runqueue to def_root_domain while dl_rebuild_rd_accounting() moves a
throttled task's bandwidth to the new housekeeping root domain.  When
dl_task_timer() later fires on that offline CPU and calls
dl_task_offline_migration(), find_later_rq() searches
task_rq(p)->rd->cpudl, which is def_root_domain.cpudl:

kernel/sched/deadline.c:find_later_rq() {
    ...
	if (!cpudl_find(&task_rq(task)->rd->cpudl, task, later_mask))
		return -1;
    ...
}

With isolcpus=domain, isolated CPUs also belong to def_root_domain.  If
p->cpus_mask includes an isolated CPU that is free or running a later
deadline in def_root_domain.cpudl, find_lock_later_rq() returns that
isolated CPU's runqueue and skips the !later_rq fallback entirely,
moving p and its bandwidth into def_root_domain without admission
control.

[ ... ]

> @@ -3669,6 +3690,29 @@ static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
>  	return cpu;
>  }
>  
> +/*
> + * Return the CPUs of the root domain that counts the bandwidth of @p, or
> + * NULL if none does. The mask of @p can hold CPUs of another root domain,
> + * with isolcpus="domain". Where a CPU is picked for @p, a CPU of this
> + * root domain comes first: no other root domain admitted @p.
> + *
> + * The caller holds the pi_lock or the rq lock of @p. With interrupts
> + * off, the root domain is not freed.
> + */
> +const struct cpumask *dl_task_bw_span(struct task_struct *p)
> +{
> +	int cpu;
> +
> +	if (!task_has_dl_policy(p) || dl_entity_is_special(&p->dl) ||
> +	    !p->dl.dl_bw)
> +		return NULL;
> +	cpu = READ_ONCE(p->dl.dl_bw_cpu);
> +	if (cpu < 0 || !cpu_active(cpu) ||
> +	    READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen))
> +		return NULL;
> +	return cpu_rq(cpu)->rd->span;
> +}

[Severity: Medium]
Does rejecting !cpu_active(cpu) here cause dl_task_bw_span() to return
NULL while a CPU is being deactivated?

In sched_cpu_deactivate(cpu), set_cpu_active(cpu, false) runs before
balance_push_set(), synchronize_rcu(), and cpuset_cpu_inactive(), and
cpuset_handle_hotplug() also calls cpuset_hotplug_update_tasks() before
rebuild_sched_domains_cpuslocked() runs dl_rebuild_rd_accounting().
Throughout that window, cpu_rq(cpu)->rd still points to the root domain
that counts p's bandwidth, but any task whose p->dl.dl_bw_cpu is the
outgoing CPU gets dl_task_bw_span(p) == NULL.  Tasks pushed by
balance_push, updated by cpuset_hotplug_update_tasks(), or woken during
deactivation can then be placed on an isolated CPU in def_root_domain
and move their bandwidth there without admission control.

> @@ -3903,6 +3947,27 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
>  	 */
>  	cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
>  	cpu = task_cpu(p);
> +	/*
> +	 * Keep the bandwidth where it was counted: with the housekeeping
> +	 * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
> +	 * domains are new, and the CPU recorded can be offline. So prefer a
> +	 * CPU of the mask of the same kind as the CPU recorded: housekeeping
> +	 * or not.
> +	 */
> +	if (recorded >= 0) {
> +		const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
> +		bool rec_hk = cpumask_test_cpu(recorded, hk);
> +		unsigned int same;
> +
> +		if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
> +			return cpu;
> +		if (cpumask_test_cpu(recorded, msk))
> +			return recorded;
> +		for_each_cpu(same, msk) {
> +			if (cpumask_test_cpu(same, hk) == rec_hk)
> +				return same;
> +		}
> +	}
>  	if (cpumask_test_cpu(cpu, msk))
>  		return cpu;
>  	if (cpumask_empty(msk)) {

[Severity: Low]
Because the new (recorded >= 0) block returns recorded whenever
cpumask_test_cpu(recorded, msk) is true, the existing else-if branch
right below this hunk is now unreachable:

kernel/sched/deadline.c:dl_task_rebuild_cpu() {
    ...
	if (cpumask_empty(msk)) {
		if (recorded >= 0 && cpu_active(recorded))
			return recorded;
		dl_get_task_effective_cpus(p, msk);
		cpumask_and(msk, msk, cpu_active_mask);
	} else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
		return recorded;
	}
    ...
}

Also, when cpumask_empty(msk) is true and recorded is inactive,
dl_get_task_effective_cpus(p, msk) refills msk after the kind-matching
loop has already run, so the fallback at the end of the function will
pick a housekeeping CPU even when recorded was an isolated CPU.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=26

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain
  2026-11-16 19:05 ` [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-16 19:51   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:51 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: missing affinity masks in
  dl_task_leaves_span() fallback
- [Low] sched/deadline: dl_task_leaves_span() rejects CPU offline for
  dead tasks
--

commit 48b62790717c3a90940a59e814da93b5ddf9ada9
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Keep the last CPU a task has in its root domain

This commit refuses to take a CPU offline with -EBUSY when doing so would
leave a deadline task with runnable CPUs only outside the root domain that
accounts for its bandwidth.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 5294546786f12..096c0ec55cdc2 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -4896,9 +4896,63 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
>  	return overflow ? -EBUSY : 0;
>  }
>  
> +/*
> + * Return true if @p has CPUs to run on once @cpu is offline, but none in
> + * the root domain of @cpu, which counts its bandwidth.
> + */
> +static bool dl_task_leaves_span(struct task_struct *p, int cpu,
> +				const struct cpumask *span)
> +{
> +	struct cpumask *msk;
> +
> +	guard(raw_spinlock_irqsave)(&p->pi_lock);
> +	if (dl_task_bw_span(p) != span)
> +		return false;
> +	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
> +	cpumask_and(msk, p->cpus_ptr, cpu_active_mask);
> +	cpumask_clear_cpu(cpu, msk);
> +	/* With no CPU left, the cpuset gives @p the CPUs of an ancestor. */
> +	if (cpumask_empty(msk))
> +		cpuset_cpus_allowed_without(p, cpu, msk);
> +	return !cpumask_empty(msk) && !cpumask_intersects(msk, span);
> +}

[Severity: Medium]
When msk is empty and dl_task_leaves_span() falls back to
cpuset_cpus_allowed_without(), does it also need to restrict the inherited
ancestor mask by p->user_cpus_ptr (when their intersection is non-empty)
and task_cpu_possible_mask(p)?

When the CPU goes offline, cpuset_update_tasks_cpumask() updates the task
via __set_cpus_allowed_ptr(), which intersects the new cpuset mask with
p->user_cpus_ptr if they overlap:

kernel/sched/core.c:__set_cpus_allowed_ptr() {
    ...
	if (p->user_cpus_ptr &&
	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
		ctx->new_mask = rq->scratch_mask;
    ...
}

Suppose CPUs 0-1 are housekeeping CPUs in span and CPUs 2-3 are isolated
(isolcpus=domain,2,3), and a task with user_cpus_ptr = {0, 2, 3} is moved
into a child cpuset with cpuset.cpus = 0 under top_cpuset (0-3).  Taking
CPU 0 offline leaves msk empty, so cpuset_cpus_allowed_without() returns
{1, 2, 3} from top_cpuset.  Because dl_task_leaves_span() sees CPU 1 in
msk & span, it allows the offline, after which __set_cpus_allowed_ptr()
intersects {1, 2, 3} with user_cpus_ptr and restricts the task to {2, 3} in
the default root domain without an admission check.

In addition, the CONFIG_CPUSETS implementation of
cpuset_cpus_allowed_without() does not intersect cs->effective_cpus with
task_cpu_possible_mask(tsk), unlike guarantee_active_cpus() and the
!CONFIG_CPUSETS stub.

> +
> +/*
> + * Return true if @cpu is the last CPU that a deadline task can run on in
> + * the root domain that counts its bandwidth. With @cpu offline, the task
> + * would run in another root domain, which did not admit it. The mask of
> + * a task holds CPUs of two root domains only with isolcpus="domain".
> + *
> + * A suspend takes the CPUs offline with the tasks frozen, and the resume
> + * puts them back. Without admission control, there is no limit to exceed.
> + * Return false in both cases.
> + */
> +static bool dl_cpu_last_of_task(int cpu)
> +{
> +	const struct cpumask *span;
> +	struct task_struct *g, *p;
> +
> +	if (cpuhp_tasks_frozen || !dl_bandwidth_enabled())
> +		return false;
> +	/* No task holds a bandwidth: do not walk the task list. */
> +	if (!atomic_read(&dl_nr_bw_tasks))
> +		return false;
> +	guard(rcu)();
> +	span = cpu_rq(cpu)->rd->span;
> +	for_each_process_thread(g, p) {
> +		if (task_has_dl_policy(p) && dl_task_leaves_span(p, cpu, span))
> +			return true;
> +	}

[Severity: Low]
Can this reject a CPU offline when p is an un-reaped exited task waiting for
its 0-lag timer?

When a deadline task exits before its 0-lag time, task_non_contending()
places it on dl_departed_list and arms inactive_timer while leaving
p->policy and p->dl.dl_bw_cpu set.  Until release_task() unhashes the task
or inactive_timer fires, for_each_process_thread() still visits it and
dl_task_leaves_span() returns true if the outgoing CPU was its only CPU in
span, causing dl_bw_deactivate() to fail with -EBUSY even though a
TASK_DEAD task will never run again or move its bandwidth to another root
domain.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=27

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to
  2026-11-16 19:05 ` [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
@ 2026-11-16 19:56   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 19:56 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] sched/deadline: unmasked offline CPUs in
  __dl_task_bw_move_cpu() span check
- [Medium] cgroup/cpuset: stale dl_bw_resv_cpu left when cpuset_attach()
  skips task loop
--

commit 1fc4cfdda6a5d5f5e22bff0342024fd03d578a7c
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to

This commit reserves and moves a deadline task's bandwidth using the
specific destination CPU recorded in task->dl.dl_bw_resv_cpu during
cpuset_can_attach(), keeping per-CPU reservation sums in cpuset_dl_bw so
that attaches to cpusets spanning multiple root domains account bandwidth
in the root domain where the task actually runs.

> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 2f39c3d12550e..0d55f092d9cbb 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
[ ... ]
> @@ -3236,15 +3258,20 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
>  				goto out_unlock;
>  		}
>  
> -		if (dl_task(task)) {
> -			/*
> -			 * Count all migrating DL tasks for cpuset task accounting.
> -			 * Only tasks that need a root-domain bandwidth move
> -			 * contribute to sum_migrate_dl_bw.
> -			 */
> -			cs->nr_migrate_dl_tasks++;
> -			if (dl_task_needs_bw_move(task, cs->effective_cpus))
> -				cs->sum_migrate_dl_bw += task->dl.dl_bw;
> +		if (!dl_task(task))
> +			continue;
> +		/*
> +		 * Sum the bandwidth of the deadline tasks that move to another
> +		 * root domain, per CPU that the move records. That CPU depends
> +		 * on the CPUs the task gets. A cpuset can span more than one
> +		 * root domain, and the affinity of a task can keep it in any
> +		 * of them.
> +		 */
> +		cpuset_attach_cpus(cs, task, cpus_attach);
> +		cpu = dl_task_bw_reserve_cpu(task, cpus_attach);
> +		if (cpu >= 0) {
> +			per_cpu_ptr(&cpuset_dl_bw, cpu)->sum += task->dl.dl_bw;
> +			cpumask_set_cpu(cpu, dl_attach_cpus);
>  		}
>  	}
[ ... ]
> @@ -3407,8 +3446,9 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>  		schedule_flush_migrate_mm();
>  	cs->old_mems_allowed = attach_ctx.nodemask_to;
>  out:
> -	clear_attach_data(&src_cs_head, false);
> -	clear_attach_data(&dst_cs_head, false);
> +	clear_attach_data(&src_cs_head);
> +	clear_attach_data(&dst_cs_head);
> +	cpuset_free_dl_bw();
>  	dec_attach_in_progress_locked();
>  
>  	mutex_unlock(&cpuset_mutex);

[Severity: Medium]
Does cpuset_attach() leave a stale task->dl.dl_bw_resv_cpu behind when it
takes the !attach_ctx.cpus_updated && !attach_ctx.mems_updated fast path?

In cpuset_can_attach(), dl_task_bw_reserve_cpu() runs whenever oldcs != cs,
even if oldcs and cs have identical effective_cpus and effective_mems (for
example, when enabling +cpuset in cgroup.subtree_control or moving a task
between v2 cgroups with the same effective masks).  If the task's bandwidth
is currently counted in another root domain, dl_task_bw_reserve_cpu() sets
task->dl.dl_bw_resv_cpu = cpu.  In cpuset_attach(), the fast path:

kernel/cgroup/cpuset.c:cpuset_attach() {
    ...
	if (cpuset_v2() && !attach_ctx.cpus_updated && !attach_ctx.mems_updated) {
		llist_for_each_entry(cs, dst_cs_head.first, attach_node)
			cs->old_mems_allowed = attach_ctx.nodemask_to;
		goto out;
	}
    ...
}

jumps past cpuset_attach_task() to out:, where cpuset_free_dl_bw() releases
the reservation in cpuset_dl_bw, but dl_task_bw_unreserve() is never called
on the tasks in tset.

If that task is later part of a v2 threaded subtree attach where oldcs == cs
for this thread while another thread in the taskset has oldcs != cs and
cpus_updated == true, cpuset_can_attach() skips this task, and
cpuset_attach_task() consumes the stale dl_bw_resv_cpu.  That moves the
task's bandwidth without a destination reservation and adds moved to
per_cpu(cpuset_dl_bw, cpu).moved, corrupting the root domain accounting and
underflowing b->sum - b->moved in cpuset_free_dl_bw().

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 5503820c5b26c..b43106a590256 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -3657,10 +3676,64 @@ bool dl_task_needs_bw_move(struct task_struct *p,
>  	 * their bandwidth: dl_add_task_root_domain() leaves them out.
>  	 */
>  	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
> -		return false;
> +		return -1;
>  
>  	cpu = dl_task_root_domain_cpu(p);
> -	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
> +	if (cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask))
> +		return -1;
> +	/*
> +	 * A task moved asleep is still on a CPU of the root domain it left.
> +	 * If the new mask holds that CPU, the task wakes up there.
> +	 */
> +	cpu = task_cpu(p);
> +	if (cpu_active(cpu) && cpumask_test_cpu(cpu, new_mask))
> +		return cpu;
> +	cpu = cpumask_first_and_and(new_mask, cpu_active_mask,
> +				    housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> +	if (cpu >= nr_cpu_ids)
> +		cpu = cpumask_any_and(new_mask, cpu_active_mask);
> +	return cpu < nr_cpu_ids ? (int)cpu : -1;
> +}

[Severity: High]
Can offline CPUs in def_root_domain.span cause __dl_task_bw_move_cpu() to
falsely return -1 when attaching a task to top_cpuset?

Every offline possible CPU is attached to def_root_domain (in sched_init()
and detach_destroy_domains()), as are CPUs isolated via isolcpus=domain or
isolated cpuset partitions.  When a task is attached to top_cpuset,
cpuset_attach_cpus() sets new_mask to task_cpu_possible_mask(task) &
~subpartitions_cpus, which includes all offline possible CPUs.

Because cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask) does not mask
against cpu_active_mask, any system with an offline CPU in
cpu_possible_mask will see def_root_domain.span intersect new_mask on that
offline CPU (unless filtered out by p->user_cpus_ptr).  Moving a deadline
task whose bandwidth is in def_root_domain back to top_cpuset will then
return -1 here, skipping both the destination reservation and the bandwidth
move out of def_root_domain.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=23

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in
  2026-11-16 19:05 ` [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
@ 2026-11-16 20:01   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 20:01 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: dl_task_rebuild_cpu() misaccounts
  offline-migrated task
--

commit 9e9e921e040902e3cbd909c81dd80911851e51ce
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Make a rebuild count a task in the root domain it runs in

This commit updates root-domain deadline bandwidth rebuilding to select the
CPU where a deadline task's bandwidth is counted based on the task's current
CPU, recorded dl_bw_cpu, active affinity mask, and cpuset, while skipping
tasks that have no active CPU available.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index b43106a590256..c765a403014a2 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c

[ ... ]

> @@ -3795,6 +3801,55 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
>  	cpuset_cpus_allowed_locked(p, cpus);
>  }
>  
> +/*
> + * Return a CPU of the root domain where a rebuild of the accounting should
> + * count the bandwidth of @p, or nr_cpu_ids if there is none.
> + *
> + * Prefer the CPU @p is on, if it is active and in the mask of @p. The
> + * task runs, or wakes up, in the root domain of that CPU. A mask can hold
> + * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
> + * isolcpus="domain". When every CPU of a mask goes offline,
> + * select_fallback_rq() can widen it to both. It moves no bandwidth.
> + *
> + * Else prefer the CPU recorded with the bandwidth, if it is active and in
> + * the mask. It is in the root domain that admitted the bandwidth, or that
> + * its last move put it in. A task moved asleep is still on a CPU of the
> + * root domain it left, out of its mask. Else take the first active CPU of
> + * the mask, a housekeeping one if there is one. A task on isolated CPUs
> + * that all went offline can have no active CPU at all.
> + */
> +static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
> +{
> +	struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
> +	int recorded = p->dl.dl_bw_cpu;
> +	unsigned int cpu;
> +
> +	/*
> +	 * Look at the mask of the task first. A change of the cpusets sets
> +	 * the mask of each task before its rebuild. The housekeeping mask
> +	 * that dl_get_task_effective_cpus() reads is updated after it. If no
> +	 * CPU of the mask is active, take the CPU recorded if it is active.
> +	 * dl_task_offline_migration() moved the task there, or an earlier
> +	 * rebuild counted it there. Else ask the cpuset.
> +	 */
> +	cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
> +	cpu = task_cpu(p);
> +	if (cpumask_test_cpu(cpu, msk))
> +		return cpu;
> +	if (cpumask_empty(msk)) {
> +		if (recorded >= 0 && cpu_active(recorded))
> +			return recorded;
> +		dl_get_task_effective_cpus(p, msk);
> +		cpumask_and(msk, msk, cpu_active_mask);
> +	} else if (recorded >= 0 && cpumask_test_cpu(recorded, msk)) {
> +		return recorded;
> +	}
> +	cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> +	if (cpu < nr_cpu_ids)
> +		return cpu;
> +	return cpumask_first(msk);
> +}

[Severity: Medium]
Can dl_task_rebuild_cpu() count a task migrated by
dl_task_offline_migration() in the wrong root domain once a CPU in
p->cpus_mask comes back online?

When admission control is disabled and all CPUs in p->cpus_ptr are offline,
dl_task_offline_migration() falls back to cpumask_any(cpu_active_mask) and
moves a throttled task's bandwidth and runqueue placement to that fallback
CPU without changing p->cpus_mask:

kernel/sched/deadline.c:dl_task_offline_migration() {
    ...
		cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
		if (cpu >= nr_cpu_ids) {
			...
			cpu = cpumask_any(cpu_active_mask);
		}
		later_rq = cpu_rq(cpu);
    ...
}

If p was pinned to a single offline CPU (p->nr_cpus_allowed == 1), it is not
pushable and continues running and replenishing on the fallback CPU even
after its original CPU comes back online.  During the rebuild triggered by
that cpu_up(), msk (&p->cpus_mask & cpu_active_mask) is no longer empty, and
neither task_cpu(p) nor recorded (p->dl.dl_bw_cpu) is in msk.
dl_task_rebuild_cpu() then returns a CPU from msk rather than task_cpu(p),
accounting the bandwidth in the root domain of p->cpus_mask instead of the
root domain where the task is still queued and running.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=24

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change
  2026-11-16 19:05 ` [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
@ 2026-11-16 20:04   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 20:04 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: dl_new_params cleared while boosted in
  enqueue_dl_entity()
- [Medium] sched/deadline: unchecked laxity in update_dl_entity() on
  dl_deadline shrink
--

commit b62a12a066a3a8cec83cd122301c3f7fcc56bd77
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Check the runtime of a task whose parameters change

This commit adds a dl_new_params flag to struct sched_dl_entity so that
when sched_setattr() modifies a deadline task's runtime, deadline, or
period, the next enqueue checks the remaining runtime against the new
parameters via update_dl_entity().

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index b47c13c8714d9..3b62ee43c2099 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -2674,7 +2674,10 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
>  static void
>  enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
>  {
> +	bool new_params = dl_se->dl_new_params;
> +
>  	WARN_ON_ONCE(on_dl_rq(dl_se));
> +	dl_se->dl_new_params = 0;

[Severity: Medium]
Does clearing dl_se->dl_new_params unconditionally here lose the mark if
the task is priority-inheritance boosted when sched_setattr() shrinks its
reservation?

When is_dl_boosted(dl_se) is true, the new_params branch below is skipped
(or on a wakeup while boosted, update_dl_entity() checks against
pi_of(dl_se) rather than the task's own parameters).  If the boost took no
donor refill (dl_donor_refill == 0), dl_deboost() later leaves
dl_se->runtime and dl_se->deadline intact, and the subsequent enqueue sees
dl_se->dl_new_params == 0 and lets the task run on its old larger runtime
after admission control already released the bandwidth difference.

>  	update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
>  
> @@ -2723,12 +2726,13 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
>  		update_dl_entity(dl_se);
>  	} else if (flags & ENQUEUE_REPLENISH) {
>  		replenish_dl_entity(dl_se);
> -	} else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
> +	} else if (((flags & ENQUEUE_CLASS) || new_params) &&
> +		   !is_dl_boosted(dl_se)) {
>  		/*
> -		 * The task enters the class runnable. It may come back before
> -		 * its deadline, with the runtime left from older parameters.
> -		 * Check that runtime against the new parameters, as a wakeup
> -		 * does.
> +		 * The task enters the class, or its parameters change, while
> +		 * it is runnable. It may have runtime left from older
> +		 * parameters. Check that runtime against the new parameters,
> +		 * as a wakeup does.
>  		 */
>  		update_dl_entity(dl_se);

[Severity: Medium]
Can update_dl_entity() leave or set dl_se->runtime above the new
dl_se->dl_runtime when sched_setattr() shrinks dl_deadline to a
constrained reservation while dl_se->deadline - rq_clock(rq) still exceeds
the new dl_deadline?

Both dl_entity_overflow() and update_dl_revised_wakeup() assume that
laxity = dl_se->deadline - rq_clock(rq) is at most dl_se->dl_deadline:

kernel/sched/deadline.c:update_dl_revised_wakeup() {
	u64 laxity = dl_se->deadline - rq_clock(rq);

	WARN_ON(dl_time_before(dl_se->deadline, rq_clock(rq)));

	dl_se->runtime = (dl_se->dl_density * laxity) >> BW_SHIFT;
}

For example, if a task running at 50ms/100ms/100ms has runtime = 30ms and
laxity = 80ms when changed to 10ms/20ms/100ms (density 50%),
dl_entity_overflow() sees 30ms / 80ms <= 10ms / 20ms and leaves runtime at
30ms; if changed instead at runtime = 49ms and laxity = 99ms to
8ms/20ms/100ms (density 40%), update_dl_revised_wakeup() sets
dl_se->runtime to 39.6ms against an 8ms budget.  Besides letting the task
overrun the bandwidth it just gave back in sched_dl_overflow(), blocking
with dl_se->runtime > dl_se->dl_runtime makes task_non_contending()
compute a negative zerolag_time and drop running_bw or total_bw
immediately.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=33

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable
  2026-11-16 19:05 ` [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
@ 2026-11-16 20:08   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 20:08 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: missed update_dl_entity() when setattr runs
  while PI-boosted
--

commit d9e5d198497bbfbacaaacba33587790f1f7b2f0f
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Check the runtime of a task that comes back runnable

This commit calls update_dl_entity() when a runnable unboosted task enters
SCHED_DEADLINE so its remaining runtime is checked against its new
scheduling parameters.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index ffa8b946e8047..b47c13c8714d9 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -2723,6 +2723,14 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
>  		update_dl_entity(dl_se);
>  	} else if (flags & ENQUEUE_REPLENISH) {
>  		replenish_dl_entity(dl_se);
> +	} else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
> +		/*
> +		 * The task enters the class runnable. It may come back before
> +		 * its deadline, with the runtime left from older parameters.
> +		 * Check that runtime against the new parameters, as a wakeup
> +		 * does.
> +		 */
> +		update_dl_entity(dl_se);
>  	} else if ((flags & ENQUEUE_MOVE) &&
>  		   !is_dl_boosted(dl_se) &&
>  		   dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {

[Severity: Medium]
Can a runnable task bypass update_dl_entity() when returning to
SCHED_DEADLINE with smaller runtime if it is PI-boosted at the time of
sched_setattr()?

If a SCHED_DEADLINE task switches to SCHED_OTHER before its 0-lag time and
is then PI-boosted back into dl_sched_class by an rt_mutex waiter,
replenish_dl_entity() sees non-zero dl_se->dl_deadline and positive
dl_se->runtime, so dl_se->dl_donor_refill stays 0.

When sched_setattr() then returns the task to SCHED_DEADLINE with a
smaller runtime while still boosted, __sched_setscheduler() sees
prev_class == next_class == &dl_sched_class (so ENQUEUE_CLASS is not set)
and is_dl_boosted(dl_se) is true, skipping update_dl_entity().  Even with
commit "sched/deadline: Check the runtime of a task whose parameters
change", enqueue_dl_entity() clears dl_se->dl_new_params unconditionally
at the top while skipping the check because is_dl_boosted(dl_se) is true.

When the task later unlocks the rt_mutex, dl_deboost() sees
dl_donor_refill == 0 and leaves the old larger dl_se->runtime intact:

kernel/sched/deadline.c:dl_deboost() {
	if (dl_se->dl_donor_refill) {
		dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
		dl_se->runtime = dl_se->dl_runtime;
	}
	dl_drop_boost(dl_se);
}

Because the subsequent enqueue only passes ENQUEUE_MOVE with a future
deadline, update_dl_entity() is never called, and the task runs on its old
larger runtime while admission control only accounts for the new smaller
bandwidth.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=32

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of its root domain
  2026-11-16 19:05 ` [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
@ 2026-11-16 20:08   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 20:08 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] cgroup/cpuset: excpus clobbered via tmp->new_cpus in
  remote_cpus_update()
- [High] cgroup/cpuset: partition_xcpus_del() called on invalid
  partition in update_parent_effective_cpumask()
- [Medium] cgroup/cpuset: wrong descendant effective_cpus in
  dl_tasks_keep_without()
- [Medium] sched/deadline: user_cpus_ptr ignored in
  dl_task_keeps_root_domain()
- [Medium] cgroup/cpuset: dl_tasks_keep_root_domain() skips invalidated
  partitions
--

commit 2c62aa35faa810e650ef3867961558c694f57e4c
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

cgroup/cpuset: Refuse to take a deadline task out of its root domain

This commit rejects cpuset CPU mask updates and invalidates partition roots
when the resulting effective CPUs would move a SCHED_DEADLINE task out of the
root domain accounting its bandwidth without admission control.

> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index ef5ad8e38f06f..5b4f7d4ef3ae2 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
[ ... ]
> @@ -1126,6 +1127,92 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
[ ... ]
> +static bool dl_tasks_keep_without(struct cpuset *root, struct cpuset *part,
> +				  const struct cpumask *xcpus, bool enable,
> +				  struct cpumask *new_cpus)
> +{
> +	struct cgroup_subsys_state *pos_css;
> +	struct cpuset *cp, *a;
> +	bool keep = true;
> +
> +	rcu_read_lock();
> +	cpuset_for_each_descendant_pre(cp, pos_css, root) {
> +		if (cp == part || (cp != root && is_partition_valid(cp))) {
> +			pos_css = css_rightmost_descendant(pos_css);
> +			continue;
> +		}
> +		if (!cpumask_intersects(cp->effective_cpus, xcpus))
> +			continue;
> +		for (a = cp; ; a = parent_cs(a)) {
> +			if (cpumask_andnot(new_cpus, a->effective_cpus, xcpus) ||
> +			    a == root)
> +				break;
> +		}
> +		keep = dl_tasks_keep_cpus(cp, new_cpus);
> +		if (!keep)
> +			break;
> +	}
> +	if (keep && enable) {
> +		cpuset_for_each_descendant_pre(cp, pos_css, part) {
> +			if (cp != part && is_partition_valid(cp)) {
> +				pos_css = css_rightmost_descendant(pos_css);
> +				continue;
> +			}
> +			if (cpumask_subset(cp->effective_cpus, xcpus))
> +				continue;
> +			if (!cpumask_and(new_cpus, cp->effective_cpus, xcpus))
> +				cpumask_copy(new_cpus, xcpus);
> +			keep = dl_tasks_keep_cpus(cp, new_cpus);
> +			if (!keep)
> +				break;
> +		}
> +	}
> +	rcu_read_unlock();
> +	return keep;
> +}

[Severity: Medium]
Can dl_tasks_keep_without() miscompute new_cpus for descendant cpusets and
let a deadline task move to another root domain without admission control?

In the first loop, when cp loses all of its effective_cpus to xcpus and
inherits from an ancestor a above parent_cs(cp), new_cpus is set to
a->effective_cpus & ~xcpus without filtering top-down through the
cpus_allowed masks of the cpusets between a and cp as
compute_effective_cpumask() does. If root has effective_cpus {0, 1, 2}
(0-1 housekeeping, 2 isolated), child P has cpus_allowed {0}, and grandchild
C has cpus_allowed {2} (so C inherits effective_cpus {0}), a partition taking
xcpus {0} tests C against root's {1, 2} and sees housekeeping CPU 1, whereas
update_cpumasks_hier() gives P {1, 2} and gives C {2}, moving a deadline
task in C into the isolated root domain.

In the second loop (when enable is true), cp's new mask is derived from
cp->effective_cpus & xcpus instead of filtering xcpus & cpu_active_mask
top-down through each descendant's cpus_allowed. Because enabling part
switches its effective CPUs from cpus_allowed to effective_xcpus, a child C
whose cpus_allowed only intersects part->exclusive_cpus previously inherited
part->cpus_allowed, passes the cpumask_subset(cp->effective_cpus, xcpus)
check if part->cpus_allowed is a subset of xcpus, and then switches to
C->cpus_allowed once part is enabled; in addition, xcpus is not masked with
cpu_active_mask, so offline CPUs in xcpus can match def_root_domain->span.

[ ... ]

> @@ -1779,6 +1868,9 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
>  		else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
>  						       excpus, tmp->delmask))
>  			WRITE_ONCE(cs->prs_err, PERR_HKEEPING);
> +		else if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->addmask,
> +						false, tmp->new_cpus))
> +			WRITE_ONCE(cs->prs_err, PERR_DLTASK);
>  		if (cs->prs_err)
>  			goto invalidate;
>  	}

[Severity: High]
Does passing tmp->new_cpus to dl_tasks_keep_without() clobber excpus before
it is copied to cs->effective_xcpus?

update_cpumasks_hier() calls remote_cpus_update() with excpus pointing to
tmp->new_cpus:

kernel/cgroup/cpuset.c:update_cpumasks_hier() {
    ...
			compute_excpus(cp, tmp->new_cpus);
			...
			rcu_read_unlock();
			remote_cpus_update(cp, NULL, tmp->new_cpus, tmp);
			rcu_read_lock();
    ...
}

When a descendant remote partition gains exclusive CPUs during
update_cpumasks_hier(), dl_tasks_keep_without() overwrites tmp->new_cpus
with top_cpuset's remaining effective CPUs, and remote_cpus_update() then
copies that clobbered mask from excpus into cs->effective_xcpus.

[ ... ]

> @@ -2013,6 +2109,15 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
>  			deleting = false;
>  			adding = cpumask_and(tmp->addmask,
>  					     cs->effective_xcpus, parent->effective_xcpus);
> +		} else if (deleting &&
> +			   !dl_tasks_keep_without(parent, cs, tmp->delmask,
> +						  is_partition_invalid(cs),
> +						  tmp->new_cpus)) {
> +			/* As above: the partition becomes invalid. */
> +			part_error = PERR_DLTASK;
> +			deleting = false;
> +			adding = cpumask_and(tmp->addmask,
> +					     cs->effective_xcpus, parent->effective_xcpus);
>  		}
>  	} else {

[Severity: High]
Can setting adding here when is_partition_invalid(cs) is true call
partition_xcpus_del() on an already-invalid partition?

An invalid local partition root (old_prs < 0) with cpuset.cpus.exclusive set
keeps a non-empty cs->effective_xcpus even though those CPUs were not removed
from parent->effective_cpus. When updating cpuset.cpus or
cpuset.cpus.exclusive on that invalid partition takes the newmask branch and
dl_tasks_keep_without() returns false, adding becomes true while old_prs < 0,
so partition_xcpus_del(old_prs, parent, tmp->addmask) triggers
WARN_ON_ONCE(old_prs < 0) and corrupts subpartitions_cpus, isolated_cpus, and
parent->effective_cpus by releasing CPUs that cs never held. Guarding adding
with is_partition_valid(cs) avoids releasing unowned CPUs.

[ ... ]

> @@ -2496,6 +2604,66 @@ static void partition_cpus_change(struct cpuset *cs, struct cpuset *trialcs,
[ ... ]
> +static bool dl_tasks_keep_root_domain(struct cpuset *cs, struct cpuset *trialcs,
> +				      struct cpumask *new_cpus)
> +{
> +	struct cgroup_subsys_state *pos_css;
> +	struct cpuset *cp;
> +	bool keep = true;
> +
> +	if (is_partition_valid(cs))
> +		return true;
> +	rcu_read_lock();
> +	cpuset_for_each_descendant_pre(cp, pos_css, cs) {
> +		if (is_partition_valid(cp)) {
> +			pos_css = css_rightmost_descendant(pos_css);
> +			continue;
> +		}
> +		trial_effective_cpus(cp, cs, trialcs, new_cpus);
> +		if (cpumask_subset(cp->effective_cpus, new_cpus))
> +			continue;
> +		keep = dl_tasks_keep_cpus(cp, new_cpus);
> +		if (!keep)
> +			break;
> +	}
> +	rcu_read_unlock();
> +	return keep;
> +}
> +
>  /**
>   * update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it
>   * @cs: the cpuset to consider
> @@ -2528,6 +2696,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,
>  	if (alloc_tmpmasks(&tmp))
>  		return -ENOMEM;
>  
> +	if (!dl_tasks_keep_root_domain(cs, trialcs, tmp.new_cpus)) {
> +		free_tmpmasks(&tmp);
> +		return -EBUSY;
> +	}
> +
>  	/*
>  	 * Check all the descendants in update_cpumasks_hier() if
>  	 * effective_xcpus is to be changed.

[Severity: Medium]
What happens when a write to cpuset.cpus (or cpuset.cpus.exclusive)
invalidates a valid partition root?

dl_tasks_keep_root_domain() runs before partition_cpus_change() and returns
true immediately when is_partition_valid(cs) is true, and
update_exclusive_cpumask() does not check dl_tasks_keep_root_domain() at all.
If cs is a valid root partition on a housekeeping CPU containing a deadline
task and a user writes a boot-isolated CPU to cs/cpuset.cpus,
partition_cpus_change() invalidates cs (PERR_HKEEPING), and
update_cpumasks_hier() recomputes cs->effective_cpus as a non-partition
cpuset on the isolated CPU; dl_rebuild_rd_accounting() then adds the task's
bandwidth to def_root_domain via dl_add_task_root_domain() without admission
control.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 096c0ec55cdc2..61b50266468f9 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -4955,6 +4955,25 @@ int dl_bw_deactivate(int cpu)
>  	return ret;
>  }
>  
> +/*
> + * Return false if @p has CPUs in the root domain that counts its
> + * bandwidth and @new_mask holds none of them. A cpuset refuses such a
> + * mask: only a move to another cpuset makes room in another root domain.
> + */
> +bool dl_task_keeps_root_domain(struct task_struct *p,
> +			       const struct cpumask *new_mask)
> +{
> +	const struct cpumask *span;
> +
> +	if (!dl_bandwidth_enabled() || !task_has_dl_policy(p))
> +		return true;
> +	guard(raw_spinlock_irqsave)(&p->pi_lock);
> +	guard(rcu)();
> +	span = dl_task_bw_span(p);
> +	return !span || !cpumask_intersects(span, p->cpus_ptr) ||
> +	       cpumask_intersects(span, new_mask);
> +}

[Severity: Medium]
Should dl_task_keeps_root_domain() account for p->user_cpus_ptr the way
__set_cpus_allowed_ptr() and dl_task_bw_reserve_cpu() do?

When cpuset_update_tasks_cpumask() calls set_cpus_allowed_ptr(task, new_cpus),
__set_cpus_allowed_ptr() restricts new_cpus to new_cpus & p->user_cpus_ptr
whenever their intersection is non-empty. Because dl_task_keeps_root_domain()
tests span against new_mask directly (and dl_tasks_keep_root_domain() and
dl_tasks_keep_without() skip cpusets where cpumask_subset(cp->effective_cpus,
new_cpus) holds), a deadline task counted in housekeeping span {0, 1} whose
p->user_cpus_ptr only intersects new_mask on isolated CPU {2} passes this
check (or the subset check when widening effective_cpus from {0, 1} to
{0, 1, 2}) and has its affinity restricted to {2} in the isolated root domain
without admission control. Intersecting new_mask with p->user_cpus_ptr when
cpumask_intersects(new_mask, p->user_cpus_ptr) is true and checking tasks even
when new_cpus grows would match __set_cpus_allowed_ptr().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=28

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked
  2026-11-16 19:05 ` [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
@ 2026-11-16 20:12   ` sashiko-bot
  0 siblings, 0 replies; 53+ messages in thread
From: sashiko-bot @ 2026-11-16 20:12 UTC (permalink / raw)
  To: Mathieu Desnoyers; +Cc: linux-rt-devel, tj, hannes, mkoutny, linux-kernel

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] sched/deadline: unstarted period in switched_to_dl() when
  !pi_blocked_on
- [Medium] sched/deadline: switched_to_dl() skips period reset on
  boosted pre-0-lag task
- [Medium] sched/deadline: stale deadline on rt_mutex waiter when donor
  or policy changes
--

commit 97588bc2bebff0e4575bb7d4c8f530868b8856a3
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Start a period for a task that enters the class blocked

This commit starts a new period in switched_to_dl() when a task blocked on
an rt_mutex enters dl_sched_class with an expired deadline so that the
rt_mutex waiter tree does not sort it with a deadline of 0.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 69a338c1526d0..ffa8b946e8047 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c

[ ... ]

> @@ -4352,6 +4363,17 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
>  	if (!task_on_rq_queued(p)) {
>  		add_rq_bw(&p->dl, &rq->dl);
>  
> +		/*
> +		 * A task that waits on an rt_mutex cannot wait for its wakeup.
> +		 * The chain walk sorts its waiter by its deadline first. A task
> +		 * new to the class has a deadline of 0, which goes before every
> +		 * real one. Start a new period now, as setup_new_dl_entity()
> +		 * does at an enqueue.
> +		 */
> +		if (dl_task_pi_blocked(p) && !p->dl.dl_throttled &&
> +		    dl_time_before(p->dl.deadline, rq_clock(rq)))
> +			replenish_dl_new_period(&p->dl, rq);

[Severity: Medium]
Does gating replenish_dl_new_period() on dl_task_pi_blocked(p) leave
p->dl.deadline at 0 when a sleeping task participates in rt_mutex priority
inheritance before waking up?

While a task sleeps in futex_wait_requeue_pi() on a non-PI futex,
p->pi_blocked_on is NULL.  If sched_setattr() or a PI boost moves p into
dl_sched_class there, switched_to_dl() leaves p->dl.deadline at 0, and a
later FUTEX_CMP_REQUEUE_PI calls rt_mutex_start_proxy_lock() ->
task_blocks_on_rt_mutex() without waking p, queuing its waiter ahead of all
deadline waiters with deadline 0.  Likewise, if a task holding an rt_mutex
is switched to SCHED_DEADLINE while sleeping on a non-rt_mutex wait,
p->dl.deadline stays 0 and rt_mutex_setprio() fails to boost p when a
deadline waiter with an earlier deadline blocks on that rt_mutex
(dl_entity_preempt(&pi_task->dl, &p->dl) is false against 0).

[Severity: Medium]
What happens when a task that left SCHED_DEADLINE before its 0-lag time is
PI-boosted back into dl_sched_class while blocked on an rt_mutex?

switched_from_dl() preserves p->dl.deadline and p->dl.dl_throttled until
inactive_timer fires.  If p then blocks on an rt_mutex while holding
another and is boosted back into dl_sched_class, enqueue_task_dl() is not
called because p is not queued, and switched_to_dl() checks
!p->dl.dl_throttled && dl_time_before(p->dl.deadline, rq_clock(rq)) without
checking is_dl_boosted(&p->dl).

If p->dl.dl_throttled is still set, switched_to_dl() skips
replenish_dl_new_period() and leaves p throttled with its expired deadline;
if p->dl.dl_timer then fires while p is boosted, dl_task_timer() bails out
on is_dl_boosted(dl_se) without clearing dl_throttled, which prevents
enqueue_dl_entity() from queuing p after deboost.  Even when p is not
throttled, if its old pre-0-lag p->dl.deadline is still in the future,
switched_to_dl() skips replenish_dl_new_period() and the chain walk sorts p
using its abandoned reservation's deadline instead of the donor's.

[Severity: Medium]
Does starting a period in switched_to_dl() leave p->dl.deadline stale if the
top PI donor on a blocked task changes while it remains in dl_sched_class?

If a non-deadline task p owns rt_mutex M1 and blocks on rt_mutex M2, an
initial deadline waiter W1 on M1 moves p into dl_sched_class and
switched_to_dl() sets p->dl.deadline from W1.  If a second deadline waiter
W2 with a much earlier deadline then blocks on M1, rt_mutex_setprio(p, W2)
keeps prev_class == next_class == &dl_sched_class and sets p->dl.pi_se =
W2->dl.pi_se with ENQUEUE_REPLENISH.  Because p is not queued,
sched_change_end() calls prio_changed_dl() instead of enqueue_task_dl() or
switched_to_dl(), leaving p->dl.deadline at W1's later deadline.
rt_mutex_adjust_prio_chain() then sees an unchanged waiter deadline and
stops the chain walk on M2, and when p later wakes up, update_dl_entity()
keeps W1's future deadline if dl_entity_overflow() is not triggered.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116190549.230119-1-mathieu.desnoyers@efficios.com?part=31

^ permalink raw reply	[flat|nested] 53+ messages in thread

end of thread, other threads:[~2026-11-16 20:12 UTC | newest]

Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 02/37] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 03/37] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 04/37] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
2026-11-16 19:30   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 06/37] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 07/37] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
2026-11-16 19:42   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 09/37] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-16 19:38   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 11/37] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 12/37] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 13/37] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 14/37] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 15/37] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-16 19:44   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 17/37] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 18/37] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 19/37] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 20/37] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-16 19:31   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 22/37] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-16 19:56   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-16 20:01   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-16 19:47   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-16 19:50   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-16 19:51   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-16 20:08   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 29/37] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 30/37] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-16 20:12   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-16 20:08   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-16 20:04   ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 34/37] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 35/37] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 36/37] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
2026-11-16 19:49   ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®