[PATCH v2 00/21] sched/deadline: bandwidth accounting fixes and selftests

[PATCH v2 00/21] sched/deadline: bandwidth accounting fixes and selftests

[Mathieu Desnoyers]

Hi Juri,

Please find in this series seventeen fixes to SCHED_DEADLINE bandwidth
accounting, on top of your four in sched/urgent. Twenty-one patches:
the seventeen fixes, three that add the selftests catching them, and
one that shows a runqueue's deadline bandwidth in sched_debug. Nine of
the seventeen are new in v2, from the reviews of v1 and of the patches
those reviews brought; the changes are listed at the end.

One word before the details: I am at home with the concepts behind
SCHED_DEADLINE, but not with its implementation, and these patches were
written with an LLM assistant under my review ("How this was developed"
below). So I am asking those who know this code for a careful reading of
the fixes themselves. Most of them have a reproducer and were measured
in VMs, which says that the symptom goes away, not that the fix is the
one you would have written.

Three take the kernel down: a divide by zero in __dl_sub(), reachable
from a debugfs write to a fair server while its CPU goes offline or
comes online, a BUG() when a deadline task's affinity has no active CPU
left, which priority inheritance and CPU hotplug reach on a default
kernel, and a BUG() in the rebuild of the accounting for a task whose
isolated CPUs all went offline. One unlocks a cpuset_mutex it does not
hold. Two break the enforcement of a reservation: a throttled task
switched out of the class and back is left throttled with nothing to
replenish it, and a task whose replenishment timer stays armed from the
reservation it left is replenished at a time of that one. The other
eleven leave a root domain's total_bw, or a runqueue's this_bw, holding
what its tasks do not: the root domain then admits deadline tasks beyond
the capacity of its CPUs, or refuses every one of them and refuses to
offline its last CPU, and on the runqueue side the kernel warns of the
underflow.

They came out of a cgroup feature under development whose selftests hung
and flaked where the accounting drifts. Eight of the seventeen fail a
case of the selftests here, one more on a kernel built without cpusets,
five have standalone reproducers I can post, and three have none: patch
3 closes a window a test cannot time, patch 18 wants isolated CPUs
taken offline, and patch 20 a replenishment timed to the wrong period.

The fixes, by patch number
==========================

The four patches that are not fixes are missing from the list: 8, 10
and 14 add the selftests, 12 the sched_debug output.

 1  sched_setattr() with SCHED_FLAG_KEEP_PARAMS from SCHED_OTHER to
    SCHED_DEADLINE fails after reserving bandwidth that nothing gives
    back.
 2  SCHED_FLAG_KEEP_PARAMS keeps the policy as well, so a request that
    asks for another one is carried out by half and succeeds. On a task
    priority inheritance boosted into the deadline class the class does
    not change, so the request is admitted and reserves bandwidth the
    task never gets, nor releases.
 3  the same flag has the parameters read from the task with no lock
    held: a change of its own that lands in between leaves the request
    stale, and the admission moves total_bw and the runqueue's sums to a
    bandwidth the task does not have. Nothing is to be admitted for
    parameters that are kept.
 4  __dl_sub() and __dl_add() divide a change of total_bw by the number
    of active CPUs of the root domain, which is 0 while a CPU passes
    through the default one.
 5  dl_task_root_domain_cpu() BUG()s when a deadline task's mask has no
    active CPU, as a boosted mutex owner's does when its CPU goes
    offline.
 6  a throttled task switched out of SCHED_DEADLINE and back before its
    0-lag time is left throttled with nothing to replenish it: it never
    runs again, and a fatal signal cannot make it run.
 7  a sleeper leaving the class releases bandwidth the root domain no
    longer holds, or never held.
 9  a rebuild of the root domains' accounting (any cpuset partition
    change, CPU hotplug) forgets the bandwidth of the tasks that left
    the class, or died, before their 0-lag time: the inactive timer then
    removes it a second time, or from a root domain that never had it.
 11 a task whose wakeup is left to the target CPU has its bandwidth on
    no runqueue while task_cpu() already points at the destination: a
    parameter change meanwhile takes it off a runqueue that never held
    it, and this_bw underflows there for good.
 13 dl_rebuild_rd_accounting() is a stub without CONFIG_CPUSETS, so such
    a kernel loses the bandwidth of every deadline task at the first
    hotplug. A rebuild has nothing to serialize it against admission
    there either, where cpuset_mutex does not exist.
 15 three answers live side by side to "which root domain counts this
    task's bandwidth": the one of the CPU it is on (admission), the one
    of its cpuset (a rebuild of the accounting) and the one of the first
    active CPU of its own mask (a release, a move). They part ways where
    a CPU goes offline or a sleeping task is moved, and the bandwidth is
    then taken from a root domain that never had it. Every site that
    hands it out records a CPU of the root domain that counts it, and
    every site that takes it back or moves it reads that.
 16 a move takes a task's bandwidth off the root domain that counts it
    whenever the new mask leaves that root domain, on the strength of
    room nothing checks was made: a task that became a deadline task
    after cpuset_can_attach() counted them, and sched_setaffinity()
    with admission control off, have the room of no one. total_bw of a
    one-CPU partition root reaches 2^64 - 576716 that way.
 17 what cpuset_can_attach() set aside for a task that left the class,
    or died, before the attach stays in the destination's total_bw: 200
    moves of a task that flips its policy leave hundreds of thousands
    of units of a CPU behind. The tally of what the attach moved comes
    from the move itself, under the runqueue lock that decides it, so
    that CPU hotplug moving a task meanwhile cannot leave the
    reservation standing for bandwidth that stayed where it was.
 18 a rebuild of the accounting credits the bandwidth of a task that
    can run on housekeeping CPUs to the default root domain where an
    isolated CPU comes first in its mask, and BUG()s for a task whose
    isolated CPUs all went offline.
 19 the policy check of __sched_setscheduler() retries without clearing
    the flag that says it holds cpuset_mutex, so the exit path unlocks
    a cpuset_mutex that is someone else's by then. sched_setparam(),
    and sched_setattr() with SCHED_FLAG_KEEP_POLICY, reach it against a
    task leaving SCHED_DEADLINE.
 20 the replenishment timer of a task throttled before the end of its
    period stays armed where the task leaves the class and its
    parameters are cleared at its 0-lag time: back in the class, its
    next throttle does not reprogram that timer, and the replenishment
    comes at the period of the reservation it left.
 21 the runqueue a sleeping task is on counts none of the bandwidth the
    task gains where priority inheritance boosted it into the class:
    this_bw is then short of what running_bw takes at the wake-up,
    which the kernel warns about, and GRUB reclaims from sums that
    disagree.

Patch 1 moves the admission check last, where nothing between it and the
change can refuse the request after it reserved; patch 3 makes the leak
it fixes unreachable by a second route, and it keeps a refused request
from changing sched_reset_on_fork. Patch 12 gives sched_debug each
runqueue's this_bw and running_bw, which is what makes patch 11's bug
visible from userspace, and patches 8, 10 and 14 carry the selftests,
49 cases in three programs.

Patches 15 to 18 go together: 15 records the root domain that counts a
task's bandwidth, 16 and 17 are the two ways a cgroup attach moves or
strands bandwidth that record then exposes, and 18 is what a rebuild
does for a task no CPU of its root domain can hold. Patch 15 comes
after patch 9 because it rests on its record and generation; it is also
what makes the fallback of patch 5 moot, so that patch stands on its own
if you would rather take the crash fix alone. Patch 21 completes patch
11: where 11 left the bandwidth of a task waking elsewhere to the
enqueue to come, 21 hands it to the runqueue the task will wake on.

Testing
=======

In 8 QEMU guests of 8 vCPUs, the 49 cases pass on the series, every
time. No case fails on 4 vCPUs either, with PREEMPT_RT, with
CONFIG_SCHED_PROXY_EXEC on and off, or on a kernel built without
CONFIG_CPUSETS, where the cases that want a cpuset partition root skip.
No kernel warning in any run, in 16 guests over those five
configurations.

A bisect over the series, one kernel per patch that changes kernel code,
2 guests each and the same test binaries throughout, says what each
patch is worth in the suite:

    base  params_keep_class_change, params_keep_boosted,
          throttled_round_trip, pi_former_owner_asleep,
          dies_before_rebuild, switched_out_before_rebuild,
          moved_after_switch_out and random_mix fail
    +1    params_keep_class_change passes
    +2    params_keep_boosted passes
    +6    throttled_round_trip passes
    +7    pi_former_owner_asleep passes
    +9    the three rebuild cases and random_mix pass

params_waking needs patch 12's sched_debug sums to run at all: on a
kernel with patch 12 and without patch 11 it fails in 6 guests of 6, and
random_mix catches that bug too, in 1 of 3. Patch 13 shows in the five
hotplug cases of a kernel built without cpusets: three fail and two skip
for want of room without it, and the five pass with it. Patches 3, 4 and
5 move nothing in the suite, by design: patch 3 closes a window a test
cannot time, and an unfixed kernel dies on the other two, so their
reproducers stand apart.

That bisect covers the fifteen patches it was run over. Patches 16 and
17 came after it, from random_mix failing on a kernel carrying all
fifteen, and the suite passes with them as it did before. Patches 18 to
21 move nothing in it either; 19 and 21 have reproducers of their own,
the one of 19 reported by lockdep in under two seconds, and the one of
21 leaving this_bw of a runqueue at 157285 where 262142 is due.

How this was developed
======================

These patches, their messages and the tests were written by an LLM
coding assistant under my direction and review.

It started from the cgroup selftests of a feature I am developing, which
the assistant also wrote: one of them hung about once in 1350 runs, and
another failed about once in ten. Asking it to root-cause those two
flakes, rather than work around them, gave patches 6 and 9. Reviewing
your four fixes in sched/urgent with it gave patches 1 and 7. I then
asked for a selftest of the bandwidth accounting in its own right, and
for one area of coverage at a time: priority inheritance, parameter
changes, cpuset moves, deaths, sched_yield(), runtime overruns, GRUB
reclaiming, the fair servers, CPU hotplug, proxy execution, and a
randomized mix of all of them. That is deadline_bw_test, and building it
turned up patch 11, through the kernel's own this_bw underflow warning
under a parameter-change load, and patch 13, on a build without
cpusets. Patches 4 and 5 came from reading the code instead; their
reproducers were written afterwards, and each takes an unfixed kernel
down on the first round.

Patches 16 and 17 came out of the randomized case failing on its own
terms, which is what I wanted that case for.

Reviews of v1 and of the rework gave the rest, each one checked against
the code before anything was written; the ones that were already
answered by a later patch of the series were answered, not fixed twice.
The assistant got things wrong on the way as well: one diagnosis of
patch 11's bug was wrong twice over and was settled only by
instrumenting a kernel; one assertion it had put in a test was itself
racy and had to go; and the first shape of patch 15 wrote a flag under
one lock that the runqueue lock writes beside, which failed 14 VMs of
16 and is why patch 11 looks as it does. The per-patch bisect above
exists because I asked whether every fix had a reproducer of its own.

Changes since v1
================

Nine reviews of v1 by Sashiko AI, and one of the patches they brought by
a second model. The new patches, in the order of the series:

 - patches 2 and 3: a bandwidth leak through SCHED_FLAG_KEEP_PARAMS
   that patch 1 does not fix. On a task that priority inheritance
   boosted into the deadline class the class does not change, so the
   request is admitted and reserves bandwidth the kept parameters never
   give to the task: a SCHED_OTHER owner of a PTHREAD_PRIO_INHERIT
   mutex with a slice of 40ms, boosted by a deadline waiter, left 0.4
   of a CPU behind at every call. Patch 2 refuses a KEEP_PARAMS request
   that would change the policy, where only a change of class was
   refused before. That is an ABI change, so say if you would rather
   keep accepting it. New selftest case params_keep_boosted;
 - patch 7 (v1's 5) cleared the parameters of a task leaving the class,
   and that task can be the donor of a priority inheritance boost until
   the chain walk of rt_mutex_adjust_pi(). A replenishment of the
   boosted owner in that window would never leave its loop over a
   dl_period and a dl_runtime of 0, with the runqueue lock held and
   interrupts off. It clears the task's bandwidth alone now, which is
   what the double release needed; shown with a kernel carrying
   mdelay(20) in that window, which warns in replenish_dl_entity() and
   stops with the parameters cleared, and survives without;
 - patch 9 gained a generation counter, so that a release which follows
   a rebuild that dropped the task's bandwidth takes nothing;
 - patch 11 is a different fix from v1's 9: the bit that says a
   runqueue counts a task's bandwidth lives inside add_rq_bw() and
   sub_rq_bw(), which are idempotent now;
 - patch 13 gained dl_accounting_mutex, since without CONFIG_CPUSETS
   nothing serializes a rebuild of the accounting against admission;
 - patch 15 records the root domain that counts a task's bandwidth,
   because the fallback of patch 5 can point at one the last rebuild
   had not credited. A deadline task pinned to a CPU that goes offline,
   with admission control off, leaves its bandwidth behind in the root
   domain it was counted in without it: total_bw 471853 where 366996
   is due, the task's 10% of a CPU;
 - patches 16 and 17, the two ways a cgroup attach moves bandwidth no
   one made room for, or strands what it set aside;
 - patch 18, the rebuild of the accounting against isolated CPUs;
 - patches 19 and 20, two bugs of the baseline that the review of v1
   reported without tying them to a patch of it: the unbalanced
   cpuset_unlock(), which patch 13 would double, and the replenishment
   timer left armed. Patch 19's reproducer is a policy flipper against
   sched_setattr(SCHED_FLAG_KEEP_POLICY), which lockdep reports in
   under two seconds;
 - patch 21, the runqueue side of a task priority inheritance boosted
   into the class.

Also, in the selftest: the fair servers are put back after the 0-lag
times of the randomized mix's workers rather than before, where a
decrease of a server can be refused for want of room, which failed
random_mix about one run in ten; the error handling of every helper it
has; a settings file, since the suite takes longer than the 45 seconds
of the kselftest default; and the hotplug case of a departed task
offlines the CPU it checked it could, rather than the one the task last
ran on.

Two things the reviews turned up that this series does not change:

 - a rebuild drops the bandwidth of a departed task where the CPU
   recorded for it is no longer active, even where its root domain
   lives on. Fixing it means recording the root domain rather than a
   CPU, which root domains being freed and rebuilt does not make cheap.
   The undercount is bounded by the task's 0-lag time, and the
   generation of patch 9 keeps the release out, so nothing underflows;
 - sched_setscheduler() of a task that priority inheritance boosted
   into the deadline class to any policy outside it returns -EBUSY:
   sched_dl_overflow() has no branch for a task of the class with
   another policy, and its caller takes -1 for a refusal. Long-standing,
   and untouched here.

Based on tip sched/urgent, at commit a2b67e904401 ("sched/deadline:
Reject SCHED_DEADLINE on exited tasks"); patch 7 builds on
commit 41a7968022ca ("sched/deadline: Fix total_bw leak when switching
away from SCHED_DEADLINE while !queued") there.

Mathieu Desnoyers (21):
  sched: Reject SCHED_FLAG_KEEP_PARAMS before reserving deadline
    bandwidth
  sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy
  sched/deadline: Do not admit bandwidth for parameters that are kept
  sched/deadline: Do not divide by zero active CPUs of a root domain
  sched/deadline: Do not BUG() on a deadline task with no active CPU
    allowed
  sched/deadline: Do not lose a throttled task switched out and back
  sched/deadline: Do not release a departing sleeper's bandwidth twice
  selftests/sched: Add a test of SCHED_DEADLINE policy round trips
  sched/deadline: Keep departing bandwidth across a root domain rebuild
  selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain
    rebuild
  sched/deadline: Do not take a waking task's bandwidth off a runqueue
    it left
  sched/debug: Show the deadline bandwidth of each runqueue
  sched/deadline: Rebuild the root domains' accounting without cpusets
    too
  selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting
  sched/deadline: Record the root domain holding a task's bandwidth
  sched/deadline: Move a task's bandwidth only where room was made
  cgroup/cpuset: Give back the bandwidth an attach did not move
  sched/deadline: Credit a rebuild to a CPU that can hold the bandwidth
  sched: Clear cpuset_locked where the policy check retries
  sched/deadline: Cancel the replenishment timer with the parameters
  sched/deadline: Count the bandwidth a boosted task is given

 include/linux/cpuset.h                        |    5 -
 include/linux/sched.h                         |   22 +
 include/linux/sched/deadline.h                |   21 +-
 kernel/cgroup/cpuset-internal.h               |    2 +
 kernel/cgroup/cpuset.c                        |   51 +-
 kernel/sched/core.c                           |   19 +
 kernel/sched/deadline.c                       |  551 ++-
 kernel/sched/debug.c                          |    2 +
 kernel/sched/sched.h                          |    4 +
 kernel/sched/syscalls.c                       |   71 +-
 tools/testing/selftests/sched/.gitignore      |    3 +
 tools/testing/selftests/sched/Makefile        |    1 +
 .../selftests/sched/deadline_bw_test.c        | 3353 +++++++++++++++++
 .../selftests/sched/deadline_rebuild_test.c   |  462 +++
 .../selftests/sched/deadline_switch_test.c    |  332 ++
 tools/testing/selftests/sched/settings        |    1 +
 16 files changed, 4782 insertions(+), 118 deletions(-)
 create mode 100644 tools/testing/selftests/sched/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/deadline_rebuild_test.c
 create mode 100644 tools/testing/selftests/sched/deadline_switch_test.c
 create mode 100644 tools/testing/selftests/sched/settings

-- 
2.47.3

[PATCH v2 01/21] sched: Reject SCHED_FLAG_KEEP_PARAMS before reserving deadline bandwidth

[Mathieu Desnoyers]

__sched_setscheduler() returns -EINVAL for a SCHED_FLAG_KEEP_PARAMS
request that would change scheduling class, after sched_dl_overflow()
has reserved the bandwidth of the request, and nothing gives it back.
Switching a SCHED_OTHER task to SCHED_DEADLINE that way fails and leaks
the runtime that sched_setattr() filled in from the task's slice, up to
100ms with a custom one, into the root domain's total_bw. Repeated, it
makes the root domain refuse every deadline task, and refuse to offline
its last CPU.

Reserve the bandwidth after that check, last of the checks. A refused
request also leaves sched_reset_on_fork as it was again.

Fixes: 8a70415351ba ("sched: Reject SCHED_FLAG_KEEP_PARAMS when it would change scheduling class")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 3fa447aabe7f..f0b7b2d52614 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -655,17 +655,6 @@ int __sched_setscheduler(struct task_struct *p,
 		goto recheck;
 	}
 
-	/*
-	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
-	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
-	 * is available.
-	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
-		retval = -EBUSY;
-		goto unlock;
-	}
-
-	p->sched_reset_on_fork = reset_on_fork;
 	oldprio = p->prio;
 
 	newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
@@ -699,6 +688,19 @@ int __sched_setscheduler(struct task_struct *p,
 		queue_flags |= DEQUEUE_CLASS;
 	}
 
+	/*
+	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
+	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
+	 * is available. Last of the checks: nothing gives back the
+	 * bandwidth it reserves.
+	 */
+	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+		retval = -EBUSY;
+		goto unlock;
+	}
+
+	p->sched_reset_on_fork = reset_on_fork;
+
 	scx_prepare_setscheduler(p, policy);
 
 	scoped_guard (sched_change, p, queue_flags) {
-- 
2.47.3

[PATCH v2 02/21] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy

[Mathieu Desnoyers]

SCHED_FLAG_KEEP_PARAMS keeps the parameters of a task: it skips
__setscheduler_params(), which is what sets p->policy. A request that
asks for another policy is then carried out by half, and succeeds:
SCHED_OTHER to SCHED_BATCH with SCHED_FLAG_KEEP_PARAMS returns 0 and
leaves the task where it was.

For SCHED_DEADLINE it costs bandwidth. A task that priority inheritance
boosted into the deadline class is of that class already, so
prev_class == next_class and the class check of
commit 8a70415351ba ("sched: Reject SCHED_FLAG_KEEP_PARAMS when it would
change scheduling class") lets the request through: sched_dl_overflow()
admits it and adds
its bandwidth, which the skipped __setscheduler_params() never gives to
the task. It keeps SCHED_NORMAL and a dl_bw of 0, so nothing releases
that bandwidth, and total_bw of the root domain holds it for good.

Refuse a policy change as well, where the class change was refused
alone.

A SCHED_OTHER owner of a PTHREAD_PRIO_INHERIT mutex with a slice of
40ms, boosted by a deadline waiter blocked on it, leaves 0.4 of a CPU
behind at every such call: total_bw 419424 -> 838854 and there to stay,
in 8-CPU VMs of v7.4-rc1 with the four deadline fixes of sched/urgent,
and of this series up to here. With this change the call is refused and
total_bw is as it was. The selftest added later in this series has the
case, params_keep_boosted.

Found by the Sashiko AI review of the first posting of this series.

Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index f0b7b2d52614..ab0a573e9b37 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -674,20 +674,25 @@ int __sched_setscheduler(struct task_struct *p,
 	prev_class = p->sched_class;
 	next_class = __setscheduler_class(policy, newprio);
 
-	if (prev_class != next_class) {
-		/*
-		 * SCHED_FLAG_KEEP_PARAMS prevents the class change from
-		 * actually happening (it skips __setscheduler_params and
-		 * the sched_class assignment). Reject contradictory
-		 * requests that ask for a class change with KEEP_PARAMS.
-		 */
-		if (attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) {
-			retval = -EINVAL;
-			goto unlock;
-		}
-		queue_flags |= DEQUEUE_CLASS;
+	/*
+	 * SCHED_FLAG_KEEP_PARAMS prevents the policy and the class change
+	 * from actually happening: it skips __setscheduler_params() and the
+	 * sched_class assignment. Reject the contradictory requests that
+	 * ask for either, rather than carry out half of what they ask.
+	 *
+	 * A class change with the policy kept is one of them: the class of a
+	 * policy depends on the priority that priority inheritance leaves,
+	 * and on whether a BPF scheduler is in charge.
+	 */
+	if ((attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (policy != p->policy || prev_class != next_class)) {
+		retval = -EINVAL;
+		goto unlock;
 	}
 
+	if (prev_class != next_class)
+		queue_flags |= DEQUEUE_CLASS;
+
 	/*
 	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
 	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
-- 
2.47.3

[PATCH v2 03/21] sched/deadline: Do not admit bandwidth for parameters that are kept

[Mathieu Desnoyers]

sys_sched_setattr() fills a SCHED_FLAG_KEEP_PARAMS request in from the
task itself (get_params()), with no lock held, and
__sched_setscheduler() then skips __setscheduler_params(): the task
keeps the parameters it has, and its dl_bw with them.
sched_dl_overflow() runs all the same.

A parameter change of the task's own that lands in between leaves the
request stale, and dl_param_changed() takes it for a change: total_bw of
the root domain, and this_bw and running_bw of its runqueue through
dl_change_utilization(), then hold a bandwidth the task does not have,
and the release at its 0-lag time, which takes the task's own dl_bw,
leaves the difference behind.

There is nothing to admit for parameters that are kept: skip the check.

Found by reading, after the Sashiko AI review of the first posting of
this series raised the case the previous patch refuses. Not reproduced:
it needs a parameter change of a deadline task to land inside the window
of a SCHED_FLAG_KEEP_PARAMS request on the same task.

Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index ab0a573e9b37..8eddb9d47d8e 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -698,8 +698,14 @@ int __sched_setscheduler(struct task_struct *p,
 	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
 	 * is available. Last of the checks: nothing gives back the
 	 * bandwidth it reserves.
+	 *
+	 * Nothing to admit for parameters that are kept
+	 * (SCHED_FLAG_KEEP_PARAMS): they are the ones the task has, and
+	 * sys_sched_setattr() read them from it with no lock held, so a
+	 * change of its own that landed in between would leave them stale.
 	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
 		retval = -EBUSY;
 		goto unlock;
 	}
-- 
2.47.3

[PATCH v2 04/21] sched/deadline: Do not divide by zero active CPUs of a root domain

[Mathieu Desnoyers]

__dl_sub() and __dl_add() spread a change of a root domain's total_bw
over the extra_bw of its active CPUs, dividing it by their number, which
their callers take from dl_bw_cpus(). That number is 0 for a root domain
whose CPUs all went offline, such as the default root domain, which
holds the offline CPUs: the division then traps.

A debugfs write of a fair server's runtime or period reaches it while
its CPU goes offline or comes online. From the rebuild of the root
domains in sched_cpu_deactivate() until the CPU is offline, and from the
CPU online until sched_cpu_activate(), the CPU sits in the default root
domain, with no active CPU. A write that leaves the server's bandwidth
as it is passes admission there, __dl_overflow() comparing that root
domain's capacity, 0, with total_bw less the old bandwidth plus the new,
0 as well, and dl_server_apply_params() divides. Any other value is
refused with EBUSY, higher directly, lower through the unsigned
subtraction. Once the CPU is offline the write is refused, since
commit 4043f5498416 ("sched/deadline: Reject debugfs dl_server writes
for offline CPUs").

Writing a fair server's own runtime back in a loop while its CPU goes
offline and online oopses on the first round, in 8-CPU VMs of v7.4-rc1
with the four deadline fixes of sched/urgent:

  Oops: divide error: 0000 [#1] SMP NOPTI
  CPU: 6 UID: 0 PID: 110 Comm: dl_server_div0
  RIP: 0010:dl_server_apply_params+0x335/0x380
  Call Trace:
   <TASK>
   sched_server_write_common.isra.0+0x1c0/0x260
   full_proxy_write+0x58/0x90
   vfs_write+0xcb/0x550
   ksys_write+0x6c/0xe0
   do_syscall_64+0x109/0x5d0
   entry_SYSCALL_64_after_hwframe+0x77/0x7f

inactive_task_timer() can reach it too, found by reading: it releases
the bandwidth of a task that left SCHED_DEADLINE, or died, before its
0-lag time from the root domain of the CPU the task is on, and that CPU
may go offline before the timer fires. With admission control off, so
may sched_setattr() to SCHED_DEADLINE of a task asleep on an offline
CPU.

With no active CPU there is no extra_bw to update: change total_bw
alone. 20 rounds of the reproducer leave no warning then.

Fixes: daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 963fda6ddb7f..dadda1a528ec 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
 	}
 }
 
+/*
+ * @cpus is the number of active CPUs of the root domain, whose extra_bw
+ * shares the change: none once they all went offline, and total_bw alone
+ * changes then.
+ */
 static inline
 void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw -= tsk_bw;
-	__dl_update(dl_b, (s32)tsk_bw / cpus);
+	if (cpus)
+		__dl_update(dl_b, (s32)tsk_bw / cpus);
 }
 
 static inline
 void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw += tsk_bw;
-	__dl_update(dl_b, -((s32)tsk_bw / cpus));
+	if (cpus)
+		__dl_update(dl_b, -((s32)tsk_bw / cpus));
 }
 
 static inline bool
-- 
2.47.3

[PATCH v2 05/21] sched/deadline: Do not BUG() on a deadline task with no active CPU allowed

[Mathieu Desnoyers]

dl_task_root_domain_cpu() finds the root domain that holds a deadline
task's bandwidth from the first active CPU of the task's mask, and
BUG()s when the mask has none. A mask can have none:

 - a task boosted into the deadline class by priority inheritance keeps
   the affinity of its own policy, which may allow one CPU alone, or a
   cpuset of one. dl_task_needs_bw_move() and set_cpus_allowed_dl()
   take it for a deadline task, dl_task(p) being true, and look the root
   domain up from the mask they are replacing, which allows that CPU,
   inactive by then: the cpuset code updates the masks of a cpuset's
   tasks once the CPU is inactive, and select_fallback_rq() widens the
   mask of a task it pushes off that CPU;
 - with admission control off (sched_rt_runtime_us at -1), a deadline
   task may be pinned to fewer CPUs than its root domain spans, and
   those may all go offline.

Both take the kernel down on the first try, in 8-CPU VMs of v7.4-rc1
with the four deadline fixes of sched/urgent. The priority inheritance
one, which asks for no unusual configuration, through the wakeup of the
boosted owner:

  kernel BUG at kernel/sched/deadline.c:3340!
  CPU: 7 UID: 0 PID: 0 Comm: swapper/7
  RIP: 0010:set_cpus_allowed_dl+0xf0/0x100
  Call Trace:
   <IRQ>
   set_cpus_allowed_force+0x9f/0xf0
   cpuset_cpus_allowed_fallback+0x12b/0x1d0
   select_fallback_rq+0x238/0x340
   try_to_wake_up+0x2a0/0xb70

Fall back to the CPU the task last ran on, which these callers used
before the commit this one fixes. Its root domain may have no active CPU
left: the previous patch has __dl_sub() and __dl_add() take that. Ten
rounds of each case leave no warning with it.

Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index dadda1a528ec..30a52e42c26d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3338,13 +3338,19 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
  * current cpus_mask (not yet updated to the new mask) to find the
  * actual root domain. Use cpus_mask rather than cpus_ptr, which
  * may be temporarily narrowed by migrate_disable().
+ *
+ * The mask has no active CPU left when the CPUs it allows went offline,
+ * which the affinity of a deadline task allows with admission control
+ * off, and that of a task boosted into the deadline class by priority
+ * inheritance always: fall back to the CPU the task last ran on.
  */
 static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
 {
 	unsigned int cpu;
 
 	cpu = cpumask_first_and(&p->cpus_mask, cpu_active_mask);
-	BUG_ON(cpu >= nr_cpu_ids);
+	if (cpu >= nr_cpu_ids)
+		cpu = task_cpu(p);
 	return cpu;
 }
 
-- 
2.47.3

[PATCH v2 06/21] sched/deadline: Do not lose a throttled task switched out and back

[Mathieu Desnoyers]

A SCHED_DEADLINE task that is throttled, switched to another policy and
switched back before its 0-lag time can be left throttled with nothing
to replenish it: runnable, queued as far as the core is concerned, but
on no dl_rq and with no timer pending. It never runs again, and a fatal
signal cannot make it run; only another switch out of SCHED_DEADLINE
requeues it.

 1. the task exhausts its runtime: dl_throttled is set, and the
    replenishment timer armed for the start of its next period;
 2. sched_setscheduler() switches it to SCHED_OTHER: switched_from_dl()
    calls task_non_contending(), which arms the inactive timer for the
    0-lag time; the parameters are kept until then, so that leaving and
    reentering SCHED_DEADLINE gains no bandwidth;
 3. the replenishment timer fires: dl_task_timer() finds a task that is
    not a deadline task and returns, dl_throttled still set;
 4. sched_setattr() switches it back before the 0-lag time:
    switched_to_dl() cancels the inactive timer, and enqueue_dl_entity()
    leaves a throttled task to its replenishment timer, which has fired
    already.

The window between the deadline and the 0-lag time is the overrun
scaled by period/runtime: about 100us for an overrun of 20us at 20%
bandwidth. The replenishment timer cleared the parameters, dl_throttled
included, of a task it found out of SCHED_DEADLINE until
commit 209a0cbda7a0 ("sched/deadline: Improve the tracking of active
utilization") moved that reset to the inactive timer, at the 0-lag
time.

Give the task the replenishment the timer owes it, as the timer already
does for one that is throttled and not queued: the overrun comes off the
periods that follow, the deadline moves past it, and dl_throttled and
dl_yielded are cleared. A return before the 0-lag time then finds an
entity it can run on, and a return after it finds the parameters reset
by the inactive timer, as before.

Clearing dl_throttled alone would leave the overrun in the runtime, for
the new period of the enqueue to forgive, the deadline behind, for
dl_check_constrained_dl() to throttle the task on again, and a yield of
that period set, for update_curr_dl_se() to throttle it at once. The
first version of this patch did, and the Sashiko AI review of the first
posting of this series pointed the three out.

The selftest added two patches later fails its throttled_round_trip
case 8 times out of 8, on v7.3 as on v7.4-rc1 with the four deadline
fixes of sched/urgent, and passes it 8 times out of 8 with this change.
It was found by a cgroup selftest under development,
test_concurrency_deadline, which hung about once in 1350 runs.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 30a52e42c26d..b646063a7b0d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1232,10 +1232,23 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 
 	/*
 	 * The task might have changed its scheduling policy to something
-	 * different than SCHED_DEADLINE (through switched_from_dl()).
+	 * different than SCHED_DEADLINE (through switched_from_dl()). It
+	 * keeps its parameters until its 0-lag time and may come back to
+	 * SCHED_DEADLINE before then, switched_to_dl() cancelling the
+	 * inactive timer: give it the replenishment this timer owes it, as
+	 * the !queued case below does, and leave the enqueue to its return.
+	 * A throttle left behind would never be lifted; an overrun left in
+	 * its runtime would be forgiven by the new period its enqueue
+	 * starts; and a yield left set would throttle it again at once.
 	 */
-	if (!dl_task(p))
+	if (!dl_task(p)) {
+		if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
+			sched_clock_tick();
+			update_rq_clock(rq);
+			replenish_dl_entity(dl_se);
+		}
 		goto unlock;
+	}
 
 	/*
 	 * The task might have been boosted by someone else and might be in the
-- 
2.47.3

[PATCH v2 07/21] sched/deadline: Do not release a departing sleeper's bandwidth twice

[Mathieu Desnoyers]

Commit 41a7968022ca ("sched/deadline: Fix total_bw leak when switching
away from SCHED_DEADLINE while !queued") releases the bandwidth of a
task that leaves SCHED_DEADLINE asleep with no inactive timer pending.
Three cases release what the root domain no longer holds, or never held:

 - the task keeps its parameters. inactive_task_timer() resets them
   once it has released the bandwidth (__dl_clear_params()); left in
   place, task_contending() and task_non_contending() take the task,
   now of another policy, for one with bandwidth when priority
   inheritance boosts it into the deadline class, and a deboost while
   it sleeps releases that bandwidth from total_bw a second time. The
   root domain then admits deadline tasks beyond its capacity;
 - a task deboosted from priority inheritance with no parameters of its
   own, and a special one (SCHED_FLAG_SUGOV), have no bandwidth in
   total_bw. Releasing 0 changes nothing, and the root domain that
   dl_task_root_domain_cpu() finds from the task's mask means nothing
   either: a sleeping task of another policy may have no active CPU in
   its mask, a CPU it is pinned to having gone offline, until its next
   wakeup widens it;
 - migrate_task_rq_dl() clears dl_non_contending when it cannot cancel
   the inactive timer because its handler runs. That handler, once it
   takes the task's locks, finds it out of SCHED_DEADLINE and releases
   total_bw itself.

Release for a task with parameters of its own, not special, and clear
its bandwidth afterwards, once sub_rq_bw(), which reads it, is done:
every path that would release it again then takes a dl_bw of 0, and a
handler that runs late finds nothing to release. A test of
hrtimer_active() would not tell that handler apart: it stays true after
the handler returns, until the hrtimer code clears its running state,
and the bandwidth of a task switched out meanwhile would never be
released.

The parameters themselves stay, where inactive_task_timer() resets them
along with the bandwidth, because the task may still be the donor of a
priority inheritance boost: rt_mutex_adjust_pi() walks the chain and
deboosts the owner after the runqueue lock of the policy change is
dropped, and until it does, a replenishment of that owner, which
update_curr_dl_se() does at once for a boosted entity, would run
replenish_dl_entity() on a donor whose dl_runtime and dl_period are 0.
It would warn and then never leave

	while (dl_se->runtime <= 0) {
		dl_se->deadline += pi_of(dl_se)->dl_period;
		dl_se->runtime += pi_of(dl_se)->dl_runtime;
	}

with the runqueue lock held and interrupts off. __setparam_dl() sets
the parameters anew if the task comes back, and inactive_task_timer()
resets them at its 0-lag time if it does not.

The deadline_bw_test selftest, added later in this series, reproduces
the first case as pi_former_owner_asleep: in a one-CPU partition with
room, the waiter of its second round is refused as a deadline task,
total_bw having wrapped. The other two were found by reading.

The donor window was found by the Sashiko AI review of the first posting
of this series, and shown with a kernel carrying mdelay(20) before
rt_mutex_adjust_pi() to widen it: a deadline task blocks on a
PTHREAD_PRIO_INHERIT mutex with its runtime unspent, and is switched to
SCHED_OTHER while the owner it boosts spins. Clearing the parameters
there, that kernel warns in replenish_dl_entity() and stops, on round 4
and round 3 of two 8-CPU VMs; clearing the bandwidth alone, both go
through their eight rounds.

Fixes: 41a7968022ca ("sched/deadline: Fix total_bw leak when switching away from SCHED_DEADLINE while !queued")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b646063a7b0d..624cabdec965 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3573,6 +3573,8 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	dec_dl_tasks_cs(p);
 
 	if (!task_on_rq_queued(p)) {
+		bool released = false;
+
 		if (p->dl.dl_non_contending) {
 			/*
 			 * Inactive timer is armed. However, p is leaving
@@ -3585,12 +3587,18 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * release total_bw via __dl_sub().
 			 */
 			sub_running_bw(&p->dl, &rq->dl);
-		} else {
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
 			/*
 			 * No pending inactive timer: the task never ran as
 			 * DEADLINE or the timer already fired while still
 			 * DEADLINE (only dropping running_bw at that time).
 			 * Nobody else will release total_bw, do it now.
+			 *
+			 * Not for a task without parameters of its own, one
+			 * deboosted from priority inheritance, whose mask may
+			 * have no active CPU for dl_task_root_domain_cpu(),
+			 * nor for a special one: neither has bandwidth in
+			 * total_bw.
 			 */
 			unsigned int cpu = dl_task_root_domain_cpu(p);
 			struct dl_bw *dl_b = &cpu_rq(cpu)->rd->dl_bw;
@@ -3598,8 +3606,29 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			raw_spin_lock(&dl_b->lock);
 			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
 			raw_spin_unlock(&dl_b->lock);
+			released = true;
 		}
 		sub_rq_bw(&p->dl, &rq->dl);
+		/*
+		 * Released: its bandwidth is no one's to release again, and
+		 * a dl_bw of 0 is what every path that would takes. Kept, it
+		 * would make the task one with bandwidth to task_contending()
+		 * and task_non_contending() once priority inheritance boosts
+		 * it, and a deboost while it sleeps would release it a second
+		 * time. So would the inactive timer's handler when it runs
+		 * late, dl_non_contending cleared under it by
+		 * migrate_task_rq_dl(): it finds !dl_task(p).
+		 *
+		 * The parameters themselves stay, where
+		 * inactive_task_timer() resets them with the bandwidth: the
+		 * task may still be the donor of a priority inheritance
+		 * boost until rt_mutex_adjust_pi() walks the chain, outside
+		 * this runqueue lock, and replenish_dl_entity() of the
+		 * boosted owner would never leave its loop over a dl_period
+		 * and a dl_runtime of 0.
+		 */
+		if (released)
+			p->dl.dl_bw = 0;
 	}
 
 	/*
-- 
2.47.3

[PATCH v2 08/21] selftests/sched: Add a test of SCHED_DEADLINE policy round trips

[Mathieu Desnoyers]

Add deadline_switch_test, with a case for the fix two patches back and
one for commit 41a7968022ca ("sched/deadline: Fix total_bw leak when
switching away from SCHED_DEADLINE while !queued"):

 - throttled_round_trip: a task with a runtime of 10ms per 2s period
   spins, so that each replenishment throttles it again at once. Right
   after a throttle it is switched to SCHED_OTHER, and back 200us after
   its deadline: past its replenishment timer, and before its 0-lag
   time, which the overrun scaled by 200 puts later. It must run again
   within a period and a half. Three attempts;
 - asleep_round_trip: a task of 9ms every 10ms, running 0.2ms and
   sleeping 50ms, is switched to SCHED_OTHER and back while it sleeps
   past its 0-lag time, enough times that a leak of its bandwidth at
   each round trip fills the root domain: sched_setattr() then fails
   with EBUSY. Skipped without admission control (sched_rt_runtime_us
   at -1). A kernel that leaks is left without SCHED_DEADLINE bandwidth
   until a reboot, so this case runs last.

In 8 VMs of 8 vCPUs each, v7.3 fails both cases, and v7.4-rc1 with the
four deadline fixes of sched/urgent fails the first; with the two
previous patches as well both pass; every time.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/sched/.gitignore      |   1 +
 tools/testing/selftests/sched/Makefile        |   1 +
 .../selftests/sched/deadline_switch_test.c    | 332 ++++++++++++++++++
 3 files changed, 334 insertions(+)
 create mode 100644 tools/testing/selftests/sched/deadline_switch_test.c

diff --git a/tools/testing/selftests/sched/.gitignore b/tools/testing/selftests/sched/.gitignore
index 6996d4654d92..d97f40c21ab5 100644
--- a/tools/testing/selftests/sched/.gitignore
+++ b/tools/testing/selftests/sched/.gitignore
@@ -1 +1,2 @@
 cs_prctl_test
+deadline_switch_test
diff --git a/tools/testing/selftests/sched/Makefile b/tools/testing/selftests/sched/Makefile
index 099ee9213557..ae9421eabf15 100644
--- a/tools/testing/selftests/sched/Makefile
+++ b/tools/testing/selftests/sched/Makefile
@@ -10,5 +10,6 @@ LDLIBS += -lpthread
 
 TEST_GEN_FILES := cs_prctl_test
 TEST_PROGS := cs_prctl_test
+TEST_GEN_PROGS := deadline_switch_test
 
 include ../lib.mk
diff --git a/tools/testing/selftests/sched/deadline_switch_test.c b/tools/testing/selftests/sched/deadline_switch_test.c
new file mode 100644
index 000000000000..81125d81b618
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline_switch_test.c
@@ -0,0 +1,332 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A SCHED_DEADLINE task switched to SCHED_OTHER and back:
+ *
+ *  - throttled: switched out after its runtime ran out, and back after
+ *    its deadline but before its 0-lag time, it must run again, the
+ *    replenishment it missed while out of SCHED_DEADLINE or not;
+ *  - asleep past its 0-lag time: its bandwidth must leave the root domain
+ *    when it leaves SCHED_DEADLINE, or the switch back adds it again and
+ *    sched_setattr() soon refuses SCHED_DEADLINE with EBUSY.
+ *
+ * On a kernel that leaks, the second case leaves the root domain without
+ * bandwidth for new SCHED_DEADLINE tasks until a reboot.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+/* Written by the child, read by the parent. */
+struct shared {
+	uint64_t stamp;		/* last time the child ran */
+	uint64_t run_start;	/* when it last resumed after a gap */
+	uint64_t runs;		/* resumptions after a gap */
+	uint64_t sleeps;	/* sleeps begun */
+};
+
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void sleep_until(uint64_t t)
+{
+	struct timespec ts = { .tv_sec = t / NSEC_PER_SEC, .tv_nsec = t % NSEC_PER_SEC };
+
+	while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL) == EINTR)
+		;
+}
+
+static int set_deadline(pid_t pid, uint64_t runtime, uint64_t period)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+static struct shared *sh;
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	/* A task lost in SCHED_DEADLINE runs again, and dies, once out of it. */
+	set_other(pid);			/* gone already: nothing to switch */
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+/* The child died before the case was done: @status says whether it skips. */
+static int gone(int status)
+{
+	if (WIFEXITED(status) && WEXITSTATUS(status) == KSFT_SKIP) {
+		ksft_print_msg("SCHED_DEADLINE refused\n");
+		return KSFT_SKIP;
+	}
+	if (WIFSIGNALED(status))
+		ksft_print_msg("the task died of signal %d\n", WTERMSIG(status));
+	else
+		ksft_print_msg("the task exited with %d\n", WEXITSTATUS(status));
+	return KSFT_FAIL;
+}
+
+/* Spins: runs out of its runtime at once, and is throttled. */
+static void spinner(void)
+{
+	uint64_t last = now_ns();
+
+	for (;;) {
+		uint64_t t = now_ns();
+
+		if (t - last > 2 * NSEC_PER_MSEC) {
+			__atomic_store_n(&sh->run_start, t, __ATOMIC_RELAXED);
+			__atomic_fetch_add(&sh->runs, 1, __ATOMIC_RELEASE);
+		}
+		__atomic_store_n(&sh->stamp, t, __ATOMIC_RELEASE);
+		last = t;
+	}
+}
+
+#define LOST_RUNTIME	(10 * NSEC_PER_MSEC)
+#define LOST_PERIOD	(2 * NSEC_PER_SEC)
+#define LOST_MARGIN	(200 * NSEC_PER_USEC)
+
+/*
+ * The window between the deadline and the 0-lag time is the overrun
+ * scaled by period/runtime: 200 here, so that an overrun of 2us leaves
+ * 400us. A runtime longer than a tick keeps an overrun caught at the tick
+ * from carrying past the next replenishment, which comes a period after
+ * the last. The switch back comes LOST_MARGIN after the deadline, once
+ * the replenishment timer has fired on a task out of SCHED_DEADLINE.
+ */
+static int test_throttled_round_trip(void)
+{
+	int attempt, ret = KSFT_PASS, err;
+	pid_t pid;
+
+	memset(sh, 0, sizeof(*sh));
+	pid = fork();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+			_exit(KSFT_SKIP);
+		spinner();
+	}
+	for (attempt = 0; attempt < 3 && ret == KSFT_PASS; attempt++) {
+		uint64_t runs, r, s, t;
+
+		/* A resumption at a replenishment, D, then its throttle. */
+		runs = __atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE) == runs) {
+			if (now_ns() - t > 3 * LOST_PERIOD) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not resume within 3 periods\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		r = __atomic_load_n(&sh->run_start, __ATOMIC_RELAXED);
+		do {
+			s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+			sleep_ns(2 * NSEC_PER_MSEC);
+		} while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) != s);
+		/* Throttled until D + period: out, and back after that. */
+		err = set_other(pid);
+		if (!err) {
+			sleep_until(r + LOST_PERIOD + LOST_MARGIN);
+			err = set_deadline(pid, LOST_RUNTIME, LOST_PERIOD);
+		}
+		if (err) {
+			ksft_print_msg("policy switch: %s\n", strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(10 * NSEC_PER_MSEC);
+		s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s &&
+		       now_ns() - t < LOST_PERIOD + LOST_PERIOD / 2)
+			sleep_ns(NSEC_PER_MSEC);
+		if (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s) {
+			ksft_print_msg("throttled, switched out and back: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+#define LEAK_RUNTIME	(9 * NSEC_PER_MSEC)
+#define LEAK_PERIOD	(10 * NSEC_PER_MSEC)
+
+/* Runs 0.2ms, sleeps 50ms. */
+static void sleeper(void)
+{
+	for (;;) {
+		uint64_t end = now_ns() + 200 * NSEC_PER_USEC;
+
+		while (now_ns() < end)
+			;
+		__atomic_fetch_add(&sh->sleeps, 1, __ATOMIC_RELEASE);
+		sleep_ns(50 * NSEC_PER_MSEC);
+	}
+}
+
+/*
+ * 90% of a CPU per round trip: a leak fills the root domain within about
+ * as many round trips as it has CPUs.
+ */
+static int test_asleep_round_trip(void)
+{
+	int i, trips = get_nprocs() * 10 / 9 + 3, ret = KSFT_PASS, err;
+	pid_t pid;
+	FILE *f;
+
+	f = fopen("/proc/sys/kernel/sched_rt_runtime_us", "r");
+	if (f) {
+		long rt = 0;
+
+		if (fscanf(f, "%ld", &rt) == 1 && rt < 0) {
+			fclose(f);
+			ksft_print_msg("no admission control: a leak cannot be seen\n");
+			return KSFT_SKIP;
+		}
+		fclose(f);
+	}
+	memset(sh, 0, sizeof(*sh));
+	pid = fork();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LEAK_RUNTIME, LEAK_PERIOD))
+			_exit(KSFT_SKIP);
+		sleeper();
+	}
+	for (i = 0; i < trips; i++) {
+		uint64_t sleeps = __atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE);
+		uint64_t t = now_ns();
+
+		while (__atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE) == sleeps) {
+			if (now_ns() - t > NSEC_PER_SEC) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not run within 1s\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		/* Asleep, well past its 0-lag time. */
+		sleep_ns(5 * NSEC_PER_MSEC);
+		err = set_other(pid);
+		if (!err)
+			err = set_deadline(pid, LEAK_RUNTIME, LEAK_PERIOD);
+		if (err) {
+			ksft_print_msg("switch back refused after %d of %d round trips: %s\n",
+				       i, trips, strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	sh = mmap(NULL, sizeof(*sh), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (sh == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	ksft_print_header();
+	ksft_set_plan(2);
+	/* The first one first: the second fills the root domain where it leaks. */
+	report("throttled_round_trip", test_throttled_round_trip());
+	report("asleep_round_trip", test_asleep_round_trip());
+	ksft_finished();
+}
-- 
2.47.3

[PATCH v2 09/21] sched/deadline: Keep departing bandwidth across a root domain rebuild

[Mathieu Desnoyers]

A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
its bandwidth in the total_bw of its root domain until that time, when
inactive_task_timer() removes it from the root domain of the CPU the
task is on (commit 387e31300b57 ("sched/deadline: Fix the update of the
total -deadline utilization")). Two things break that:

 1. a rebuild of the root domains' accounting,
    dl_rebuild_rd_accounting() on any change of the cpuset partitions
    and on CPU hotplug, clears every root domain and adds back the
    bandwidth of the deadline tasks it finds in the cpusets. A task out
    of the policy is not one, and a dead one is in no cpuset any more:
    its bandwidth is dropped, and the timer then removes it again, from
    whichever root domain its CPU is in by then. total_bw wraps around,
    and that root domain admits deadline tasks beyond its capacity
    until the next rebuild;
 2. a task out of the policy may move meanwhile, as any other task, to
    a CPU of another root domain, moved to another cpuset partition:
    the timer then removes the bandwidth from that root domain, which
    never had it, and leaves it for good in the one that has it.

Keep each departing task on a list, from its departure to its timer,
with a CPU of the root domain that holds its bandwidth: the CPU a dying
task runs on, and for one that leaves the policy, a CPU of its mask
(dl_task_root_domain_cpu()), since the CPU it last ran on may be of
another root domain, when it was moved asleep. A rebuild puts that
bandwidth back in the root domain of that CPU, and the timer removes it
from there.

dl_non_contending tells a task departed, its inactive timer armed and
that timer's handler yet to run; hrtimer_active() stays true for a
while after the handler returns, and would leave the task on the list
with nothing to take it off. A count of the rebuilds, which each task
records when its bandwidth is counted, settles the races with a death
and with a CPU going inactive: a rebuild that dropped the bandwidth
since leaves nothing to remove, as for a task that exits while a
rebuild runs, which neither the cpuset walk, skipping exiting tasks,
nor the list, if it is not dead yet when the list is put back, finds. A
switch back to SCHED_DEADLINE before the timer has sched_dl_overflow()
release the old bandwidth from the root domain that keeps it, the new
one reserved in the task's current root domain; a boost into
the deadline class by priority inheritance in that window, whose
switched_to_dl() cancels the timer, takes the task off the list, its
bandwidth removed when it leaves the class again, from the root domain
of its mask by then, as before. The list's lock nests inside the rq and
pi locks, and outside the root domains' dl_bw locks.

dl_cookie and dl_bw_visited() have no user outside deadline.c left:
make them static.

The selftest added by the next patch fails its three cases every time
without this change, and passes them with it. A standalone reproducer,
a task of 400ms every second that runs 380ms and then exits, switches
to SCHED_OTHER or moves, a partition root made or unmade before its
0-lag time, has a one-CPU partition root admit two tasks of 600ms every
second on v7.3, every time, and refuse the second with this change, in
36 runs out of 36. It was found by a cgroup selftest under development,
test_concurrency_deadline_domain, which failed about once in ten runs
when it followed a test whose deadline task had died just before.

Fixes: 387e31300b57 ("sched/deadline: Fix the update of the total -deadline utilization")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h          |  14 ++
 include/linux/sched/deadline.h |   5 +-
 kernel/cgroup/cpuset.c         |  11 +-
 kernel/sched/deadline.c        | 230 ++++++++++++++++++++++++++++-----
 4 files changed, 216 insertions(+), 44 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e..0633a73579db 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -734,6 +734,20 @@ struct sched_dl_entity {
 	 */
 	struct hrtimer			inactive_timer;
 
+	/*
+	 * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
+	 * time" keeps its bandwidth in its root domain until the inactive
+	 * timer removes it. @dl_departed_cpu is then a CPU of the root domain
+	 * that counts that bandwidth, and @dl_departed_node links it on the
+	 * list that a rebuild of the root domains' accounting puts that
+	 * bandwidth back from; -1 otherwise. @dl_bw_gen is the
+	 * rebuild that last counted the task's bandwidth: one since then
+	 * dropped it, and nothing is to be removed.
+	 */
+	struct list_head		dl_departed_node;
+	int				dl_departed_cpu;
+	unsigned int			dl_bw_gen;
+
 	/*
 	 * Bits for DL-server functionality. Also see the comment near
 	 * dl_server_update().
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..bdf39ac1d43d 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -32,7 +32,7 @@ static inline bool dl_time_before(u64 a, u64 b)
 struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
-extern void dl_clear_root_domain_cpu(int cpu);
+extern void dl_clear_root_domains(void);
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
@@ -43,9 +43,6 @@ extern void dl_clear_root_domain_cpu(int cpu);
 extern bool dl_task_needs_bw_move(struct task_struct *p,
 				  const struct cpumask *new_mask);
 
-extern u64 dl_cookie;
-extern bool dl_bw_visited(int cpu, u64 cookie);
-
 static inline bool dl_server(struct sched_dl_entity *dl_se)
 {
 	return dl_se->dl_server;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..4159abfbbbb5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -985,21 +985,14 @@ void dl_rebuild_rd_accounting(void)
 {
 	struct cpuset *cs = NULL;
 	struct cgroup_subsys_state *pos_css;
-	int cpu;
-	u64 cookie = ++dl_cookie;
 
 	lockdep_assert_cpuset_lock_held();
 	lockdep_assert_cpus_held();
 	lockdep_assert_held(&sched_domains_mutex);
 
-	rcu_read_lock();
-
-	for_each_possible_cpu(cpu) {
-		if (dl_bw_visited(cpu, cookie))
-			continue;
+	dl_clear_root_domains();
 
-		dl_clear_root_domain_cpu(cpu);
-	}
+	rcu_read_lock();
 
 	cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
 
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 624cabdec965..2729c108b24b 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -163,7 +163,13 @@ static inline unsigned long dl_bw_capacity(int i)
 	}
 }
 
-bool dl_bw_visited(int cpu, u64 cookie)
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+static u64 dl_cookie;
+
+static bool dl_bw_visited(int cpu, u64 cookie)
 {
 	struct root_domain *rd = cpu_rq(cpu)->rd;
 
@@ -217,6 +223,88 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
 	       cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
 }
 
+/*
+ * A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
+ * its bandwidth in total_bw until then: the inactive timer removes it. A
+ * rebuild of the root domains' accounting meanwhile (dl_clear_root_domains()
+ * and dl_add_task_root_domain(), on any change of the partitions or CPU
+ * hotplug) finds no such task among the deadline tasks of the cpusets: it
+ * puts that bandwidth back from dl_departed_list, which holds them from
+ * their departure to their timer, in the root domain each left, found by
+ * its mask then (dl_task_root_domain_cpu()), or by the CPU it dies on: the
+ * CPU it last ran on may be of another one, when it was moved asleep. The
+ * task may run anywhere meanwhile, as another policy; its timer holds a
+ * reference to it.
+ *
+ * dl_bw_gen counts the rebuilds, and each task records the one that last
+ * counted its bandwidth. A rebuild that dropped it since then leaves
+ * nothing to remove: the CPU recorded was no longer active, or the task
+ * exited while the rebuild ran, which the cpuset walk skips, and was not
+ * dead yet when the list was put back. Both counts, and the list, are under
+ * dl_departed_lock, taken with interrupts disabled (under the task's rq
+ * lock or pi_lock, or by the rebuild) and before any dl_bw lock.
+ */
+static LIST_HEAD(dl_departed_list);
+static DEFINE_RAW_SPINLOCK(dl_departed_lock);
+static unsigned int dl_bw_gen;
+
+/*
+ * @p leaves SCHED_DEADLINE, or dies, its bandwidth kept for now in the root
+ * domain of @cpu. Under its rq lock.
+ */
+static void dl_task_depart(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	if (dl_se->dl_departed_cpu >= 0)
+		return;
+	guard(raw_spinlock)(&dl_departed_lock);
+	dl_se->dl_departed_cpu = cpu;
+	list_add(&dl_se->dl_departed_node, &dl_departed_list);
+}
+
+/*
+ * @p comes back to the deadline class, its bandwidth kept as a deadline
+ * task's again. Under its rq lock.
+ */
+static void dl_task_undepart(struct task_struct *p)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	dl_se->dl_departed_cpu = -1;
+	list_del(&dl_se->dl_departed_node);
+}
+
+/*
+ * Remove @p's bandwidth from the root domain that counts it: that of the
+ * CPU recorded when it departed, else that of @cpu. Nothing to remove
+ * when a rebuild of the accounting has dropped it; and no rebuild running
+ * counts it again. Under @p's rq lock.
+ */
+static void dl_task_release_bw(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct dl_bw *dl_b;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	if (dl_se->dl_departed_cpu >= 0) {
+		cpu = dl_se->dl_departed_cpu;
+		dl_se->dl_departed_cpu = -1;
+		list_del(&dl_se->dl_departed_node);
+	}
+	if (dl_se->dl_bw_gen == dl_bw_gen) {
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+	}
+	dl_se->dl_bw_gen = dl_bw_gen;
+}
+
 static inline
 void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
@@ -449,13 +537,9 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				sub_running_bw(dl_se, dl_rq);
 
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
-				struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 				if (READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				raw_spin_lock(&dl_b->lock);
-				__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
-				raw_spin_unlock(&dl_b->lock);
+				dl_task_release_bw(p, task_cpu(p));
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -464,8 +548,18 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 	}
 
 	dl_se->dl_non_contending = 1;
-	if (!dl_server(dl_se))
-		get_task_struct(dl_task_of(dl_se));
+	if (!dl_server(dl_se)) {
+		struct task_struct *p = dl_task_of(dl_se);
+
+		get_task_struct(p);
+		/*
+		 * Dying: its bandwidth is in the root domain of the CPU it
+		 * runs on. switched_from_dl() records one that leaves
+		 * SCHED_DEADLINE.
+		 */
+		if (READ_ONCE(p->__state) == TASK_DEAD)
+			dl_task_depart(p, task_cpu(p));
+	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
 }
@@ -2191,17 +2285,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		goto no_task;
 
 	if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
-		struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 		if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
 			sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
 			sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
 			dl_se->dl_non_contending = 0;
 		}
 
-		raw_spin_lock(&dl_b->lock);
-		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-		raw_spin_unlock(&dl_b->lock);
+		dl_task_release_bw(p, task_cpu(p));
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3485,12 +3575,33 @@ void dl_add_task_root_domain(struct task_struct *p)
 	dl_get_task_effective_cpus(p, msk);
 	cpu = cpumask_first_and(cpu_active_mask, msk);
 	BUG_ON(cpu >= nr_cpu_ids);
+
+	raw_spin_lock(&dl_departed_lock);
+	/*
+	 * Counted already by this rebuild: put back by
+	 * dl_clear_root_domains() as a departed task, which priority
+	 * inheritance boosted into the class since, or its bandwidth
+	 * gone since (dl_task_release_bw()), as that of one that died
+	 * after the walk found it. One that died since with its 0-lag
+	 * time ahead goes back to the root domain of the CPU recorded
+	 * (dl_task_depart()), if active.
+	 */
+	if (p->dl.dl_bw_gen == dl_bw_gen)
+		goto unlock;
+	if (p->dl.dl_departed_cpu >= 0) {
+		cpu = p->dl.dl_departed_cpu;
+		if (!cpu_active(cpu))
+			goto unlock;
+	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
 
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_gen = dl_bw_gen;
+unlock:
+	raw_spin_unlock(&dl_departed_lock);
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 }
 
@@ -3548,9 +3659,45 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
-void dl_clear_root_domain_cpu(int cpu)
+/*
+ * Clear every root domain ahead of a rebuild of their accounting, which
+ * dl_add_task_root_domain() then fills with the deadline tasks of the
+ * cpusets: put back their servers, and the bandwidth of the tasks that
+ * left SCHED_DEADLINE, or died, before their 0-lag time, which no cpuset
+ * holds as a deadline task, in the root domain of the CPU recorded for
+ * each; dropped if it is no longer active.
+ */
+void dl_clear_root_domains(void)
 {
-	dl_clear_root_domain(cpu_rq(cpu)->rd);
+	struct sched_dl_entity *dl_se;
+	u64 cookie = ++dl_cookie;
+	struct dl_bw *dl_b;
+	int cpu;
+
+	guard(raw_spinlock_irqsave)(&dl_departed_lock);
+	dl_bw_gen++;
+	for_each_possible_cpu(cpu) {
+		if (!dl_bw_visited(cpu, cookie))
+			dl_clear_root_domain(cpu_rq(cpu)->rd);
+	}
+	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
+		cpu = dl_se->dl_departed_cpu;
+		/*
+		 * Counted in the generation that ends here, and nowhere
+		 * since: one a rebuild dropped, for want of an active CPU to
+		 * count it, is not to be put back, and its release takes
+		 * nothing away either (dl_task_release_bw()).
+		 */
+		if (dl_se->dl_bw_gen != dl_bw_gen - 1)
+			continue;
+		if (!cpu_active(cpu))
+			continue;
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_add(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+		dl_se->dl_bw_gen = dl_bw_gen;
+	}
 }
 
 static void switched_from_dl(struct rq *rq, struct task_struct *p)
@@ -3584,7 +3731,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * or sub_rq_bw (below) will complain.
 			 *
 			 * inactive_task_timer() will find !dl_task(p) and
-			 * release total_bw via __dl_sub().
+			 * release total_bw (dl_task_release_bw()).
 			 */
 			sub_running_bw(&p->dl, &rq->dl);
 		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
@@ -3600,12 +3747,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * nor for a special one: neither has bandwidth in
 			 * total_bw.
 			 */
-			unsigned int cpu = dl_task_root_domain_cpu(p);
-			struct dl_bw *dl_b = &cpu_rq(cpu)->rd->dl_bw;
-
-			raw_spin_lock(&dl_b->lock);
-			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
-			raw_spin_unlock(&dl_b->lock);
+			dl_task_release_bw(p, dl_task_root_domain_cpu(p));
 			released = true;
 		}
 		sub_rq_bw(&p->dl, &rq->dl);
@@ -3635,9 +3777,19 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 * We cannot use inactive_task_timer() to invoke sub_running_bw()
 	 * at the 0-lag time, because the task could have been migrated
 	 * while SCHED_OTHER in the meanwhile.
+	 *
+	 * Nor can it take the task's bandwidth from the root domain of the
+	 * CPU the task is on by then, and a rebuild of the root domains
+	 * meanwhile would not find it among the deadline tasks: record a
+	 * CPU of the root domain that keeps that bandwidth until the
+	 * inactive timer removes it (dl_task_depart()). dl_non_contending
+	 * tells that timer armed, its handler yet to run; hrtimer_active()
+	 * stays true for a while after that handler returns.
 	 */
-	if (p->dl.dl_non_contending)
+	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
+		dl_task_depart(p, dl_task_root_domain_cpu(p));
+	}
 
 	/*
 	 * Since this might be the only -deadline task on the rq,
@@ -3657,6 +3809,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 static void switched_to_dl(struct rq *rq, struct task_struct *p)
 {
 	cancel_inactive_timer(&p->dl);
+	/*
+	 * Boosted back into the class before its 0-lag time (a policy change
+	 * released that bandwidth in sched_dl_overflow()): the inactive
+	 * timer will not remove it, but its leaving the class again. That
+	 * forgets the CPU recorded: moved to another root domain meanwhile,
+	 * the task then removes the bandwidth from that one, as it did
+	 * before departures were recorded.
+	 */
+	if (p->dl.dl_departed_cpu >= 0)
+		dl_task_undepart(p);
 
 	/*
 	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -3771,12 +3933,6 @@ DEFINE_SCHED_CLASS(dl) = {
 #endif
 };
 
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
 int sched_dl_global_validate(void)
 {
 	u64 runtime = global_rt_runtime();
@@ -3897,8 +4053,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 
 	if (dl_policy(policy) && !task_has_dl_policy(p) &&
 	    !__dl_overflow(dl_b, cap, 0, new_bw)) {
-		if (hrtimer_active(&p->dl.inactive_timer))
-			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
@@ -3924,6 +4078,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	}
 	raw_spin_unlock(&dl_b->lock);
 
+	/*
+	 * Back before its 0-lag time, the task still has the bandwidth it
+	 * left with, which the new one replaces: removed from the root
+	 * domain that keeps it (dl_task_release_bw()). A rebuild cannot
+	 * run meanwhile: the caller holds cpuset_mutex.
+	 */
+	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
+		lockdep_assert_cpuset_lock_held();
+		if (p->dl.dl_departed_cpu >= 0)
+			dl_task_release_bw(p, -1);
+		p->dl.dl_bw_gen = dl_bw_gen;
+	}
+
 	return err;
 }
 
@@ -4059,6 +4226,7 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
 	__dl_clear_params(dl_se);
+	dl_se->dl_departed_cpu = -1;
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-- 
2.47.3

[PATCH v2 10/21] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild

[Mathieu Desnoyers]

Add deadline_rebuild_test, for the previous patch. In each case a task
of 400ms every second runs 380ms, then leaves SCHED_DEADLINE, its 0-lag
time some 570ms ahead; past that time, the one-CPU partition root P must
admit one task of 600ms every second and refuse a second with EBUSY:

 - dies_before_rebuild: the task, in P, exits, and a second partition
   root is made meanwhile, which rebuilds the accounting of every root
   domain;
 - switched_out_before_rebuild: the same, the task switched to
   SCHED_OTHER and asleep in P;
 - moved_after_switch_out: the task, outside P, switches to SCHED_OTHER,
   moves into P and dies there.

A kernel that removes the bandwidth from P's root domain when a rebuild
dropped it, or when P never had it, admits both. Skipped without root,
the cgroup v2 cpuset controller, deadline admission control (that is,
with sched_rt_runtime_us at -1) or 3 CPUs.

In 8 VMs of 8 vCPUs each, three runs in each, v7.4-rc1 with the four
deadline fixes of sched/urgent fails the three cases, and so does it
with the previous patches but the last; with it, the three pass; every
time. v7.3 fails them as well (2 VMs, three runs in each).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/sched/.gitignore      |   1 +
 tools/testing/selftests/sched/Makefile        |   2 +-
 .../selftests/sched/deadline_rebuild_test.c   | 462 ++++++++++++++++++
 3 files changed, 464 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline_rebuild_test.c

diff --git a/tools/testing/selftests/sched/.gitignore b/tools/testing/selftests/sched/.gitignore
index d97f40c21ab5..fc7ab1ddbbe7 100644
--- a/tools/testing/selftests/sched/.gitignore
+++ b/tools/testing/selftests/sched/.gitignore
@@ -1,2 +1,3 @@
 cs_prctl_test
 deadline_switch_test
+deadline_rebuild_test
diff --git a/tools/testing/selftests/sched/Makefile b/tools/testing/selftests/sched/Makefile
index ae9421eabf15..728e046b463c 100644
--- a/tools/testing/selftests/sched/Makefile
+++ b/tools/testing/selftests/sched/Makefile
@@ -10,6 +10,6 @@ LDLIBS += -lpthread
 
 TEST_GEN_FILES := cs_prctl_test
 TEST_PROGS := cs_prctl_test
-TEST_GEN_PROGS := deadline_switch_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
 
 include ../lib.mk
diff --git a/tools/testing/selftests/sched/deadline_rebuild_test.c b/tools/testing/selftests/sched/deadline_rebuild_test.c
new file mode 100644
index 000000000000..d1f948f5de77
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline_rebuild_test.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The bandwidth of a SCHED_DEADLINE task that leaves the policy, or dies,
+ * before its 0-lag time stays allocated in its root domain until the
+ * inactive timer removes it. It must be removed once, from the root
+ * domain that keeps it:
+ *
+ *  - dies: a task of a one-CPU partition root P dies, and a second
+ *    partition root Q is made meanwhile, which rebuilds the accounting
+ *    of every root domain;
+ *  - switched out: the same, the task switched to SCHED_OTHER;
+ *  - moved: a task outside P switched to SCHED_OTHER, then moved into P
+ *    before its 0-lag time.
+ *
+ * Then P, past the timer, must admit one task of 600ms every second and
+ * refuse a second one (EBUSY). A kernel that removes the bandwidth from
+ * P's root domain when the rebuild dropped it, or when P never held it,
+ * leaves it negative: it admits both.
+ *
+ * Needs root, the cgroup v2 cpuset controller, deadline admission
+ * control and 3 CPUs.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+static int set_deadline(pid_t pid, uint64_t runtime)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = NSEC_PER_SEC,
+		.sched_period = NSEC_PER_SEC,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+/* These clocks cannot fail: both are always supported, the pointer is valid. */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	set_other(pid);			/* gone already: nothing to switch */
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* A partition root of the one CPU @cpu, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+	char path[PATH_MAX], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	if (write_file(cg, "cpuset.cpus", buf) || write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+	uint64_t end;
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	end = now_ns() + NSEC_PER_SEC;
+	while (rmdir(cg)) {
+		if (errno == ENOENT)
+			return;
+		if (errno != EBUSY || now_ns() > end) {
+			ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+			return;
+		}
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+}
+
+/* Both clocks are always supported and the pointer is valid: no error. */
+static uint64_t cpu_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+enum depart { DIES, SWITCHED_OUT };
+
+/*
+ * In @cg if given, made SCHED_DEADLINE at 400ms every second, the child
+ * runs 380ms of it, then dies or switches to SCHED_OTHER and sleeps:
+ * its 0-lag time is then some 570ms ahead. The pipe @fd gets a byte once
+ * it is out of the policy; for one that dies, its exit status tells. The
+ * caller keeps the read end alone.
+ */
+static pid_t departing(const char *cg, enum depart how, int fd[2])
+{
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		uint64_t t0;
+
+		close(fd[0]);
+		if (cg && enter(cg, getpid()))
+			_exit(2);
+		if (set_deadline(0, 400 * NSEC_PER_MSEC))
+			_exit(3);
+		t0 = cpu_time_ns();
+		while (cpu_time_ns() - t0 < 380 * NSEC_PER_MSEC)
+			;
+		if (how == DIES)
+			_exit(0);
+		if (set_other(0) || write(fd[1], "x", 1) != 1)
+			_exit(4);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	fd[1] = -1;
+	return pid;
+}
+
+/* Wait until the child is out of SCHED_DEADLINE, dead or switched out. */
+static int departed(pid_t *pid, enum depart how, int fd)
+{
+	int status;
+	char c;
+
+	if (*pid < 0)
+		return -1;
+	/* No byte: the child failed, and its exit status tells. */
+	if (how == SWITCHED_OUT) {
+		ssize_t n;
+
+		do {
+			n = read(fd, &c, 1);
+		} while (n < 0 && errno == EINTR);
+		if (n == 1)
+			return 0;
+	}
+	if (waitpid(*pid, &status, 0) != *pid)
+		return -1;
+	*pid = -1;
+	return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
+}
+
+/*
+ * A child that enters @cg and sleeps there, or -1. It moves itself, so
+ * that it last ran on a CPU of @cg, and says so once it is there.
+ */
+static pid_t sleeper(const char *cg)
+{
+	int fd[2];
+	pid_t pid;
+	char c;
+
+	if (pipe(fd))
+		return -1;
+	pid = fork();
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		close(fd[0]);
+		if (enter(cg, getpid()) || write(fd[1], "x", 1) != 1)
+			_exit(1);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	if (pid > 0 && read(fd[0], &c, 1) != 1) {
+		reap(pid);
+		pid = -1;
+	}
+	close(fd[0]);
+	return pid;
+}
+
+/* P, past every timer, admits one task of 600ms every second, not two. */
+static int check_p(void)
+{
+	pid_t a = sleeper(cg_p), b = sleeper(cg_p);
+	int ret = KSFT_FAIL, e1, e2;
+
+	if (a < 0 || b < 0) {
+		ksft_print_msg("no task could enter P\n");
+		goto out;
+	}
+	e1 = set_deadline(a, 600 * NSEC_PER_MSEC);
+	e2 = set_deadline(b, 600 * NSEC_PER_MSEC);
+	if (e1)
+		ksft_print_msg("a first task refused in P: %s\n", strerror(-e1));
+	else if (!e2)
+		ksft_print_msg("1.2 CPUs of deadline tasks admitted in a partition of one CPU\n");
+	else if (e2 != -EBUSY)
+		ksft_print_msg("the second task refused with %s, not EBUSY\n", strerror(-e2));
+	else
+		ret = KSFT_PASS;
+out:
+	reap(a);
+	reap(b);
+	return ret;
+}
+
+/* A task of P leaves, and Q is made before its 0-lag time. */
+static int test_rebuild(enum depart how)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(cg_p, how, fd);
+	err = departed(&pid, how, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (make_partition(cg_q, ncpus - 2)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 2);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	/* Past the 0-lag time; one switched out sleeps on in P. */
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_q);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+/* A task outside P switches out, then moves into P before its 0-lag time. */
+static int test_move(void)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(NULL, SWITCHED_OUT, fd);
+	err = departed(&pid, SWITCHED_OUT, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (enter(cg_p, pid)) {
+		ksft_print_msg("the task cannot move into P\n");
+		goto out;
+	}
+	/* It wakes up, and dies, on P's CPU, before its 0-lag time. */
+	reap(pid);
+	pid = -1;
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	char buf[256], path[PATH_MAX + 32];
+	int added;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	ncpus = get_nprocs();
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs\n");
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	if (find_cgroup2())
+		ksft_exit_skip("cgroup v2 is not mounted\n");
+	/* The cpuset controller for the root's children while the test runs. */
+	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+	if (read_file(path, buf, sizeof(buf)))
+		ksft_exit_skip("no cgroup.subtree_control\n");
+	added = !strstr(buf, "cpuset");
+	if (added && write_file(root, "cgroup.subtree_control", "+cpuset"))
+		ksft_exit_skip("no cpuset controller\n");
+	if (snprintf(cg_p, sizeof(cg_p), "%s/deadline_rebuild_p", root) >= (int)sizeof(cg_p) ||
+	    snprintf(cg_q, sizeof(cg_q), "%s/deadline_rebuild_q", root) >= (int)sizeof(cg_q))
+		ksft_exit_skip("cgroup v2 mount path too long\n");
+
+	ksft_set_plan(3);
+	report("dies_before_rebuild", test_rebuild(DIES));
+	report("switched_out_before_rebuild", test_rebuild(SWITCHED_OUT));
+	report("moved_after_switch_out", test_move());
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+	ksft_finished();
+}
-- 
2.47.3

[PATCH v2 11/21] sched/deadline: Do not take a waking task's bandwidth off a runqueue it left

[Mathieu Desnoyers]

A deadline task that wakes on another CPU has its bandwidth taken off
the runqueue it slept on by migrate_task_rq_dl(), from set_task_cpu(),
and charged to the one it wakes on by the enqueue that follows. That
enqueue can be left to the target CPU through the wake list, and
try_to_wake_up() returns before it runs: the task's task_cpu() is then
the CPU it wakes on while its bandwidth is on no runqueue at all.

Whatever happens to the task in that window takes its bandwidth from a
runqueue that does not have it, or gives it to one that already does:

 - sched_setattr() finds it off the runqueues, and
   dl_change_utilization() takes its old bandwidth off the runqueue of
   task_cpu(), which never held it. this_bw of that CPU underflows,
   which costs every other task and the fair server there their own, for
   good. switched_from_dl() leaves the class through the same hole;
 - a switch out of SCHED_DEADLINE and back in that window has
   switched_to_dl() charge the runqueue of task_cpu(), and the enqueue
   that lands later charges it a second time, since task_contending()
   goes by ENQUEUE_MIGRATED. this_bw of that CPU then counts the task
   twice, and the release of one of the two underflows it later.

Say in the entity whether a runqueue counts its bandwidth, as
dl_bw_oncpu, and have add_rq_bw() and sub_rq_bw() put it there and take
it away once each: an add on a runqueue that counts it already, and a
subtraction from one that does not, then change nothing, and
dl_change_utilization() has a runqueue to act on only while one counts
it. The servers keep their own tally, dl_bw_attached, and go through
__add_rq_bw() and __sub_rq_bw() directly.

Reaching the first needs a deadline task whose wakeup migrates, which
asks for more than one CPU in its root domain and for them to be taken.
It is easier to reach with CONFIG_SCHED_PROXY_EXEC=y, where a task
blocked on a mutex is migrated to the CPU of its owner and wakes back on
its own.

In 8 VMs of 8 vCPUs, a task of 10ms every 100ms that sleeps and wakes in
a partition root of two CPUs, both taken by deadline tasks of an earlier
deadline, its runtime changed between 10ms and 20ms from a CPU outside,
warns in all 8 within a second, and the warnings spread to the idle
task's accounting of both CPUs:

  WARNING: kernel/sched/deadline.c:352 at dl_rq_change_utilization+0x71/0x2e0, CPU#0: wake_race/109
   __sub_rq_bw(): this_bw underflow
  WARNING: kernel/sched/deadline.c:316 at task_contending+0x244/0x260, CPU#6: swapper/6/0
   __add_running_bw(): running_bw overflow

The 8 VMs above warn no more with this change. deadline_bw_test, added
later in this series, has params_waking for it: without this change it
fails on every one of those VMs, and random_mix on one of three.

The double charge of the second case was found by the Sashiko AI review
of the first posting of this series, which also pointed out that the
first version of this patch left its bit set for an entity with no
bandwidth of its own; a bit that says where the bandwidth is, rather
than where it is not, has neither fault.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  1 +
 kernel/sched/deadline.c | 35 +++++++++++++++++++++++++++++++----
 2 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 0633a73579db..6803df5ab13c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -718,6 +718,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer_running  : 1;
 	unsigned int			dl_defer_idle     : 1;
 	unsigned int			dl_bw_attached    : 1;
+	unsigned int			dl_bw_oncpu       : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 2729c108b24b..cbcc00ea46d3 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -355,18 +355,31 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
 	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
 }
 
+/*
+ * @dl_bw_oncpu tells whether this_bw of a runqueue counts @dl_se's
+ * bandwidth, so that these two put it there and take it away once each:
+ * a wakeup takes it off the runqueue the task slept on and the enqueue on
+ * the one it wakes on puts it back, and what happens to the task in
+ * between must not count it twice, nor take it from a runqueue that does
+ * not have it. The servers keep their own tally, dl_bw_attached, and go
+ * through __add_rq_bw() and __sub_rq_bw() directly.
+ */
 static inline
 void add_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__add_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || dl_se->dl_bw_oncpu)
+		return;
+	__add_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 1;
 }
 
 static inline
 void sub_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || !dl_se->dl_bw_oncpu)
+		return;
+	__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 0;
 }
 
 static inline
@@ -435,6 +448,14 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 	if (task_on_rq_queued(p))
 		return;
 
+	/*
+	 * Waking up on another CPU, the task has its bandwidth on no
+	 * runqueue: the enqueue to come charges the new one, with the new
+	 * parameters, which __setparam_dl() sets right after this.
+	 */
+	if (!p->dl.dl_bw_oncpu)
+		return;
+
 	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
 }
 
@@ -2790,6 +2811,11 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
 		 */
 		cancel_inactive_timer(&p->dl);
 	}
+	/*
+	 * Until the enqueue on the CPU it wakes on charges it there, this
+	 * bandwidth is on no runqueue: the rq of task_cpu() is not the one
+	 * it was taken from, which dl_bw_oncpu now says.
+	 */
 	sub_rq_bw(&p->dl, &rq->dl);
 	rq_unlock(rq, &rf);
 }
@@ -4214,6 +4240,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer			= 0;
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
+	dl_se->dl_bw_oncpu		= 0;
 
 #ifdef CONFIG_RT_MUTEXES
 	dl_se->pi_se			= dl_se;
-- 
2.47.3

[PATCH v2 12/21] sched/debug: Show the deadline bandwidth of each runqueue

[Mathieu Desnoyers]

sched_debug prints the bandwidth each root domain has handed out
(dl_bw->total_bw), and nothing of the two sums each runqueue keeps:
this_bw, the bandwidth of every deadline task of that CPU, and
running_bw, the part of it whose tasks are active. GRUB drains the
runtime of a reclaiming task at the rate the two give, the bandwidth of
its CPU that is reserved and active (grub_reclaim(): max_bw - (this_bw -
running_bw) - extra_bw): the less of its CPU is active, the slower its
runtime drains, and the longer it runs on.

Both sums have invariants a test can check, which the kernel itself
warns about when they break: neither may wrap, and running_bw never
exceeds this_bw. Print them.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/debug.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..8a726817baa0 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1166,6 +1166,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 	SEQ_printf(m, "  .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
 
 	PU(dl_nr_running);
+	PU(running_bw);
+	PU(this_bw);
 	dl_bw = &cpu_rq(cpu)->rd->dl_bw;
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
-- 
2.47.3

[PATCH v2 13/21] sched/deadline: Rebuild the root domains' accounting without cpusets too

[Mathieu Desnoyers]

A change of the root domains, which CPU hotplug and suspend and resume
make, clears their deadline bandwidth accounting and fills it back: the
servers of each CPU, and the bandwidth of the deadline tasks of each
cpuset. The walk that puts the tasks back lives in the cpuset code,
dl_rebuild_rd_accounting(), and is a stub without CONFIG_CPUSETS: such
a kernel loses the bandwidth of every deadline task at the first
hotplug, total_bw of the new root domain holding its servers alone.

The tasks are never told. Each still takes its bandwidth out of
total_bw once it leaves SCHED_DEADLINE, or dies: it underflows, and
admission afterwards refuses every deadline task of the root domain.

Walk the tasks instead of the cpusets, which is all there is without
them, and keep one declaration beside dl_clear_root_domains() for both.

In 8-CPU VMs built without CONFIG_CPUSETS, the hotplug_sleepers case of
the selftest added later in this series admits 9 tasks of 10ms every
100ms before taking a CPU offline and back, and 0 after, on v7.4-rc1 as
with the patches before this one; hotplug_departed and
hotplug_server_refused fail after it, and two more hotplug cases skip
for want of room. The five pass with this change, in 4 VMs, with no
warning.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/cpuset.h         |  5 ----
 include/linux/sched/deadline.h | 16 ++++++++++++
 kernel/sched/deadline.c        | 48 ++++++++++++++++++++++++++++++++++
 kernel/sched/syscalls.c        | 13 ++++++---
 4 files changed, 74 insertions(+), 8 deletions(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..608c3c5a9535 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -129,7 +129,6 @@ static inline int cpuset_do_page_mem_spread(void)
 
 extern bool current_cpuset_is_being_rebound(void);
 
-extern void dl_rebuild_rd_accounting(void);
 extern void rebuild_sched_domains(void);
 
 extern void cpuset_print_current_mems_allowed(void);
@@ -273,10 +272,6 @@ static inline bool current_cpuset_is_being_rebound(void)
 	return false;
 }
 
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
-
 static inline void rebuild_sched_domains(void)
 {
 	guard(cpus_read_lock)();
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdf39ac1d43d..d69c461a3eda 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,22 @@ struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
 extern void dl_clear_root_domains(void);
+extern void dl_rebuild_rd_accounting(void);
+/*
+ * The lock that keeps a rebuild of the root domains' accounting from running
+ * against an admission of bandwidth: cpuset_mutex where cpusets are built in,
+ * since every rebuild comes from there, and a lock of the deadline class's own
+ * where they are not.
+ */
+#ifdef CONFIG_CPUSETS
+static inline void dl_accounting_lock(void) { }
+static inline void dl_accounting_unlock(void) { }
+static inline void lockdep_assert_dl_accounting_held(void) { }
+#else
+extern void dl_accounting_lock(void);
+extern void dl_accounting_unlock(void);
+extern void lockdep_assert_dl_accounting_held(void);
+#endif
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index cbcc00ea46d3..85df2b0044be 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3685,6 +3685,53 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
+#ifndef CONFIG_CPUSETS
+/*
+ * Where cpusets are built in, cpuset_mutex keeps an admission of bandwidth
+ * from running against a rebuild of the accounting, and
+ * __sched_setscheduler() takes it for every change of or into
+ * SCHED_DEADLINE. Where they are not, cpuset_lock() is a stub and this
+ * takes its place.
+ */
+static DEFINE_MUTEX(dl_accounting_mutex);
+
+void dl_accounting_lock(void)
+{
+	mutex_lock(&dl_accounting_mutex);
+}
+
+void dl_accounting_unlock(void)
+{
+	mutex_unlock(&dl_accounting_mutex);
+}
+
+void lockdep_assert_dl_accounting_held(void)
+{
+	lockdep_assert_held(&dl_accounting_mutex);
+}
+
+/*
+ * Rebuild the root domains' accounting, where no cpuset holds the
+ * deadline tasks of each: they are found by walking them all, which
+ * dl_rebuild_rd_accounting() of the cpusets does per cpuset instead.
+ */
+void dl_rebuild_rd_accounting(void)
+{
+	struct task_struct *g, *p;
+
+	lockdep_assert_cpus_held();
+	lockdep_assert_held(&sched_domains_mutex);
+
+	guard(mutex)(&dl_accounting_mutex);
+	dl_clear_root_domains();
+
+	read_lock(&tasklist_lock);
+	for_each_process_thread(g, p)
+		dl_add_task_root_domain(p);
+	read_unlock(&tasklist_lock);
+}
+#endif /* !CONFIG_CPUSETS */
+
 /*
  * Clear every root domain ahead of a rebuild of their accounting, which
  * dl_add_task_root_domain() then fills with the deadline tasks of the
@@ -4112,6 +4159,7 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 */
 	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
 		lockdep_assert_cpuset_lock_held();
+		lockdep_assert_dl_accounting_held();
 		if (p->dl.dl_departed_cpu >= 0)
 			dl_task_release_bw(p, -1);
 		p->dl.dl_bw_gen = dl_bw_gen;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 8eddb9d47d8e..b1fc4fadd689 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -559,6 +559,7 @@ int __sched_setscheduler(struct task_struct *p,
 	if (dl_policy(policy) || dl_policy(p->policy)) {
 		cpuset_locked = true;
 		cpuset_lock();
+		dl_accounting_lock();
 	}
 
 	/*
@@ -650,8 +651,10 @@ int __sched_setscheduler(struct task_struct *p,
 	if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
 		policy = oldpolicy = -1;
 		task_rq_unlock(rq, p, &rf);
-		if (cpuset_locked)
+		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
+		}
 		goto recheck;
 	}
 
@@ -740,8 +743,10 @@ int __sched_setscheduler(struct task_struct *p,
 	task_rq_unlock(rq, p, &rf);
 
 	if (pi) {
-		if (cpuset_locked)
+		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
+		}
 		rt_mutex_adjust_pi(p);
 	}
 
@@ -753,8 +758,10 @@ int __sched_setscheduler(struct task_struct *p,
 
 unlock:
 	task_rq_unlock(rq, p, &rf);
-	if (cpuset_locked)
+	if (cpuset_locked) {
+		dl_accounting_unlock();
 		cpuset_unlock();
+	}
 	return retval;
 }
 
-- 
2.47.3

[PATCH v2 14/21] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

[Mathieu Desnoyers]

Add deadline_bw_test, 44 cases that take deadline tasks through priority
inheritance, parameter changes, cpuset moves, deaths, sched_yield(),
runtime overruns, bandwidth reclaiming, changes to the fair servers, a
randomized mix of all of those, and CPU hotplug.

Each case measures the free bandwidth of a root domain before and after
it runs, as the number of tasks of 10ms every 100ms the root domain
admits, and wants the two equal once the case's tasks are gone and their
0-lag time is past: fewer is bandwidth leaked, more is bandwidth
released twice. A deadline task that a case keeps running must also make
progress.

The reclaiming and server cases measure instead the share of a CPU a
deadline task with SCHED_FLAG_RECLAIM gets, which tells of the bandwidth
its CPU keeps as reclaimable (extra_bw) where the admission of probe()
tells of the root domain's own (total_bw): of a CPU of its own, a task
of 10ms every 100ms got 66% of it, where one without the flag got 9%,
and it follows the fair server of that CPU, 49% with a server of 10% and
24% with one of 30%, as GRUB has it.

params_waking and random_mix check the two sums each runqueue keeps,
this_bw and running_bw, which "sched/debug: Show the deadline bandwidth
of each runqueue" prints: parameter changes of a deadline task whose
wakeups move it between the two CPUs of a partition root, and six tasks
taking random transitions of their own while the case moves, kills and
reparameterizes them and changes the fair servers, three thousand
transitions in three seconds. Both want this_bw of each CPU back where
it started. random_mix's seed is fixed, so a failure can be replayed.

The cases in the partition root each get a new one, the accounting
rebuilt in between, so that a case that breaks it does not fail the next
ones. The server cases put the fair servers back as they found them.
Skipped without root or deadline admission control, and the cases that
need them without a cpuset partition root, debugfs, CPU hotplug or
enough CPUs.

Kprobes confirmed the states the cases mean to reach: the inactive timer
pending at a kill, the replenishment, the boost, and the fair server
stopped, deferred and serving at the debugfs writes. A write cannot find
a server in two of its states: throttled after serving, which the last
of the served task's runtime, charged as background running, turns at
once into deferred anew; and its idle wait, which on a CPU with no tick
in idle begins and ends in one timer callback.

In 8 VMs of 8 vCPUs, v7.4-rc1 with the four deadline fixes of
sched/urgent fails four cases, every time:

 - pi_former_owner_asleep: the waiter of its second round is refused
   with EBUSY in a partition with room, total_bw having wrapped when the
   deboost of the first round released the owner's bandwidth a second
   time;
 - params_keep_class_change: the refused request leaves 40% of the
   partition reserved;
 - params_keep_boosted: the same request on an owner that priority
   inheritance boosted into the class is not refused at all, and leaves
   the owner's slice reserved;
 - random_mix: the partition root admits 5, 8 or 10 tasks of 10% where
   it admitted 9 before, bandwidth leaked and bandwidth released twice
   both.

params_waking fails on every VM of 8 vCPUs and of 4 without
"sched/deadline: Do not take a waking task's bandwidth off a runqueue it
left": this_bw of one of the two CPUs ends elsewhere than it started,
having lost or gained the bandwidth of a task or of a fair server, and
the kernel warns of the underflow that got it there. random_mix catches
that one on 1 VM of 3, a race being what it is.

With this series, all 44 cases pass every time, with PREEMPT_RT as well,
and with no warning. Built without cpusets they need the previous patch:
without it hotplug_sleepers fails, and the hotplug cases after it fail
or skip for want of room, every time, the root domain that a CPU hotplug
makes getting back the bandwidth of its servers alone.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/sched/.gitignore      |    1 +
 tools/testing/selftests/sched/Makefile        |    2 +-
 .../selftests/sched/deadline_bw_test.c        | 3353 +++++++++++++++++
 tools/testing/selftests/sched/settings        |    1 +
 4 files changed, 3356 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/settings

diff --git a/tools/testing/selftests/sched/.gitignore b/tools/testing/selftests/sched/.gitignore
index fc7ab1ddbbe7..27018878c063 100644
--- a/tools/testing/selftests/sched/.gitignore
+++ b/tools/testing/selftests/sched/.gitignore
@@ -1,3 +1,4 @@
 cs_prctl_test
 deadline_switch_test
 deadline_rebuild_test
+deadline_bw_test
diff --git a/tools/testing/selftests/sched/Makefile b/tools/testing/selftests/sched/Makefile
index 728e046b463c..a32a0f7aa76b 100644
--- a/tools/testing/selftests/sched/Makefile
+++ b/tools/testing/selftests/sched/Makefile
@@ -10,6 +10,6 @@ LDLIBS += -lpthread
 
 TEST_GEN_FILES := cs_prctl_test
 TEST_PROGS := cs_prctl_test
-TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test deadline_bw_test
 
 include ../lib.mk
diff --git a/tools/testing/selftests/sched/deadline_bw_test.c b/tools/testing/selftests/sched/deadline_bw_test.c
new file mode 100644
index 000000000000..98f66b770143
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline_bw_test.c
@@ -0,0 +1,3353 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * SCHED_DEADLINE bandwidth through priority inheritance, parameter changes,
+ * cpuset moves, deaths, yields, runtime overruns, bandwidth reclaiming,
+ * changes to the fair servers and CPU hotplug.
+ *
+ * Each case measures the free bandwidth of a root domain before and after
+ * it runs: the number of tasks of 10ms every 100ms that root domain admits
+ * (probe()). Once every task of the case is gone and its 0-lag time is
+ * past, the two must be equal: fewer is bandwidth leaked, more is
+ * bandwidth released twice, which lets the root domain admit deadline
+ * tasks beyond its capacity. A deadline task a case keeps running must
+ * also make progress.
+ *
+ * The reclaiming cases measure instead the share of a CPU a deadline task
+ * with SCHED_FLAG_RECLAIM gets, which tells of the free bandwidth its CPU
+ * keeps of the root domain (extra_bw), where probe() tells of the root
+ * domain's own (total_bw).
+ *
+ * Every case but the hotplug ones uses P, a partition root of one CPU,
+ * the last one, made anew for each case: the rebuild of the accounting
+ * that makes it clears what an earlier case left. The hotplug cases run in
+ * the root domain of all CPUs. The server cases change the fair server of a
+ * CPU through debugfs, and the test puts it back as it found it. Needs
+ * root, deadline admission control, and for the cases that use them, the
+ * cgroup v2 cpuset controller, debugfs and CPU hotplug.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+#include <linux/futex.h>
+
+#include "../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+#ifndef SCHED_FLAG_RECLAIM
+#define SCHED_FLAG_RECLAIM	0x02
+#endif
+#ifndef SCHED_FLAG_DL_OVERRUN
+#define SCHED_FLAG_DL_OVERRUN	0x04
+#endif
+#ifndef SCHED_FLAG_KEEP_PARAMS
+#define SCHED_FLAG_KEEP_PARAMS	0x10
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+/* The threads of probe() and hold() do nothing: a small stack will do. */
+#define THREAD_STACK	(64 * 1024 > PTHREAD_STACK_MIN ? 64 * 1024 : PTHREAD_STACK_MIN)
+
+#define PROBE_RUNTIME	(10 * NSEC_PER_MSEC)
+#define PROBE_PERIOD	(100 * NSEC_PER_MSEC)
+#define PERIOD		(100 * NSEC_PER_MSEC)
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+/* Both clocks are always supported and the pointer is valid: no error. */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static uint64_t thread_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void spin_ns(uint64_t ns)
+{
+	uint64_t end = thread_time_ns() + ns;
+
+	while (thread_time_ns() < end)
+		;
+}
+
+/*
+ * Polls @cond every @step, at most @ms: 0 once it holds, -1 on a timeout.
+ * Every wait here is bounded, so that a child that never started, or was
+ * refused SCHED_DEADLINE, fails its case instead of hanging it.
+ */
+#define wait_for_step(ms, step, cond)					\
+({									\
+	uint64_t wait_end = now_ns() + (uint64_t)(ms) * NSEC_PER_MSEC;	\
+	int wait_ret = -1;						\
+									\
+	while (1) {							\
+		if (cond) {						\
+			wait_ret = 0;					\
+			break;						\
+		}							\
+		if (now_ns() >= wait_end)				\
+			break;						\
+		sleep_ns(step);						\
+	}								\
+	wait_ret;							\
+})
+
+#define wait_for(ms, cond)	wait_for_step(ms, NSEC_PER_MSEC, cond)
+
+/* sem_wait(), retrying the signals that interrupt it. */
+static void sem_wait_sig(sem_t *s)
+{
+	while (sem_wait(s)) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("sem_wait: %s\n", strerror(errno));
+		return;
+	}
+}
+
+/* The same, at most @ms: 0, or -1. For what a thread or a child must post. */
+static int sem_wait_for(sem_t *s, int ms)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+	ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	while (sem_timedwait(s, &ts)) {
+		if (errno == EINTR)
+			continue;
+		if (errno != ETIMEDOUT)
+			ksft_print_msg("sem_timedwait: %s\n", strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+/* fork(), which tells of its own failure: the caller handles -1. */
+static pid_t fork_child(const char *what)
+{
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork for %s: %s\n", what, strerror(errno));
+	return pid;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static void join(pthread_t t)
+{
+	int err = pthread_join(t, NULL);
+
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+}
+
+static pid_t gettid_(void)
+{
+	return syscall(SYS_gettid);
+}
+
+static int set_deadline_flags(pid_t tid, uint64_t runtime, uint64_t period,
+			      uint64_t flags)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = flags,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int set_deadline(pid_t tid, uint64_t runtime, uint64_t period)
+{
+	return set_deadline_flags(tid, runtime, period, 0);
+}
+
+static int set_other(pid_t tid)
+{
+	struct dl_attr attr = { .size = sizeof(attr), .sched_policy = SCHED_OTHER };
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* @cg, a partition root of the CPUs @cpus lists: 0, or -1. */
+static int make_partition_cpus(const char *cg, const char *cpus)
+{
+	char path[PATH_MAX + 32], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	if (write_file(cg, "cpuset.cpus", cpus) ||
+	    write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+/* @cg, a partition root of CPU @cpu: 0, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+	char buf[32];
+
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	return make_partition_cpus(cg, buf);
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	if (wait_for(1000, !rmdir(cg) || errno == ENOENT))
+		ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+}
+
+/* P, a partition root of the last CPU: 0, or -1. */
+static int make_p(void)
+{
+	return make_partition(cg_p, ncpus - 1);
+}
+
+static void remove_p(void)
+{
+	remove_partition(cg_p);
+}
+
+/* The CPU @pid last ran on, from /proc. */
+static int task_cpu_of(pid_t pid)
+{
+	char path[64], buf[1024], *p;
+	int i;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+	if (read_file(path, buf, sizeof(buf)))
+		return -1;
+	p = strrchr(buf, ')');
+	for (i = 0; p && i < 37; i++)
+		p = strchr(p + 1, ' ');
+	return p ? atoi(p + 1) : -1;
+}
+
+/* The CPU time @tid has used. */
+static uint64_t cpu_time_of(pthread_t t)
+{
+	static int told;
+	struct timespec ts;
+	clockid_t c;
+
+	if (pthread_getcpuclockid(t, &c) || clock_gettime(c, &ts)) {
+		if (!told++)
+			ksft_print_msg("the CPU time of a thread cannot be read: %s\n",
+				       strerror(errno));
+		return 0;
+	}
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/* The share of a CPU, in percent, that @t uses over @ns, a period later. */
+static int share_of(pthread_t t, uint64_t ns)
+{
+	uint64_t cpu, wall;
+
+	sleep_ns(PERIOD);		/* a period of its own to settle in */
+	cpu = cpu_time_of(t);
+	wall = now_ns();
+	sleep_ns(ns);
+	cpu = cpu_time_of(t) - cpu;
+	wall = now_ns() - wall;
+	return wall ? (int)(100 * cpu / wall) : -1;
+}
+
+/*
+ * The probe: from a process that joins @cg (the root domain of its CPUs),
+ * threads each make themselves a deadline task of PROBE_RUNTIME every
+ * PROBE_PERIOD, one at a time, until one is refused; then they all exit.
+ * Returns how many were admitted, or -1. The threads never run past their
+ * first instants, so their bandwidth goes at their death; probe() waits for
+ * their 0-lag times all the same.
+ */
+struct probe_slot {
+	pthread_t t;
+	int err;
+};
+
+static sem_t probe_done;
+static int probe_hold[2];
+
+static void *probe_thread(void *arg)
+{
+	struct probe_slot *s = arg;
+	char c;
+
+	s->err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+	if (sem_post(&probe_done))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	/* Asleep until the probe is over: the write end closes. */
+	while (!s->err) {
+		ssize_t r = read(probe_hold[0], &c, 1);
+
+		if (r >= 0)
+			break;
+		if (errno != EINTR) {
+			s->err = -errno;
+			break;
+		}
+	}
+	return NULL;
+}
+
+static int probe(const char *cg)
+{
+	int fd[2], n = -1, max = 16 * ncpus + 16;
+	pid_t pid;
+
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		return -1;
+	}
+	pid = fork_child("the admission check");
+	if (!pid) {
+		struct probe_slot *s = calloc(max, sizeof(*s));
+		pthread_attr_t a;
+		int i, k = 0;
+
+		close(fd[0]);
+		if (!s || sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+		    (cg && enter(cg, getpid())))
+			_exit(1);
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			_exit(1);
+		for (i = 0; i < max; i++) {
+			if (pthread_create(&s[i].t, &a, probe_thread, &s[i])) {
+				k = -1;	/* not an admission refused */
+				break;
+			}
+			if (sem_wait_for(&probe_done, 10000)) {
+				k = -1;
+				i++;	/* that thread is to be joined as well */
+				break;
+			}
+			if (s[i].err) {
+				if (s[i].err != -EBUSY)
+					k = -1;
+				join(s[i].t);
+				break;
+			}
+			if (k >= 0)
+				k++;
+		}
+		if (i == max)
+			k = -1;
+		close(probe_hold[1]);
+		while (--i >= 0)
+			join(s[i].t);
+		if (write(fd[1], &k, sizeof(k)) != sizeof(k))
+			_exit(1);
+		_exit(0);
+	}
+	close(fd[1]);
+	if (pid < 0 || read(fd[0], &n, sizeof(n)) != sizeof(n))
+		n = -1;
+	close(fd[0]);
+	reap(pid);
+	sleep_ns(3 * PROBE_PERIOD);
+	return n;
+}
+
+/*
+ * A case runs in a child process, in @cg, so that all its threads go with
+ * it: @fn returns KSFT_PASS or KSFT_FAIL, or KSFT_SKIP.
+ */
+static int run_in(const char *cg, int (*fn)(void))
+{
+	pid_t pid = fork_child("a case");
+	int status;
+
+	if (!pid) {
+		if (cg && enter(cg, getpid()))
+			_exit(KSFT_SKIP);
+		_exit(fn());
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!WIFEXITED(status)) {
+		ksft_print_msg("the case died of signal %d\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return WEXITSTATUS(status);
+}
+
+/* @fn in @cg, with the free bandwidth of @cg's root domain the same after. */
+static int measured(const char *name, const char *cg, int (*fn)(void))
+{
+	int before, after, ret;
+
+	before = probe(cg);
+	if (before <= 0) {
+		ksft_print_msg("%s: no room to probe (%d)\n", name, before);
+		return KSFT_SKIP;
+	}
+	ret = run_in(cg, fn);
+	sleep_ns(3 * PERIOD);
+	after = probe(cg);
+	if (ret == KSFT_PASS && after != before) {
+		ksft_print_msg("%s: %d probe tasks admitted before, %d after: %s\n",
+			       name, before, after,
+			       after < before ? "bandwidth leaked" : "bandwidth released twice");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Priority inheritance */
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_ready, owner_go;
+static atomic_int owner_holds, owner_tid;
+
+/* 0, or -1 where a priority inheritance mutex is not to be had. */
+static int pi_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err)
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+	if (!err)
+		err = pthread_mutex_init(&pi_lock, &a);
+	if (err) {
+		ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	if (sem_init(&owner_ready, 0, 0) || sem_init(&owner_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	owner_holds = 0;
+	return 0;
+}
+
+/*
+ * The calling thread, a deadline task meanwhile, waits for the lock up to
+ * @ms: 0 when it times out, or gets the lock, as @timeout expects.
+ */
+static int wait_lock_for(int ms, int timeout)
+{
+	struct timespec ts;
+	int err;
+
+	err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return -1;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)ms * 1000000L;
+	ts.tv_sec += ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	err = pthread_mutex_timedlock(&pi_lock, &ts);
+	if (!err) {
+		int e = pthread_mutex_unlock(&pi_lock);
+
+		if (e)
+			ksft_print_msg("a waiter cannot unlock: %s\n", strerror(e));
+	}
+	if (set_other(0)) {
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+		return -1;
+	}
+	if (err != (timeout ? ETIMEDOUT : 0)) {
+		ksft_print_msg("a waiter of the lock: %s, not %s\n", err ? strerror(err) : "got it",
+			       timeout ? "a timeout" : "the lock");
+		return -1;
+	}
+	return 0;
+}
+
+/* The lock of an owner: -1 told of, and owner_holds says so. */
+static int take_pi_lock(void)
+{
+	int err = pthread_mutex_lock(&pi_lock);
+
+	if (err) {
+		ksft_print_msg("an owner cannot take the lock: %s\n", strerror(err));
+		owner_holds = -1;
+	}
+	return err ? -1 : 0;
+}
+
+static void drop_pi_lock(void)
+{
+	int err = pthread_mutex_unlock(&pi_lock);
+
+	if (err)
+		ksft_print_msg("an owner cannot unlock: %s\n", strerror(err));
+}
+
+/* Owner: was a deadline task (@was_dl), takes the lock and sleeps with it. */
+static void *sleeping_owner(void *arg)
+{
+	int was_dl = (long)arg;
+
+	owner_tid = gettid_();
+	if (was_dl) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			owner_holds = -1;
+			sem_post(&owner_ready);
+			return (void *)1;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		/* Asleep past its 0-lag time, switched out by the main thread. */
+		sem_post(&owner_ready);
+		sem_wait_sig(&owner_go);
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	sleep_ns(400 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_asleep(int was_dl, int rounds)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)(long)was_dl)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (was_dl) {
+		if (sem_wait_for(&owner_ready, 5000)) {
+			ksft_print_msg("the owner did not become a deadline task\n");
+			ret = KSFT_FAIL;
+		} else {
+			sleep_ns(200 * NSEC_PER_MSEC);
+			if (set_other(owner_tid)) {
+				ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+				ret = KSFT_FAIL;
+			}
+		}
+		sem_post(&owner_go);
+	}
+	if (ret == KSFT_PASS && (wait_for(5000, owner_holds) || owner_holds != 2)) {
+		ksft_print_msg("the owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS)
+		sleep_ns(20 * NSEC_PER_MSEC);
+	/* Each attempt boosts the sleeping owner, its timeout deboosts it asleep. */
+	for (r = 0; r < rounds && ret == KSFT_PASS; r++) {
+		if (wait_lock_for(20, 1)) {
+			ksft_print_msg("round %d of %d failed\n", r + 1, rounds);
+			ret = KSFT_FAIL;
+		}
+	}
+	join(o);
+	return ret;
+}
+
+static int pi_other_owner_asleep(void)
+{
+	return pi_asleep(0, 5);
+}
+
+static int pi_former_owner_asleep(void)
+{
+	return pi_asleep(1, 5);
+}
+
+/* A deadline owner boosted by a deadline waiter of an earlier deadline. */
+static void *dl_owner(void *arg)
+{
+	uint64_t end;
+
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		owner_holds = -1;
+		return (void *)1;
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	end = now_ns() + 300 * NSEC_PER_MSEC;
+	while (now_ns() < end) {
+		spin_ns(2 * NSEC_PER_MSEC);
+		sleep_ns(5 * NSEC_PER_MSEC);
+	}
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_deadline_owner(void)
+{
+	int r, ret = KSFT_PASS, err;
+	pthread_t o;
+	void *res;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, dl_owner, NULL)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the deadline owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++)
+		if (wait_lock_for(20, 1))
+			ret = KSFT_FAIL;
+	err = pthread_join(o, &res);
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+	return res || err ? KSFT_FAIL : ret;
+}
+
+/* The owner runs when boosted, and its unlock deboosts it. */
+static void *running_owner(void *arg)
+{
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	spin_ns(50 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_unlock_running(void)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (pi_init())
+			return KSFT_SKIP;
+		if (pthread_create(&o, NULL, running_owner, NULL)) {
+			ksft_print_msg("the owner thread could not start\n");
+			return KSFT_FAIL;
+		}
+		if (wait_for_step(5000, 100 * NSEC_PER_USEC, owner_holds) ||
+		    owner_holds != 2) {
+			ksft_print_msg("the owner never took the lock\n");
+			ret = KSFT_FAIL;
+		} else if (wait_lock_for(1000, 0)) {
+			ret = KSFT_FAIL;
+		}
+		join(o);
+	}
+	return ret;
+}
+
+/* Parameter changes */
+
+static atomic_int t_state;	/* 1: about to sleep, or spinning */
+static _Atomic pid_t t_tid;
+static sem_t t_exit;
+
+/* A deadline task that runs @run_ns of its runtime, then sleeps. */
+static void *runs_then_sleeps(void *arg)
+{
+	uint64_t run = (uint64_t)(long)arg;
+
+	t_tid = gettid_();
+	if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	sleep_ns(PERIOD);	/* a fresh period */
+	spin_ns(run);
+	t_state = 1;
+	sem_wait_sig(&t_exit);
+	return NULL;
+}
+
+/* Starts runs_then_sleeps(): 0 once it is about to sleep, or -1. */
+static int start_sleeper(pthread_t *t, uint64_t run)
+{
+	t_state = 0;
+	if (sem_init(&t_exit, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	if (pthread_create(t, NULL, runs_then_sleeps, (void *)(long)run)) {
+		ksft_print_msg("the sleeper thread could not start\n");
+		return -1;
+	}
+	if (wait_for_step(5000, 100 * NSEC_PER_USEC, t_state) || t_state != 1) {
+		ksft_print_msg("the sleeper was refused SCHED_DEADLINE\n");
+		join(*t);
+		return -1;
+	}
+	return 0;
+}
+
+static void stop_sleeper(pthread_t t)
+{
+	if (sem_post(&t_exit))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	join(t);
+}
+
+/* The root domain of @cg admits @want probe tasks now. */
+static int admits_in(const char *cg, int want, const char *what)
+{
+	int n = probe(cg);
+
+	if (n != want) {
+		ksft_print_msg("%s: %d probe tasks admitted, %d expected\n", what, n, want);
+		return -1;
+	}
+	return 0;
+}
+
+/* The root domain of the caller's cgroup admits @want probe tasks now. */
+static int admits(int want, const char *what)
+{
+	return admits_in(NULL, want, what);
+}
+
+static int params_asleep(void)
+{
+	static const uint64_t rt[] = { 40, 10, 30, 20 };
+	int base, i, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	/* Probed from here, in P: this process is P's. */
+	base = probe(NULL) + 2;		/* the task's 20% back */
+	for (i = 0; i < 4 && ret == KSFT_PASS; i++) {
+		char what[64];
+
+		if (set_deadline(t_tid, rt[i] * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		snprintf(what, sizeof(what), "asleep, changed to %llums every 100ms",
+			 (unsigned long long)rt[i]);
+		if (admits(base - (int)(rt[i] / 10), what))
+			ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+static int params_non_contending(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS;
+
+	/* 5ms of 20ms run: its 0-lag time some 20ms after it sleeps. */
+	if (start_sleeper(&t, 5 * NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(NSEC_PER_MSEC);
+	if (set_deadline(t_tid, 40 * NSEC_PER_MSEC, PERIOD) ||
+	    set_deadline(t_tid, 10 * NSEC_PER_MSEC, PERIOD))
+		ret = KSFT_FAIL;
+	stop_sleeper(t);
+	return ret;
+}
+
+static atomic_int spin_stop;
+static uint64_t spin_runtime, spin_flags;	/* set before the thread starts */
+
+static void *dl_spinner(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, spin_runtime, PERIOD, spin_flags)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop)
+		;
+	return NULL;
+}
+
+/* Starts dl_spinner() of @runtime every 100ms with @flags: 0, or -1. */
+static int start_spinner(pthread_t *t, uint64_t runtime, uint64_t flags)
+{
+	t_state = 0;
+	spin_stop = 0;
+	spin_runtime = runtime;
+	spin_flags = flags;
+	if (pthread_create(t, NULL, dl_spinner, NULL)) {
+		ksft_print_msg("the spinner thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state == 1)
+		return 0;
+	join(*t);
+	return -1;
+}
+
+static void stop_spinner(pthread_t t)
+{
+	spin_stop = 1;
+	set_other(t_tid);	/* out of the class already: no matter */
+	join(t);
+}
+
+/* Waits until @t stops using CPU time: throttled. 0, or -1 after 1s. */
+static int wait_throttled(pthread_t t)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC, last = cpu_time_of(t);
+
+	while (now_ns() < end) {
+		uint64_t cur;
+
+		sleep_ns(3 * NSEC_PER_MSEC);
+		cur = cpu_time_of(t);
+		if (cur == last)
+			return 0;
+		last = cur;
+	}
+	return -1;
+}
+
+/* @t runs again within @ns. */
+static int progresses(pthread_t t, uint64_t ns)
+{
+	uint64_t start = cpu_time_of(t), end = now_ns() + ns;
+
+	while (now_ns() < end) {
+		if (cpu_time_of(t) > start + NSEC_PER_MSEC)
+			return 0;
+		sleep_ns(2 * NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+static int params_throttled(void)
+{
+	int r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0))
+		return KSFT_FAIL;
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (wait_throttled(t) ||
+		    set_deadline(t_tid, (r & 1 ? 10 : 20) * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (progresses(t, 3 * PERIOD)) {
+			ksft_print_msg("throttled, parameters changed: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	return ret;
+}
+
+static int params_refused(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS, base, err;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	base = probe(NULL);
+	/*
+	 * P's free bandwidth is at least @base probe tasks of 10%, and less
+	 * than one more: growing the task by one more than that is refused,
+	 * whatever the admission limit.
+	 */
+	if (20 + 10 * (base + 1) > 100) {
+		stop_sleeper(t);
+		ksft_print_msg("P has more room than one of its tasks can take\n");
+		return KSFT_SKIP;
+	}
+	err = set_deadline(t_tid, (20 + 10 * (uint64_t)(base + 1)) * NSEC_PER_MSEC, PERIOD);
+	if (err != -EBUSY) {
+		ksft_print_msg("an increase beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	} else if (admits(base, "after a refused increase")) {
+		ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+/*
+ * SCHED_FLAG_KEEP_PARAMS with a change of class is refused, with nothing
+ * reserved for the deadline parameters it would keep: the task's own,
+ * its slice the runtime.
+ */
+static int params_keep_class_change(void)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,
+	};
+	int err;
+
+	if (syscall(SYS_sched_setattr, 0, &attr, 0)) {
+		ksft_print_msg("a slice of its own is refused: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	attr.sched_policy = SCHED_DEADLINE;
+	attr.sched_flags = SCHED_FLAG_KEEP_PARAMS;
+	attr.sched_deadline = PERIOD;
+	attr.sched_period = PERIOD;
+	err = syscall(SYS_sched_setattr, 0, &attr, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("to SCHED_DEADLINE with SCHED_FLAG_KEEP_PARAMS: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* A deadline task that blocks on the lock: it boosts its owner. */
+static void *boosting_waiter(void *arg)
+{
+	return (void *)(long)wait_lock_for(300, 1);
+}
+
+/*
+ * SCHED_FLAG_KEEP_PARAMS with SCHED_DEADLINE on an owner that priority
+ * inheritance has boosted into the class: the class of the request is the
+ * one the owner already has, so a class change is not what it asks for,
+ * and the policy it asks for is one the kept parameters would not give
+ * it. It is refused with EINVAL; admitted, it reserves the bandwidth of
+ * the owner's slice and nothing ever releases it.
+ */
+static int params_keep_boosted(void)
+{
+	struct dl_attr slice = {
+		.size = sizeof(slice),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,	/* what the request would keep */
+	};
+	struct dl_attr keep = {
+		.size = sizeof(keep),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = SCHED_FLAG_KEEP_PARAMS,
+		.sched_deadline = PERIOD,
+		.sched_period = PERIOD,
+	};
+	int ret = KSFT_PASS, err;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (syscall(SYS_sched_setattr, owner_tid, &slice, 0)) {
+		ksft_print_msg("the owner cannot take a slice of its own: %s\n",
+			       strerror(errno));
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);		/* it blocks, the owner is boosted */
+	err = syscall(SYS_sched_setattr, owner_tid, &keep, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("KEEP_PARAMS to SCHED_DEADLINE on a boosted owner: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	join(w);
+out:
+	join(o);
+	return ret;
+}
+
+/* cpuset moves, from the main process, outside P */
+
+/* Shared with the processes of a case: atomic in shared memory. */
+static atomic_int *mv_state;
+
+/* mv_state[0] says a child is up (1) or was refused SCHED_DEADLINE (-1). */
+static int child_up(const char *what, int fast)
+{
+	if (wait_for_step(5000, fast ? 100 * NSEC_PER_USEC : NSEC_PER_MSEC, mv_state[0])) {
+		ksft_print_msg("%s: the task never started\n", what);
+		return -1;
+	}
+	if (mv_state[0] != 1) {
+		ksft_print_msg("%s: the task was refused SCHED_DEADLINE\n", what);
+		return -1;
+	}
+	return 0;
+}
+
+/* A process whose one deadline thread does what @mode says, in the root cgroup. */
+enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_YIELDED, MV_NON_CONTENDING };
+
+static pid_t mover(enum mv_mode mode, uint64_t runtime)
+{
+	pid_t pid = fork_child("a deadline task");
+
+	if (!pid) {
+		if (set_deadline(0, runtime, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		if (mode == MV_ASLEEP) {
+			spin_ns(NSEC_PER_MSEC);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_NON_CONTENDING) {
+			sleep_ns(PERIOD);
+			spin_ns(runtime / 4);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_YIELDED) {
+			*mv_state = 1;
+			for (;;) {
+				spin_ns(NSEC_PER_MSEC / 2);
+				mv_state[1]++;
+				sched_yield();
+			}
+		}
+		*mv_state = 1;
+		for (;;)
+			mv_state[1]++;
+	}
+	return pid;
+}
+
+static int moves(enum mv_mode mode, const char *name)
+{
+	int base_p, base_r, after_p, after_r, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up(name, mode == MV_NON_CONTENDING)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (mode == MV_NON_CONTENDING)
+		sleep_ns(NSEC_PER_MSEC);
+	else
+		sleep_ns(2 * PERIOD);
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		ret = KSFT_FAIL;
+	}
+	if (mode == MV_THROTTLED || mode == MV_YIELDED) {
+		int last = mv_state[1];
+
+		sleep_ns(3 * PERIOD);
+		if (mv_state[1] == last) {
+			ksft_print_msg("%s: moved, never ran again\n", name);
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after_p = probe(cg_p);
+	after_r = probe(root);
+	if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+		ksft_print_msg("%s: probe tasks admitted, P %d -> %d, the other CPUs %d -> %d\n",
+			       name, base_p, after_p, base_r, after_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int move_refused(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, err;
+	pid_t in_p, pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	/* 60% in P, then a task of 60% moved in: refused. */
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	pid = mover(MV_ASLEEP, 60 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a task of 60% to move into P", 0)) {
+		reap(pid);
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	err = enter(cg_p, pid);
+	if (err != -EBUSY) {
+		ksft_print_msg("a move beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	reap(in_p);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a refused move changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int move_asleep(void)
+{
+	return moves(MV_ASLEEP, "move_asleep");
+}
+
+static int move_throttled(void)
+{
+	return moves(MV_THROTTLED, "move_throttled");
+}
+
+static int move_non_contending(void)
+{
+	return moves(MV_NON_CONTENDING, "move_non_contending");
+}
+
+/* Deaths, in P */
+
+enum die_mode { DIE_RUNNING, DIE_SWITCHED_OUT, DIE_SHRUNK };
+
+/*
+ * A deadline task of 30% that runs 5ms of a fresh period, its 0-lag time
+ * ahead, and dies there: as it is, or switched to SCHED_OTHER, its
+ * bandwidth then going at that time; or shrunk to 10%, which puts that
+ * time behind it, and the bandwidth goes at its death.
+ */
+static int die_case(enum die_mode mode, const char *what)
+{
+	pid_t pid = fork_child("a task that dies");
+	int status;
+
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		sleep_ns(PERIOD);
+		spin_ns(5 * NSEC_PER_MSEC);
+		if (mode == DIE_SWITCHED_OUT && set_other(0))
+			_exit(2);
+		if (mode == DIE_SHRUNK && set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+			_exit(3);
+		_exit(0);
+	}
+	if (pid < 0 || waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status)) {
+		ksft_print_msg("a task that dies %s failed\n", what);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+static int death_running(void)
+{
+	return die_case(DIE_RUNNING, "running");
+}
+
+static int death_switched_out(void)
+{
+	return die_case(DIE_SWITCHED_OUT, "switched out");
+}
+
+static int death_shrunk(void)
+{
+	return die_case(DIE_SHRUNK, "shrunk");
+}
+
+/*
+ * Waits until the spinner's mv_state[1] moves (@moving): it runs; or stops
+ * for 5ms: it is throttled. 0, or -1 after 1s.
+ */
+static int wait_spinner(int moving)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+	int last = mv_state[1];
+
+	while (now_ns() < end) {
+		sleep_ns(moving ? NSEC_PER_MSEC : 5 * NSEC_PER_MSEC);
+		if (moving ? mv_state[1] != last : mv_state[1] == last)
+			return 0;
+		last = mv_state[1];
+	}
+	return -1;
+}
+
+enum kill_when { KILL_AFTER, KILL_THROTTLED, KILL_RUNNING };
+
+/*
+ * A deadline task of 30% (mover()) killed in the state @mode leaves it in,
+ * @after that, or once it is throttled, or while it runs. The killer
+ * leaves P first, to watch and kill from another CPU without taking the
+ * task's.
+ */
+static int kill_case(enum mv_mode mode, enum kill_when when, uint64_t after, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t pid;
+
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (enter(root, getpid())) {
+		ksft_print_msg("killed %s: the killer cannot leave P\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (child_up(what, 1)) {
+		ret = KSFT_FAIL;
+	} else if (when != KILL_AFTER && wait_spinner(when == KILL_RUNNING)) {
+		ksft_print_msg("killed %s: never %s\n", what,
+			       when == KILL_RUNNING ? "ran" : "throttled");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(after);
+	}
+out:
+	reap(pid);
+	return ret;
+}
+
+static int death_killed_asleep(void)
+{
+	return kill_case(MV_ASLEEP, KILL_AFTER, 2 * PERIOD, "asleep");
+}
+
+static int death_killed_non_contending(void)
+{
+	return kill_case(MV_NON_CONTENDING, KILL_AFTER, 0, "before its 0-lag time");
+}
+
+static int death_killed_throttled(void)
+{
+	return kill_case(MV_THROTTLED, KILL_THROTTLED, 0, "throttled");
+}
+
+static int death_killed_running(void)
+{
+	return kill_case(MV_THROTTLED, KILL_RUNNING, 0, "running");
+}
+
+/*
+ * A robust priority inheritance lock that processes share: mv_state[0]
+ * tells its owner's state, mv_state[2] its waiter's.
+ */
+static pthread_mutex_t *shm_lock;
+
+/* 0, or -1 where such a mutex is not to be had. */
+static int shm_lock_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err)
+		err = pthread_mutexattr_setpshared(&a, PTHREAD_PROCESS_SHARED);
+	if (!err)
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+	if (!err)
+		err = pthread_mutexattr_setrobust(&a, PTHREAD_MUTEX_ROBUST);
+	if (!err)
+		err = pthread_mutex_init(shm_lock, &a);
+	if (err) {
+		ksft_print_msg("no shared priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	mv_state[0] = 0;
+	mv_state[2] = 0;
+	return 0;
+}
+
+/* Takes the lock, as a deadline task of 30% if @dl, and sleeps with it. */
+static pid_t lock_owner(int dl)
+{
+	pid_t pid = fork_child("the owner of the lock");
+
+	if (!pid) {
+		if ((dl && set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) ||
+		    pthread_mutex_lock(shm_lock)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/* A deadline task of 5ms every 50ms that waits for the lock: 2 once it got it. */
+static pid_t lock_waiter(void)
+{
+	pid_t pid = fork_child("a waiter of the lock");
+
+	if (!pid) {
+		int err;
+
+		if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+			mv_state[2] = -1;
+			_exit(1);
+		}
+		mv_state[2] = 1;
+		err = pthread_mutex_lock(shm_lock);
+		if (err == EOWNERDEAD)
+			err = pthread_mutex_consistent(shm_lock);
+		if (!err)
+			pthread_mutex_unlock(shm_lock);
+		mv_state[2] = err ? -2 : 2;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/* @var reaches @want within 1s: 0, or -1. */
+static int wait_state(atomic_int *var, int want)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+
+	while (now_ns() < end) {
+		if (*var == want)
+			return 0;
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+/*
+ * The owner sleeps with the lock, a deadline task if @dl_owner, and the
+ * waiter boosts it; then @kill_owner, or the waiter, dies. A killed owner
+ * does not die boosted: its exit releases its priority inheritance
+ * futexes first (futex_exit_release()), which deboosts it.
+ */
+static int pi_death(int dl_owner, int kill_owner, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t owner, waiter = -1;
+
+	if (shm_lock_init())
+		return KSFT_SKIP;
+	owner = lock_owner(dl_owner);
+	if (owner < 0)
+		return KSFT_FAIL;
+	if (wait_state(&mv_state[0], 1)) {
+		ksft_print_msg("%s: no owner\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	waiter = lock_waiter();
+	if (waiter < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_state(&mv_state[2], 1)) {
+		ksft_print_msg("%s: no waiter\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);	/* blocked, boosting the owner */
+	if (kill_owner) {
+		reap(owner);
+		owner = -1;
+		if (wait_state(&mv_state[2], 2)) {
+			ksft_print_msg("%s: the waiter did not get the lock\n", what);
+			ret = KSFT_FAIL;
+		}
+	} else {
+		reap(waiter);
+		waiter = -1;
+		sleep_ns(50 * NSEC_PER_MSEC);	/* the owner deboosted asleep */
+	}
+out:
+	reap(waiter);
+	reap(owner);
+	return ret;
+}
+
+static int death_pi_waiter(void)
+{
+	return pi_death(0, 0, "a waiter killed");
+}
+
+static int death_pi_owner(void)
+{
+	return pi_death(0, 1, "a boosted owner killed");
+}
+
+static int death_pi_deadline_owner(void)
+{
+	return pi_death(1, 1, "a boosted deadline owner killed");
+}
+
+/* Deadline threads of one process, asleep, throttled and running, killed at once. */
+static void *group_thread(void *arg)
+{
+	long kind = (long)arg;
+
+	if (set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1000;
+		return NULL;
+	}
+	spin_ns(NSEC_PER_MSEC);
+	mv_state[0]++;
+	if (!kind)
+		for (;;)
+			pause();
+	for (;;)
+		;
+	return NULL;
+}
+
+static int death_group(void)
+{
+	pid_t pid;
+
+	mv_state[0] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (!pid) {
+		pthread_t t;
+		long i;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, group_thread, (void *)i))
+				mv_state[0] = -1000;
+		for (;;)
+			pause();
+	}
+	if (wait_for(5000, mv_state[0] < 0 || mv_state[0] == 3))
+		ksft_print_msg("only %d threads of the group started\n", (int)mv_state[0]);
+	sleep_ns(PERIOD / 2);
+	reap(pid);
+	if (mv_state[0] != 3) {
+		ksft_print_msg("a thread of the group refused SCHED_DEADLINE\n");
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* sched_yield() */
+
+static atomic_int y_rounds;	/* the yields of the yielder */
+
+/*
+ * A deadline task of 30% that yields half a millisecond into every
+ * period: it spends nearly all of its time throttled by sched_yield(),
+ * with runtime left, so its 0-lag time is a few milliseconds away and the
+ * inactive timer that releases its bandwidth is armed for that long.
+ * @t_state is 2 while it is in the yield.
+ */
+static void *yielder(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop) {
+		spin_ns(NSEC_PER_MSEC / 2);
+		y_rounds++;
+		t_state = 2;
+		sched_yield();
+		t_state = 1;
+	}
+	return NULL;
+}
+
+static int start_yielder(pthread_t *t)
+{
+	t_state = 0;
+	spin_stop = 0;
+	y_rounds = 0;
+	if (pthread_create(t, NULL, yielder, NULL)) {
+		ksft_print_msg("the yielder thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state > 0)
+		return 0;
+	join(*t);
+	return -1;
+}
+
+/*
+ * A task switched to SCHED_OTHER while it is throttled by sched_yield():
+ * its bandwidth comes back, and it runs again.
+ */
+static int yield_switched_out(void)
+{
+	int base, ret = KSFT_PASS;
+	pthread_t t;
+
+	base = probe(NULL);
+	if (base < 3) {
+		ksft_print_msg("no room for a task of 30%%\n");
+		return KSFT_SKIP;
+	}
+	if (start_yielder(&t))
+		return KSFT_FAIL;
+	if (wait_throttled(t) || t_state != 2) {
+		ksft_print_msg("the task never yielded\n");
+		ret = KSFT_FAIL;
+	} else if (set_other(t_tid)) {
+		ret = KSFT_FAIL;
+	} else {
+		int last = y_rounds;
+
+		sleep_ns(2 * PERIOD);
+		if (y_rounds == last) {
+			ksft_print_msg("yielded, then switched out: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "yielded, then switched out"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+static int yield_moved(void)
+{
+	return moves(MV_YIELDED, "yield_moved");
+}
+
+/* A task killed while it is throttled by sched_yield(). */
+static int yield_killed(void)
+{
+	pid_t pid;
+
+	mv_state[0] = 0;
+	pid = fork_child("a task that yields");
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			spin_ns(NSEC_PER_MSEC / 2);
+			sched_yield();
+		}
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a task that yields", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	reap(pid);
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_DL_OVERRUN */
+
+static atomic_int got_xcpu;
+
+static void catch_xcpu(int sig)
+{
+	got_xcpu = 1;
+}
+
+/* A deadline task of 10% with SCHED_FLAG_DL_OVERRUN that runs past it. */
+static void *overruns(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD, SCHED_FLAG_DL_OVERRUN)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	spin_ns(15 * NSEC_PER_MSEC);
+	while (!spin_stop)
+		sleep_ns(10 * NSEC_PER_MSEC);
+	return NULL;
+}
+
+/*
+ * The thread group of a task with SCHED_FLAG_DL_OVERRUN that overruns its
+ * runtime gets SIGXCPU. Caught here: the overrun leaves the task's
+ * bandwidth reserved, and it comes back when the task leaves
+ * SCHED_DEADLINE.
+ */
+static int overrun_caught(void)
+{
+	int base, ret = KSFT_PASS;
+	uint64_t end;
+	pthread_t t;
+
+	if (signal(SIGXCPU, catch_xcpu) == SIG_ERR)
+		return KSFT_FAIL;
+	base = probe(NULL);
+	if (base < 1) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	got_xcpu = 0;
+	t_state = 0;
+	spin_stop = 0;
+	if (pthread_create(&t, NULL, overruns, NULL))
+		return KSFT_FAIL;
+	while (!t_state)
+		sleep_ns(NSEC_PER_MSEC);
+	end = now_ns() + 2 * NSEC_PER_SEC;
+	while (!got_xcpu && now_ns() < end)
+		sleep_ns(5 * NSEC_PER_MSEC);
+	if (t_state != 1) {
+		ret = KSFT_FAIL;
+	} else if (!got_xcpu) {
+		ksft_print_msg("it overran its runtime, no SIGXCPU in 2s\n");
+		ret = KSFT_FAIL;
+	} else if (admits(base - 1, "an overrun told")) {
+		ret = KSFT_FAIL;	/* the overrun gave its 10% back */
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "after an overrun was told"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* The deadline threads of overrun_group_death(). */
+static void *og_thread(void *arg)
+{
+	switch ((int)(long)arg) {
+	case 0:		/* overruns its runtime: SIGXCPU to the group */
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_DL_OVERRUN))
+			_exit(1);
+		/*
+		 * The other two first: of one CPU, this one has the earliest
+		 * deadline of the three and would overrun, and kill them,
+		 * before they are where the signal is to find them.
+		 */
+		if (wait_for(5000, mv_state[0] >= 2))
+			_exit(2);
+		spin_ns(30 * NSEC_PER_MSEC);
+		break;
+	case 1:		/* asleep, its 0-lag time past */
+		if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		spin_ns(NSEC_PER_MSEC);
+		mv_state[0]++;
+		break;
+	case 2:		/* asleep, mostly before its 0-lag time */
+		if (set_deadline(0, 15 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			spin_ns(3 * NSEC_PER_MSEC);
+			sleep_ns(22 * NSEC_PER_MSEC);
+		}
+	}
+	for (;;)
+		pause();
+}
+
+/*
+ * The SIGXCPU of an overrun kills the whole thread group: the bandwidth
+ * of its three deadline threads comes back.
+ */
+static int overrun_group_death(void)
+{
+	pid_t pid;
+	int i, status;
+
+	mv_state[0] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (!pid) {
+		pthread_t t;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, og_thread, (void *)(long)i))
+				_exit(1);
+		for (;;)
+			pause();
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	for (i = 0; i < 300; i++) {
+		if (waitpid(pid, &status, WNOHANG) == pid)
+			break;
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+	if (i == 300) {
+		reap(pid);
+		ksft_print_msg("an overrun of 10ms every 100ms: no SIGXCPU in 3s\n");
+		return KSFT_FAIL;
+	}
+	if (WIFEXITED(status)) {
+		ksft_print_msg("a thread of the group was refused SCHED_DEADLINE\n");
+		return KSFT_SKIP;
+	}
+	if (!WIFSIGNALED(status) || WTERMSIG(status) != SIGXCPU) {
+		ksft_print_msg("the group of an overrun died of signal %d, not SIGXCPU\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_RECLAIM */
+
+#define RECLAIM_MIN	40	/* % of a CPU a reclaiming task of 10% gets */
+#define PLAIN_MAX	25	/* % one without the flag gets */
+#define SHARE_SLACK	15	/* % two measures of the same may differ by */
+#define SHARE_NS	(400 * NSEC_PER_MSEC)
+
+/*
+ * A deadline task with SCHED_FLAG_RECLAIM runs past its runtime, taking
+ * the bandwidth of its root domain that no deadline task uses: of a CPU
+ * of its own, a task of 10% every 100ms gets most of it, where one
+ * without the flag gets its 10%.
+ */
+static int reclaim_runs(void)
+{
+	int plain, grub, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	plain = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM))
+		return KSFT_FAIL;
+	grub = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (plain > PLAIN_MAX || grub < RECLAIM_MIN) {
+		ksft_print_msg("of a CPU, a task of 10%% got %d%%, with SCHED_FLAG_RECLAIM %d%%\n",
+			       plain, grub);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A reclaiming task switched out of SCHED_DEADLINE before its 0-lag time
+ * and back, which leaves its bandwidth standing until the inactive timer
+ * releases it: what it reclaims afterwards is what it reclaimed before.
+ */
+static int reclaim_switched(void)
+{
+	int before, after = 0, r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	before = share_of(t, SHARE_NS);
+	for (r = 0; r < 4; r++) {
+		if (set_other(t_tid)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (r & 1)
+			sleep_ns(5 * NSEC_PER_MSEC);
+		if (set_deadline_flags(t_tid, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			ksft_print_msg("round %d of 4: back to SCHED_DEADLINE refused\n",
+				       r + 1);
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(PERIOD);
+	}
+	if (ret == KSFT_PASS)
+		after = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (ret == KSFT_PASS &&
+	    (after < before - SHARE_SLACK || after > before + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%% of a CPU, %d%% after 4 round trips\n",
+			       before, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A reclaiming deadline task of 10% that spins, publishing the CPU time
+ * it has used in milliseconds, and idles while mv_state[2] is set. The
+ * moves happen then: a cpuset move waits for an RCU grace period, which
+ * a task that reclaims a whole CPU of its own keeps from ending where
+ * RCU runs in kthreads of its own (PREEMPT_RT).
+ */
+static pid_t reclaim_mover(void)
+{
+	pid_t pid = fork_child("a reclaiming task");
+
+	if (!pid) {
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			if (mv_state[2]) {
+				sleep_ns(10 * NSEC_PER_MSEC);
+				continue;
+			}
+			spin_ns(NSEC_PER_MSEC);
+			mv_state[1] = (int)(thread_time_ns() / NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* Moves the mover to @cg, idle: 0, or -1. */
+static int move_idle(const char *cg, pid_t pid)
+{
+	int err;
+
+	mv_state[2] = 1;
+	sleep_ns(20 * NSEC_PER_MSEC);	/* it stops spinning */
+	err = enter(cg, pid);
+	mv_state[2] = 0;
+	return err;
+}
+
+/* The share of a CPU, in percent, the mover uses over the next @ns. */
+static int mover_share(uint64_t ns)
+{
+	uint64_t wall = now_ns();
+	int cpu = mv_state[1];
+
+	sleep_ns(ns);
+	cpu = mv_state[1] - cpu;
+	wall = now_ns() - wall;
+	return (int)(100 * (uint64_t)cpu * NSEC_PER_MSEC / wall);
+}
+
+/*
+ * A reclaiming task moved into P and back, then into P again: what it
+ * reclaims of P's CPU is what it reclaimed there before, and the free
+ * bandwidth of both root domains is the one it found.
+ */
+static int reclaim_moved(void)
+{
+	int base_p, base_r, in_p, again = 0, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	mv_state[2] = 1;		/* idle until it is in P */
+	pid = reclaim_mover();
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a reclaiming task", 0) || move_idle(cg_p, pid)) {
+		mv_state[2] = 1;
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	in_p = mover_share(SHARE_NS);
+	if (move_idle(root, pid) || move_idle(cg_p, pid)) {
+		ksft_print_msg("a move of a reclaiming task refused\n");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(2 * PERIOD);
+		again = mover_share(SHARE_NS);
+		if (in_p < RECLAIM_MIN || again < in_p - SHARE_SLACK ||
+		    again > in_p + SHARE_SLACK) {
+			ksft_print_msg("reclaiming: %d%% of P's CPU, %d%% after a move\n",
+				       in_p, again);
+			ret = KSFT_FAIL;
+		}
+	}
+	mv_state[2] = 1;
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a reclaiming task moved: the free bandwidth changed\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Deadline servers */
+
+#define SERVER_RUNTIME	(100 * NSEC_PER_MSEC)	/* 10%, where the cases start from */
+#define SERVER_PERIOD	(1000 * NSEC_PER_MSEC)
+
+static const char server_dir[] = "/sys/kernel/debug/sched/fair_server";
+static uint64_t *server_found;	/* runtime and period of each CPU's fair server */
+
+static int server_get(int cpu, const char *name, uint64_t *val)
+{
+	char path[96], buf[32];
+
+	snprintf(path, sizeof(path), "%s/cpu%d/%s", server_dir, cpu, name);
+	if (read_file(path, buf, sizeof(buf)))
+		return -1;
+	*val = strtoull(buf, NULL, 10);
+	return 0;
+}
+
+static int server_put(int cpu, const char *name, uint64_t val)
+{
+	char dir[96], buf[32];
+
+	snprintf(dir, sizeof(dir), "%s/cpu%d", server_dir, cpu);
+	snprintf(buf, sizeof(buf), "%llu", (unsigned long long)val);
+	return write_file(dir, name, buf);
+}
+
+/* The fair server of @cpu runs @runtime every @period: 0, or -errno. */
+static int server_set(int cpu, uint64_t runtime, uint64_t period)
+{
+	uint64_t cur;
+	int err;
+
+	if (!server_found || server_get(cpu, "period", &cur))
+		return -ENOENT;
+	/* Each write keeps the runtime within the period. */
+	if (runtime <= cur) {
+		err = server_put(cpu, "runtime", runtime);
+		return err ? err : server_put(cpu, "period", period);
+	}
+	err = server_put(cpu, "period", period);
+	return err ? err : server_put(cpu, "runtime", runtime);
+}
+
+/* The fair servers as the test finds them, or none without debugfs. */
+static void server_init(void)
+{
+	int cpu;
+
+	server_found = calloc(2 * ncpus, sizeof(*server_found));
+	for (cpu = 0; server_found && cpu < ncpus; cpu++) {
+		if (server_get(cpu, "runtime", &server_found[2 * cpu]) ||
+		    server_get(cpu, "period", &server_found[2 * cpu + 1])) {
+			free(server_found);
+			server_found = NULL;
+			ksft_print_msg("no fair server in debugfs: server cases skipped\n");
+		}
+	}
+}
+
+/* Puts the fair servers back as the test found them. */
+static void server_restore(void)
+{
+	uint64_t runtime, period;
+	int cpu;
+
+	for (cpu = 0; server_found && cpu < ncpus; cpu++) {
+		int err;
+
+		if (server_get(cpu, "runtime", &runtime) || server_get(cpu, "period", &period))
+			continue;
+		if (runtime == server_found[2 * cpu] && period == server_found[2 * cpu + 1])
+			continue;
+		/*
+		 * A root domain whose departing tasks still hold their
+		 * bandwidth has room for nothing, a decrease included:
+		 * their 0-lag times are a period or two away.
+		 */
+		err = wait_for(1000, !server_set(cpu, server_found[2 * cpu],
+						 server_found[2 * cpu + 1]));
+		if (err)
+			ksft_print_msg("the fair server of CPU %d stays changed\n", cpu);
+	}
+}
+
+/* Deadline threads of 10% every 100ms that sleep until released. */
+static pthread_t *held;
+static int nheld, held_err;
+static sem_t held_up, held_go;
+
+static void *holder(void *arg)
+{
+	int err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+
+	held_err = err;
+	if (sem_post(&held_up))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	if (!err)
+		sem_wait_sig(&held_go);
+	return NULL;
+}
+
+/* Starts @n more holders: 0 once they are all admitted, or -1. */
+static int hold(int n)
+{
+	int max = 16 * ncpus + 16, err;
+	pthread_attr_t a;
+
+	if (!held) {
+		held = calloc(max, sizeof(*held));
+		if (!held || sem_init(&held_up, 0, 0) || sem_init(&held_go, 0, 0)) {
+			ksft_print_msg("the holders cannot be set up\n");
+			return -1;
+		}
+	}
+	err = pthread_attr_init(&a);
+	if (!err)
+		err = pthread_attr_setstacksize(&a, THREAD_STACK);
+	if (err) {
+		ksft_print_msg("pthread_attr: %s\n", strerror(err));
+		return -1;
+	}
+	for (; n > 0; n--) {
+		if (nheld == max || pthread_create(&held[nheld], &a, holder, NULL)) {
+			ksft_print_msg("a holder thread could not start\n");
+			return -1;
+		}
+		if (sem_wait_for(&held_up, 10000)) {
+			ksft_print_msg("a holder did not answer\n");
+			return -1;
+		}
+		if (held_err) {
+			join(held[nheld]);
+			return -1;
+		}
+		nheld++;
+	}
+	return 0;
+}
+
+static void release_held(void)
+{
+	int i;
+
+	for (i = 0; i < nheld; i++)
+		if (sem_post(&held_go))
+			ksft_print_msg("sem_post: %s\n", strerror(errno));
+	for (i = 0; i < nheld; i++)
+		join(held[i]);
+	nheld = 0;
+	sleep_ns(3 * PERIOD);	/* their 0-lag times */
+}
+
+/*
+ * The fair server of P's CPU, changed at runtime, takes from P's root
+ * domain what deadline tasks then cannot: from 10% every second, P admits
+ * two probe tasks fewer with 30%, one fewer with 100ms every 500ms, one
+ * more with the server off. A change P has no room for is refused.
+ */
+static int server_params(void)
+{
+	static const struct {
+		uint64_t runtime, period;
+		int more;
+		const char *what;
+	} step[] = {
+		{ 3 * SERVER_RUNTIME, SERVER_PERIOD, -2, "a fair server of 30%" },
+		{ SERVER_RUNTIME, SERVER_PERIOD / 2, -1, "a fair server of 100ms every 500ms" },
+		{ 0, SERVER_PERIOD, 1, "the fair server off" },
+		{ SERVER_RUNTIME, SERVER_PERIOD, 0, "the fair server back to 10%" },
+	};
+	int cpu = ncpus - 1, base, err, i, ret = KSFT_PASS;
+	uint64_t runtime;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	for (i = 0; i < (int)ARRAY_SIZE(step); i++) {
+		err = server_set(cpu, step[i].runtime, step[i].period);
+		if (err) {
+			ksft_print_msg("%s: %s\n", step[i].what, strerror(-err));
+			return KSFT_FAIL;
+		}
+		if (admits(base + step[i].more, step[i].what))
+			return KSFT_FAIL;
+	}
+	/*
+	 * With base - 1 probe tasks held, P has room for 10% more and less
+	 * than 20%: the server may grow to 20%, not to 30%.
+	 */
+	if (hold(base - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = server_put(cpu, "runtime", 2 * SERVER_RUNTIME);
+	if (err) {
+		ksft_print_msg("a fair server of 20%% with room for it: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	} else {
+		err = server_put(cpu, "runtime", 3 * SERVER_RUNTIME);
+		if (err != -EBUSY) {
+			ksft_print_msg("a fair server of 30%% with no room for it: %s, not EBUSY\n",
+				       err ? strerror(-err) : "accepted");
+			ret = KSFT_FAIL;
+		} else if (server_get(cpu, "runtime", &runtime) ||
+			   runtime != 2 * SERVER_RUNTIME) {
+			ksft_print_msg("a refused change changed the fair server\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	release_held();
+	if (ret == KSFT_PASS && admits(base, "after a refused change of the fair server"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * The rebuilds of the root domains' accounting put the fair servers'
+ * bandwidth back as it is set, not as it was at boot: P's own, when a
+ * partition root Q is made and unmade beside it; that of the other CPUs,
+ * when P is unmade and its CPU joins them with its server; P's again,
+ * when it is made anew.
+ */
+static int server_rebuilds(void)
+{
+	int cpu = ncpus - 1, base, n10, n30;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base - 2, "P, a fair server of 30%"))
+		return KSFT_FAIL;
+	if (make_partition(cg_q, ncpus - 2)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root Q beside P\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P, Q made beside it")) {
+		remove_partition(cg_q);
+		return KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	if (admits_in(cg_p, base - 2, "P, Q unmade beside it"))
+		return KSFT_FAIL;
+	/* P unmade: its CPU and its server join the root domain of the others. */
+	remove_p();
+	n30 = probe(root);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		return KSFT_FAIL;
+	n10 = probe(root);
+	if (n10 != n30 + 2) {
+		ksft_print_msg("P unmade: %d probe tasks, its CPU's server at 30%%, %d at 10%%\n",
+			       n30, n10);
+		return KSFT_FAIL;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) || make_p()) {
+		ksft_print_msg("P made anew, a fair server of 30%%: failed\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P made anew, a fair server of 30%") ||
+	    server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base, "P, the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/* Waits for @f to be set: a futex in shared memory. */
+static void wait_set(atomic_int *f)
+{
+	while (!*f)
+		syscall(SYS_futex, f, FUTEX_WAIT, 0, NULL, NULL, 0);
+}
+
+static void set_flag(atomic_int *f, int val)
+{
+	*f = val;
+	syscall(SYS_futex, f, FUTEX_WAKE, INT_MAX, NULL, NULL, 0);
+}
+
+/*
+ * A process of P that spins while mv_state[4 + @rt] is set, and sleeps
+ * otherwise: a fair task, or a real-time one (@rt) that leaves P's CPU
+ * for a millisecond every 50ms to the real-time kthreads of that CPU
+ * (PREEMPT_RT runs RCU and the timers there).
+ */
+static pid_t server_client(int rt)
+{
+	pid_t pid = fork_child(rt ? "a real-time task of P" : "a fair task of P");
+
+	if (!pid) {
+		struct sched_param sp = { .sched_priority = 1 };
+
+		if (enter(cg_p, getpid()) ||
+		    (rt && sched_setscheduler(0, SCHED_FIFO, &sp)))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			wait_set(&mv_state[4 + rt]);
+			spin_ns(rt ? 50 * NSEC_PER_MSEC : NSEC_PER_MSEC);
+			if (rt)
+				sleep_ns(NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* The share of a CPU, in percent, that process @pid uses over @ns. */
+static int proc_share(pid_t pid, uint64_t ns)
+{
+	struct timespec t0, t1;
+	uint64_t wall = now_ns();
+	clockid_t c;
+
+	if (clock_getcpuclockid(pid, &c) || clock_gettime(c, &t0))
+		return -1;
+	sleep_ns(ns);
+	if (clock_gettime(c, &t1))
+		return -1;
+	wall = now_ns() - wall;
+	return (int)(100 * ((t1.tv_sec - t0.tv_sec) * NSEC_PER_SEC +
+			    t1.tv_nsec - t0.tv_nsec) / wall);
+}
+
+#define STARVED_SLACK	8	/* % a starved fair task's share may be off its server's */
+
+/*
+ * The fair server of P's CPU changed from outside P in the states a change
+ * can find it in: stopped, deferred while a fair task of P runs, deferred
+ * while a real-time task starves it, serving it, deferred again after
+ * serving it, and deferred while the CPU idles. Each change starts a
+ * period of the server, which defers serving a starved task to its
+ * zero-laxity time, 0.7s in at 30% and 0.9s in at 10%. Afterwards, the
+ * starved task gets the server's bandwidth as set last, and P's free
+ * bandwidth is as it was. (Throttled after serving, the server is
+ * deferred anew at once: the last of the served task's runtime is charged
+ * to it as running in the background. Its idle wait ends in the timer
+ * callback that starts it, on a CPU with no tick in idle.)
+ */
+static int server_states(void)
+{
+	static const struct {
+		int fair, rt, ms;
+	} step[] = {
+		{ 0, 0, 1500 },	/* stopped: nothing of P ran for a period of the server */
+		{ 1, 0, 300 },	/* deferred while the fair task runs */
+		{ 1, 1, 300 },	/* deferred, the fair task starved: before 0.9s */
+		{ 1, 1, 800 },	/* serving it: after 0.7s */
+		{ 1, 1, 1500 },	/* deferred again: served from 0.9s to 1s */
+		{ 0, 0, 600 },	/* deferred while the CPU idles */
+	};
+	int cpu = ncpus - 1, base, i, s30 = -1, s10 = -1, ret = KSFT_PASS;
+	pid_t fair, rt;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base <= 0) {
+		ksft_print_msg("no room in P to probe\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	set_flag(&mv_state[4], 0);
+	set_flag(&mv_state[5], 0);
+	fair = server_client(0);
+	rt = server_client(1);
+	if (fair < 0 || rt < 0 || wait_for(5000, mv_state[0] >= 2)) {
+		if (fair >= 0 && rt >= 0)
+			ksft_print_msg("a fair or real-time task of P could not start\n");
+		ret = KSFT_FAIL;
+	}
+	/* Even steps set the server to 30%, odd ones back to 10%: its runtime alone. */
+	for (i = 0; i < (int)ARRAY_SIZE(step) && ret == KSFT_PASS; i++) {
+		set_flag(&mv_state[4], step[i].fair);
+		set_flag(&mv_state[5], step[i].rt);
+		sleep_ns(step[i].ms * NSEC_PER_MSEC);
+		if (server_put(cpu, "runtime", (i & 1 ? 1 : 3) * SERVER_RUNTIME)) {
+			ksft_print_msg("change %d of the fair server refused\n", i + 1);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS) {
+		set_flag(&mv_state[4], 1);
+		set_flag(&mv_state[5], 1);
+		if (server_put(cpu, "runtime", 3 * SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s30 = proc_share(fair, 2 * SERVER_PERIOD);
+		if (server_put(cpu, "runtime", SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s10 = proc_share(fair, 2 * SERVER_PERIOD);
+	}
+	set_flag(&mv_state[5], 0);
+	set_flag(&mv_state[4], 0);
+	reap(rt);
+	reap(fair);
+	if (ret == KSFT_PASS &&
+	    (s30 < 30 - STARVED_SLACK || s30 > 30 + STARVED_SLACK ||
+	     s10 < 10 - STARVED_SLACK || s10 > 10 + STARVED_SLACK)) {
+		ksft_print_msg("a starved fair task got %d%% of a server of 30%%, %d%% of 10%%\n",
+			       s30, s10);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits_in(cg_p, base, "after the fair server's changes"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * A reclaiming deadline task of P gets the share of P's CPU its root
+ * domain leaves free, which P's fair server takes from: less once the
+ * server grows, as much after a rebuild of the accounting (Q made and
+ * unmade), and what it had once the server is back. The server is then
+ * put back as the test found it, as P's free bandwidth must be.
+ */
+static int server_reclaim(void)
+{
+	int cpu = ncpus - 1, s10, s30, rebuilt, back, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	s10 = share_of(t, SHARE_NS);
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	s30 = share_of(t, SHARE_NS);
+	if (make_partition(cg_q, ncpus - 2))
+		ret = KSFT_FAIL;
+	remove_partition(cg_q);
+	rebuilt = share_of(t, SHARE_NS);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	back = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	server_restore();
+	if (ret == KSFT_PASS &&
+	    (s30 > s10 - SHARE_SLACK ||
+	     rebuilt < s30 - SHARE_SLACK || rebuilt > s30 + SHARE_SLACK ||
+	     back < s10 - SHARE_SLACK || back > s10 + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%%, %d%% at 30%%, %d%% rebuilt, %d%% back at 10%%\n",
+			       s10, s30, rebuilt, back);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Parameter changes while a task wakes on another CPU */
+
+#define WAKE_NS		(5 * NSEC_PER_SEC)	/* how long the changes go on */
+#define WAKE_BURST	2000			/* changes between two clock readings */
+
+/* this_bw and running_bw of @cpu, from sched_debug: 0, or -1. */
+static int rq_bw_of(int cpu, long *this_bw, long *running_bw)
+{
+	char line[256], want[32];
+	FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+	int in = 0, got = 0;
+
+	if (!f)
+		return -1;
+	snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+	while (got < 2 && fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, want, strlen(want)))
+			in = 1;
+		else if (!in || !val)
+			continue;
+		else if (strstr(line, ".running_bw") && sscanf(val + 1, "%ld", running_bw) == 1)
+			got++;
+		else if (strstr(line, ".this_bw") && sscanf(val + 1, "%ld", this_bw) == 1)
+			got++;
+	}
+	fclose(f);
+	return got == 2 ? 0 : -1;
+}
+
+/*
+ * A deadline task of Q, a partition root of two CPUs taken by two
+ * deadline tasks of an earlier deadline, that sleeps and wakes: it wakes
+ * on whichever of the two has the later deadline to run, which moves it.
+ * Its parameters change meanwhile, from a CPU outside Q.
+ *
+ * A wakeup that moves a task takes its bandwidth off the runqueue it
+ * slept on, and the enqueue on the one it wakes on, which that CPU may
+ * be left to do, charges it there. A change in between must not take it
+ * off the runqueue of either: once every task of the case is gone,
+ * this_bw of both CPUs is what it was before.
+ *
+ * Only then, with nothing left to change: sched_debug prints running_bw
+ * and this_bw of a CPU one after the other, holding no lock in between,
+ * and every parameter change takes a task's bandwidth off its runqueue
+ * and puts it back, so a reading of the two while the changes go on
+ * straddles one of those and tells nothing.
+ */
+static atomic_int *wk_state;		/* 0: tasks up, 1: wakeups, 2: stop */
+
+static pid_t waking_task(uint64_t runtime, uint64_t period, int sleeps)
+{
+	pid_t pid = fork_child("a waking deadline task");
+
+	if (!pid) {
+		if (enter(cg_q, getpid()) || set_deadline(0, runtime, period)) {
+			wk_state[0] = -1;
+			_exit(1);
+		}
+		wk_state[0]++;
+		for (;;) {
+			if (wk_state[2])
+				_exit(0);
+			spin_ns(sleeps ? 20 * NSEC_PER_USEC : NSEC_PER_MSEC);
+			if (sleeps) {
+				sleep_ns(80 * NSEC_PER_USEC);
+				wk_state[1]++;
+			}
+		}
+	}
+	return pid;
+}
+
+/* Changes the parameters of @pid over and over, until told to stop. */
+static pid_t changer_proc(pid_t pid)
+{
+	pid_t self = fork_child("a changer of the parameters");
+
+	if (!self) {
+		while (!wk_state[2]) {
+			if (set_deadline(pid, 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid, 10 * NSEC_PER_MSEC, PERIOD))
+				break;
+		}
+		_exit(0);
+	}
+	return self;
+}
+
+static int params_waking(void)
+{
+	int cpu_a = ncpus - 3, cpu_b = ncpus - 2, i, ret = KSFT_PASS, changes = 0;
+	long ta, ra, tb, rb, ta2, ra2, tb2, rb2;
+	pid_t pid[3] = { -1, -1, -1 }, changer = -1;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d-%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (rq_bw_of(cpu_a, &ta, &ra) || rq_bw_of(cpu_b, &tb, &rb)) {
+		remove_partition(cg_q);
+		ksft_print_msg("sched_debug tells of neither CPU's this_bw\n");
+		return KSFT_SKIP;
+	}
+	wk_state = mv_state + 8;
+	wk_state[0] = wk_state[1] = wk_state[2] = 0;
+	/* Two of 40% hold both CPUs with an earlier deadline than the third. */
+	pid[0] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[1] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[2] = waking_task(10 * NSEC_PER_MSEC, PERIOD, 1);
+	if (pid[0] < 0 || pid[1] < 0 || pid[2] < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	wait_for(5000, wk_state[0] >= 3 || wk_state[0] < 0);
+	if (wk_state[0] != 3) {
+		ksft_print_msg("no room for the deadline tasks of a partition of two CPUs\n");
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	changer = changer_proc(pid[2]);
+	if (changer < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	end = now_ns() + WAKE_NS;
+	while (now_ns() < end) {
+		for (i = 0; i < WAKE_BURST; i++, changes += 2) {
+			if (set_deadline(pid[2], 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid[2], 10 * NSEC_PER_MSEC, PERIOD)) {
+				ksft_print_msg("a change of the waking task refused\n");
+				ret = KSFT_FAIL;
+				goto out;
+			}
+		}
+	}
+	if (!wk_state[1]) {
+		ksft_print_msg("the waking task never woke\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	wk_state[2] = 1;
+	reap(changer);
+	for (i = 0; i < 3; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb) {
+			ksft_print_msg("%d changes: this_bw %ld and %ld, was %ld and %ld\n",
+				       changes, ta2, tb2, ta, tb);
+			ret = KSFT_FAIL;
+		}
+	}
+	remove_partition(cg_q);
+	return ret;
+}
+
+/* A random mix of the transitions above */
+
+#define MIX_WORKERS	6
+#define MIX_NS		(3 * NSEC_PER_SEC)
+#define MIX_SEED	20261103	/* fixed, so that a failure can be replayed */
+
+/*
+ * Each worker takes a random transition of its own over and over: into
+ * SCHED_DEADLINE with random parameters and flags, out of it, asleep,
+ * spinning past its runtime, through sched_yield(), through the shared
+ * mutex of the priority inheritance cases, and out of life. The case
+ * takes the ones only another task can: new parameters, a move between
+ * the cgroups, a kill, and the fair server of a CPU. Every worker it
+ * finds dead it starts anew, so the mix keeps its tasks.
+ *
+ * Nothing here looks at what a single transition does, which the cases
+ * above do one at a time. What it wants is the accounting whole
+ * afterwards: once every worker is gone and its 0-lag time is past, each
+ * of the three root domains admits what it admitted before, and this_bw
+ * of the three CPUs is what it was. The kernel's own warnings of
+ * running_bw and this_bw over- and underflow count as well, which the
+ * suite's dmesg tells of.
+ */
+static atomic_int *mix_state;		/* 0: stop, 1 + i: worker i's rounds */
+
+static pid_t mix_worker(int idx, unsigned int seed)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	static const uint64_t period[] = { 50, 100 };
+	static const uint64_t flag[] = { 0, SCHED_FLAG_RECLAIM, SCHED_FLAG_DL_OVERRUN };
+	pid_t pid = fork_child("a worker of the mix");
+
+	if (!pid) {
+		struct sigaction sa = { .sa_handler = catch_xcpu };
+		int err;
+
+		if (sigaction(SIGXCPU, &sa, NULL))
+			_exit(1);
+		seed += idx;
+		/*
+		 * A refusal is part of the mix: each call below is made in
+		 * whatever state the worker is in, and an error is what the
+		 * kernel is entitled to answer.
+		 */
+		while (!mix_state[0]) {
+			unsigned int r = rand_r(&seed), v = r / 8;
+
+			switch (r % 8) {
+			case 0:
+				set_deadline_flags(0, runtime[v % 4] * NSEC_PER_MSEC,
+						   period[(v / 4) % 2] * NSEC_PER_MSEC,
+						   flag[(v / 8) % 3]);
+				break;
+			case 1:
+				set_other(0);
+				break;
+			case 2:
+				sleep_ns((v % 20 + 1) * NSEC_PER_MSEC);
+				break;
+			case 3:
+				spin_ns((v % 15 + 1) * NSEC_PER_MSEC);
+				break;
+			case 4:
+				sched_yield();
+				break;
+			case 5:
+				err = pthread_mutex_trylock(shm_lock);
+				if (err == EOWNERDEAD)	/* its owner died holding it */
+					err = pthread_mutex_consistent(shm_lock);
+				if (!err) {
+					spin_ns(NSEC_PER_MSEC);
+					if (pthread_mutex_unlock(shm_lock))
+						_exit(1);
+				}
+				break;
+			case 6:
+				_exit(0);	/* the case starts another */
+			default:
+				sleep_ns(NSEC_PER_MSEC / 2);
+				break;
+			}
+			mix_state[1 + idx]++;
+		}
+		_exit(0);
+	}
+	return pid;
+}
+
+static int random_mix(void)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	int cpu_a = ncpus - 3, cpu_b = ncpus - 2, cpu_p = ncpus - 1;
+	int base_p, base_q, base_r, i, ret = KSFT_PASS, rounds = 0, sums;
+	long ta = 0, ra, tb = 0, rb, tp = 0, rp, ta2, ra2, tb2, rb2, tp2, rp2;
+	pid_t pid[MIX_WORKERS] = { -1, -1, -1, -1, -1, -1 };
+	unsigned int seed = MIX_SEED;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d-%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (shm_lock_init()) {
+		remove_partition(cg_q);
+		return KSFT_SKIP;	/* no shared mutex for the workers */
+	}
+	/* Without the two sums of sched_debug, the admissions alone tell. */
+	sums = !rq_bw_of(cpu_a, &ta, &ra) && !rq_bw_of(cpu_b, &tb, &rb) &&
+	       !rq_bw_of(cpu_p, &tp, &rp);
+	base_p = probe(cg_p);
+	base_q = probe(cg_q);
+	base_r = probe(root);
+	if (base_p <= 0 || base_q <= 0 || base_r <= 0) {
+		remove_partition(cg_q);
+		ksft_print_msg("no room to probe P, Q or the rest\n");
+		return KSFT_SKIP;
+	}
+	mix_state = mv_state + 16;
+	for (i = 0; i <= MIX_WORKERS; i++)
+		mix_state[i] = 0;
+	for (i = 0; i < MIX_WORKERS; i++) {
+		pid[i] = mix_worker(i, seed);
+		if (pid[i] < 0) {
+			ret = KSFT_FAIL;
+			goto out;
+		}
+	}
+
+	end = now_ns() + MIX_NS;
+	while (now_ns() < end) {
+		unsigned int r = rand_r(&seed);
+		int w = r % MIX_WORKERS;
+
+		switch ((r / MIX_WORKERS) % 7) {
+		case 0:
+			set_deadline(pid[w], runtime[r % 4] * NSEC_PER_MSEC, PERIOD);
+			break;
+		case 1:
+			set_other(pid[w]);
+			break;
+		case 2:
+			enter(cg_p, pid[w]);
+			break;
+		case 3:
+			enter(cg_q, pid[w]);
+			break;
+		case 4:
+			enter(root, pid[w]);
+			break;
+		case 5:
+			/* Dead and reaped already: ESRCH, and it starts anew below. */
+			if (kill(pid[w], SIGKILL) && errno != ESRCH)
+				ksft_print_msg("kill %d: %s\n", pid[w], strerror(errno));
+			break;
+		default:
+			server_set(r % 2 ? cpu_p : cpu_a,
+				   (r % 3 + 1) * SERVER_RUNTIME, SERVER_PERIOD);
+			break;
+		}
+		sleep_ns((r % 2000 + 100) * NSEC_PER_USEC);
+		/* Whoever died, of its own or killed, runs again. */
+		for (i = 0; i < MIX_WORKERS; i++) {
+			pid_t done = waitpid(pid[i], NULL, WNOHANG);
+
+			if (done < 0 && errno != EINTR) {
+				ksft_print_msg("waitpid %d: %s\n", pid[i], strerror(errno));
+				ret = KSFT_FAIL;
+				goto out;
+			}
+			if (done == pid[i]) {
+				pid[i] = mix_worker(i, seed + rounds);
+				if (pid[i] < 0) {
+					ret = KSFT_FAIL;
+					goto out;
+				}
+				rounds++;
+			}
+		}
+	}
+out:
+	mix_state[0] = 1;
+	for (i = 0; i < MIX_WORKERS; i++)
+		reap(pid[i]);
+	sleep_ns(5 * PERIOD);		/* their 0-lag times */
+	server_restore();		/* which needs the room they held */
+	if (ret != KSFT_PASS) {
+		remove_partition(cg_q);
+		return ret;
+	}
+	for (i = 0, rounds = 0; i < MIX_WORKERS; i++)
+		rounds += mix_state[1 + i];
+	if (!rounds) {
+		ksft_print_msg("the workers of the mix took no transition\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && sums) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2) ||
+		    rq_bw_of(cpu_p, &tp2, &rp2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb || tp2 != tp) {
+			ksft_print_msg("this_bw %ld %ld %ld, was %ld %ld %ld\n",
+				       ta2, tb2, tp2, ta, tb, tp);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, base_p, "P after the mix") ||
+	     admits_in(cg_q, base_q, "Q after the mix") ||
+	     admits_in(root, base_r, "the root domain of the rest after the mix"))) {
+		ksft_print_msg("seed %u, %d transitions\n", MIX_SEED, rounds);
+		ret = KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	return ret;
+}
+
+/* CPU hotplug, from the main process, with no P */
+
+static int hp_cpu = -1;
+
+/* Whether the online file of @cpu can be written, as hp_cpu's can. */
+static int cpu_hotpluggable(int cpu)
+{
+	char path[64];
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	return !access(path, W_OK);
+}
+
+static int cpu_online(int cpu, int on)
+{
+	char dir[64];
+
+	snprintf(dir, sizeof(dir), "/sys/devices/system/cpu/cpu%d", cpu);
+	return write_file(dir, "online", on ? "1" : "0");
+}
+
+/* Back online: every case that follows needs that CPU. */
+static void cpu_up(int cpu)
+{
+	int err = cpu_online(cpu, 1);
+
+	if (err)
+		ksft_print_msg("CPU %d cannot come back online: %s\n", cpu, strerror(-err));
+}
+
+/* Deadline sleepers of 30% each: offline and online a CPU meanwhile. */
+static int hotplug_sleepers(void)
+{
+	int base = probe(NULL), after, i, ret = KSFT_PASS;
+	pid_t pid[4];
+
+	for (i = 0; i < 4; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			return KSFT_FAIL;
+		}
+	}
+	if (cpu_online(hp_cpu, 0) || cpu_online(hp_cpu, 1)) {
+		ksft_print_msg("offline and online of CPU %d with deadline sleepers failed\n",
+			       hp_cpu);
+		ret = KSFT_FAIL;
+		cpu_up(hp_cpu);
+	}
+	for (i = 0; i < 4; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("hotplug with sleepers: %d probe tasks admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* The deadline tasks fill the CPUs: going offline must be refused. */
+static int hotplug_refused(void)
+{
+	int base = probe(NULL), n, i, ret = KSFT_PASS, err;
+	pid_t *pid;
+
+	if (base < 12) {
+		ksft_print_msg("needs room for 12 probe tasks\n");
+		return KSFT_SKIP;
+	}
+	/* 30% each, up to the free bandwidth but 30% to 60%: less than a CPU's. */
+	n = base / 3 - 1;
+	pid = calloc(n, sizeof(*pid));
+	if (!pid) {
+		ksft_print_msg("calloc: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			free(pid);
+			return KSFT_FAIL;
+		}
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (!err) {
+		ksft_print_msg("CPU %d went offline with its deadline bandwidth in use\n", hp_cpu);
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else if (err != -EBUSY) {
+		ksft_print_msg("offline refused with %s, not EBUSY\n", strerror(-err));
+		ret = KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++)
+		reap(pid[i]);
+	free(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && probe(NULL) != base) {
+		ksft_print_msg("a refused offline changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A deadline task leaves the class while it runs, its 0-lag time ahead,
+ * and the CPU it is on goes offline before that time.
+ */
+static int hotplug_departed(void)
+{
+	int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+	pid_t pid;
+
+	/*
+	 * What the root domain admits with the CPU offline and no deadline
+	 * task of ours: the release of the departed task's bandwidth
+	 * happens there, and the online that follows would rebuild the
+	 * accounting over it.
+	 */
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	base_off = probe(NULL);
+	cpu_up(hp_cpu);
+	if (base <= 0 || base_off <= 0) {
+		ksft_print_msg("no room to probe with the CPU online or offline\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(MV_THROTTLED, 40 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a departing deadline task", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(PERIOD / 2);
+	/* Running and queued: its bandwidth stays until its 0-lag time. */
+	if (set_other(pid))
+		ret = KSFT_FAIL;
+	cpu = task_cpu_of(pid);
+	if (cpu <= 0 || !cpu_hotpluggable(cpu)) {
+		cpu_set_t set;
+
+		/* Out of the class now: it may be moved to the CPU we offline. */
+		cpu = hp_cpu;
+		CPU_ZERO(&set);
+		CPU_SET(cpu, &set);
+		if (sched_setaffinity(pid, sizeof(set), &set))
+			ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && cpu_online(cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", cpu);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);		/* its 0-lag time, the CPU offline */
+	if (ret == KSFT_PASS) {
+		after = probe(NULL);
+		if (after != base_off) {
+			ksft_print_msg("departed, CPU %d offline: %d admitted, %d expected\n",
+				       cpu, after, base_off);
+			ret = KSFT_FAIL;
+		}
+	}
+	cpu_up(cpu);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("departed, CPU back online: %d admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * The fair server of the CPU set to 30%: its bandwidth leaves the root
+ * domain with the CPU, as it does at 10%, and comes back with the CPU as
+ * it was set.
+ */
+static int hotplug_server(void)
+{
+	int base, off10, off30;
+
+	if (server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off10 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (server_set(hp_cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base - 2, "a fair server of 30%"))
+		return KSFT_FAIL;
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused, its fair server at 30%%\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off30 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (off30 != off10) {
+		ksft_print_msg("CPU %d offline: %d probe tasks, its server at 10%%, %d at 30%%\n",
+			       hp_cpu, off10, off30);
+		return KSFT_FAIL;
+	}
+	if (admits(base - 2, "a fair server of 30%, its CPU back online") ||
+	    server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base, "the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/*
+ * Going offline, a CPU takes its fair server's bandwidth along with its
+ * capacity. With the server at the admission limit less 10%, the CPU goes
+ * offline with as many probe tasks held as the root domain admits but
+ * one, and no longer with one more.
+ */
+static int hotplug_server_refused(void)
+{
+	uint64_t rt_runtime, rt_period, runtime;
+	int n, err, ret = KSFT_PASS;
+	char buf[32];
+
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) ||
+	    read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf))) {
+		ksft_print_msg("no sched_rt_runtime_us or sched_rt_period_us\n");
+		return KSFT_SKIP;
+	}
+	read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf));
+	rt_runtime = atoll(buf);
+	read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf));
+	rt_period = atoll(buf);
+	if (!rt_period || 5 * rt_runtime < rt_period) {
+		ksft_print_msg("the admission limit is below a fifth of a CPU\n");
+		return KSFT_SKIP;
+	}
+	runtime = SERVER_PERIOD * rt_runtime / rt_period - SERVER_PERIOD / 10;
+	if (server_set(hp_cpu, runtime, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	n = probe(NULL);
+	if (n < 2 || hold(n - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (err) {
+		ksft_print_msg("CPU %d, room for one probe task left: offline refused (%s)\n",
+			       hp_cpu, strerror(-err));
+		ret = KSFT_FAIL;
+	} else if (cpu_online(hp_cpu, 1) || hold(1)) {
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else {
+		err = cpu_online(hp_cpu, 0);
+		if (err != -EBUSY) {
+			ksft_print_msg("CPU %d, no room left: offline %s, not EBUSY\n",
+				       hp_cpu, err ? strerror(-err) : "accepted");
+			if (!err)
+				cpu_up(hp_cpu);
+			ret = KSFT_FAIL;
+		}
+	}
+	release_held();
+	return ret;
+}
+
+/* Setup */
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+enum where {
+	MEASURED_IN_P,	/* in P, measured before and after */
+	IN_P,		/* in P, measuring itself */
+	FROM_OUTSIDE,	/* from the main process */
+};
+
+static const struct p_case {
+	const char *name;
+	int (*fn)(void);
+	enum where where;
+} p_cases[] = {
+	{ "pi_deadline_owner", pi_deadline_owner, MEASURED_IN_P },
+	{ "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
+	{ "pi_former_owner_asleep", pi_former_owner_asleep, MEASURED_IN_P },
+	{ "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
+	{ "params_asleep", params_asleep, IN_P },
+	{ "params_non_contending", params_non_contending, MEASURED_IN_P },
+	{ "params_throttled", params_throttled, MEASURED_IN_P },
+	{ "params_refused", params_refused, IN_P },
+	{ "params_keep_class_change", params_keep_class_change, MEASURED_IN_P },
+	{ "params_keep_boosted", params_keep_boosted, MEASURED_IN_P },
+	{ "move_asleep", move_asleep, FROM_OUTSIDE },
+	{ "move_throttled", move_throttled, FROM_OUTSIDE },
+	{ "move_non_contending", move_non_contending, FROM_OUTSIDE },
+	{ "move_refused", move_refused, FROM_OUTSIDE },
+	{ "death_running", death_running, MEASURED_IN_P },
+	{ "death_switched_out", death_switched_out, MEASURED_IN_P },
+	{ "death_shrunk", death_shrunk, MEASURED_IN_P },
+	{ "death_killed_asleep", death_killed_asleep, MEASURED_IN_P },
+	{ "death_killed_non_contending", death_killed_non_contending, MEASURED_IN_P },
+	{ "death_killed_throttled", death_killed_throttled, MEASURED_IN_P },
+	{ "death_killed_running", death_killed_running, MEASURED_IN_P },
+	{ "death_pi_waiter", death_pi_waiter, MEASURED_IN_P },
+	{ "death_pi_owner", death_pi_owner, MEASURED_IN_P },
+	{ "death_pi_deadline_owner", death_pi_deadline_owner, MEASURED_IN_P },
+	{ "death_group", death_group, MEASURED_IN_P },
+	{ "yield_switched_out", yield_switched_out, IN_P },
+	{ "yield_moved", yield_moved, FROM_OUTSIDE },
+	{ "yield_killed", yield_killed, MEASURED_IN_P },
+	{ "overrun_caught", overrun_caught, IN_P },
+	{ "overrun_group_death", overrun_group_death, MEASURED_IN_P },
+	{ "reclaim_runs", reclaim_runs, MEASURED_IN_P },
+	{ "reclaim_switched", reclaim_switched, MEASURED_IN_P },
+	{ "reclaim_moved", reclaim_moved, FROM_OUTSIDE },
+	{ "server_params", server_params, IN_P },
+	{ "server_rebuilds", server_rebuilds, FROM_OUTSIDE },
+	{ "server_states", server_states, FROM_OUTSIDE },
+	{ "server_reclaim", server_reclaim, MEASURED_IN_P },
+	{ "params_waking", params_waking, FROM_OUTSIDE },
+	{ "random_mix", random_mix, FROM_OUTSIDE },
+};
+
+#define N_P_CASES	((int)ARRAY_SIZE(p_cases))
+
+static const struct hp_case {
+	const char *name;
+	int (*fn)(void);
+} hp_cases[] = {
+	{ "hotplug_sleepers", hotplug_sleepers },
+	{ "hotplug_refused", hotplug_refused },
+	{ "hotplug_departed", hotplug_departed },
+	{ "hotplug_server", hotplug_server },
+	{ "hotplug_server_refused", hotplug_server_refused },
+};
+
+#define N_HP_CASES	((int)ARRAY_SIZE(hp_cases))
+
+/* The cases in P, each in a P of its own. */
+static void run_p_cases(void)
+{
+	char path[PATH_MAX + 32], buf[256];
+	int i, ret, added = 0, told = 0, ok;
+
+	ok = !find_cgroup2() &&
+	     snprintf(cg_p, sizeof(cg_p), "%s/deadline_bw_p", root) < (int)sizeof(cg_p) &&
+	     snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) < (int)sizeof(cg_q);
+	if (ok) {
+		/* The cpuset controller for the root's children while the cases run. */
+		snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+		ok = !read_file(path, buf, sizeof(buf));
+		added = ok && !strstr(buf, "cpuset");
+		if (added && write_file(root, "cgroup.subtree_control", "+cpuset")) {
+			ok = 0;
+			added = 0;
+		}
+	}
+	for (i = 0; i < N_P_CASES; i++) {
+		const struct p_case *c = &p_cases[i];
+
+		if (ok && make_p()) {
+			remove_p();
+			ok = 0;
+		}
+		if (!ok) {
+			if (!told++)
+				ksft_print_msg("no one-CPU partition root: cases in P skipped\n");
+			ksft_test_result_skip("%s\n", c->name);
+			continue;
+		}
+		if (c->where == MEASURED_IN_P)
+			ret = measured(c->name, cg_p, c->fn);
+		else if (c->where == IN_P)
+			ret = run_in(cg_p, c->fn);
+		else
+			ret = c->fn();
+		remove_p();
+		server_restore();
+		report(c->name, ret);
+	}
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+int main(void)
+{
+	char buf[64];
+	int i;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	ncpus = get_nprocs();
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs\n");
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	mv_state = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (mv_state == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	shm_lock = (pthread_mutex_t *)((char *)mv_state + 256);
+
+	ksft_set_plan(N_P_CASES + N_HP_CASES);
+	server_init();
+	run_p_cases();
+
+	for (i = ncpus - 1; i > 0 && hp_cpu < 0; i--) {
+		if (cpu_hotpluggable(i))
+			hp_cpu = i;
+	}
+	if (hp_cpu < 0)
+		ksft_print_msg("no CPU hotplug: hotplug cases skipped\n");
+	for (i = 0; i < N_HP_CASES; i++) {
+		if (hp_cpu < 0) {
+			ksft_test_result_skip("%s\n", hp_cases[i].name);
+			continue;
+		}
+		report(hp_cases[i].name, hp_cases[i].fn());
+		server_restore();
+	}
+	ksft_finished();
+}
diff --git a/tools/testing/selftests/sched/settings b/tools/testing/selftests/sched/settings
new file mode 100644
index 000000000000..a62d2fa1275c
--- /dev/null
+++ b/tools/testing/selftests/sched/settings
@@ -0,0 +1 @@
+timeout=600
-- 
2.47.3

[PATCH v2 15/21] sched/deadline: Record the root domain holding a task's bandwidth

[Mathieu Desnoyers]

Three answers to "which root domain counts this task's bandwidth" live
side by side:

 - admission takes the root domain of the CPU the task is on,
   sched_dl_overflow() through dl_bw_of(task_cpu(p));
 - a rebuild of the accounting takes the one of the task's cpuset,
   dl_add_task_root_domain() through dl_get_task_effective_cpus();
 - a release and a move take the one of the first active CPU of the
   task's own mask, dl_task_root_domain_cpu().

They agree while the task's mask, its cpuset and the CPU it last ran on
belong to one root domain, and part ways where they do not: a task
pinned to a CPU that goes offline, which admission control off allows,
has its bandwidth put by the rebuild in the root domain of its cpuset
while its mask has no active CPU and the CPU it is on belongs to the
default root domain; a sleeping task moved to another partition has it
in the destination's root domain while the CPU it last ran on is in the
one it left. The release then takes the bandwidth from a root domain
that never had it, which underflows, and leaves it in the one that has
it.

Record the CPU instead. Every site that hands bandwidth to a task
records a CPU of the root domain that counts it, in the entity:
admission, the rebuild, the move between root domains of
set_cpus_allowed_dl() (whose room was made by cpuset_can_attach()), and
the move of dl_task_offline_migration(). Every site that takes it back
or moves it reads that CPU, and the release clears it. Where dl_bw_gen
says a rebuild dropped the bandwidth of a task, a change of its
parameters and a move take nothing back, and record where the new
bandwidth is for the next rebuild. The mask is no longer consulted, so
dl_task_root_domain_cpu() has nothing left to guess and nothing to fall
back on but the CPU of a task whose bandwidth is counted nowhere.

The record is also what the departed list of "sched/deadline: Keep
departing bandwidth across a root domain rebuild" needs, which carried
a CPU of its own until here, and a task boosted back into the class
keeps it now, where dropping it left the release to the mask again.

Found by the Sashiko AI review of the first posting of this series,
which pointed at the fallback of "sched/deadline: Do not BUG() on a
deadline task with no active CPU allowed" pointing at a root domain that
rebuild had not credited. A review of this patch found the two sites
that kept taking the root domain of the CPU the task is on: the change
of parameters of sched_dl_overflow(), and the move of
dl_task_offline_migration().

Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  20 +++--
 kernel/sched/deadline.c | 167 +++++++++++++++++++++++++++-------------
 2 files changed, 124 insertions(+), 63 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6803df5ab13c..da3a3a4f7246 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -736,17 +736,21 @@ struct sched_dl_entity {
 	struct hrtimer			inactive_timer;
 
 	/*
+	 * @dl_bw_cpu is a CPU of the root domain that counts @dl_bw, -1
+	 * where none does: every site that hands that bandwidth out records
+	 * it, and every site that takes it back or moves it reads it.
+	 * @dl_bw_gen is the rebuild of the root domains' accounting that
+	 * last counted it: one since then dropped it, and there is nothing
+	 * to take back.
+	 *
 	 * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
-	 * time" keeps its bandwidth in its root domain until the inactive
-	 * timer removes it. @dl_departed_cpu is then a CPU of the root domain
-	 * that counts that bandwidth, and @dl_departed_node links it on the
-	 * list that a rebuild of the root domains' accounting puts that
-	 * bandwidth back from; -1 otherwise. @dl_bw_gen is the
-	 * rebuild that last counted the task's bandwidth: one since then
-	 * dropped it, and nothing is to be removed.
+	 * time" keeps its bandwidth until the inactive timer removes it:
+	 * @dl_departed_node links it meanwhile on the list that a rebuild
+	 * puts that bandwidth back from, and tells of that by not being
+	 * empty.
 	 */
 	struct list_head		dl_departed_node;
-	int				dl_departed_cpu;
+	int				dl_bw_cpu;
 	unsigned int			dl_bw_gen;
 
 	/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 85df2b0044be..1dbde0fd199b 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -252,15 +252,14 @@ static unsigned int dl_bw_gen;
  * @p leaves SCHED_DEADLINE, or dies, its bandwidth kept for now in the root
  * domain of @cpu. Under its rq lock.
  */
-static void dl_task_depart(struct task_struct *p, int cpu)
+static void dl_task_depart(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 
 	lockdep_assert_irqs_disabled();
-	if (dl_se->dl_departed_cpu >= 0)
-		return;
 	guard(raw_spinlock)(&dl_departed_lock);
-	dl_se->dl_departed_cpu = cpu;
+	if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
+		return;
 	list_add(&dl_se->dl_departed_node, &dl_departed_list);
 }
 
@@ -273,9 +272,10 @@ static void dl_task_undepart(struct task_struct *p)
 	struct sched_dl_entity *dl_se = &p->dl;
 
 	lockdep_assert_irqs_disabled();
+	if (list_empty(&dl_se->dl_departed_node))
+		return;
 	guard(raw_spinlock)(&dl_departed_lock);
-	dl_se->dl_departed_cpu = -1;
-	list_del(&dl_se->dl_departed_node);
+	list_del_init(&dl_se->dl_departed_node);
 }
 
 /*
@@ -284,24 +284,25 @@ static void dl_task_undepart(struct task_struct *p)
  * when a rebuild of the accounting has dropped it; and no rebuild running
  * counts it again. Under @p's rq lock.
  */
-static void dl_task_release_bw(struct task_struct *p, int cpu)
+static void dl_task_release_bw(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 	struct dl_bw *dl_b;
+	int cpu;
 
 	lockdep_assert_irqs_disabled();
 	guard(raw_spinlock)(&dl_departed_lock);
-	if (dl_se->dl_departed_cpu >= 0) {
-		cpu = dl_se->dl_departed_cpu;
-		dl_se->dl_departed_cpu = -1;
-		list_del(&dl_se->dl_departed_node);
-	}
-	if (dl_se->dl_bw_gen == dl_bw_gen) {
+	if (!list_empty(&dl_se->dl_departed_node))
+		list_del_init(&dl_se->dl_departed_node);
+	cpu = dl_se->dl_bw_cpu;
+	WARN_ON_ONCE(dl_se->dl_bw && cpu < 0);
+	if (cpu >= 0 && dl_se->dl_bw_gen == dl_bw_gen) {
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
 		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
 		raw_spin_unlock(&dl_b->lock);
 	}
+	dl_se->dl_bw_cpu = -1;
 	dl_se->dl_bw_gen = dl_bw_gen;
 }
 
@@ -460,6 +461,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p);
 
 /*
  * The utilization of a task cannot be immediately removed from
@@ -560,7 +562,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
 				if (READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				dl_task_release_bw(p, task_cpu(p));
+				dl_task_release_bw(p);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -579,7 +581,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 		 * SCHED_DEADLINE.
 		 */
 		if (READ_ONCE(p->__state) == TASK_DEAD)
-			dl_task_depart(p, task_cpu(p));
+			dl_task_depart(p);
 	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
@@ -817,19 +819,30 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 	}
 
 	/*
-	 * And we finally need to fix up root_domain(s) bandwidth accounting,
-	 * since p is still hanging out in the old (now moved to default) root
-	 * domain.
+	 * And we finally need to fix up root_domain(s) bandwidth accounting:
+	 * the bandwidth of @p is counted in the root domain that took it,
+	 * which is not the one of the runqueue it leaves if it was moved
+	 * asleep. A rebuild that dropped it has nothing here to move, and
+	 * the record left behind is what makes the next one count it.
 	 */
-	dl_b = &rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_sub(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
-
-	dl_b = &later_rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
+	scoped_guard (raw_spinlock, &dl_departed_lock) {
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
+
+			dl_b = &src_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw,
+				 cpumask_weight(src_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+
+			dl_b = &later_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_add(dl_b, p->dl.dl_bw,
+				 cpumask_weight(later_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+		}
+		p->dl.dl_bw_cpu = later_rq->cpu;
+	}
 
 	set_task_cpu(p, later_rq->cpu);
 	double_unlock_balance(later_rq, rq);
@@ -2312,7 +2325,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 			dl_se->dl_non_contending = 0;
 		}
 
-		dl_task_release_bw(p, task_cpu(p));
+		dl_task_release_bw(p);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3475,11 +3488,14 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
  */
 static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
 {
-	unsigned int cpu;
+	int cpu = p->dl.dl_bw_cpu;
 
-	cpu = cpumask_first_and(&p->cpus_mask, cpu_active_mask);
-	if (cpu >= nr_cpu_ids)
-		cpu = task_cpu(p);
+	/*
+	 * No bandwidth of @p's counted anywhere: nothing to take back or to
+	 * move, and the CPU it is on answers for its root domain.
+	 */
+	if (cpu < 0)
+		return task_cpu(p);
 	return cpu;
 }
 
@@ -3497,15 +3513,28 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
 		unsigned int src_cpu = dl_task_root_domain_cpu(p);
 		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
+		unsigned int cpu;
 
 		/*
 		 * We now free resources of the root_domain we are migrating
 		 * off. In the worst case, sched_setattr() may temporary fail
-		 * until we complete the update.
+		 * until we complete the update. A rebuild that dropped the
+		 * bandwidth of @p leaves nothing to free.
+		 */
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			raw_spin_lock(&src_dl_b->lock);
+			__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+			raw_spin_unlock(&src_dl_b->lock);
+		}
+
+		/*
+		 * The room in the root domain of the mask to come was made
+		 * by cpuset_can_attach(): that one counts the bandwidth from
+		 * here on.
 		 */
-		raw_spin_lock(&src_dl_b->lock);
-		__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
-		raw_spin_unlock(&src_dl_b->lock);
+		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
+		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
+		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
 	set_cpus_allowed_common(p, ctx);
@@ -3614,10 +3643,12 @@ void dl_add_task_root_domain(struct task_struct *p)
 	 */
 	if (p->dl.dl_bw_gen == dl_bw_gen)
 		goto unlock;
-	if (p->dl.dl_departed_cpu >= 0) {
-		cpu = p->dl.dl_departed_cpu;
-		if (!cpu_active(cpu))
+	if (!list_empty(&p->dl.dl_departed_node)) {
+		int recorded = p->dl.dl_bw_cpu;
+
+		if (recorded < 0 || !cpu_active(recorded))
 			goto unlock;
+		cpu = recorded;
 	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
@@ -3625,6 +3656,7 @@ void dl_add_task_root_domain(struct task_struct *p)
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_cpu = cpu;
 	p->dl.dl_bw_gen = dl_bw_gen;
 unlock:
 	raw_spin_unlock(&dl_departed_lock);
@@ -3754,7 +3786,7 @@ void dl_clear_root_domains(void)
 			dl_clear_root_domain(cpu_rq(cpu)->rd);
 	}
 	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
-		cpu = dl_se->dl_departed_cpu;
+		cpu = dl_se->dl_bw_cpu;
 		/*
 		 * Counted in the generation that ends here, and nowhere
 		 * since: one a rebuild dropped, for want of an active CPU to
@@ -3763,7 +3795,7 @@ void dl_clear_root_domains(void)
 		 */
 		if (dl_se->dl_bw_gen != dl_bw_gen - 1)
 			continue;
-		if (!cpu_active(cpu))
+		if (cpu < 0 || !cpu_active(cpu))
 			continue;
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
@@ -3820,7 +3852,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * nor for a special one: neither has bandwidth in
 			 * total_bw.
 			 */
-			dl_task_release_bw(p, dl_task_root_domain_cpu(p));
+			dl_task_release_bw(p);
 			released = true;
 		}
 		sub_rq_bw(&p->dl, &rq->dl);
@@ -3861,7 +3893,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 */
 	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
-		dl_task_depart(p, dl_task_root_domain_cpu(p));
+		dl_task_depart(p);
 	}
 
 	/*
@@ -3890,8 +3922,7 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	 * the task then removes the bandwidth from that one, as it did
 	 * before departures were recorded.
 	 */
-	if (p->dl.dl_departed_cpu >= 0)
-		dl_task_undepart(p);
+	dl_task_undepart(p);
 
 	/*
 	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -4104,9 +4135,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	u64 period = attr->sched_period ?: attr->sched_deadline;
 	u64 runtime = attr->sched_runtime;
 	u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
-	int cpus, err = -1, cpu = task_cpu(p);
-	struct dl_bw *dl_b = dl_bw_of(cpu);
+	int cpus, err = -1, cpu;
+	struct dl_bw *dl_b;
 	unsigned long cap;
+	bool counted;
 
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
@@ -4115,6 +4147,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
 		return 0;
 
+	/*
+	 * The bandwidth of a task of the class is counted in the root
+	 * domain that took it, which the CPU the task is on does not tell:
+	 * moved asleep, it has not run there yet. Change it where it is
+	 * counted; a rebuild that dropped it, and a task that enters the
+	 * class, have the root domain of that CPU take the new one. The
+	 * caller holds cpuset_mutex, so no rebuild runs meanwhile.
+	 */
+	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
+		  p->dl.dl_bw_gen == dl_bw_gen;
+	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
+	dl_b = dl_bw_of(cpu);
+
 	/*
 	 * Either if a task, enters, leave, or stays -deadline but changes
 	 * its parameters, we may need to update accordingly the total
@@ -4129,7 +4174,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
-		   !__dl_overflow(dl_b, cap, p->dl.dl_bw, new_bw)) {
+		   !__dl_overflow(dl_b, cap, counted ? p->dl.dl_bw : 0,
+				  new_bw)) {
 		/*
 		 * XXX this is slightly incorrect: when the task
 		 * utilization decreases, we should delay the total
@@ -4137,7 +4183,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		 * But this would require to set the task's "inactive
 		 * timer" when the task is not inactive.
 		 */
-		__dl_sub(dl_b, p->dl.dl_bw, cpus);
+		if (counted)
+			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		dl_change_utilization(p, new_bw);
 		err = 0;
@@ -4157,11 +4204,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 * domain that keeps it (dl_task_release_bw()). A rebuild cannot
 	 * run meanwhile: the caller holds cpuset_mutex.
 	 */
-	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
-		lockdep_assert_cpuset_lock_held();
-		lockdep_assert_dl_accounting_held();
-		if (p->dl.dl_departed_cpu >= 0)
-			dl_task_release_bw(p, -1);
+	if (!err && dl_policy(policy)) {
+		if (!task_has_dl_policy(p)) {
+			lockdep_assert_cpuset_lock_held();
+			lockdep_assert_dl_accounting_held();
+			/*
+			 * Whatever bandwidth the task still has counted
+			 * somewhere, the new one replaces: on the departed
+			 * list, or taken off it by a boost into the class
+			 * (switched_to_dl()), the record says where.
+			 */
+			if (p->dl.dl_bw_cpu >= 0)
+				dl_task_release_bw(p);
+		}
+		p->dl.dl_bw_cpu = cpu;
 		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
@@ -4301,7 +4357,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
 	__dl_clear_params(dl_se);
-	dl_se->dl_departed_cpu = -1;
+	dl_se->dl_bw_cpu = -1;
+	INIT_LIST_HEAD(&dl_se->dl_departed_node);
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-- 
2.47.3

[PATCH v2 16/21] sched/deadline: Move a task's bandwidth only where room was made

[Mathieu Desnoyers]

set_cpus_allowed_dl() moves the bandwidth of a deadline task off the
root domain that counts it whenever the new mask leaves that root
domain, on the strength of the room cpuset_can_attach() made in the one
the task moves to, as its comment says. Nothing checks that the room
was made, and two paths reach it without any:

 - cpuset_can_attach() counts the deadline tasks of the attach it is
   asked about. A task that becomes one after that has nothing set
   aside, and the attach moves its bandwidth all the same: the root
   domain it lands in ends short of what its tasks hold, and underflows
   when they leave. A task that flips between SCHED_DEADLINE and
   SCHED_OTHER while it is moved in and out of a one-CPU partition root
   drives total_bw of that partition to 2^64 - 576716 in 8-CPU VMs of
   v7.4-rc1 with the four deadline fixes of sched/urgent, in 1 VM of 4;
 - sched_setaffinity() of a deadline task takes the same path. With
   admission control on, a mask that does not cover the task's root
   domain is refused before this (__sched_setaffinity()), but with
   sched_rt_runtime_us at -1 nothing refuses it.

Say in the affinity context whether the caller made that room, as
SCA_DL_BW_RESERVED, and move the bandwidth only then. The bandwidth of a
task whose move no one made room for stays where it is counted, and the
next rebuild of the accounting attributes it to the root domain of the
task's cpuset.

Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  2 ++
 kernel/cgroup/cpuset.c  | 13 ++++++++++++-
 kernel/sched/core.c     | 14 ++++++++++++++
 kernel/sched/deadline.c | 15 ++++++++++-----
 kernel/sched/sched.h    |  2 ++
 5 files changed, 40 insertions(+), 6 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index da3a3a4f7246..52b450d212c6 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1960,6 +1960,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  * Return: zero if successful, or a negative error code
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
+extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
+					const struct cpumask *new_mask);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 4159abfbbbb5..5b8f1d9552a6 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3326,6 +3326,7 @@ static cpumask_var_t cpus_attach;
 
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
+	bool reserved = false;
 	struct mm_struct *mm;
 
 	lockdep_assert_cpuset_lock_held();
@@ -3335,11 +3336,21 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	else
 		cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
 			       subpartitions_cpus);
+	/*
+	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
+	 * deadline tasks cpuset_can_attach() counted: say so, so that this
+	 * task's bandwidth moves there with it.
+	 */
+	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+		reserved = true;
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	if (reserved)
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
+	else
+		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 04b62c16fa8c..0ff07258cf0c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3269,6 +3269,20 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
+/*
+ * The same, for a caller that has made room in the root domain of @new_mask
+ * for the deadline bandwidth of @p: that bandwidth moves there with it.
+ */
+int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+{
+	struct affinity_context ac = {
+		.new_mask  = new_mask,
+		.flags     = SCA_DL_BW_RESERVED,
+	};
+
+	return __set_cpus_allowed_ptr(p, &ac);
+}
+
 /*
  * Change a given task's CPU affinity to the intersection of its current
  * affinity mask and @subset_mask, writing the resulting mask to @new_mask.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 1dbde0fd199b..7a58e78ac5ce 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3505,12 +3505,17 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	WARN_ON_ONCE(!dl_task(p));
 
 	/*
-	 * Migrating a SCHED_DEADLINE task between exclusive
-	 * cpusets (different root_domains) entails a bandwidth
-	 * update. We already made space for us in the destination
-	 * domain (see cpuset_can_attach()).
+	 * Migrating a SCHED_DEADLINE task between exclusive cpusets
+	 * (different root_domains) entails a bandwidth update, and only
+	 * where the caller has made room for it: cpuset_attach_task() says
+	 * so of the tasks cpuset_can_attach() counted, and of no others.
+	 * The bandwidth of a task whose move no one made room for stays
+	 * where it is counted, for the next rebuild of the accounting to
+	 * attribute, rather than leaving the root domain it moves to short
+	 * of what its tasks hold.
 	 */
-	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
+	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+	    dl_task_needs_bw_move(p, ctx->new_mask)) {
 		unsigned int src_cpu = dl_task_root_domain_cpu(p);
 		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
 		unsigned int cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c7c5633524a..3e5bc8b6f887 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2953,6 +2953,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 #define SCA_MIGRATE_DISABLE	0x02
 #define SCA_MIGRATE_ENABLE	0x04
 #define SCA_USER		0x08
+/* The caller made room for @p's deadline bandwidth where the mask points. */
+#define SCA_DL_BW_RESERVED	0x10
 
 extern void update_group_capacity(struct sched_domain *sd, int cpu);
 
-- 
2.47.3

[PATCH v2 17/21] cgroup/cpuset: Give back the bandwidth an attach did not move

[Mathieu Desnoyers]

cpuset_can_attach() counts the bandwidth of every migrating deadline
task in sum_migrate_dl_bw, and cpuset_reserve_dl_bw() sets that sum
aside in the destination's root domain. The attach that follows moves
the bandwidth of each task that is still a deadline task, and nothing
answers for the reservation of one that is not: a task that left
SCHED_DEADLINE, or died, since it was counted moves none of it, and what
was set aside for it stays in the destination's total_bw. The root
domain then admits that much less, until the next rebuild of the
accounting.

A task that flips between SCHED_DEADLINE and SCHED_OTHER every
millisecond while it is moved in and out of a one-CPU partition root
leaves, in 200 moves, hundreds of thousands of units of a CPU behind, in
8-CPU VMs of v7.4-rc1 with the four deadline fixes of sched/urgent:
total_bw of the partition 52428 (its fair server) before, 367000 and
681572 after, in 2 VMs of 4.

Keep the tally of what the attach moves, and give the rest back once it
is through. The move says what it moved: cpuset_attach_task() reads the
root domain recorded for a task without the runqueue lock that
set_cpus_allowed_dl() decides under, so CPU hotplug moving the task to
another root domain in between, in dl_task_offline_migration(), would
leave the tally counting a move that did not happen and a reservation
standing for bandwidth no task of the destination holds.
set_cpus_allowed_dl() writes the bandwidth it hands to the root domain
of the new mask into the affinity context, and the tally takes it from
there: the offer of the reservation rests on the unlocked read, the
count on what the move did. With this change and the previous patch,
total_bw of the partition is 52428 after those 200 moves, in 6 VMs of 6.

Found by the randomized case of the selftest added earlier in this
series, which caught a fair server it could no longer lower in a
partition root with no room left.

Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h           |  3 ++-
 kernel/cgroup/cpuset-internal.h |  2 ++
 kernel/cgroup/cpuset.c          | 39 ++++++++++++++++++++++++++-------
 kernel/sched/core.c             | 11 +++++++---
 kernel/sched/deadline.c         |  5 ++++-
 kernel/sched/sched.h            |  2 ++
 6 files changed, 49 insertions(+), 13 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 52b450d212c6..24e28f2819d9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1961,7 +1961,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
-					const struct cpumask *new_mask);
+					const struct cpumask *new_mask,
+					u64 *moved);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 1ba94ffaff9d..52fc09f905c3 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -174,6 +174,8 @@ struct cpuset {
 	int nr_migrate_dl_tasks;
 	/* DL bandwidth that needs destination reservation for this attach. */
 	u64 sum_migrate_dl_bw;
+	/* Of that bandwidth, what the attach moved: the rest is given back. */
+	u64 sum_moved_dl_bw;
 	/*
 	 * CPU used for temporary DL bandwidth allocation during attach;
 	 * -1 if no DL bandwidth was allocated in the current attach.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 5b8f1d9552a6..263db5c4c662 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3183,10 +3183,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
 		if (cs->nr_migrate_dl_tasks) {
 			if (!cancel)
 				atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
-			else if (cs->dl_bw_cpu >= 0) /* && cancel */
-				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+			/*
+			 * Give back the bandwidth the attach did not move:
+			 * all of the reservation when it is cancelled, and
+			 * what was set aside for the tasks that are no
+			 * longer deadline tasks otherwise. Theirs is
+			 * released by the class they left, from the root
+			 * domain that counts it.
+			 */
+			if (cs->dl_bw_cpu >= 0) {
+				u64 bw = cancel ? cs->sum_migrate_dl_bw :
+					 cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+
+				if (bw)
+					dl_bw_free(cs->dl_bw_cpu, bw);
+			}
 			cs->nr_migrate_dl_tasks = 0;
 			cs->sum_migrate_dl_bw = 0;
+			cs->sum_moved_dl_bw = 0;
 			cs->dl_bw_cpu = -1;
 		}
 	}
@@ -3338,19 +3352,28 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 			       subpartitions_cpus);
 	/*
 	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
-	 * deadline tasks cpuset_can_attach() counted: say so, so that this
-	 * task's bandwidth moves there with it.
+	 * deadline tasks cpuset_can_attach() counted: offer it, so that this
+	 * task's bandwidth moves there with it. A task that became a
+	 * deadline task since has nothing set aside, so it moves its
+	 * bandwidth only where the rest of the reservation covers it.
 	 */
-	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus) &&
+	    cs->sum_moved_dl_bw + task->dl.dl_bw <= cs->sum_migrate_dl_bw)
 		reserved = true;
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	if (reserved)
-		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
-	else
+	if (reserved) {
+		u64 moved;
+
+		/* The move decides: the tally counts what it moved. */
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
+							  &moved));
+		cs->sum_moved_dl_bw += moved;
+	} else {
 		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	}
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0ff07258cf0c..120128032c15 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3271,16 +3271,21 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
 /*
  * The same, for a caller that has made room in the root domain of @new_mask
- * for the deadline bandwidth of @p: that bandwidth moves there with it.
+ * for the deadline bandwidth of @p: that bandwidth moves there with it, and
+ * @moved takes what the move counts there, 0 where it moved none.
  */
-int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+int set_cpus_allowed_bw_reserved(struct task_struct *p,
+				 const struct cpumask *new_mask, u64 *moved)
 {
 	struct affinity_context ac = {
 		.new_mask  = new_mask,
 		.flags     = SCA_DL_BW_RESERVED,
 	};
+	int ret;
 
-	return __set_cpus_allowed_ptr(p, &ac);
+	ret = __set_cpus_allowed_ptr(p, &ac);
+	*moved = ac.dl_bw_moved;
+	return ret;
 }
 
 /*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 7a58e78ac5ce..029ec2ba5a66 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3512,7 +3512,9 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * The bandwidth of a task whose move no one made room for stays
 	 * where it is counted, for the next rebuild of the accounting to
 	 * attribute, rather than leaving the root domain it moves to short
-	 * of what its tasks hold.
+	 * of what its tasks hold. @ctx tells the caller what moved: the
+	 * record of a task is read here, under its runqueue lock, and the
+	 * caller cannot keep a reservation for a move this decides against.
 	 */
 	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
 	    dl_task_needs_bw_move(p, ctx->new_mask)) {
@@ -3540,6 +3542,7 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
 		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
 		p->dl.dl_bw_gen = dl_bw_gen;
+		ctx->dl_bw_moved = p->dl.dl_bw;
 	}
 
 	set_cpus_allowed_common(p, ctx);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3e5bc8b6f887..0d5908f58594 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2659,6 +2659,8 @@ extern const u32		sched_prio_to_wmult[40];
 struct affinity_context {
 	const struct cpumask	*new_mask;
 	struct cpumask		*user_mask;
+	/* Out: the deadline bandwidth the move counts in the new root domain. */
+	u64			dl_bw_moved;
 	unsigned int		flags;
 };
 
-- 
2.47.3

[PATCH v2 18/21] sched/deadline: Credit a rebuild to a CPU that can hold the bandwidth

[Mathieu Desnoyers]

A rebuild of the root domains' accounting takes a CPU of the task's
cpuset for each deadline task it finds, dl_add_task_root_domain()
through dl_get_task_effective_cpus(), and two things can go wrong with
the CPUs isolated by isolcpu="domain", which belong to the default root
domain:

 - the cpuset of a task that may run on a housekeeping CPU can include
   isolated CPUs beside it, and without cpusets the mask is every CPU
   the task can run on. The first active CPU of it is then an isolated
   one where it comes first, and the task's bandwidth is counted in the
   default root domain instead of the one it belongs to. The root
   domain it belongs to underflows when the task leaves the class, or
   dies;
 - a task confined to isolated CPUs has them all taken offline, which
   nothing refuses: a deadline task of the default root domain does not
   keep the last of its CPUs online, dl_bw_deactivate() asking the root
   domain of the CPU going offline. The rebuild then finds no active
   CPU for it and BUG()s.

Leave the housekeeping CPUs of a task that has them, and leave a task
with no active CPU out of the rebuild: the generation of
"sched/deadline: Keep departing bandwidth across a root domain rebuild"
keeps the release of its bandwidth out as well, so that nothing is taken
from a root domain that no longer counts it, and the next rebuild
counts it again where it can.

Found by the Sashiko AI review of the first posting of this series.

Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 029ec2ba5a66..c0af06d0f368 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3617,6 +3617,16 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	 * hierarchy up to the partition root.
 	 */
 	cpuset_cpus_allowed_locked(p, cpus);
+	/*
+	 * Which the CPUs isolated by isolcpu="domain" are not part of: a
+	 * cpuset can include them beside housekeeping CPUs, and without
+	 * cpusets the mask is every CPU the task can run on. The first
+	 * active CPU of it would then be an isolated one, of the default
+	 * root domain, rather than one of the root domain the task's
+	 * bandwidth belongs to.
+	 */
+	if (housekeeping_enabled(HK_TYPE_DOMAIN))
+		cpumask_and(cpus, cpus, hk_msk);
 }
 
 /* The caller should hold cpuset_mutex */
@@ -3637,7 +3647,18 @@ void dl_add_task_root_domain(struct task_struct *p)
 	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
 	dl_get_task_effective_cpus(p, msk);
 	cpu = cpumask_first_and(cpu_active_mask, msk);
-	BUG_ON(cpu >= nr_cpu_ids);
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * No active CPU to count this task's bandwidth: the ones it
+		 * may run on all went offline, which the CPUs isolated by
+		 * isolcpu="domain" allow, no deadline task of theirs keeping
+		 * the last of them online. Leave it out of this rebuild,
+		 * where dl_bw_gen keeps its release out as well, until one
+		 * has a CPU for it.
+		 */
+		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+		return;
+	}
 
 	raw_spin_lock(&dl_departed_lock);
 	/*
-- 
2.47.3

[PATCH v2 19/21] sched: Clear cpuset_locked where the policy check retries

[Mathieu Desnoyers]

The retry of the policy check in __sched_setscheduler() drops
cpuset_mutex and leaves cpuset_locked saying it is held:

	if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
		policy = oldpolicy = -1;
		task_rq_unlock(rq, p, &rf);
		if (cpuset_locked) {
			dl_accounting_unlock();
			cpuset_unlock();
		}
		goto recheck;
	}

sched_setparam(), and sched_setattr() with SCHED_FLAG_KEEP_POLICY, take
the policy they check from the task with no lock held, and take
cpuset_mutex for a task of SCHED_DEADLINE. A switch of that task out of
the class meanwhile makes the check retry; the second pass finds no
deadline policy to take the lock for, and the exit path unlocks a
cpuset_mutex that belongs to whoever holds it by then.
dl_accounting_mutex, added earlier in this series, goes the same way.
CONFIG_DEBUG_MUTEXES reports the unlock of a mutex the task does not
own.

Clear the flag with the lock.

Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b1fc4fadd689..d541f4fc7252 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -654,6 +654,7 @@ int __sched_setscheduler(struct task_struct *p,
 		if (cpuset_locked) {
 			dl_accounting_unlock();
 			cpuset_unlock();
+			cpuset_locked = false;
 		}
 		goto recheck;
 	}
-- 
2.47.3

[PATCH v2 20/21] sched/deadline: Cancel the replenishment timer with the parameters

[Mathieu Desnoyers]

The replenishment timer of a throttled task is armed for its next
period, which can come after its 0-lag time: a task that yielded keeps
runtime, and a constrained-deadline task that overruns by little
reaches its 0-lag time within the same period. Leaving SCHED_DEADLINE,
such a task has its parameters cleared at the 0-lag time with that
timer still armed.

Back in the class before it fires, the task throttles again, and
start_dl_timer() leaves the armed timer where it is:

	if (!hrtimer_is_queued(timer)) {
		if (!dl_server(dl_se))
			get_task_struct(dl_task_of(dl_se));
		hrtimer_start(timer, act, HRTIMER_MODE_ABS_HARD);
	}

so the replenishment comes at the period of the reservation the task
left: too early, which hands it runtime it has not earned, or up to a
period too late.

Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole") left
that timer armed on purpose, as what reset a task that stayed out of
the class. The 0-lag time took that over and left the timer behind:
cancel it where the parameters go.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c0af06d0f368..15d224846eba 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -563,6 +563,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				if (READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
 				dl_task_release_bw(p);
+				cancel_replenish_timer(dl_se);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -2326,6 +2327,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		}
 
 		dl_task_release_bw(p);
+		cancel_replenish_timer(dl_se);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
-- 
2.47.3

[PATCH v2 21/21] sched/deadline: Count the bandwidth a boosted task is given

[Mathieu Desnoyers]

The runqueue a sleeping deadline task is on counts the bandwidth of
that task in this_bw, which switched_to_dl() hands it as the task
enters the class. A task that priority inheritance boosted into the
class is in it already: __sched_setscheduler() neither dequeues it nor
enqueues it back, so switched_to_dl() does not run, and sched_setattr()
of SCHED_DEADLINE on such a task leaves this_bw without the bandwidth
the task now has. Its wake-up gives that bandwidth to running_bw alone:

	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);

and the two sums GRUB reclaims from disagree by the bandwidth of the
task until it migrates, where the enqueue adds it (add_rq_bw()).

Change the bandwidth the runqueue counts where the parameters change,
in __setparam_dl(), which every task that gains deadline parameters
goes through, rather than in sched_dl_overflow(), which only a task of
the class reaches that way. A runqueue that counts nothing of the task
takes the new bandwidth, where the previous patch left it to the
enqueue to come: that enqueue finds the bandwidth there and leaves it
(add_rq_bw() of a task a runqueue counts), and a task waking up has
task_cpu() pointing at the runqueue it wakes on. The bandwidth of a
special entity stays counted nowhere.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 37 +++++++++++++++++++++++++++++--------
 1 file changed, 29 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 15d224846eba..a3b45bb4f35e 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -442,22 +442,32 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+/*
+ * The runqueue a sleeping task is on counts the bandwidth the task has:
+ * give it @new_bw in place of what it counts of the task's.
+ */
 static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 {
-	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct rq *rq = task_rq(p);
 
-	if (task_on_rq_queued(p))
+	if (task_on_rq_queued(p) || dl_entity_is_special(dl_se))
 		return;
 
 	/*
-	 * Waking up on another CPU, the task has its bandwidth on no
-	 * runqueue: the enqueue to come charges the new one, with the new
-	 * parameters, which __setparam_dl() sets right after this.
+	 * No runqueue counts bandwidth of the task's: this one takes the
+	 * new one, and the enqueue to come leaves it there (add_rq_bw() of
+	 * a task a runqueue counts). A task waking up on another CPU has
+	 * none of its bandwidth on a runqueue, and task_cpu() points at the
+	 * one it wakes on.
 	 */
-	if (!p->dl.dl_bw_oncpu)
+	if (!dl_se->dl_bw_oncpu) {
+		__add_rq_bw(new_bw, &rq->dl);
+		dl_se->dl_bw_oncpu = 1;
 		return;
+	}
 
-	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+	dl_rq_change_utilization(rq, dl_se, new_bw);
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4217,7 +4227,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		if (counted)
 			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
-		dl_change_utilization(p, new_bw);
 		err = 0;
 	} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
 		/*
@@ -4266,6 +4275,18 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	u64 period = attr->sched_period ?: attr->sched_deadline;
+
+	/*
+	 * The bandwidth of a special entity is counted on no runqueue.
+	 * The rest have theirs changed here, where their parameters are:
+	 * switched_to_dl() hands it to the runqueue of a task that enters
+	 * the class, and a task that priority inheritance boosted into the
+	 * class is in it already, so nothing dequeues it and enqueues it
+	 * back for that.
+	 */
+	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
 
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
-- 
2.47.3

Re: [PATCH v2 04/21] sched/deadline: Do not divide by zero active CPUs of a root domain

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] sched/deadline: dl_b->total_bw underflow in __dl_sub() and __dl_add()
--

commit 1283d9f058013be0f43eec132085929ecd9c1f99
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Do not divide by zero active CPUs of a root domain

This commit guards the calls to __dl_update() in __dl_sub() and __dl_add()
with a check that cpus is non-zero to avoid dividing by zero when all CPUs in
a root domain are inactive. When cpus is zero, it leaves the dl_b->total_bw
updates in place without updating per-CPU extra_bw.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 963fda6ddb7f6..dadda1a528ec6 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
>  	}
>  }
>  
> +/*
> + * @cpus is the number of active CPUs of the root domain, whose extra_bw
> + * shares the change: none once they all went offline, and total_bw alone
> + * changes then.
> + */
>  static inline
>  void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
>  {
>  	dl_b->total_bw -= tsk_bw;
> -	__dl_update(dl_b, (s32)tsk_bw / cpus);
> +	if (cpus)
> +		__dl_update(dl_b, (s32)tsk_bw / cpus);
>  }
>  
>  static inline
>  void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
>  {
>  	dl_b->total_bw += tsk_bw;
> -	__dl_update(dl_b, -((s32)tsk_bw / cpus));
> +	if (cpus)
> +		__dl_update(dl_b, -((s32)tsk_bw / cpus));
>  }

[Severity: Medium]
Can updating dl_b->total_bw when cpus == 0 (or on an inactive CPU) underflow
or corrupt dl_b->total_bw in the paths mentioned in the commit message?

During a root-domain rebuild in dl_rebuild_rd_accounting(),
dl_clear_root_domain() resets rd->dl_bw.total_bw to 0 and dl_server_add_bw()
only adds server bandwidth back for active CPUs:

kernel/sched/deadline.c:dl_clear_root_domain() {
	rd->dl_bw.total_bw = 0;
	for_each_cpu(i, rd->span)
		cpu_rq(i)->dl.extra_bw = cpu_rq(i)->dl.max_bw;

	for_each_cpu(i, rd->span)
		dl_server_add_bw(rd, i);
}

kernel/sched/deadline.c:dl_server_add_bw() {
	dl_se = &cpu_rq(cpu)->fair_server;
	if (dl_server(dl_se) && dl_se->dl_bw_attached && cpu_active(cpu))
		__dl_add(&rd->dl_bw, dl_se->dl_bw, dl_bw_cpus(cpu));
}

Likewise, __dl_server_attach_bw_locked() and __dl_server_detach_bw_locked()
only update dl_b->total_bw when cpu_active(cpu_of(rq)) is true, because an
inactive CPU's server bandwidth is not accounted in dl_b->total_bw.

In dl_server_apply_params(), however, dl_b is updated unconditionally when
dl_se->dl_bw_attached is set:

kernel/sched/deadline.c:dl_server_apply_params() {
	if (__dl_overflow(dl_b, cap, old_bw, new_bw))
		return -EBUSY;

	if (init) {
		__add_rq_bw(new_bw, &rq->dl);
		__dl_add(dl_b, new_bw, cpus);
		dl_se->dl_bw_attached = 1;
	} else if (dl_se->dl_bw_attached) {
		__dl_sub(dl_b, dl_se->dl_bw, cpus);
		__dl_add(dl_b, new_bw, cpus);

		dl_rq_change_utilization(rq, dl_se, new_bw);
	}
}

While a CPU sits in def_root_domain with !cpu_active(cpu) (between
sched_cpu_deactivate() and CPU offline, or between CPU online and
sched_cpu_activate()), its fair_server bandwidth is not in
def_root_domain.dl_bw.total_bw. If admission control is disabled
(dl_b->bw == -1) and a debugfs write sets new_bw < dl_se->dl_bw,
__dl_overflow() passes and __dl_sub() followed by __dl_add() subtracts the
unaccounted dl_se->dl_bw from 0, underflowing def_root_domain.dl_bw.total_bw.
Even with admission control enabled, if def_root_domain also contains active
isolated CPUs (cpus > 0), dl_server_apply_params() on the inactive
hotplugging CPU will subtract unaccounted dl_se->dl_bw and add new_bw to
def_root_domain.dl_bw.total_bw and the active CPUs' extra_bw. Should
dl_server_apply_params() gate the __dl_overflow(), __dl_sub(), and __dl_add()
calls on cpu_active(cpu), matching __dl_server_attach_bw_locked() and
__dl_server_detach_bw_locked()?

For the task paths mentioned in the commit message, at this commit:

1. In inactive_task_timer() (and task_non_contending()), if a task leaves
SCHED_DEADLINE or dies before its 0-lag time and task_cpu(p) goes offline
before the timer fires, the root-domain rebuild has already reset
def_root_domain.dl_bw.total_bw to 0 without re-adding the departed task:

kernel/sched/deadline.c:inactive_task_timer() {
	if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
		struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
		...
		raw_spin_lock(&dl_b->lock);
		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
		raw_spin_unlock(&dl_b->lock);
}

Calling __dl_sub() with cpus == 0 then subtracts p->dl.dl_bw from 0 and
underflows def_root_domain.dl_bw.total_bw. (A later commit in this series,
commit 1e8adcf8ed27 ("sched/deadline: Keep departing bandwidth across a root
domain rebuild"), addresses the rebuild race via dl_departed_list and
dl_bw_gen.)

2. With admission control off, if sched_setattr() switches a task asleep on
an offline CPU to SCHED_DEADLINE, sched_dl_overflow() calls __dl_add() on
def_root_domain (where cpus == 0), adding new_bw to
def_root_domain.dl_bw.total_bw. Subsequent removal in switched_from_dl() or
set_cpus_allowed_dl() uses dl_task_root_domain_cpu(p) in an active root
domain, leaving def_root_domain.dl_bw.total_bw elevated and underflowing the
active root domain's total_bw. (A later commit in this series, commit
90cd336704a1 ("sched/deadline: Record the root domain holding a task's
bandwidth"), records dl_bw_cpu on the entity.)

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=4

Re: [PATCH v2 06/21] sched/deadline: Do not lose a throttled task switched out and back

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched/deadline: replenish_dl_entity() in dl_task_timer() skips 0-lag reset
--

commit a554b85972acb72b5ec447330ac07adc4a64ee34
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Do not lose a throttled task switched out and back

This commit calls replenish_dl_entity() from dl_task_timer() when a throttled
task has switched out of SCHED_DEADLINE before the replenishment timer fires,
so that if the task switches back before its 0-lag time, it does not remain
throttled with no pending timer.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 30a52e42c26d8..b646063a7b0d5 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -1232,10 +1232,23 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
>  
>  	/*
>  	 * The task might have changed its scheduling policy to something
> -	 * different than SCHED_DEADLINE (through switched_from_dl()).
> +	 * different than SCHED_DEADLINE (through switched_from_dl()). It
> +	 * keeps its parameters until its 0-lag time and may come back to
> +	 * SCHED_DEADLINE before then, switched_to_dl() cancelling the
> +	 * inactive timer: give it the replenishment this timer owes it, as
> +	 * the !queued case below does, and leave the enqueue to its return.
> +	 * A throttle left behind would never be lifted; an overrun left in
> +	 * its runtime would be forgiven by the new period its enqueue
> +	 * starts; and a yield left set would throttle it again at once.
>  	 */
> -	if (!dl_task(p))
> +	if (!dl_task(p)) {
> +		if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
> +			sched_clock_tick();
> +			update_rq_clock(rq);
> +			replenish_dl_entity(dl_se);
> +		}
>  		goto unlock;
> +	}

[Severity: High]
What happens if the task stays out of SCHED_DEADLINE past its 0-lag time, and
then switches back to SCHED_DEADLINE while runnable before the replenished
dl_se->deadline expires?

When replenish_dl_entity() runs here in dl_task_timer() at the end of the
period, it advances dl_se->deadline by one or more periods (past the 0-lag
time) and sets a positive dl_se->runtime:

kernel/sched/deadline.c:replenish_dl_entity() {
	...
	while (dl_se->runtime <= 0) {
		dl_se->deadline += pi_of(dl_se)->dl_period;
		dl_se->runtime += pi_of(dl_se)->dl_runtime;
	}
	...
}

If the task does not return before the 0-lag time, inactive_task_timer()
fires, subtracts the old bandwidth from dl_b, and calls __dl_clear_params():

kernel/sched/deadline.c:inactive_task_timer() {
	...
	raw_spin_lock(&dl_b->lock);
	__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
	raw_spin_unlock(&dl_b->lock);
	__dl_clear_params(dl_se);
	...
}

However, __dl_clear_params() only zeroes the static reservation parameters and
flags; it leaves the dynamic dl_se->deadline and dl_se->runtime untouched:

kernel/sched/deadline.c:__dl_clear_params() {
	dl_se->dl_runtime		= 0;
	dl_se->dl_deadline		= 0;
	dl_se->dl_period		= 0;
	dl_se->flags			= 0;
	dl_se->dl_bw			= 0;
	dl_se->dl_density		= 0;
	...
}

If the runnable task then switches back to SCHED_DEADLINE via sched_setattr()
after the 0-lag time has passed, but before the replenished dl_se->deadline,
__setparam_dl() writes the new static parameters and sched_change_end() calls
enqueue_task_dl() -> enqueue_dl_entity() with ENQUEUE_MOVE (and without
ENQUEUE_WAKEUP):

kernel/sched/deadline.c:enqueue_dl_entity() {
	...
	} else if ((flags & ENQUEUE_MOVE) &&
		   !is_dl_boosted(dl_se) &&
		   dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
		setup_new_dl_entity(dl_se);
	}
	...
}

Previously, dl_se->deadline stayed at the old deadline (which is earlier than
the 0-lag time), so dl_time_before(dl_se->deadline, rq_clock(rq)) was always
true after the 0-lag time and setup_new_dl_entity() initialized a fresh period
from the new static parameters.

With dl_se->deadline pushed past the 0-lag time by replenish_dl_entity(),
won't dl_time_before(dl_se->deadline, rq_clock(rq)) evaluate to false during
the window between the 0-lag time and the replenished dl_se->deadline,
skipping setup_new_dl_entity()?

If setup_new_dl_entity() is skipped after the 0-lag time has already freed the
old bandwidth and reset the reservation, the task will run with stale dynamic
dl_se->runtime and dl_se->deadline from its previous reservation. For example,
a task could re-enter SCHED_DEADLINE with a tiny new runtime reservation after
its old bandwidth was released at 0-lag, yet still consume the large
dl_se->runtime replenished from its previous reservation, or conversely remain
penalized by a previous overrun after its 0-lag time has already elapsed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=6

Re: [PATCH v2 07/21] sched/deadline: Do not release a departing sleeper's bandwidth twice

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched/deadline: switched_from_dl() leaves stale params after clearing dl_bw
--

commit 3a2d0df92966c3d09add14a543e55863d8af1e1e
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Do not release a departing sleeper's bandwidth twice

This commit updates switched_from_dl() to only release total_bw when a
departing sleeping task has its own deadline parameters and is not a
special entity, and clears p->dl.dl_bw afterward so that subsequent paths
do not subtract its bandwidth a second time.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index b646063a7b0d5..624cabdec9652 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3573,6 +3573,8 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  	dec_dl_tasks_cs(p);
>  
>  	if (!task_on_rq_queued(p)) {
> +		bool released = false;
> +
>  		if (p->dl.dl_non_contending) {
>  			/*
>  			 * Inactive timer is armed. However, p is leaving
> @@ -3585,12 +3587,18 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  			 * release total_bw via __dl_sub().
>  			 */
>  			sub_running_bw(&p->dl, &rq->dl);
> -		} else {
> +		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
>  			/*
>  			 * No pending inactive timer: the task never ran as
>  			 * DEADLINE or the timer already fired while still
>  			 * DEADLINE (only dropping running_bw at that time).
>  			 * Nobody else will release total_bw, do it now.
> +			 *
> +			 * Not for a task without parameters of its own, one
> +			 * deboosted from priority inheritance, whose mask may
> +			 * have no active CPU for dl_task_root_domain_cpu(),
> +			 * nor for a special one: neither has bandwidth in
> +			 * total_bw.
>  			 */
>  			unsigned int cpu = dl_task_root_domain_cpu(p);
>  			struct dl_bw *dl_b = &cpu_rq(cpu)->rd->dl_bw;
> @@ -3598,8 +3606,29 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  			raw_spin_lock(&dl_b->lock);
>  			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
>  			raw_spin_unlock(&dl_b->lock);
> +			released = true;
>  		}
>  		sub_rq_bw(&p->dl, &rq->dl);
> +		/*
> +		 * Released: its bandwidth is no one's to release again, and
> +		 * a dl_bw of 0 is what every path that would takes. Kept, it
> +		 * would make the task one with bandwidth to task_contending()
> +		 * and task_non_contending() once priority inheritance boosts
> +		 * it, and a deboost while it sleeps would release it a second
> +		 * time. So would the inactive timer's handler when it runs
> +		 * late, dl_non_contending cleared under it by
> +		 * migrate_task_rq_dl(): it finds !dl_task(p).
> +		 *
> +		 * The parameters themselves stay, where
> +		 * inactive_task_timer() resets them with the bandwidth: the
> +		 * task may still be the donor of a priority inheritance
> +		 * boost until rt_mutex_adjust_pi() walks the chain, outside
> +		 * this runqueue lock, and replenish_dl_entity() of the
> +		 * boosted owner would never leave its loop over a dl_period
> +		 * and a dl_runtime of 0.
> +		 */
> +		if (released)
> +			p->dl.dl_bw = 0;

[Severity: High]
When switched_from_dl() takes the !p->dl.dl_non_contending branch, there is
no pending inactive timer to run inactive_task_timer() and call
__dl_clear_params(&p->dl) later. Does clearing only p->dl.dl_bw here leave
the departed task permanently carrying stale deadline parameters
(dl_runtime, dl_deadline, dl_period, dl_density, flags, runtime, and
deadline) after it switches to another policy such as SCHED_OTHER?

Several paths in kernel/sched/deadline.c check dl_deadline, dl_runtime, or
flags rather than dl_bw to distinguish a PI-boosted non-deadline task from
a task with its own deadline reservation:

1. In replenish_dl_entity(), initial replenishment for a PI-boosted task
checks dl_se->dl_deadline == 0:

kernel/sched/deadline.c:replenish_dl_entity() {
	if (dl_se->dl_deadline == 0 ||
	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
		dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
		dl_se->runtime = pi_of(dl_se)->dl_runtime;
	}
}

If the departed task is later PI-boosted by a SCHED_DEADLINE waiter while
holding an rt_mutex, dl_se->dl_deadline is still non-zero. If its stale
p->dl.runtime is still positive and p->dl.deadline is still in the future,
replenish_dl_entity() skips both the while (dl_se->runtime <= 0) loop and
the dl_time_before(dl_se->deadline, rq_clock(rq)) check, so the boosted
task runs with its own stale runtime and deadline instead of inheriting the
donor's parameters. Conversely, if the task never ran as SCHED_DEADLINE
(p->dl.deadline == 0, p->dl.runtime == 0) or its stale deadline is in the
past, skipping the dl_se->dl_deadline == 0 initialization causes the loop
to advance from the stale deadline and trigger a false replenishment lag
warning.

2. If the task previously had SCHED_FLAG_RECLAIM set in p->dl.flags,
retaining p->dl.flags while p->dl.dl_bw == 0 causes dl_scaled_delta_exec()
to call grub_reclaim() when the task is later PI-boosted:

kernel/sched/deadline.c:grub_reclaim() {
	if (u_inact + rq->dl.extra_bw > rq->dl.max_bw - dl_se->dl_bw)
		u_act = dl_se->dl_bw;
	else
		u_act = rq->dl.max_bw - u_inact - rq->dl.extra_bw;
}

Whenever u_inact + rq->dl.extra_bw >= rq->dl.max_bw, u_act becomes
dl_se->dl_bw (0), scaling execution time to 0 so the boosted task's runtime
never depletes. Stale SCHED_FLAG_DL_OVERRUN in p->dl.flags can likewise
deliver SIGXCPU to the boosted non-deadline task.

3. Both task_non_contending() and task_contending() guard against
non-deadline boosted tasks by checking dl_se->dl_runtime == 0:

kernel/sched/deadline.c:task_non_contending() {
	if (dl_se->dl_runtime == 0)
		return;
}

Because p->dl.dl_runtime remains non-zero, blocking while PI-boosted (or
deboosting while runnable) runs task_non_contending() using the stale
parameters and sets p->dl.dl_non_contending = 1 with inactive_timer armed
Even though p->dl.dl_bw is 0 at that moment, if the task then re-enters
SCHED_DEADLINE via sched_setattr() while sleeping and PI-boosted
(prev_class == next_class == &dl_sched_class), __setparam_dl() assigns a
non-zero p->dl.dl_bw while dl_non_contending is still 1 without having
added the new bandwidth to rq->dl.running_bw, causing inactive_task_timer()
or migrate_task_rq_dl() to underflow rq->dl.running_bw.

4. If the departed task wakes up in SCHED_OTHER and later switches back to
SCHED_DEADLINE while runnable before its old p->dl.deadline passes,
enqueue_dl_entity() checks dl_time_before(dl_se->deadline, rq_clock(rq)):

kernel/sched/deadline.c:enqueue_dl_entity() {
	} else if ((flags & ENQUEUE_MOVE) &&
		   !is_dl_boosted(dl_se) &&
		   dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
		setup_new_dl_entity(dl_se);
	}
}

Because the stale p->dl.deadline is still in the future,
setup_new_dl_entity() is skipped and the task reuses its old unspent
runtime.

Should switched_from_dl() clear the remaining deadline parameters (or reset
them once any PI donor relationship is adjusted) instead of leaving them
populated when no inactive timer is pending?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=7

Re: [PATCH v2 09/21] sched/deadline: Keep departing bandwidth across a root domain rebuild

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched/deadline: departing bandwidth accounting corrupted in switched_to_dl() on PI boost
--

commit 1e8adcf8ed2712137f083c0716509491291e2298
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Keep departing bandwidth across a root domain rebuild

This commit tracks tasks that leave SCHED_DEADLINE or exit before their
0-lag time on dl_departed_list along with the CPU of their root domain and a
rebuild generation counter. This preserves their departing bandwidth across
dl_rebuild_rd_accounting() and ensures inactive_task_timer() removes the
bandwidth from the original root domain even if the task migrates.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 624cabdec9652..2729c108b24bc 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c

[ ... ]

> @@ -3485,12 +3575,33 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	dl_get_task_effective_cpus(p, msk);
>  	cpu = cpumask_first_and(cpu_active_mask, msk);
>  	BUG_ON(cpu >= nr_cpu_ids);
> +
> +	raw_spin_lock(&dl_departed_lock);
> +	/*
> +	 * Counted already by this rebuild: put back by
> +	 * dl_clear_root_domains() as a departed task, which priority
> +	 * inheritance boosted into the class since, or its bandwidth
> +	 * gone since (dl_task_release_bw()), as that of one that died
> +	 * after the walk found it. One that died since with its 0-lag
> +	 * time ahead goes back to the root domain of the CPU recorded
> +	 * (dl_task_depart()), if active.
> +	 */
> +	if (p->dl.dl_bw_gen == dl_bw_gen)
> +		goto unlock;
> +	if (p->dl.dl_departed_cpu >= 0) {
> +		cpu = p->dl.dl_departed_cpu;
> +		if (!cpu_active(cpu))
> +			goto unlock;
> +	}
>  	rq = cpu_rq(cpu);
>  	dl_b = &rq->rd->dl_bw;
>  
>  	raw_spin_lock(&dl_b->lock);
>  	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
>  	raw_spin_unlock(&dl_b->lock);
> +	p->dl.dl_bw_gen = dl_bw_gen;
> +unlock:
> +	raw_spin_unlock(&dl_departed_lock);
>  	raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
>  }

[ ... ]

> @@ -3584,7 +3731,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  			 * or sub_rq_bw (below) will complain.
>  			 *
>  			 * inactive_task_timer() will find !dl_task(p) and
> -			 * release total_bw via __dl_sub().
> +			 * release total_bw (dl_task_release_bw()).
>  			 */
>  			sub_running_bw(&p->dl, &rq->dl);
>  		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
> @@ -3600,12 +3747,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  			 * nor for a special one: neither has bandwidth in
>  			 * total_bw.
>  			 */
> -			unsigned int cpu = dl_task_root_domain_cpu(p);
> -			struct dl_bw *dl_b = &cpu_rq(cpu)->rd->dl_bw;
> -
> -			raw_spin_lock(&dl_b->lock);
> -			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
> -			raw_spin_unlock(&dl_b->lock);
> +			dl_task_release_bw(p, dl_task_root_domain_cpu(p));
>  			released = true;
>  		}
>  		sub_rq_bw(&p->dl, &rq->dl);

[ ... ]

> @@ -3657,6 +3809,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  static void switched_to_dl(struct rq *rq, struct task_struct *p)
>  {
>  	cancel_inactive_timer(&p->dl);
> +	/*
> +	 * Boosted back into the class before its 0-lag time (a policy change
> +	 * released that bandwidth in sched_dl_overflow()): the inactive
> +	 * timer will not remove it, but its leaving the class again. That
> +	 * forgets the CPU recorded: moved to another root domain meanwhile,
> +	 * the task then removes the bandwidth from that one, as it did
> +	 * before departures were recorded.
> +	 */
> +	if (p->dl.dl_departed_cpu >= 0)
> +		dl_task_undepart(p);

[ ... ]

> @@ -3924,6 +4078,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	}
>  	raw_spin_unlock(&dl_b->lock);
>  
> +	/*
> +	 * Back before its 0-lag time, the task still has the bandwidth it
> +	 * left with, which the new one replaces: removed from the root
> +	 * domain that keeps it (dl_task_release_bw()). A rebuild cannot
> +	 * run meanwhile: the caller holds cpuset_mutex.
> +	 */
> +	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
> +		lockdep_assert_cpuset_lock_held();
> +		if (p->dl.dl_departed_cpu >= 0)
> +			dl_task_release_bw(p, -1);
> +		p->dl.dl_bw_gen = dl_bw_gen;
> +	}

[Severity: High]
As reported by Sashiko on v1 [1], can priority-inheritance boosting of a
departed task before its 0-lag time corrupt root-domain total_bw accounting?

When a SCHED_DEADLINE task p switches to a non-deadline policy such as
SCHED_OTHER before its 0-lag time, switched_from_dl() arms inactive_timer,
clears p->dl.dl_non_contending, keeps p->dl.dl_runtime and p->dl.dl_bw, and
calls dl_task_depart() to record the CPU of its original root domain (RD_A)
and link p onto dl_departed_list. If p holds an rt_mutex and is boosted into
dl_sched_class via rt_mutex_setprio() before its 0-lag time,
switched_to_dl() calls cancel_inactive_timer(&p->dl) and
dl_task_undepart(p), unlinking p from dl_departed_list and resetting
p->dl.dl_departed_cpu to -1 in this commit.

In this commit, resetting p->dl.dl_departed_cpu to -1 causes
sched_dl_overflow() to skip dl_task_release_bw() if p is setscheduled back
to SCHED_DEADLINE while boosted, leaking the old p->dl.dl_bw in RD_A, and
causes switched_from_dl() on deboost after a migration to RD_B to subtract
p->dl.dl_bw from RD_B instead of RD_A.

Later commits in the series ("sched/deadline: Record the root domain holding
a task's bandwidth" and "sched/deadline: Count the bandwidth a boosted task
is given") replace dl_departed_cpu with dl_bw_cpu (preserving dl_bw_cpu
across dl_task_undepart()) and update rq->dl.this_bw in __setparam_dl().
However, even with those later commits applied, don't the following three
scenarios still corrupt bandwidth accounting when a departed task is
PI-boosted before its 0-lag time?

1. If p migrates to another cpuset partition / root domain (RD_B) while in
SCHED_OTHER (!dl_task(p)), cpuset_can_attach() does not reserve or move
p->dl.dl_bw into RD_B, so p->dl.dl_bw remains in RD_A. If p is then
PI-boosted in RD_B, switched_to_dl() calls dl_task_undepart(p) and unlinks
p->dl.dl_departed_node from dl_departed_list. If dl_rebuild_rd_accounting()
runs while p is PI-boosted, dl_clear_root_domains() does not see p on
dl_departed_list, and dl_add_task_root_domain(p) sees that p is not on
dl_departed_list (list_empty(&p->dl.dl_departed_node), or
p->dl.dl_departed_cpu < 0 here). Doesn't dl_add_task_root_domain(p) then
ignore the recorded CPU in RD_A and add p->dl.dl_bw to RD_B (via
dl_get_task_effective_cpus()), dropping p's departing bandwidth from RD_A
before its 0-lag time and charging RD_B without admission control?

2. Because rt_mutex_setprio() only holds p->pi_lock and rq->__lock (not
cpuset_mutex or dl_accounting_mutex), can a PI deboost race with a
concurrent dl_rebuild_rd_accounting()? If p is PI-boosted when
dl_clear_root_domains() runs, p is not on dl_departed_list, so
dl_clear_root_domains() clears the root domains without restoring
p->dl.dl_bw. If rt_mutex_setprio() then deboosts p before
dl_add_task_root_domain(p) inspects it, switched_from_dl() re-adds p to
dl_departed_list after dl_clear_root_domains() has already finished, and
dl_add_task_root_domain(p) sees !dl_task(p) and returns early, dropping p's
departing bandwidth before its 0-lag time.

3. If p is sleeping (!task_on_rq_queued(p)) after leaving SCHED_DEADLINE
with inactive_timer armed (so p->dl.dl_non_contending is 0), and a
SCHED_DEADLINE waiter blocks on an rt_mutex held by p, switched_to_dl()
cancels inactive_timer and calls dl_task_undepart(p). If that waiter times
out or is interrupted while p is still sleeping, rt_mutex_setprio() deboosts
p and calls switched_from_dl(). Since !task_on_rq_queued(p),
p->dl.dl_non_contending == 0, and p->dl.dl_runtime > 0, doesn't
switched_from_dl() enter the else-if branch and call dl_task_release_bw()
immediately, releasing p->dl.dl_bw before p's 0-lag time has elapsed?

[1] https://lore.kernel.org/r/9a2c58b84302e5ed3765f773eca3c71c7d5ea7bb.1793804635.git.mathieu.desnoyers@efficios.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=9

Re: [PATCH v2 13/21] sched/deadline: Rebuild the root domains' accounting without cpusets too

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched: lock inversion and leaked dl_accounting_mutex in __sched_setscheduler()
- [High] sched/deadline: late dl_accounting_mutex lock in dl_rebuild_rd_accounting()
- [High] sched/deadline: dl_rebuild_rd_accounting() walks dead and zombie tasks
--

commit 9c6ff828105f0cf8abfab2f4dbfe02756a888bc8
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Rebuild the root domains' accounting without cpusets too

This commit implements dl_rebuild_rd_accounting() and dl_accounting_mutex for
!CONFIG_CPUSETS kernels so that root domain SCHED_DEADLINE bandwidth
accounting is rebuilt across all tasks during CPU hotplug and suspend/resume.

[ ... ]

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index cbcc00ea46d31..85df2b0044bed 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3685,6 +3685,53 @@ void dl_clear_root_domain(struct root_domain *rd)
>  		dl_server_add_bw(rd, i);
>  }
>  
> +#ifndef CONFIG_CPUSETS
> +/*
> + * Where cpusets are built in, cpuset_mutex keeps an admission of bandwidth
> + * from running against a rebuild of the accounting, and
> + * __sched_setscheduler() takes it for every change of or into
> + * SCHED_DEADLINE. Where they are not, cpuset_lock() is a stub and this
> + * takes its place.
> + */
> +static DEFINE_MUTEX(dl_accounting_mutex);
> +
> +void dl_accounting_lock(void)
> +{
> +	mutex_lock(&dl_accounting_mutex);
> +}
> +
> +void dl_accounting_unlock(void)
> +{
> +	mutex_unlock(&dl_accounting_mutex);
> +}
> +
> +void lockdep_assert_dl_accounting_held(void)
> +{
> +	lockdep_assert_held(&dl_accounting_mutex);
> +}
> +
> +/*
> + * Rebuild the root domains' accounting, where no cpuset holds the
> + * deadline tasks of each: they are found by walking them all, which
> + * dl_rebuild_rd_accounting() of the cpusets does per cpuset instead.
> + */
> +void dl_rebuild_rd_accounting(void)
> +{
> +	struct task_struct *g, *p;
> +
> +	lockdep_assert_cpus_held();
> +	lockdep_assert_held(&sched_domains_mutex);
> +
> +	guard(mutex)(&dl_accounting_mutex);
> +	dl_clear_root_domains();

[Severity: High]
As reported by Sashiko on v1 [1], is acquiring dl_accounting_mutex inside
dl_rebuild_rd_accounting() too late to serialize against
__sched_setscheduler() -> sched_dl_overflow()?

Under CONFIG_CPUSETS, rebuild_sched_domains_cpuslocked() and
cpuset_reset_sched_domains() acquire cpuset_mutex before calling
partition_sched_domains(), holding it across the entire execution of
partition_sched_domains_locked().

Under !CONFIG_CPUSETS, partition_sched_domains_locked() runs
detach_destroy_domains() and build_sched_domains() -> cpu_attach_domain() ->
rq_attach_root() before calling dl_rebuild_rd_accounting():

kernel/sched/topology.c:partition_sched_domains_locked() {
    ...
    detach_destroy_domains(doms_cur);
    ...
    build_sched_domains(doms_new[i], ...);
    ...
    update_sched_domain_debugfs();
    dl_rebuild_rd_accounting();
}

At that point, newly allocated root_domain instances (whose dl_bw.total_bw
only holds dl_server bandwidth and zero task bandwidth) are already attached
to cpu_rq(cpu)->rd, while dl_bw_gen has not yet been incremented by
dl_clear_root_domains().

If __sched_setscheduler() runs on another CPU during that window, it can
acquire dl_accounting_mutex before dl_rebuild_rd_accounting() is reached and
call sched_dl_overflow() against the unpopulated new root domain:

- A task entering SCHED_DEADLINE checks __dl_overflow() against a new
  root_domain with zero task bandwidth, allowing total_bw to be overcommitted
  once dl_rebuild_rd_accounting() runs and adds both existing tasks and the
  newly admitted task.

- A departed task re-entering SCHED_DEADLINE before its 0-lag time calls
  dl_task_release_bw(p, -1) in sched_dl_overflow(); because dl_bw_gen has not
  been incremented yet, dl_task_release_bw() sees dl_se->dl_bw_gen ==
  dl_bw_gen and subtracts dl_se->dl_bw from dl_bw_of(dl_departed_cpu),
  underflowing the newly attached root_domain's total_bw.

- An existing SCHED_DEADLINE task reducing its bandwidth calls
  __dl_overflow() and __dl_sub() on the unpopulated new root_domain, either
  failing with -EBUSY on unsigned wrap or underflowing total_bw.

Should dl_accounting_mutex be held across partition_sched_domains_locked()
(or acquired by its callers) just like cpuset_mutex?

> +
> +	read_lock(&tasklist_lock);
> +	for_each_process_thread(g, p)
> +		dl_add_task_root_domain(p);
> +	read_unlock(&tasklist_lock);
> +}

[Severity: High]
Can walking all tasks with for_each_process_thread() in
dl_rebuild_rd_accounting() corrupt rd->dl_bw.total_bw when encountering dying
(TASK_DEAD) or unreaped zombie (EXIT_ZOMBIE) deadline tasks?

Unlike css_task_iter_next() in the CONFIG_CPUSETS implementation (which skips
dying tasks and exiting tasks with no live threads), for_each_process_thread()
visits tasks in do_task_dead() as well as unreaped EXIT_ZOMBIE processes until
they are reaped, while p->prio remains < MAX_DL_PRIO so dl_task(p) is true:

1. When a SCHED_DEADLINE task dies at or after its 0-lag time via
do_task_dead() -> __schedule() -> deactivate_task() -> dequeue_task_dl() ->
task_non_contending(), it runs under rq->__lock without holding p->pi_lock or
dl_accounting_mutex:

kernel/sched/deadline.c:task_non_contending() {
    ...
    if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
        if (READ_ONCE(p->__state) == TASK_DEAD)
            sub_rq_bw(dl_se, &rq->dl);
        dl_task_release_bw(p, task_cpu(p));
        __dl_clear_params(dl_se);
    }
    ...
}

If dl_rebuild_rd_accounting() runs on another CPU between
dl_task_release_bw() (which releases dl_departed_lock) and
__dl_clear_params() (which zeroes dl_se->dl_bw), dl_clear_root_domains()
increments dl_bw_gen and dl_add_task_root_domain(p) sees
p->dl.dl_bw_gen != dl_bw_gen with a non-zero p->dl.dl_bw, adding the dying
task's bandwidth back to dl_b->total_bw where it is permanently leaked.

2. If a task dies before its 0-lag time on CPU X (entering dl_departed_list
with dl_departed_cpu = X and dl_bw_gen = G) and remains an unreaped
EXIT_ZOMBIE while CPU X goes offline (rebuild G + 1), both
dl_clear_root_domains() and dl_add_task_root_domain() skip it because
!cpu_active(X), dropping its bandwidth with p->dl.dl_bw_gen left at G. If CPU
X comes back online (rebuild G + 2) before inactive_task_timer() fires,
dl_clear_root_domains() skips the entry because dl_se->dl_bw_gen !=
dl_bw_gen - 1, but for_each_process_thread() then passes the zombie to
dl_add_task_root_domain(p), which checks only p->dl.dl_bw_gen == dl_bw_gen
and cpu_active(cpu), adding the dropped bandwidth back.

[ ... ]

> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 8eddb9d47d8e7..b1fc4fadd6899 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -559,6 +559,7 @@ int __sched_setscheduler(struct task_struct *p,
>  	if (dl_policy(policy) || dl_policy(p->policy)) {
>  		cpuset_locked = true;
>  		cpuset_lock();
> +		dl_accounting_lock();
>  	}
>  
>  	/*
> @@ -650,8 +651,10 @@ int __sched_setscheduler(struct task_struct *p,
>  	if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
>  		policy = oldpolicy = -1;
>  		task_rq_unlock(rq, p, &rf);
> -		if (cpuset_locked)
> +		if (cpuset_locked) {
> +			dl_accounting_unlock();
> +			cpuset_unlock();
> +		}
>  		goto recheck;
>  	}
>  
> @@ -740,8 +743,10 @@ int __sched_setscheduler(struct task_struct *p,
>  	task_rq_unlock(rq, p, &rf);
>  
>  	if (pi) {
> -		if (cpuset_locked)
> +		if (cpuset_locked) {
> +			dl_accounting_unlock();
> +			cpuset_unlock();
> +		}
>  		rt_mutex_adjust_pi(p);
>  	}

[Severity: High]
What happens when normalize_rt_tasks() calls __sched_setscheduler(p, &attr,
false, false) on a SCHED_DEADLINE task while holding read_lock(&tasklist_lock)?

kernel/sched/core.c:normalize_rt_tasks() {
    read_lock(&tasklist_lock);
    for_each_process_thread(g, p) {
        ...
        if (!rt_or_dl_task(p))
            continue;
        ...
        __sched_setscheduler(p, &attr, false, false);
    }
    read_unlock(&tasklist_lock);
}

Because dl_policy(p->policy) is true, __sched_setscheduler() calls
dl_accounting_lock() without checking whether pi is true. On !CONFIG_CPUSETS
kernels (where cpuset_lock() was previously a no-op stub), acquiring
dl_accounting_mutex under read_lock(&tasklist_lock) sleeps in atomic context
and inverts the lock order relative to dl_rebuild_rd_accounting(), which
acquires dl_accounting_mutex before read_lock(&tasklist_lock).

In addition, on the normal return path of __sched_setscheduler(),
dl_accounting_unlock() and cpuset_unlock() are only called inside if (pi), so
calling __sched_setscheduler() with pi == false returns 0 with
dl_accounting_mutex (and cpuset_mutex) still locked.

[1] https://lore.kernel.org/r/7ea39be6af23940c33a63002ea832838fe69bb9e.1793804635.git.mathieu.desnoyers@efficios.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=13

Re: [PATCH v2 15/21] sched/deadline: Record the root domain holding a task's bandwidth

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/deadline: broken rd rebuild for PI-boosted departed tasks
- [Medium] sched/deadline: stale task_cpu() root domain on sleeping moved tasks
- [Low] sched/deadline: stale comments in switched_to_dl() and dl_bw_cpu helpers
--

commit 90cd336704a1119534fc7c444edd810ae1302642
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Record the root domain holding a task's bandwidth

This commit replaces dl_departed_cpu and cpumask-based root domain lookups
with a recorded dl_bw_cpu in struct sched_dl_entity so that deadline
bandwidth is released or moved from the same root domain where it was
accounted.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 85df2b0044bed..1dbde0fd199b4 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -252,15 +252,14 @@ static unsigned int dl_bw_gen;
>   * @p leaves SCHED_DEADLINE, or dies, its bandwidth kept for now in the root
>   * domain of @cpu. Under its rq lock.
>   */
> -static void dl_task_depart(struct task_struct *p, int cpu)
> +static void dl_task_depart(struct task_struct *p)
>  {
>  	struct sched_dl_entity *dl_se = &p->dl;
>  
>  	lockdep_assert_irqs_disabled();
> -	if (dl_se->dl_departed_cpu >= 0)
> -		return;
>  	guard(raw_spinlock)(&dl_departed_lock);
> -	dl_se->dl_departed_cpu = cpu;
> +	if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
> +		return;
>  	list_add(&dl_se->dl_departed_node, &dl_departed_list);
>  }
>  
> @@ -273,9 +272,10 @@ static void dl_task_undepart(struct task_struct *p)
>  	struct sched_dl_entity *dl_se = &p->dl;
>  
>  	lockdep_assert_irqs_disabled();
> +	if (list_empty(&dl_se->dl_departed_node))
> +		return;
>  	guard(raw_spinlock)(&dl_departed_lock);
> -	dl_se->dl_departed_cpu = -1;
> -	list_del(&dl_se->dl_departed_node);
> +	list_del_init(&dl_se->dl_departed_node);
>  }
>  
>  /*
> @@ -284,24 +284,25 @@ static void dl_task_undepart(struct task_struct *p)
>   * when a rebuild of the accounting has dropped it; and no rebuild running
>   * counts it again. Under @p's rq lock.
>   */
> -static void dl_task_release_bw(struct task_struct *p, int cpu)
> +static void dl_task_release_bw(struct task_struct *p)

[ ... ]

> @@ -3475,11 +3488,14 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
>   */
>  static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
>  {
> -	unsigned int cpu;
> +	int cpu = p->dl.dl_bw_cpu;
>  
> -	cpu = cpumask_first_and(&p->cpus_mask, cpu_active_mask);
> -	if (cpu >= nr_cpu_ids)
> -		cpu = task_cpu(p);
> +	/*
> +	 * No bandwidth of @p's counted anywhere: nothing to take back or to
> +	 * move, and the CPU it is on answers for its root domain.
> +	 */
> +	if (cpu < 0)
> +		return task_cpu(p);
>  	return cpu;
>  }

[ ... ]

> @@ -3614,10 +3643,12 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	 */
>  	if (p->dl.dl_bw_gen == dl_bw_gen)
>  		goto unlock;
> -	if (p->dl.dl_departed_cpu >= 0) {
> -		cpu = p->dl.dl_departed_cpu;
> -		if (!cpu_active(cpu))
> +	if (!list_empty(&p->dl.dl_departed_node)) {
> +		int recorded = p->dl.dl_bw_cpu;
> +
> +		if (recorded < 0 || !cpu_active(recorded))
>  			goto unlock;
> +		cpu = recorded;
>  	}
>  	rq = cpu_rq(cpu);
>  	dl_b = &rq->rd->dl_bw;
> @@ -3625,6 +3656,7 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	raw_spin_lock(&dl_b->lock);
>  	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
>  	raw_spin_unlock(&dl_b->lock);
> +	p->dl.dl_bw_cpu = cpu;
>  	p->dl.dl_bw_gen = dl_bw_gen;
>  unlock:
>  	raw_spin_unlock(&dl_departed_lock);

[Severity: High]
Can root domain bandwidth accounting be corrupted during a rebuild when a
departed task is boosted into dl_sched_class via priority inheritance before
its 0-lag time expires?

When a SCHED_DEADLINE task leaves the deadline policy before its 0-lag time,
switched_from_dl() arms inactive_timer and calls dl_task_depart(p) to link
p->dl.dl_departed_node onto dl_departed_list with p->dl.dl_bw_cpu recording
its root domain CPU. If p is then PI-boosted via rt_mutex_setprio() ->
switched_to_dl(), dl_task_undepart(p) unlinks p->dl.dl_departed_node while
preserving p->dl.dl_bw_cpu.

If dl_rebuild_rd_accounting() runs while p is PI-boosted,
dl_add_task_root_domain(p) sees list_empty(&p->dl.dl_departed_node) == true:

1. If p was moved to another cpuset partition while in SCHED_NORMAL (where no
deadline admission check ran because !dl_task(p)), dl_add_task_root_domain(p)
ignores p->dl.dl_bw_cpu, adds p->dl.dl_bw to the new cpuset's root domain via
dl_get_task_effective_cpus(p), and overwrites p->dl.dl_bw_cpu with cpu. Or,
if an earlier rebuild had already dropped p's departing bandwidth, this path
resurrects p->dl.dl_bw into p's effective cpuset.

2. Because rt_mutex_setprio() only holds p->pi_lock and rq->__lock (not
cpuset_mutex or dl_accounting_mutex), p can be PI-boosted when
dl_clear_root_domains() runs (so it is not on dl_departed_list and is not
restored there) and then deboost via rt_mutex_setprio() -> switched_from_dl()
before the task walk reaches p. switched_from_dl() puts p back onto
dl_departed_list after dl_clear_root_domains() has already finished, and
dl_add_task_root_domain(p) skips p because !dl_task(p) is now true, dropping
p's active 0-lag bandwidth completely.

3. For a regular non-deadline task with p->dl.dl_bw == 0 that is PI-boosted
during a rebuild, dl_add_task_root_domain(p) unconditionally sets
p->dl.dl_bw_cpu = cpu, leaving a non-negative dl_bw_cpu on a task with zero
bandwidth after it deboosts.

Should a departed task stay on dl_departed_list across PI boost/deboost until
its bandwidth is actually released, and should dl_add_task_root_domain(p) skip
tasks with p->dl.dl_bw == 0?

[ ... ]

> @@ -3890,8 +3922,7 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
>  	 * the task then removes the bandwidth from that one, as it did
>  	 * before departures were recorded.
>  	 */
> -	if (p->dl.dl_departed_cpu >= 0)
> -		dl_task_undepart(p);
> +	dl_task_undepart(p);

[Severity: Low]
This isn't a bug, but several comments in kernel/sched/deadline.c were not
updated to match the new dl_bw_cpu behavior:

1. In switched_to_dl(), the comment right above dl_task_undepart(p) still says:

	 * Boosted back into the class before its 0-lag time (a policy change
	 * released that bandwidth in sched_dl_overflow()): the inactive
	 * timer will not remove it, but its leaving the class again. That
	 * forgets the CPU recorded: moved to another root domain meanwhile,
	 * the task then removes the bandwidth from that one, as it did
	 * before departures were recorded.

However, dl_task_undepart(p) now preserves p->dl.dl_bw_cpu instead of
forgetting it.

2. Above dl_task_root_domain_cpu(), the function comment still states that it
uses p's current cpus_mask to find the root domain and falls back when
cpus_mask has no active CPU:

 * Find the root domain where a DL task's bandwidth is currently
 * accounted. For a sleeping task that was moved between cpusets
 * without running, task_rq(p)->rd may be stale. Use the task's
 * current cpus_mask (not yet updated to the new mask) to find the
 * actual root domain. Use cpus_mask rather than cpus_ptr, which
 * may be temporarily narrowed by migrate_disable().

whereas dl_task_root_domain_cpu() now reads p->dl.dl_bw_cpu directly.

3. Above dl_task_depart() and dl_task_release_bw(), the comments still refer
to the @cpu parameter and departed-CPU fallback that were removed from both
functions.

Could these comments be updated to reflect the new dl_bw_cpu logic?

[ ... ]

> @@ -4115,6 +4147,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
>  		return 0;
>  
> +	/*
> +	 * The bandwidth of a task of the class is counted in the root
> +	 * domain that took it, which the CPU the task is on does not tell:
> +	 * moved asleep, it has not run there yet. Change it where it is
> +	 * counted; a rebuild that dropped it, and a task that enters the
> +	 * class, have the root domain of that CPU take the new one. The
> +	 * caller holds cpuset_mutex, so no rebuild runs meanwhile.
> +	 */
> +	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
> +		  p->dl.dl_bw_gen == dl_bw_gen;
> +	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
> +	dl_b = dl_bw_of(cpu);

[Severity: Medium]
Does updating a task's deadline parameters while it is asleep after moving to
a new cpuset partition still use the stale task_cpu(p) root domain in
__sched_setscheduler() and sched_dl_overflow()?

When a task sleeping on a CPU in partition A is moved to a disjoint cpuset
partition B via cpuset_attach_task() -> set_cpus_allowed_ptr(),
affine_move_task() does not migrate the sleeping task's runqueue, so
task_cpu(p) remains in partition A while p->cpus_ptr and p->cpus_mask are
updated to partition B (and for a SCHED_DEADLINE task, set_cpus_allowed_dl()
updates p->dl.dl_bw_cpu to partition B).

If __sched_setscheduler() is then called on the still-sleeping task:

1. In kernel/sched/syscalls.c:__sched_setscheduler(), when
dl_bandwidth_enabled() is true, the affinity check before sched_dl_overflow()
still inspects rq->rd->span from task_rq(p) (partition A):

		if (dl_bandwidth_enabled() && dl_policy(policy) &&
				!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
			cpumask_t *span = rq->rd->span;

			if (!cpumask_subset(span, p->cpus_ptr) ||
			    rq->rd->dl_bw.bw == 0) {
				retval = -EPERM;
				goto unlock;
			}
		}

Because partition A's span is disjoint from p->cpus_ptr (partition B),
!cpumask_subset(span, p->cpus_ptr) triggers and __sched_setscheduler() fails
with -EPERM before reaching sched_dl_overflow().

2. In sched_dl_overflow(), when counted is false (for example, when a sleeping
non-deadline task moved to partition B is switched to SCHED_DEADLINE while
still asleep, or when a prior rebuild dropped the task's bandwidth),
cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p) falls back to task_cpu(p) in
partition A, crediting the bandwidth to partition A's root domain even though
inc_dl_tasks_cs(p) and the task's wakeup place it in partition B.

Should __sched_setscheduler() and the uncounted path in sched_dl_overflow()
resolve the destination root domain from the task's effective cpuset or
allowed mask rather than task_cpu(p)?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=15

Re: [PATCH v2 17/21] cgroup/cpuset: Give back the bandwidth an attach did not move

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] cgroup/cpuset: clear_attach_data() frees rebuilt bandwidth on concurrent rebuild
--

commit 8c5e4a02e2c52ea15855fb0f7b705db9ce328dd1
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

cgroup/cpuset: Give back the bandwidth an attach did not move

This commit tracks the deadline bandwidth actually moved during
cpuset_attach_task() in sum_moved_dl_bw via set_cpus_allowed_bw_reserved()
and frees any unused reservation (sum_migrate_dl_bw - sum_moved_dl_bw) in
clear_attach_data().

> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index ae0bbe23a6332..1ae86338f3c2a 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -3197,10 +3197,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
>  		if (cs->nr_migrate_dl_tasks) {
>  			if (!cancel)
>  				atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
> -			else if (cs->dl_bw_cpu >= 0) /* && cancel */
> -				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
> +			/*
> +			 * Give back the bandwidth the attach did not move:
> +			 * all of the reservation when it is cancelled, and
> +			 * what was set aside for the tasks that are no
> +			 * longer deadline tasks otherwise. Theirs is
> +			 * released by the class they left, from the root
> +			 * domain that counts it.
> +			 */
> +			if (cs->dl_bw_cpu >= 0) {
> +				u64 bw = cancel ? cs->sum_migrate_dl_bw :
> +					 cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
> +
> +				if (bw)
> +					dl_bw_free(cs->dl_bw_cpu, bw);
> +			}

[Severity: High]
Can clear_attach_data() free bandwidth that was already wiped or attributed
to the migrated tasks if rebuild_sched_domains() runs between
cpuset_can_attach() and cpuset_attach()?

During cgroup migration, cgroup_attach_lock() holds cpus_read_lock(), and
cpuset_can_attach() reserves cs->sum_migrate_dl_bw in cs->dl_bw_cpu's root
domain via cpuset_reserve_dl_bw(), increments attach_ctx.in_progress, and
drops cpuset_mutex.

Before cpuset_attach() runs, another thread calling rebuild_sched_domains()
(for instance via sched_rt_handler() or rebuild_sched_domains_energy()) only
takes cpus_read_lock() and mutex_lock(&cpuset_mutex) directly rather than
waiting on attach_ctx.in_progress via wait_attach_done_lock():

kernel/cgroup/cpuset.c:rebuild_sched_domains_cpuslocked() {
	mutex_lock(&cpuset_mutex);
	rebuild_sched_domains_locked();
	mutex_unlock(&cpuset_mutex);
}

rebuild_sched_domains_locked() -> partition_sched_domains() ->
dl_rebuild_rd_accounting() calls dl_clear_root_domains(), resetting
rd->dl_bw.total_bw to 0 and wiping the reservation made by
cpuset_reserve_dl_bw().

If dl_rebuild_rd_accounting() runs after css_set_move_task() moves the task
into cs, dl_add_task_root_domain() accounts the task's bandwidth into cs's
root domain and sets task->dl.dl_bw_cpu to a CPU in cs. When
cpuset_attach_task() then runs, dl_task_needs_bw_move(task,
cs->effective_cpus) returns false because task->dl.dl_bw_cpu is already in
cs's root domain, leaving cs->sum_moved_dl_bw at 0. Even if the rebuild runs
before css_set_move_task() and the task leaves SCHED_DEADLINE before
cpuset_attach_task(), the reservation in cs's root domain has already been
cleared by dl_clear_root_domains().

When clear_attach_data() subsequently calls dl_bw_free(cs->dl_bw_cpu,
cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw), wouldn't it subtract
cs->sum_migrate_dl_bw from the rebuilt total_bw and leave the destination
root domain's total_bw short of what its tasks hold?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=17

Re: [PATCH v2 18/21] sched/deadline: Credit a rebuild to a CPU that can hold the bandwidth

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/deadline: offline hk_msk CPUs break dl_get_task_effective_cpus()
- [High] sched/deadline: stale HK_TYPE_DOMAIN mask in dl_get_task_effective_cpus()
- [High] sched/deadline: dl_add_task_root_domain() ignores dl_bw_cpu on empty msk
--

commit 119330e07c6138cad461765e2b7413e00f7f5e31
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Credit a rebuild to a CPU that can hold the bandwidth

This commit filters out isolated CPUs from a deadline task's effective CPU
mask when housekeeping CPUs are present, and skips tasks that have no active
CPU during a root domain bandwidth accounting rebuild.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 029ec2ba5a663..c0af06d0f3688 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3617,6 +3617,16 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
>  	 * hierarchy up to the partition root.
>  	 */
>  	cpuset_cpus_allowed_locked(p, cpus);
> +	/*
> +	 * Which the CPUs isolated by isolcpu="domain" are not part of: a
> +	 * cpuset can include them beside housekeeping CPUs, and without
> +	 * cpusets the mask is every CPU the task can run on. The first
> +	 * active CPU of it would then be an isolated one, of the default
> +	 * root domain, rather than one of the root domain the task's
> +	 * bandwidth belongs to.
> +	 */
> +	if (housekeeping_enabled(HK_TYPE_DOMAIN))
> +		cpumask_and(cpus, cpus, hk_msk);
>  }

[Severity: High]
Can offline or unpopulated housekeeping CPUs in hk_msk cause
dl_get_task_effective_cpus() to drop or misattribute the bandwidth of a
deadline task running in def_root_domain?

In housekeeping_setup() and housekeeping_update(),
housekeeping_cpumask(HK_TYPE_DOMAIN) is built from cpu_possible_mask minus
the isolated CPUs, so hk_msk continues to include housekeeping CPUs when
they are offline or unpopulated:

kernel/sched/deadline.c:dl_get_task_effective_cpus() {
	hk_msk = housekeeping_cpumask(HK_TYPE_DOMAIN);
	if (housekeeping_enabled(HK_TYPE_DOMAIN)) {
		if (!cpumask_intersects(p->cpus_ptr, hk_msk)) {
			cpumask_andnot(cpus, cpu_active_mask, hk_msk);
			return;
		}
	}
	cpuset_cpus_allowed_locked(p, cpus);
	if (housekeeping_enabled(HK_TYPE_DOMAIN))
		cpumask_and(cpus, cpus, hk_msk);
}

For a deadline task running on an active isolated CPU in def_root_domain,
p->cpus_ptr can still contain offline or unpopulated housekeeping CPUs
(for example, __sched_setscheduler() requires cpumask_subset(rq->rd->span,
p->cpus_ptr) where def_root_domain.span contains offline possible CPUs,
and top_cpuset keeps offline non-subpartition CPUs in p->cpus_ptr).

Because p->cpus_ptr and hk_msk both contain those offline housekeeping CPUs,
cpumask_intersects(p->cpus_ptr, hk_msk) evaluates to true even when p has no
active housekeeping CPU in p->cpus_ptr. Execution then falls through and
cpumask_and(cpus, cpus, hk_msk) strips all active isolated CPUs from cpus:

- If cpus contains no active housekeeping CPUs (for example, all
  housekeeping CPUs in top_cpuset are offline, or p is in a child cpuset cs
  where guarantee_active_cpus() stops at cs because cs->effective_cpus
  intersects cpu_active_mask on the active isolated CPU),
  cpumask_first_and(cpu_active_mask, msk) in dl_add_task_root_domain()
  returns >= nr_cpu_ids and drops the active task's bandwidth from
  def_root_domain.

- If another housekeeping CPU outside p->cpus_ptr is active in top_cpuset
  (or with !CONFIG_CPUSETS), stripping the isolated CPUs causes
  dl_add_task_root_domain() to credit p's bandwidth to that housekeeping
  root domain instead of def_root_domain. Similarly, a task admitted on an
  isolated CPU in def_root_domain with an unnarrowed affinity mask has its
  bandwidth moved to a housekeeping root domain on rebuild while continuing
  to run in def_root_domain.

- Conversely, if a housekeeping deadline task is preempted on an isolated
  CPU while in a migrate_disable() section, migrate_disable_switch() narrows
  p->cpus_ptr to cpumask_of(rq->cpu), so !cpumask_intersects(p->cpus_ptr,
  hk_msk) evaluates to true and credits its bandwidth to def_root_domain
  instead of its housekeeping root domain.

[Severity: High]
Also in dl_get_task_effective_cpus(), does reading HK_TYPE_DOMAIN observe a
stale mask during cgroup v2 isolated partition transitions?

When a cgroup v2 PRS_ISOLATED partition transitions to PRS_ROOT or
PRS_MEMBER, or is invalidated during CPU hotplug,
cpuset_update_sd_hk_unlock() and cpuset_handle_hotplug() invoke
rebuild_sched_domains_locked() -> partition_sched_domains_locked() ->
dl_rebuild_rd_accounting() -> dl_add_task_root_domain() ->
dl_get_task_effective_cpus() before housekeeping_update() updates
HK_TYPE_DOMAIN:

kernel/cgroup/cpuset.c:cpuset_update_sd_hk_unlock() {
	if (force_sd_rebuild)
		rebuild_sched_domains_locked();

	if (update_housekeeping) {
		update_housekeeping = false;
		cpumask_copy(isolated_hk_cpus, isolated_cpus);

		mutex_unlock(&cpuset_mutex);
		cpus_read_unlock();
		WARN_ON_ONCE(housekeeping_update(isolated_hk_cpus));
	}
}

While generate_sched_domains() uses housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT),
dl_get_task_effective_cpus() queries HK_TYPE_DOMAIN, which still excludes the
CPUs transitioning out of PRS_ISOLATED while dl_rebuild_rd_accounting() runs.

If a deadline task in that partition has p->cpus_ptr intersecting the stale
hk_msk (for example, on an offline CPU), cpumask_and(cpus, cpus, hk_msk)
clears the newly un-isolated active CPUs from cpus and drops the task's
bandwidth from the rebuild. If p->cpus_ptr does not intersect the stale
hk_msk, dl_get_task_effective_cpus() returns cpu_active_mask & ~hk_msk,
which can pick a lower-numbered isolated CPU in def_root_domain instead of
the task's new root domain.

>  
>  /* The caller should hold cpuset_mutex */
> @@ -3637,7 +3647,18 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
>  	dl_get_task_effective_cpus(p, msk);
>  	cpu = cpumask_first_and(cpu_active_mask, msk);
> -	BUG_ON(cpu >= nr_cpu_ids);
> +	if (cpu >= nr_cpu_ids) {
> +		/*
> +		 * No active CPU to count this task's bandwidth: the ones it
> +		 * may run on all went offline, which the CPUs isolated by
> +		 * isolcpu="domain" allow, no deadline task of theirs keeping
> +		 * the last of them online. Leave it out of this rebuild,
> +		 * where dl_bw_gen keeps its release out as well, until one
> +		 * has a CPU for it.
> +		 */
> +		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
> +		return;
> +	}

[Severity: High]
Does returning early when cpu >= nr_cpu_ids skip tasks whose bandwidth was
already moved to an active fallback CPU by dl_task_offline_migration(), or
recorded on an active CPU in p->dl.dl_bw_cpu when departing SCHED_DEADLINE?

When a deadline task confined to isolated CPUs is throttled while all of its
isolated CPUs go offline, dl_task_timer() -> dl_task_offline_migration()
picks an active fallback CPU via cpumask_any(cpu_active_mask), migrates p to
later_rq->cpu, and records p->dl.dl_bw_cpu = later_rq->cpu without modifying
p->cpus_ptr:

kernel/sched/deadline.c:dl_task_offline_migration() {
	scoped_guard (raw_spinlock, &dl_departed_lock) {
		if (p->dl.dl_bw_gen == dl_bw_gen) {
			...
		}
		p->dl.dl_bw_cpu = later_rq->cpu;
	}

	set_task_cpu(p, later_rq->cpu);
}

Because p->cpus_ptr is still restricted to the offline isolated CPUs,
dl_get_task_effective_cpus(p, msk) returns an empty mask on rebuild. In
dl_add_task_root_domain(), cpu = cpumask_first_and(cpu_active_mask, msk) is
then >= nr_cpu_ids, and this early return exits before consulting
p->dl.dl_bw_cpu or crediting later_rq->rd->dl_bw.

Similarly, because the cpu >= nr_cpu_ids return happens before acquiring
dl_departed_lock and checking !list_empty(&p->dl.dl_departed_node), a task
that departed SCHED_DEADLINE with an active CPU recorded in p->dl.dl_bw_cpu
and then had its affinity restricted to offline isolated CPUs (before dying
during the rebuild walk or being PI-boosted back into dl_sched_class prior to
its 0-lag time) will also return early and have its bandwidth omitted from
the recorded active CPU's root domain.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=18

Re: [PATCH v2 20/21] sched/deadline: Cancel the replenishment timer with the parameters

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] sched/deadline: stale deadline and runtime left by __dl_clear_params()
- [Medium] sched/deadline: uncancelled dl_timer in switched_from_dl() for sleepers
--

commit 88a7a115f1789bc812516cdb6cea3464930cdccf
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Cancel the replenishment timer with the parameters

This commit cancels a deadline task's replenishment timer in
task_non_contending() and inactive_task_timer() when clearing its parameters
at the 0-lag time after leaving SCHED_DEADLINE. This prevents a task that
re-enters SCHED_DEADLINE before the old period expires from keeping the
armed replenishment timer of its previous reservation.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index c0af06d0f3688..15d224846ebad 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -563,6 +563,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
>  				if (READ_ONCE(p->__state) == TASK_DEAD)
>  					sub_rq_bw(dl_se, &rq->dl);
>  				dl_task_release_bw(p);
> +				cancel_replenish_timer(dl_se);
>  				__dl_clear_params(dl_se);
>  			}
>  		}

[Severity: Medium]
When cancel_replenish_timer() and __dl_clear_params() run at the 0-lag time
in task_non_contending() or inactive_task_timer(), __dl_clear_params()
clears dl_throttled and dl_yielded along with the static reservation
parameters, but leaves the dynamic dl_se->deadline and dl_se->runtime from
the old reservation intact:

kernel/sched/deadline.c:__dl_clear_params() {
	dl_se->dl_runtime		= 0;
	dl_se->dl_deadline		= 0;
	dl_se->dl_period		= 0;
	dl_se->flags			= 0;
	dl_se->dl_bw			= 0;
	dl_se->dl_density		= 0;

	dl_se->dl_throttled		= 0;
	dl_se->dl_yielded		= 0;
	...
}

For the two cases described in the commit message where the 0-lag time is
reached before the old period ends, could leaving dl_se->deadline and
dl_se->runtime set cause problems if the task returns to SCHED_DEADLINE
before the old deadline or period expires?

In the first case, a task yields (or leaves with remaining runtime > 0), so
its 0-lag time is earlier than dl_se->deadline. After the 0-lag time passes
and releases the old bandwidth and clears dl_throttled and dl_yielded, if
the task returns to SCHED_DEADLINE before the old dl_se->deadline,
enqueue_dl_entity() checks whether the deadline has passed on the
ENQUEUE_MOVE path:

kernel/sched/deadline.c:enqueue_dl_entity() {
	...
	} else if ((flags & ENQUEUE_MOVE) &&
		   !is_dl_boosted(dl_se) &&
		   dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
		setup_new_dl_entity(dl_se);
	}
}

Because the stale dl_se->deadline is still in the future,
setup_new_dl_entity() is skipped and the task runs with its leftover
dl_se->runtime and old dl_se->deadline (regaining runtime it forfeited when
yielding, or exceeding a smaller new dl_runtime after its old bandwidth was
already released).

In the second case, a constrained-deadline task overruns slightly and
reaches its 0-lag time after dl_se->deadline, but before dl_next_period().
If it switches back to SCHED_DEADLINE as a constrained-deadline task before
the old period ends, enqueue_dl_entity() runs dl_check_constrained_dl()
before reaching the ENQUEUE_MOVE check:

kernel/sched/deadline.c:enqueue_dl_entity() {
	...
	if (!dl_se->dl_throttled && !dl_is_implicit(dl_se))
		dl_check_constrained_dl(dl_se);
	...
	if (!dl_se->dl_defer && dl_se->dl_throttled && !(flags & ENQUEUE_REPLENISH)) {
		if (flags & ENQUEUE_WAKEUP)
			task_contending(dl_se, flags);

		return;
	}
}

Seeing dl_se->deadline in the past and rq_clock(rq) before dl_next_period(),
dl_check_constrained_dl() sets dl_se->dl_throttled = 1 and re-arms dl_timer
for the old period, causing enqueue_dl_entity() to return early without
initializing the new reservation.

Should __dl_clear_params() also reset dl_se->deadline and dl_se->runtime?

> @@ -2326,6 +2327,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
>  		}
>  
>  		dl_task_release_bw(p);
> +		cancel_replenish_timer(dl_se);
>  		__dl_clear_params(dl_se);
>  
>  		goto unlock;

[Severity: Medium]
What happens if a constrained-deadline task overruns slightly as it blocks,
reaches its 0-lag time while still sleeping in SCHED_DEADLINE, and only then
switches out of SCHED_DEADLINE before its replenishment timer fires?

When the task blocks in dequeue_task_dl(), update_curr_dl_se() sees the
runtime overrun, sets dl_se->dl_throttled = 1, and starts dl_se->dl_timer
for dl_next_period(). Then dequeue_dl_entity() calls
task_non_contending(&p->dl, true).

If the 0-lag time passes while the task is still asleep and still in
SCHED_DEADLINE, task_non_contending() or inactive_task_timer() drops
running_bw and clears dl_non_contending, while dl_throttled remains 1 and
dl_timer remains armed.

If the sleeping task is then switched to another scheduling class before
dl_timer fires, switched_from_dl() takes the !task_on_rq_queued(p) &&
!p->dl.dl_non_contending branch:

kernel/sched/deadline.c:switched_from_dl() {
	...
	if (!task_on_rq_queued(p)) {
		bool released = false;

		if (p->dl.dl_non_contending) {
			sub_running_bw(&p->dl, &rq->dl);
		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
			dl_task_release_bw(p);
			released = true;
		}
		sub_rq_bw(&p->dl, &rq->dl);
		if (released)
			p->dl.dl_bw = 0;
	}
}

Because neither task_non_contending() nor inactive_task_timer() runs on this
path, dl_task_release_bw(p) releases the bandwidth while leaving
p->dl.dl_timer armed and p->dl.dl_throttled set to 1. If the task returns to
SCHED_DEADLINE before the old dl_timer expires, enqueue_dl_entity() sees
dl_se->dl_throttled == 1 and returns early without enqueuing the task until
the old period ends.

Should switched_from_dl() also cancel the replenishment timer and clear the
throttled state when releasing a sleeping task's bandwidth past its 0-lag
time?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=20

Re: [PATCH v2 21/21] sched/deadline: Count the bandwidth a boosted task is given

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] sched/deadline: dl_change_utilization() checks old flags before update
--

commit f342e630523d7367eb408895a6057749ace71951
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Count the bandwidth a boosted task is given

This commit moves the runqueue bandwidth accounting update from
sched_dl_overflow() to __setparam_dl() so that tasks already in the deadline
class due to priority inheritance have their new bandwidth added to the
runqueue's this_bw when sched_setattr() sets SCHED_DEADLINE parameters.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 15d224846ebad..a3b45bb4f35e1 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -442,22 +442,32 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
>  	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
>  }
>  
> +/*
> + * The runqueue a sleeping task is on counts the bandwidth the task has:
> + * give it @new_bw in place of what it counts of the task's.
> + */
>  static void dl_change_utilization(struct task_struct *p, u64 new_bw)
>  {
> -	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
> +	struct sched_dl_entity *dl_se = &p->dl;
> +	struct rq *rq = task_rq(p);
>  
> -	if (task_on_rq_queued(p))
> +	if (task_on_rq_queued(p) || dl_entity_is_special(dl_se))
>  		return;

[ ... ]

> @@ -4266,6 +4275,18 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
>  {
>  	struct sched_dl_entity *dl_se = &p->dl;
> +	u64 period = attr->sched_period ?: attr->sched_deadline;
> +
> +	/*
> +	 * The bandwidth of a special entity is counted on no runqueue.
> +	 * The rest have theirs changed here, where their parameters are:
> +	 * switched_to_dl() hands it to the runqueue of a task that enters
> +	 * the class, and a task that priority inheritance boosted into the
> +	 * class is in it already, so nothing dequeues it and enqueues it
> +	 * back for that.
> +	 */
> +	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
> +		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
>  
>  	dl_se->dl_runtime = attr->sched_runtime;
>  	dl_se->dl_deadline = attr->sched_deadline;

[Severity: Medium]
Does dl_change_utilization() check the old dl_se->flags before __setparam_dl()
updates them?

In __setparam_dl(), dl_change_utilization() is called when attr->sched_flags
does not have SCHED_FLAG_SUGOV set, before dl_se->flags is updated from
attr->sched_flags:

kernel/sched/deadline.c:__setparam_dl() {
	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));

	dl_se->dl_runtime = attr->sched_runtime;
	dl_se->dl_deadline = attr->sched_deadline;
	dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
	dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
}

If a sleeping task that previously had SCHED_FLAG_SUGOV set in dl_se->flags is
reconfigured via __sched_setscheduler() -> __setscheduler_params() ->
__setparam_dl() without SCHED_FLAG_SUGOV, dl_entity_is_special(dl_se) in
dl_change_utilization() still observes the old SCHED_FLAG_SUGOV flag and
returns early without adding new_bw to rq->dl.this_bw or setting
dl_se->dl_bw_oncpu = 1.

Immediately afterward, __setparam_dl() clears SCHED_FLAG_SUGOV in dl_se->flags,
and switched_to_dl() does not run because the task is already in
dl_sched_class. When the task later wakes up on the same CPU, task_contending()
sees !dl_entity_is_special(dl_se) and adds dl_se->dl_bw to rq->dl.running_bw
while rq->dl.this_bw does not include it, triggering:

	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);

Since __setparam_dl() is the only caller of dl_change_utilization() and already
checks !(attr->sched_flags & SCHED_FLAG_SUGOV), should the
dl_entity_is_special(dl_se) check in dl_change_utilization() be dropped?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104211207.1392343-1-mathieu.desnoyers@efficios.com?part=21

Re: [PATCH v2 01/21] sched: Reject SCHED_FLAG_KEEP_PARAMS before reserving deadline bandwidth

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> __sched_setscheduler() returns -EINVAL for a SCHED_FLAG_KEEP_PARAMS
> request that would change scheduling class, after sched_dl_overflow()
> has reserved the bandwidth of the request, and nothing gives it back.
> Switching a SCHED_OTHER task to SCHED_DEADLINE that way fails and leaks
> the runtime that sched_setattr() filled in from the task's slice, up to
> 100ms with a custom one, into the root domain's total_bw. Repeated, it
> makes the root domain refuse every deadline task, and refuse to offline
> its last CPU.
> 
> Reserve the bandwidth after that check, last of the checks. A refused
> request also leaves sched_reset_on_fork as it was again.
> 
> Fixes: 8a70415351ba ("sched: Reject SCHED_FLAG_KEEP_PARAMS when it would change scheduling class")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---

Makes sense to me.

Reviewed-by: Juri Lelli <juri.lelli@redhat.com>

Thanks,
Juri

Re: [PATCH v2 02/21] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> SCHED_FLAG_KEEP_PARAMS keeps the parameters of a task: it skips
> __setscheduler_params(), which is what sets p->policy. A request that
> asks for another policy is then carried out by half, and succeeds:
> SCHED_OTHER to SCHED_BATCH with SCHED_FLAG_KEEP_PARAMS returns 0 and
> leaves the task where it was.
> 
> For SCHED_DEADLINE it costs bandwidth. A task that priority inheritance
> boosted into the deadline class is of that class already, so
> prev_class == next_class and the class check of
> commit 8a70415351ba ("sched: Reject SCHED_FLAG_KEEP_PARAMS when it would
> change scheduling class") lets the request through: sched_dl_overflow()
> admits it and adds
> its bandwidth, which the skipped __setscheduler_params() never gives to
> the task. It keeps SCHED_NORMAL and a dl_bw of 0, so nothing releases
> that bandwidth, and total_bw of the root domain holds it for good.
> 
> Refuse a policy change as well, where the class change was refused
> alone.
> 
> A SCHED_OTHER owner of a PTHREAD_PRIO_INHERIT mutex with a slice of
> 40ms, boosted by a deadline waiter blocked on it, leaves 0.4 of a CPU
> behind at every such call: total_bw 419424 -> 838854 and there to stay,
> in 8-CPU VMs of v7.4-rc1 with the four deadline fixes of sched/urgent,
> and of this series up to here. With this change the call is refused and
> total_bw is as it was. The selftest added later in this series has the
> case, params_keep_boosted.
> 
> Found by the Sashiko AI review of the first posting of this series.
> 
> Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---

Makes sense to me.

Reviewed-by: Juri Lelli <juri.lelli@redhat.com>

Thanks,
Juri

Re: [PATCH v2 03/21] sched/deadline: Do not admit bandwidth for parameters that are kept

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> sys_sched_setattr() fills a SCHED_FLAG_KEEP_PARAMS request in from the
> task itself (get_params()), with no lock held, and
> __sched_setscheduler() then skips __setscheduler_params(): the task
> keeps the parameters it has, and its dl_bw with them.
> sched_dl_overflow() runs all the same.
> 
> A parameter change of the task's own that lands in between leaves the
> request stale, and dl_param_changed() takes it for a change: total_bw of
> the root domain, and this_bw and running_bw of its runqueue through
> dl_change_utilization(), then hold a bandwidth the task does not have,
> and the release at its 0-lag time, which takes the task's own dl_bw,
> leaves the difference behind.
> 
> There is nothing to admit for parameters that are kept: skip the check.
> 
> Found by reading, after the Sashiko AI review of the first posting of
> this series raised the case the previous patch refuses. Not reproduced:
> it needs a parameter change of a deadline task to land inside the window
> of a SCHED_FLAG_KEEP_PARAMS request on the same task.
> 
> Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---

Reviewed-by: Juri Lelli <juri.lelli@redhat.com>

Thanks,
Juri

Re: [PATCH v2 04/21] sched/deadline: Do not divide by zero active CPUs of a root domain

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> __dl_sub() and __dl_add() spread a change of a root domain's total_bw
> over the extra_bw of its active CPUs, dividing it by their number, which
> their callers take from dl_bw_cpus(). That number is 0 for a root domain
> whose CPUs all went offline, such as the default root domain, which
> holds the offline CPUs: the division then traps.
> 
> A debugfs write of a fair server's runtime or period reaches it while
> its CPU goes offline or comes online. From the rebuild of the root
> domains in sched_cpu_deactivate() until the CPU is offline, and from the
> CPU online until sched_cpu_activate(), the CPU sits in the default root
> domain, with no active CPU. A write that leaves the server's bandwidth
> as it is passes admission there, __dl_overflow() comparing that root
> domain's capacity, 0, with total_bw less the old bandwidth plus the new,
> 0 as well, and dl_server_apply_params() divides. Any other value is
> refused with EBUSY, higher directly, lower through the unsigned
> subtraction. Once the CPU is offline the write is refused, since
> commit 4043f5498416 ("sched/deadline: Reject debugfs dl_server writes
> for offline CPUs").
> 
> Writing a fair server's own runtime back in a loop while its CPU goes
> offline and online oopses on the first round, in 8-CPU VMs of v7.4-rc1
> with the four deadline fixes of sched/urgent:
> 
>   Oops: divide error: 0000 [#1] SMP NOPTI
>   CPU: 6 UID: 0 PID: 110 Comm: dl_server_div0
>   RIP: 0010:dl_server_apply_params+0x335/0x380
>   Call Trace:
>    <TASK>
>    sched_server_write_common.isra.0+0x1c0/0x260
>    full_proxy_write+0x58/0x90
>    vfs_write+0xcb/0x550
>    ksys_write+0x6c/0xe0
>    do_syscall_64+0x109/0x5d0
>    entry_SYSCALL_64_after_hwframe+0x77/0x7f
> 
> inactive_task_timer() can reach it too, found by reading: it releases
> the bandwidth of a task that left SCHED_DEADLINE, or died, before its
> 0-lag time from the root domain of the CPU the task is on, and that CPU
> may go offline before the timer fires. With admission control off, so
> may sched_setattr() to SCHED_DEADLINE of a task asleep on an offline
> CPU.
> 
> With no active CPU there is no extra_bw to update: change total_bw
> alone. 20 rounds of the reproducer leave no warning then.
> 
> Fixes: daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl tasks")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>  kernel/sched/deadline.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 963fda6ddb7f..dadda1a528ec 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
>  	}
>  }
>  
> +/*
> + * @cpus is the number of active CPUs of the root domain, whose extra_bw
> + * shares the change: none once they all went offline, and total_bw alone
> + * changes then.
> + */
>  static inline
>  void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
>  {
>  	dl_b->total_bw -= tsk_bw;
> -	__dl_update(dl_b, (s32)tsk_bw / cpus);
> +	if (cpus)
> +		__dl_update(dl_b, (s32)tsk_bw / cpus);
>  }
>  
>  static inline
>  void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
>  {
>  	dl_b->total_bw += tsk_bw;
> -	__dl_update(dl_b, -((s32)tsk_bw / cpus));
> +	if (cpus)
> +		__dl_update(dl_b, -((s32)tsk_bw / cpus));
>  }

I wonder if it makes sense to add/sub from total_bw when cpus == 0.

Wouldn't be better, if possible, to avoid calls with 0 cpus?

Thanks,
Juri

Re: [PATCH v2 05/21] sched/deadline: Do not BUG() on a deadline task with no active CPU allowed

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> dl_task_root_domain_cpu() finds the root domain that holds a deadline
> task's bandwidth from the first active CPU of the task's mask, and
> BUG()s when the mask has none. A mask can have none:
> 
>  - a task boosted into the deadline class by priority inheritance keeps
>    the affinity of its own policy, which may allow one CPU alone, or a
>    cpuset of one. dl_task_needs_bw_move() and set_cpus_allowed_dl()
>    take it for a deadline task, dl_task(p) being true, and look the root
>    domain up from the mask they are replacing, which allows that CPU,
>    inactive by then: the cpuset code updates the masks of a cpuset's
>    tasks once the CPU is inactive, and select_fallback_rq() widens the
>    mask of a task it pushes off that CPU;
>  - with admission control off (sched_rt_runtime_us at -1), a deadline
>    task may be pinned to fewer CPUs than its root domain spans, and
>    those may all go offline.
> 
> Both take the kernel down on the first try, in 8-CPU VMs of v7.4-rc1
> with the four deadline fixes of sched/urgent. The priority inheritance
            ^^^
Not sure how much sense will this info make when reading this changelog
in the future. :)

> one, which asks for no unusual configuration, through the wakeup of the
> boosted owner:
> 
>   kernel BUG at kernel/sched/deadline.c:3340!
>   CPU: 7 UID: 0 PID: 0 Comm: swapper/7
>   RIP: 0010:set_cpus_allowed_dl+0xf0/0x100
>   Call Trace:
>    <IRQ>
>    set_cpus_allowed_force+0x9f/0xf0
>    cpuset_cpus_allowed_fallback+0x12b/0x1d0
>    select_fallback_rq+0x238/0x340
>    try_to_wake_up+0x2a0/0xb70
> 
> Fall back to the CPU the task last ran on, which these callers used
> before the commit this one fixes. Its root domain may have no active CPU
> left: the previous patch has __dl_sub() and __dl_add() take that. Ten
> rounds of each case leave no warning with it.
> 
> Fixes: b4a61fdc19c5 ("sched/deadline: Fix stale root domain in cpuset bandwidth transfer")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>  kernel/sched/deadline.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index dadda1a528ec..30a52e42c26d 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3338,13 +3338,19 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
>   * current cpus_mask (not yet updated to the new mask) to find the
>   * actual root domain. Use cpus_mask rather than cpus_ptr, which
>   * may be temporarily narrowed by migrate_disable().
> + *
> + * The mask has no active CPU left when the CPUs it allows went offline,
> + * which the affinity of a deadline task allows with admission control
> + * off, and that of a task boosted into the deadline class by priority
> + * inheritance always: fall back to the CPU the task last ran on.
>   */
>  static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
>  {
>  	unsigned int cpu;
>  
>  	cpu = cpumask_first_and(&p->cpus_mask, cpu_active_mask);
> -	BUG_ON(cpu >= nr_cpu_ids);
> +	if (cpu >= nr_cpu_ids)
> +		cpu = task_cpu(p);
>  	return cpu;
>  }

At least (worst case) we fall back to how things were before my change
introduced the regression.

Acked-by: Juri Lelli <juri.lelli@redhat.com>

Thanks,
Juri

Re: [PATCH v2 00/21] sched/deadline: bandwidth accounting fixes and selftests

[Marek Szyprowski]

On 04.11.2026 22:11, Mathieu Desnoyers wrote:
> Please find in this series seventeen fixes to SCHED_DEADLINE bandwidth
> accounting, on top of your four in sched/urgent. Twenty-one patches:
> the seventeen fixes, three that add the selftests catching them, and
> one that shows a runqueue's deadline bandwidth in sched_debug. Nine of
> the seventeen are new in v2, from the reviews of v1 and of the patches
> those reviews brought; the changes are listed at the end.
> ...

Thanks! This fixes the issue I've reported in:

https://lore.kernel.org/all/eb0e386b-da07-4a2b-bff3-79441fd6be1f@samsung.com/

Feel free to add:

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v2 06/21] sched/deadline: Do not lose a throttled task switched out and back

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> A SCHED_DEADLINE task that is throttled, switched to another policy and
> switched back before its 0-lag time can be left throttled with nothing
> to replenish it: runnable, queued as far as the core is concerned, but
> on no dl_rq and with no timer pending. It never runs again, and a fatal
> signal cannot make it run; only another switch out of SCHED_DEADLINE
> requeues it.
> 
>  1. the task exhausts its runtime: dl_throttled is set, and the
>     replenishment timer armed for the start of its next period;
>  2. sched_setscheduler() switches it to SCHED_OTHER: switched_from_dl()
>     calls task_non_contending(), which arms the inactive timer for the
>     0-lag time; the parameters are kept until then, so that leaving and
>     reentering SCHED_DEADLINE gains no bandwidth;
>  3. the replenishment timer fires: dl_task_timer() finds a task that is
>     not a deadline task and returns, dl_throttled still set;
>  4. sched_setattr() switches it back before the 0-lag time:
>     switched_to_dl() cancels the inactive timer, and enqueue_dl_entity()
>     leaves a throttled task to its replenishment timer, which has fired
>     already.
> 
> The window between the deadline and the 0-lag time is the overrun
> scaled by period/runtime: about 100us for an overrun of 20us at 20%
> bandwidth. The replenishment timer cleared the parameters, dl_throttled
> included, of a task it found out of SCHED_DEADLINE until
> commit 209a0cbda7a0 ("sched/deadline: Improve the tracking of active
> utilization") moved that reset to the inactive timer, at the 0-lag
> time.
> 
> Give the task the replenishment the timer owes it, as the timer already
> does for one that is throttled and not queued: the overrun comes off the
> periods that follow, the deadline moves past it, and dl_throttled and
> dl_yielded are cleared. A return before the 0-lag time then finds an
> entity it can run on, and a return after it finds the parameters reset
> by the inactive timer, as before.
> 
> Clearing dl_throttled alone would leave the overrun in the runtime, for
> the new period of the enqueue to forgive, the deadline behind, for
> dl_check_constrained_dl() to throttle the task on again, and a yield of
> that period set, for update_curr_dl_se() to throttle it at once. The
> first version of this patch did, and the Sashiko AI review of the first
> posting of this series pointed the three out.
> 
> The selftest added two patches later fails its throttled_round_trip
> case 8 times out of 8, on v7.3 as on v7.4-rc1 with the four deadline
> fixes of sched/urgent, and passes it 8 times out of 8 with this change.
> It was found by a cgroup selftest under development,
> test_concurrency_deadline, which hung about once in 1350 runs.
> 
> Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>  kernel/sched/deadline.c | 17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 30a52e42c26d..b646063a7b0d 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -1232,10 +1232,23 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
>  
>  	/*
>  	 * The task might have changed its scheduling policy to something
> -	 * different than SCHED_DEADLINE (through switched_from_dl()).
> +	 * different than SCHED_DEADLINE (through switched_from_dl()). It
> +	 * keeps its parameters until its 0-lag time and may come back to
> +	 * SCHED_DEADLINE before then, switched_to_dl() cancelling the
> +	 * inactive timer: give it the replenishment this timer owes it, as
> +	 * the !queued case below does, and leave the enqueue to its return.
> +	 * A throttle left behind would never be lifted; an overrun left in
> +	 * its runtime would be forgiven by the new period its enqueue
> +	 * starts; and a yield left set would throttle it again at once.
>  	 */
> -	if (!dl_task(p))
> +	if (!dl_task(p)) {
> +		if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
> +			sched_clock_tick();
> +			update_rq_clock(rq);
> +			replenish_dl_entity(dl_se);
> +		}
>  		goto unlock;
> +	}

This should fix the subject issue. Sashiko points out a follow-up
re-entry case that looks problematic (and I think I'm now seeing more),
but that should all only concern limited bandwidth overruns. This change
fix an actual hang, so we might want to take it and work on the
remaining issues separately.

Thanks,
Juri

Re: [PATCH v2 08/21] selftests/sched: Add a test of SCHED_DEADLINE policy round trips

[Juri Lelli]

Hello,

On 04/11/26 16:11, Mathieu Desnoyers wrote:
> Add deadline_switch_test, with a case for the fix two patches back and
> one for commit 41a7968022ca ("sched/deadline: Fix total_bw leak when
> switching away from SCHED_DEADLINE while !queued"):
> 
>  - throttled_round_trip: a task with a runtime of 10ms per 2s period
>    spins, so that each replenishment throttles it again at once. Right
>    after a throttle it is switched to SCHED_OTHER, and back 200us after
>    its deadline: past its replenishment timer, and before its 0-lag
>    time, which the overrun scaled by 200 puts later. It must run again
>    within a period and a half. Three attempts;
>  - asleep_round_trip: a task of 9ms every 10ms, running 0.2ms and
>    sleeping 50ms, is switched to SCHED_OTHER and back while it sleeps
>    past its 0-lag time, enough times that a leak of its bandwidth at
>    each round trip fills the root domain: sched_setattr() then fails
>    with EBUSY. Skipped without admission control (sched_rt_runtime_us
>    at -1). A kernel that leaks is left without SCHED_DEADLINE bandwidth
>    until a reboot, so this case runs last.
> 
> In 8 VMs of 8 vCPUs each, v7.3 fails both cases, and v7.4-rc1 with the
> four deadline fixes of sched/urgent fails the first; with the two
> previous patches as well both pass; every time.
> 
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---

Thanks for the selftests! Very useful.

FWIW I did propose some already in the past, but then didn't get time/
tokens for a follow up:

https://lore.kernel.org/all/20260306-upstream-deadline-kselftests-v1-0-2b23ef74c46a@redhat.com/

We can probably re-start the effort with yours and keep adding as
needed.

Do we want to have per-policy subdirs under sched?

Thanks,
Juri