[PATCH v4 00/26] sched/deadline: bandwidth accounting fixes and selftests

[PATCH v4 00/26] sched/deadline: bandwidth accounting fixes and selftests

[Mathieu Desnoyers]

Hi Ingo, Juri,

This series fixes twenty-two bugs of SCHED_DEADLINE, most of them in its
bandwidth accounting, then adds each runqueue's deadline bandwidth to
sched_debug and three selftests: twenty-six patches on tip sched/urgent,
the fixes first, as you asked, so that stable can take 1 to 22. Three of
the bugs take the kernel down (patches 5, 8 and 20), two leave
cpuset_mutex unusable (1 and 6), and most of the others leave a root
domain's total_bw or a runqueue's this_bw wrong: the root domain then
admits deadline tasks beyond its capacity, or refuses every one.

It applies on top of "locking/rtmutex: priority inheritance fixes and
tests", posted for Peter and Thomas: patch 8 rests on its patch 3. Patch
1 is Andrea Parri's, reviewed by Waiman, Kayra and Juri, carried here
because patches 6 and 14 build on it, now with my Signed-off-by; take it
from whichever route you prefer.

Your four fixes, Juri, which Ingo dropped from sched/urgent after the
first crashed Marek Szyprowski's hotplug test, are patches 16 (1/4,
without the task's mask, which has no active CPU during hotplug), 10
(2/4, without its double release, and without your Signed-off-by, which
is yours to add back), 2 (3/4) and 11 (4/4, extended). Marek found v2
fixed on the board that crashed.

The fixes, by patch number
==========================

 1  a retry in __sched_setscheduler() unlocks a cpuset_mutex it no
    longer holds (Andrea Parri);
 2  SCHED_FLAG_KEEP_PARAMS carries out half of a policy or class change,
    and leaks the bandwidth of a boosted task;
 3  SCHED_FLAG_KEEP_PARAMS admits stale parameters, read without a lock;
 4  a task boosted by a deadline waiter cannot leave for another policy
    (EBUSY);
 5  a debugfs write to a fair server divides by zero active CPUs while
    its CPU goes offline or comes online;
 6  sysrq-n takes cpuset_mutex under tasklist_lock and never releases
    it;
 7  a throttled task switched out and back before its 0-lag time never
    runs again;
 8  the owner boosted by a deadline waiter that leaves the class loops
    forever in replenish_dl_entity(), under the runqueue lock;
 9  a task back in the class runs on the deadline and runtime of the
    period it left, or of the donor that boosted it;
 10 a task that leaves the class asleep keeps its bandwidth in total_bw
    for good;
 11 SCHED_DEADLINE on a zombie reserves bandwidth nothing releases, and
    a zombie switched out of the class is taken off its cpuset's count
    twice;
 12 a rebuild of the accounting forgets the bandwidth of tasks that left
    the class, or died, before their 0-lag time, which is then released
    twice;
 13 a parameter change while a wakeup is in flight underflows the
    this_bw of a runqueue that never held the task's bandwidth;
 14 without cpusets, the first hotplug drops the bandwidth of every
    deadline task; with them, a drifting count has every rebuild drop a
    cpuset's tasks;
 15 a rebuild counts the bandwidth of a task at its last schedule, which
    nothing releases;
 16 admission, moves and rebuilds disagree on the root domain that
    counts a task's bandwidth, and take it from one that never had it;
 17 a cpuset move takes bandwidth off a root domain with no room made
    for it: total_bw of a partition reaches 2^64 - 262144;
 18 a rebuild of the sched domains between cpuset_can_attach() and
    cpuset_attach() drops the attach's reservation;
 19 an attach leaves in the destination's total_bw the reservation of a
    task that left the class, or died, before it;
 20 a rebuild counts a task on the first active CPU of its cpuset: an
    isolated one, none at all (BUG()), or not where
    dl_task_offline_migration() moved it;
 21 a replenishment timer left armed across a class round trip fires at
    the period of the reservation the task left;
 22 sched_setattr(SCHED_DEADLINE) on a boosted task leaves its
    runqueue's this_bw without the task's bandwidth, and a sugov
    kthread's own reservation is never admitted.

Patch 23 shows patch 13's bug from userspace; patches 24 to 26 add 55
cases in tools/testing/selftests/sched/deadline, as Juri suggested.
Patches 16 to 20 go together: 16 records the root domain that counts a
task's bandwidth, and 17 to 20 fix the attach and rebuild paths that the
record exposes. Patch 22 completes patch 13.

Patch 8 touches the rt_mutex code, Peter and Thomas: it records a
deadline waiter's reservation in its rt_mutex node, which grows struct
rt_mutex_waiter by 48 bytes, and takes a deadline boost from there; the
other classes keep the waiter's own priority, as since acd58620e415
("sched/rtmutex: Refactor rt_mutex_setprio()").

Testing
=======

The 55 cases pass every time with no kernel warning in 16 QEMU guests,
lockdep on: 8 of 8 vCPUs, and 2 each of 4 vCPUs, PREEMPT_RT, proxy
execution off and no CONFIG_CPUSETS. v7.4-rc1 fails 19 of them, which
patches 24 to 26 list, and three hotplug cases without cpusets. Kernels
stopped at single patches confirm the claims of patches 10, 12, 18 and
19. Patch 8 has no selftest: a debug delay or a stress test locks
v7.4-rc1 up, and the series runs the stress test for 600 seconds; the
locking API, ww_mutex, locktorture and futex selftests pass, with and
without PREEMPT_RT.

How this was developed
======================

These patches, their messages and the tests were written by an LLM
coding assistant under my direction and review. I am at home with the
concepts behind SCHED_DEADLINE, but not with its implementation, so I am
asking those who know this code for a careful reading of the fixes: a
reproducer that stops failing says that the symptom goes away, not that
the fix is the one you would have written. The bugs came from flaky
cgroup selftests of a feature I am developing, from reviewing Juri's
four fixes, from building deadline_bw_test, from reading the code, and
from the Sashiko AI reviews of v1 to v3, each finding checked against
the code first; the patches changed since v3 also went through a local
Sashiko run before posting.

Changes since v3
================

 - rebased on tip sched/urgent, which took Nickolai Zeldovich's this_bw
   fix for a dying task; patch 12 keeps its dl_task test;
 - new patches 4 (a boosted task may change its policy) and 8 (a
   deadline boost from the waiter's rt_mutex node);
 - 1: my Signed-off-by; 5: rq->online instead of cpu_active(); 6:
   cpuset_mutex decided again under the runqueue lock; 9: the deboost of
   a task with no parameters resets the donor's deadline and runtime;
 - 10: an inactive timer where the 0-lag time is ahead, and the
   parameters cleared with the bandwidth; 14: the rebuild walks the task
   list with cpusets too, without the per-cpuset count, and leaves out
   tasks with no bandwidth;
 - 16: special and boosted tasks with no bandwidth of their own are not
   moved; 18: waits in cpuset_full_lock() too; 20: the task's own mask
   first, and dl_task_offline_migration() counts a task a rebuild left
   out; 22: checks that the bandwidth is counted before a no-op, and
   leaves the runqueue alone when it is kept;
 - deadline_bw_test: boosted_owner_policy, deboost_resets_period and
   deboost_released_resets_period added, three cases fixed to reach the
   state they are named for, CPUs from the online set, error paths that
   join their threads;
 - deadline_switch_test: its children die with the test
   (PR_SET_PDEATHSIG);
 - shorter commit messages.

Not changed: a departed task boosted into the class is counted
conservatively across a rebuild, never twice, and the EPERM of a
sleeping task moved to another partition is the baseline's affinity
check.

Based on tip sched/urgent, 006991aa8997, with "locking/rtmutex: priority
inheritance fixes and tests" applied first:
https://lore.kernel.org/lkml/20261106235931.1478800-1-mathieu.desnoyers@efficios.com

v3: https://lore.kernel.org/all/20261105193359.1450487-1-mathieu.desnoyers@efficios.com/

Andrea Parri (1):
  sched: Fix cpuset_mutex double unlock in __sched_setscheduler()

Juri Lelli (1):
  sched/deadline: Reject SCHED_DEADLINE on exited tasks

Mathieu Desnoyers (24):
  sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or
    the class
  sched/deadline: Do not admit bandwidth for parameters that are kept
  sched: Let a task boosted by a deadline waiter change its policy
  sched/deadline: Do not divide by zero active CPUs of a root domain
  sched: Take cpuset_mutex for a deadline policy change only with pi
  sched/deadline: Do not lose a throttled task switched out and back
  sched/deadline: Take a deadline boost from the waiter's rt_mutex node
  sched/deadline: Reset the deadline and the runtime with the parameters
  sched/deadline: Release the bandwidth of a task that leaves the class
    asleep
  sched/deadline: Keep departing bandwidth across a root domain rebuild
  sched/deadline: Do not take a waking task's bandwidth off a runqueue
    it left
  sched/deadline: Rebuild the root domains' accounting from the task
    list
  sched/deadline: Leave a dying task out of a rebuild of the accounting
  sched/deadline: Record the root domain holding a task's bandwidth
  sched/deadline: Move a task's bandwidth only where room was made
  cgroup/cpuset: Wait for an attach before rebuilding the sched domains
  cgroup/cpuset: Give back the bandwidth an attach did not move
  sched/deadline: Credit a rebuild to the CPU recorded with the task's
    bandwidth
  sched/deadline: Cancel the replenishment timer with the parameters
  sched/deadline: Count the bandwidth a boosted task is given
  sched/debug: Show the deadline bandwidth of each runqueue
  selftests/sched: Add a test of SCHED_DEADLINE policy round trips
  selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain
    rebuild
  selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

 include/linux/cpuset.h                        |    9 -
 include/linux/sched.h                         |   47 +-
 include/linux/sched/deadline.h                |   38 +-
 include/linux/sched/rt.h                      |   11 -
 kernel/cgroup/cgroup.c                        |    3 -
 kernel/cgroup/cpuset-internal.h               |    8 +-
 kernel/cgroup/cpuset.c                        |  126 +-
 kernel/locking/rtmutex.c                      |   41 +-
 kernel/locking/rtmutex_api.c                  |    2 +-
 kernel/locking/rtmutex_common.h               |    5 +
 kernel/sched/core.c                           |   88 +-
 kernel/sched/deadline.c                       |  799 +++-
 kernel/sched/debug.c                          |   11 +-
 kernel/sched/sched.h                          |   23 +-
 kernel/sched/syscalls.c                       |   87 +-
 kernel/sched/topology.c                       |    9 +
 tools/testing/selftests/Makefile              |    1 +
 .../selftests/sched/deadline/.gitignore       |    3 +
 .../testing/selftests/sched/deadline/Makefile |    8 +
 tools/testing/selftests/sched/deadline/config |    4 +
 .../sched/deadline/deadline_bw_test.c         | 3902 +++++++++++++++++
 .../sched/deadline/deadline_rebuild_test.c    |  462 ++
 .../sched/deadline/deadline_switch_test.c     |  440 ++
 .../testing/selftests/sched/deadline/settings |    1 +
 24 files changed, 5847 insertions(+), 281 deletions(-)
 create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
 create mode 100644 tools/testing/selftests/sched/deadline/Makefile
 create mode 100644 tools/testing/selftests/sched/deadline/config
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/settings


base-commit: 006991aa89974378ee579416dba87adc22cd60fd
prerequisite-patch-id: 382feddeb27d226d8711d23036c8ae0c8629c680
prerequisite-patch-id: 72e83c74aa52fcd32afb2909b071f40c82b0ef10
prerequisite-patch-id: cfd50da1cef9606aafc687658a1ac79f74969703
prerequisite-patch-id: 2e79dc888982d901d6e8b3784790ef71037482a9
prerequisite-patch-id: f7338b3af8aeb24a47d19dff76d7e3180120b9aa
-- 
2.47.3

[PATCH v4 01/26] sched: Fix cpuset_mutex double unlock in __sched_setscheduler()

[Mathieu Desnoyers]

From: Andrea Parri <parri.andrea@gmail.com>

A sched_setattr() with SCHED_FLAG_KEEP_POLICY on a SCHED_DEADLINE task
unlocks cpuset_mutex twice if the task's policy changes concurrently:

  WARNING: bad unlock balance detected!
  7.4.0-rc1 #1 Not tainted
  -------------------------------------
  repro/126 is trying to release lock (cpuset_mutex) at:
  [<ffffffff9a17cacc>] __sched_setscheduler+0x9cc/0xc40
  but there are no more locks to release!
  ...
  Call Trace:
   print_unlock_imbalance_bug.part.0+0xfb/0x110
   lock_release+0x286/0x2e0
   __mutex_unlock_slowpath+0x1a/0x400
   __sched_setscheduler+0x9cc/0xc40
   __x64_sys_sched_setattr+0xa3/0x1b0
   do_syscall_64+0x106/0x5f0
   entry_SYSCALL_64_after_hwframe+0x77/0x7f

__sched_setscheduler() samples p->policy before taking any lock, and
restarts from the recheck label if p->policy differs under the rq lock.
The restart drops cpuset_mutex but leaves cpuset_locked set, so a pass
that no longer needs the mutex still unlocks it on its way out:

  CPU 1                                CPU 2
  __sched_setscheduler(KEEP_POLICY)
    policy = p->policy /* DEADLINE */
                                       __sched_setscheduler(SCHED_OTHER)
                                         p->policy = SCHED_OTHER
    cpuset_locked = true
    cpuset_lock()
    rq = task_rq_lock(p)
    oldpolicy != p->policy
      cpuset_unlock()
      goto recheck
    /* no cpuset_lock() this time */
    ...
    if (cpuset_locked)
      cpuset_unlock()

Without CONFIG_DEBUG_MUTEXES, or CONFIG_DEBUG_RT_MUTEXES on PREEMPT_RT,
nothing warns. The second unlock still clears the owner field regardless
of who holds the mutex, so another task can lose cpuset_mutex inside its
critical section. Entering SCHED_DEADLINE needs CAP_SYS_NICE, but a
racing caller that owns the task does not need it if it samples a policy
other than SCHED_DEADLINE.

Clear cpuset_locked when dropping the mutex for the restart. This should be
safe because the flag is only tested on the paths leaving the next pass,
after that pass has set it again if it took the mutex.

The following program, run as root, reproduces the splat, which no longer
appears with this applied. On PREEMPT_RT it crashes the kernel within
seconds, which this also fixes:

  #define _GNU_SOURCE
  #include <pthread.h>
  #include <stdint.h>
  #include <sys/syscall.h>
  #include <unistd.h>

  struct attr {
      uint32_t size, policy;
      uint64_t flags;
      int32_t nice;
      uint32_t prio;
      uint64_t runtime, deadline, period;
  };

  static volatile int stop;
  static volatile pid_t target;

  static void set(struct attr a)
  {
      a.size = sizeof(a);
      syscall(SYS_sched_setattr, target, &a, 0);
  }

  static void *sleeper(void *arg)
  {
      target = syscall(SYS_gettid);
      while (!stop)
          usleep(1000);
      return arg;
  }

  static void *switcher(void *arg)
  {
      while (!stop) {
          /* SCHED_DEADLINE, 2us every 1s */
          set((struct attr){ .policy = 6, .runtime = 2000,
                             .deadline = 1000000000,
                             .period = 1000000000 });
          set((struct attr){ .policy = 0 }); /* SCHED_OTHER */
      }
      return arg;
  }

  static void *racer(void *arg)
  {
      /* KEEP_POLICY | KEEP_PARAMS */
      while (!stop)
          set((struct attr){ .flags = 0x18 });
      return arg;
  }

  int main(void)
  {
      pthread_t t[5];
      int i;

      pthread_create(&t[0], NULL, sleeper, NULL);
      while (!target)
          usleep(100);
      pthread_create(&t[1], NULL, switcher, NULL);
      for (i = 2; i < 5; i++)
          pthread_create(&t[i], NULL, racer, NULL);
      sleep(30);
      stop = 1;
      for (i = 0; i < 5; i++)
          pthread_join(t[i], NULL);
      return 0;
  }

An LLM helped find the PREEMPT_RT crash and write this patch and its test.

Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20261016062629.2102883-1-sh_def%40163.com
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 45a127deeadd..2af550f92ac6 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -640,8 +640,10 @@ int __sched_setscheduler(struct task_struct *p,
 	if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
 		policy = oldpolicy = -1;
 		task_rq_unlock(rq, p, &rf);
-		if (cpuset_locked)
+		if (cpuset_locked) {
 			cpuset_unlock();
+			cpuset_locked = false;
+		}
 		goto recheck;
 	}
 
-- 
2.47.3

[PATCH v4 02/26] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class

[Mathieu Desnoyers]

SCHED_FLAG_KEEP_PARAMS keeps the parameters of a task:
__sched_setscheduler() skips __setscheduler_params(), which is what sets
p->policy, and the sched_class assignment with it. A request that asks
for another policy is then carried out by half, and succeeds:

 - a class change, SCHED_DEADLINE to SCHED_OTHER, sets DEQUEUE_CLASS:
   sched_change_begin() runs switched_from_dl() on a task that stays in
   the class, and sched_change_end() switched_to_dl(). A sleeping task
   has its bandwidth taken off its runqueue there, and taken off again
   when it does leave: this_bw underflows;
 - a policy change within the class, SCHED_OTHER to SCHED_BATCH, returns
   0 and leaves the task where it was;
 - a task that priority inheritance boosted into the deadline class is
   of that class already, and a request for SCHED_DEADLINE finds
   prev_class == next_class: sched_dl_overflow() admits it and adds its
   bandwidth, which the skipped __setscheduler_params() never gives to
   the task. It keeps SCHED_NORMAL and a dl_bw of 0, so nothing releases
   that bandwidth, and total_bw of the root domain holds it for good.

Refuse a request for another policy or another class, before
sched_dl_overflow() reserves bandwidth, since nothing gives back what it
reserves: a refusal after it would leak the runtime that sched_setattr()
fills in from the task's slice, up to 100ms with a custom one, into
total_bw. A refused request also leaves sched_reset_on_fork as it was.

A SCHED_OTHER owner of a PTHREAD_PRIO_INHERIT mutex with a slice of
40ms, boosted by a deadline waiter blocked on it, leaves 0.4 of a CPU
behind at every such call: total_bw 419424 -> 838854 and there to stay,
in 8-CPU VMs of v7.4-rc1. With this change the call is refused and
total_bw is as it was. The selftest added later in this series has the
case, params_keep_boosted.

The class change was found by Juri Lelli from a report of Tejun Heo, and
refused, after the reservation, by his "sched: Reject
SCHED_FLAG_KEEP_PARAMS when it would change scheduling class", since
dropped from tip. The policy change and the boosted case were found by
the Sashiko AI review of the first posting of this series.

Link: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-3-dd761fe005b4@redhat.com/
Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 2af550f92ac6..6819f0fea3c5 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -647,17 +647,6 @@ int __sched_setscheduler(struct task_struct *p,
 		goto recheck;
 	}
 
-	/*
-	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
-	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
-	 * is available.
-	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
-		retval = -EBUSY;
-		goto unlock;
-	}
-
-	p->sched_reset_on_fork = reset_on_fork;
 	oldprio = p->prio;
 
 	newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
@@ -677,9 +666,38 @@ int __sched_setscheduler(struct task_struct *p,
 	prev_class = p->sched_class;
 	next_class = __setscheduler_class(policy, newprio);
 
+	/*
+	 * SCHED_FLAG_KEEP_PARAMS prevents the policy and the class change
+	 * from actually happening: it skips __setscheduler_params() and the
+	 * sched_class assignment. Reject the contradictory requests that
+	 * ask for either, rather than carry out half of what they ask.
+	 *
+	 * A class change with the policy kept is one of them: the class of a
+	 * policy depends on the priority that priority inheritance leaves,
+	 * and on whether a BPF scheduler is in charge.
+	 */
+	if ((attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (policy != p->policy || prev_class != next_class)) {
+		retval = -EINVAL;
+		goto unlock;
+	}
+
 	if (prev_class != next_class)
 		queue_flags |= DEQUEUE_CLASS;
 
+	/*
+	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
+	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
+	 * is available. Last of the checks: nothing gives back the
+	 * bandwidth it reserves.
+	 */
+	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+		retval = -EBUSY;
+		goto unlock;
+	}
+
+	p->sched_reset_on_fork = reset_on_fork;
+
 	scx_prepare_setscheduler(p, policy);
 
 	scoped_guard (sched_change, p, queue_flags) {
-- 
2.47.3

[PATCH v4 03/26] sched/deadline: Do not admit bandwidth for parameters that are kept

[Mathieu Desnoyers]

sys_sched_setattr() fills a SCHED_FLAG_KEEP_PARAMS request in from the
task itself (get_params()), with no lock held, and
__sched_setscheduler() then skips __setscheduler_params(): the task
keeps the parameters it has, and its dl_bw with them.
sched_dl_overflow() runs all the same.

A parameter change of the task's own that lands in between leaves the
request stale, and dl_param_changed() takes it for a change: total_bw of
the root domain, and this_bw and running_bw of its runqueue through
dl_change_utilization(), then hold a bandwidth the task does not have,
and the release at its 0-lag time, which takes the task's own dl_bw,
leaves the difference behind.

There is nothing to admit for parameters that are kept: skip the check.

Found by reading, after the Sashiko AI review of the first posting of
this series raised the case the previous patch refuses. Not reproduced:
it needs a parameter change of a deadline task to land inside the window
of a SCHED_FLAG_KEEP_PARAMS request on the same task.

Fixes: a509a7cd7974 ("sched/uclamp: Extend sched_setattr() to support utilization clamping")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 6819f0fea3c5..b8c036d03d6f 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -690,8 +690,14 @@ int __sched_setscheduler(struct task_struct *p,
 	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
 	 * is available. Last of the checks: nothing gives back the
 	 * bandwidth it reserves.
+	 *
+	 * Nothing to admit for parameters that are kept
+	 * (SCHED_FLAG_KEEP_PARAMS): they are the ones the task has, and
+	 * sys_sched_setattr() read them from it with no lock held, so a
+	 * change of its own that landed in between would leave them stale.
 	 */
-	if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
+	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
 		retval = -EBUSY;
 		goto unlock;
 	}
-- 
2.47.3

[PATCH v4 04/26] sched: Let a task boosted by a deadline waiter change its policy

[Mathieu Desnoyers]

__sched_setscheduler() admits bandwidth for a change of or into
SCHED_DEADLINE: it calls sched_dl_overflow() where the new policy is
SCHED_DEADLINE or dl_task(p). dl_task() tests the priority, not the
policy: it holds for a task of another class boosted by a deadline waiter
blocked on an rt_mutex it holds. sched_dl_overflow() has no branch for a
change of such a task to another policy, which does not have the policy
to leave, and admits nothing: it returns -1, and the change fails with
EBUSY. sched_setscheduler() to SCHED_OTHER or SCHED_FIFO on the owner of
a priority-inheriting mutex fails while a deadline task waits on it, and
sysrq-n leaves the owner as it is.

Test the policy: a task of the class has it, and a boosted one has
nothing to admit or to leave. The deadline_bw_test selftest added later
in this series has the case, boosted_owner_policy.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b8c036d03d6f..bdffe5452dc8 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -689,7 +689,9 @@ int __sched_setscheduler(struct task_struct *p,
 	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
 	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
 	 * is available. Last of the checks: nothing gives back the
-	 * bandwidth it reserves.
+	 * bandwidth it reserves. A task boosted into the class by priority
+	 * inheritance has nothing to admit nor to leave: the policy
+	 * decides, not the priority.
 	 *
 	 * Nothing to admit for parameters that are kept
 	 * (SCHED_FLAG_KEEP_PARAMS): they are the ones the task has, and
@@ -697,7 +699,8 @@ int __sched_setscheduler(struct task_struct *p,
 	 * change of its own that landed in between would leave them stale.
 	 */
 	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
-	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
+	    (dl_policy(policy) || task_has_dl_policy(p)) &&
+	    sched_dl_overflow(p, policy, attr)) {
 		retval = -EBUSY;
 		goto unlock;
 	}
-- 
2.47.3

[PATCH v4 05/26] sched/deadline: Do not divide by zero active CPUs of a root domain

[Mathieu Desnoyers]

__dl_sub() and __dl_add() spread a change of a root domain's total_bw
over the extra_bw of its active CPUs, dividing it by their number,
dl_bw_cpus(). That number is 0 for a root domain whose CPUs all went
offline, such as the default root domain, which holds the offline CPUs:
the division traps.

A debugfs write of a fair server's runtime or period divides there while
its CPU goes offline or comes online: the CPU sits in the default root
domain from the rebuild in sched_cpu_deactivate() until it is offline,
and from its online until sched_cpu_activate(), and
commit 4043f5498416 ("sched/deadline: Reject debugfs dl_server writes
for offline CPUs") refuses the write only once the CPU is offline. A
write that leaves the server's bandwidth as it is passes admission
there. Writing a fair server's own runtime back in a loop while its CPU
goes offline and online oopses on the first round, in 8-CPU VMs of
v7.4-rc1:

  Oops: divide error: 0000 [#1] SMP NOPTI
  RIP: 0010:dl_server_apply_params+0x335/0x380
  Call Trace:
   sched_server_write_common.isra.0+0x1c0/0x260
   full_proxy_write+0x58/0x90
   vfs_write+0xcb/0x550

inactive_task_timer() can divide too, found by reading: it releases the
bandwidth of a task that left SCHED_DEADLINE, or died, from the root
domain of its CPU, which may have gone offline since. With admission
control off, so can sched_setattr(SCHED_DEADLINE) of a task asleep on an
offline CPU.

Refuse the write while the runqueue is offline, as once the CPU is: the
server's bandwidth is not counted then, and the rebuild at the CPU's
activation counts what the server has. rq->online is set under the
runqueue lock by the attach to a root domain that counts the CPU, and
cleared before the rebuild that drops it, where cpu_active() is set
before the attach and would be read without cpus_read_lock(). With no
active CPU there is no extra_bw to update either: change total_bw alone,
for the paths that have no write to refuse. 20 rounds of the reproducer
leave no warning then, its writes refused with EBUSY in the window.

Fixes: daec57983670 ("sched/deadline: Reclaim bandwidth not used by dl tasks")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 11 +++++++++--
 kernel/sched/debug.c    |  9 ++++++++-
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index dfae08625342..00983487bff3 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -189,18 +189,25 @@ void __dl_update(struct dl_bw *dl_b, s64 bw)
 	}
 }
 
+/*
+ * @cpus is the number of active CPUs of the root domain, whose extra_bw
+ * shares the change: none once they all went offline, and total_bw alone
+ * changes then.
+ */
 static inline
 void __dl_sub(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw -= tsk_bw;
-	__dl_update(dl_b, (s32)tsk_bw / cpus);
+	if (cpus)
+		__dl_update(dl_b, (s32)tsk_bw / cpus);
 }
 
 static inline
 void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 {
 	dl_b->total_bw += tsk_bw;
-	__dl_update(dl_b, -((s32)tsk_bw / cpus));
+	if (cpus)
+		__dl_update(dl_b, -((s32)tsk_bw / cpus));
 }
 
 static inline bool
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..5f8303630296 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -410,7 +410,14 @@ static ssize_t sched_server_write_common(struct file *filp, const char __user *u
 			return  -EINVAL;
 		}
 
-		if (!cpu_online(cpu_of(rq)))
+		/*
+		 * Nor while its runqueue is offline, the CPU going offline or
+		 * coming online: the server's bandwidth is not counted in a
+		 * root domain then, which has no active CPU to spread a
+		 * change over. rq->online follows the attach to a root domain
+		 * under this lock; cpu_active() is set before it.
+		 */
+		if (!rq->online)
 			return -EBUSY;
 
 		update_rq_clock(rq);
-- 
2.47.3

[PATCH v4 06/26] sched: Take cpuset_mutex for a deadline policy change only with pi

[Mathieu Desnoyers]

__sched_setscheduler() takes cpuset_mutex for a change of or into
SCHED_DEADLINE, and releases it on success only where pi is set, with
rt_mutex_adjust_pi(). normalize_rt_tasks(), the one caller with pi
unset, then takes a mutex under read_lock(&tasklist_lock) for every
deadline task it finds, and never releases it: sysrq-n on a system with
a deadline task sleeps in atomic context, and every later change of or
into SCHED_DEADLINE blocks on cpuset_mutex for good.

Commit 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex") made the
lock depend on the policy where it depended on pi before, and left the
release where it was. Take it where pi is set, as before:
normalize_rt_tasks() takes tasks out of the class, which admits no
bandwidth and needs no stable cpusets.

The policy the decision reads is the task's before the runqueue lock, and
only a caller that gave none has it read again under the lock. Given
another policy than SCHED_DEADLINE, a change into the class by another
caller in between makes the call a change out of it without the mutex.
Decide again under the lock, where the policy is settled: a change out
of the class needs no mutex, but the decision follows the policy the
call acts on.

sysrq-n on v7.4-rc1 with a deadline task, lockdep on, in 8-CPU VMs:

  sysrq: Nice All RT Tasks
  BUG: sleeping function called from invalid context at kernel/locking/mutex.c:623
  in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 4356, name: deadline_switch
  locks held by deadline_switch/4356: 4, last CPU#3:
   ...
   #3: ffffffff83006098 (tasklist_lock){.+.+}-{3:3}, at: normalize_rt_tasks+0x34/0x150

and a sched_setattr(SCHED_DEADLINE) after it never returns, in 2 VMs of
2. The deadline_switch_test selftest added later in this series has the
case, sysrq_normalize, which passes with this change.

Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index bdffe5452dc8..c3f404cddb9e 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -554,9 +554,10 @@ int __sched_setscheduler(struct task_struct *p,
 
 	/*
 	 * SCHED_DEADLINE bandwidth accounting relies on stable cpusets
-	 * information.
+	 * information. Not where pi is unset: normalize_rt_tasks() takes
+	 * tasks out of the class under tasklist_lock, which admits none.
 	 */
-	if (dl_policy(policy) || dl_policy(p->policy)) {
+	if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
 		cpuset_locked = true;
 		cpuset_lock();
 	}
@@ -647,6 +648,16 @@ int __sched_setscheduler(struct task_struct *p,
 		goto recheck;
 	}
 
+	/*
+	 * The policy read before this lock decided against cpuset_mutex: a
+	 * change into the class since makes this one a change out of it.
+	 * Decide again with the lock held.
+	 */
+	if (pi && !cpuset_locked && task_has_dl_policy(p)) {
+		task_rq_unlock(rq, p, &rf);
+		goto recheck;
+	}
+
 	oldprio = p->prio;
 
 	newprio = __normal_prio(policy, attr->sched_priority, attr->sched_nice);
-- 
2.47.3

[PATCH v4 07/26] sched/deadline: Do not lose a throttled task switched out and back

[Mathieu Desnoyers]

A SCHED_DEADLINE task that is throttled, switched to another policy and
switched back before its 0-lag time can be left throttled with nothing
to replenish it: on no dl_rq and with no timer pending, it never runs
again, not even for a fatal signal, until another switch out of
SCHED_DEADLINE requeues it:

 1. the task exhausts its runtime: dl_throttled is set, and the
    replenishment timer armed;
 2. a switch to SCHED_OTHER arms the inactive timer for the 0-lag time,
    task_non_contending(), the parameters staying until then;
 3. the replenishment timer fires, finds a task out of the class and
    returns, dl_throttled still set;
 4. a switch back before the 0-lag time cancels the inactive timer, and
    enqueue_dl_entity() leaves a throttled task to its replenishment
    timer, which fired already.

The window is the overrun scaled by period/runtime: about 100us for an
overrun of 20us at 20% bandwidth. The replenishment timer reset a task
it found out of the class until commit 209a0cbda7a0 ("sched/deadline:
Improve the tracking of active utilization") moved that reset to the
inactive timer.

Give the task the replenishment the timer owes it, as the timer does for
a throttled task that is not queued: the overrun comes off the periods
that follow, the deadline moves past it, and dl_throttled and dl_yielded
are cleared. Clearing dl_throttled alone, as the first version of this
patch did, left the overrun, the deadline and a yield to throttle the
task again, as the Sashiko AI review of the first posting of this series
pointed out.

The deadline_switch_test selftest added later in this series fails its
throttled_round_trip case 8 times out of 8 on v7.3 and v7.4-rc1, and
passes it 8 of 8 with this change. A cgroup selftest under development
found it, hanging about once in 1350 runs.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 00983487bff3..6cf33045d93a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1232,10 +1232,23 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 
 	/*
 	 * The task might have changed its scheduling policy to something
-	 * different than SCHED_DEADLINE (through switched_from_dl()).
+	 * different than SCHED_DEADLINE (through switched_from_dl()). It
+	 * keeps its parameters until its 0-lag time and may come back to
+	 * SCHED_DEADLINE before then, switched_to_dl() cancelling the
+	 * inactive timer: give it the replenishment this timer owes it, as
+	 * the !queued case below does, and leave the enqueue to its return.
+	 * A throttle left behind would never be lifted; an overrun left in
+	 * its runtime would be forgiven by the new period its enqueue
+	 * starts; and a yield left set would throttle it again at once.
 	 */
-	if (!dl_task(p))
+	if (!dl_task(p)) {
+		if (dl_se->dl_throttled && !is_dl_boosted(dl_se)) {
+			sched_clock_tick();
+			update_rq_clock(rq);
+			replenish_dl_entity(dl_se);
+		}
 		goto unlock;
+	}
 
 	/*
 	 * The task might have been boosted by someone else and might be in the
-- 
2.47.3

[PATCH v4 08/26] sched/deadline: Take a deadline boost from the waiter's rt_mutex node

[Mathieu Desnoyers]

A task boosted by a deadline waiter of an rt_mutex it holds runs on the
waiter's reservation: pi_se points at the waiter's entity, whose
dl_runtime, dl_deadline and dl_period replenish_dl_entity() reads. The
waiter can leave SCHED_DEADLINE meanwhile. __sched_setscheduler()
changes it under its runqueue lock, and walks the chain to the owner,
rt_mutex_adjust_pi(), once that lock is released. In between,
inactive_task_timer() at the waiter's 0-lag time, or
task_non_contending() in the change itself for a waiter woken but not
yet off the lock, clears the waiter's parameters. The owner is still
boosted, and the next time its runtime runs out, at a tick or a switch,
replenish_dl_entity() never leaves, under the runqueue lock with
interrupts off:

	while (dl_se->runtime <= 0) {
		dl_se->deadline += pi_of(dl_se)->dl_period;
		dl_se->runtime += pi_of(dl_se)->dl_runtime;
	}

Its CPU locks up, and so does the one walking the chain, waiting for the
owner's runqueue lock.

A copy of the waiter's parameters taken at the boost is not enough:
rt_mutex_setprio() reads the waiter with no lock of the waiter's, and a
copy taken as the waiter leaves the class takes them cleared. The
waiter's rt_mutex node records its priority and deadline under its
pi_lock, which every change of its parameters holds, and every waiter
enters its owner's pi_waiters with them current since "locking/rtmutex:
Clone a waiter's sort keys as it enters pi_waiters", which this series
applies on top of. Record there the reservation the waiter runs on, its
own or the one a boost lent it, and take a deadline boost from the node
of the top pi waiter: the priority, the deadline that decides whether a
deadline owner is boosted, and a copy of the reservation, which the
boosted task runs on from then on.

rt_mutex_setprio(), rt_effective_prio() and __setscheduler_dl_pi() read
that node, rt_effective_prio() from the top pi waiter rather than
p->pi_top_task, which try_to_take_rt_mutex() can leave stale. The chain
walk and rt_mutex_adjust_pi() compare a deadline waiter's reservation
too, so that a change of its parameters alone reaches the owner. struct
rt_mutex_waiter, on the stack of every rt_mutex sleeper and, with
PREEMPT_RT, of every sleeping lock waiter, grows from 112 to 160 bytes.
rt_mutex_get_top_task() has no user left.

Tested in VMs of 8 CPUs, 2 or 3 per case, with the same result in each:

 - a debug delay of 300ms before rt_mutex_adjust_pi() opens the window:
   a 1ms/100ms deadline waiter of a spinning owner, switched to
   SCHED_OTHER with its inactive timer armed, locks up v7.4-rc1 and this
   series without this patch, the owner's CPU warning in
   replenish_dl_entity(); with this patch the owner is deboosted when
   the change returns;
 - without the delay, a deadline waiter switched in and out of
   SCHED_DEADLINE 20000 times a second while a thread boosts the owner
   again thousands of times a second, with a 50us timed lock: a copy
   taken from the waiter itself locks the owner's CPU up within a
   second, and this patch runs 600 seconds;
 - a deadline waiter that changes class while another is ahead of it,
   then is promoted, boosts the owner as its class says, either way;
 - the locking API, ww_mutex, locktorture rtmutex_lock and futex
   selftests pass, with and without PREEMPT_RT.

Found by the Sashiko AI review of the third posting of this series; the
window of a copy taken from the waiter itself, and a node not current on
promotion, by reviews of this posting by a second model.

Fixes: a5e7be3b28a2 ("sched/deadline: Clear dl_entity params when setscheduling to different class")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h           | 21 +++++++--
 include/linux/sched/deadline.h  | 17 ++++++++
 include/linux/sched/rt.h        | 11 -----
 kernel/locking/rtmutex.c        | 41 +++++++++++++++---
 kernel/locking/rtmutex_api.c    |  2 +-
 kernel/locking/rtmutex_common.h |  5 +++
 kernel/sched/core.c             | 69 ++++++++++++++++++++++++++----
 kernel/sched/deadline.c         | 76 ++++++++++++++++++++++++---------
 kernel/sched/sched.h            | 19 +++------
 kernel/sched/syscalls.c         |  6 +--
 10 files changed, 198 insertions(+), 69 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..f0b6a8a91323 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -640,6 +640,13 @@ struct sched_rt_entity {
 struct rq_flags;
 typedef struct task_struct *(*dl_server_pick_f)(struct sched_dl_entity *, struct rq_flags *rf);
 
+/* The parameters of a deadline reservation: runtime, relative deadline, period. */
+struct sched_dl_params {
+	u64				runtime;
+	u64				deadline;
+	u64				period;
+};
+
 struct sched_dl_entity {
 	struct rb_node			rb_node;
 
@@ -706,6 +713,9 @@ struct sched_dl_entity {
 	 * contributes to the root domain's total_bw. Only meaningful for server
 	 * entities (@dl_server == 1). Allows toggling the reservation on/off
 	 * without losing the configured @dl_runtime/@dl_period.
+	 *
+	 * @dl_boosted tells if priority inheritance boosted the entity: it
+	 * runs on the parameters copied in @pi_params.
 	 */
 	unsigned int			dl_throttled      : 1;
 	unsigned int			dl_yielded        : 1;
@@ -718,6 +728,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer_running  : 1;
 	unsigned int			dl_defer_idle     : 1;
 	unsigned int			dl_bw_attached    : 1;
+	unsigned int			dl_boosted        : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
@@ -745,11 +756,13 @@ struct sched_dl_entity {
 
 #ifdef CONFIG_RT_MUTEXES
 	/*
-	 * Priority Inheritance. When a DEADLINE scheduling entity is boosted
-	 * pi_se points to the donor, otherwise points to the dl_se it belongs
-	 * to (the original one/itself).
+	 * Priority inheritance: the parameters of the reservation the
+	 * entity runs on while @dl_boosted, copied at the boost from the
+	 * node of the waiter that lends them (rt_waiter_node). Not read
+	 * from the waiter: it may leave SCHED_DEADLINE and have its own
+	 * cleared before the chain walk deboosts this one.
 	 */
-	struct sched_dl_entity *pi_se;
+	struct sched_dl_params		pi_params;
 #endif
 };
 
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..416ca548a038 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -10,6 +10,23 @@
 
 #include <linux/sched.h>
 
+#ifdef CONFIG_RT_MUTEXES
+/*
+ * The reservation @dl_se runs on: its own, or the one a priority
+ * inheritance boost lent it. Under its task's pi_lock or rq lock.
+ */
+static inline struct sched_dl_params dl_entity_params(const struct sched_dl_entity *dl_se)
+{
+	if (dl_se->dl_boosted)
+		return dl_se->pi_params;
+	return (struct sched_dl_params){
+		.runtime	= dl_se->dl_runtime,
+		.deadline	= dl_se->dl_deadline,
+		.period		= dl_se->dl_period,
+	};
+}
+#endif
+
 static inline bool dl_prio(int prio)
 {
 	return unlikely(prio < MAX_DL_PRIO);
diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h
index 922935cc3383..a790428734db 100644
--- a/include/linux/sched/rt.h
+++ b/include/linux/sched/rt.h
@@ -57,20 +57,9 @@ extern void rt_mutex_schedule(void);
 extern void rt_mutex_post_schedule(void);
 extern void rt_mutex_futex_post_schedule(void);
 
-/*
- * Must hold either p->pi_lock or task_rq(p)->lock.
- */
-static inline struct task_struct *rt_mutex_get_top_task(struct task_struct *p)
-{
-	return p->pi_top_task;
-}
 extern void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task);
 extern void rt_mutex_adjust_pi(struct task_struct *p);
 #else
-static inline struct task_struct *rt_mutex_get_top_task(struct task_struct *task)
-{
-	return NULL;
-}
 # define rt_mutex_adjust_pi(p)		do { } while (0)
 #endif
 
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index e89f0f0e604b..c0869c63a476 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -363,16 +363,21 @@ static __always_inline int __waiter_prio(struct task_struct *task)
 }
 
 /*
- * Update the waiter->tree copy of the sort keys.
+ * Update the waiter->tree copy of the sort keys, and of the reservation a
+ * deadline waiter lends the owner: under @task's pi_lock, which every
+ * change of its parameters holds, so that a boost never runs on those of
+ * a waiter that left the class before the chain walk came.
  */
 static __always_inline void
 waiter_update_prio(struct rt_mutex_waiter *waiter, struct task_struct *task)
 {
 	lockdep_assert_held(&waiter->lock->wait_lock);
+	lockdep_assert_held(&task->pi_lock);
 	lockdep_assert(RB_EMPTY_NODE(&waiter->tree.entry));
 
 	waiter->tree.prio = __waiter_prio(task);
 	waiter->tree.deadline = task->dl.deadline;
+	waiter->tree.dl = dl_entity_params(&task->dl);
 }
 
 /*
@@ -387,6 +392,7 @@ waiter_clone_prio(struct rt_mutex_waiter *waiter, struct task_struct *task)
 
 	waiter->pi_tree.prio = waiter->tree.prio;
 	waiter->pi_tree.deadline = waiter->tree.deadline;
+	waiter->pi_tree.dl = waiter->tree.dl;
 }
 
 /*
@@ -433,6 +439,27 @@ static __always_inline int rt_waiter_node_equal(struct rt_waiter_node *left,
 	return 1;
 }
 
+/*
+ * Whether the lock tree copy of @waiter's node still has what @task would
+ * record in it: its sort keys and, for a deadline task, the reservation
+ * it lends the owner, which a change of its parameters moves without them
+ * (sched_setattr() leaves the deadline of a task that is not running).
+ */
+static __always_inline bool rt_waiter_node_current(struct rt_mutex_waiter *waiter,
+						   struct task_struct *task)
+{
+	struct sched_dl_params dl;
+
+	if (!rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task)))
+		return false;
+	if (!dl_prio(waiter->tree.prio))
+		return true;
+	dl = dl_entity_params(&task->dl);
+	return dl.runtime == waiter->tree.dl.runtime &&
+	       dl.deadline == waiter->tree.dl.deadline &&
+	       dl.period == waiter->tree.dl.period;
+}
+
 static inline bool rt_mutex_steal(struct rt_mutex_waiter *waiter,
 				  struct rt_mutex_waiter *top_waiter)
 {
@@ -831,13 +858,13 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
 	}
 
 	/*
-	 * If the waiter priority is the same as the task priority
-	 * then there is no further priority adjustment necessary.  If
-	 * deadlock detection is off, we stop the chain walk. If its
-	 * enabled we continue, but stop the requeueing in the chain
-	 * walk.
+	 * If the waiter priority is the same as the task priority, and
+	 * so is the reservation of a deadline task, then there is no
+	 * further priority adjustment necessary.  If deadlock detection
+	 * is off, we stop the chain walk. If its enabled we continue,
+	 * but stop the requeueing in the chain walk.
 	 */
-	if (rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+	if (rt_waiter_node_current(waiter, task)) {
 		if (!detect_deadlock)
 			goto out_unlock_pi;
 		else
diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c
index eb18b094473c..ba907636cc3c 100644
--- a/kernel/locking/rtmutex_api.c
+++ b/kernel/locking/rtmutex_api.c
@@ -510,7 +510,7 @@ void __sched rt_mutex_adjust_pi(struct task_struct *task)
 	raw_spin_lock_irqsave(&task->pi_lock, flags);
 
 	waiter = task->pi_blocked_on;
-	if (!waiter || rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+	if (!waiter || rt_waiter_node_current(waiter, task)) {
 		raw_spin_unlock_irqrestore(&task->pi_lock, flags);
 		return;
 	}
diff --git a/kernel/locking/rtmutex_common.h b/kernel/locking/rtmutex_common.h
index 1f56d40c3e66..9abd3184cfa3 100644
--- a/kernel/locking/rtmutex_common.h
+++ b/kernel/locking/rtmutex_common.h
@@ -33,6 +33,11 @@ struct rt_waiter_node {
 	struct rb_node	entry;
 	int		prio;
 	u64		deadline;
+	/*
+	 * A deadline waiter's reservation, its relative runtime, deadline
+	 * and period, recorded with @prio and the absolute @deadline.
+	 */
+	struct sched_dl_params dl;
 };
 
 /*
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..55a4f2d8294e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -95,6 +95,9 @@
 #include "smp.h"
 
 #include "../workqueue_internal.h"
+#ifdef CONFIG_RT_MUTEXES
+#include "../locking/rtmutex_common.h"
+#endif
 #include "../../io_uring/io-wq.h"
 #include "../smpboot.h"
 #include "../locking/mutex.h"
@@ -7742,6 +7745,55 @@ void rt_mutex_futex_post_schedule(void)
 	lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0));
 }
 
+/*
+ * The priority @pi_task, the top pi waiter of node @pi, lends a task of
+ * priority @prio: its own, read without its lock, as the chain walk that
+ * follows a change of it brings the change; but that of its node where
+ * either is a deadline one. A deadline boost runs on the reservation the
+ * node recorded with it under the waiter's pi_lock: past a change of the
+ * waiter's own, its parameters may be cleared before the walk comes
+ * (__dl_clear_params()), and replenish_dl_entity() never ends over a
+ * runtime of 0.
+ */
+static int __rt_effective_prio(struct task_struct *pi_task,
+			       struct rt_waiter_node *pi, int prio)
+{
+	int pi_prio;
+
+	if (!pi_task)
+		return prio;
+	pi_prio = pi_task->prio;
+	if (dl_prio(pi_prio) || dl_prio(pi->prio))
+		pi_prio = pi->prio;
+	return min(prio, pi_prio);
+}
+
+/* The node of @p's top pi waiter, or NULL. Under @p->pi_lock. */
+static struct rt_waiter_node *rt_mutex_top_pi_node(struct task_struct *p)
+{
+	lockdep_assert_held(&p->pi_lock);
+	return task_has_pi_waiters(p) ? &task_top_pi_waiter(p)->pi_tree : NULL;
+}
+
+/* The priority @p, of priority @prio of its own, runs at with its boost. */
+int rt_effective_prio(struct task_struct *p, int prio)
+{
+	struct rt_waiter_node *pi = rt_mutex_top_pi_node(p);
+
+	return __rt_effective_prio(pi ? task_top_pi_waiter(p)->task : NULL, pi, prio);
+}
+
+/*
+ * The reservation the top pi waiter of @p lends it, as its node recorded
+ * it, or NULL if that node is not a deadline one. Under @p->pi_lock.
+ */
+const struct sched_dl_params *rt_mutex_top_dl_params(struct task_struct *p)
+{
+	struct rt_waiter_node *pi = rt_mutex_top_pi_node(p);
+
+	return pi && dl_prio(pi->prio) ? &pi->dl : NULL;
+}
+
 /*
  * rt_mutex_setprio - set the current priority of a task
  * @p: task to boost
@@ -7758,11 +7810,11 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 	int prio, oldprio, queue_flag =
 		DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 	const struct sched_class *prev_class, *next_class;
+	struct rt_waiter_node *pi = pi_task ? rt_mutex_top_pi_node(p) : NULL;
 	struct rq_flags rf;
 	struct rq *rq;
 
-	/* XXX used to be waiter->prio, not waiter->task->prio */
-	prio = __rt_effective_prio(pi_task, p->normal_prio);
+	prio = __rt_effective_prio(pi_task, pi, p->normal_prio);
 
 	/*
 	 * If nothing changed; bail early.
@@ -7832,21 +7884,22 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 		 */
 		if (dl_prio(prio)) {
 			if (!dl_prio(p->normal_prio) ||
-			    (pi_task && dl_prio(pi_task->prio) &&
-			     dl_entity_preempt(&pi_task->dl, &p->dl))) {
-				p->dl.pi_se = pi_task->dl.pi_se;
+			    (pi_task && dl_prio(pi->prio) &&
+			     (dl_entity_is_special(&pi_task->dl) ||
+			      dl_time_before(pi->deadline, p->dl.deadline)))) {
+				dl_boost(&p->dl, &pi->dl);
 				scope->flags |= ENQUEUE_REPLENISH;
 			} else {
-				p->dl.pi_se = &p->dl;
+				p->dl.dl_boosted = 0;
 			}
 		} else if (rt_prio(prio)) {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				p->dl.dl_boosted = 0;
 			if (oldprio < prio)
 				scope->flags |= ENQUEUE_HEAD;
 		} else {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				p->dl.dl_boosted = 0;
 			if (rt_prio(oldprio))
 				p->rt.timeout = 0;
 		}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 6cf33045d93a..b8b0076870a6 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -84,25 +84,63 @@ static inline int on_dl_rq(struct sched_dl_entity *dl_se)
 }
 
 #ifdef CONFIG_RT_MUTEXES
-static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
+static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
 {
-	return dl_se->pi_se;
+	return dl_se->dl_boosted;
 }
 
-static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
+/*
+ * The parameters of the reservation @dl_se runs on: its own, or the
+ * donor's, copied by dl_boost().
+ */
+static inline u64 pi_runtime_of(struct sched_dl_entity *dl_se)
 {
-	return pi_of(dl_se) != dl_se;
+	return dl_se->dl_boosted ? dl_se->pi_params.runtime : dl_se->dl_runtime;
 }
-#else /* !CONFIG_RT_MUTEXES: */
-static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
+
+static inline u64 pi_deadline_of(struct sched_dl_entity *dl_se)
 {
-	return dl_se;
+	return dl_se->dl_boosted ? dl_se->pi_params.deadline : dl_se->dl_deadline;
 }
 
+static inline u64 pi_period_of(struct sched_dl_entity *dl_se)
+{
+	return dl_se->dl_boosted ? dl_se->pi_params.period : dl_se->dl_period;
+}
+
+/*
+ * Boost @dl_se with the reservation @params a waiter's rt_mutex node
+ * recorded, under the waiter's pi_lock: a copy, as the waiter may leave
+ * SCHED_DEADLINE, and __dl_clear_params() clear its own parameters,
+ * before the chain walk that follows deboosts @dl_se, and a replenishment
+ * over a dl_period and a dl_runtime of 0 never ends. Under the rq lock and
+ * pi_lock of @dl_se's task.
+ */
+void dl_boost(struct sched_dl_entity *dl_se, const struct sched_dl_params *params)
+{
+	dl_se->pi_params = *params;
+	dl_se->dl_boosted = 1;
+}
+#else /* !CONFIG_RT_MUTEXES: */
 static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
 {
 	return false;
 }
+
+static inline u64 pi_runtime_of(struct sched_dl_entity *dl_se)
+{
+	return dl_se->dl_runtime;
+}
+
+static inline u64 pi_deadline_of(struct sched_dl_entity *dl_se)
+{
+	return dl_se->dl_deadline;
+}
+
+static inline u64 pi_period_of(struct sched_dl_entity *dl_se)
+{
+	return dl_se->dl_period;
+}
 #endif /* !CONFIG_RT_MUTEXES */
 
 static inline u8 dl_get_type(struct sched_dl_entity *dl_se, struct rq *rq)
@@ -731,9 +769,8 @@ static void wakeup_preempt_dl(struct rq *rq, struct task_struct *p, int flags);
 static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
 					    struct rq *rq)
 {
-	/* for non-boosted task, pi_of(dl_se) == dl_se */
-	dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
-	dl_se->runtime = pi_of(dl_se)->dl_runtime;
+	dl_se->deadline = rq_clock(rq) + pi_deadline_of(dl_se);
+	dl_se->runtime = pi_runtime_of(dl_se);
 
 	/*
 	 * If it is a deferred reservation, and the server
@@ -808,7 +845,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
 	struct rq *rq = rq_of_dl_rq(dl_rq);
 
-	WARN_ON_ONCE(pi_of(dl_se)->dl_runtime <= 0);
+	WARN_ON_ONCE(pi_runtime_of(dl_se) <= 0);
 
 	/*
 	 * This could be the case for a !-dl task that is boosted.
@@ -822,8 +859,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 */
 	if (dl_se->dl_deadline == 0 ||
 	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
-		dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
-		dl_se->runtime = pi_of(dl_se)->dl_runtime;
+		dl_se->deadline = rq_clock(rq) + pi_deadline_of(dl_se);
+		dl_se->runtime = pi_runtime_of(dl_se);
 	}
 
 	if (dl_se->dl_yielded && dl_se->runtime > 0)
@@ -836,8 +873,8 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 * arbitrary large.
 	 */
 	while (dl_se->runtime <= 0) {
-		dl_se->deadline += pi_of(dl_se)->dl_period;
-		dl_se->runtime += pi_of(dl_se)->dl_runtime;
+		dl_se->deadline += pi_period_of(dl_se);
+		dl_se->runtime += pi_runtime_of(dl_se);
 	}
 
 	/*
@@ -946,9 +983,9 @@ static bool dl_entity_overflow(struct sched_dl_entity *dl_se, u64 t)
 	 * of anything below microseconds resolution is actually fiction
 	 * (but still we want to give the user that illusion >;).
 	 */
-	left = (pi_of(dl_se)->dl_deadline >> DL_SCALE) * (dl_se->runtime >> DL_SCALE);
+	left = (pi_deadline_of(dl_se) >> DL_SCALE) * (dl_se->runtime >> DL_SCALE);
 	right = ((dl_se->deadline - t) >> DL_SCALE) *
-		(pi_of(dl_se)->dl_runtime >> DL_SCALE);
+		(pi_runtime_of(dl_se) >> DL_SCALE);
 
 	return dl_time_before(right, left);
 }
@@ -3977,10 +4014,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer			= 0;
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
-
-#ifdef CONFIG_RT_MUTEXES
-	dl_se->pi_se			= dl_se;
-#endif
+	dl_se->dl_boosted		= 0;
 }
 
 void init_dl_entity(struct sched_dl_entity *dl_se)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1d76950dbb0a..a06cf2e0d983 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -356,6 +356,9 @@ extern int  sched_dl_global_validate(void);
 extern void sched_dl_do_global(void);
 extern int  sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
 extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+#ifdef CONFIG_RT_MUTEXES
+extern void dl_boost(struct sched_dl_entity *dl_se, const struct sched_dl_params *params);
+#endif
 extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
 extern bool __checkparam_dl(const struct sched_attr *attr);
 extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
@@ -4244,20 +4247,8 @@ bool task_is_pushable(struct rq *rq, struct task_struct *p, int cpu)
 
 #ifdef CONFIG_RT_MUTEXES
 
-static inline int __rt_effective_prio(struct task_struct *pi_task, int prio)
-{
-	if (pi_task)
-		prio = min(prio, pi_task->prio);
-
-	return prio;
-}
-
-static inline int rt_effective_prio(struct task_struct *p, int prio)
-{
-	struct task_struct *pi_task = rt_mutex_get_top_task(p);
-
-	return __rt_effective_prio(pi_task, prio);
-}
+extern int rt_effective_prio(struct task_struct *p, int prio);
+extern const struct sched_dl_params *rt_mutex_top_dl_params(struct task_struct *p);
 
 #else /* !CONFIG_RT_MUTEXES: */
 
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index c3f404cddb9e..89c185f7d498 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -297,10 +297,10 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
 	 */
 
 	if (dl_prio(newprio) && !dl_policy(policy)) {
-		struct task_struct *pi_task = rt_mutex_get_top_task(p);
+		const struct sched_dl_params *params = rt_mutex_top_dl_params(p);
 
-		if (pi_task) {
-			p->dl.pi_se = pi_task->dl.pi_se;
+		if (params) {
+			dl_boost(&p->dl, params);
 			scope->flags |= ENQUEUE_REPLENISH;
 		}
 	}
-- 
2.47.3

[PATCH v4 09/26] sched/deadline: Reset the deadline and the runtime with the parameters

[Mathieu Desnoyers]

A task that leaves SCHED_DEADLINE keeps its parameters until its 0-lag
time, so that leaving and coming back gains nothing. __dl_clear_params()
then resets them, in inactive_task_timer() or in task_non_contending(),
but leaves the deadline and the runtime of the period the task was in.

A task that comes back after its 0-lag time, and before that deadline,
is enqueued on them: enqueue_dl_entity() starts a period anew for a
task whose deadline passed, setup_new_dl_entity(), and for no other. The
task spends the runtime of the reservation it gave back before the new
one begins, and a constrained task whose deadline passed before the end
of its period is throttled on them instead, dl_check_constrained_dl()
arming the replenishment timer of the old period. The previous patch
widens the window to a period for a task that was throttled, moving its
deadline on where its timer fires outside the class.

A task boosted into the class by priority inheritance, with no
parameters of its own, runs on the donor's deadline and runtime, and
keeps them after its deboost: switched_from_dl() resets nothing for it,
and a change into the class before that deadline is enqueued on them as
well.

Reset them with the parameters, and on the deboost of a task that has
none: the next enqueue then starts a period from the parameters the
task has. The deadline_bw_test selftest added later in this series has
the case, deboost_resets_period.

Found by the Sashiko AI review of the second posting of this series, the
deboost by that of the third.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b8b0076870a6..f389620e6fc9 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3580,6 +3580,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	if (task_on_rq_queued(p) && p->dl.dl_runtime)
 		task_non_contending(&p->dl, false);
 
+	/*
+	 * Deboosted from priority inheritance with no parameters of its own:
+	 * the deadline and the runtime are the donor's. A change into the
+	 * class then starts a period from the parameters it is given.
+	 */
+	if (!p->dl.dl_runtime) {
+		p->dl.deadline = 0;
+		p->dl.runtime = 0;
+	}
+
 	/*
 	 * In case a task is setscheduled out from SCHED_DEADLINE we need to
 	 * keep track of that on its cpuset (for correct bandwidth tracking).
@@ -4006,6 +4016,9 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_bw			= 0;
 	dl_se->dl_density		= 0;
 
+	dl_se->deadline			= 0;
+	dl_se->runtime			= 0;
+
 	dl_se->dl_throttled		= 0;
 	dl_se->dl_yielded		= 0;
 	dl_se->dl_non_contending	= 0;
-- 
2.47.3

[PATCH v4 10/26] sched/deadline: Release the bandwidth of a task that leaves the class asleep

[Mathieu Desnoyers]

A task that leaves SCHED_DEADLINE while it is not queued, asleep or
stopped, keeps its bandwidth in the root domain's total_bw for good.
Repeated, it fills total_bw up to the admission cap, after which
sched_setattr(SCHED_DEADLINE) and the offlining of a CPU fail with
EBUSY. switched_from_dl() releases the bandwidth of a queued task alone,
through task_non_contending(), and the inactive timer, the one other
path, is not armed: the task never ran in the class, or the timer fired
while it still was, taking running_bw alone.

If the 0-lag time is still ahead, arm the inactive timer, to release the
bandwidth then: a task migrating to the CPU it wakes on, whose timer
migrate_task_rq_dl() cancels, and one that came back to the class asleep
can have it ahead with no timer armed. Past it, release the bandwidth in
switched_from_dl(), for a task with parameters of its own that is not
special, and clear its dl_bw, so that no later path, a deboost or a late
inactive timer, releases it again. Clear its parameters and its period
with it, as inactive_task_timer() does; a boost the task still gives
reads none of them since "sched/deadline: Take a deadline boost from the
waiter's rt_mutex node".

Reported by Tejun Heo. Juri Lelli's "sched/deadline: Fix total_bw leak
when switching away from SCHED_DEADLINE while !queued", since dropped
from tip, released on this path for every task without clearing dl_bw,
and released the bandwidth of a boosted task twice. The deadline_bw_test
selftest added later in this series has the cases,
pi_former_owner_asleep and deboost_released_resets_period; the Sashiko
AI review of the third posting of this series found the two states with
the 0-lag time ahead.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 97 +++++++++++++++++++++++++++++++++--------
 1 file changed, 80 insertions(+), 17 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index f389620e6fc9..cb0cf9c15108 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -390,6 +390,22 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
 
+/*
+ * The 0-lag time of @dl_se, relative to the clock of its runqueue: its
+ * utilization counts until then once it blocks or leaves the class.
+ */
+static s64 dl_zero_lag(struct sched_dl_entity *dl_se)
+{
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	/*
+	 * Using relative times instead of the absolute "0-lag time"
+	 * allows to simplify the code
+	 */
+	return dl_se->deadline - rq_clock(rq) -
+	       div64_long((dl_se->runtime * dl_se->dl_period), dl_se->dl_runtime);
+}
+
 /*
  * The utilization of a task cannot be immediately removed from
  * the rq active utilization (running_bw) when the task blocks.
@@ -463,15 +479,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 
 	WARN_ON(dl_se->dl_non_contending);
 
-	zerolag_time = dl_se->deadline -
-		 div64_long((dl_se->runtime * dl_se->dl_period),
-			dl_se->dl_runtime);
-
-	/*
-	 * Using relative times instead of the absolute "0-lag time"
-	 * allows to simplify the code
-	 */
-	zerolag_time -= rq_clock(rq);
+	zerolag_time = dl_zero_lag(dl_se);
 
 	/*
 	 * If the "0-lag time" already passed, decrease the active
@@ -3597,15 +3605,69 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	dec_dl_tasks_cs(p);
 
 	if (!task_on_rq_queued(p)) {
-		/*
-		 * Inactive timer is armed. However, p is leaving DEADLINE and
-		 * might migrate away from this rq while continuing to run on
-		 * some other class. We need to remove its contribution from
-		 * this rq running_bw now, or sub_rq_bw (below) will complain.
-		 */
-		if (p->dl.dl_non_contending)
+		bool released = false;
+
+		if (p->dl.dl_non_contending) {
+			/*
+			 * Inactive timer is armed. However, p is leaving
+			 * DEADLINE and might migrate away from this rq while
+			 * continuing to run on some other class. We need to
+			 * remove its contribution from this rq running_bw now,
+			 * or sub_rq_bw (below) will complain.
+			 *
+			 * inactive_task_timer() will find !dl_task(p) and
+			 * release total_bw via __dl_sub().
+			 */
 			sub_running_bw(&p->dl, &rq->dl);
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl) &&
+			   dl_zero_lag(&p->dl) >= 0 &&
+			   !hrtimer_active(&p->dl.inactive_timer)) {
+			/*
+			 * No inactive timer armed, the 0-lag time ahead: the
+			 * task is in flight to the CPU it wakes on, which
+			 * cancelled the timer (migrate_task_rq_dl()), or it
+			 * came back to the class asleep, which cancelled it,
+			 * and leaves again. Arm the timer, as for a queued
+			 * task: its handler releases total_bw at the 0-lag
+			 * time, finding the task out of the class. running_bw
+			 * is off this runqueue already, or never was on it.
+			 */
+			task_non_contending(&p->dl, false);
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
+			/*
+			 * No inactive timer armed, the 0-lag time past: the
+			 * task never ran as DEADLINE, or the timer fired while
+			 * it still was (only dropping running_bw then). Or the
+			 * handler of a timer that fired runs late: it finds
+			 * nothing left to release.
+			 * Nobody else will release total_bw, do it now.
+			 *
+			 * Not for a task without parameters of its own, one
+			 * deboosted from priority inheritance, nor for a
+			 * special one: neither has bandwidth in total_bw.
+			 */
+			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+			raw_spin_unlock(&dl_b->lock);
+			released = true;
+		}
 		sub_rq_bw(&p->dl, &rq->dl);
+		/*
+		 * Released: its 0-lag time is past, and its parameters go with
+		 * the bandwidth, as where inactive_task_timer() resets them.
+		 * Its bandwidth is no one's to release again, and a dl_bw of 0
+		 * is what every path that would takes. Kept, it would make the
+		 * task one with bandwidth to task_contending() and
+		 * task_non_contending() once priority inheritance boosts it,
+		 * and a deboost while it sleeps would release it a second time.
+		 * So would the inactive timer's handler when it runs late,
+		 * dl_non_contending cleared under it by migrate_task_rq_dl():
+		 * it finds !dl_task(p).
+		 */
+		if (released)
+			__dl_clear_params(&p->dl);
 	}
 
 	/*
@@ -4005,7 +4067,8 @@ bool __checkparam_dl(const struct sched_attr *attr)
 }
 
 /*
- * This function clears the sched_dl_entity static params.
+ * This function clears the sched_dl_entity static params, and the state
+ * of the period the entity is in.
  */
 static void __dl_clear_params(struct sched_dl_entity *dl_se)
 {
-- 
2.47.3

[PATCH v4 11/26] sched/deadline: Reject SCHED_DEADLINE on exited tasks

[Mathieu Desnoyers]

From: Juri Lelli <juri.lelli@redhat.com>

sched_setattr(SCHED_DEADLINE) on a zombie task succeeds and charges
total_bw via sched_dl_overflow(), but the task has already done its
final schedule() and will never run again. Nothing releases the reserved
bandwidth. Repeating this fills total_bw to the admission cap, after
which sched_setattr(SCHED_DEADLINE) and CPU offline fail with -EBUSY.

Reject sched_setattr(SCHED_DEADLINE) on exited tasks with -ESRCH.

A task of the class that exited, switched to another policy, runs
switched_from_dl(), which decrements the count of deadline tasks of its
cpuset that cgroup_task_dead() decremented already: at 0 with a deadline
task left in it, the cpuset is skipped by the next rebuild of the root
domains' accounting, and that task's bandwidth with it. Reject that as
well. The deadline_bw_test selftest added later in this series has the
case, death_zombie_switched: on v7.4-rc1 the switch is accepted, and
the one-CPU partition root the zombie ran in admits 9 tasks of 10% after
a rebuild, with a task of 30% still in it, where 6 are due.

Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/3f6e6c87-645b-45fd-8fc6-8da5bc5b82d8@gmail.com/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-4-dd761fe005b4@redhat.com/
Cc: stable@vger.kernel.org
Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
[mathieu: reject a task of the class leaving it too, found by the
 Sashiko AI review of the patch]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/syscalls.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 89c185f7d498..12830e4f8842 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -584,6 +584,19 @@ int __sched_setscheduler(struct task_struct *p,
 	if (retval)
 		goto unlock;
 
+	/*
+	 * Reject SCHED_DEADLINE on exited tasks. A zombie will never be
+	 * scheduled again, so sched_dl_overflow() would charge total_bw
+	 * that nothing will ever release. And reject a change of policy of
+	 * one in the class: cgroup_task_dead() took it off the count of
+	 * deadline tasks of its cpuset, which switched_from_dl() would
+	 * decrement a second time.
+	 */
+	if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
+		retval = -ESRCH;
+		goto unlock;
+	}
+
 	/*
 	 * If not changing anything there's no need to proceed further,
 	 * but store a possible modification of reset_on_fork.
-- 
2.47.3

[PATCH v4 12/26] sched/deadline: Keep departing bandwidth across a root domain rebuild

[Mathieu Desnoyers]

A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
its bandwidth in the total_bw of its root domain until then, when
inactive_task_timer() removes it from the root domain of the CPU the
task is on (commit 387e31300b57 ("sched/deadline: Fix the update of the
total -deadline utilization")). Two things break that:

 1. a rebuild of the root domains' accounting, on any change of the
    cpuset partitions and on CPU hotplug, clears every root domain and
    adds back the bandwidth of the deadline tasks of the cpusets, which
    a task out of the policy is not, and a dead one is in no cpuset: its
    bandwidth is dropped, and the timer then removes it again, from
    whichever root domain its CPU is in by then. total_bw wraps around,
    and that root domain admits deadline tasks beyond its capacity until
    the next rebuild;
 2. the task may move meanwhile to a CPU of another cpuset partition:
    the timer then removes the bandwidth from that root domain, which
    never had it, and leaves it for good in the one that has it.

Keep each departing task on a list, from its departure to its timer,
with the CPU it is on as it leaves: a CPU of the root domain that holds
its bandwidth, but for a task moved asleep to another partition, which a
later patch records. A rebuild puts that bandwidth back in the root
domain of that CPU, and the timer removes it from there.

dl_non_contending tells a task departed, where hrtimer_active() stays
true for a while after the handler returns. A count of the rebuilds,
which each task records when its bandwidth is counted, keeps the timer
from removing what a rebuild dropped since, such as the bandwidth of a
task that exits while a rebuild runs, which neither the cpuset walk nor
the list finds. The list's lock nests inside the rq and pi locks, and
outside the root domains' dl_bw locks. dl_cookie and dl_bw_visited()
have no user outside deadline.c left: make them static.

The deadline_rebuild_test selftest added later in this series fails its
three cases every time without this change, on v7.3 as on v7.4-rc1, and
passes them with it: a task of 400ms every second runs 380ms, then
exits, switches to SCHED_OTHER or moves, before a rebuild, and a one-CPU
partition root admits two tasks of 600ms every second where it has room
for one. A cgroup selftest under development found it, failing about
once in ten runs after a test whose deadline task had just died.

Fixes: 387e31300b57 ("sched/deadline: Fix the update of the total -deadline utilization")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h          |  14 ++
 include/linux/sched/deadline.h |   5 +-
 kernel/cgroup/cpuset.c         |  11 +-
 kernel/sched/deadline.c        | 227 ++++++++++++++++++++++++++++-----
 4 files changed, 214 insertions(+), 43 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index f0b6a8a91323..eb06c541927e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -745,6 +745,20 @@ struct sched_dl_entity {
 	 */
 	struct hrtimer			inactive_timer;
 
+	/*
+	 * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
+	 * time" keeps its bandwidth in its root domain until the inactive
+	 * timer removes it. @dl_departed_cpu is then a CPU of the root domain
+	 * that counts that bandwidth, and @dl_departed_node links it on the
+	 * list that a rebuild of the root domains' accounting puts that
+	 * bandwidth back from; -1 otherwise. @dl_bw_gen is the
+	 * rebuild that last counted the task's bandwidth: one since then
+	 * dropped it, and nothing is to be removed.
+	 */
+	struct list_head		dl_departed_node;
+	int				dl_departed_cpu;
+	unsigned int			dl_bw_gen;
+
 	/*
 	 * Bits for DL-server functionality. Also see the comment near
 	 * dl_server_update().
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 416ca548a038..12c5d8486b09 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -49,7 +49,7 @@ static inline bool dl_time_before(u64 a, u64 b)
 struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
-extern void dl_clear_root_domain_cpu(int cpu);
+extern void dl_clear_root_domains(void);
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
@@ -60,9 +60,6 @@ extern void dl_clear_root_domain_cpu(int cpu);
 extern bool dl_task_needs_bw_move(struct task_struct *p,
 				  const struct cpumask *new_mask);
 
-extern u64 dl_cookie;
-extern bool dl_bw_visited(int cpu, u64 cookie);
-
 static inline bool dl_server(struct sched_dl_entity *dl_se)
 {
 	return dl_se->dl_server;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..4159abfbbbb5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -985,21 +985,14 @@ void dl_rebuild_rd_accounting(void)
 {
 	struct cpuset *cs = NULL;
 	struct cgroup_subsys_state *pos_css;
-	int cpu;
-	u64 cookie = ++dl_cookie;
 
 	lockdep_assert_cpuset_lock_held();
 	lockdep_assert_cpus_held();
 	lockdep_assert_held(&sched_domains_mutex);
 
-	rcu_read_lock();
-
-	for_each_possible_cpu(cpu) {
-		if (dl_bw_visited(cpu, cookie))
-			continue;
+	dl_clear_root_domains();
 
-		dl_clear_root_domain_cpu(cpu);
-	}
+	rcu_read_lock();
 
 	cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
 
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index cb0cf9c15108..4f69ac1846ac 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -201,7 +201,13 @@ static inline unsigned long dl_bw_capacity(int i)
 	}
 }
 
-bool dl_bw_visited(int cpu, u64 cookie)
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+static u64 dl_cookie;
+
+static bool dl_bw_visited(int cpu, u64 cookie)
 {
 	struct root_domain *rd = cpu_rq(cpu)->rd;
 
@@ -255,6 +261,86 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
 	       cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
 }
 
+/*
+ * A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
+ * its bandwidth in total_bw until then: the inactive timer removes it. A
+ * rebuild of the root domains' accounting meanwhile (dl_clear_root_domains()
+ * and dl_add_task_root_domain(), on any change of the partitions or CPU
+ * hotplug) finds no such task among the deadline tasks of the cpusets: it
+ * puts that bandwidth back from dl_departed_list, which holds them from
+ * their departure to their timer, in the root domain of the CPU each was
+ * on when it left, or died. The task may run anywhere meanwhile, as
+ * another policy; its timer holds a reference to it.
+ *
+ * dl_bw_gen counts the rebuilds, and each task records the one that last
+ * counted its bandwidth. A rebuild that dropped it since then leaves
+ * nothing to remove: the CPU recorded was no longer active, or the task
+ * exited while the rebuild ran, which the cpuset walk skips, and was not
+ * dead yet when the list was put back. Both counts, and the list, are under
+ * dl_departed_lock, taken with interrupts disabled (under the task's rq
+ * lock or pi_lock, or by the rebuild) and before any dl_bw lock.
+ */
+static LIST_HEAD(dl_departed_list);
+static DEFINE_RAW_SPINLOCK(dl_departed_lock);
+static unsigned int dl_bw_gen;
+
+/*
+ * @p leaves SCHED_DEADLINE, or dies, its bandwidth kept for now in the root
+ * domain of @cpu. Under its rq lock.
+ */
+static void dl_task_depart(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	if (dl_se->dl_departed_cpu >= 0)
+		return;
+	guard(raw_spinlock)(&dl_departed_lock);
+	dl_se->dl_departed_cpu = cpu;
+	list_add(&dl_se->dl_departed_node, &dl_departed_list);
+}
+
+/*
+ * @p comes back to the deadline class, its bandwidth kept as a deadline
+ * task's again. Under its rq lock.
+ */
+static void dl_task_undepart(struct task_struct *p)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	dl_se->dl_departed_cpu = -1;
+	list_del(&dl_se->dl_departed_node);
+}
+
+/*
+ * Remove @p's bandwidth from the root domain that counts it: that of the
+ * CPU recorded when it departed, else that of @cpu. Nothing to remove
+ * when a rebuild of the accounting has dropped it; and no rebuild running
+ * counts it again. Under @p's rq lock.
+ */
+static void dl_task_release_bw(struct task_struct *p, int cpu)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct dl_bw *dl_b;
+
+	lockdep_assert_irqs_disabled();
+	guard(raw_spinlock)(&dl_departed_lock);
+	if (dl_se->dl_departed_cpu >= 0) {
+		cpu = dl_se->dl_departed_cpu;
+		dl_se->dl_departed_cpu = -1;
+		list_del(&dl_se->dl_departed_node);
+	}
+	if (dl_se->dl_bw_gen == dl_bw_gen) {
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+	}
+	dl_se->dl_bw_gen = dl_bw_gen;
+}
+
 static inline
 void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
@@ -495,13 +581,9 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				sub_running_bw(dl_se, dl_rq);
 
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
-				struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				raw_spin_lock(&dl_b->lock);
-				__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
-				raw_spin_unlock(&dl_b->lock);
+				dl_task_release_bw(p, task_cpu(p));
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -510,8 +592,18 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 	}
 
 	dl_se->dl_non_contending = 1;
-	if (!dl_server(dl_se))
-		get_task_struct(dl_task_of(dl_se));
+	if (!dl_server(dl_se)) {
+		struct task_struct *p = dl_task_of(dl_se);
+
+		get_task_struct(p);
+		/*
+		 * Dying: its bandwidth is in the root domain of the CPU it
+		 * runs on. switched_from_dl() records one that leaves
+		 * SCHED_DEADLINE.
+		 */
+		if (READ_ONCE(p->__state) == TASK_DEAD)
+			dl_task_depart(p, task_cpu(p));
+	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
 }
@@ -2236,17 +2328,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		goto no_task;
 
 	if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
-		struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
 		if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
 			sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
 			sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
 			dl_se->dl_non_contending = 0;
 		}
 
-		raw_spin_lock(&dl_b->lock);
-		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-		raw_spin_unlock(&dl_b->lock);
+		dl_task_release_bw(p, task_cpu(p));
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3507,12 +3595,33 @@ void dl_add_task_root_domain(struct task_struct *p)
 	dl_get_task_effective_cpus(p, msk);
 	cpu = cpumask_first_and(cpu_active_mask, msk);
 	BUG_ON(cpu >= nr_cpu_ids);
+
+	raw_spin_lock(&dl_departed_lock);
+	/*
+	 * Counted already by this rebuild: put back by
+	 * dl_clear_root_domains() as a departed task, which priority
+	 * inheritance boosted into the class since, or its bandwidth
+	 * gone since (dl_task_release_bw()), as that of one that died
+	 * after the walk found it. One that died since with its 0-lag
+	 * time ahead goes back to the root domain of the CPU recorded
+	 * (dl_task_depart()), if active.
+	 */
+	if (p->dl.dl_bw_gen == dl_bw_gen)
+		goto unlock;
+	if (p->dl.dl_departed_cpu >= 0) {
+		cpu = p->dl.dl_departed_cpu;
+		if (!cpu_active(cpu))
+			goto unlock;
+	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
 
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_gen = dl_bw_gen;
+unlock:
+	raw_spin_unlock(&dl_departed_lock);
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 }
 
@@ -3570,9 +3679,45 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
-void dl_clear_root_domain_cpu(int cpu)
+/*
+ * Clear every root domain ahead of a rebuild of their accounting, which
+ * dl_add_task_root_domain() then fills with the deadline tasks of the
+ * cpusets: put back their servers, and the bandwidth of the tasks that
+ * left SCHED_DEADLINE, or died, before their 0-lag time, which no cpuset
+ * holds as a deadline task, in the root domain of the CPU recorded for
+ * each; dropped if it is no longer active.
+ */
+void dl_clear_root_domains(void)
 {
-	dl_clear_root_domain(cpu_rq(cpu)->rd);
+	struct sched_dl_entity *dl_se;
+	u64 cookie = ++dl_cookie;
+	struct dl_bw *dl_b;
+	int cpu;
+
+	guard(raw_spinlock_irqsave)(&dl_departed_lock);
+	dl_bw_gen++;
+	for_each_possible_cpu(cpu) {
+		if (!dl_bw_visited(cpu, cookie))
+			dl_clear_root_domain(cpu_rq(cpu)->rd);
+	}
+	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
+		cpu = dl_se->dl_departed_cpu;
+		/*
+		 * Counted in the generation that ends here, and nowhere
+		 * since: one a rebuild dropped, for want of an active CPU to
+		 * count it, is not to be put back, and its release takes
+		 * nothing away either (dl_task_release_bw()).
+		 */
+		if (dl_se->dl_bw_gen != dl_bw_gen - 1)
+			continue;
+		if (!cpu_active(cpu))
+			continue;
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_add(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+		dl_se->dl_bw_gen = dl_bw_gen;
+	}
 }
 
 static void switched_from_dl(struct rq *rq, struct task_struct *p)
@@ -3616,7 +3761,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * or sub_rq_bw (below) will complain.
 			 *
 			 * inactive_task_timer() will find !dl_task(p) and
-			 * release total_bw via __dl_sub().
+			 * release total_bw (dl_task_release_bw()).
 			 */
 			sub_running_bw(&p->dl, &rq->dl);
 		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl) &&
@@ -3646,11 +3791,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * deboosted from priority inheritance, nor for a
 			 * special one: neither has bandwidth in total_bw.
 			 */
-			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
-			raw_spin_lock(&dl_b->lock);
-			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-			raw_spin_unlock(&dl_b->lock);
+			dl_task_release_bw(p, task_cpu(p));
 			released = true;
 		}
 		sub_rq_bw(&p->dl, &rq->dl);
@@ -3674,9 +3815,19 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 * We cannot use inactive_task_timer() to invoke sub_running_bw()
 	 * at the 0-lag time, because the task could have been migrated
 	 * while SCHED_OTHER in the meanwhile.
+	 *
+	 * Nor can it take the task's bandwidth from the root domain of the
+	 * CPU the task is on by then, and a rebuild of the root domains
+	 * meanwhile would not find it among the deadline tasks: record a
+	 * CPU of the root domain that keeps that bandwidth until the
+	 * inactive timer removes it (dl_task_depart()). dl_non_contending
+	 * tells that timer armed, its handler yet to run; hrtimer_active()
+	 * stays true for a while after that handler returns.
 	 */
-	if (p->dl.dl_non_contending)
+	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
+		dl_task_depart(p, task_cpu(p));
+	}
 
 	/*
 	 * Since this might be the only -deadline task on the rq,
@@ -3696,6 +3847,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 static void switched_to_dl(struct rq *rq, struct task_struct *p)
 {
 	cancel_inactive_timer(&p->dl);
+	/*
+	 * Boosted back into the class before its 0-lag time (a policy change
+	 * released that bandwidth in sched_dl_overflow()): the inactive
+	 * timer will not remove it, but its leaving the class again. That
+	 * forgets the CPU recorded: moved to another root domain meanwhile,
+	 * the task then removes the bandwidth from that one, as it did
+	 * before departures were recorded.
+	 */
+	if (p->dl.dl_departed_cpu >= 0)
+		dl_task_undepart(p);
 
 	/*
 	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -3810,12 +3971,6 @@ DEFINE_SCHED_CLASS(dl) = {
 #endif
 };
 
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
 int sched_dl_global_validate(void)
 {
 	u64 runtime = global_rt_runtime();
@@ -3936,8 +4091,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 
 	if (dl_policy(policy) && !task_has_dl_policy(p) &&
 	    !__dl_overflow(dl_b, cap, 0, new_bw)) {
-		if (hrtimer_active(&p->dl.inactive_timer))
-			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
@@ -3963,6 +4116,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	}
 	raw_spin_unlock(&dl_b->lock);
 
+	/*
+	 * Back before its 0-lag time, the task still has the bandwidth it
+	 * left with, which the new one replaces: removed from the root
+	 * domain that keeps it (dl_task_release_bw()). A rebuild cannot
+	 * run meanwhile: the caller holds cpuset_mutex.
+	 */
+	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
+		lockdep_assert_cpuset_lock_held();
+		if (p->dl.dl_departed_cpu >= 0)
+			dl_task_release_bw(p, -1);
+		p->dl.dl_bw_gen = dl_bw_gen;
+	}
+
 	return err;
 }
 
@@ -4099,6 +4265,7 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
 	__dl_clear_params(dl_se);
+	dl_se->dl_departed_cpu = -1;
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-- 
2.47.3

[PATCH v4 13/26] sched/deadline: Do not take a waking task's bandwidth off a runqueue it left

[Mathieu Desnoyers]

A deadline task that wakes on another CPU has its bandwidth taken off
the runqueue it slept on by migrate_task_rq_dl(), from set_task_cpu(),
and charged to the one it wakes on by the enqueue that follows. That
enqueue can be left to the target CPU through the wake list, and
try_to_wake_up() returns before it runs: the task's task_cpu() is then
the CPU it wakes on while its bandwidth is on no runqueue at all. What
happens to the task in that window takes its bandwidth from a runqueue
that does not have it, or gives it to one that already does:

 - sched_setattr(), finding the task off the runqueues, has
   dl_change_utilization() take its old bandwidth off the runqueue of
   task_cpu(): this_bw of that CPU underflows, costing every other task
   and the fair server there their own, for good. switched_from_dl()
   leaves the class through the same hole;
 - a switch out of SCHED_DEADLINE and back in has switched_to_dl()
   charge the runqueue of task_cpu(), and the enqueue charges it again,
   task_contending() going by ENQUEUE_MIGRATED: this_bw counts the task
   twice, and underflows later.

Record in the entity whether a runqueue counts its bandwidth,
dl_bw_oncpu, and have add_rq_bw() and sub_rq_bw() add and subtract it
once each: an add where it is counted, or a subtraction where it is not,
changes nothing, and dl_change_utilization() acts on a runqueue only
while one counts it. The servers keep their own tally, dl_bw_attached,
and call __add_rq_bw() and __sub_rq_bw() directly.

The first case needs a wakeup that migrates, in a root domain of more
than one CPU, all taken; CONFIG_SCHED_PROXY_EXEC=y makes it easier,
migrating a task blocked on a mutex to the CPU of its owner and waking
it back on its own. In 8 VMs of 8 vCPUs, a task of 10ms every 100ms that
sleeps and wakes in a two-CPU partition root whose CPUs run deadline
tasks of an earlier deadline, its runtime changed between 10ms and 20ms
from outside, warns in all 8 within a second, and the warnings spread to
the idle task's accounting of both CPUs:

  WARNING: kernel/sched/deadline.c:352 at dl_rq_change_utilization+0x71/0x2e0, CPU#0: wake_race/109
   __sub_rq_bw(): this_bw underflow
  WARNING: kernel/sched/deadline.c:316 at task_contending+0x244/0x260, CPU#6: swapper/6/0
   __add_running_bw(): running_bw overflow

With this change none of the 8 warns. deadline_bw_test, added later in
this series, has the case, params_waking, which fails on every one of
those VMs without this change, and random_mix on one of three. The
double charge was found by the Sashiko AI review of the first posting of
this series, as was a bit the first version of this patch left set for
an entity with no bandwidth of its own.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  5 +++++
 kernel/sched/deadline.c | 35 +++++++++++++++++++++++++++++++----
 2 files changed, 36 insertions(+), 4 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index eb06c541927e..6df7d20cbb97 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -714,6 +714,10 @@ struct sched_dl_entity {
 	 * entities (@dl_server == 1). Allows toggling the reservation on/off
 	 * without losing the configured @dl_runtime/@dl_period.
 	 *
+	 * @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
+	 * @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
+	 * runqueue's lock. Not for servers, which keep @dl_bw_attached.
+	 *
 	 * @dl_boosted tells if priority inheritance boosted the entity: it
 	 * runs on the parameters copied in @pi_params.
 	 */
@@ -728,6 +732,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer_running  : 1;
 	unsigned int			dl_defer_idle     : 1;
 	unsigned int			dl_bw_attached    : 1;
+	unsigned int			dl_bw_oncpu       : 1;
 	unsigned int			dl_boosted        : 1;
 
 	/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 4f69ac1846ac..6b0ec2003857 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -391,18 +391,31 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
 	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
 }
 
+/*
+ * @dl_bw_oncpu tells whether this_bw of a runqueue counts @dl_se's
+ * bandwidth, so that these two put it there and take it away once each:
+ * a wakeup takes it off the runqueue the task slept on and the enqueue on
+ * the one it wakes on puts it back, and what happens to the task in
+ * between must not count it twice, nor take it from a runqueue that does
+ * not have it. The servers keep their own tally, dl_bw_attached, and go
+ * through __add_rq_bw() and __sub_rq_bw() directly.
+ */
 static inline
 void add_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__add_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || dl_se->dl_bw_oncpu)
+		return;
+	__add_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 1;
 }
 
 static inline
 void sub_rq_bw(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
-	if (!dl_entity_is_special(dl_se))
-		__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	if (dl_entity_is_special(dl_se) || !dl_se->dl_bw_oncpu)
+		return;
+	__sub_rq_bw(dl_se->dl_bw, dl_rq);
+	dl_se->dl_bw_oncpu = 0;
 }
 
 static inline
@@ -471,6 +484,14 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 	if (task_on_rq_queued(p))
 		return;
 
+	/*
+	 * Waking up on another CPU, the task has its bandwidth on no
+	 * runqueue: the enqueue to come charges the new one, with the new
+	 * parameters, which __setparam_dl() sets right after this.
+	 */
+	if (!p->dl.dl_bw_oncpu)
+		return;
+
 	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
 }
 
@@ -2833,6 +2854,11 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
 		 */
 		cancel_inactive_timer(&p->dl);
 	}
+	/*
+	 * Until the enqueue on the CPU it wakes on charges it there, this
+	 * bandwidth is on no runqueue: the rq of task_cpu() is not the one
+	 * it was taken from, which dl_bw_oncpu now says.
+	 */
 	sub_rq_bw(&p->dl, &rq->dl);
 	rq_unlock(rq, &rf);
 }
@@ -4256,6 +4282,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer			= 0;
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
+	dl_se->dl_bw_oncpu		= 0;
 	dl_se->dl_boosted		= 0;
 }
 
-- 
2.47.3

[PATCH v4 14/26] sched/deadline: Rebuild the root domains' accounting from the task list

[Mathieu Desnoyers]

A change of the root domains, which CPU hotplug and suspend and resume
make, clears their deadline accounting and fills it back with the
servers of each CPU and the deadline tasks of each cpuset. The task walk
lives in the cpuset code, dl_rebuild_rd_accounting(), and is a stub
without CONFIG_CPUSETS: such a kernel loses the bandwidth of every
deadline task at the first hotplug. Each task still takes its bandwidth
out of total_bw once it leaves SCHED_DEADLINE, or dies: total_bw
underflows, and admission afterwards refuses every deadline task of the
root domain.

With cpusets, the walk skips a cpuset whose nr_deadline_tasks is 0. An
attach moves that count by a tally taken in cpuset_can_attach(), while a
change of class counts the task on the cpuset it is in at the time: a
task that leaves the class between the tally and the move is taken off
the old cpuset twice, and the next deadline task there brings the count
to 0. Every rebuild then drops the bandwidth of the deadline tasks of
that cpuset, for good. A boost into the class and its deboost count
without cpuset_mutex, so nothing orders them against the attach.

Walk the tasks instead, with or without cpusets, under tasklist_lock
held for reading, which fork and exit wait for meanwhile: the walk finds
every deadline task, and the count goes, with the hooks that kept it. A
rebuild then tests the class of every task, without its lock, where the
count let it skip a cpuset with no deadline task. The walk finds exited
tasks too: one whose dl_bw is 0 is left out, as the cpuset walk left it.
Admission must not run against the rebuild: cpuset_mutex sees to it with
cpusets, held by __sched_setscheduler() over sched_dl_overflow() and by
the cpuset code from partition_sched_domains() to the rebuild, and a
mutex of the deadline code's own takes its place at both without.

In 8-CPU VMs built without CONFIG_CPUSETS, the hotplug_sleepers case of
the selftest added later in this series admits 9 tasks of 10ms every
100ms before taking a CPU offline and back, and 0 after, on v7.4-rc1 as
with the patches before this one; hotplug_departed and
hotplug_server_refused fail after it, and two more hotplug cases skip
for want of room. The five pass with this change, in 4 VMs, with no
warning. The drift of the count was found by the Sashiko AI review of
the third posting of this series, and the exited tasks by its review of
v4 before posting.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/cpuset.h          |  9 ---
 include/linux/sched/deadline.h  | 16 ++++++
 kernel/cgroup/cgroup.c          |  3 -
 kernel/cgroup/cpuset-internal.h |  6 +-
 kernel/cgroup/cpuset.c          | 67 +----------------------
 kernel/sched/deadline.c         | 97 +++++++++++++++++++++++++--------
 kernel/sched/syscalls.c         | 14 +++--
 kernel/sched/topology.c         |  9 +++
 8 files changed, 111 insertions(+), 110 deletions(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..db1cc2fc0324 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -72,8 +72,6 @@ extern int cpuset_init(void);
 extern void cpuset_init_smp(void);
 extern void cpuset_force_rebuild(void);
 extern void cpuset_update_active_cpus(void);
-extern void inc_dl_tasks_cs(struct task_struct *task);
-extern void dec_dl_tasks_cs(struct task_struct *task);
 extern void cpuset_lock(void);
 extern void cpuset_unlock(void);
 extern void lockdep_assert_cpuset_lock_held(void);
@@ -129,7 +127,6 @@ static inline int cpuset_do_page_mem_spread(void)
 
 extern bool current_cpuset_is_being_rebound(void);
 
-extern void dl_rebuild_rd_accounting(void);
 extern void rebuild_sched_domains(void);
 
 extern void cpuset_print_current_mems_allowed(void);
@@ -194,8 +191,6 @@ static inline void cpuset_update_active_cpus(void)
 	partition_sched_domains(1, NULL, NULL);
 }
 
-static inline void inc_dl_tasks_cs(struct task_struct *task) { }
-static inline void dec_dl_tasks_cs(struct task_struct *task) { }
 static inline void cpuset_lock(void) { }
 static inline void cpuset_unlock(void) { }
 static inline void lockdep_assert_cpuset_lock_held(void) { }
@@ -273,10 +268,6 @@ static inline bool current_cpuset_is_being_rebound(void)
 	return false;
 }
 
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
-
 static inline void rebuild_sched_domains(void)
 {
 	guard(cpus_read_lock)();
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 12c5d8486b09..7c99776af136 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -50,6 +50,22 @@ struct root_domain;
 extern void dl_add_task_root_domain(struct task_struct *p);
 extern void dl_clear_root_domain(struct root_domain *rd);
 extern void dl_clear_root_domains(void);
+extern void dl_rebuild_rd_accounting(void);
+/*
+ * The lock that keeps a rebuild of the root domains' accounting from running
+ * against an admission of bandwidth: cpuset_mutex where cpusets are built in,
+ * since every rebuild comes from there, and a lock of the deadline class's own
+ * where they are not.
+ */
+#ifdef CONFIG_CPUSETS
+static inline void dl_accounting_lock(void) { }
+static inline void dl_accounting_unlock(void) { }
+static inline void lockdep_assert_dl_accounting_held(void) { }
+#else
+extern void dl_accounting_lock(void);
+extern void dl_accounting_unlock(void);
+extern void lockdep_assert_dl_accounting_held(void);
+#endif
 /*
  * Return whether moving DL task @p to @new_mask requires moving DL
  * bandwidth accounting between root domains. This helper is specific to
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..2cfa770c2dcb 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -7240,9 +7240,6 @@ static void do_cgroup_task_dead(struct task_struct *tsk)
 	if (thread_group_leader(tsk) && atomic_read(&tsk->signal->live))
 		list_add_tail(&tsk->cg_list, &cset->dying_tasks);
 
-	if (dl_task(tsk))
-		dec_dl_tasks_cs(tsk);
-
 	WARN_ON_ONCE(cgroup_task_frozen(tsk));
 	if (unlikely(!(tsk->flags & PF_KTHREAD) &&
 		     test_bit(CGRP_FREEZE, &task_dfl_cgroup(tsk)->flags)))
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 1ba94ffaff9d..153067b164c1 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,11 +166,7 @@ struct cpuset {
 	 */
 	bool remote_partition_updating;
 
-	/*
-	 * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
-	 * know when to rebuild associated root domain bandwidth information.
-	 */
-	atomic_t nr_deadline_tasks;
+	/* SCHED_DEADLINE tasks of the attach in progress moving here. */
 	int nr_migrate_dl_tasks;
 	/* DL bandwidth that needs destination reservation for this attach. */
 	u64 sum_migrate_dl_bw;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 4159abfbbbb5..7628cf77bdfd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -225,20 +225,6 @@ struct tmpmasks {
 	cpumask_var_t new_cpus;		/* For update_cpumasks_hier() */
 };
 
-void inc_dl_tasks_cs(struct task_struct *p)
-{
-	struct cpuset *cs = task_cs(p);
-
-	atomic_inc(&cs->nr_deadline_tasks);
-}
-
-void dec_dl_tasks_cs(struct task_struct *p)
-{
-	struct cpuset *cs = task_cs(p);
-
-	atomic_dec(&cs->nr_deadline_tasks);
-}
-
 static inline bool is_partition_valid(const struct cpuset *cs)
 {
 	return cs->partition_root_state > 0;
@@ -965,54 +951,6 @@ static int generate_sched_domains(cpumask_var_t **domains,
 	return ndoms;
 }
 
-static void dl_update_tasks_root_domain(struct cpuset *cs)
-{
-	struct css_task_iter it;
-	struct task_struct *task;
-
-	if (atomic_read(&cs->nr_deadline_tasks) == 0)
-		return;
-
-	css_task_iter_start(&cs->css, 0, &it);
-
-	while ((task = css_task_iter_next(&it)))
-		dl_add_task_root_domain(task);
-
-	css_task_iter_end(&it);
-}
-
-void dl_rebuild_rd_accounting(void)
-{
-	struct cpuset *cs = NULL;
-	struct cgroup_subsys_state *pos_css;
-
-	lockdep_assert_cpuset_lock_held();
-	lockdep_assert_cpus_held();
-	lockdep_assert_held(&sched_domains_mutex);
-
-	dl_clear_root_domains();
-
-	rcu_read_lock();
-
-	cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
-
-		if (cpumask_empty(cs->effective_cpus)) {
-			pos_css = css_rightmost_descendant(pos_css);
-			continue;
-		}
-
-		css_get(&cs->css);
-
-		rcu_read_unlock();
-
-		dl_update_tasks_root_domain(cs);
-
-		rcu_read_lock();
-		css_put(&cs->css);
-	}
-	rcu_read_unlock();
-}
-
 /*
  * Rebuild scheduler domains.
  *
@@ -3181,9 +3119,7 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
 	llist_for_each_entry_safe(cs, next, lnode, attach_node) {
 		init_llist_node(&cs->attach_node);
 		if (cs->nr_migrate_dl_tasks) {
-			if (!cancel)
-				atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
-			else if (cs->dl_bw_cpu >= 0) /* && cancel */
+			if (cancel && cs->dl_bw_cpu >= 0)
 				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
 			cs->nr_migrate_dl_tasks = 0;
 			cs->sum_migrate_dl_bw = 0;
@@ -3272,7 +3208,6 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 			 * contribute to sum_migrate_dl_bw.
 			 */
 			cs->nr_migrate_dl_tasks++;
-			oldcs->nr_migrate_dl_tasks--;
 			if (dl_task_needs_bw_move(task, cs->effective_cpus))
 				cs->sum_migrate_dl_bw += task->dl.dl_bw;
 		}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 6b0ec2003857..45a896a4f461 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -266,7 +266,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * its bandwidth in total_bw until then: the inactive timer removes it. A
  * rebuild of the root domains' accounting meanwhile (dl_clear_root_domains()
  * and dl_add_task_root_domain(), on any change of the partitions or CPU
- * hotplug) finds no such task among the deadline tasks of the cpusets: it
+ * hotplug) finds no such task among the deadline tasks: it
  * puts that bandwidth back from dl_departed_list, which holds them from
  * their departure to their timer, in the root domain of the CPU each was
  * on when it left, or died. The task may run anywhere meanwhile, as
@@ -275,7 +275,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
  * dl_bw_gen counts the rebuilds, and each task records the one that last
  * counted its bandwidth. A rebuild that dropped it since then leaves
  * nothing to remove: the CPU recorded was no longer active, or the task
- * exited while the rebuild ran, which the cpuset walk skips, and was not
+ * exited while the rebuild ran, which the walk does not find, and was not
  * dead yet when the list was put back. Both counts, and the list, are under
  * dl_departed_lock, taken with interrupts disabled (under the task's rq
  * lock or pi_lock, or by the rebuild) and before any dl_bw lock.
@@ -3602,7 +3602,7 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	cpuset_cpus_allowed_locked(p, cpus);
 }
 
-/* The caller should hold cpuset_mutex */
+/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
 void dl_add_task_root_domain(struct task_struct *p)
 {
 	struct rq_flags rf;
@@ -3612,7 +3612,12 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct cpumask *msk;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
-	if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
+	/*
+	 * Nothing to count for a special entity, nor for a task with no
+	 * bandwidth, such as an exited one whose bandwidth went with its
+	 * parameters, which the walk of the task list finds too.
+	 */
+	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 		return;
 	}
@@ -3705,13 +3710,71 @@ void dl_clear_root_domain(struct root_domain *rd)
 		dl_server_add_bw(rd, i);
 }
 
+#ifndef CONFIG_CPUSETS
+/*
+ * Where cpusets are built in, cpuset_mutex keeps an admission of bandwidth
+ * from running against a rebuild of the accounting: __sched_setscheduler()
+ * takes it for every change of or into SCHED_DEADLINE, and the callers of
+ * partition_sched_domains() hold it from the attachment of the new root
+ * domains to the rebuild. Where they are not, cpuset_lock() is a stub and
+ * this takes its place, in both.
+ */
+static DEFINE_MUTEX(dl_accounting_mutex);
+
+void dl_accounting_lock(void)
+{
+	mutex_lock(&dl_accounting_mutex);
+}
+
+void dl_accounting_unlock(void)
+{
+	mutex_unlock(&dl_accounting_mutex);
+}
+
+void lockdep_assert_dl_accounting_held(void)
+{
+	lockdep_assert_held(&dl_accounting_mutex);
+}
+#endif /* !CONFIG_CPUSETS */
+
+/*
+ * Rebuild the root domains' accounting: clear it, then count every
+ * deadline task of the system again, found by walking them all. No
+ * cpuset tells which hold any: a count of theirs went wrong across an
+ * attach.
+ */
+void dl_rebuild_rd_accounting(void)
+{
+	struct task_struct *g, *p;
+
+	lockdep_assert_cpus_held();
+	lockdep_assert_held(&sched_domains_mutex);
+	lockdep_assert_cpuset_lock_held();
+	lockdep_assert_dl_accounting_held();
+
+	dl_clear_root_domains();
+
+	read_lock(&tasklist_lock);
+	for_each_process_thread(g, p) {
+		/*
+		 * Tested without the task's lock first, to spare the others
+		 * the locks: a task enters the class under the locks this
+		 * rebuild holds, or by a boost, which brings it no bandwidth
+		 * of its own to count.
+		 */
+		if (dl_task(p))
+			dl_add_task_root_domain(p);
+	}
+	read_unlock(&tasklist_lock);
+}
+
 /*
  * Clear every root domain ahead of a rebuild of their accounting, which
- * dl_add_task_root_domain() then fills with the deadline tasks of the
- * cpusets: put back their servers, and the bandwidth of the tasks that
- * left SCHED_DEADLINE, or died, before their 0-lag time, which no cpuset
- * holds as a deadline task, in the root domain of the CPU recorded for
- * each; dropped if it is no longer active.
+ * dl_add_task_root_domain() then fills with the deadline tasks: put back
+ * their servers, and the bandwidth of the tasks that left SCHED_DEADLINE,
+ * or died, before their 0-lag time, which that walk does not count, in
+ * the root domain of the CPU recorded for each; dropped if it is no
+ * longer active.
  */
 void dl_clear_root_domains(void)
 {
@@ -3769,12 +3832,6 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 		p->dl.runtime = 0;
 	}
 
-	/*
-	 * In case a task is setscheduled out from SCHED_DEADLINE we need to
-	 * keep track of that on its cpuset (for correct bandwidth tracking).
-	 */
-	dec_dl_tasks_cs(p);
-
 	if (!task_on_rq_queued(p)) {
 		bool released = false;
 
@@ -3884,12 +3941,6 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	if (p->dl.dl_departed_cpu >= 0)
 		dl_task_undepart(p);
 
-	/*
-	 * In case a task is setscheduled to SCHED_DEADLINE we need to keep
-	 * track of that on its cpuset (for correct bandwidth tracking).
-	 */
-	inc_dl_tasks_cs(p);
-
 	/* If p is not queued we will update its parameters at next wakeup. */
 	if (!task_on_rq_queued(p)) {
 		add_rq_bw(&p->dl, &rq->dl);
@@ -4146,10 +4197,12 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 * Back before its 0-lag time, the task still has the bandwidth it
 	 * left with, which the new one replaces: removed from the root
 	 * domain that keeps it (dl_task_release_bw()). A rebuild cannot
-	 * run meanwhile: the caller holds cpuset_mutex.
+	 * run meanwhile: the caller holds cpuset_mutex, or the lock of
+	 * dl_accounting_lock() without cpusets.
 	 */
 	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
 		lockdep_assert_cpuset_lock_held();
+		lockdep_assert_dl_accounting_held();
 		if (p->dl.dl_departed_cpu >= 0)
 			dl_task_release_bw(p, -1);
 		p->dl.dl_bw_gen = dl_bw_gen;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 12830e4f8842..29a2c1db2343 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -560,6 +560,7 @@ int __sched_setscheduler(struct task_struct *p,
 	if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
 		cpuset_locked = true;
 		cpuset_lock();
+		dl_accounting_lock();
 	}
 
 	/*
@@ -588,9 +589,7 @@ int __sched_setscheduler(struct task_struct *p,
 	 * Reject SCHED_DEADLINE on exited tasks. A zombie will never be
 	 * scheduled again, so sched_dl_overflow() would charge total_bw
 	 * that nothing will ever release. And reject a change of policy of
-	 * one in the class: cgroup_task_dead() took it off the count of
-	 * deadline tasks of its cpuset, which switched_from_dl() would
-	 * decrement a second time.
+	 * one in the class: its bandwidth went at its last schedule.
 	 */
 	if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
 		retval = -ESRCH;
@@ -655,6 +654,7 @@ int __sched_setscheduler(struct task_struct *p,
 		policy = oldpolicy = -1;
 		task_rq_unlock(rq, p, &rf);
 		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
 			cpuset_locked = false;
 		}
@@ -759,8 +759,10 @@ int __sched_setscheduler(struct task_struct *p,
 	task_rq_unlock(rq, p, &rf);
 
 	if (pi) {
-		if (cpuset_locked)
+		if (cpuset_locked) {
+			dl_accounting_unlock();
 			cpuset_unlock();
+		}
 		rt_mutex_adjust_pi(p);
 	}
 
@@ -772,8 +774,10 @@ int __sched_setscheduler(struct task_struct *p,
 
 unlock:
 	task_rq_unlock(rq, p, &rf);
-	if (cpuset_locked)
+	if (cpuset_locked) {
+		dl_accounting_unlock();
 		cpuset_unlock();
+	}
 	return retval;
 }
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 9d3a1b8dd393..2fce33783e81 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3528,7 +3528,16 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
 void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[],
 			     struct sched_domain_attr *dattr_new)
 {
+	/*
+	 * The root domains attached here count no task's bandwidth until
+	 * dl_rebuild_rd_accounting() at the end: no admission of bandwidth
+	 * may run meanwhile. The callers of this hold cpuset_mutex, which
+	 * keeps admissions out, where cpusets are built in; the lock
+	 * dl_accounting_lock() takes stands in for it where they are not.
+	 */
+	dl_accounting_lock();
 	sched_domains_mutex_lock();
 	partition_sched_domains_locked(ndoms_new, doms_new, dattr_new);
 	sched_domains_mutex_unlock();
+	dl_accounting_unlock();
 }
-- 
2.47.3

[PATCH v4 15/26] sched/deadline: Leave a dying task out of a rebuild of the accounting

[Mathieu Desnoyers]

A rebuild of the root domains' accounting finds the tasks of a cpuset
through css_task_iter, which hides an exiting task once its process has
no live thread left, and hides a dying thread of a process with live
threads only once cgroup_task_dead() takes it off the cpuset, after its
last schedule. The walk without cpusets finds every task. Either can
find a task at its last schedule, between the release of its bandwidth
there, task_non_contending() past its 0-lag time, and the reset of its
dl_bw: both run under the runqueue lock, which the walk does not take.
The walk then counts, in a root domain the rebuild has just cleared, a
bandwidth that nothing releases again.

Leave out a dead task that is not on the departed list: its bandwidth
is released, or being released. do_task_dead() sets TASK_DEAD under
pi_lock, which the walk holds, so a task found alive dies after the
walk, and releases what the walk counted. A dead task on the list keeps
its bandwidth until its timer, and the rebuild counts it where the list
says, as before.

Found by the Sashiko AI review of the second posting of this series.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 45a896a4f461..3f3de4b74a2d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3643,6 +3643,15 @@ void dl_add_task_root_domain(struct task_struct *p)
 		cpu = p->dl.dl_departed_cpu;
 		if (!cpu_active(cpu))
 			goto unlock;
+	} else if (READ_ONCE(p->__state) == TASK_DEAD) {
+		/*
+		 * Dead and not departed: its bandwidth was released at its
+		 * last schedule, under its runqueue lock and not this one,
+		 * and dl_bw may still say otherwise. do_task_dead() sets
+		 * TASK_DEAD under pi_lock before that, so a task found alive
+		 * here dies after this walk, and releases what it counted.
+		 */
+		goto unlock;
 	}
 	rq = cpu_rq(cpu);
 	dl_b = &rq->rd->dl_bw;
-- 
2.47.3

[PATCH v4 16/26] sched/deadline: Record the root domain holding a task's bandwidth

[Mathieu Desnoyers]

A task's bandwidth can be taken from a root domain that never held it.
Three answers to "which root domain counts this task's bandwidth" live
side by side: admission and the release at the 0-lag time take the root
domain of the CPU the task is on, a move between cpusets that of its
runqueue, and a rebuild of the accounting that of its cpuset. They part
ways where the task's CPU and its cpuset are in different root domains:

 - a sleeping task moved to another partition and back before it runs
   has no move seen the second time: its bandwidth stays in the
   partition, and the root domain it returns to admits deadline tasks
   beyond its capacity, and underflows when the task exits;
 - a task pinned to a CPU that goes offline, admission control off, is
   counted by the rebuild in its cpuset's root domain and released from
   the default one.

Record, wherever a task is handed bandwidth, a CPU of the root domain
that counts it: admission, the rebuild, and the moves of
set_cpus_allowed_dl() and dl_task_offline_migration(). Every release or
move of the bandwidth reads that CPU, dl_task_root_domain_cpu(), and the
release clears it. Where dl_bw_gen says a rebuild dropped the bandwidth,
a change of parameters or a move takes nothing back, and records where
the new bandwidth goes. A task counted nowhere has the CPU it is on
answer for it, and a task boosted back into the class keeps its record.
The departed list of "sched/deadline: Keep departing bandwidth across a
root domain rebuild" uses the record in place of a CPU of its own. A
special entity, a sugov kthread counted nowhere, and a task boosted into
the class with no bandwidth of its own are left out of the moves, as the
rebuild leaves them out: a move took a sugov kthread's bandwidth from a
root domain that never held it.

The moved sleeper was reported by Weiheng Qiu and analysed by Juri
Lelli, whose "sched/deadline: Fix stale root domain in cpuset bandwidth
transfer", since dropped from tip, took the first active CPU of the
task's mask instead, which a mask lacks during CPU hotplug: a sugov
kthread then crashed on its BUG_ON(). The Sashiko AI reviews of this
series found the rest: a mask naming a root domain no rebuild had
credited, sched_dl_overflow() and dl_task_offline_migration() keeping
the CPU the task is on, and the special and boosted tasks moved.

Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/14508cb2-805f-41ac-9f6b-bfd7c3d7cb1c@gmail.com/
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  20 ++--
 kernel/sched/deadline.c | 212 +++++++++++++++++++++++++++-------------
 2 files changed, 155 insertions(+), 77 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6df7d20cbb97..28b579bff9e1 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -751,17 +751,21 @@ struct sched_dl_entity {
 	struct hrtimer			inactive_timer;
 
 	/*
+	 * @dl_bw_cpu is a CPU of the root domain that counts @dl_bw, -1
+	 * where none does: every site that hands that bandwidth out records
+	 * it, and every site that takes it back or moves it reads it.
+	 * @dl_bw_gen is the rebuild of the root domains' accounting that
+	 * last counted it: one since then dropped it, and there is nothing
+	 * to take back.
+	 *
 	 * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
-	 * time" keeps its bandwidth in its root domain until the inactive
-	 * timer removes it. @dl_departed_cpu is then a CPU of the root domain
-	 * that counts that bandwidth, and @dl_departed_node links it on the
-	 * list that a rebuild of the root domains' accounting puts that
-	 * bandwidth back from; -1 otherwise. @dl_bw_gen is the
-	 * rebuild that last counted the task's bandwidth: one since then
-	 * dropped it, and nothing is to be removed.
+	 * time" keeps its bandwidth until the inactive timer removes it:
+	 * @dl_departed_node links it meanwhile on the list that a rebuild
+	 * puts that bandwidth back from, and tells of that by not being
+	 * empty.
 	 */
 	struct list_head		dl_departed_node;
-	int				dl_departed_cpu;
+	int				dl_bw_cpu;
 	unsigned int			dl_bw_gen;
 
 	/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 3f3de4b74a2d..7a7101e6f7a0 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -286,17 +286,16 @@ static unsigned int dl_bw_gen;
 
 /*
  * @p leaves SCHED_DEADLINE, or dies, its bandwidth kept for now in the root
- * domain of @cpu. Under its rq lock.
+ * domain of the CPU recorded with it. Under its rq lock.
  */
-static void dl_task_depart(struct task_struct *p, int cpu)
+static void dl_task_depart(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 
 	lockdep_assert_irqs_disabled();
-	if (dl_se->dl_departed_cpu >= 0)
-		return;
 	guard(raw_spinlock)(&dl_departed_lock);
-	dl_se->dl_departed_cpu = cpu;
+	if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
+		return;
 	list_add(&dl_se->dl_departed_node, &dl_departed_list);
 }
 
@@ -309,35 +308,37 @@ static void dl_task_undepart(struct task_struct *p)
 	struct sched_dl_entity *dl_se = &p->dl;
 
 	lockdep_assert_irqs_disabled();
+	if (list_empty(&dl_se->dl_departed_node))
+		return;
 	guard(raw_spinlock)(&dl_departed_lock);
-	dl_se->dl_departed_cpu = -1;
-	list_del(&dl_se->dl_departed_node);
+	list_del_init(&dl_se->dl_departed_node);
 }
 
 /*
- * Remove @p's bandwidth from the root domain that counts it: that of the
- * CPU recorded when it departed, else that of @cpu. Nothing to remove
- * when a rebuild of the accounting has dropped it; and no rebuild running
- * counts it again. Under @p's rq lock.
+ * Remove @p's bandwidth from the root domain that counts it, that of the
+ * CPU recorded with it, and clear the record. Nothing to remove when a
+ * rebuild of the accounting has dropped it; and no rebuild running counts
+ * it again. Under @p's rq lock.
  */
-static void dl_task_release_bw(struct task_struct *p, int cpu)
+static void dl_task_release_bw(struct task_struct *p)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
 	struct dl_bw *dl_b;
+	int cpu;
 
 	lockdep_assert_irqs_disabled();
 	guard(raw_spinlock)(&dl_departed_lock);
-	if (dl_se->dl_departed_cpu >= 0) {
-		cpu = dl_se->dl_departed_cpu;
-		dl_se->dl_departed_cpu = -1;
-		list_del(&dl_se->dl_departed_node);
-	}
-	if (dl_se->dl_bw_gen == dl_bw_gen) {
+	if (!list_empty(&dl_se->dl_departed_node))
+		list_del_init(&dl_se->dl_departed_node);
+	cpu = dl_se->dl_bw_cpu;
+	WARN_ON_ONCE(dl_se->dl_bw && cpu < 0);
+	if (cpu >= 0 && dl_se->dl_bw_gen == dl_bw_gen) {
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
 		__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
 		raw_spin_unlock(&dl_b->lock);
 	}
+	dl_se->dl_bw_cpu = -1;
 	dl_se->dl_bw_gen = dl_bw_gen;
 }
 
@@ -496,6 +497,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p);
 
 /*
  * The 0-lag time of @dl_se, relative to the clock of its runqueue: its
@@ -604,7 +606,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 			if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
-				dl_task_release_bw(p, task_cpu(p));
+				dl_task_release_bw(p);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -623,7 +625,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 		 * SCHED_DEADLINE.
 		 */
 		if (READ_ONCE(p->__state) == TASK_DEAD)
-			dl_task_depart(p, task_cpu(p));
+			dl_task_depart(p);
 	}
 
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
@@ -861,19 +863,30 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 	}
 
 	/*
-	 * And we finally need to fix up root_domain(s) bandwidth accounting,
-	 * since p is still hanging out in the old (now moved to default) root
-	 * domain.
+	 * And we finally need to fix up root_domain(s) bandwidth accounting:
+	 * the bandwidth of @p is counted in the root domain that took it,
+	 * which is not the one of the runqueue it leaves if it was moved
+	 * asleep. A rebuild that dropped it has nothing here to move, and
+	 * the record left behind is what makes the next one count it.
 	 */
-	dl_b = &rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_sub(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
-
-	dl_b = &later_rq->rd->dl_bw;
-	raw_spin_lock(&dl_b->lock);
-	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span));
-	raw_spin_unlock(&dl_b->lock);
+	scoped_guard (raw_spinlock, &dl_departed_lock) {
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
+
+			dl_b = &src_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw,
+				 cpumask_weight(src_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+
+			dl_b = &later_rq->rd->dl_bw;
+			raw_spin_lock(&dl_b->lock);
+			__dl_add(dl_b, p->dl.dl_bw,
+				 cpumask_weight(later_rq->rd->span));
+			raw_spin_unlock(&dl_b->lock);
+		}
+		p->dl.dl_bw_cpu = later_rq->cpu;
+	}
 
 	set_task_cpu(p, later_rq->cpu);
 	double_unlock_balance(later_rq, rq);
@@ -2355,7 +2368,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 			dl_se->dl_non_contending = 0;
 		}
 
-		dl_task_release_bw(p, task_cpu(p));
+		dl_task_release_bw(p);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -3503,14 +3516,26 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
 	}
 }
 
+/*
+ * A CPU of the root domain that counts the bandwidth of @p: the one
+ * recorded with it. None where no bandwidth of @p's is counted anywhere:
+ * nothing to take back or to move then, and the CPU @p is on answers for
+ * its root domain.
+ */
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
+{
+	int cpu = p->dl.dl_bw_cpu;
+
+	if (cpu < 0)
+		return task_cpu(p);
+	return cpu;
+}
+
 static void set_cpus_allowed_dl(struct task_struct *p,
 				struct affinity_context *ctx)
 {
-	struct rq *rq;
-
 	WARN_ON_ONCE(!dl_task(p));
 
-	rq = task_rq(p);
 	/*
 	 * Migrating a SCHED_DEADLINE task between exclusive
 	 * cpusets (different root_domains) entails a bandwidth
@@ -3518,17 +3543,30 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * domain (see cpuset_can_attach()).
 	 */
 	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
-		struct dl_bw *src_dl_b;
+		unsigned int src_cpu = dl_task_root_domain_cpu(p);
+		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
+		unsigned int cpu;
 
-		src_dl_b = dl_bw_of(cpu_of(rq));
 		/*
 		 * We now free resources of the root_domain we are migrating
 		 * off. In the worst case, sched_setattr() may temporary fail
-		 * until we complete the update.
+		 * until we complete the update. A rebuild that dropped the
+		 * bandwidth of @p leaves nothing to free.
+		 */
+		if (p->dl.dl_bw_gen == dl_bw_gen) {
+			raw_spin_lock(&src_dl_b->lock);
+			__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+			raw_spin_unlock(&src_dl_b->lock);
+		}
+
+		/*
+		 * The room in the root domain of the mask to come was made
+		 * by cpuset_can_attach(): that one counts the bandwidth from
+		 * here on.
 		 */
-		raw_spin_lock(&src_dl_b->lock);
-		__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-		raw_spin_unlock(&src_dl_b->lock);
+		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
+		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
+		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
 	set_cpus_allowed_common(p, ctx);
@@ -3537,10 +3575,18 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 bool dl_task_needs_bw_move(struct task_struct *p,
 			   const struct cpumask *new_mask)
 {
-	if (!dl_task(p))
+	unsigned int cpu;
+
+	/*
+	 * Nothing to move for a special entity, nor for a task boosted into
+	 * the class with no bandwidth of its own: neither is counted, as
+	 * dl_add_task_root_domain() leaves them out.
+	 */
+	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
 		return false;
 
-	return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
+	cpu = dl_task_root_domain_cpu(p);
+	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
 }
 
 /* Assumes rq->lock is held */
@@ -3614,8 +3660,9 @@ void dl_add_task_root_domain(struct task_struct *p)
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
 	/*
 	 * Nothing to count for a special entity, nor for a task with no
-	 * bandwidth, such as an exited one whose bandwidth went with its
-	 * parameters, which the walk of the task list finds too.
+	 * bandwidth: an exited one whose bandwidth went with its
+	 * parameters, which the walk of the task list finds too, and a task
+	 * boosted into the class with none of its own, which gets no record.
 	 */
 	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
@@ -3639,10 +3686,12 @@ void dl_add_task_root_domain(struct task_struct *p)
 	 */
 	if (p->dl.dl_bw_gen == dl_bw_gen)
 		goto unlock;
-	if (p->dl.dl_departed_cpu >= 0) {
-		cpu = p->dl.dl_departed_cpu;
-		if (!cpu_active(cpu))
+	if (!list_empty(&p->dl.dl_departed_node)) {
+		int recorded = p->dl.dl_bw_cpu;
+
+		if (recorded < 0 || !cpu_active(recorded))
 			goto unlock;
+		cpu = recorded;
 	} else if (READ_ONCE(p->__state) == TASK_DEAD) {
 		/*
 		 * Dead and not departed: its bandwidth was released at its
@@ -3659,6 +3708,7 @@ void dl_add_task_root_domain(struct task_struct *p)
 	raw_spin_lock(&dl_b->lock);
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
 	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_cpu = cpu;
 	p->dl.dl_bw_gen = dl_bw_gen;
 unlock:
 	raw_spin_unlock(&dl_departed_lock);
@@ -3799,7 +3849,7 @@ void dl_clear_root_domains(void)
 			dl_clear_root_domain(cpu_rq(cpu)->rd);
 	}
 	list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
-		cpu = dl_se->dl_departed_cpu;
+		cpu = dl_se->dl_bw_cpu;
 		/*
 		 * Counted in the generation that ends here, and nowhere
 		 * since: one a rebuild dropped, for want of an active CPU to
@@ -3808,7 +3858,7 @@ void dl_clear_root_domains(void)
 		 */
 		if (dl_se->dl_bw_gen != dl_bw_gen - 1)
 			continue;
-		if (!cpu_active(cpu))
+		if (cpu < 0 || !cpu_active(cpu))
 			continue;
 		dl_b = dl_bw_of(cpu);
 		raw_spin_lock(&dl_b->lock);
@@ -3883,7 +3933,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 			 * deboosted from priority inheritance, nor for a
 			 * special one: neither has bandwidth in total_bw.
 			 */
-			dl_task_release_bw(p, task_cpu(p));
+			dl_task_release_bw(p);
 			released = true;
 		}
 		sub_rq_bw(&p->dl, &rq->dl);
@@ -3918,7 +3968,7 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	 */
 	if (p->dl.dl_non_contending) {
 		p->dl.dl_non_contending = 0;
-		dl_task_depart(p, task_cpu(p));
+		dl_task_depart(p);
 	}
 
 	/*
@@ -3942,13 +3992,10 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
 	/*
 	 * Boosted back into the class before its 0-lag time (a policy change
 	 * released that bandwidth in sched_dl_overflow()): the inactive
-	 * timer will not remove it, but its leaving the class again. That
-	 * forgets the CPU recorded: moved to another root domain meanwhile,
-	 * the task then removes the bandwidth from that one, as it did
-	 * before departures were recorded.
+	 * timer will not remove it, but its leaving the class again, from
+	 * the root domain of the CPU recorded, which it keeps.
 	 */
-	if (p->dl.dl_departed_cpu >= 0)
-		dl_task_undepart(p);
+	dl_task_undepart(p);
 
 	/* If p is not queued we will update its parameters at next wakeup. */
 	if (!task_on_rq_queued(p)) {
@@ -4155,9 +4202,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	u64 period = attr->sched_period ?: attr->sched_deadline;
 	u64 runtime = attr->sched_runtime;
 	u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
-	int cpus, err = -1, cpu = task_cpu(p);
-	struct dl_bw *dl_b = dl_bw_of(cpu);
+	int cpus, err = -1, cpu;
+	struct dl_bw *dl_b;
 	unsigned long cap;
+	bool counted;
 
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
@@ -4166,6 +4214,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
 		return 0;
 
+	/*
+	 * The bandwidth of a task of the class is counted in the root
+	 * domain that took it, which the CPU the task is on does not tell:
+	 * moved asleep, it has not run there yet. Change it where it is
+	 * counted; a rebuild that dropped it, and a task that enters the
+	 * class, have the root domain of that CPU take the new one. The
+	 * caller holds cpuset_mutex, or the lock of dl_accounting_lock()
+	 * without cpusets, so no rebuild runs meanwhile.
+	 */
+	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
+		  p->dl.dl_bw_gen == dl_bw_gen;
+	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
+	dl_b = dl_bw_of(cpu);
+
 	/*
 	 * Either if a task, enters, leave, or stays -deadline but changes
 	 * its parameters, we may need to update accordingly the total
@@ -4180,7 +4242,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		__dl_add(dl_b, new_bw, cpus);
 		err = 0;
 	} else if (dl_policy(policy) && task_has_dl_policy(p) &&
-		   !__dl_overflow(dl_b, cap, p->dl.dl_bw, new_bw)) {
+		   !__dl_overflow(dl_b, cap, counted ? p->dl.dl_bw : 0,
+				  new_bw)) {
 		/*
 		 * XXX this is slightly incorrect: when the task
 		 * utilization decreases, we should delay the total
@@ -4188,7 +4251,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		 * But this would require to set the task's "inactive
 		 * timer" when the task is not inactive.
 		 */
-		__dl_sub(dl_b, p->dl.dl_bw, cpus);
+		if (counted)
+			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
 		dl_change_utilization(p, new_bw);
 		err = 0;
@@ -4209,11 +4273,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 * run meanwhile: the caller holds cpuset_mutex, or the lock of
 	 * dl_accounting_lock() without cpusets.
 	 */
-	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
-		lockdep_assert_cpuset_lock_held();
-		lockdep_assert_dl_accounting_held();
-		if (p->dl.dl_departed_cpu >= 0)
-			dl_task_release_bw(p, -1);
+	if (!err && dl_policy(policy)) {
+		if (!task_has_dl_policy(p)) {
+			lockdep_assert_cpuset_lock_held();
+			lockdep_assert_dl_accounting_held();
+			/*
+			 * Whatever bandwidth the task still has counted
+			 * somewhere, the new one replaces: on the departed
+			 * list, or taken off it by a boost into the class
+			 * (switched_to_dl()), the record says where.
+			 */
+			if (p->dl.dl_bw_cpu >= 0)
+				dl_task_release_bw(p);
+		}
+		p->dl.dl_bw_cpu = cpu;
 		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
@@ -4354,7 +4427,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	init_dl_task_timer(dl_se);
 	init_dl_inactive_task_timer(dl_se);
 	__dl_clear_params(dl_se);
-	dl_se->dl_departed_cpu = -1;
+	dl_se->dl_bw_cpu = -1;
+	INIT_LIST_HEAD(&dl_se->dl_departed_node);
 }
 
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-- 
2.47.3

[PATCH v4 17/26] sched/deadline: Move a task's bandwidth only where room was made

[Mathieu Desnoyers]

set_cpus_allowed_dl() moves the bandwidth of a deadline task off the
root domain that counts it whenever the new mask leaves that root
domain, on the strength of the room cpuset_can_attach() made in the one
the task moves to, as its comment says. Nothing checks that the room
was made, and two paths reach it without any:

 - cpuset_can_attach() counts the deadline tasks of the attach it is
   asked about. A task that becomes one after that has nothing set
   aside, and the attach moves its bandwidth all the same: the root
   domain it lands in ends short of what its tasks hold, and underflows
   when they leave. A task that flips between SCHED_DEADLINE and
   SCHED_OTHER while it is moved in and out of a one-CPU partition root
   drives total_bw of that partition to 2^64 - 262144 in 8-CPU VMs of
   v7.4-rc1, in 2 VMs of 2;
 - sched_setaffinity() of a deadline task takes the same path. With
   admission control on, a mask that does not cover the task's root
   domain is refused before this (__sched_setaffinity()), but with
   sched_rt_runtime_us at -1 nothing refuses it.

Say in the affinity context whether the caller made that room, as
SCA_DL_BW_RESERVED, and move the bandwidth only then. The bandwidth of a
task whose move no one made room for stays where it is counted, and the
next rebuild of the accounting attributes it to the root domain of the
task's cpuset.

Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h   |  2 ++
 kernel/cgroup/cpuset.c  | 13 ++++++++++++-
 kernel/sched/core.c     | 14 ++++++++++++++
 kernel/sched/deadline.c | 15 ++++++++++-----
 kernel/sched/sched.h    |  2 ++
 5 files changed, 40 insertions(+), 6 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 28b579bff9e1..b0c36891415d 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1979,6 +1979,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  * Return: zero if successful, or a negative error code
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
+extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
+					const struct cpumask *new_mask);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7628cf77bdfd..bf9a7f88c0ed 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3261,6 +3261,7 @@ static cpumask_var_t cpus_attach;
 
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
+	bool reserved = false;
 	struct mm_struct *mm;
 
 	lockdep_assert_cpuset_lock_held();
@@ -3270,11 +3271,21 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	else
 		cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
 			       subpartitions_cpus);
+	/*
+	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
+	 * deadline tasks cpuset_can_attach() counted: say so, so that this
+	 * task's bandwidth moves there with it.
+	 */
+	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+		reserved = true;
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	if (reserved)
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
+	else
+		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 55a4f2d8294e..6a2d660ffea1 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3272,6 +3272,20 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
+/*
+ * The same, for a caller that has made room in the root domain of @new_mask
+ * for the deadline bandwidth of @p: that bandwidth moves there with it.
+ */
+int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+{
+	struct affinity_context ac = {
+		.new_mask  = new_mask,
+		.flags     = SCA_DL_BW_RESERVED,
+	};
+
+	return __set_cpus_allowed_ptr(p, &ac);
+}
+
 /*
  * Change a given task's CPU affinity to the intersection of its current
  * affinity mask and @subset_mask, writing the resulting mask to @new_mask.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 7a7101e6f7a0..1b730e22ee6c 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3537,12 +3537,17 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	WARN_ON_ONCE(!dl_task(p));
 
 	/*
-	 * Migrating a SCHED_DEADLINE task between exclusive
-	 * cpusets (different root_domains) entails a bandwidth
-	 * update. We already made space for us in the destination
-	 * domain (see cpuset_can_attach()).
+	 * Migrating a SCHED_DEADLINE task between exclusive cpusets
+	 * (different root_domains) entails a bandwidth update, and only
+	 * where the caller has made room for it: cpuset_attach_task() says
+	 * so of the tasks cpuset_can_attach() counted, and of no others.
+	 * The bandwidth of a task whose move no one made room for stays
+	 * where it is counted, for the next rebuild of the accounting to
+	 * attribute, rather than leaving the root domain it moves to short
+	 * of what its tasks hold.
 	 */
-	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
+	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+	    dl_task_needs_bw_move(p, ctx->new_mask)) {
 		unsigned int src_cpu = dl_task_root_domain_cpu(p);
 		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
 		unsigned int cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a06cf2e0d983..57aadc5b1ba6 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2956,6 +2956,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 #define SCA_MIGRATE_DISABLE	0x02
 #define SCA_MIGRATE_ENABLE	0x04
 #define SCA_USER		0x08
+/* The caller made room for @p's deadline bandwidth where the mask points. */
+#define SCA_DL_BW_RESERVED	0x20
 
 extern void update_group_capacity(struct sched_domain *sd, int cpu);
 
-- 
2.47.3

[PATCH v4 18/26] cgroup/cpuset: Wait for an attach before rebuilding the sched domains

[Mathieu Desnoyers]

cpuset_can_attach() sets aside, in the destination's root domain, the
bandwidth of the deadline tasks it is about to move, and drops
cpuset_mutex until cpuset_attach() moves them. A rebuild of the sched
domains in between, from a write of sched_rt_runtime_us or from the
energy model, takes cpuset_mutex, clears the root domains and counts
each task where its cpuset says: the reservation is gone. The attach
then moves each task's bandwidth on the strength of it, and the
destination's total_bw ends short of what its tasks hold: it admits
deadline tasks beyond its capacity, and underflows when they leave.

The writes of a cpuset's files wait for an attach in progress,
wait_attach_done_lock(), since commit 75f7a25ec6bb ("cgroup/cpuset:
Prevent race between task attach and cpuset state change"), and CPU
hotplug holds cpus_write_lock, which an attach holds for reading
throughout. Wait in rebuild_sched_domains_cpuslocked() as well, where a
rebuild from hotplug finds no attach in progress, and in
cpuset_full_lock(), which the cgroup v1 writes of sched_load_balance and
sched_relax_domain_level and hk_sd_workfn() rebuild under, in the order
the writes of cpus and cpus.partition take it. The other rebuilds, at
the online, offline and kill of a cpuset, run under cgroup_mutex, which
the attach holds throughout.

The move_vs_rebuild case of deadline_bw_test, added later in this
series, moves a process of 2049 threads, the last a deadline task
asleep, in and out of a one-CPU partition root while sched_rt_runtime_us
is written 20 times, 5ms apart, six rounds. Without this patch a round
fails in 5 VMs of 8, the partition root left admitting no task at all,
or the other CPUs admitting 3 deadline tasks of 10% more than they have
room for; with it, it passes in 8 of 8. A rebuild later in the attach
does no harm until the next patch gives back what an attach did not
move: it would then give back a reservation already gone.

Found by the Sashiko AI review of the second posting of this series, the
cgroup v1 writes by that of the third.

Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/cgroup/cpuset.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index bf9a7f88c0ed..2ee58d46da0e 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -296,17 +296,19 @@ void lockdep_assert_cpuset_lock_held(void)
 	lockdep_assert_held(&cpuset_mutex);
 }
 
+static void wait_attach_done_lock(void) __acquires(&cpuset_mutex);
+
 /**
  * cpuset_full_lock - Acquire full protection for cpuset modification
  *
  * Takes both CPU hotplug read lock (cpus_read_lock()) and cpuset mutex
- * to safely modify cpuset data.
+ * to safely modify cpuset data, the latter once no attach is in progress.
  */
 void cpuset_full_lock(void)
 {
 	mutex_lock(&cpuset_top_mutex);
 	cpus_read_lock();
-	mutex_lock(&cpuset_mutex);
+	wait_attach_done_lock();
 }
 
 void cpuset_full_unlock(void)
@@ -1005,7 +1007,13 @@ void rebuild_sched_domains_locked(void)
 
 static void rebuild_sched_domains_cpuslocked(void)
 {
-	mutex_lock(&cpuset_mutex);
+	/*
+	 * An attach in progress has set aside, in the root domain its tasks
+	 * move to, the deadline bandwidth it moves once it takes
+	 * cpuset_mutex again: the rebuild of the accounting would drop that
+	 * reservation, and the move would go through on its strength.
+	 */
+	wait_attach_done_lock();
 	rebuild_sched_domains_locked();
 	mutex_unlock(&cpuset_mutex);
 }
-- 
2.47.3

[PATCH v4 19/26] cgroup/cpuset: Give back the bandwidth an attach did not move

[Mathieu Desnoyers]

cpuset_can_attach() counts the bandwidth of every migrating deadline
task in sum_migrate_dl_bw, and cpuset_reserve_dl_bw() sets that sum
aside in the destination's root domain. The attach that follows moves
the bandwidth of each task that is still a deadline task, and nothing
answers for the reservation of one that is not: what was set aside for a
task that left SCHED_DEADLINE, or died, since it was counted stays in
the destination's total_bw, which admits that much less until the next
rebuild of the accounting. A task that flips between SCHED_DEADLINE and
SCHED_OTHER every millisecond while it is moved in and out of a one-CPU
partition root 200 times leaves total_bw of the partition at 367000,
681572 or 2^64 - 576716, where it was 52428, its fair server, in 4 VMs
of 10 of v7.4-rc1 with the patches of this series before this one.

Keep the tally of what the attach moves, and give the rest back once it
is through. The tally takes what set_cpus_allowed_dl() moved, written
into the affinity context under the runqueue lock: the root domain that
cpuset_attach_task() reads without that lock can change under it, CPU
hotplug moving the task in dl_task_offline_migration(), and the tally
would count a move that did not happen. With this change, total_bw of
the partition is 52428 after those 200 moves, in 6 VMs of 6.

Found by the randomized case of the selftest added earlier in this
series, which caught a fair server it could no longer lower in a
partition root with no room left.

Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/sched.h           |  3 ++-
 kernel/cgroup/cpuset-internal.h |  2 ++
 kernel/cgroup/cpuset.c          | 39 ++++++++++++++++++++++++++-------
 kernel/sched/core.c             | 11 +++++++---
 kernel/sched/deadline.c         |  5 ++++-
 kernel/sched/sched.h            |  2 ++
 6 files changed, 49 insertions(+), 13 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index b0c36891415d..bd611dbb57b0 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1980,7 +1980,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
  */
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
-					const struct cpumask *new_mask);
+					const struct cpumask *new_mask,
+					u64 *moved);
 extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
 extern void release_user_cpus_ptr(struct task_struct *p);
 extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 153067b164c1..6f3d43d8dbad 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -170,6 +170,8 @@ struct cpuset {
 	int nr_migrate_dl_tasks;
 	/* DL bandwidth that needs destination reservation for this attach. */
 	u64 sum_migrate_dl_bw;
+	/* Of that bandwidth, what the attach moved: the rest is given back. */
+	u64 sum_moved_dl_bw;
 	/*
 	 * CPU used for temporary DL bandwidth allocation during attach;
 	 * -1 if no DL bandwidth was allocated in the current attach.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 2ee58d46da0e..f93f5547714a 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3127,10 +3127,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
 	llist_for_each_entry_safe(cs, next, lnode, attach_node) {
 		init_llist_node(&cs->attach_node);
 		if (cs->nr_migrate_dl_tasks) {
-			if (cancel && cs->dl_bw_cpu >= 0)
-				dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+			/*
+			 * Give back the bandwidth the attach did not move:
+			 * all of the reservation when it is cancelled, and
+			 * what was set aside for the tasks that are no
+			 * longer deadline tasks otherwise. Theirs is
+			 * released by the class they left, from the root
+			 * domain that counts it.
+			 */
+			if (cs->dl_bw_cpu >= 0) {
+				u64 bw = cancel ? cs->sum_migrate_dl_bw :
+					 cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+
+				if (bw)
+					dl_bw_free(cs->dl_bw_cpu, bw);
+			}
 			cs->nr_migrate_dl_tasks = 0;
 			cs->sum_migrate_dl_bw = 0;
+			cs->sum_moved_dl_bw = 0;
 			cs->dl_bw_cpu = -1;
 		}
 	}
@@ -3281,19 +3295,28 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 			       subpartitions_cpus);
 	/*
 	 * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
-	 * deadline tasks cpuset_can_attach() counted: say so, so that this
-	 * task's bandwidth moves there with it.
+	 * deadline tasks cpuset_can_attach() counted: offer it, so that this
+	 * task's bandwidth moves there with it. A task that became a
+	 * deadline task since has nothing set aside, so it moves its
+	 * bandwidth only where the rest of the reservation covers it.
 	 */
-	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+	if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus) &&
+	    cs->sum_moved_dl_bw + task->dl.dl_bw <= cs->sum_migrate_dl_bw)
 		reserved = true;
 	/*
 	 * can_attach beforehand should guarantee that this doesn't
 	 * fail.  TODO: have a better way to handle failure here
 	 */
-	if (reserved)
-		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
-	else
+	if (reserved) {
+		u64 moved;
+
+		/* The move decides: the tally counts what it moved. */
+		WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
+							  &moved));
+		cs->sum_moved_dl_bw += moved;
+	} else {
 		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+	}
 
 	if (cpuset_v2() && !attach_ctx.mems_updated)
 		return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6a2d660ffea1..494a815c6bb0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3274,16 +3274,21 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
 /*
  * The same, for a caller that has made room in the root domain of @new_mask
- * for the deadline bandwidth of @p: that bandwidth moves there with it.
+ * for the deadline bandwidth of @p: that bandwidth moves there with it, and
+ * @moved takes what the move counts there, 0 where it moved none.
  */
-int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+int set_cpus_allowed_bw_reserved(struct task_struct *p,
+				 const struct cpumask *new_mask, u64 *moved)
 {
 	struct affinity_context ac = {
 		.new_mask  = new_mask,
 		.flags     = SCA_DL_BW_RESERVED,
 	};
+	int ret;
 
-	return __set_cpus_allowed_ptr(p, &ac);
+	ret = __set_cpus_allowed_ptr(p, &ac);
+	*moved = ac.dl_bw_moved;
+	return ret;
 }
 
 /*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 1b730e22ee6c..f169cf35e313 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3544,7 +3544,9 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * The bandwidth of a task whose move no one made room for stays
 	 * where it is counted, for the next rebuild of the accounting to
 	 * attribute, rather than leaving the root domain it moves to short
-	 * of what its tasks hold.
+	 * of what its tasks hold. @ctx tells the caller what moved: the
+	 * record of a task is read here, under its runqueue lock, and the
+	 * caller cannot keep a reservation for a move this decides against.
 	 */
 	if ((ctx->flags & SCA_DL_BW_RESERVED) &&
 	    dl_task_needs_bw_move(p, ctx->new_mask)) {
@@ -3572,6 +3574,7 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 		cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
 		p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
 		p->dl.dl_bw_gen = dl_bw_gen;
+		ctx->dl_bw_moved = p->dl.dl_bw;
 	}
 
 	set_cpus_allowed_common(p, ctx);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 57aadc5b1ba6..10a4750af6f3 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2662,6 +2662,8 @@ extern const u32		sched_prio_to_wmult[40];
 struct affinity_context {
 	const struct cpumask	*new_mask;
 	struct cpumask		*user_mask;
+	/* Out: the deadline bandwidth the move counts in the new root domain. */
+	u64			dl_bw_moved;
 	unsigned int		flags;
 };
 
-- 
2.47.3

[PATCH v4 20/26] sched/deadline: Credit a rebuild to the CPU recorded with the task's bandwidth

[Mathieu Desnoyers]

A rebuild of the root domains' accounting counts each deadline task in
the root domain of the first active CPU of its cpuset,
dl_add_task_root_domain() through dl_get_task_effective_cpus(): a guess
that admission did not make, wrong three ways:

 - the cpuset of a task on a housekeeping CPU can hold CPUs isolated by
   isolcpus="domain" beside it, as the root cpuset does, and without
   cpusets the mask is every CPU the task may run on: an isolated CPU
   coming first has the task counted in the default root domain, which
   underflows when the task leaves the class, or dies;
 - a task confined to isolated CPUs that all go offline, which nothing
   refuses, has no active CPU left, and the rebuild BUG()s;
 - a throttled task whose CPUs went offline is moved, with its
   bandwidth, to an active CPU by dl_task_offline_migration(), and the
   rebuild counts it again where its cpuset says.

Take the CPU recorded with the task's bandwidth where it is active and
the task's mask allows it: the rebuild counts the bandwidth where
admission, or the last move, put it. Otherwise take the first active CPU
of the task's own mask, a housekeeping one where it has any: a change of
partitions sets the masks of its tasks before its rebuild, while the
housekeeping mask that dl_get_task_effective_cpus() reads lags it. The
cpuset answers for a task with no active CPU of its own, as before, and
a task with no active CPU at all is left out: the generation of
"sched/deadline: Keep departing bandwidth across a root domain rebuild"
keeps its release out as well, and dl_task_offline_migration() counts it
where it moves it.

Found by the Sashiko AI reviews of the first three postings of this
series.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 74 +++++++++++++++++++++++++++++++++++------
 1 file changed, 63 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index f169cf35e313..ed951246e3d2 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -878,14 +878,20 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
 			__dl_sub(dl_b, p->dl.dl_bw,
 				 cpumask_weight(src_rq->rd->span));
 			raw_spin_unlock(&dl_b->lock);
-
-			dl_b = &later_rq->rd->dl_bw;
-			raw_spin_lock(&dl_b->lock);
-			__dl_add(dl_b, p->dl.dl_bw,
-				 cpumask_weight(later_rq->rd->span));
-			raw_spin_unlock(&dl_b->lock);
 		}
+		/*
+		 * Counted from here in the root domain the task moves to,
+		 * also where a rebuild left it out for want of an active
+		 * CPU: a walk that reaches it after this finds the
+		 * generation current and leaves it alone.
+		 */
+		dl_b = &later_rq->rd->dl_bw;
+		raw_spin_lock(&dl_b->lock);
+		__dl_add(dl_b, p->dl.dl_bw,
+			 cpumask_weight(later_rq->rd->span));
+		raw_spin_unlock(&dl_b->lock);
 		p->dl.dl_bw_cpu = later_rq->cpu;
+		p->dl.dl_bw_gen = dl_bw_gen;
 	}
 
 	set_task_cpu(p, later_rq->cpu);
@@ -3656,6 +3662,47 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	cpuset_cpus_allowed_locked(p, cpus);
 }
 
+/*
+ * A CPU of the root domain that a rebuild of the accounting counts @p's
+ * bandwidth in. The one recorded with it, where it is active and the
+ * cpuset of @p allows it: that of the root domain that admitted the
+ * bandwidth, or that the last move put it in, which the cpuset alone does
+ * not tell where it holds CPUs of two root domains, housekeeping ones
+ * beside the ones isolcpus="domain" leaves to the default root domain. The
+ * record answers as well for a task whose own CPUs went offline, moved to
+ * an active one by dl_task_offline_migration(). Else the first active CPU
+ * the cpuset allows, a housekeeping one where it has any; and nr_cpu_ids
+ * where it allows none, which the isolated CPUs allow, no deadline task
+ * of theirs keeping the last of them online.
+ */
+static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
+{
+	struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	int recorded = p->dl.dl_bw_cpu;
+	unsigned int cpu;
+
+	/*
+	 * The task's own CPUs first: a change of cpusets sets the mask of
+	 * each task before the rebuild it makes, where the housekeeping mask
+	 * that dl_get_task_effective_cpus() reads lags that change. With
+	 * none of them active, the CPU recorded is where
+	 * dl_task_offline_migration() moved the task; else its cpuset says.
+	 */
+	cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+	if (cpumask_empty(msk)) {
+		if (recorded >= 0 && cpu_active(recorded))
+			return recorded;
+		dl_get_task_effective_cpus(p, msk);
+		cpumask_and(msk, msk, cpu_active_mask);
+	}
+	if (recorded >= 0 && cpumask_test_cpu(recorded, msk))
+		return recorded;
+	cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+	if (cpu < nr_cpu_ids)
+		return cpu;
+	return cpumask_first(msk);
+}
+
 /* From dl_rebuild_rd_accounting(), under the locks it asserts. */
 void dl_add_task_root_domain(struct task_struct *p)
 {
@@ -3663,7 +3710,6 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct rq *rq;
 	struct dl_bw *dl_b;
 	unsigned int cpu;
-	struct cpumask *msk;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
 	/*
@@ -3677,10 +3723,16 @@ void dl_add_task_root_domain(struct task_struct *p)
 		return;
 	}
 
-	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
-	dl_get_task_effective_cpus(p, msk);
-	cpu = cpumask_first_and(cpu_active_mask, msk);
-	BUG_ON(cpu >= nr_cpu_ids);
+	cpu = dl_task_rebuild_cpu(p);
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * No active CPU to count this task's bandwidth: leave it out
+		 * of this rebuild, where dl_bw_gen keeps its release out as
+		 * well, until one has a CPU for it.
+		 */
+		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+		return;
+	}
 
 	raw_spin_lock(&dl_departed_lock);
 	/*
-- 
2.47.3

[PATCH v4 21/26] sched/deadline: Cancel the replenishment timer with the parameters

[Mathieu Desnoyers]

The replenishment timer of a throttled task is armed for its next
period, which can come after its 0-lag time: a task that yielded keeps
runtime, and a constrained-deadline task that overruns by little
reaches its 0-lag time within the same period. Leaving SCHED_DEADLINE,
such a task has its parameters cleared at the 0-lag time with that
timer still armed.

Back in the class before it fires, the task throttles again, and
start_dl_timer() leaves the armed timer where it is:

	if (!hrtimer_is_queued(timer)) {
		if (!dl_server(dl_se))
			get_task_struct(dl_task_of(dl_se));
		hrtimer_start(timer, act, HRTIMER_MODE_ABS_HARD);
	}

so the replenishment comes at the period of the reservation the task
left: too early, which hands it runtime it has not earned, or up to a
period too late.

Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole") left
that timer armed on purpose, as what reset a task that stayed out of
the class. The 0-lag time took that over and left the timer behind:
cancel it where the parameters go, and where switched_from_dl()
releases the bandwidth of a task that leaves asleep past its 0-lag
time, a constrained task that overran as it blocked having the timer
armed there too.

Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ed951246e3d2..a0b05ff8c6da 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -607,6 +607,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 				if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
 					sub_rq_bw(dl_se, &rq->dl);
 				dl_task_release_bw(p);
+				cancel_replenish_timer(dl_se);
 				__dl_clear_params(dl_se);
 			}
 		}
@@ -2375,6 +2376,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		}
 
 		dl_task_release_bw(p);
+		cancel_replenish_timer(dl_se);
 		__dl_clear_params(dl_se);
 
 		goto unlock;
@@ -4009,8 +4011,10 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 		 * dl_non_contending cleared under it by migrate_task_rq_dl():
 		 * it finds !dl_task(p).
 		 */
-		if (released)
+		if (released) {
+			cancel_replenish_timer(&p->dl);
 			__dl_clear_params(&p->dl);
+		}
 	}
 
 	/*
-- 
2.47.3

[PATCH v4 22/26] sched/deadline: Count the bandwidth a boosted task is given

[Mathieu Desnoyers]

sched_setattr(SCHED_DEADLINE) on a task that priority inheritance
boosted into the class leaves the this_bw of its runqueue without the
task's new bandwidth: switched_to_dl(), which hands a sleeping task's
bandwidth to this_bw as it enters the class, does not run for a task in
it already. Its wake-up gives that bandwidth to running_bw alone:

	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);

and the two sums GRUB reclaims from disagree by that bandwidth until the
task migrates, where the enqueue adds it (add_rq_bw()).

Change the bandwidth the runqueue counts in __setparam_dl(), which every
task that gains deadline parameters goes through, rather than in
sched_dl_overflow(), which only a task of the class reaches that way. A
runqueue that counts nothing of the task takes the new bandwidth now,
where "sched/deadline: Do not take a waking task's bandwidth off a
runqueue it left" left it to the next enqueue: that enqueue finds it
counted and adds nothing, and a waking task's task_cpu() is the runqueue
it wakes on already. The bandwidth of a special entity stays counted
nowhere, the request telling one, since the entity's flags are those of
its old parameters: a sugov kthread given a reservation of its own gets
that one counted.

A reservation with the bandwidth the kthread had needs more:
sched_dl_overflow() returns at once for a task of the class whose
bandwidth does not change, before asking whether that bandwidth is
counted, which a sugov kthread's is not, nor that of a task a rebuild
left out. Ask first, and admit and record bandwidth that is not counted
as new. A runqueue that counts the bandwidth a change keeps has nothing
to change: dl_rq_change_utilization() would take a sleeping task's
running_bw off before its 0-lag time and cancel its inactive timer for
nothing. The early return was found by the Sashiko AI review of the
third posting of this series.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 55 ++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a0b05ff8c6da..ac01a0023c8f 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -478,22 +478,36 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+/*
+ * The runqueue a sleeping task is on counts the bandwidth the task has:
+ * give it @new_bw in place of what it counts of the task's.
+ */
 static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 {
-	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct rq *rq = task_rq(p);
 
 	if (task_on_rq_queued(p))
 		return;
 
 	/*
-	 * Waking up on another CPU, the task has its bandwidth on no
-	 * runqueue: the enqueue to come charges the new one, with the new
-	 * parameters, which __setparam_dl() sets right after this.
+	 * No runqueue counts bandwidth of the task's: this one takes the
+	 * new one, and the enqueue to come leaves it there (add_rq_bw() of
+	 * a task a runqueue counts). A task waking up on another CPU has
+	 * none of its bandwidth on a runqueue, and task_cpu() points at the
+	 * one it wakes on.
 	 */
-	if (!p->dl.dl_bw_oncpu)
+	if (!dl_se->dl_bw_oncpu) {
+		__add_rq_bw(new_bw, &rq->dl);
+		dl_se->dl_bw_oncpu = 1;
+		return;
+	}
+
+	/* Counted with the bandwidth it keeps: nothing changes. */
+	if (new_bw == dl_se->dl_bw)
 		return;
 
-	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+	dl_rq_change_utilization(rq, dl_se, new_bw);
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4274,10 +4288,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
 
-	/* !deadline task may carry old deadline bandwidth */
-	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
-		return 0;
-
 	/*
 	 * The bandwidth of a task of the class is counted in the root
 	 * domain that took it, which the CPU the task is on does not tell:
@@ -4289,6 +4299,16 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 */
 	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
 		  p->dl.dl_bw_gen == dl_bw_gen;
+
+	/*
+	 * !deadline task may carry old deadline bandwidth. The same
+	 * bandwidth, counted, is nothing to admit; not counted, a special
+	 * entity given a reservation of its own, or a task a rebuild left
+	 * out, it is admitted as new below.
+	 */
+	if (new_bw == p->dl.dl_bw && counted)
+		return 0;
+
 	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
 	dl_b = dl_bw_of(cpu);
 
@@ -4318,7 +4338,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		if (counted)
 			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
-		dl_change_utilization(p, new_bw);
 		err = 0;
 	} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
 		/*
@@ -4368,6 +4387,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	u64 period = attr->sched_period ?: attr->sched_deadline;
+
+	/*
+	 * The bandwidth of a special entity is counted on no runqueue, and
+	 * the request tells one, the flags of the entity being those of the
+	 * parameters it had. The rest have theirs changed here, where their
+	 * parameters are:
+	 * switched_to_dl() hands it to the runqueue of a task that enters
+	 * the class, and a task that priority inheritance boosted into the
+	 * class is in it already, so nothing dequeues it and enqueues it
+	 * back for that.
+	 */
+	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
 
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
-- 
2.47.3

[PATCH v4 23/26] sched/debug: Show the deadline bandwidth of each runqueue

[Mathieu Desnoyers]

sched_debug prints the bandwidth each root domain has handed out
(dl_bw->total_bw), and nothing of the two sums each runqueue keeps:
this_bw, the bandwidth of every deadline task of that CPU, and
running_bw, the part of it whose tasks are active. GRUB drains the
runtime of a reclaiming task at the rate the two give, the bandwidth of
its CPU that is reserved and active (grub_reclaim(): max_bw - (this_bw -
running_bw) - extra_bw): the less of its CPU is active, the slower its
runtime drains, and the longer it runs on.

Both sums have invariants a test can check, which the kernel itself
warns about when they break: neither may wrap, and running_bw never
exceeds this_bw. Print them.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/debug.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 5f8303630296..463f077fa703 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1173,6 +1173,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 	SEQ_printf(m, "  .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
 
 	PU(dl_nr_running);
+	PU(running_bw);
+	PU(this_bw);
 	dl_bw = &cpu_rq(cpu)->rd->dl_bw;
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
 	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
-- 
2.47.3

[PATCH v4 24/26] selftests/sched: Add a test of SCHED_DEADLINE policy round trips

[Mathieu Desnoyers]

Add deadline_switch_test, with a case for each of "sched/deadline: Do
not lose a throttled task switched out and back" and "sched/deadline:
Release the bandwidth of a task that leaves the class asleep", earlier
in this series:

 - throttled_round_trip: a task with a runtime of 10ms per 2s period
   spins, so that each replenishment throttles it again at once. Right
   after a throttle it is switched to SCHED_OTHER, and back 200us after
   its deadline: past its replenishment timer, and before its 0-lag
   time, which the overrun scaled by 200 puts later. It must run again
   within a period and a half. Three attempts;
 - asleep_round_trip: a task of 9ms every 10ms, running 0.2ms and
   sleeping 50ms, is switched to SCHED_OTHER and back while it sleeps
   past its 0-lag time, enough times that a leak of its bandwidth at
   each round trip fills the root domain: sched_setattr() then fails
   with EBUSY. Skipped without admission control (sched_rt_runtime_us at
   -1). A kernel that leaks is left without SCHED_DEADLINE bandwidth
   until a reboot, so this case runs last.

A third case, sysrq_normalize, checks that a task can enter
SCHED_DEADLINE after sysrq-n took one out of it: "sched: Take
cpuset_mutex for a deadline policy change only with pi" has the
cpuset_mutex that sysrq-n left locked. It renices every real-time task
of the system, so it runs only with DEADLINE_TEST_SYSRQ set, and last.

In 8 VMs of 8 vCPUs each, v7.3 and v7.4-rc1 fail the two round trips,
and with the two fixes both pass, every time. sysrq_normalize finds the
admission blocked on v7.4-rc1, in 2 VMs of 2, and passes with the fix.

The tests of the deadline class get a directory of their own,
sched/deadline, for those of the other policies to follow, and that
directory joins the selftest targets.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/Makefile              |   1 +
 .../selftests/sched/deadline/.gitignore       |   1 +
 .../testing/selftests/sched/deadline/Makefile |   8 +
 .../sched/deadline/deadline_switch_test.c     | 440 ++++++++++++++++++
 4 files changed, 450 insertions(+)
 create mode 100644 tools/testing/selftests/sched/deadline/.gitignore
 create mode 100644 tools/testing/selftests/sched/deadline/Makefile
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_switch_test.c

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 07c3719f8b04..35b0c5e86ba7 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -120,6 +120,7 @@ TARGETS += rlimits
 TARGETS += rseq
 TARGETS += rtc
 TARGETS += rust
+TARGETS += sched/deadline
 TARGETS += sched_ext
 TARGETS += seccomp
 TARGETS += sgx
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
new file mode 100644
index 000000000000..9f203c455b91
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -0,0 +1 @@
+deadline_switch_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
new file mode 100644
index 000000000000..3f4dc1ec6119
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
+LDLIBS += -lpthread
+
+TEST_GEN_PROGS := deadline_switch_test
+
+include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/deadline_switch_test.c b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
new file mode 100644
index 000000000000..5c6bb0037e5a
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_switch_test.c
@@ -0,0 +1,440 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A SCHED_DEADLINE task switched to SCHED_OTHER and back:
+ *
+ *  - throttled: switched out after its runtime ran out, and back after
+ *    its deadline but before its 0-lag time, it must run again, the
+ *    replenishment it missed while out of SCHED_DEADLINE or not;
+ *  - asleep past its 0-lag time: its bandwidth must leave the root domain
+ *    when it leaves SCHED_DEADLINE, or the switch back adds it again and
+ *    sched_setattr() soon refuses SCHED_DEADLINE with EBUSY.
+ *
+ * On a kernel that leaks, the second case leaves the root domain without
+ * bandwidth for new SCHED_DEADLINE tasks until a reboot.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+/* Written by the child, read by the parent. */
+struct shared {
+	uint64_t stamp;		/* last time the child ran */
+	uint64_t run_start;	/* when it last resumed after a gap */
+	uint64_t runs;		/* resumptions after a gap */
+	uint64_t sleeps;	/* sleeps begun */
+};
+
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void sleep_until(uint64_t t)
+{
+	struct timespec ts = { .tv_sec = t / NSEC_PER_SEC, .tv_nsec = t % NSEC_PER_SEC };
+
+	while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL) == EINTR)
+		;
+}
+
+static int set_deadline(pid_t pid, uint64_t runtime, uint64_t period)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+static struct shared *sh;
+
+/* A child that dies with the test rather than keep its bandwidth. */
+static pid_t fork_child(void)
+{
+	pid_t parent = getpid();
+	pid_t pid = fork();
+
+	if (!pid && (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent))
+		_exit(KSFT_FAIL);
+	return pid;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	/* A task lost in SCHED_DEADLINE runs again, and dies, once out of it. */
+	set_other(pid);			/* gone already: nothing to switch */
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+/* The child died before the case was done: @status says whether it skips. */
+static int gone(int status)
+{
+	if (WIFEXITED(status) && WEXITSTATUS(status) == KSFT_SKIP) {
+		ksft_print_msg("SCHED_DEADLINE refused\n");
+		return KSFT_SKIP;
+	}
+	if (WIFSIGNALED(status))
+		ksft_print_msg("the task died of signal %d\n", WTERMSIG(status));
+	else
+		ksft_print_msg("the task exited with %d\n", WEXITSTATUS(status));
+	return KSFT_FAIL;
+}
+
+/* Spins: runs out of its runtime at once, and is throttled. */
+static void spinner(void)
+{
+	uint64_t last = now_ns();
+
+	for (;;) {
+		uint64_t t = now_ns();
+
+		if (t - last > 2 * NSEC_PER_MSEC) {
+			__atomic_store_n(&sh->run_start, t, __ATOMIC_RELAXED);
+			__atomic_fetch_add(&sh->runs, 1, __ATOMIC_RELEASE);
+		}
+		__atomic_store_n(&sh->stamp, t, __ATOMIC_RELEASE);
+		last = t;
+	}
+}
+
+#define LOST_RUNTIME	(10 * NSEC_PER_MSEC)
+#define LOST_PERIOD	(2 * NSEC_PER_SEC)
+#define LOST_MARGIN	(200 * NSEC_PER_USEC)
+
+/*
+ * The window between the deadline and the 0-lag time is the overrun
+ * scaled by period/runtime: 200 here, so that an overrun of 2us leaves
+ * 400us. A runtime longer than a tick keeps an overrun caught at the tick
+ * from carrying past the next replenishment, which comes a period after
+ * the last. The switch back comes LOST_MARGIN after the deadline, once
+ * the replenishment timer has fired on a task out of SCHED_DEADLINE.
+ */
+static int test_throttled_round_trip(void)
+{
+	int attempt, ret = KSFT_PASS, err;
+	pid_t pid;
+
+	memset(sh, 0, sizeof(*sh));
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+			_exit(KSFT_SKIP);
+		spinner();
+	}
+	for (attempt = 0; attempt < 3 && ret == KSFT_PASS; attempt++) {
+		uint64_t runs, r, s, t;
+
+		/* A resumption at a replenishment, D, then its throttle. */
+		runs = __atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->runs, __ATOMIC_ACQUIRE) == runs) {
+			if (now_ns() - t > 3 * LOST_PERIOD) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not resume within 3 periods\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		r = __atomic_load_n(&sh->run_start, __ATOMIC_RELAXED);
+		do {
+			s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+			sleep_ns(2 * NSEC_PER_MSEC);
+		} while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) != s);
+		/* Throttled until D + period: out, and back after that. */
+		err = set_other(pid);
+		if (!err) {
+			sleep_until(r + LOST_PERIOD + LOST_MARGIN);
+			err = set_deadline(pid, LOST_RUNTIME, LOST_PERIOD);
+		}
+		if (err) {
+			ksft_print_msg("policy switch: %s\n", strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(10 * NSEC_PER_MSEC);
+		s = __atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE);
+		t = now_ns();
+		while (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s &&
+		       now_ns() - t < LOST_PERIOD + LOST_PERIOD / 2)
+			sleep_ns(NSEC_PER_MSEC);
+		if (__atomic_load_n(&sh->stamp, __ATOMIC_ACQUIRE) == s) {
+			ksft_print_msg("throttled, switched out and back: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+#define LEAK_RUNTIME	(9 * NSEC_PER_MSEC)
+#define LEAK_PERIOD	(10 * NSEC_PER_MSEC)
+
+/* Runs 0.2ms, sleeps 50ms. */
+static void sleeper(void)
+{
+	for (;;) {
+		uint64_t end = now_ns() + 200 * NSEC_PER_USEC;
+
+		while (now_ns() < end)
+			;
+		__atomic_fetch_add(&sh->sleeps, 1, __ATOMIC_RELEASE);
+		sleep_ns(50 * NSEC_PER_MSEC);
+	}
+}
+
+/*
+ * 90% of a CPU per round trip: a leak fills the root domain within about
+ * as many round trips as it has CPUs.
+ */
+static int test_asleep_round_trip(void)
+{
+	int i, trips = get_nprocs() * 10 / 9 + 3, ret = KSFT_PASS, err;
+	pid_t pid;
+	FILE *f;
+
+	f = fopen("/proc/sys/kernel/sched_rt_runtime_us", "r");
+	if (f) {
+		long rt = 0;
+
+		if (fscanf(f, "%ld", &rt) == 1 && rt < 0) {
+			fclose(f);
+			ksft_print_msg("no admission control: a leak cannot be seen\n");
+			return KSFT_SKIP;
+		}
+		fclose(f);
+	}
+	memset(sh, 0, sizeof(*sh));
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LEAK_RUNTIME, LEAK_PERIOD))
+			_exit(KSFT_SKIP);
+		sleeper();
+	}
+	for (i = 0; i < trips; i++) {
+		uint64_t sleeps = __atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE);
+		uint64_t t = now_ns();
+
+		while (__atomic_load_n(&sh->sleeps, __ATOMIC_ACQUIRE) == sleeps) {
+			if (now_ns() - t > NSEC_PER_SEC) {
+				int status;
+
+				if (waitpid(pid, &status, WNOHANG) == pid)
+					return gone(status);
+				ksft_print_msg("the task did not run within 1s\n");
+				reap(pid);
+				return KSFT_FAIL;
+			}
+			sleep_ns(NSEC_PER_MSEC);
+		}
+		/* Asleep, well past its 0-lag time. */
+		sleep_ns(5 * NSEC_PER_MSEC);
+		err = set_other(pid);
+		if (!err)
+			err = set_deadline(pid, LEAK_RUNTIME, LEAK_PERIOD);
+		if (err) {
+			ksft_print_msg("switch back refused after %d of %d round trips: %s\n",
+				       i, trips, strerror(-err));
+			ret = KSFT_FAIL;
+			break;
+		}
+	}
+	reap(pid);
+	return ret;
+}
+
+/*
+ * sysrq-n takes every task out of the real-time classes, under
+ * tasklist_lock. A deadline task among them had __sched_setscheduler()
+ * take cpuset_mutex there, and never release it: no task could enter
+ * SCHED_DEADLINE again. The case renices every real-time task of the
+ * system: it runs with DEADLINE_TEST_SYSRQ set, and last.
+ */
+static int test_sysrq_normalize(void)
+{
+	int fd, status, ret = KSFT_PASS;
+	pid_t pid, prober, w;
+	uint64_t t;
+
+	if (!getenv("DEADLINE_TEST_SYSRQ")) {
+		ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+		return KSFT_SKIP;
+	}
+	fd = open("/proc/sysrq-trigger", O_WRONLY);
+	if (fd < 0) {
+		ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	pid = fork_child();
+	if (pid < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		close(fd);
+		return KSFT_FAIL;
+	}
+	if (!pid) {
+		if (set_deadline(0, LOST_RUNTIME, LOST_PERIOD))
+			_exit(KSFT_SKIP);
+		for (;;)
+			pause();
+	}
+	t = now_ns();
+	while (sched_getscheduler(pid) != SCHED_DEADLINE) {
+		if (waitpid(pid, &status, WNOHANG) == pid) {
+			close(fd);
+			return gone(status);
+		}
+		if (now_ns() - t > NSEC_PER_SEC) {
+			ksft_print_msg("the task did not enter SCHED_DEADLINE within 1s\n");
+			close(fd);
+			reap(pid);
+			return KSFT_FAIL;
+		}
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	if (write(fd, "n", 1) != 1) {
+		ksft_print_msg("sysrq-n: %s\n", strerror(errno));
+		close(fd);
+		reap(pid);
+		return KSFT_SKIP;
+	}
+	close(fd);
+	t = now_ns();
+	while (sched_getscheduler(pid) != SCHED_OTHER && now_ns() - t < NSEC_PER_SEC)
+		sleep_ns(NSEC_PER_MSEC);
+	if (sched_getscheduler(pid) != SCHED_OTHER) {
+		ksft_print_msg("still SCHED_DEADLINE after sysrq-n\n");
+		ret = KSFT_FAIL;
+	}
+	/* Admission still works: a task enters SCHED_DEADLINE within 5s. */
+	prober = fork_child();
+	if (prober < 0) {
+		ksft_print_msg("fork: %s\n", strerror(errno));
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (!prober)
+		_exit(set_deadline(0, LOST_RUNTIME, LOST_PERIOD) ? 1 : 0);
+	t = now_ns();
+	while ((w = waitpid(prober, &status, WNOHANG)) == 0) {
+		if (now_ns() - t > 5 * NSEC_PER_SEC) {
+			ksft_print_msg("SCHED_DEADLINE admission blocked after sysrq-n\n");
+			reap(pid);
+			return KSFT_FAIL;	/* the prober is stuck on a mutex: left behind */
+		}
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	if (w != prober) {
+		ksft_print_msg("waitpid %d: %s\n", prober, strerror(errno));
+		ret = KSFT_FAIL;
+	} else if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+		ksft_print_msg("SCHED_DEADLINE refused after sysrq-n\n");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	return ret;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	sh = mmap(NULL, sizeof(*sh), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (sh == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	ksft_print_header();
+	ksft_set_plan(3);
+	/*
+	 * In this order: the second fills the root domain where it leaks, and
+	 * the third leaves it without deadline admission where it locks.
+	 */
+	report("throttled_round_trip", test_throttled_round_trip());
+	report("asleep_round_trip", test_asleep_round_trip());
+	report("sysrq_normalize", test_sysrq_normalize());
+	ksft_finished();
+}
-- 
2.47.3

[PATCH v4 25/26] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild

[Mathieu Desnoyers]

Add deadline_rebuild_test, for "sched/deadline: Keep departing bandwidth
across a root domain rebuild", earlier in this series. In each case a
task of 400ms every second runs 380ms, then leaves SCHED_DEADLINE, its
0-lag time some 570ms ahead; past that time, the one-CPU partition root
P must admit one task of 600ms every second and refuse a second with
EBUSY:

 - dies_before_rebuild: the task, in P, exits, and a second partition
   root is made meanwhile, which rebuilds the accounting of every root
   domain;
 - switched_out_before_rebuild: the same, the task switched to
   SCHED_OTHER and asleep in P;
 - moved_after_switch_out: the task, outside P, switches to SCHED_OTHER,
   moves into P and dies there.

A kernel that removes the bandwidth from P's root domain when a rebuild
dropped it, or when P never had it, admits both. Skipped without root,
the cgroup v2 cpuset controller, deadline admission control (that is,
with sched_rt_runtime_us at -1) or 3 CPUs.

In 8 VMs of 8 vCPUs each, three runs in each, v7.4-rc1 fails the three
cases, and so does it with the patches of this series up to
"sched/deadline: Keep departing bandwidth across a root domain rebuild"
excluded; with it, the three pass; every time. v7.3 fails them as well
(2 VMs, three runs in each).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../selftests/sched/deadline/.gitignore       |   1 +
 .../testing/selftests/sched/deadline/Makefile |   2 +-
 tools/testing/selftests/sched/deadline/config |   2 +
 .../sched/deadline/deadline_rebuild_test.c    | 462 ++++++++++++++++++
 4 files changed, 466 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline/config
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c

diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index 9f203c455b91..bf7dd0da81a9 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1 +1,2 @@
 deadline_switch_test
+deadline_rebuild_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index 3f4dc1ec6119..f3a1d1746d22 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
 CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
 LDLIBS += -lpthread
 
-TEST_GEN_PROGS := deadline_switch_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
 
 include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
new file mode 100644
index 000000000000..d5bb793d742b
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/config
@@ -0,0 +1,2 @@
+CONFIG_CGROUPS=y
+CONFIG_CPUSETS=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
new file mode 100644
index 000000000000..ccf462f27dd0
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The bandwidth of a SCHED_DEADLINE task that leaves the policy, or dies,
+ * before its 0-lag time stays allocated in its root domain until the
+ * inactive timer removes it. It must be removed once, from the root
+ * domain that keeps it:
+ *
+ *  - dies: a task of a one-CPU partition root P dies, and a second
+ *    partition root Q is made meanwhile, which rebuilds the accounting
+ *    of every root domain;
+ *  - switched out: the same, the task switched to SCHED_OTHER;
+ *  - moved: a task outside P switched to SCHED_OTHER, then moved into P
+ *    before its 0-lag time.
+ *
+ * Then P, past the timer, must admit one task of 600ms every second and
+ * refuse a second one (EBUSY). A kernel that removes the bandwidth from
+ * P's root domain when the rebuild dropped it, or when P never held it,
+ * leaves it negative: it admits both.
+ *
+ * Needs root, the cgroup v2 cpuset controller, deadline admission
+ * control and 3 CPUs.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+static int set_deadline(pid_t pid, uint64_t runtime)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_runtime = runtime,
+		.sched_deadline = NSEC_PER_SEC,
+		.sched_period = NSEC_PER_SEC,
+	};
+
+	return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+	struct sched_param sp = { .sched_priority = 0 };
+
+	return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+/* These clocks cannot fail: both are always supported, the pointer is valid. */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	set_other(pid);			/* gone already: nothing to switch */
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* A partition root of the one CPU @cpu, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+	char path[PATH_MAX], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	if (write_file(cg, "cpuset.cpus", buf) || write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+	uint64_t end;
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	end = now_ns() + NSEC_PER_SEC;
+	while (rmdir(cg)) {
+		if (errno == ENOENT)
+			return;
+		if (errno != EBUSY || now_ns() > end) {
+			ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+			return;
+		}
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+}
+
+/* Both clocks are always supported and the pointer is valid: no error. */
+static uint64_t cpu_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+enum depart { DIES, SWITCHED_OUT };
+
+/*
+ * In @cg if given, made SCHED_DEADLINE at 400ms every second, the child
+ * runs 380ms of it, then dies or switches to SCHED_OTHER and sleeps:
+ * its 0-lag time is then some 570ms ahead. The pipe @fd gets a byte once
+ * it is out of the policy; for one that dies, its exit status tells. The
+ * caller keeps the read end alone.
+ */
+static pid_t departing(const char *cg, enum depart how, int fd[2])
+{
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		uint64_t t0;
+
+		close(fd[0]);
+		if (cg && enter(cg, getpid()))
+			_exit(2);
+		if (set_deadline(0, 400 * NSEC_PER_MSEC))
+			_exit(3);
+		t0 = cpu_time_ns();
+		while (cpu_time_ns() - t0 < 380 * NSEC_PER_MSEC)
+			;
+		if (how == DIES)
+			_exit(0);
+		if (set_other(0) || write(fd[1], "x", 1) != 1)
+			_exit(4);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	fd[1] = -1;
+	return pid;
+}
+
+/* Wait until the child is out of SCHED_DEADLINE, dead or switched out. */
+static int departed(pid_t *pid, enum depart how, int fd)
+{
+	int status;
+	char c;
+
+	if (*pid < 0)
+		return -1;
+	/* No byte: the child failed, and its exit status tells. */
+	if (how == SWITCHED_OUT) {
+		ssize_t n;
+
+		do {
+			n = read(fd, &c, 1);
+		} while (n < 0 && errno == EINTR);
+		if (n == 1)
+			return 0;
+	}
+	if (waitpid(*pid, &status, 0) != *pid)
+		return -1;
+	*pid = -1;
+	return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
+}
+
+/*
+ * A child that enters @cg and sleeps there, or -1. It moves itself, so
+ * that it last ran on a CPU of @cg, and says so once it is there.
+ */
+static pid_t sleeper(const char *cg)
+{
+	int fd[2];
+	pid_t pid;
+	char c;
+
+	if (pipe(fd))
+		return -1;
+	pid = fork();
+	if (pid < 0)
+		ksft_print_msg("fork: %s\n", strerror(errno));
+	if (!pid) {
+		close(fd[0]);
+		if (enter(cg, getpid()) || write(fd[1], "x", 1) != 1)
+			_exit(1);
+		for (;;)
+			pause();
+	}
+	close(fd[1]);
+	if (pid > 0 && read(fd[0], &c, 1) != 1) {
+		reap(pid);
+		pid = -1;
+	}
+	close(fd[0]);
+	return pid;
+}
+
+/* P, past every timer, admits one task of 600ms every second, not two. */
+static int check_p(void)
+{
+	pid_t a = sleeper(cg_p), b = sleeper(cg_p);
+	int ret = KSFT_FAIL, e1, e2;
+
+	if (a < 0 || b < 0) {
+		ksft_print_msg("no task could enter P\n");
+		goto out;
+	}
+	e1 = set_deadline(a, 600 * NSEC_PER_MSEC);
+	e2 = set_deadline(b, 600 * NSEC_PER_MSEC);
+	if (e1)
+		ksft_print_msg("a first task refused in P: %s\n", strerror(-e1));
+	else if (!e2)
+		ksft_print_msg("1.2 CPUs of deadline tasks admitted in a partition of one CPU\n");
+	else if (e2 != -EBUSY)
+		ksft_print_msg("the second task refused with %s, not EBUSY\n", strerror(-e2));
+	else
+		ret = KSFT_PASS;
+out:
+	reap(a);
+	reap(b);
+	return ret;
+}
+
+/* A task of P leaves, and Q is made before its 0-lag time. */
+static int test_rebuild(enum depart how)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(cg_p, how, fd);
+	err = departed(&pid, how, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (make_partition(cg_q, ncpus - 2)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 2);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	/* Past the 0-lag time; one switched out sleeps on in P. */
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_q);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+/* A task outside P switches out, then moves into P before its 0-lag time. */
+static int test_move(void)
+{
+	int ret = KSFT_FAIL, fd[2], err;
+	pid_t pid = -1;
+
+	if (pipe(fd))
+		return KSFT_FAIL;
+	if (make_partition(cg_p, ncpus - 1)) {
+		ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	pid = departing(NULL, SWITCHED_OUT, fd);
+	err = departed(&pid, SWITCHED_OUT, fd[0]);
+	if (err) {
+		ksft_print_msg("the deadline task failed (%d)\n", err);
+		goto out;
+	}
+	if (enter(cg_p, pid)) {
+		ksft_print_msg("the task cannot move into P\n");
+		goto out;
+	}
+	/* It wakes up, and dies, on P's CPU, before its 0-lag time. */
+	reap(pid);
+	pid = -1;
+	sleep_ns(1500 * NSEC_PER_MSEC);
+	ret = check_p();
+out:
+	reap(pid);
+	remove_partition(cg_p);
+	close(fd[0]);
+	if (fd[1] >= 0)
+		close(fd[1]);
+	return ret;
+}
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+	char buf[256], path[PATH_MAX + 32];
+	int added;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	ncpus = get_nprocs();
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs\n");
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	if (find_cgroup2())
+		ksft_exit_skip("cgroup v2 is not mounted\n");
+	/* The cpuset controller for the root's children while the test runs. */
+	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+	if (read_file(path, buf, sizeof(buf)))
+		ksft_exit_skip("no cgroup.subtree_control\n");
+	added = !strstr(buf, "cpuset");
+	if (added && write_file(root, "cgroup.subtree_control", "+cpuset"))
+		ksft_exit_skip("no cpuset controller\n");
+	if (snprintf(cg_p, sizeof(cg_p), "%s/deadline_rebuild_p", root) >= (int)sizeof(cg_p) ||
+	    snprintf(cg_q, sizeof(cg_q), "%s/deadline_rebuild_q", root) >= (int)sizeof(cg_q))
+		ksft_exit_skip("cgroup v2 mount path too long\n");
+
+	ksft_set_plan(3);
+	report("dies_before_rebuild", test_rebuild(DIES));
+	report("switched_out_before_rebuild", test_rebuild(SWITCHED_OUT));
+	report("moved_after_switch_out", test_move());
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+	ksft_finished();
+}
-- 
2.47.3

[PATCH v4 26/26] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

[Mathieu Desnoyers]

Add deadline_bw_test, 49 cases that take deadline tasks through priority
inheritance, parameter changes, cpuset moves, one of them racing a
rebuild of the sched domains, deaths, one of them leaving a zombie whose
policy is changed, sched_yield(), runtime overruns, bandwidth
reclaiming, changes to the fair servers, a randomized mix of all of
those, and CPU hotplug.

Each case measures the free bandwidth of a root domain before and after
it runs, as the number of tasks of 10ms every 100ms the root domain
admits, and wants the two equal once the case's tasks are gone and their
0-lag time is past: fewer is bandwidth leaked, more is bandwidth
released twice. A deadline task that a case keeps running must also make
progress. The reclaiming and server cases measure instead the share of a
CPU a task with SCHED_FLAG_RECLAIM gets, which follows the bandwidth its
CPU keeps reclaimable (extra_bw): on a CPU of its own, a task of 10ms
every 100ms got 66% of it, where one without the flag got 9%, and 49%
with a fair server of 10%, 24% with one of 30%, as GRUB has it.
params_waking, changing the parameters of a task whose wakeups move it
between two CPUs, and random_mix, six tasks taking three thousand random
transitions in three seconds with a fixed seed, check that this_bw of
each CPU, which "sched/debug: Show the deadline bandwidth of each
runqueue" prints, gets back where it started.

The cases in the partition root each get a new one, the accounting
rebuilt in between, so that a case that breaks it does not fail the next
ones, and the server cases put the fair servers back as they found them.
Skipped without root or deadline admission control, and the cases that
need them without a cpuset partition root, debugfs, CPU hotplug or
enough CPUs. Kprobes confirmed the states the cases mean to reach: the
inactive timer pending at a kill, the replenishment, the boost, and the
fair server stopped, deferred and serving at the debugfs writes.

In 8-CPU VMs, v7.4-rc1 fails thirteen cases, in 2 VMs of 2 (in probe
tasks of 10% admitted, before -> after):

 - pi_former_owner_asleep: the former owner's bandwidth leaks, 9 -> 6;
 - params_keep_class_change and params_keep_boosted: the request is
   accepted, not refused;
 - boosted_owner_policy: SCHED_OTHER on the owner a deadline waiter
   boosts fails with EBUSY;
 - deboost_resets_period: the deboosted owner runs 40ms of its first
   period as a task of 10ms, on the donor's runtime;
 - deboost_released_resets_period: the owner's former reservation is
   still held after it left the class asleep, and the waiter is refused
   with EBUSY;
 - move_asleep, move_non_contending and move_vs_rebuild: the bandwidth
   of a task moved out of the partition root asleep stays in it, 9 -> 6,
   and the root domain it returns to is short of it, 66 -> 69 (63 -> 66
   or 69 for move_vs_rebuild);
 - death_zombie_switched: SCHED_OTHER on a zombie is accepted, and the
   partition root admits 9 after a rebuild with a task of 30% in it,
   where 6 are due;
 - overrun_caught: 8 admitted where 9 are due;
 - random_mix: 9 -> 2;
 - hotplug_departed: 70 admitted where 66 are due.

params_waking fails on every VM of 8 vCPUs and of 4 without
"sched/deadline: Do not take a waking task's bandwidth off a runqueue it
left", and random_mix on 1 of 3, the kernel warning of the underflow.
With this series, all 49 cases pass every time, with PREEMPT_RT as well,
and with no warning. Built without cpusets they need "sched/deadline:
Rebuild the root domains' accounting from the task list", without which
hotplug_sleepers fails and the hotplug cases after it fail or skip for
want of room.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../selftests/sched/deadline/.gitignore       |    1 +
 .../testing/selftests/sched/deadline/Makefile |    2 +-
 tools/testing/selftests/sched/deadline/config |    2 +
 .../sched/deadline/deadline_bw_test.c         | 3902 +++++++++++++++++
 .../testing/selftests/sched/deadline/settings |    1 +
 5 files changed, 3907 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/sched/deadline/deadline_bw_test.c
 create mode 100644 tools/testing/selftests/sched/deadline/settings

diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index bf7dd0da81a9..42f6b2945e96 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1,2 +1,3 @@
 deadline_switch_test
 deadline_rebuild_test
+deadline_bw_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index f3a1d1746d22..15f1e96a21e6 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
 CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
 LDLIBS += -lpthread
 
-TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test deadline_bw_test
 
 include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/config b/tools/testing/selftests/sched/deadline/config
index d5bb793d742b..f457f79968c0 100644
--- a/tools/testing/selftests/sched/deadline/config
+++ b/tools/testing/selftests/sched/deadline/config
@@ -1,2 +1,4 @@
 CONFIG_CGROUPS=y
 CONFIG_CPUSETS=y
+CONFIG_DEBUG_FS=y
+CONFIG_HOTPLUG_CPU=y
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
new file mode 100644
index 000000000000..8ede5d5a350e
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -0,0 +1,3902 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * SCHED_DEADLINE bandwidth through priority inheritance, parameter changes,
+ * cpuset moves, deaths, yields, runtime overruns, bandwidth reclaiming,
+ * changes to the fair servers and CPU hotplug.
+ *
+ * Each case measures the free bandwidth of a root domain before and after
+ * it runs: the number of tasks of 10ms every 100ms that root domain admits
+ * (probe()). Once every task of the case is gone and its 0-lag time is
+ * past, the two must be equal: fewer is bandwidth leaked, more is
+ * bandwidth released twice, which lets the root domain admit deadline
+ * tasks beyond its capacity. A deadline task a case keeps running must
+ * also make progress.
+ *
+ * The reclaiming cases measure instead the share of a CPU a deadline task
+ * with SCHED_FLAG_RECLAIM gets, which tells of the free bandwidth its CPU
+ * keeps of the root domain (extra_bw), where probe() tells of the root
+ * domain's own (total_bw).
+ *
+ * Every case but the hotplug ones uses P, a partition root of one CPU,
+ * the last one, made anew for each case: the rebuild of the accounting
+ * that makes it clears what an earlier case left. The hotplug cases run in
+ * the root domain of all CPUs. The server cases change the fair server of a
+ * CPU through debugfs, and the test puts it back as it found it. Needs
+ * root, deadline admission control, and for the cases that use them, the
+ * cgroup v2 cpuset controller, debugfs and CPU hotplug.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+#include <linux/futex.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE	6
+#endif
+#ifndef SCHED_FLAG_RECLAIM
+#define SCHED_FLAG_RECLAIM	0x02
+#endif
+#ifndef SCHED_FLAG_DL_OVERRUN
+#define SCHED_FLAG_DL_OVERRUN	0x04
+#endif
+#ifndef SCHED_FLAG_KEEP_PARAMS
+#define SCHED_FLAG_KEEP_PARAMS	0x10
+#endif
+
+#define NSEC_PER_USEC	1000ULL
+#define NSEC_PER_MSEC	1000000ULL
+#define NSEC_PER_SEC	1000000000ULL
+
+/* The threads of probe() and hold() do nothing: a small stack will do. */
+#define THREAD_STACK	(64 * 1024 > PTHREAD_STACK_MIN ? 64 * 1024 : PTHREAD_STACK_MIN)
+
+#define PROBE_RUNTIME	(10 * NSEC_PER_MSEC)
+#define PROBE_PERIOD	(100 * NSEC_PER_MSEC)
+#define PERIOD		(100 * NSEC_PER_MSEC)
+
+struct dl_attr {
+	uint32_t size;
+	uint32_t sched_policy;
+	uint64_t sched_flags;
+	int32_t sched_nice;
+	uint32_t sched_priority;
+	uint64_t sched_runtime;
+	uint64_t sched_deadline;
+	uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus, *online;	/* the online CPUs at the start, ascending */
+
+/* The @n-th highest of the online CPUs, from 1. */
+static int top_cpu(int n)
+{
+	return online[ncpus - n];
+}
+
+/* Both clocks are always supported and the pointer is valid: no error. */
+static uint64_t now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static uint64_t thread_time_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+	struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+	while (nanosleep(&ts, &ts) && errno == EINTR)
+		;
+}
+
+static void spin_ns(uint64_t ns)
+{
+	uint64_t end = thread_time_ns() + ns;
+
+	while (thread_time_ns() < end)
+		;
+}
+
+/*
+ * Polls @cond every @step, at most @ms: 0 once it holds, -1 on a timeout.
+ * Every wait here is bounded, so that a child that never started, or was
+ * refused SCHED_DEADLINE, fails its case instead of hanging it.
+ */
+#define wait_for_step(ms, step, cond)					\
+({									\
+	uint64_t wait_end = now_ns() + (uint64_t)(ms) * NSEC_PER_MSEC;	\
+	int wait_ret = -1;						\
+									\
+	while (1) {							\
+		if (cond) {						\
+			wait_ret = 0;					\
+			break;						\
+		}							\
+		if (now_ns() >= wait_end)				\
+			break;						\
+		sleep_ns(step);						\
+	}								\
+	wait_ret;							\
+})
+
+#define wait_for(ms, cond)	wait_for_step(ms, NSEC_PER_MSEC, cond)
+
+/* sem_wait(), retrying the signals that interrupt it. */
+static void sem_wait_sig(sem_t *s)
+{
+	while (sem_wait(s)) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("sem_wait: %s\n", strerror(errno));
+		return;
+	}
+}
+
+/* The same, at most @ms: 0, or -1. For what a thread or a child must post. */
+static int sem_wait_for(sem_t *s, int ms)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)(ms % 1000) * 1000000L;
+	ts.tv_sec += ms / 1000 + ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	while (sem_timedwait(s, &ts)) {
+		if (errno == EINTR)
+			continue;
+		if (errno != ETIMEDOUT)
+			ksft_print_msg("sem_timedwait: %s\n", strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+/* fork(), which tells of its own failure: the caller handles -1. */
+static pid_t fork_child(const char *what)
+{
+	pid_t pid = fork();
+
+	if (pid < 0)
+		ksft_print_msg("fork for %s: %s\n", what, strerror(errno));
+	return pid;
+}
+
+static void reap(pid_t pid)
+{
+	int status;
+
+	if (pid <= 0)
+		return;
+	if (kill(pid, SIGKILL) && errno != ESRCH)
+		ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		break;
+	}
+}
+
+static void join(pthread_t t)
+{
+	int err = pthread_join(t, NULL);
+
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+}
+
+static pid_t gettid_(void)
+{
+	return syscall(SYS_gettid);
+}
+
+static int set_deadline_flags(pid_t tid, uint64_t runtime, uint64_t period,
+			      uint64_t flags)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = flags,
+		.sched_runtime = runtime,
+		.sched_deadline = period,
+		.sched_period = period,
+	};
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int set_deadline(pid_t tid, uint64_t runtime, uint64_t period)
+{
+	return set_deadline_flags(tid, runtime, period, 0);
+}
+
+static int set_other(pid_t tid)
+{
+	struct dl_attr attr = { .size = sizeof(attr), .sched_policy = SCHED_OTHER };
+
+	return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int fd, err = 0;
+	ssize_t n;
+
+	if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+		return -ENAMETOOLONG;
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	n = write(fd, val, strlen(val));
+	if (n < 0)
+		err = -errno;
+	else if (n != (ssize_t)strlen(val))
+		err = -EIO;
+	if (close(fd) && !err)
+		err = -errno;
+	return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+	int fd = open(path, O_RDONLY);
+	ssize_t n;
+
+	if (fd < 0)
+		return -1;
+	do {
+		n = read(fd, buf, len - 1);
+	} while (n < 0 && errno == EINTR);
+	close(fd);
+	if (n < 0)
+		return -1;
+	buf[n] = 0;
+	return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", pid);
+	return write_file(cg, "cgroup.procs", buf);
+}
+
+/* @cg, a partition root of the CPUs @cpus lists: 0, or -1. */
+static int make_partition_cpus(const char *cg, const char *cpus)
+{
+	char path[PATH_MAX + 32], buf[32];
+
+	if (mkdir(cg, 0755) && errno != EEXIST)
+		return -1;
+	if (write_file(cg, "cpuset.cpus", cpus) ||
+	    write_file(cg, "cpuset.cpus.partition", "root"))
+		return -1;
+	snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+	if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+		return -1;
+	return 0;
+}
+
+/* @cg, a partition root of CPU @cpu: 0, or -1. */
+static int make_partition(const char *cg, int cpu)
+{
+	char buf[32];
+
+	snprintf(buf, sizeof(buf), "%d", cpu);
+	return make_partition_cpus(cg, buf);
+}
+
+static void remove_partition(const char *cg)
+{
+	int err = write_file(cg, "cpuset.cpus.partition", "member");
+
+	if (err && err != -ENOENT)
+		ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+	/* A cgroup whose last task just died takes a moment to empty. */
+	if (wait_for(1000, !rmdir(cg) || errno == ENOENT))
+		ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+}
+
+/* P, a partition root of the last online CPU: 0, or -1. */
+static int make_p(void)
+{
+	return make_partition(cg_p, top_cpu(1));
+}
+
+static void remove_p(void)
+{
+	remove_partition(cg_p);
+}
+
+/* Field @n, from 3 on, of /proc/@pid/stat, or @fail. */
+static int stat_field(pid_t pid, int n, int fail)
+{
+	char path[64], buf[1024], *p;
+	int i;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+	if (read_file(path, buf, sizeof(buf)))
+		return fail;
+	p = strrchr(buf, ')');
+	for (i = 0; p && i < n - 2; i++)
+		p = strchr(p + 1, ' ');
+	return p ? atoi(p + 1) : fail;
+}
+
+/* The CPU @pid last ran on, from /proc. */
+static int task_cpu_of(pid_t pid)
+{
+	return stat_field(pid, 39, -1);
+}
+
+/* The priority /proc shows of a deadline task, or of one boosted by one. */
+#define DL_TASK_PRIO	(-101)
+
+/* The priority @pid runs at, from /proc, or 0. */
+static int prio_of(pid_t pid)
+{
+	return stat_field(pid, 18, 0);
+}
+
+/* The CPU time @t has used, or 0 if it cannot be read. */
+static uint64_t cpu_time_of(pthread_t t)
+{
+	static int told;
+	struct timespec ts;
+	clockid_t c;
+	int err;
+
+	err = pthread_getcpuclockid(t, &c);
+	if (!err && clock_gettime(c, &ts))
+		err = errno;
+	if (err) {
+		if (!told++)
+			ksft_print_msg("the CPU time of a thread cannot be read: %s\n",
+				       strerror(err));
+		return 0;
+	}
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+/* The state letter of @pid in /proc, or 0. */
+static char proc_state_of(pid_t pid)
+{
+	char path[64], buf[1024], *p;
+
+	snprintf(path, sizeof(path), "/proc/%d/stat", pid);
+	if (read_file(path, buf, sizeof(buf)))
+		return 0;
+	p = strrchr(buf, ')');
+	if (!p || p[1] != ' ')
+		return 0;
+	return p[2];
+}
+
+/* The share of a CPU, in percent, that @t uses over @ns, a period later. */
+static int share_of(pthread_t t, uint64_t ns)
+{
+	uint64_t cpu, wall;
+
+	sleep_ns(PERIOD);		/* a period of its own to settle in */
+	cpu = cpu_time_of(t);
+	wall = now_ns();
+	sleep_ns(ns);
+	cpu = cpu_time_of(t) - cpu;
+	wall = now_ns() - wall;
+	return wall ? (int)(100 * cpu / wall) : -1;
+}
+
+/*
+ * The probe: from a process that joins @cg (the root domain of its CPUs),
+ * threads each make themselves a deadline task of PROBE_RUNTIME every
+ * PROBE_PERIOD, one at a time, until one is refused; then they all exit.
+ * Returns how many were admitted, or -1. The threads never run past their
+ * first instants, so their bandwidth goes at their death; probe() waits for
+ * their 0-lag times all the same.
+ */
+struct probe_slot {
+	pthread_t t;
+	int err;
+};
+
+static sem_t probe_done;
+static int probe_hold[2];
+
+static void *probe_thread(void *arg)
+{
+	struct probe_slot *s = arg;
+	char c;
+
+	s->err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+	if (sem_post(&probe_done))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	/* Asleep until the probe is over: the write end closes. */
+	while (!s->err) {
+		ssize_t r = read(probe_hold[0], &c, 1);
+
+		if (r >= 0)
+			break;
+		if (errno != EINTR) {
+			s->err = -errno;
+			break;
+		}
+	}
+	return NULL;
+}
+
+static int probe(const char *cg)
+{
+	int fd[2], n = -1, max = 16 * ncpus + 16;
+	pid_t pid;
+
+	if (pipe(fd)) {
+		ksft_print_msg("pipe: %s\n", strerror(errno));
+		return -1;
+	}
+	pid = fork_child("the admission check");
+	if (!pid) {
+		struct probe_slot *s = calloc(max, sizeof(*s));
+		pthread_attr_t a;
+		int i, k = 0;
+
+		close(fd[0]);
+		if (!s || sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+		    (cg && enter(cg, getpid())))
+			_exit(1);
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			_exit(1);
+		for (i = 0; i < max; i++) {
+			if (pthread_create(&s[i].t, &a, probe_thread, &s[i])) {
+				k = -1;	/* not an admission refused */
+				break;
+			}
+			if (sem_wait_for(&probe_done, 10000)) {
+				k = -1;
+				i++;	/* that thread is to be joined as well */
+				break;
+			}
+			if (s[i].err) {
+				if (s[i].err != -EBUSY)
+					k = -1;
+				join(s[i].t);
+				break;
+			}
+			if (k >= 0)
+				k++;
+		}
+		if (i == max)
+			k = -1;
+		close(probe_hold[1]);
+		while (--i >= 0)
+			join(s[i].t);
+		if (write(fd[1], &k, sizeof(k)) != sizeof(k))
+			_exit(1);
+		_exit(0);
+	}
+	close(fd[1]);
+	if (pid < 0 || read(fd[0], &n, sizeof(n)) != sizeof(n))
+		n = -1;
+	close(fd[0]);
+	reap(pid);
+	sleep_ns(3 * PROBE_PERIOD);
+	return n;
+}
+
+/*
+ * A case runs in a child process, in @cg, so that all its threads go with
+ * it: @fn returns KSFT_PASS or KSFT_FAIL, or KSFT_SKIP.
+ */
+static int run_in(const char *cg, int (*fn)(void))
+{
+	pid_t pid = fork_child("a case");
+	int status;
+
+	if (!pid) {
+		int err = cg ? enter(cg, getpid()) : 0;
+
+		if (err) {
+			ksft_print_msg("the case cannot enter %s: %s\n", cg, strerror(-err));
+			_exit(KSFT_SKIP);
+		}
+		_exit(fn());
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	while (waitpid(pid, &status, 0) != pid) {
+		if (errno == EINTR)
+			continue;
+		ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+		return KSFT_FAIL;
+	}
+	if (!WIFEXITED(status)) {
+		ksft_print_msg("the case died of signal %d\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return WEXITSTATUS(status);
+}
+
+/* @fn in @cg, with the free bandwidth of @cg's root domain the same after. */
+static int measured(const char *name, const char *cg, int (*fn)(void))
+{
+	int before, after, ret;
+
+	before = probe(cg);
+	if (before <= 0) {
+		ksft_print_msg("%s: %s\n", name, before ? "the probe failed" : "no room to probe");
+		return before ? KSFT_FAIL : KSFT_SKIP;
+	}
+	ret = run_in(cg, fn);
+	sleep_ns(3 * PERIOD);
+	after = probe(cg);
+	if (after < 0) {
+		ksft_print_msg("%s: the probe failed after the case\n", name);
+		ret = KSFT_FAIL;
+	} else if (ret == KSFT_PASS && after != before) {
+		ksft_print_msg("%s: %d probe tasks admitted before, %d after: %s\n",
+			       name, before, after,
+			       after < before ? "bandwidth leaked" : "bandwidth released twice");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Priority inheritance */
+
+static pthread_mutex_t pi_lock;
+static sem_t owner_ready, owner_go;
+static atomic_int owner_holds, owner_tid;
+
+/* 0, or -1 where a priority inheritance mutex is not to be had. */
+static int pi_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err) {
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+		if (!err)
+			err = pthread_mutex_init(&pi_lock, &a);
+		pthread_mutexattr_destroy(&a);
+	}
+	if (err) {
+		ksft_print_msg("no priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	if (sem_init(&owner_ready, 0, 0) || sem_init(&owner_go, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	owner_holds = 0;
+	return 0;
+}
+
+/* Undoes pi_init(), its owner and waiters gone. */
+static void pi_fini(void)
+{
+	pthread_mutex_destroy(&pi_lock);
+	sem_destroy(&owner_ready);
+	sem_destroy(&owner_go);
+}
+
+/*
+ * The calling thread, a deadline task meanwhile, waits for the lock up to
+ * @ms: 0 when it times out, or gets the lock, as @timeout expects.
+ */
+static int wait_lock_for(int ms, int timeout)
+{
+	struct timespec ts;
+	int err;
+
+	err = set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return -1;
+	}
+	clock_gettime(CLOCK_REALTIME, &ts);
+	ts.tv_nsec += (long)ms * 1000000L;
+	ts.tv_sec += ts.tv_nsec / 1000000000L;
+	ts.tv_nsec %= 1000000000L;
+	err = pthread_mutex_timedlock(&pi_lock, &ts);
+	if (!err) {
+		int e = pthread_mutex_unlock(&pi_lock);
+
+		if (e)
+			ksft_print_msg("a waiter cannot unlock: %s\n", strerror(e));
+	}
+	if (set_other(0)) {
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+		return -1;
+	}
+	if (err != (timeout ? ETIMEDOUT : 0)) {
+		ksft_print_msg("a waiter of the lock: %s, not %s\n", err ? strerror(err) : "got it",
+			       timeout ? "a timeout" : "the lock");
+		return -1;
+	}
+	return 0;
+}
+
+/* The lock of an owner: -1 told of, and owner_holds says so. */
+static int take_pi_lock(void)
+{
+	int err = pthread_mutex_lock(&pi_lock);
+
+	if (err) {
+		ksft_print_msg("an owner cannot take the lock: %s\n", strerror(err));
+		owner_holds = -1;
+	}
+	return err ? -1 : 0;
+}
+
+static void drop_pi_lock(void)
+{
+	int err = pthread_mutex_unlock(&pi_lock);
+
+	if (err)
+		ksft_print_msg("an owner cannot unlock: %s\n", strerror(err));
+}
+
+/* Owner: was a deadline task (@was_dl), takes the lock and sleeps with it. */
+static void *sleeping_owner(void *arg)
+{
+	int was_dl = (long)arg;
+
+	owner_tid = gettid_();
+	if (was_dl) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			owner_holds = -1;
+			sem_post(&owner_ready);
+			return (void *)1;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		/* Asleep past its 0-lag time, switched out by the main thread. */
+		sem_post(&owner_ready);
+		sem_wait_sig(&owner_go);
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	sleep_ns(400 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_asleep(int was_dl, int rounds)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)(long)was_dl)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (was_dl) {
+		if (sem_wait_for(&owner_ready, 5000)) {
+			ksft_print_msg("the owner did not become a deadline task\n");
+			ret = KSFT_FAIL;
+		} else {
+			sleep_ns(200 * NSEC_PER_MSEC);
+			if (set_other(owner_tid)) {
+				ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+				ret = KSFT_FAIL;
+			}
+		}
+		sem_post(&owner_go);
+	}
+	if (ret == KSFT_PASS && (wait_for(5000, owner_holds) || owner_holds != 2)) {
+		ksft_print_msg("the owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS)
+		sleep_ns(20 * NSEC_PER_MSEC);
+	/* Each attempt boosts the sleeping owner, its timeout deboosts it asleep. */
+	for (r = 0; r < rounds && ret == KSFT_PASS; r++) {
+		if (wait_lock_for(20, 1)) {
+			ksft_print_msg("round %d of %d failed\n", r + 1, rounds);
+			ret = KSFT_FAIL;
+		}
+	}
+	join(o);
+	return ret;
+}
+
+static int pi_other_owner_asleep(void)
+{
+	return pi_asleep(0, 5);
+}
+
+static int pi_former_owner_asleep(void)
+{
+	return pi_asleep(1, 5);
+}
+
+/* A deadline owner boosted by a deadline waiter of an earlier deadline. */
+static void *dl_owner(void *arg)
+{
+	uint64_t end;
+
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		owner_holds = -1;
+		return (void *)1;
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	end = now_ns() + 300 * NSEC_PER_MSEC;
+	while (now_ns() < end) {
+		spin_ns(2 * NSEC_PER_MSEC);
+		sleep_ns(5 * NSEC_PER_MSEC);
+	}
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_deadline_owner(void)
+{
+	int r, ret = KSFT_PASS, err;
+	pthread_t o;
+	void *res;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, dl_owner, NULL)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the deadline owner never took the lock\n");
+		ret = KSFT_FAIL;
+	}
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++)
+		if (wait_lock_for(20, 1))
+			ret = KSFT_FAIL;
+	err = pthread_join(o, &res);
+	if (err)
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+	return res || err ? KSFT_FAIL : ret;
+}
+
+/* The owner runs when boosted, and its unlock deboosts it. */
+static void *running_owner(void *arg)
+{
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	spin_ns(50 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	return NULL;
+}
+
+static int pi_unlock_running(void)
+{
+	pthread_t o;
+	int r, ret = KSFT_PASS;
+
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (pi_init())
+			return KSFT_SKIP;
+		if (pthread_create(&o, NULL, running_owner, NULL)) {
+			ksft_print_msg("the owner thread could not start\n");
+			return KSFT_FAIL;
+		}
+		if (wait_for_step(5000, 100 * NSEC_PER_USEC, owner_holds) ||
+		    owner_holds != 2) {
+			ksft_print_msg("the owner never took the lock\n");
+			ret = KSFT_FAIL;
+		} else if (wait_lock_for(1000, 0)) {
+			ret = KSFT_FAIL;
+		}
+		join(o);
+		pi_fini();
+	}
+	return ret;
+}
+
+/* Parameter changes */
+
+static atomic_int t_state;	/* 1: about to sleep, or spinning */
+static _Atomic pid_t t_tid;
+static sem_t t_exit;
+
+/* A deadline task that runs @run_ns of its runtime, then sleeps. */
+static void *runs_then_sleeps(void *arg)
+{
+	uint64_t run = (uint64_t)(long)arg;
+
+	t_tid = gettid_();
+	if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	sleep_ns(PERIOD);	/* a fresh period */
+	spin_ns(run);
+	t_state = 1;
+	sem_wait_sig(&t_exit);
+	return NULL;
+}
+
+/* Starts runs_then_sleeps(): 0 once it is about to sleep, or -1. */
+static int start_sleeper(pthread_t *t, uint64_t run)
+{
+	t_state = 0;
+	if (sem_init(&t_exit, 0, 0)) {
+		ksft_print_msg("sem_init: %s\n", strerror(errno));
+		return -1;
+	}
+	if (pthread_create(t, NULL, runs_then_sleeps, (void *)(long)run)) {
+		ksft_print_msg("the sleeper thread could not start\n");
+		return -1;
+	}
+	if (wait_for_step(5000, 100 * NSEC_PER_USEC, t_state) || t_state != 1) {
+		ksft_print_msg("the sleeper was refused SCHED_DEADLINE\n");
+		sem_post(&t_exit);	/* in case it gets there late */
+		join(*t);
+		return -1;
+	}
+	return 0;
+}
+
+static void stop_sleeper(pthread_t t)
+{
+	if (sem_post(&t_exit))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	join(t);
+}
+
+/* The root domain of @cg admits @want probe tasks now. */
+static int admits_in(const char *cg, int want, const char *what)
+{
+	int n = probe(cg);
+
+	if (n != want) {
+		ksft_print_msg("%s: %d probe tasks admitted, %d expected\n", what, n, want);
+		return -1;
+	}
+	return 0;
+}
+
+/* The root domain of the caller's cgroup admits @want probe tasks now. */
+static int admits(int want, const char *what)
+{
+	return admits_in(NULL, want, what);
+}
+
+static int params_asleep(void)
+{
+	static const uint64_t rt[] = { 40, 10, 30, 20 };
+	int base, i, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	/* Probed from here, in P: this process is P's. */
+	base = probe(NULL) + 2;		/* the task's 20% back */
+	for (i = 0; i < 4 && ret == KSFT_PASS; i++) {
+		char what[64];
+
+		if (set_deadline(t_tid, rt[i] * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		snprintf(what, sizeof(what), "asleep, changed to %llums every 100ms",
+			 (unsigned long long)rt[i]);
+		if (admits(base - (int)(rt[i] / 10), what))
+			ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+static int params_non_contending(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS;
+
+	/* 5ms of 20ms run: its 0-lag time some 20ms after it sleeps. */
+	if (start_sleeper(&t, 5 * NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(NSEC_PER_MSEC);
+	if (set_deadline(t_tid, 40 * NSEC_PER_MSEC, PERIOD) ||
+	    set_deadline(t_tid, 10 * NSEC_PER_MSEC, PERIOD))
+		ret = KSFT_FAIL;
+	stop_sleeper(t);
+	return ret;
+}
+
+static atomic_int spin_stop;
+static uint64_t spin_runtime, spin_flags;	/* set before the thread starts */
+
+static void *dl_spinner(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, spin_runtime, PERIOD, spin_flags)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop)
+		;
+	return NULL;
+}
+
+/* Starts dl_spinner() of @runtime every 100ms with @flags: 0, or -1. */
+static int start_spinner(pthread_t *t, uint64_t runtime, uint64_t flags)
+{
+	t_state = 0;
+	spin_stop = 0;
+	spin_runtime = runtime;
+	spin_flags = flags;
+	if (pthread_create(t, NULL, dl_spinner, NULL)) {
+		ksft_print_msg("the spinner thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state == 1)
+		return 0;
+	spin_stop = 1;		/* in case it gets there late */
+	join(*t);
+	return -1;
+}
+
+static void stop_spinner(pthread_t t)
+{
+	spin_stop = 1;
+	set_other(t_tid);	/* out of the class already: no matter */
+	join(t);
+}
+
+/* Waits until @t stops using CPU time: throttled. 0, or -1 after 1s. */
+static int wait_throttled(pthread_t t)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC, last = cpu_time_of(t);
+
+	while (last && now_ns() < end) {
+		uint64_t cur;
+
+		sleep_ns(3 * NSEC_PER_MSEC);
+		cur = cpu_time_of(t);
+		if (cur == last)
+			return 0;
+		last = cur;
+	}
+	return -1;
+}
+
+/* @t runs again within @ns. */
+static int progresses(pthread_t t, uint64_t ns)
+{
+	uint64_t start = cpu_time_of(t), end = now_ns() + ns;
+
+	while (now_ns() < end) {
+		if (cpu_time_of(t) > start + NSEC_PER_MSEC)
+			return 0;
+		sleep_ns(2 * NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+static int params_throttled(void)
+{
+	int r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0))
+		return KSFT_FAIL;
+	for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+		if (wait_throttled(t) ||
+		    set_deadline(t_tid, (r & 1 ? 10 : 20) * NSEC_PER_MSEC, PERIOD)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (progresses(t, 3 * PERIOD)) {
+			ksft_print_msg("throttled, parameters changed: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	return ret;
+}
+
+static int params_refused(void)
+{
+	pthread_t t;
+	int ret = KSFT_PASS, base, err;
+
+	if (start_sleeper(&t, NSEC_PER_MSEC))
+		return KSFT_FAIL;
+	sleep_ns(2 * PERIOD);
+	base = probe(NULL);
+	/*
+	 * P's free bandwidth is at least @base probe tasks of 10%, and less
+	 * than one more: growing the task by one more than that is refused,
+	 * whatever the admission limit.
+	 */
+	if (20 + 10 * (base + 1) > 100) {
+		stop_sleeper(t);
+		ksft_print_msg("P has more room than one of its tasks can take\n");
+		return KSFT_SKIP;
+	}
+	err = set_deadline(t_tid, (20 + 10 * (uint64_t)(base + 1)) * NSEC_PER_MSEC, PERIOD);
+	if (err != -EBUSY) {
+		ksft_print_msg("an increase beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	} else if (admits(base, "after a refused increase")) {
+		ret = KSFT_FAIL;
+	}
+	stop_sleeper(t);
+	return ret;
+}
+
+/*
+ * SCHED_FLAG_KEEP_PARAMS with a change of class is refused, with nothing
+ * reserved for the deadline parameters it would keep: the task's own,
+ * its slice the runtime.
+ */
+static int params_keep_class_change(void)
+{
+	struct dl_attr attr = {
+		.size = sizeof(attr),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,
+	};
+	int err;
+
+	if (syscall(SYS_sched_setattr, 0, &attr, 0)) {
+		ksft_print_msg("a slice of its own is refused: %s\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+	attr.sched_policy = SCHED_DEADLINE;
+	attr.sched_flags = SCHED_FLAG_KEEP_PARAMS;
+	attr.sched_deadline = PERIOD;
+	attr.sched_period = PERIOD;
+	err = syscall(SYS_sched_setattr, 0, &attr, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("to SCHED_DEADLINE with SCHED_FLAG_KEEP_PARAMS: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* A deadline task that blocks on the lock: it boosts its owner. */
+static void *boosting_waiter(void *arg)
+{
+	return (void *)(long)wait_lock_for(300, 1);
+}
+
+/*
+ * SCHED_FLAG_KEEP_PARAMS with SCHED_DEADLINE on an owner that priority
+ * inheritance has boosted into the class: the class of the request is the
+ * one the owner already has, so a class change is not what it asks for,
+ * and the policy it asks for is one the kept parameters would not give
+ * it. It is refused with EINVAL; admitted, it reserves the bandwidth of
+ * the owner's slice and nothing ever releases it.
+ */
+static int params_keep_boosted(void)
+{
+	struct dl_attr slice = {
+		.size = sizeof(slice),
+		.sched_policy = SCHED_OTHER,
+		.sched_runtime = 40 * NSEC_PER_MSEC,	/* what the request would keep */
+	};
+	struct dl_attr keep = {
+		.size = sizeof(keep),
+		.sched_policy = SCHED_DEADLINE,
+		.sched_flags = SCHED_FLAG_KEEP_PARAMS,
+		.sched_deadline = PERIOD,
+		.sched_period = PERIOD,
+	};
+	int ret = KSFT_PASS, err;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (syscall(SYS_sched_setattr, owner_tid, &slice, 0)) {
+		ksft_print_msg("the owner cannot take a slice of its own: %s\n",
+			       strerror(errno));
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);		/* it blocks, the owner is boosted */
+	err = syscall(SYS_sched_setattr, owner_tid, &keep, 0) ? -errno : 0;
+	if (err != -EINVAL) {
+		ksft_print_msg("KEEP_PARAMS to SCHED_DEADLINE on a boosted owner: %s, not EINVAL\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	join(w);
+out:
+	join(o);
+	return ret;
+}
+
+/* Boosted by a deadline waiter, the owner may still change its policy. */
+static int boosted_owner_policy(void)
+{
+	int ret = KSFT_PASS, err;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);		/* it blocks, the owner is boosted */
+	err = set_other(owner_tid);
+	if (err) {
+		ksft_print_msg("SCHED_OTHER on a boosted owner: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	}
+	join(w);
+	join(o);
+	return ret;
+}
+
+static atomic_int owner_first_period_ms;
+
+/*
+ * Holds the lock spinning until the waiter boosts it, and 5ms more,
+ * deboosted as it unlocks, then enters the class on 10ms of 100ms: its
+ * first period is on its own parameters, not on what the donor left.
+ * With @arg, a deadline task of 60ms of 100ms first, which runs 1ms and
+ * sleeps until it is switched out of the class, its 0-lag time past: kept,
+ * parameters larger than what the donor leaves it would have the deboost
+ * put its 0-lag time ahead, not clear them.
+ */
+static void *deboosted_owner(void *arg)
+{
+	uint64_t t0, cpu0;
+	int err;
+
+	owner_tid = gettid_();
+	if (arg) {
+		err = set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD);
+		if (err) {
+			ksft_print_msg("the owner refused as a deadline task: %s\n",
+				       strerror(-err));
+			return (void *)1;
+		}
+		spin_ns(NSEC_PER_MSEC);
+		sem_post(&owner_ready);
+		sem_wait_sig(&owner_go);
+	}
+	if (take_pi_lock())
+		return (void *)1;
+	owner_holds = 2;
+	t0 = now_ns();
+	while (prio_of(owner_tid) != DL_TASK_PRIO) {
+		if (now_ns() - t0 > NSEC_PER_SEC) {
+			drop_pi_lock();
+			return (void *)2;	/* never boosted */
+		}
+	}
+	spin_ns(5 * NSEC_PER_MSEC);
+	drop_pi_lock();
+	err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	t0 = now_ns();
+	cpu0 = thread_time_ns();
+	while (now_ns() - t0 < PERIOD)
+		;
+	owner_first_period_ms = (thread_time_ns() - cpu0) / NSEC_PER_MSEC;
+	return NULL;
+}
+
+/* A deadline waiter of 40ms of 100ms: the owner spends 5 of them boosted. */
+static void *donor_waiter(void *arg)
+{
+	int err;
+
+	err = set_deadline(0, 40 * NSEC_PER_MSEC, PERIOD);
+	if (err) {
+		ksft_print_msg("a waiter refused as a deadline task: %s\n", strerror(-err));
+		return (void *)1;
+	}
+	err = take_pi_lock();
+	if (!err)
+		drop_pi_lock();
+	if (set_other(0))
+		ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+	return (void *)(long)err;
+}
+
+/* @released: the owner left the class asleep past its 0-lag time before. */
+static int deboost_case(int released)
+{
+	int err, ret = KSFT_PASS;
+	void *o_ret, *w_ret;
+	pthread_t o, w;
+
+	if (pi_init())
+		return KSFT_SKIP;
+	owner_first_period_ms = -1;
+	if (pthread_create(&o, NULL, deboosted_owner, (void *)(long)released)) {
+		ksft_print_msg("the owner thread could not start\n");
+		return KSFT_FAIL;
+	}
+	if (released) {
+		if (sem_wait_for(&owner_ready, 5000)) {
+			sem_post(&owner_go);	/* only late: let it end */
+			join(o);
+			return KSFT_FAIL;	/* the owner said why */
+		}
+		sleep_ns(3 * PERIOD);
+		err = proc_state_of(owner_tid) == 'S' ? set_other(owner_tid) : -EAGAIN;
+		sem_post(&owner_go);
+		if (err) {
+			ksft_print_msg("the owner could not leave the class asleep: %s\n",
+				       strerror(-err));
+			join(o);
+			return KSFT_FAIL;
+		}
+	}
+	if (wait_for(5000, owner_holds) || owner_holds != 2) {
+		ksft_print_msg("the owner never took the lock\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	if (pthread_create(&w, NULL, donor_waiter, NULL)) {
+		ksft_print_msg("the waiter thread could not start\n");
+		join(o);
+		return KSFT_FAIL;
+	}
+	err = pthread_join(w, &w_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	} else if (w_ret) {
+		ret = KSFT_FAIL;	/* the waiter said why */
+	}
+	err = pthread_join(o, &o_ret);
+	if (err) {
+		ksft_print_msg("pthread_join: %s\n", strerror(err));
+		ret = KSFT_FAIL;
+	} else if (o_ret == (void *)2) {
+		ksft_print_msg("the waiter never boosted the owner\n");
+		ret = KSFT_FAIL;
+	} else if (owner_first_period_ms < 0) {
+		ret = KSFT_FAIL;
+	} else if (owner_first_period_ms > 20) {
+		ksft_print_msg("the owner ran %d ms of its first 100: the donor's runtime\n",
+			       (int)owner_first_period_ms);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int deboost_resets_period(void)
+{
+	return deboost_case(0);
+}
+
+/*
+ * The same for an owner released from the class with its bandwidth: its
+ * parameters go too, and the deboost resets what the donor left.
+ */
+static int deboost_released_resets_period(void)
+{
+	return deboost_case(1);
+}
+
+/* cpuset moves, from the main process, outside P */
+
+/* Shared with the processes of a case: atomic in shared memory. */
+static atomic_int *mv_state;
+
+/*
+ * mv_state[0] says a child is up (1), was refused SCHED_DEADLINE (-1), or
+ * could not start its threads (-2).
+ */
+static int child_up(const char *what, int fast)
+{
+	if (wait_for_step(5000, fast ? 100 * NSEC_PER_USEC : NSEC_PER_MSEC, mv_state[0])) {
+		ksft_print_msg("%s: the task never started\n", what);
+		return -1;
+	}
+	if (mv_state[0] == -2) {
+		ksft_print_msg("%s: the process cannot start its threads\n", what);
+		return -1;
+	}
+	if (mv_state[0] != 1) {
+		ksft_print_msg("%s: the task was refused SCHED_DEADLINE\n", what);
+		return -1;
+	}
+	return 0;
+}
+
+/* A process whose one deadline thread does what @mode says, in the root cgroup. */
+enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_YIELDED, MV_NON_CONTENDING };
+
+static pid_t mover(enum mv_mode mode, uint64_t runtime)
+{
+	pid_t pid = fork_child("a deadline task");
+
+	if (!pid) {
+		if (set_deadline(0, runtime, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		if (mode == MV_ASLEEP) {
+			spin_ns(NSEC_PER_MSEC);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_NON_CONTENDING) {
+			sleep_ns(PERIOD);
+			spin_ns(runtime / 4);
+			*mv_state = 1;
+			for (;;)
+				pause();
+		}
+		if (mode == MV_YIELDED) {
+			*mv_state = 1;
+			for (;;) {
+				spin_ns(NSEC_PER_MSEC / 2);
+				mv_state[1]++;
+				sched_yield();
+			}
+		}
+		*mv_state = 1;
+		for (;;)
+			mv_state[1]++;
+	}
+	return pid;
+}
+
+static int wait_spinner(int moving);
+
+static int moves(enum mv_mode mode, const char *name)
+{
+	int base_p, base_r, after_p, after_r, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up(name, mode == MV_NON_CONTENDING)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (mode == MV_NON_CONTENDING) {
+		sleep_ns(NSEC_PER_MSEC);
+	} else if (mode == MV_ASLEEP) {
+		sleep_ns(2 * PERIOD);
+	} else if (wait_spinner(0)) {
+		/* Throttled, or yielded: its counter stops. */
+		ksft_print_msg("%s: never stopped\n", name);
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	if (enter(cg_p, pid) || enter(root, pid)) {
+		ksft_print_msg("%s: a move refused\n", name);
+		ret = KSFT_FAIL;
+	}
+	if (mode == MV_THROTTLED || mode == MV_YIELDED) {
+		int last = mv_state[1];
+
+		sleep_ns(3 * PERIOD);
+		if (mv_state[1] == last) {
+			ksft_print_msg("%s: moved, never ran again\n", name);
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after_p = probe(cg_p);
+	after_r = probe(root);
+	if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+		ksft_print_msg("%s: probe tasks admitted, P %d -> %d, the other CPUs %d -> %d\n",
+			       name, base_p, after_p, base_r, after_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int move_refused(void)
+{
+	int base_p, base_r, ret = KSFT_PASS, err;
+	pid_t in_p, pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	/* 60% in P, then a task of 60% moved in: refused. */
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 60% in P", 0)) {
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	pid = mover(MV_ASLEEP, 60 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a task of 60% to move into P", 0)) {
+		reap(pid);
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	err = enter(cg_p, pid);
+	if (err != -EBUSY) {
+		ksft_print_msg("a move beyond P's room: %s, not EBUSY\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	reap(pid);
+	reap(in_p);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a refused move changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+static int move_asleep(void)
+{
+	return moves(MV_ASLEEP, "move_asleep");
+}
+
+static int move_throttled(void)
+{
+	return moves(MV_THROTTLED, "move_throttled");
+}
+
+static int move_non_contending(void)
+{
+	return moves(MV_NON_CONTENDING, "move_non_contending");
+}
+
+/*
+ * Writes sched_rt_runtime_us back to itself @n times, 5ms apart, a rebuild
+ * of the sched domains each, and exits.
+ */
+static pid_t rebuilder(int n)
+{
+	pid_t pid = fork_child("a sysctl writer");
+
+	if (!pid) {
+		char buf[64];
+
+		if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)))
+			_exit(1);
+		while (n-- > 0) {
+			write_file("/proc/sys/kernel", "sched_rt_runtime_us", buf);
+			sleep_ns(5 * NSEC_PER_MSEC);
+		}
+		_exit(0);
+	}
+	return pid;
+}
+
+static void *pauser(void *arg)
+{
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * A process of one deadline thread, asleep, and @threads other ones: a
+ * cgroup move of it walks them all between cpuset_can_attach() and
+ * cpuset_attach(), which is the window a rebuild has to land in.
+ */
+static void *dl_pauser(void *arg)
+{
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1;
+		return arg;
+	}
+	spin_ns(NSEC_PER_MSEC);
+	*mv_state = 1;
+	for (;;)
+		pause();
+	return arg;
+}
+
+/*
+ * A process of @threads pausers and, created last so that a cgroup move
+ * reaches it last, one deadline thread asleep: a rebuild that lands in the
+ * move finds it still in the cgroup it leaves.
+ */
+static pid_t wide_mover(int threads)
+{
+	pid_t pid = fork_child("a wide deadline process");
+
+	if (!pid) {
+		pthread_attr_t a;
+		pthread_t t;
+		int i;
+
+		if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+			goto no_threads;
+		for (i = 0; i < threads; i++) {
+			if (pthread_create(&t, &a, pauser, NULL))
+				goto no_threads;
+		}
+		if (pthread_create(&t, NULL, dl_pauser, NULL))
+			goto no_threads;
+		for (;;)
+			pause();
+no_threads:
+		mv_state[0] = -2;
+		_exit(1);
+	}
+	return pid;
+}
+
+/*
+ * A task moved in and out of P while the sched domains are rebuilt from
+ * the sysctl every 5ms: a rebuild between cpuset_can_attach() and
+ * cpuset_attach() drops the bandwidth the former set aside in the
+ * destination, and the move goes through on its strength. Written back
+ * to back, the 20 rebuilds would be over before the first move reaches
+ * that window. A rebuild puts the accounting right again, so the moves
+ * go on until the writer is done, and the last move it raced is what the
+ * probes see; six rounds.
+ */
+static int move_vs_rebuild(void)
+{
+	int base_p, base_r, with_p, with_r, after_p, after_r, round, ret = KSFT_PASS;
+	const char *name = "move_vs_rebuild";
+	pid_t pid, writer;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	pid = wide_mover(2048);
+	if (pid < 0 || child_up(name, 0)) {
+		reap(pid);
+		return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+	}
+	/* The reference of the rounds: the task asleep in the root cgroup. */
+	sleep_ns(2 * PERIOD);
+	with_p = probe(cg_p);
+	with_r = probe(root);
+	for (round = 0; round < 6 && ret == KSFT_PASS; round++) {
+		int status, err = 0;
+
+		writer = rebuilder(20);
+		if (writer < 0) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		while (!err && waitpid(writer, &status, WNOHANG) == 0) {
+			err = enter(cg_p, pid);
+			if (!err)
+				err = enter(root, pid);
+		}
+		if (err) {
+			ksft_print_msg("%s: a move refused: %s\n", name, strerror(-err));
+			reap(writer);
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(3 * PERIOD);
+		after_p = probe(cg_p);
+		after_r = probe(root);
+		if (after_p != with_p || after_r != with_r) {
+			ksft_print_msg("%s: round %d: P %d -> %d probe tasks, the rest %d -> %d\n",
+				       name, round, with_p, after_p, with_r, after_r);
+			ret = KSFT_FAIL;
+		}
+	}
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after_p = probe(cg_p);
+	after_r = probe(root);
+	if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+		ksft_print_msg("%s: task gone: P %d -> %d probe tasks, the rest %d -> %d\n",
+			       name, base_p, after_p, base_r, after_r);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Deaths, in P */
+
+enum die_mode { DIE_RUNNING, DIE_SWITCHED_OUT, DIE_SHRUNK };
+
+/*
+ * A deadline task of 30% that runs 5ms of a fresh period, its 0-lag time
+ * ahead, and dies there: as it is, or switched to SCHED_OTHER, its
+ * bandwidth then going at that time; or shrunk to 10%, which puts that
+ * time behind it, and the bandwidth goes at its death.
+ */
+static int die_case(enum die_mode mode, const char *what)
+{
+	pid_t pid = fork_child("a task that dies");
+	int status;
+
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		sleep_ns(PERIOD);
+		spin_ns(5 * NSEC_PER_MSEC);
+		if (mode == DIE_SWITCHED_OUT && set_other(0))
+			_exit(2);
+		if (mode == DIE_SHRUNK && set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+			_exit(3);
+		_exit(0);
+	}
+	if (pid < 0 || waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status)) {
+		ksft_print_msg("a task that dies %s failed\n", what);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+static int death_running(void)
+{
+	return die_case(DIE_RUNNING, "running");
+}
+
+static int death_switched_out(void)
+{
+	return die_case(DIE_SWITCHED_OUT, "switched out");
+}
+
+static int death_shrunk(void)
+{
+	return die_case(DIE_SHRUNK, "shrunk");
+}
+
+/*
+ * Waits until the spinner's mv_state[1] moves (@moving): it runs; or stops
+ * for 5ms: it is throttled. 0, or -1 after 1s.
+ */
+static int wait_spinner(int moving)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+	int last = mv_state[1];
+
+	while (now_ns() < end) {
+		sleep_ns(moving ? NSEC_PER_MSEC : 5 * NSEC_PER_MSEC);
+		if (moving ? mv_state[1] != last : mv_state[1] == last)
+			return 0;
+		last = mv_state[1];
+	}
+	return -1;
+}
+
+enum kill_when { KILL_AFTER, KILL_THROTTLED, KILL_RUNNING };
+
+/*
+ * A deadline task of 30% (mover()) killed in the state @mode leaves it in,
+ * @after that, or once it is throttled, or while it runs. The killer
+ * leaves P first, to watch and kill from another CPU without taking the
+ * task's.
+ */
+static int kill_case(enum mv_mode mode, enum kill_when when, uint64_t after, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t pid;
+
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(mode, 30 * NSEC_PER_MSEC);
+	if (pid < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (enter(root, getpid())) {
+		ksft_print_msg("killed %s: the killer cannot leave P\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (child_up(what, 1)) {
+		ret = KSFT_FAIL;
+	} else if (when != KILL_AFTER && wait_spinner(when == KILL_RUNNING)) {
+		ksft_print_msg("killed %s: never %s\n", what,
+			       when == KILL_RUNNING ? "ran" : "throttled");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(after);
+	}
+out:
+	reap(pid);
+	return ret;
+}
+
+static int death_killed_asleep(void)
+{
+	return kill_case(MV_ASLEEP, KILL_AFTER, 2 * PERIOD, "asleep");
+}
+
+static int death_killed_non_contending(void)
+{
+	return kill_case(MV_NON_CONTENDING, KILL_AFTER, 0, "before its 0-lag time");
+}
+
+static int death_killed_throttled(void)
+{
+	return kill_case(MV_THROTTLED, KILL_THROTTLED, 0, "throttled");
+}
+
+static int death_killed_running(void)
+{
+	return kill_case(MV_THROTTLED, KILL_RUNNING, 0, "running");
+}
+
+/*
+ * A robust priority inheritance lock that processes share: mv_state[0]
+ * tells its owner's state, mv_state[2] its waiter's.
+ */
+static pthread_mutex_t *shm_lock;
+
+/* 0, or -1 where such a mutex is not to be had. */
+static int shm_lock_init(void)
+{
+	pthread_mutexattr_t a;
+	int err;
+
+	err = pthread_mutexattr_init(&a);
+	if (!err)
+		err = pthread_mutexattr_setpshared(&a, PTHREAD_PROCESS_SHARED);
+	if (!err)
+		err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+	if (!err)
+		err = pthread_mutexattr_setrobust(&a, PTHREAD_MUTEX_ROBUST);
+	if (!err)
+		err = pthread_mutex_init(shm_lock, &a);
+	if (err) {
+		ksft_print_msg("no shared priority inheritance mutex: %s\n", strerror(err));
+		return -1;
+	}
+	mv_state[0] = 0;
+	mv_state[2] = 0;
+	return 0;
+}
+
+/* Takes the lock, as a deadline task of 30% if @dl, and sleeps with it. */
+static pid_t lock_owner(int dl)
+{
+	pid_t pid = fork_child("the owner of the lock");
+
+	if (!pid) {
+		if ((dl && set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) ||
+		    pthread_mutex_lock(shm_lock)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/* A deadline task of 5ms every 50ms that waits for the lock: 2 once it got it. */
+static pid_t lock_waiter(void)
+{
+	pid_t pid = fork_child("a waiter of the lock");
+
+	if (!pid) {
+		int err;
+
+		if (set_deadline(0, 5 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC)) {
+			mv_state[2] = -1;
+			_exit(1);
+		}
+		mv_state[2] = 1;
+		err = pthread_mutex_lock(shm_lock);
+		if (err == EOWNERDEAD)
+			err = pthread_mutex_consistent(shm_lock);
+		if (!err)
+			pthread_mutex_unlock(shm_lock);
+		mv_state[2] = err ? -2 : 2;
+		for (;;)
+			pause();
+	}
+	return pid;
+}
+
+/* @var reaches @want within 1s: 0, or -1. */
+static int wait_state(atomic_int *var, int want)
+{
+	uint64_t end = now_ns() + NSEC_PER_SEC;
+
+	while (now_ns() < end) {
+		if (*var == want)
+			return 0;
+		sleep_ns(NSEC_PER_MSEC);
+	}
+	return -1;
+}
+
+/*
+ * The owner sleeps with the lock, a deadline task if @dl_owner, and the
+ * waiter boosts it; then @kill_owner, or the waiter, dies. A killed owner
+ * does not die boosted: its exit releases its priority inheritance
+ * futexes first (futex_exit_release()), which deboosts it.
+ */
+static int pi_death(int dl_owner, int kill_owner, const char *what)
+{
+	int ret = KSFT_PASS;
+	pid_t owner, waiter = -1;
+
+	if (shm_lock_init())
+		return KSFT_SKIP;
+	owner = lock_owner(dl_owner);
+	if (owner < 0)
+		return KSFT_FAIL;
+	if (wait_state(&mv_state[0], 1)) {
+		ksft_print_msg("%s: no owner\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	waiter = lock_waiter();
+	if (waiter < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_state(&mv_state[2], 1)) {
+		ksft_print_msg("%s: no waiter\n", what);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	sleep_ns(50 * NSEC_PER_MSEC);	/* blocked, boosting the owner */
+	if (kill_owner) {
+		reap(owner);
+		owner = -1;
+		if (wait_state(&mv_state[2], 2)) {
+			ksft_print_msg("%s: the waiter did not get the lock\n", what);
+			ret = KSFT_FAIL;
+		}
+	} else {
+		reap(waiter);
+		waiter = -1;
+		sleep_ns(50 * NSEC_PER_MSEC);	/* the owner deboosted asleep */
+	}
+out:
+	reap(waiter);
+	reap(owner);
+	return ret;
+}
+
+static int death_pi_waiter(void)
+{
+	return pi_death(0, 0, "a waiter killed");
+}
+
+static int death_pi_owner(void)
+{
+	return pi_death(0, 1, "a boosted owner killed");
+}
+
+static int death_pi_deadline_owner(void)
+{
+	return pi_death(1, 1, "a boosted deadline owner killed");
+}
+
+/* Deadline threads of one process, asleep, throttled and running, killed at once. */
+/*
+ * Thread 0 sleeps; thread 1 spins on 10ms of 100ms, throttled first; thread
+ * 2, on 60ms, starts once the killer says so and counts while it runs.
+ */
+static void *group_thread(void *arg)
+{
+	long kind = (long)arg;
+
+	if (set_deadline(0, (kind == 2 ? 60 : 10) * NSEC_PER_MSEC, PERIOD)) {
+		mv_state[0] = -1000;
+		return NULL;
+	}
+	if (kind == 2)
+		while (!mv_state[2])
+			sleep_ns(NSEC_PER_MSEC);
+	spin_ns(NSEC_PER_MSEC);
+	mv_state[0]++;
+	if (!kind)
+		for (;;)
+			pause();
+	for (;;)
+		mv_state[kind == 2 ? 3 : 1]++;
+	return NULL;
+}
+
+static int death_group(void)
+{
+	int last, ret = KSFT_PASS;
+	pid_t pid;
+
+	mv_state[0] = mv_state[1] = mv_state[2] = mv_state[3] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (!pid) {
+		pthread_t t;
+		long i;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, group_thread, (void *)i))
+				mv_state[0] = -1000;
+		for (;;)
+			pause();
+	}
+	/* From outside P: in it, the killer never runs while a thread does. */
+	if (enter(root, getpid())) {
+		ksft_print_msg("the killer cannot leave P\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_for(5000, mv_state[0] < 0 || mv_state[0] == 2) || mv_state[0] != 2) {
+		ksft_print_msg("only %d threads of the group started\n", (int)mv_state[0]);
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	if (wait_spinner(0)) {
+		ksft_print_msg("the first spinner never throttled\n");
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	last = mv_state[3];
+	mv_state[2] = 1;
+	if (wait_for(5000, mv_state[0] < 0 || mv_state[3] != last) || mv_state[0] != 3) {
+		ksft_print_msg("the second spinner never ran\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	/* One asleep, one throttled, one running. */
+	reap(pid);
+	return ret;
+}
+
+/*
+ * A deadline task of P exits and stays a zombie: cgroup_task_dead() took
+ * it off P's count of deadline tasks. Switched to SCHED_OTHER as a zombie,
+ * which is refused, it would run switched_from_dl() and be taken off that
+ * count a second time: at 0 with a deadline task left in P, a rebuild of
+ * the accounting skips P, and that task's bandwidth with it.
+ */
+static int death_zombie_switched(void)
+{
+	int base, ret = KSFT_PASS, err;
+	pid_t in_p, zombie;
+
+	base = probe(cg_p);
+	if (base < 4) {
+		ksft_print_msg("no room in P for a task of 30%% and a probe\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	in_p = fork_child("a deadline task of P");
+	if (!in_p) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		for (;;)
+			pause();
+	}
+	if (in_p < 0 || child_up("a task of 30% in P", 0)) {
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	mv_state[0] = 0;
+	zombie = fork_child("a deadline task that exits");
+	if (!zombie) {
+		if (enter(cg_p, getpid()) || set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0] = 1;
+		_exit(0);
+	}
+	if (zombie < 0 || child_up("a task of 10% in P that exits", 0) ||
+	    wait_for(1000, proc_state_of(zombie) == 'Z')) {
+		ksft_print_msg("no zombie deadline task\n");
+		reap(zombie);
+		reap(in_p);
+		return KSFT_FAIL;
+	}
+	/* Its bandwidth goes at its 0-lag time, within a period. */
+	sleep_ns(3 * PERIOD);
+	err = set_other(zombie);
+	if (err != -ESRCH) {
+		ksft_print_msg("SCHED_OTHER on a zombie deadline task: %s, not ESRCH\n",
+			       err ? strerror(-err) : "accepted");
+		ret = KSFT_FAIL;
+	}
+	/* A rebuild of the accounting: Q made beside P, and unmade. */
+	if (make_partition(cg_q, top_cpu(2))) {
+		ksft_print_msg("no partition root Q beside P\n");
+		ret = KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	if (admits_in(cg_p, base - 3, "P rebuilt, a task of 30% and a zombie in it"))
+		ret = KSFT_FAIL;
+	reap(zombie);
+	reap(in_p);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits_in(cg_p, base, "P, its tasks gone"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* sched_yield() */
+
+static atomic_int y_rounds;	/* the yields of the yielder */
+
+/*
+ * A deadline task of 30% that yields half a millisecond into every
+ * period: it spends nearly all of its time throttled by sched_yield(),
+ * with runtime left, so its 0-lag time is a few milliseconds away and the
+ * inactive timer that releases its bandwidth is armed for that long.
+ * @t_state is 2 while it is in the yield.
+ */
+static void *yielder(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	while (!spin_stop) {
+		spin_ns(NSEC_PER_MSEC / 2);
+		y_rounds++;
+		t_state = 2;
+		sched_yield();
+		t_state = 1;
+	}
+	return NULL;
+}
+
+static int start_yielder(pthread_t *t)
+{
+	t_state = 0;
+	spin_stop = 0;
+	y_rounds = 0;
+	if (pthread_create(t, NULL, yielder, NULL)) {
+		ksft_print_msg("the yielder thread could not start\n");
+		return -1;
+	}
+	if (!wait_for(5000, t_state) && t_state > 0)
+		return 0;
+	spin_stop = 1;		/* in case it gets there late */
+	join(*t);
+	return -1;
+}
+
+/*
+ * A task switched to SCHED_OTHER while it is throttled by sched_yield():
+ * its bandwidth comes back, and it runs again.
+ */
+static int yield_switched_out(void)
+{
+	int base, ret = KSFT_PASS;
+	pthread_t t;
+
+	base = probe(NULL);
+	if (base < 3) {
+		ksft_print_msg("no room for a task of 30%%\n");
+		return KSFT_SKIP;
+	}
+	if (start_yielder(&t))
+		return KSFT_FAIL;
+	if (wait_throttled(t) || t_state != 2) {
+		ksft_print_msg("the task never yielded\n");
+		ret = KSFT_FAIL;
+	} else if (set_other(t_tid)) {
+		ret = KSFT_FAIL;
+	} else {
+		int last = y_rounds;
+
+		sleep_ns(2 * PERIOD);
+		if (y_rounds == last) {
+			ksft_print_msg("yielded, then switched out: never ran again\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "yielded, then switched out"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+static int yield_moved(void)
+{
+	return moves(MV_YIELDED, "yield_moved");
+}
+
+/* A task killed while it is throttled by sched_yield(). */
+static int yield_killed(void)
+{
+	pid_t pid;
+
+	mv_state[0] = 0;
+	pid = fork_child("a task that yields");
+	if (!pid) {
+		if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			spin_ns(NSEC_PER_MSEC / 2);
+			sched_yield();
+		}
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a task that yields", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	reap(pid);
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_DL_OVERRUN */
+
+static atomic_int got_xcpu;
+
+static void catch_xcpu(int sig)
+{
+	got_xcpu = 1;
+}
+
+/* A deadline task of 10% with SCHED_FLAG_DL_OVERRUN that runs past it. */
+static void *overruns(void *arg)
+{
+	t_tid = gettid_();
+	if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD, SCHED_FLAG_DL_OVERRUN)) {
+		t_state = -1;
+		return (void *)1;
+	}
+	t_state = 1;
+	spin_ns(15 * NSEC_PER_MSEC);
+	while (!spin_stop)
+		sleep_ns(10 * NSEC_PER_MSEC);
+	return NULL;
+}
+
+/*
+ * The thread group of a task with SCHED_FLAG_DL_OVERRUN that overruns its
+ * runtime gets SIGXCPU. Caught here: the overrun leaves the task's
+ * bandwidth reserved, and it comes back when the task leaves
+ * SCHED_DEADLINE.
+ */
+static int overrun_caught(void)
+{
+	int base, ret = KSFT_PASS;
+	uint64_t end;
+	pthread_t t;
+
+	if (signal(SIGXCPU, catch_xcpu) == SIG_ERR)
+		return KSFT_FAIL;
+	base = probe(NULL);
+	if (base < 1) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	got_xcpu = 0;
+	t_state = 0;
+	spin_stop = 0;
+	if (pthread_create(&t, NULL, overruns, NULL))
+		return KSFT_FAIL;
+	while (!t_state)
+		sleep_ns(NSEC_PER_MSEC);
+	end = now_ns() + 2 * NSEC_PER_SEC;
+	while (!got_xcpu && now_ns() < end)
+		sleep_ns(5 * NSEC_PER_MSEC);
+	if (t_state != 1) {
+		ret = KSFT_FAIL;
+	} else if (!got_xcpu) {
+		ksft_print_msg("it overran its runtime, no SIGXCPU in 2s\n");
+		ret = KSFT_FAIL;
+	} else if (admits(base - 1, "an overrun told")) {
+		ret = KSFT_FAIL;	/* the overrun gave its 10% back */
+	}
+	stop_spinner(t);
+	sleep_ns(3 * PERIOD);		/* its 0-lag time */
+	if (ret == KSFT_PASS && admits(base, "after an overrun was told"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/* The deadline threads of overrun_group_death(). */
+static void *og_thread(void *arg)
+{
+	switch ((int)(long)arg) {
+	case 0:		/* overruns its runtime: SIGXCPU to the group */
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_DL_OVERRUN))
+			_exit(1);
+		/*
+		 * The other two first: of one CPU, this one has the earliest
+		 * deadline of the three and would overrun, and kill them,
+		 * before they are where the signal is to find them.
+		 */
+		if (wait_for(5000, mv_state[0] >= 2))
+			_exit(2);
+		spin_ns(30 * NSEC_PER_MSEC);
+		break;
+	case 1:		/* asleep, its 0-lag time past */
+		if (set_deadline(0, 20 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		spin_ns(NSEC_PER_MSEC);
+		mv_state[0]++;
+		break;
+	case 2:		/* asleep, mostly before its 0-lag time */
+		if (set_deadline(0, 15 * NSEC_PER_MSEC, PERIOD))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			spin_ns(3 * NSEC_PER_MSEC);
+			sleep_ns(22 * NSEC_PER_MSEC);
+		}
+	}
+	for (;;)
+		pause();
+}
+
+/*
+ * The SIGXCPU of an overrun kills the whole thread group: the bandwidth
+ * of its three deadline threads comes back.
+ */
+static int overrun_group_death(void)
+{
+	pid_t pid;
+	int i, status;
+
+	mv_state[0] = 0;
+	pid = fork_child("a group of deadline threads");
+	if (!pid) {
+		pthread_t t;
+
+		for (i = 0; i < 3; i++)
+			if (pthread_create(&t, NULL, og_thread, (void *)(long)i))
+				_exit(3);
+		for (;;)
+			pause();
+	}
+	if (pid < 0)
+		return KSFT_FAIL;
+	for (i = 0; i < 300; i++) {
+		if (waitpid(pid, &status, WNOHANG) == pid)
+			break;
+		sleep_ns(10 * NSEC_PER_MSEC);
+	}
+	if (i == 300) {
+		reap(pid);
+		ksft_print_msg("an overrun of 10ms every 100ms: no SIGXCPU in 3s\n");
+		return KSFT_FAIL;
+	}
+	if (WIFEXITED(status)) {
+		switch (WEXITSTATUS(status)) {
+		case 1:
+			ksft_print_msg("a thread of the group was refused SCHED_DEADLINE\n");
+			return KSFT_SKIP;
+		case 2:
+			ksft_print_msg("the other two threads of the group were not ready in 5s\n");
+			break;
+		default:
+			ksft_print_msg("a thread of the group could not start\n");
+		}
+		return KSFT_FAIL;
+	}
+	if (!WIFSIGNALED(status) || WTERMSIG(status) != SIGXCPU) {
+		ksft_print_msg("the group of an overrun died of signal %d, not SIGXCPU\n",
+			       WIFSIGNALED(status) ? WTERMSIG(status) : 0);
+		return KSFT_FAIL;
+	}
+	return KSFT_PASS;
+}
+
+/* SCHED_FLAG_RECLAIM */
+
+#define RECLAIM_MIN	40	/* % of a CPU a reclaiming task of 10% gets */
+#define PLAIN_MAX	25	/* % one without the flag gets */
+#define SHARE_SLACK	15	/* % two measures of the same may differ by */
+#define SHARE_NS	(400 * NSEC_PER_MSEC)
+
+/*
+ * A deadline task with SCHED_FLAG_RECLAIM runs past its runtime, taking
+ * the bandwidth of its root domain that no deadline task uses: of a CPU
+ * of its own, a task of 10% every 100ms gets most of it, where one
+ * without the flag gets its 10%.
+ */
+static int reclaim_runs(void)
+{
+	int plain, grub, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, 0)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	plain = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM))
+		return KSFT_FAIL;
+	grub = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (plain > PLAIN_MAX || grub < RECLAIM_MIN) {
+		ksft_print_msg("of a CPU, a task of 10%% got %d%%, with SCHED_FLAG_RECLAIM %d%%\n",
+			       plain, grub);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A reclaiming task switched out of SCHED_DEADLINE before its 0-lag time
+ * and back, which leaves its bandwidth standing until the inactive timer
+ * releases it: what it reclaims afterwards is what it reclaimed before.
+ */
+static int reclaim_switched(void)
+{
+	int before, after = 0, r, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	before = share_of(t, SHARE_NS);
+	for (r = 0; r < 4; r++) {
+		if (set_other(t_tid)) {
+			ret = KSFT_FAIL;
+			break;
+		}
+		if (r & 1)
+			sleep_ns(5 * NSEC_PER_MSEC);
+		if (set_deadline_flags(t_tid, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			ksft_print_msg("round %d of 4: back to SCHED_DEADLINE refused\n",
+				       r + 1);
+			ret = KSFT_FAIL;
+			break;
+		}
+		sleep_ns(PERIOD);
+	}
+	if (ret == KSFT_PASS)
+		after = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	if (ret == KSFT_PASS &&
+	    (after < before - SHARE_SLACK || after > before + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%% of a CPU, %d%% after 4 round trips\n",
+			       before, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A reclaiming deadline task of 10% that spins, publishing the CPU time
+ * it has used in milliseconds, and idles while mv_state[2] is set. The
+ * moves happen then: a cpuset move waits for an RCU grace period, which
+ * a task that reclaims a whole CPU of its own keeps from ending where
+ * RCU runs in kthreads of its own (PREEMPT_RT).
+ */
+static pid_t reclaim_mover(void)
+{
+	pid_t pid = fork_child("a reclaiming task");
+
+	if (!pid) {
+		if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+				       SCHED_FLAG_RECLAIM)) {
+			mv_state[0] = -1;
+			_exit(1);
+		}
+		mv_state[0] = 1;
+		for (;;) {
+			if (mv_state[2]) {
+				sleep_ns(10 * NSEC_PER_MSEC);
+				continue;
+			}
+			spin_ns(NSEC_PER_MSEC);
+			mv_state[1] = (int)(thread_time_ns() / NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* Moves the mover to @cg, idle: 0, or -1. */
+static int move_idle(const char *cg, pid_t pid)
+{
+	int err;
+
+	mv_state[2] = 1;
+	sleep_ns(20 * NSEC_PER_MSEC);	/* it stops spinning */
+	err = enter(cg, pid);
+	mv_state[2] = 0;
+	return err;
+}
+
+/* The share of a CPU, in percent, the mover uses over the next @ns. */
+static int mover_share(uint64_t ns)
+{
+	uint64_t wall = now_ns();
+	int cpu = mv_state[1];
+
+	sleep_ns(ns);
+	cpu = mv_state[1] - cpu;
+	wall = now_ns() - wall;
+	return (int)(100 * (uint64_t)cpu * NSEC_PER_MSEC / wall);
+}
+
+/*
+ * A reclaiming task moved into P and back, then into P again: what it
+ * reclaims of P's CPU is what it reclaimed there before, and the free
+ * bandwidth of both root domains is the one it found.
+ */
+static int reclaim_moved(void)
+{
+	int base_p, base_r, in_p, again = 0, ret = KSFT_PASS;
+	pid_t pid;
+
+	base_p = probe(cg_p);
+	base_r = probe(root);
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	mv_state[2] = 1;		/* idle until it is in P */
+	pid = reclaim_mover();
+	if (pid < 0)
+		return KSFT_FAIL;
+	if (child_up("a reclaiming task", 0) || move_idle(cg_p, pid)) {
+		mv_state[2] = 1;
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(2 * PERIOD);
+	in_p = mover_share(SHARE_NS);
+	if (move_idle(root, pid) || move_idle(cg_p, pid)) {
+		ksft_print_msg("a move of a reclaiming task refused\n");
+		ret = KSFT_FAIL;
+	} else {
+		sleep_ns(2 * PERIOD);
+		again = mover_share(SHARE_NS);
+		if (in_p < RECLAIM_MIN || again < in_p - SHARE_SLACK ||
+		    again > in_p + SHARE_SLACK) {
+			ksft_print_msg("reclaiming: %d%% of P's CPU, %d%% after a move\n",
+				       in_p, again);
+			ret = KSFT_FAIL;
+		}
+	}
+	mv_state[2] = 1;
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+		ksft_print_msg("a reclaiming task moved: the free bandwidth changed\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Deadline servers */
+
+#define SERVER_RUNTIME	(100 * NSEC_PER_MSEC)	/* 10%, where the cases start from */
+#define SERVER_PERIOD	(1000 * NSEC_PER_MSEC)
+
+static const char server_dir[] = "/sys/kernel/debug/sched/fair_server";
+static uint64_t *server_found;	/* runtime and period of each CPU's fair server */
+
+static int server_get(int cpu, const char *name, uint64_t *val)
+{
+	char path[96], buf[32];
+
+	snprintf(path, sizeof(path), "%s/cpu%d/%s", server_dir, cpu, name);
+	if (read_file(path, buf, sizeof(buf)))
+		return -1;
+	*val = strtoull(buf, NULL, 10);
+	return 0;
+}
+
+static int server_put(int cpu, const char *name, uint64_t val)
+{
+	char dir[96], buf[32];
+
+	snprintf(dir, sizeof(dir), "%s/cpu%d", server_dir, cpu);
+	snprintf(buf, sizeof(buf), "%llu", (unsigned long long)val);
+	return write_file(dir, name, buf);
+}
+
+/* The fair server of @cpu runs @runtime every @period: 0, or -errno. */
+static int server_set(int cpu, uint64_t runtime, uint64_t period)
+{
+	uint64_t cur;
+	int err;
+
+	if (!server_found || server_get(cpu, "period", &cur))
+		return -ENOENT;
+	/* Each write keeps the runtime within the period. */
+	if (runtime <= cur) {
+		err = server_put(cpu, "runtime", runtime);
+		return err ? err : server_put(cpu, "period", period);
+	}
+	err = server_put(cpu, "period", period);
+	return err ? err : server_put(cpu, "runtime", runtime);
+}
+
+/* The fair servers as the test finds them, or none without debugfs. */
+static void server_init(void)
+{
+	int i;
+
+	server_found = calloc(2 * ncpus, sizeof(*server_found));
+	for (i = 0; server_found && i < ncpus; i++) {
+		if (server_get(online[i], "runtime", &server_found[2 * i]) ||
+		    server_get(online[i], "period", &server_found[2 * i + 1])) {
+			free(server_found);
+			server_found = NULL;
+			ksft_print_msg("no fair server in debugfs: server cases skipped\n");
+		}
+	}
+}
+
+/* Puts the fair servers back as the test found them. */
+static void server_restore(void)
+{
+	uint64_t runtime, period;
+	int i;
+
+	for (i = 0; server_found && i < ncpus; i++) {
+		int cpu = online[i], err;
+
+		if (server_get(cpu, "runtime", &runtime) || server_get(cpu, "period", &period))
+			continue;
+		if (runtime == server_found[2 * i] && period == server_found[2 * i + 1])
+			continue;
+		/*
+		 * A root domain whose departing tasks still hold their
+		 * bandwidth has room for nothing, a decrease included:
+		 * their 0-lag times are a period or two away.
+		 */
+		err = wait_for(1000, !server_set(cpu, server_found[2 * i],
+						 server_found[2 * i + 1]));
+		if (err)
+			ksft_print_msg("the fair server of CPU %d stays changed\n", cpu);
+	}
+}
+
+/* Deadline threads of 10% every 100ms that sleep until released. */
+static pthread_t *held;
+static int nheld, held_err;
+static sem_t held_up, held_go;
+
+static void *holder(void *arg)
+{
+	int err = set_deadline(0, PROBE_RUNTIME, PROBE_PERIOD);
+
+	held_err = err;
+	if (sem_post(&held_up))
+		ksft_print_msg("sem_post: %s\n", strerror(errno));
+	if (!err)
+		sem_wait_sig(&held_go);
+	return NULL;
+}
+
+/* Starts @n more holders: 0 once they are all admitted, or -1. */
+static int hold(int n)
+{
+	int max = 16 * ncpus + 16, err;
+	pthread_attr_t a;
+
+	if (!held) {
+		held = calloc(max, sizeof(*held));
+		if (!held || sem_init(&held_up, 0, 0) || sem_init(&held_go, 0, 0)) {
+			ksft_print_msg("the holders cannot be set up\n");
+			return -1;
+		}
+	}
+	err = pthread_attr_init(&a);
+	if (!err)
+		err = pthread_attr_setstacksize(&a, THREAD_STACK);
+	if (err) {
+		ksft_print_msg("pthread_attr: %s\n", strerror(err));
+		return -1;
+	}
+	for (; n > 0; n--) {
+		if (nheld == max || pthread_create(&held[nheld], &a, holder, NULL)) {
+			ksft_print_msg("a holder thread could not start\n");
+			return -1;
+		}
+		if (sem_wait_for(&held_up, 10000)) {
+			ksft_print_msg("a holder did not answer\n");
+			nheld++;	/* released and joined with the others */
+			return -1;
+		}
+		if (held_err) {
+			join(held[nheld]);
+			return -1;
+		}
+		nheld++;
+	}
+	return 0;
+}
+
+static void release_held(void)
+{
+	int i;
+
+	for (i = 0; i < nheld; i++)
+		if (sem_post(&held_go))
+			ksft_print_msg("sem_post: %s\n", strerror(errno));
+	for (i = 0; i < nheld; i++)
+		join(held[i]);
+	/* What a holder that answered late, or was refused, left over. */
+	while (held && (!sem_trywait(&held_up) || !sem_trywait(&held_go)))
+		;
+	nheld = 0;
+	sleep_ns(3 * PERIOD);	/* their 0-lag times */
+}
+
+/*
+ * The fair server of P's CPU, changed at runtime, takes from P's root
+ * domain what deadline tasks then cannot: from 10% every second, P admits
+ * two probe tasks fewer with 30%, one fewer with 100ms every 500ms, one
+ * more with the server off. A change P has no room for is refused.
+ */
+static int server_params(void)
+{
+	static const struct {
+		uint64_t runtime, period;
+		int more;
+		const char *what;
+	} step[] = {
+		{ 3 * SERVER_RUNTIME, SERVER_PERIOD, -2, "a fair server of 30%" },
+		{ SERVER_RUNTIME, SERVER_PERIOD / 2, -1, "a fair server of 100ms every 500ms" },
+		{ 0, SERVER_PERIOD, 1, "the fair server off" },
+		{ SERVER_RUNTIME, SERVER_PERIOD, 0, "the fair server back to 10%" },
+	};
+	int cpu = top_cpu(1), base, err, i, ret = KSFT_PASS;
+	uint64_t runtime;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	for (i = 0; i < (int)ARRAY_SIZE(step); i++) {
+		err = server_set(cpu, step[i].runtime, step[i].period);
+		if (err) {
+			ksft_print_msg("%s: %s\n", step[i].what, strerror(-err));
+			return KSFT_FAIL;
+		}
+		if (admits(base + step[i].more, step[i].what))
+			return KSFT_FAIL;
+	}
+	/*
+	 * With base - 1 probe tasks held, P has room for 10% more and less
+	 * than 20%: the server may grow to 20%, not to 30%.
+	 */
+	if (hold(base - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = server_put(cpu, "runtime", 2 * SERVER_RUNTIME);
+	if (err) {
+		ksft_print_msg("a fair server of 20%% with room for it: %s\n", strerror(-err));
+		ret = KSFT_FAIL;
+	} else {
+		err = server_put(cpu, "runtime", 3 * SERVER_RUNTIME);
+		if (err != -EBUSY) {
+			ksft_print_msg("a fair server of 30%% with no room for it: %s, not EBUSY\n",
+				       err ? strerror(-err) : "accepted");
+			ret = KSFT_FAIL;
+		} else if (server_get(cpu, "runtime", &runtime) ||
+			   runtime != 2 * SERVER_RUNTIME) {
+			ksft_print_msg("a refused change changed the fair server\n");
+			ret = KSFT_FAIL;
+		}
+	}
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	release_held();
+	if (ret == KSFT_PASS && admits(base, "after a refused change of the fair server"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * The rebuilds of the root domains' accounting put the fair servers'
+ * bandwidth back as it is set, not as it was at boot: P's own, when a
+ * partition root Q is made and unmade beside it; that of the other CPUs,
+ * when P is unmade and its CPU joins them with its server; P's again,
+ * when it is made anew.
+ */
+static int server_rebuilds(void)
+{
+	int cpu = top_cpu(1), base, n10, n30;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base < 2) {
+		ksft_print_msg("no room in P for two probe tasks\n");
+		return KSFT_SKIP;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base - 2, "P, a fair server of 30%"))
+		return KSFT_FAIL;
+	if (make_partition(cg_q, top_cpu(2))) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root Q beside P\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P, Q made beside it")) {
+		remove_partition(cg_q);
+		return KSFT_FAIL;
+	}
+	remove_partition(cg_q);
+	if (admits_in(cg_p, base - 2, "P, Q unmade beside it"))
+		return KSFT_FAIL;
+	/* P unmade: its CPU and its server join the root domain of the others. */
+	remove_p();
+	n30 = probe(root);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		return KSFT_FAIL;
+	n10 = probe(root);
+	if (n10 != n30 + 2) {
+		ksft_print_msg("P unmade: %d probe tasks, its CPU's server at 30%%, %d at 10%%\n",
+			       n30, n10);
+		return KSFT_FAIL;
+	}
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) || make_p()) {
+		ksft_print_msg("P made anew, a fair server of 30%%: failed\n");
+		return KSFT_FAIL;
+	}
+	if (admits_in(cg_p, base - 2, "P made anew, a fair server of 30%") ||
+	    server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits_in(cg_p, base, "P, the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/* Waits for @f to be set: a futex in shared memory. */
+static void wait_set(atomic_int *f)
+{
+	while (!*f)
+		syscall(SYS_futex, f, FUTEX_WAIT, 0, NULL, NULL, 0);
+}
+
+static void set_flag(atomic_int *f, int val)
+{
+	*f = val;
+	syscall(SYS_futex, f, FUTEX_WAKE, INT_MAX, NULL, NULL, 0);
+}
+
+/*
+ * A process of P that spins while mv_state[4 + @rt] is set, and sleeps
+ * otherwise: a fair task, or a real-time one (@rt) that leaves P's CPU
+ * for a millisecond every 50ms to the real-time kthreads of that CPU
+ * (PREEMPT_RT runs RCU and the timers there).
+ */
+static pid_t server_client(int rt)
+{
+	pid_t pid = fork_child(rt ? "a real-time task of P" : "a fair task of P");
+
+	if (!pid) {
+		struct sched_param sp = { .sched_priority = 1 };
+
+		if (enter(cg_p, getpid()) ||
+		    (rt && sched_setscheduler(0, SCHED_FIFO, &sp)))
+			_exit(1);
+		mv_state[0]++;
+		for (;;) {
+			wait_set(&mv_state[4 + rt]);
+			spin_ns(rt ? 50 * NSEC_PER_MSEC : NSEC_PER_MSEC);
+			if (rt)
+				sleep_ns(NSEC_PER_MSEC);
+		}
+	}
+	return pid;
+}
+
+/* The share of a CPU, in percent, that process @pid uses over @ns. */
+static int proc_share(pid_t pid, uint64_t ns)
+{
+	struct timespec t0, t1;
+	uint64_t wall = now_ns();
+	clockid_t c;
+
+	if (clock_getcpuclockid(pid, &c) || clock_gettime(c, &t0))
+		return -1;
+	sleep_ns(ns);
+	if (clock_gettime(c, &t1))
+		return -1;
+	wall = now_ns() - wall;
+	return (int)(100 * ((t1.tv_sec - t0.tv_sec) * NSEC_PER_SEC +
+			    t1.tv_nsec - t0.tv_nsec) / wall);
+}
+
+#define STARVED_SLACK	8	/* % a starved fair task's share may be off its server's */
+
+/*
+ * The fair server of P's CPU changed from outside P in the states a change
+ * can find it in: stopped, deferred while a fair task of P runs, deferred
+ * while a real-time task starves it, serving it, deferred again after
+ * serving it, and deferred while the CPU idles. Each change starts a
+ * period of the server, which defers serving a starved task to its
+ * zero-laxity time, 0.7s in at 30% and 0.9s in at 10%. Afterwards, the
+ * starved task gets the server's bandwidth as set last, and P's free
+ * bandwidth is as it was. (Throttled after serving, the server is
+ * deferred anew at once: the last of the served task's runtime is charged
+ * to it as running in the background. Its idle wait ends in the timer
+ * callback that starts it, on a CPU with no tick in idle.)
+ */
+static int server_states(void)
+{
+	static const struct {
+		int fair, rt, ms;
+	} step[] = {
+		{ 0, 0, 1500 },	/* stopped: nothing of P ran for a period of the server */
+		{ 1, 0, 300 },	/* deferred while the fair task runs */
+		{ 1, 1, 300 },	/* deferred, the fair task starved: before 0.9s */
+		{ 1, 1, 800 },	/* serving it: after 0.7s */
+		{ 1, 1, 1500 },	/* deferred again: served from 0.9s to 1s */
+		{ 0, 0, 600 },	/* deferred while the CPU idles */
+	};
+	int cpu = top_cpu(1), base, i, s30 = -1, s10 = -1, ret = KSFT_PASS;
+	pid_t fair, rt;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(cg_p);
+	if (base <= 0) {
+		ksft_print_msg("no room in P to probe\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	set_flag(&mv_state[4], 0);
+	set_flag(&mv_state[5], 0);
+	fair = server_client(0);
+	rt = server_client(1);
+	if (fair < 0 || rt < 0 || wait_for(5000, mv_state[0] >= 2)) {
+		if (fair >= 0 && rt >= 0)
+			ksft_print_msg("a fair or real-time task of P could not start\n");
+		ret = KSFT_FAIL;
+	}
+	/* Even steps set the server to 30%, odd ones back to 10%: its runtime alone. */
+	for (i = 0; i < (int)ARRAY_SIZE(step) && ret == KSFT_PASS; i++) {
+		set_flag(&mv_state[4], step[i].fair);
+		set_flag(&mv_state[5], step[i].rt);
+		sleep_ns(step[i].ms * NSEC_PER_MSEC);
+		if (server_put(cpu, "runtime", (i & 1 ? 1 : 3) * SERVER_RUNTIME)) {
+			ksft_print_msg("change %d of the fair server refused\n", i + 1);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS) {
+		set_flag(&mv_state[4], 1);
+		set_flag(&mv_state[5], 1);
+		if (server_put(cpu, "runtime", 3 * SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s30 = proc_share(fair, 2 * SERVER_PERIOD);
+		if (server_put(cpu, "runtime", SERVER_RUNTIME))
+			ret = KSFT_FAIL;
+		s10 = proc_share(fair, 2 * SERVER_PERIOD);
+	}
+	set_flag(&mv_state[5], 0);
+	set_flag(&mv_state[4], 0);
+	reap(rt);
+	reap(fair);
+	if (ret == KSFT_PASS &&
+	    (s30 < 30 - STARVED_SLACK || s30 > 30 + STARVED_SLACK ||
+	     s10 < 10 - STARVED_SLACK || s10 > 10 + STARVED_SLACK)) {
+		ksft_print_msg("a starved fair task got %d%% of a server of 30%%, %d%% of 10%%\n",
+			       s30, s10);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && admits_in(cg_p, base, "after the fair server's changes"))
+		ret = KSFT_FAIL;
+	return ret;
+}
+
+/*
+ * A reclaiming deadline task of P gets the share of P's CPU its root
+ * domain leaves free, which P's fair server takes from: less once the
+ * server grows, as much after a rebuild of the accounting (Q made and
+ * unmade), and what it had once the server is back. The server is then
+ * put back as the test found it, as P's free bandwidth must be.
+ */
+static int server_reclaim(void)
+{
+	int cpu = top_cpu(1), s10, s30, rebuilt, back, ret = KSFT_PASS;
+	pthread_t t;
+
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", cpu);
+		return KSFT_SKIP;
+	}
+	if (start_spinner(&t, 10 * NSEC_PER_MSEC, SCHED_FLAG_RECLAIM)) {
+		ksft_print_msg("no room for a task of 10%%\n");
+		return KSFT_SKIP;
+	}
+	s10 = share_of(t, SHARE_NS);
+	if (server_set(cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	s30 = share_of(t, SHARE_NS);
+	if (make_partition(cg_q, top_cpu(2)))
+		ret = KSFT_FAIL;
+	remove_partition(cg_q);
+	rebuilt = share_of(t, SHARE_NS);
+	if (server_set(cpu, SERVER_RUNTIME, SERVER_PERIOD))
+		ret = KSFT_FAIL;
+	back = share_of(t, SHARE_NS);
+	stop_spinner(t);
+	server_restore();
+	if (ret == KSFT_PASS &&
+	    (s30 > s10 - SHARE_SLACK ||
+	     rebuilt < s30 - SHARE_SLACK || rebuilt > s30 + SHARE_SLACK ||
+	     back < s10 - SHARE_SLACK || back > s10 + SHARE_SLACK)) {
+		ksft_print_msg("reclaiming: %d%%, %d%% at 30%%, %d%% rebuilt, %d%% back at 10%%\n",
+			       s10, s30, rebuilt, back);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* Parameter changes while a task wakes on another CPU */
+
+#define WAKE_NS		(5 * NSEC_PER_SEC)	/* how long the changes go on */
+#define WAKE_BURST	2000			/* changes between two clock readings */
+
+/* this_bw and running_bw of @cpu, from sched_debug: 0, or -1. */
+static int rq_bw_of(int cpu, long *this_bw, long *running_bw)
+{
+	char line[256], want[32];
+	FILE *f = fopen("/sys/kernel/debug/sched/debug", "r");
+	int in = 0, got = 0;
+
+	if (!f)
+		return -1;
+	snprintf(want, sizeof(want), "dl_rq[%d]:", cpu);
+	while (got < 2 && fgets(line, sizeof(line), f)) {
+		char *val = strchr(line, ':');
+
+		if (!strncmp(line, want, strlen(want)))
+			in = 1;
+		else if (in && !strncmp(line, "dl_rq[", 6))
+			break;			/* the next CPU's */
+		else if (!in || !val)
+			continue;
+		else if (strstr(line, ".running_bw") && sscanf(val + 1, "%ld", running_bw) == 1)
+			got++;
+		else if (strstr(line, ".this_bw") && sscanf(val + 1, "%ld", this_bw) == 1)
+			got++;
+	}
+	fclose(f);
+	return got == 2 ? 0 : -1;
+}
+
+/*
+ * A deadline task of Q, a partition root of two CPUs taken by two
+ * deadline tasks of an earlier deadline, that sleeps and wakes: it wakes
+ * on whichever of the two has the later deadline to run, which moves it.
+ * Its parameters change meanwhile, from a CPU outside Q.
+ *
+ * A wakeup that moves a task takes its bandwidth off the runqueue it
+ * slept on, and the enqueue on the one it wakes on, which that CPU may
+ * be left to do, charges it there. A change in between must not take it
+ * off the runqueue of either: once every task of the case is gone,
+ * this_bw of both CPUs is what it was before.
+ *
+ * Only then, with nothing left to change: sched_debug prints running_bw
+ * and this_bw of a CPU one after the other, holding no lock in between,
+ * and every parameter change takes a task's bandwidth off its runqueue
+ * and puts it back, so a reading of the two while the changes go on
+ * straddles one of those and tells nothing.
+ */
+static atomic_int *wk_state;		/* 0: tasks up, 1: wakeups, 2: stop */
+
+static pid_t waking_task(uint64_t runtime, uint64_t period, int sleeps)
+{
+	pid_t pid = fork_child("a waking deadline task");
+
+	if (!pid) {
+		if (enter(cg_q, getpid()) || set_deadline(0, runtime, period)) {
+			wk_state[0] = -1;
+			_exit(1);
+		}
+		wk_state[0]++;
+		for (;;) {
+			if (wk_state[2])
+				_exit(0);
+			spin_ns(sleeps ? 20 * NSEC_PER_USEC : NSEC_PER_MSEC);
+			if (sleeps) {
+				sleep_ns(80 * NSEC_PER_USEC);
+				wk_state[1]++;
+			}
+		}
+	}
+	return pid;
+}
+
+/* Changes the parameters of @pid over and over, until told to stop. */
+static pid_t changer_proc(pid_t pid)
+{
+	pid_t self = fork_child("a changer of the parameters");
+
+	if (!self) {
+		while (!wk_state[2]) {
+			if (set_deadline(pid, 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid, 10 * NSEC_PER_MSEC, PERIOD))
+				break;
+		}
+		_exit(0);
+	}
+	return self;
+}
+
+static int params_waking(void)
+{
+	int cpu_a = top_cpu(3), cpu_b = top_cpu(2), i, ret = KSFT_PASS, changes = 0;
+	long ta, ra, tb, rb, ta2, ra2, tb2, rb2;
+	pid_t pid[3] = { -1, -1, -1 }, changer = -1;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (rq_bw_of(cpu_a, &ta, &ra) || rq_bw_of(cpu_b, &tb, &rb)) {
+		remove_partition(cg_q);
+		ksft_print_msg("sched_debug tells of neither CPU's this_bw\n");
+		return KSFT_SKIP;
+	}
+	wk_state = mv_state + 8;
+	wk_state[0] = wk_state[1] = wk_state[2] = 0;
+	/* Two of 40% hold both CPUs with an earlier deadline than the third. */
+	pid[0] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[1] = waking_task(20 * NSEC_PER_MSEC, 50 * NSEC_PER_MSEC, 0);
+	pid[2] = waking_task(10 * NSEC_PER_MSEC, PERIOD, 1);
+	if (pid[0] < 0 || pid[1] < 0 || pid[2] < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	wait_for(5000, wk_state[0] >= 3 || wk_state[0] < 0);
+	if (wk_state[0] != 3) {
+		ksft_print_msg("no room for the deadline tasks of a partition of two CPUs\n");
+		ret = KSFT_SKIP;
+		goto out;
+	}
+	changer = changer_proc(pid[2]);
+	if (changer < 0) {
+		ret = KSFT_FAIL;
+		goto out;
+	}
+	end = now_ns() + WAKE_NS;
+	while (now_ns() < end) {
+		for (i = 0; i < WAKE_BURST; i++, changes += 2) {
+			if (set_deadline(pid[2], 20 * NSEC_PER_MSEC, PERIOD) ||
+			    set_deadline(pid[2], 10 * NSEC_PER_MSEC, PERIOD)) {
+				ksft_print_msg("a change of the waking task refused\n");
+				ret = KSFT_FAIL;
+				goto out;
+			}
+		}
+	}
+	if (!wk_state[1]) {
+		ksft_print_msg("the waking task never woke\n");
+		ret = KSFT_FAIL;
+	}
+out:
+	wk_state[2] = 1;
+	reap(changer);
+	for (i = 0; i < 3; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb) {
+			ksft_print_msg("%d changes: this_bw %ld and %ld, was %ld and %ld\n",
+				       changes, ta2, tb2, ta, tb);
+			ret = KSFT_FAIL;
+		}
+	}
+	remove_partition(cg_q);
+	return ret;
+}
+
+/* A random mix of the transitions above */
+
+#define MIX_WORKERS	6
+#define MIX_NS		(3 * NSEC_PER_SEC)
+#define MIX_SEED	20261103	/* fixed: a failing run takes the same mix again */
+
+/*
+ * Each worker takes a random transition of its own over and over: into
+ * SCHED_DEADLINE with random parameters and flags, out of it, asleep,
+ * spinning past its runtime, through sched_yield(), through the shared
+ * mutex of the priority inheritance cases, and out of life. The case
+ * takes the ones only another task can: new parameters, a move between
+ * the cgroups, a kill, and the fair server of a CPU. Every worker it
+ * finds dead it starts anew, so the mix keeps its tasks.
+ *
+ * Nothing here looks at what a single transition does, which the cases
+ * above do one at a time. What it wants is the accounting whole
+ * afterwards: once every worker is gone and its 0-lag time is past, each
+ * of the three root domains admits what it admitted before, and this_bw
+ * of the three CPUs is what it was. An over- or underflow of running_bw
+ * or this_bw also shows as a warning in the kernel log, which the test
+ * does not read.
+ */
+static atomic_int *mix_state;		/* 0: stop, 1 + i: worker i's rounds */
+
+static pid_t mix_worker(int idx, unsigned int seed)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	static const uint64_t period[] = { 50, 100 };
+	static const uint64_t flag[] = { 0, SCHED_FLAG_RECLAIM, SCHED_FLAG_DL_OVERRUN };
+	pid_t pid = fork_child("a worker of the mix");
+
+	if (!pid) {
+		struct sigaction sa = { .sa_handler = catch_xcpu };
+		int err;
+
+		if (sigaction(SIGXCPU, &sa, NULL))
+			_exit(1);
+		seed += idx;
+		/*
+		 * A refusal is part of the mix: each call below is made in
+		 * whatever state the worker is in, and an error is what the
+		 * kernel is entitled to answer.
+		 */
+		while (!mix_state[0]) {
+			unsigned int r = rand_r(&seed), v = r / 8;
+
+			switch (r % 8) {
+			case 0:
+				set_deadline_flags(0, runtime[v % 4] * NSEC_PER_MSEC,
+						   period[(v / 4) % 2] * NSEC_PER_MSEC,
+						   flag[(v / 8) % 3]);
+				break;
+			case 1:
+				set_other(0);
+				break;
+			case 2:
+				sleep_ns((v % 20 + 1) * NSEC_PER_MSEC);
+				break;
+			case 3:
+				spin_ns((v % 15 + 1) * NSEC_PER_MSEC);
+				break;
+			case 4:
+				sched_yield();
+				break;
+			case 5:
+				err = pthread_mutex_trylock(shm_lock);
+				if (err == EOWNERDEAD)	/* its owner died holding it */
+					err = pthread_mutex_consistent(shm_lock);
+				if (!err) {
+					spin_ns(NSEC_PER_MSEC);
+					if (pthread_mutex_unlock(shm_lock))
+						_exit(1);
+				}
+				break;
+			case 6:
+				_exit(0);	/* the case starts another */
+			default:
+				sleep_ns(NSEC_PER_MSEC / 2);
+				break;
+			}
+			mix_state[1 + idx]++;
+		}
+		_exit(0);
+	}
+	return pid;
+}
+
+static int random_mix(void)
+{
+	static const uint64_t runtime[] = { 5, 10, 20, 30 };
+	int cpu_a = top_cpu(3), cpu_b = top_cpu(2), cpu_p = top_cpu(1);
+	int base_p, base_q, base_r, i, ret = KSFT_PASS, rounds = 0, sums;
+	long ta = 0, ra, tb = 0, rb, tp = 0, rp, ta2, ra2, tb2, rb2, tp2, rp2;
+	pid_t pid[MIX_WORKERS] = { -1, -1, -1, -1, -1, -1 };
+	unsigned int seed = MIX_SEED;
+	uint64_t end;
+	char cpus[32];
+
+	if (ncpus < 4) {
+		ksft_print_msg("needs 4 CPUs\n");
+		return KSFT_SKIP;
+	}
+	snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+	if (make_partition_cpus(cg_q, cpus)) {
+		remove_partition(cg_q);
+		ksft_print_msg("no partition root of two CPUs\n");
+		return KSFT_SKIP;
+	}
+	if (shm_lock_init()) {
+		remove_partition(cg_q);
+		return KSFT_SKIP;	/* no shared mutex for the workers */
+	}
+	/* Without the two sums of sched_debug, the admissions alone tell. */
+	sums = !rq_bw_of(cpu_a, &ta, &ra) && !rq_bw_of(cpu_b, &tb, &rb) &&
+	       !rq_bw_of(cpu_p, &tp, &rp);
+	base_p = probe(cg_p);
+	base_q = probe(cg_q);
+	base_r = probe(root);
+	if (base_p <= 0 || base_q <= 0 || base_r <= 0) {
+		remove_partition(cg_q);
+		ksft_print_msg("no room to probe P, Q or the rest\n");
+		return KSFT_SKIP;
+	}
+	mix_state = mv_state + 16;
+	for (i = 0; i <= MIX_WORKERS; i++)
+		mix_state[i] = 0;
+	for (i = 0; i < MIX_WORKERS; i++) {
+		pid[i] = mix_worker(i, seed);
+		if (pid[i] < 0) {
+			ret = KSFT_FAIL;
+			goto out;
+		}
+	}
+
+	end = now_ns() + MIX_NS;
+	while (now_ns() < end) {
+		unsigned int r = rand_r(&seed);
+		int w = r % MIX_WORKERS;
+
+		switch ((r / MIX_WORKERS) % 7) {
+		case 0:
+			set_deadline(pid[w], runtime[r % 4] * NSEC_PER_MSEC, PERIOD);
+			break;
+		case 1:
+			set_other(pid[w]);
+			break;
+		case 2:
+			enter(cg_p, pid[w]);
+			break;
+		case 3:
+			enter(cg_q, pid[w]);
+			break;
+		case 4:
+			enter(root, pid[w]);
+			break;
+		case 5:
+			/* Dead and reaped already: ESRCH, and it starts anew below. */
+			if (kill(pid[w], SIGKILL) && errno != ESRCH)
+				ksft_print_msg("kill %d: %s\n", pid[w], strerror(errno));
+			break;
+		default:
+			server_set(r % 2 ? cpu_p : cpu_a,
+				   (r % 3 + 1) * SERVER_RUNTIME, SERVER_PERIOD);
+			break;
+		}
+		sleep_ns((r % 2000 + 100) * NSEC_PER_USEC);
+		/* Whoever died, of its own or killed, runs again. */
+		for (i = 0; i < MIX_WORKERS; i++) {
+			pid_t done = waitpid(pid[i], NULL, WNOHANG);
+
+			if (done < 0 && errno != EINTR) {
+				ksft_print_msg("waitpid %d: %s\n", pid[i], strerror(errno));
+				ret = KSFT_FAIL;
+				goto out;
+			}
+			if (done == pid[i]) {
+				pid[i] = mix_worker(i, seed + rounds);
+				if (pid[i] < 0) {
+					ret = KSFT_FAIL;
+					goto out;
+				}
+				rounds++;
+			}
+		}
+	}
+out:
+	mix_state[0] = 1;
+	for (i = 0; i < MIX_WORKERS; i++)
+		reap(pid[i]);
+	sleep_ns(5 * PERIOD);		/* their 0-lag times */
+	server_restore();		/* which needs the room they held */
+	if (ret != KSFT_PASS) {
+		remove_partition(cg_q);
+		return ret;
+	}
+	for (i = 0, rounds = 0; i < MIX_WORKERS; i++)
+		rounds += mix_state[1 + i];
+	if (!rounds) {
+		ksft_print_msg("the workers of the mix took no transition\n");
+		ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && sums) {
+		if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2) ||
+		    rq_bw_of(cpu_p, &tp2, &rp2)) {
+			ret = KSFT_FAIL;
+		} else if (ta2 != ta || tb2 != tb || tp2 != tp) {
+			ksft_print_msg("this_bw %ld %ld %ld, was %ld %ld %ld\n",
+				       ta2, tb2, tp2, ta, tb, tp);
+			ret = KSFT_FAIL;
+		}
+	}
+	if (ret == KSFT_PASS &&
+	    (admits_in(cg_p, base_p, "P after the mix") ||
+	     admits_in(cg_q, base_q, "Q after the mix") ||
+	     admits_in(root, base_r, "the root domain of the rest after the mix")))
+		ret = KSFT_FAIL;
+	if (ret != KSFT_PASS)
+		ksft_print_msg("seed %u, %d transitions\n", MIX_SEED, rounds);
+	remove_partition(cg_q);
+	return ret;
+}
+
+/* CPU hotplug, from the main process, with no P */
+
+static int hp_cpu = -1;
+
+/* Whether the online file of @cpu can be written, as hp_cpu's can. */
+static int cpu_hotpluggable(int cpu)
+{
+	char path[64];
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	return !access(path, W_OK);
+}
+
+static int cpu_online(int cpu, int on)
+{
+	char dir[64];
+
+	snprintf(dir, sizeof(dir), "/sys/devices/system/cpu/cpu%d", cpu);
+	return write_file(dir, "online", on ? "1" : "0");
+}
+
+/* Back online: every case that follows needs that CPU. */
+static void cpu_up(int cpu)
+{
+	int err = cpu_online(cpu, 1);
+
+	if (err)
+		ksft_print_msg("CPU %d cannot come back online: %s\n", cpu, strerror(-err));
+}
+
+/* Deadline sleepers of 30% each: offline and online a CPU meanwhile. */
+static int hotplug_sleepers(void)
+{
+	int base = probe(NULL), after, i, ret = KSFT_PASS;
+	pid_t pid[4];
+
+	for (i = 0; i < 4; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			return KSFT_FAIL;
+		}
+	}
+	if (cpu_online(hp_cpu, 0) || cpu_online(hp_cpu, 1)) {
+		ksft_print_msg("offline and online of CPU %d with deadline sleepers failed\n",
+			       hp_cpu);
+		ret = KSFT_FAIL;
+		cpu_up(hp_cpu);
+	}
+	for (i = 0; i < 4; i++)
+		reap(pid[i]);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("hotplug with sleepers: %d probe tasks admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/* The deadline tasks fill the CPUs: going offline must be refused. */
+static int hotplug_refused(void)
+{
+	int base = probe(NULL), n, i, ret = KSFT_PASS, err;
+	pid_t *pid;
+
+	if (base < 12) {
+		ksft_print_msg("needs room for 12 probe tasks\n");
+		return KSFT_SKIP;
+	}
+	/* 30% each, up to the free bandwidth but 30% to 60%: less than a CPU's. */
+	n = base / 3 - 1;
+	pid = calloc(n, sizeof(*pid));
+	if (!pid) {
+		ksft_print_msg("calloc: %s\n", strerror(errno));
+		return KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++) {
+		mv_state[0] = 0;
+		pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+		if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+			while (i >= 0)
+				reap(pid[i--]);
+			free(pid);
+			return KSFT_FAIL;
+		}
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (!err) {
+		ksft_print_msg("CPU %d went offline with its deadline bandwidth in use\n", hp_cpu);
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else if (err != -EBUSY) {
+		ksft_print_msg("offline refused with %s, not EBUSY\n", strerror(-err));
+		ret = KSFT_FAIL;
+	}
+	for (i = 0; i < n; i++)
+		reap(pid[i]);
+	free(pid);
+	sleep_ns(3 * PERIOD);
+	if (ret == KSFT_PASS && probe(NULL) != base) {
+		ksft_print_msg("a refused offline changed the free bandwidth\n");
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * A deadline task leaves the class while it runs, its 0-lag time ahead,
+ * and the CPU it is on goes offline before that time.
+ */
+static int hotplug_departed(void)
+{
+	int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+	pid_t pid;
+
+	/*
+	 * What the root domain admits with the CPU offline and no deadline
+	 * task of ours: the release of the departed task's bandwidth
+	 * happens there, and the online that follows would rebuild the
+	 * accounting over it.
+	 */
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	base_off = probe(NULL);
+	cpu_up(hp_cpu);
+	if (base <= 0 || base_off <= 0) {
+		ksft_print_msg("no room to probe with the CPU online or offline\n");
+		return KSFT_SKIP;
+	}
+	mv_state[0] = 0;
+	mv_state[1] = 0;
+	pid = mover(MV_THROTTLED, 40 * NSEC_PER_MSEC);
+	if (pid < 0 || child_up("a departing deadline task", 0)) {
+		reap(pid);
+		return KSFT_FAIL;
+	}
+	sleep_ns(PERIOD / 2);
+	/* Running and queued: its bandwidth stays until its 0-lag time. */
+	if (set_other(pid))
+		ret = KSFT_FAIL;
+	cpu = task_cpu_of(pid);
+	if (cpu <= 0 || !cpu_hotpluggable(cpu)) {
+		cpu_set_t set;
+
+		/* Out of the class now: it may be moved to the CPU we offline. */
+		cpu = hp_cpu;
+		CPU_ZERO(&set);
+		CPU_SET(cpu, &set);
+		if (sched_setaffinity(pid, sizeof(set), &set))
+			ret = KSFT_FAIL;
+	}
+	if (ret == KSFT_PASS && cpu_online(cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", cpu);
+		ret = KSFT_FAIL;
+	}
+	sleep_ns(3 * PERIOD);		/* its 0-lag time, the CPU offline */
+	if (ret == KSFT_PASS) {
+		after = probe(NULL);
+		if (after != base_off) {
+			ksft_print_msg("departed, CPU %d offline: %d admitted, %d expected\n",
+				       cpu, after, base_off);
+			ret = KSFT_FAIL;
+		}
+	}
+	cpu_up(cpu);
+	reap(pid);
+	sleep_ns(3 * PERIOD);
+	after = probe(NULL);
+	if (ret == KSFT_PASS && after != base) {
+		ksft_print_msg("departed, CPU back online: %d admitted before, %d after\n",
+			       base, after);
+		ret = KSFT_FAIL;
+	}
+	return ret;
+}
+
+/*
+ * The fair server of the CPU set to 30%: its bandwidth leaves the root
+ * domain with the CPU, as it does at 10%, and comes back with the CPU as
+ * it was set.
+ */
+static int hotplug_server(void)
+{
+	int base, off10, off30;
+
+	if (server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	base = probe(NULL);
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off10 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (server_set(hp_cpu, 3 * SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base - 2, "a fair server of 30%"))
+		return KSFT_FAIL;
+	if (cpu_online(hp_cpu, 0)) {
+		ksft_print_msg("offline of CPU %d refused, its fair server at 30%%\n", hp_cpu);
+		return KSFT_FAIL;
+	}
+	off30 = probe(NULL);
+	cpu_up(hp_cpu);
+	if (off30 != off10) {
+		ksft_print_msg("CPU %d offline: %d probe tasks, its server at 10%%, %d at 30%%\n",
+			       hp_cpu, off10, off30);
+		return KSFT_FAIL;
+	}
+	if (admits(base - 2, "a fair server of 30%, its CPU back online") ||
+	    server_set(hp_cpu, SERVER_RUNTIME, SERVER_PERIOD) ||
+	    admits(base, "the fair server back to 10%"))
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/*
+ * Going offline, a CPU takes its fair server's bandwidth along with its
+ * capacity. With the server at the admission limit less 10%, the CPU goes
+ * offline with as many probe tasks held as the root domain admits but
+ * one, and no longer with one more.
+ */
+static int hotplug_server_refused(void)
+{
+	uint64_t rt_runtime, rt_period, runtime;
+	int n, err, ret = KSFT_PASS;
+	char buf[32];
+
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) ||
+	    read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf))) {
+		ksft_print_msg("no sched_rt_runtime_us or sched_rt_period_us\n");
+		return KSFT_SKIP;
+	}
+	read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf));
+	rt_runtime = atoll(buf);
+	read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf));
+	rt_period = atoll(buf);
+	if (!rt_period || 5 * rt_runtime < rt_period) {
+		ksft_print_msg("the admission limit is below a fifth of a CPU\n");
+		return KSFT_SKIP;
+	}
+	runtime = SERVER_PERIOD * rt_runtime / rt_period - SERVER_PERIOD / 10;
+	if (server_set(hp_cpu, runtime, SERVER_PERIOD)) {
+		ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+		return KSFT_SKIP;
+	}
+	n = probe(NULL);
+	if (n < 2 || hold(n - 1)) {
+		release_held();
+		return KSFT_FAIL;
+	}
+	err = cpu_online(hp_cpu, 0);
+	if (err) {
+		ksft_print_msg("CPU %d, room for one probe task left: offline refused (%s)\n",
+			       hp_cpu, strerror(-err));
+		ret = KSFT_FAIL;
+	} else if (cpu_online(hp_cpu, 1) || hold(1)) {
+		cpu_up(hp_cpu);
+		ret = KSFT_FAIL;
+	} else {
+		err = cpu_online(hp_cpu, 0);
+		if (err != -EBUSY) {
+			ksft_print_msg("CPU %d, no room left: offline %s, not EBUSY\n",
+				       hp_cpu, err ? strerror(-err) : "accepted");
+			if (!err)
+				cpu_up(hp_cpu);
+			ret = KSFT_FAIL;
+		}
+	}
+	release_held();
+	return ret;
+}
+
+/* Setup */
+
+static int find_cgroup2(void)
+{
+	char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+	FILE *f = fopen("/proc/self/mounts", "r");
+	int found = 0;
+
+	while (f && !found && fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+		    !strcmp(type, "cgroup2")) {
+			snprintf(root, sizeof(root), "%s", dir);
+			found = 1;
+		}
+	}
+	if (f)
+		fclose(f);
+	return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+	if (ret == KSFT_PASS)
+		ksft_test_result_pass("%s\n", name);
+	else if (ret == KSFT_SKIP)
+		ksft_test_result_skip("%s\n", name);
+	else
+		ksft_test_result_fail("%s\n", name);
+}
+
+enum where {
+	MEASURED_IN_P,	/* in P, measured before and after */
+	IN_P,		/* in P, measuring itself */
+	FROM_OUTSIDE,	/* from the main process */
+};
+
+static const struct p_case {
+	const char *name;
+	int (*fn)(void);
+	enum where where;
+} p_cases[] = {
+	{ "pi_deadline_owner", pi_deadline_owner, MEASURED_IN_P },
+	{ "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
+	{ "pi_former_owner_asleep", pi_former_owner_asleep, MEASURED_IN_P },
+	{ "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
+	{ "params_asleep", params_asleep, IN_P },
+	{ "params_non_contending", params_non_contending, MEASURED_IN_P },
+	{ "params_throttled", params_throttled, MEASURED_IN_P },
+	{ "params_refused", params_refused, IN_P },
+	{ "params_keep_class_change", params_keep_class_change, MEASURED_IN_P },
+	{ "params_keep_boosted", params_keep_boosted, MEASURED_IN_P },
+	{ "boosted_owner_policy", boosted_owner_policy, MEASURED_IN_P },
+	{ "deboost_resets_period", deboost_resets_period, MEASURED_IN_P },
+	{ "deboost_released_resets_period", deboost_released_resets_period, MEASURED_IN_P },
+	{ "move_asleep", move_asleep, FROM_OUTSIDE },
+	{ "move_throttled", move_throttled, FROM_OUTSIDE },
+	{ "move_non_contending", move_non_contending, FROM_OUTSIDE },
+	{ "move_refused", move_refused, FROM_OUTSIDE },
+	{ "move_vs_rebuild", move_vs_rebuild, FROM_OUTSIDE },
+	{ "death_running", death_running, MEASURED_IN_P },
+	{ "death_switched_out", death_switched_out, MEASURED_IN_P },
+	{ "death_shrunk", death_shrunk, MEASURED_IN_P },
+	{ "death_killed_asleep", death_killed_asleep, MEASURED_IN_P },
+	{ "death_killed_non_contending", death_killed_non_contending, MEASURED_IN_P },
+	{ "death_killed_throttled", death_killed_throttled, MEASURED_IN_P },
+	{ "death_killed_running", death_killed_running, MEASURED_IN_P },
+	{ "death_pi_waiter", death_pi_waiter, MEASURED_IN_P },
+	{ "death_pi_owner", death_pi_owner, MEASURED_IN_P },
+	{ "death_pi_deadline_owner", death_pi_deadline_owner, MEASURED_IN_P },
+	{ "death_group", death_group, MEASURED_IN_P },
+	{ "death_zombie_switched", death_zombie_switched, FROM_OUTSIDE },
+	{ "yield_switched_out", yield_switched_out, IN_P },
+	{ "yield_moved", yield_moved, FROM_OUTSIDE },
+	{ "yield_killed", yield_killed, MEASURED_IN_P },
+	{ "overrun_caught", overrun_caught, IN_P },
+	{ "overrun_group_death", overrun_group_death, MEASURED_IN_P },
+	{ "reclaim_runs", reclaim_runs, MEASURED_IN_P },
+	{ "reclaim_switched", reclaim_switched, MEASURED_IN_P },
+	{ "reclaim_moved", reclaim_moved, FROM_OUTSIDE },
+	{ "server_params", server_params, IN_P },
+	{ "server_rebuilds", server_rebuilds, FROM_OUTSIDE },
+	{ "server_states", server_states, FROM_OUTSIDE },
+	{ "server_reclaim", server_reclaim, MEASURED_IN_P },
+	{ "params_waking", params_waking, FROM_OUTSIDE },
+	{ "random_mix", random_mix, FROM_OUTSIDE },
+};
+
+#define N_P_CASES	((int)ARRAY_SIZE(p_cases))
+
+static const struct hp_case {
+	const char *name;
+	int (*fn)(void);
+} hp_cases[] = {
+	{ "hotplug_sleepers", hotplug_sleepers },
+	{ "hotplug_refused", hotplug_refused },
+	{ "hotplug_departed", hotplug_departed },
+	{ "hotplug_server", hotplug_server },
+	{ "hotplug_server_refused", hotplug_server_refused },
+};
+
+#define N_HP_CASES	((int)ARRAY_SIZE(hp_cases))
+
+/* The cases in P, each in a P of its own. */
+static void run_p_cases(void)
+{
+	char path[PATH_MAX + 32], buf[256];
+	int i, ret, added = 0, told = 0, ok;
+
+	ok = !find_cgroup2() &&
+	     snprintf(cg_p, sizeof(cg_p), "%s/deadline_bw_p", root) < (int)sizeof(cg_p) &&
+	     snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) < (int)sizeof(cg_q);
+	if (ok) {
+		/* The cpuset controller for the root's children while the cases run. */
+		snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+		ok = !read_file(path, buf, sizeof(buf));
+		added = ok && !strstr(buf, "cpuset");
+		if (added && write_file(root, "cgroup.subtree_control", "+cpuset")) {
+			ok = 0;
+			added = 0;
+		}
+	}
+	for (i = 0; i < N_P_CASES; i++) {
+		const struct p_case *c = &p_cases[i];
+
+		if (ok && make_p()) {
+			remove_p();
+			ok = 0;
+		}
+		if (!ok) {
+			if (!told++)
+				ksft_print_msg("no one-CPU partition root: cases in P skipped\n");
+			ksft_test_result_skip("%s\n", c->name);
+			continue;
+		}
+		if (c->where == MEASURED_IN_P)
+			ret = measured(c->name, cg_p, c->fn);
+		else if (c->where == IN_P)
+			ret = run_in(cg_p, c->fn);
+		else
+			ret = c->fn();
+		remove_p();
+		server_restore();
+		report(c->name, ret);
+	}
+	if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+		ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+/* The online CPUs, ascending, into online: their count, or -1. */
+static int read_online(void)
+{
+	static char buf[65536];		/* a sysfs list fits in a page */
+	char *p = buf;
+	int n = 0, max = 0;
+
+	if (read_file("/sys/devices/system/cpu/online", buf, sizeof(buf)) ||
+	    strlen(buf) == sizeof(buf) - 1)
+		return -1;
+	while (*p >= '0' && *p <= '9') {
+		long lo = strtol(p, &p, 10), hi = lo;
+
+		if (*p == '-')
+			hi = strtol(p + 1, &p, 10);
+		for (; lo <= hi; lo++) {
+			if (n == max) {
+				int *more = realloc(online, (max = 2 * max + 64) * sizeof(*online));
+
+				if (!more)
+					return -1;
+				online = more;
+			}
+			online[n++] = lo;
+		}
+		if (*p == ',')
+			p++;
+	}
+	return n;
+}
+
+int main(void)
+{
+	char buf[64];
+	int i;
+
+	ksft_print_header();
+	if (geteuid())
+		ksft_exit_skip("needs root\n");
+	ncpus = read_online();
+	if (ncpus < 0)
+		ksft_exit_fail_msg("the online CPUs cannot be read\n");
+	if (ncpus < 3)
+		ksft_exit_skip("needs 3 CPUs\n");
+	if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+		ksft_exit_skip("no deadline admission control\n");
+	mv_state = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	if (mv_state == MAP_FAILED)
+		ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
+	shm_lock = (pthread_mutex_t *)((char *)mv_state + 256);
+
+	ksft_set_plan(N_P_CASES + N_HP_CASES);
+	server_init();
+	run_p_cases();
+
+	for (i = 1; i < ncpus && hp_cpu < 0; i++) {
+		if (cpu_hotpluggable(top_cpu(i)))
+			hp_cpu = top_cpu(i);
+	}
+	if (hp_cpu < 0)
+		ksft_print_msg("no CPU hotplug: hotplug cases skipped\n");
+	for (i = 0; i < N_HP_CASES; i++) {
+		if (hp_cpu < 0) {
+			ksft_test_result_skip("%s\n", hp_cases[i].name);
+			continue;
+		}
+		report(hp_cases[i].name, hp_cases[i].fn());
+		server_restore();
+	}
+	ksft_finished();
+}
diff --git a/tools/testing/selftests/sched/deadline/settings b/tools/testing/selftests/sched/deadline/settings
new file mode 100644
index 000000000000..a62d2fa1275c
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/settings
@@ -0,0 +1 @@
+timeout=600
-- 
2.47.3

Re: [PATCH v4 15/26] sched/deadline: Leave a dying task out of a rebuild of the accounting

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: premature bandwidth drop in
  dl_add_task_root_domain()
--

commit 961e96ebfc4954b76eeee2f5e097ac6c0a279fa7
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Leave a dying task out of a rebuild of the accounting

This commit skips tasks in TASK_DEAD state that are not on the departed list
when rebuilding root domain deadline bandwidth accounting in
dl_add_task_root_domain().

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 45a896a4f4615..3f3de4b74a2dc 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3643,6 +3643,15 @@ void dl_add_task_root_domain(struct task_struct *p)
>  		cpu = p->dl.dl_departed_cpu;
>  		if (!cpu_active(cpu))
>  			goto unlock;
> +	} else if (READ_ONCE(p->__state) == TASK_DEAD) {
> +		/*
> +		 * Dead and not departed: its bandwidth was released at its
> +		 * last schedule, under its runqueue lock and not this one,
> +		 * and dl_bw may still say otherwise. do_task_dead() sets
> +		 * TASK_DEAD under pi_lock before that, so a task found alive
> +		 * here dies after this walk, and releases what it counted.
> +		 */
> +		goto unlock;
>  	}

[Severity: Medium]
Can this drop the bandwidth of a dying task whose 0-lag time is still in the
future?

do_task_dead() calls set_special_state(TASK_DEAD), which sets p->__state =
TASK_DEAD under p->pi_lock and immediately releases p->pi_lock before
calling __schedule().  The task only sets p->dl.dl_departed_cpu later inside
__schedule() -> ... -> dequeue_task_dl() -> task_non_contending() ->
dl_task_depart():

CPU0                                  CPU1
do_task_dead()                        dl_rebuild_rd_accounting()
  set_special_state(TASK_DEAD)          dl_clear_root_domains()
    lock(&p->pi_lock)                     dl_bw_gen++
    p->__state = TASK_DEAD                rd->dl_bw.total_bw = 0
    unlock(&p->pi_lock)                 ...
                                        dl_add_task_root_domain(p)
                                          p->dl.dl_departed_cpu < 0
                                          p->__state == TASK_DEAD
                                          goto unlock  <-- skips p->dl.dl_bw
  __schedule(SM_NONE)
    ...
    task_non_contending(p)
      /* zerolag_time >= 0 */
      dl_task_depart(p, ...)
      start_inactive_timer(p)

Until inactive_timer fires at the 0-lag time, p's bandwidth is missing from
rd->dl_bw.total_bw, allowing sched_dl_overflow() to over-admit new deadline
tasks beyond the root domain's capacity.  In addition, because p joins
dl_departed_list with a stale p->dl.dl_bw_gen, a second rebuild before
inactive_timer fires will take the p->dl.dl_departed_cpu >= 0 branch in
dl_add_task_root_domain() and add p->dl.dl_bw back on top of any newly
admitted tasks.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=15

Re: [PATCH v4 04/26] sched: Let a task boosted by a deadline waiter change its policy

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched: dangling pi_se in __sched_setscheduler() when pi is
  false
- [Medium] sched: unconditional ENQUEUE_REPLENISH in
  __setscheduler_dl_pi()
--

commit 8fd411e98dd72e517f278200f2a112aeff758f77
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched: Let a task boosted by a deadline waiter change its policy

This commit changes __sched_setscheduler() to test task_has_dl_policy(p)
rather than dl_task(p) when deciding whether to run sched_dl_overflow(),
allowing a task boosted by a deadline waiter to change its policy.

> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index b8c036d03d6f5..bdffe5452dc8c 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -689,7 +689,9 @@ int __sched_setscheduler(struct task_struct *p,
>  	 * If setscheduling to SCHED_DEADLINE (or changing the parameters
>  	 * of a SCHED_DEADLINE task) we need to check if enough bandwidth
>  	 * is available. Last of the checks: nothing gives back the
> -	 * bandwidth it reserves.
> +	 * bandwidth it reserves. A task boosted into the class by priority
> +	 * inheritance has nothing to admit nor to leave: the policy
> +	 * decides, not the priority.
>  	 *
>  	 * Nothing to admit for parameters that are kept
>  	 * (SCHED_FLAG_KEEP_PARAMS): they are the ones the task has, and
> @@ -697,7 +699,8 @@ int __sched_setscheduler(struct task_struct *p,
>  	 * change of its own that landed in between would leave them stale.
>  	 */
>  	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS) &&
> -	    (dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
> +	    (dl_policy(policy) || task_has_dl_policy(p)) &&
> +	    sched_dl_overflow(p, policy, attr)) {
>  		retval = -EBUSY;
>  		goto unlock;
>  	}

[Severity: Medium]
Does this leave a dangling p->dl.pi_se pointer when normalize_rt_tasks()
(sysrq-n) calls __sched_setscheduler() with pi set to false on a non-deadline
task boosted by a deadline rt_mutex waiter?

With pi set to false, __sched_setscheduler() skips rt_effective_prio(), so
newprio is normal priority (120) and the task moves to fair_sched_class.
However, __setscheduler_dl_pi() only updates p->dl.pi_se when dl_prio(newprio)
is true:

kernel/sched/syscalls.c:__setscheduler_dl_pi() {
	if (dl_prio(newprio) && !dl_policy(policy)) {
		struct task_struct *pi_task = rt_mutex_get_top_task(p);

		if (pi_task) {
			p->dl.pi_se = pi_task->dl.pi_se;
			scope->flags |= ENQUEUE_REPLENISH;
		}
	}
}

switched_from_dl() does not reset p->dl.pi_se either because p->dl.dl_runtime
is 0, and when p later releases the rt_mutex, rt_mutex_setprio() sees
prio == p->prio && !dl_prio(prio) and returns early without resetting
p->dl.pi_se = &p->dl.  If the waiter later exits (or has its deadline
parameters cleared) and p subsequently enters the deadline policy,
is_dl_boosted(&p->dl) remains true and replenish_dl_entity() dereferences the
stale p->dl.pi_se pointer, causing a use-after-free or an infinite loop under
rq->__lock.

[Severity: Medium]
Can a non-deadline task boosted by a deadline waiter also bypass deadline
bandwidth enforcement by repeatedly calling sched_setscheduler() or
sched_setattr()?

With this change, a boosted non-deadline task calling __sched_setscheduler()
with pi set to true (for example, toggling between normal and batch policies)
keeps dl_prio(newprio) and reaches __setscheduler_dl_pi(), which
unconditionally sets ENQUEUE_REPLENISH on scope->flags.  When
sched_change_end() re-enqueues the task, enqueue_task_dl() clears
dl_se->dl_throttled and replenish_dl_entity() sees dl_se->dl_deadline == 0:

kernel/sched/deadline.c:replenish_dl_entity() {
	if (dl_se->dl_deadline == 0 ||
	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
		dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
		dl_se->runtime = pi_of(dl_se)->dl_runtime;
	}
}

This refills dl_se->runtime to the donor's full runtime and unthrottles the
task on every call, allowing the mutex owner to replenish its inherited budget
indefinitely and starve lower-priority tasks on that CPU.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=4

Re: [PATCH v4 09/26] sched/deadline: Reset the deadline and the runtime with the parameters

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: unreset donor period in switched_from_dl()
  when dl_runtime != 0
- [Medium] sched/deadline: donor deadline and runtime kept on deboost
  after setattr
--

commit e10fc72d0029b6302955712a86b151e84624b177
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Reset the deadline and the runtime with the parameters

This commit resets dl_se->deadline and dl_se->runtime in
__dl_clear_params() and in switched_from_dl() when a deboosted task has no
dl_runtime of its own.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index b8b0076870a64..f389620e6fc9a 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -3580,6 +3580,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  	if (task_on_rq_queued(p) && p->dl.dl_runtime)
>  		task_non_contending(&p->dl, false);
>  
> +	/*
> +	 * Deboosted from priority inheritance with no parameters of its own:
> +	 * the deadline and the runtime are the donor's. A change into the
> +	 * class then starts a period from the parameters it is given.
> +	 */
> +	if (!p->dl.dl_runtime) {
> +		p->dl.deadline = 0;
> +		p->dl.runtime = 0;
> +	}

[Severity: Medium]
What happens in switched_from_dl() when a task that still carries pre-0-lag
parameters (p->dl.dl_runtime != 0) is PI-boosted by a SCHED_DEADLINE waiter
and then deboosted?

When the task is boosted before its inactive_timer fires, switched_to_dl()
cancels inactive_timer and replenish_dl_entity() overwrites p->dl.deadline
and p->dl.runtime from the donor's pi_params.  On deboost, p->dl.dl_runtime
is non-zero, so switched_from_dl() skips the reset and passes the
donor-modified deadline and runtime to task_non_contending(), mixing them
with the task's old dl_period and dl_runtime.

Depending on the donor's parameters, this either releases the task's old
bandwidth before its real 0-lag time or leaves the donor's future deadline
and runtime in place for a subsequent entry into SCHED_DEADLINE.

[Severity: Medium]
Can a task keep the donor's deadline and runtime if sched_setattr() gives
it SCHED_DEADLINE parameters while it is PI-boosted?

If a non-deadline task is boosted into dl_sched_class and then configured
as SCHED_DEADLINE via sched_setattr() before unlocking the rt_mutex, both
__sched_setscheduler() and the subsequent deboost in rt_mutex_setprio()
keep the task in dl_sched_class without calling switched_from_dl():

kernel/sched/core.c:rt_mutex_setprio() {
    ...
		} else {
			p->dl.dl_boosted = 0;
		}
    ...
}

When rt_mutex_setprio() clears dl_boosted, enqueue_dl_entity() only calls
setup_new_dl_entity() if dl_se->deadline is already in the past, so the
task runs unboosted on the donor's remaining runtime and deadline instead
of starting a period from its own parameters.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=9

Re: [PATCH v4 14/26] sched/deadline: Rebuild the root domains' accounting from the task list

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: missing dl_bw_gen check in
  dl_add_task_root_domain()
--

commit e697aca18463b223769b3e38b19e21df9b87aad8
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Rebuild the root domains' accounting from the task list

This commit moves dl_rebuild_rd_accounting() into the deadline scheduler
to walk the task list under tasklist_lock instead of iterating over
cpusets, and adds dl_accounting_mutex to serialize bandwidth admission
against rebuilds when CONFIG_CPUSETS is disabled.

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 6b0ec20038573..45a896a4f4615 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c

[ ... ]

> @@ -3705,13 +3710,71 @@ void dl_clear_root_domain(struct root_domain *rd)
>  		dl_server_add_bw(rd, i);
>  }

[ ... ]

> +void dl_rebuild_rd_accounting(void)
> +{
> +	struct task_struct *g, *p;
> +
> +	lockdep_assert_cpus_held();
> +	lockdep_assert_held(&sched_domains_mutex);
> +	lockdep_assert_cpuset_lock_held();
> +	lockdep_assert_dl_accounting_held();
> +
> +	dl_clear_root_domains();
> +
> +	read_lock(&tasklist_lock);
> +	for_each_process_thread(g, p) {
> +		/*
> +		 * Tested without the task's lock first, to spare the others
> +		 * the locks: a task enters the class under the locks this
> +		 * rebuild holds, or by a boost, which brings it no bandwidth
> +		 * of its own to count.
> +		 */
> +		if (dl_task(p))
> +			dl_add_task_root_domain(p);
> +	}
> +	read_unlock(&tasklist_lock);
> +}

[Severity: Medium]
Can dl_add_task_root_domain() resurrect departed bandwidth that an earlier
rebuild already dropped?

Unlike css_task_iter, for_each_process_thread() visits un-reaped
EXIT_ZOMBIE tasks.  When a SCHED_DEADLINE task on CPU X exits before its
0-lag time and stays a zombie, it remains dl_task(p) with p->dl.dl_bw > 0
and p->dl.dl_departed_cpu == X on dl_departed_list until its
inactive_timer fires.

If CPU X goes offline, the first rebuild drops the zombie's bandwidth
because !cpu_active(X), leaving p->dl.dl_bw_gen behind at generation G.
If CPU X comes back online before inactive_timer expires, the second
rebuild (generation G + 2) skips the task in dl_clear_root_domains()
because dl_se->dl_bw_gen != dl_bw_gen - 1, but then visits it in the
tasklist walk and calls dl_add_task_root_domain():

kernel/sched/deadline.c:dl_add_task_root_domain() {
	...
	if (p->dl.dl_bw_gen == dl_bw_gen)
		goto unlock;
	if (p->dl.dl_departed_cpu >= 0) {
		cpu = p->dl.dl_departed_cpu;
		if (!cpu_active(cpu))
			goto unlock;
	}
	rq = cpu_rq(cpu);
	dl_b = &rq->rd->dl_bw;

	raw_spin_lock(&dl_b->lock);
	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
	raw_spin_unlock(&dl_b->lock);
	p->dl.dl_bw_gen = dl_bw_gen;
	...
}

Because p->dl.dl_bw_gen == dl_bw_gen is false (G != G + 2) and cpu_active(X)
is now true, dl_add_task_root_domain() adds p->dl.dl_bw back into
dl_b->total_bw, over-accounting total_bw until inactive_timer expires and
causing new admissions to fail with -EBUSY or pushing total_bw past the
root domain capacity if tasks were admitted while CPU X was offline.
Checking p->dl.dl_bw_gen != dl_bw_gen - 1 when p->dl.dl_departed_cpu >= 0
(matching dl_clear_root_domains()) would avoid re-adding dropped
bandwidth.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=14

Re: [PATCH v4 10/26] sched/deadline: Release the bandwidth of a task that leaves the class asleep

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: dl_zero_lag() mixes PI donor state with kept
  task params
--

commit 82ae18ee2c32ac7a69958d0da90b2ae54505ac04
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Release the bandwidth of a task that leaves the class asleep

This commit releases or arms the inactive timer for a task's SCHED_DEADLINE
bandwidth in switched_from_dl() when the task leaves the class while not
queued, and clears its deadline parameters once the bandwidth is released.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index f389620e6fc9a..cb0cf9c15108f 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -390,6 +390,22 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
>  
>  static void __dl_clear_params(struct sched_dl_entity *dl_se);
>  
> +/*
> + * The 0-lag time of @dl_se, relative to the clock of its runqueue: its
> + * utilization counts until then once it blocks or leaves the class.
> + */
> +static s64 dl_zero_lag(struct sched_dl_entity *dl_se)
> +{
> +	struct rq *rq = rq_of_dl_se(dl_se);
> +
> +	/*
> +	 * Using relative times instead of the absolute "0-lag time"
> +	 * allows to simplify the code
> +	 */
> +	return dl_se->deadline - rq_clock(rq) -
> +	       div64_long((dl_se->runtime * dl_se->dl_period), dl_se->dl_runtime);
> +}
[ ... ]
> @@ -3597,15 +3605,69 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
>  	dec_dl_tasks_cs(p);
>  
>  	if (!task_on_rq_queued(p)) {
> -		/*
> -		 * Inactive timer is armed. However, p is leaving DEADLINE and
> -		 * might migrate away from this rq while continuing to run on
> -		 * some other class. We need to remove its contribution from
> -		 * this rq running_bw now, or sub_rq_bw (below) will complain.
> -		 */
> -		if (p->dl.dl_non_contending)
> +		bool released = false;
> +
> +		if (p->dl.dl_non_contending) {
> +			/*
> +			 * Inactive timer is armed. However, p is leaving
> +			 * DEADLINE and might migrate away from this rq while
> +			 * continuing to run on some other class. We need to
> +			 * remove its contribution from this rq running_bw now,
> +			 * or sub_rq_bw (below) will complain.
> +			 *
> +			 * inactive_task_timer() will find !dl_task(p) and
> +			 * release total_bw via __dl_sub().
> +			 */
>  			sub_running_bw(&p->dl, &rq->dl);
> +		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl) &&
> +			   dl_zero_lag(&p->dl) >= 0 &&
> +			   !hrtimer_active(&p->dl.inactive_timer)) {
> +			/*
> +			 * No inactive timer armed, the 0-lag time ahead: the
> +			 * task is in flight to the CPU it wakes on, which
> +			 * cancelled the timer (migrate_task_rq_dl()), or it
> +			 * came back to the class asleep, which cancelled it,
> +			 * and leaves again. Arm the timer, as for a queued
> +			 * task: its handler releases total_bw at the 0-lag
> +			 * time, finding the task out of the class. running_bw
> +			 * is off this runqueue already, or never was on it.
> +			 */
> +			task_non_contending(&p->dl, false);
> +		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {

[Severity: Medium]
Can dl_zero_lag(&p->dl) mix a PI donor's dynamic period state with p's own
unreleased reservation parameters?

When a SCHED_DEADLINE task leaves the class before its 0-lag time (or
changes policy away from SCHED_DEADLINE while already PI-boosted), it keeps
its static parameters (dl_runtime, dl_deadline, dl_period, dl_bw) until
0-lag time.  If a SCHED_DEADLINE waiter boosts the task into dl_sched_class
before inactive_timer fires, switched_to_dl() cancels inactive_timer.

During the boost, replenish_dl_entity() skips initializing the donor's
period at first because p->dl.dl_deadline is still non-zero:

kernel/sched/deadline.c:replenish_dl_entity() {
    ...
	if (dl_se->dl_deadline == 0 ||
	    (dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
		dl_se->deadline = rq_clock(rq) + pi_deadline_of(dl_se);
		dl_se->runtime = pi_runtime_of(dl_se);
	}
    ...
}

Once p->dl.deadline and p->dl.runtime are replenished from the donor's
pi_params (on wakeup via replenish_dl_new_period() or after exhausting the
old runtime), p->dl.deadline and p->dl.runtime hold the donor's state while
p->dl.dl_period and p->dl.dl_runtime still hold p's old reservation.  When
p later blocks or is deboosted in switched_from_dl(), dl_zero_lag(&p->dl)
combines the donor's deadline and runtime with p's dl_period and
dl_runtime.  Depending on the donor's parameters, this either releases p's
dl_bw and clears its parameters before p's true 0-lag time, or sees a
positive 0-lag time after p's true 0-lag time has already passed, delaying
the bandwidth release and leaving the donor's deadline and runtime on
p->dl.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=10

Re: [PATCH v4 08/26] sched/deadline: Take a deadline boost from the waiter's rt_mutex node

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] sched/deadline: missing reset of dl_boosted in
  __sched_setscheduler()
--

commit 58101a4480d1230ff10b589ddb99ec9388dce3f3
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Take a deadline boost from the waiter's rt_mutex node

This commit records a deadline waiter's reservation parameters in its
rt_mutex_waiter node under pi_lock and copies them into the boosted owner's
sched_dl_entity so replenishment does not spin if the waiter leaves
SCHED_DEADLINE before the PI chain walk completes.

> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index c3f404cddb9e7..89c185f7d4980 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -297,10 +297,10 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
>  	 */
>  
>  	if (dl_prio(newprio) && !dl_policy(policy)) {
> -		struct task_struct *pi_task = rt_mutex_get_top_task(p);
> +		const struct sched_dl_params *params = rt_mutex_top_dl_params(p);
>  
> -		if (pi_task) {
> -			p->dl.pi_se = pi_task->dl.pi_se;
> +		if (params) {
> +			dl_boost(&p->dl, params);
>  			scope->flags |= ENQUEUE_REPLENISH;
>  		}
>  	}

[Severity: Medium]
Can __sched_setscheduler() leave p->dl.dl_boosted set when a task boosted by
a deadline waiter changes its own scheduling policy or parameters?

If a task holding an rt_mutex with a deadline waiter switches its own policy
to SCHED_DEADLINE with an earlier deadline than the waiter's, __setparam_dl()
updates p->dl.dl_runtime, dl_deadline, and dl_period without clearing
p->dl.dl_boosted, and __setscheduler_dl_pi() does nothing because
dl_policy(policy) is true.  The task continues to run on the waiter's
pi_params and skips setup_new_dl_entity() in enqueue_dl_entity() until unlock.

Similarly, if normalize_rt_tasks() resets a deadline-boosted non-deadline task
to SCHED_NORMAL with pi == false, switched_from_dl() skips __dl_clear_params()
because p->dl.dl_runtime is 0, and __setscheduler_dl_pi() does nothing because
!dl_prio(newprio).  When the task later unlocks the rt_mutex,
rt_mutex_setprio() sees prio == p->prio and !dl_prio(prio) and returns at
out_unlock without clearing p->dl.dl_boosted, leaving is_dl_boosted(&p->dl)
permanently true if the task later enters SCHED_DEADLINE.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=8

Re: [PATCH v4 26/26] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] selftests/sched: unchecked boosting_waiter() return and boost
  state
--

commit e5ace2481bd0093710ca8b43f5f8c80f64160b52
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting

This commit adds deadline_bw_test to test SCHED_DEADLINE bandwidth
accounting across priority inheritance, parameter changes, cpuset moves,
task exits, sched_yield(), runtime overruns, bandwidth reclaiming, fair
server changes, and CPU hotplug.

> diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
> new file mode 100644
> index 0000000000000..8ede5d5a350e3
> --- /dev/null
> +++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c

[ ... ]

> +/* A deadline task that blocks on the lock: it boosts its owner. */
> +static void *boosting_waiter(void *arg)
> +{
> +	return (void *)(long)wait_lock_for(300, 1);
> +}
> +
> +/*
> + * SCHED_FLAG_KEEP_PARAMS with SCHED_DEADLINE on an owner that priority
> + * inheritance has boosted into the class: the class of the request is the
> + * one the owner already has, so a class change is not what it asks for,
> + * and the policy it asks for is one the kept parameters would not give
> + * it. It is refused with EINVAL; admitted, it reserves the bandwidth of
> + * the owner's slice and nothing ever releases it.
> + */
> +static int params_keep_boosted(void)
> +{
> +	struct dl_attr slice = {
> +		.size = sizeof(slice),
> +		.sched_policy = SCHED_OTHER,
> +		.sched_runtime = 40 * NSEC_PER_MSEC,	/* what the request would keep */
> +	};
> +	struct dl_attr keep = {
> +		.size = sizeof(keep),
> +		.sched_policy = SCHED_DEADLINE,
> +		.sched_flags = SCHED_FLAG_KEEP_PARAMS,
> +		.sched_deadline = PERIOD,
> +		.sched_period = PERIOD,
> +	};
> +	int ret = KSFT_PASS, err;
> +	pthread_t o, w;
> +
> +	if (pi_init())
> +		return KSFT_SKIP;
> +	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
> +		ksft_print_msg("the owner thread could not start\n");
> +		return KSFT_FAIL;
> +	}
> +	if (wait_for(5000, owner_holds) || owner_holds != 2) {
> +		ksft_print_msg("the owner never took the lock\n");
> +		join(o);
> +		return KSFT_FAIL;
> +	}
> +	if (syscall(SYS_sched_setattr, owner_tid, &slice, 0)) {
> +		ksft_print_msg("the owner cannot take a slice of its own: %s\n",
> +			       strerror(errno));
> +		ret = KSFT_SKIP;
> +		goto out;
> +	}
> +	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
> +		ksft_print_msg("the waiter thread could not start\n");
> +		ret = KSFT_FAIL;
> +		goto out;
> +	}
> +	sleep_ns(50 * NSEC_PER_MSEC);		/* it blocks, the owner is boosted */
> +	err = syscall(SYS_sched_setattr, owner_tid, &keep, 0) ? -errno : 0;
> +	if (err != -EINVAL) {
> +		ksft_print_msg("KEEP_PARAMS to SCHED_DEADLINE on a boosted owner: %s, not EINVAL\n",
> +			       err ? strerror(-err) : "accepted");
> +		ret = KSFT_FAIL;
> +	}
> +	join(w);
> +out:
> +	join(o);
> +	return ret;
> +}
> +
> +/* Boosted by a deadline waiter, the owner may still change its policy. */
> +static int boosted_owner_policy(void)
> +{
> +	int ret = KSFT_PASS, err;
> +	pthread_t o, w;
> +
> +	if (pi_init())
> +		return KSFT_SKIP;
> +	if (pthread_create(&o, NULL, sleeping_owner, (void *)0L)) {
> +		ksft_print_msg("the owner thread could not start\n");
> +		return KSFT_FAIL;
> +	}
> +	if (wait_for(5000, owner_holds) || owner_holds != 2) {
> +		ksft_print_msg("the owner never took the lock\n");
> +		join(o);
> +		return KSFT_FAIL;
> +	}
> +	if (pthread_create(&w, NULL, boosting_waiter, NULL)) {
> +		ksft_print_msg("the waiter thread could not start\n");
> +		join(o);
> +		return KSFT_FAIL;
> +	}
> +	sleep_ns(50 * NSEC_PER_MSEC);		/* it blocks, the owner is boosted */
> +	err = set_other(owner_tid);
> +	if (err) {
> +		ksft_print_msg("SCHED_OTHER on a boosted owner: %s\n", strerror(-err));
> +		ret = KSFT_FAIL;
> +	}
> +	join(w);
> +	join(o);
> +	return ret;
> +}

[Severity: Low]
Can params_keep_boosted() and boosted_owner_policy() report KSFT_PASS
even when owner_tid was never PI-boosted?

boosting_waiter() returns the status of wait_lock_for(), which is
non-zero if set_deadline() or pthread_mutex_timedlock() fails, but join()
calls pthread_join(t, NULL) and discards that return value.  Neither test
checks prio_of(owner_tid) == DL_TASK_PRIO before calling sched_setattr()
either, so if wait_lock_for() fails or w is delayed past the 50ms sleep,
both tests exercise an unboosted SCHED_OTHER owner and still pass.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=26

Re: [PATCH v4 16/26] sched/deadline: Record the root domain holding a task's bandwidth

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] sched/deadline: dl_task_undepart() loses departed state on PI
  boost
- [Medium] sched/deadline: stale dl_bw_cpu when dl_bw is zero misroutes
  bandwidth
--

commit 750b37778a0ba278821539fc3ca7bca85e0c4a0a
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

sched/deadline: Record the root domain holding a task's bandwidth

This commit records a CPU from the root domain accounting for a deadline
task's bandwidth in sched_dl_entity::dl_bw_cpu so that subsequent moves,
parameter updates, rebuilds, and releases operate on the root domain that
took the bandwidth.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/

> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 3f3de4b74a2dc..7a7101e6f7a03 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c

[ ... ]

> @@ -3537,10 +3575,18 @@ static void set_cpus_allowed_dl(struct task_struct *p,
>  bool dl_task_needs_bw_move(struct task_struct *p,
>  			   const struct cpumask *new_mask)
>  {
> -	if (!dl_task(p))
> +	unsigned int cpu;
> +
> +	/*
> +	 * Nothing to move for a special entity, nor for a task boosted into
> +	 * the class with no bandwidth of its own: neither is counted, as
> +	 * dl_add_task_root_domain() leaves them out.
> +	 */
> +	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
>  		return false;
>  
> -	return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
> +	cpu = dl_task_root_domain_cpu(p);
> +	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
>  }
>  
>  /* Assumes rq->lock is held */
> @@ -3614,8 +3660,9 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
>  	/*
>  	 * Nothing to count for a special entity, nor for a task with no
> -	 * bandwidth, such as an exited one whose bandwidth went with its
> -	 * parameters, which the walk of the task list finds too.
> +	 * bandwidth: an exited one whose bandwidth went with its
> +	 * parameters, which the walk of the task list finds too, and a task
> +	 * boosted into the class with none of its own, which gets no record.
>  	 */
>  	if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
>  		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
> @@ -3639,10 +3686,12 @@ void dl_add_task_root_domain(struct task_struct *p)
>  	 */
>  	if (p->dl.dl_bw_gen == dl_bw_gen)
>  		goto unlock;
> -	if (p->dl.dl_departed_cpu >= 0) {
> -		cpu = p->dl.dl_departed_cpu;
> -		if (!cpu_active(cpu))
> +	if (!list_empty(&p->dl.dl_departed_node)) {
> +		int recorded = p->dl.dl_bw_cpu;
> +
> +		if (recorded < 0 || !cpu_active(recorded))
>  			goto unlock;
> +		cpu = recorded;
>  	} else if (READ_ONCE(p->__state) == TASK_DEAD) {

[ ... ]

> @@ -3942,13 +3992,10 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
>  	/*
>  	 * Boosted back into the class before its 0-lag time (a policy change
>  	 * released that bandwidth in sched_dl_overflow()): the inactive
> -	 * timer will not remove it, but its leaving the class again. That
> -	 * forgets the CPU recorded: moved to another root domain meanwhile,
> -	 * the task then removes the bandwidth from that one, as it did
> -	 * before departures were recorded.
> +	 * timer will not remove it, but its leaving the class again, from
> +	 * the root domain of the CPU recorded, which it keeps.
>  	 */
> -	if (p->dl.dl_departed_cpu >= 0)
> -		dl_task_undepart(p);
> +	dl_task_undepart(p);

[Severity: Medium]
When a task that left SCHED_DEADLINE before its 0-lag time is boosted back
into dl_sched_class via priority inheritance, does calling
dl_task_undepart(p) lose the fact that its bandwidth is departing?

Unlinking p->dl.dl_departed_node while !task_has_dl_policy(p) makes
list_empty(&p->dl.dl_departed_node) true while dl_task(p) and p->dl.dl_bw
are both non-zero.  If the task's affinity or cpuset changed to another
root domain after it left SCHED_DEADLINE, a root-domain rebuild in
dl_add_task_root_domain() will skip the !list_empty(&p->dl.dl_departed_node)
branch, charge the departing bandwidth to the new root domain that never
admitted it, and overwrite p->dl.dl_bw_cpu.

For the same reason, dl_task_needs_bw_move() checks !dl_task(p) rather
than !task_has_dl_policy(p), so migrating that PI-boosted task across
exclusive cpusets moves its departing bandwidth into the destination root
domain.  If a prior rebuild had already dropped the departing bandwidth,
dl_add_task_root_domain() also puts it back because it does not check
dl_bw_gen == dl_bw_gen - 1 as dl_clear_root_domains() does.  And because
rt_mutex_setprio() does not serialize against dl_rebuild_rd_accounting(),
a departed task that is PI-boosted when dl_clear_root_domains() runs and
deboosted before the task-list walk reaches it is missed by both, dropping
its active 0-lag bandwidth from the root domain.

[ ... ]

> @@ -4155,9 +4202,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	u64 period = attr->sched_period ?: attr->sched_deadline;
>  	u64 runtime = attr->sched_runtime;
>  	u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
> -	int cpus, err = -1, cpu = task_cpu(p);
> -	struct dl_bw *dl_b = dl_bw_of(cpu);
> +	int cpus, err = -1, cpu;
> +	struct dl_bw *dl_b;
>  	unsigned long cap;
> +	bool counted;
>  
>  	if (attr->sched_flags & SCHED_FLAG_SUGOV)
>  		return 0;
> @@ -4166,6 +4214,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
>  		return 0;
>  
> +	/*
> +	 * The bandwidth of a task of the class is counted in the root
> +	 * domain that took it, which the CPU the task is on does not tell:
> +	 * moved asleep, it has not run there yet. Change it where it is
> +	 * counted; a rebuild that dropped it, and a task that enters the
> +	 * class, have the root domain of that CPU take the new one. The
> +	 * caller holds cpuset_mutex, or the lock of dl_accounting_lock()
> +	 * without cpusets, so no rebuild runs meanwhile.
> +	 */
> +	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
> +		  p->dl.dl_bw_gen == dl_bw_gen;
> +	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
> +	dl_b = dl_bw_of(cpu);

[ ... ]

> @@ -4209,11 +4273,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
>  	 * run meanwhile: the caller holds cpuset_mutex, or the lock of
>  	 * dl_accounting_lock() without cpusets.
>  	 */
> -	if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
> -		lockdep_assert_cpuset_lock_held();
> -		lockdep_assert_dl_accounting_held();
> -		if (p->dl.dl_departed_cpu >= 0)
> -			dl_task_release_bw(p, -1);
> +	if (!err && dl_policy(policy)) {
> +		if (!task_has_dl_policy(p)) {
> +			lockdep_assert_cpuset_lock_held();
> +			lockdep_assert_dl_accounting_held();
> +			/*
> +			 * Whatever bandwidth the task still has counted
> +			 * somewhere, the new one replaces: on the departed
> +			 * list, or taken off it by a boost into the class
> +			 * (switched_to_dl()), the record says where.
> +			 */
> +			if (p->dl.dl_bw_cpu >= 0)
> +				dl_task_release_bw(p);
> +		}
> +		p->dl.dl_bw_cpu = cpu;
> +		p->dl.dl_bw_gen = dl_bw_gen;
>  	}

[Severity: Medium]
Can recording p->dl.dl_bw_cpu when new_bw is 0 cause a later sched_setattr()
to charge the task's bandwidth to a stale root domain?

__checkparam_dl() permits parameters where (runtime << BW_SHIFT) < period
(for example, runtime = 1024 ns and period = 2 s), for which to_ratio()
returns new_bw == 0.  sched_dl_overflow() (and dl_task_offline_migration())
still sets p->dl.dl_bw_cpu to a CPU in the current root domain RD0.  If
the task is then moved to another exclusive cpuset partition RD1,
dl_task_needs_bw_move() returns false because !p->dl.dl_bw, leaving
p->dl.dl_bw_cpu pointing at RD0.  When the task runs in RD1 and calls
sched_setattr() with new_bw > 0, sched_dl_overflow() sees counted == true
and validates and adds new_bw to RD0's dl_bw instead of RD1's, bypassing
admission control in RD1.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107014451.1487825-1-mathieu.desnoyers@efficios.com?part=16