[PATCH v3 08/24] sched/deadline: Release the bandwidth of a task that leaves the class asleep

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

A task that leaves SCHED_DEADLINE while it is not queued, asleep or
stopped, keeps its bandwidth in the root domain's total_bw for good:

 - switched_from_dl() hands the release to task_non_contending() for a
   queued task alone;
 - for one that is not queued, it takes the bandwidth off the
   runqueue's running_bw and this_bw, and nothing takes it off
   total_bw;
 - the inactive timer is the one other path that does, and it is not
   armed: the task never ran as SCHED_DEADLINE, or the timer fired
   while it still was, taking running_bw alone.

Repeated, it fills total_bw up to the admission cap, after which
sched_setattr(SCHED_DEADLINE) and the offlining of a CPU fail with
EBUSY.

Release it in switched_from_dl() where the task is not queued and no
inactive timer is pending, for a task with parameters of its own, not
special: a task deboosted from priority inheritance with no parameters
of its own, and a special one (SCHED_FLAG_SUGOV), have no bandwidth in
total_bw. Then clear its bandwidth, once sub_rq_bw(), which reads it, is
done: every path that would release it again takes a dl_bw of 0. Kept,
it would make the task one with bandwidth to task_contending() and
task_non_contending() once priority inheritance boosts it into the
class, and a deboost while it sleeps would release it a second time. So
would the inactive timer's handler when it runs late, dl_non_contending
cleared under it by migrate_task_rq_dl(): it finds the task out of the
class. A test of hrtimer_active() would not tell that handler apart: it
stays true after the handler returns, until the hrtimer code clears its
running state, and the bandwidth of a task switched out meanwhile would
never be released.

With the bandwidth goes the state of the period the task was in, as
where inactive_task_timer() resets it: its 0-lag time is past, a return
to the class starts a period anew, and a throttle left behind would hold
a returning task until the old timer fires. The parameters themselves,
dl_runtime, dl_deadline and dl_period, stay, because the task may still
be the donor of a priority inheritance boost: rt_mutex_adjust_pi() walks
the chain and deboosts the owner after the runqueue lock of the policy
change is dropped, and until it does, a replenishment of that owner,
which update_curr_dl_se() does at once for a boosted entity, would run
replenish_dl_entity() on a donor whose dl_runtime and dl_period are 0.
It would warn and then never leave

	while (dl_se->runtime <= 0) {
		dl_se->deadline += pi_of(dl_se)->dl_period;
		dl_se->runtime += pi_of(dl_se)->dl_runtime;
	}

with the runqueue lock held and interrupts off. __setparam_dl() sets
the parameters anew if the task comes back, and the inactive timer of a
boost into the class resets them if it does not.

The leak was reported by Tejun Heo, and first fixed by Juri Lelli's
"sched/deadline: Fix total_bw leak when switching away from
SCHED_DEADLINE while !queued", since dropped from tip, which released
for every task on this path and kept the bandwidth: the double release
of a boosted task is the pi_former_owner_asleep case of the
deadline_bw_test selftest added later in this series, in a one-CPU
partition with room, the waiter of its second round refused as a
deadline task, total_bw having wrapped.

The donor window was found by the Sashiko AI review of the first
posting of this series, and shown with a kernel carrying mdelay(20)
before rt_mutex_adjust_pi() to widen it: a deadline task blocks on a
PTHREAD_PRIO_INHERIT mutex with its runtime unspent, and is switched to
SCHED_OTHER while the owner it boosts spins. Clearing the parameters
there, that kernel warns in replenish_dl_entity() and stops, on round 4
and round 3 of two 8-CPU VMs; keeping them, both go through their eight
rounds.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 86 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 73 insertions(+), 13 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 088e84fb9b10..867111a0822e 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -350,6 +350,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
 }
 
+static void __dl_clear_state(struct sched_dl_entity *dl_se);
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
 
 /*
@@ -3550,15 +3551,62 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	dec_dl_tasks_cs(p);
 
 	if (!task_on_rq_queued(p)) {
-		/*
-		 * Inactive timer is armed. However, p is leaving DEADLINE and
-		 * might migrate away from this rq while continuing to run on
-		 * some other class. We need to remove its contribution from
-		 * this rq running_bw now, or sub_rq_bw (below) will complain.
-		 */
-		if (p->dl.dl_non_contending)
+		bool released = false;
+
+		if (p->dl.dl_non_contending) {
+			/*
+			 * Inactive timer is armed. However, p is leaving
+			 * DEADLINE and might migrate away from this rq while
+			 * continuing to run on some other class. We need to
+			 * remove its contribution from this rq running_bw now,
+			 * or sub_rq_bw (below) will complain.
+			 *
+			 * inactive_task_timer() will find !dl_task(p) and
+			 * release total_bw via __dl_sub().
+			 */
 			sub_running_bw(&p->dl, &rq->dl);
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
+			/*
+			 * No pending inactive timer: the task never ran as
+			 * DEADLINE or the timer already fired while still
+			 * DEADLINE (only dropping running_bw at that time).
+			 * Nobody else will release total_bw, do it now.
+			 *
+			 * Not for a task without parameters of its own, one
+			 * deboosted from priority inheritance, nor for a
+			 * special one: neither has bandwidth in total_bw.
+			 */
+			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+			raw_spin_unlock(&dl_b->lock);
+			released = true;
+		}
 		sub_rq_bw(&p->dl, &rq->dl);
+		/*
+		 * Released: its 0-lag time is past, and the state of the
+		 * period it was in goes with the bandwidth, as where
+		 * inactive_task_timer() resets it. Its bandwidth is no one's
+		 * to release again, and a dl_bw of 0 is what every path that
+		 * would takes. Kept, it would make the task one with
+		 * bandwidth to task_contending() and task_non_contending()
+		 * once priority inheritance boosts it, and a deboost while
+		 * it sleeps would release it a second time. So would the
+		 * inactive timer's handler when it runs late,
+		 * dl_non_contending cleared under it by
+		 * migrate_task_rq_dl(): it finds !dl_task(p).
+		 *
+		 * The parameters themselves stay, where
+		 * inactive_task_timer() resets them with the bandwidth: the
+		 * task may still be the donor of a priority inheritance
+		 * boost until rt_mutex_adjust_pi() walks the chain, outside
+		 * this runqueue lock, and replenish_dl_entity() of the
+		 * boosted owner would never leave its loop over a dl_period
+		 * and a dl_runtime of 0.
+		 */
+		if (released)
+			__dl_clear_state(&p->dl);
 	}
 
 	/*
@@ -3960,22 +4008,34 @@ bool __checkparam_dl(const struct sched_attr *attr)
 /*
  * This function clears the sched_dl_entity static params.
  */
-static void __dl_clear_params(struct sched_dl_entity *dl_se)
+/*
+ * The bandwidth of @dl_se and the state of the period it is in: what a
+ * task that left the class past its 0-lag time has no use for. The
+ * parameters stay, where a boosted owner may still read them.
+ */
+static void __dl_clear_state(struct sched_dl_entity *dl_se)
 {
-	dl_se->dl_runtime		= 0;
-	dl_se->dl_deadline		= 0;
-	dl_se->dl_period		= 0;
 	dl_se->flags			= 0;
 	dl_se->dl_bw			= 0;
-	dl_se->dl_density		= 0;
 
 	dl_se->deadline			= 0;
 	dl_se->runtime			= 0;
 
 	dl_se->dl_throttled		= 0;
 	dl_se->dl_yielded		= 0;
-	dl_se->dl_non_contending	= 0;
 	dl_se->dl_overrun		= 0;
+}
+
+static void __dl_clear_params(struct sched_dl_entity *dl_se)
+{
+	__dl_clear_state(dl_se);
+
+	dl_se->dl_runtime		= 0;
+	dl_se->dl_deadline		= 0;
+	dl_se->dl_period		= 0;
+	dl_se->dl_density		= 0;
+
+	dl_se->dl_non_contending	= 0;
 	dl_se->dl_server		= 0;
 	dl_se->dl_defer			= 0;
 	dl_se->dl_defer_running		= 0;
-- 
2.47.3