[PATCH v4 10/26] sched/deadline: Release the bandwidth of a task that leaves the class asleep

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

A task that leaves SCHED_DEADLINE while it is not queued, asleep or
stopped, keeps its bandwidth in the root domain's total_bw for good.
Repeated, it fills total_bw up to the admission cap, after which
sched_setattr(SCHED_DEADLINE) and the offlining of a CPU fail with
EBUSY. switched_from_dl() releases the bandwidth of a queued task alone,
through task_non_contending(), and the inactive timer, the one other
path, is not armed: the task never ran in the class, or the timer fired
while it still was, taking running_bw alone.

If the 0-lag time is still ahead, arm the inactive timer, to release the
bandwidth then: a task migrating to the CPU it wakes on, whose timer
migrate_task_rq_dl() cancels, and one that came back to the class asleep
can have it ahead with no timer armed. Past it, release the bandwidth in
switched_from_dl(), for a task with parameters of its own that is not
special, and clear its dl_bw, so that no later path, a deboost or a late
inactive timer, releases it again. Clear its parameters and its period
with it, as inactive_task_timer() does; a boost the task still gives
reads none of them since "sched/deadline: Take a deadline boost from the
waiter's rt_mutex node".

Reported by Tejun Heo. Juri Lelli's "sched/deadline: Fix total_bw leak
when switching away from SCHED_DEADLINE while !queued", since dropped
from tip, released on this path for every task without clearing dl_bw,
and released the bandwidth of a boosted task twice. The deadline_bw_test
selftest added later in this series has the cases,
pi_former_owner_asleep and deboost_released_resets_period; the Sashiko
AI review of the third posting of this series found the two states with
the 0-lag time ahead.

Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-2-dd761fe005b4@redhat.com/
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/all/2b5fb3f22bca6aa383be8f4781923392@kernel.org/
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 97 +++++++++++++++++++++++++++++++++--------
 1 file changed, 80 insertions(+), 17 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index f389620e6fc9..cb0cf9c15108 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -390,6 +390,22 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
 
+/*
+ * The 0-lag time of @dl_se, relative to the clock of its runqueue: its
+ * utilization counts until then once it blocks or leaves the class.
+ */
+static s64 dl_zero_lag(struct sched_dl_entity *dl_se)
+{
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	/*
+	 * Using relative times instead of the absolute "0-lag time"
+	 * allows to simplify the code
+	 */
+	return dl_se->deadline - rq_clock(rq) -
+	       div64_long((dl_se->runtime * dl_se->dl_period), dl_se->dl_runtime);
+}
+
 /*
  * The utilization of a task cannot be immediately removed from
  * the rq active utilization (running_bw) when the task blocks.
@@ -463,15 +479,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
 
 	WARN_ON(dl_se->dl_non_contending);
 
-	zerolag_time = dl_se->deadline -
-		 div64_long((dl_se->runtime * dl_se->dl_period),
-			dl_se->dl_runtime);
-
-	/*
-	 * Using relative times instead of the absolute "0-lag time"
-	 * allows to simplify the code
-	 */
-	zerolag_time -= rq_clock(rq);
+	zerolag_time = dl_zero_lag(dl_se);
 
 	/*
 	 * If the "0-lag time" already passed, decrease the active
@@ -3597,15 +3605,69 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	dec_dl_tasks_cs(p);
 
 	if (!task_on_rq_queued(p)) {
-		/*
-		 * Inactive timer is armed. However, p is leaving DEADLINE and
-		 * might migrate away from this rq while continuing to run on
-		 * some other class. We need to remove its contribution from
-		 * this rq running_bw now, or sub_rq_bw (below) will complain.
-		 */
-		if (p->dl.dl_non_contending)
+		bool released = false;
+
+		if (p->dl.dl_non_contending) {
+			/*
+			 * Inactive timer is armed. However, p is leaving
+			 * DEADLINE and might migrate away from this rq while
+			 * continuing to run on some other class. We need to
+			 * remove its contribution from this rq running_bw now,
+			 * or sub_rq_bw (below) will complain.
+			 *
+			 * inactive_task_timer() will find !dl_task(p) and
+			 * release total_bw via __dl_sub().
+			 */
 			sub_running_bw(&p->dl, &rq->dl);
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl) &&
+			   dl_zero_lag(&p->dl) >= 0 &&
+			   !hrtimer_active(&p->dl.inactive_timer)) {
+			/*
+			 * No inactive timer armed, the 0-lag time ahead: the
+			 * task is in flight to the CPU it wakes on, which
+			 * cancelled the timer (migrate_task_rq_dl()), or it
+			 * came back to the class asleep, which cancelled it,
+			 * and leaves again. Arm the timer, as for a queued
+			 * task: its handler releases total_bw at the 0-lag
+			 * time, finding the task out of the class. running_bw
+			 * is off this runqueue already, or never was on it.
+			 */
+			task_non_contending(&p->dl, false);
+		} else if (p->dl.dl_runtime && !dl_entity_is_special(&p->dl)) {
+			/*
+			 * No inactive timer armed, the 0-lag time past: the
+			 * task never ran as DEADLINE, or the timer fired while
+			 * it still was (only dropping running_bw then). Or the
+			 * handler of a timer that fired runs late: it finds
+			 * nothing left to release.
+			 * Nobody else will release total_bw, do it now.
+			 *
+			 * Not for a task without parameters of its own, one
+			 * deboosted from priority inheritance, nor for a
+			 * special one: neither has bandwidth in total_bw.
+			 */
+			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+			raw_spin_unlock(&dl_b->lock);
+			released = true;
+		}
 		sub_rq_bw(&p->dl, &rq->dl);
+		/*
+		 * Released: its 0-lag time is past, and its parameters go with
+		 * the bandwidth, as where inactive_task_timer() resets them.
+		 * Its bandwidth is no one's to release again, and a dl_bw of 0
+		 * is what every path that would takes. Kept, it would make the
+		 * task one with bandwidth to task_contending() and
+		 * task_non_contending() once priority inheritance boosts it,
+		 * and a deboost while it sleeps would release it a second time.
+		 * So would the inactive timer's handler when it runs late,
+		 * dl_non_contending cleared under it by migrate_task_rq_dl():
+		 * it finds !dl_task(p).
+		 */
+		if (released)
+			__dl_clear_params(&p->dl);
 	}
 
 	/*
@@ -4005,7 +4067,8 @@ bool __checkparam_dl(const struct sched_attr *attr)
 }
 
 /*
- * This function clears the sched_dl_entity static params.
+ * This function clears the sched_dl_entity static params, and the state
+ * of the period the entity is in.
  */
 static void __dl_clear_params(struct sched_dl_entity *dl_se)
 {
-- 
2.47.3