[PATCH v4 22/26] sched/deadline: Count the bandwidth a boosted task is given

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

sched_setattr(SCHED_DEADLINE) on a task that priority inheritance
boosted into the class leaves the this_bw of its runqueue without the
task's new bandwidth: switched_to_dl(), which hands a sleeping task's
bandwidth to this_bw as it enters the class, does not run for a task in
it already. Its wake-up gives that bandwidth to running_bw alone:

	WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);

and the two sums GRUB reclaims from disagree by that bandwidth until the
task migrates, where the enqueue adds it (add_rq_bw()).

Change the bandwidth the runqueue counts in __setparam_dl(), which every
task that gains deadline parameters goes through, rather than in
sched_dl_overflow(), which only a task of the class reaches that way. A
runqueue that counts nothing of the task takes the new bandwidth now,
where "sched/deadline: Do not take a waking task's bandwidth off a
runqueue it left" left it to the next enqueue: that enqueue finds it
counted and adds nothing, and a waking task's task_cpu() is the runqueue
it wakes on already. The bandwidth of a special entity stays counted
nowhere, the request telling one, since the entity's flags are those of
its old parameters: a sugov kthread given a reservation of its own gets
that one counted.

A reservation with the bandwidth the kthread had needs more:
sched_dl_overflow() returns at once for a task of the class whose
bandwidth does not change, before asking whether that bandwidth is
counted, which a sugov kthread's is not, nor that of a task a rebuild
left out. Ask first, and admit and record bandwidth that is not counted
as new. A runqueue that counts the bandwidth a change keeps has nothing
to change: dl_rq_change_utilization() would take a sleeping task's
running_bw off before its 0-lag time and cancel its inactive timer for
nothing. The early return was found by the Sashiko AI review of the
third posting of this series.

Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 55 ++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a0b05ff8c6da..ac01a0023c8f 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -478,22 +478,36 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+/*
+ * The runqueue a sleeping task is on counts the bandwidth the task has:
+ * give it @new_bw in place of what it counts of the task's.
+ */
 static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 {
-	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+	struct sched_dl_entity *dl_se = &p->dl;
+	struct rq *rq = task_rq(p);
 
 	if (task_on_rq_queued(p))
 		return;
 
 	/*
-	 * Waking up on another CPU, the task has its bandwidth on no
-	 * runqueue: the enqueue to come charges the new one, with the new
-	 * parameters, which __setparam_dl() sets right after this.
+	 * No runqueue counts bandwidth of the task's: this one takes the
+	 * new one, and the enqueue to come leaves it there (add_rq_bw() of
+	 * a task a runqueue counts). A task waking up on another CPU has
+	 * none of its bandwidth on a runqueue, and task_cpu() points at the
+	 * one it wakes on.
 	 */
-	if (!p->dl.dl_bw_oncpu)
+	if (!dl_se->dl_bw_oncpu) {
+		__add_rq_bw(new_bw, &rq->dl);
+		dl_se->dl_bw_oncpu = 1;
+		return;
+	}
+
+	/* Counted with the bandwidth it keeps: nothing changes. */
+	if (new_bw == dl_se->dl_bw)
 		return;
 
-	dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+	dl_rq_change_utilization(rq, dl_se, new_bw);
 }
 
 static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4274,10 +4288,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return 0;
 
-	/* !deadline task may carry old deadline bandwidth */
-	if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
-		return 0;
-
 	/*
 	 * The bandwidth of a task of the class is counted in the root
 	 * domain that took it, which the CPU the task is on does not tell:
@@ -4289,6 +4299,16 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 	 */
 	counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
 		  p->dl.dl_bw_gen == dl_bw_gen;
+
+	/*
+	 * !deadline task may carry old deadline bandwidth. The same
+	 * bandwidth, counted, is nothing to admit; not counted, a special
+	 * entity given a reservation of its own, or a task a rebuild left
+	 * out, it is admitted as new below.
+	 */
+	if (new_bw == p->dl.dl_bw && counted)
+		return 0;
+
 	cpu = counted ? p->dl.dl_bw_cpu : task_cpu(p);
 	dl_b = dl_bw_of(cpu);
 
@@ -4318,7 +4338,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 		if (counted)
 			__dl_sub(dl_b, p->dl.dl_bw, cpus);
 		__dl_add(dl_b, new_bw, cpus);
-		dl_change_utilization(p, new_bw);
 		err = 0;
 	} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
 		/*
@@ -4368,6 +4387,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
 void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
+	u64 period = attr->sched_period ?: attr->sched_deadline;
+
+	/*
+	 * The bandwidth of a special entity is counted on no runqueue, and
+	 * the request tells one, the flags of the entity being those of the
+	 * parameters it had. The rest have theirs changed here, where their
+	 * parameters are:
+	 * switched_to_dl() hands it to the runqueue of a task that enters
+	 * the class, and a task that priority inheritance boosted into the
+	 * class is in it already, so nothing dequeues it and enqueues it
+	 * back for that.
+	 */
+	if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+		dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
 
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
-- 
2.47.3