[PATCH v5 23/29] sched/deadline: Move a task's bandwidth when it must leave its CPU

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

When a CPU goes offline, a deadline task asleep on it moves at its
wakeup. select_fallback_rq() picks the new CPU. balance_push() moves a
queued task the same way. The new CPU can be in another root domain.
With isolcpus="domain", the isolated CPUs are in the default root
domain, and the housekeeping CPUs in another one. A cpuset that is not
a partition can hold both.

Commit 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all
levels after offline migration") fixed this for a throttled task.
dl_task_offline_migration() moves its bandwidth with it. A task that
select_fallback_rq() places still moves no bandwidth.

The rebuild of the accounting at the offline runs before the wakeup.
So the task runs in a root domain that does not count its bandwidth,
until the next rebuild. That root domain can admit deadline tasks
beyond its capacity.

Fix this in migrate_task_rq_dl(). When a task leaves a CPU that is
inactive or not in its mask, move its bandwidth to the root domain of
the new CPU. Record that CPU. As dl_task_offline_migration() does, add
the bandwidth even without room, since the task runs there. Add it also
if a rebuild left the task out.

This also covers a change of the mask that made no room for the task,
such as sched_setaffinity() without admission control. The bandwidth
now moves when the task leaves its CPU. Before, it stayed in the old
root domain until the next rebuild.

migrate_task_rq_dl() runs under the task's pi_lock for a wakeup, and
under its rq lock otherwise. With dl_departed_lock, either keeps the
record stable.

The hotplug_fallback cases of deadline_bw_test, added later in this
series, fail without this patch. A task woken on an isolated CPU stays
counted in the housekeeping root domain. This follows the Sashiko AI
review of v4.

Fixes: 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all levels after offline migration")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/deadline.c | 58 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 51 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index bc31045e4686..a35609a10ec5 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2873,11 +2873,54 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
 	return cpu;
 }
 
-static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
+/*
+ * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
+ * also if a rebuild left @p out. Add it even without room, since @p runs
+ * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
+ */
+static void dl_task_move_bw(struct task_struct *p, int new_cpu)
+{
+	struct dl_bw *dl_b;
+	int cpu;
+
+	if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
+		return;
+
+	guard(raw_spinlock)(&dl_departed_lock);
+	cpu = p->dl.dl_bw_cpu;
+	if (cpu < 0)
+		return;
+	if (p->dl.dl_bw_gen == dl_bw_gen) {
+		if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
+			return;
+		dl_b = dl_bw_of(cpu);
+		raw_spin_lock(&dl_b->lock);
+		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
+		raw_spin_unlock(&dl_b->lock);
+	}
+	dl_b = dl_bw_of(new_cpu);
+	raw_spin_lock(&dl_b->lock);
+	__dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
+	raw_spin_unlock(&dl_b->lock);
+	p->dl.dl_bw_cpu = new_cpu;
+	p->dl.dl_bw_gen = dl_bw_gen;
+}
+
+static void migrate_task_rq_dl(struct task_struct *p, int new_cpu)
 {
+	int cpu = task_cpu(p);
 	struct rq_flags rf;
 	struct rq *rq;
 
+	/*
+	 * The task must leave a CPU that is no longer active, or that its
+	 * mask no longer holds. The new CPU can then be in another root
+	 * domain. Move the bandwidth there, as dl_task_offline_migration()
+	 * does for a throttled task.
+	 */
+	if (unlikely(!cpu_active(cpu) || !cpumask_test_cpu(cpu, p->cpus_ptr)))
+		dl_task_move_bw(p, new_cpu);
+
 	if (READ_ONCE(p->__state) != TASK_WAKING)
 		return;
 
@@ -3577,9 +3620,10 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * (different root_domains) entails a bandwidth update. Move the
 	 * bandwidth only if the caller made room for it in the destination,
 	 * as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
-	 * Otherwise, the bandwidth stays where it is counted, until the next
-	 * rebuild of the accounting. Moving it without room would leave no
-	 * root domain counting it.
+	 * Otherwise, the bandwidth stays where it is counted, until the task
+	 * leaves a CPU that the new mask does not hold (migrate_task_rq_dl()),
+	 * or the next rebuild of the accounting. Freeing it here without room
+	 * would leave no root domain counting it.
 	 *
 	 * The caller made room in the root domain of ctx->dl_bw_cpu, the CPU
 	 * it expects the move to record. Move the bandwidth only if the move
@@ -3734,7 +3778,7 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
  * task runs, or wakes up, in the root domain of that CPU. A mask can hold
  * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
  * isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * select_fallback_rq() can widen it to both.
  *
  * Else prefer the CPU recorded with the bandwidth, if it is active and in
  * the mask. It is in the root domain that admitted the bandwidth, or that
@@ -3801,8 +3845,8 @@ void dl_add_task_root_domain(struct task_struct *p)
 		/*
 		 * No active CPU can count the bandwidth of this task. Leave it
 		 * out of this rebuild. dl_bw_gen then keeps its release from
-		 * removing anything. A later rebuild, or
-		 * dl_task_offline_migration(), counts it again.
+		 * removing anything. A later rebuild counts it again. So does
+		 * the move of the task to an active CPU.
 		 */
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 		return;
-- 
2.47.3