[PATCH] sched/rt: queue a push when a queued RT task is boosted

[Chaosheng Qu <chaosheng.qu.linux@gmail.com>]

A PI boost can leave a queued RT task stuck on a CPU where it cannot run.
The pull side only considers tasks whose affinity includes the CPU that
became idle, and prio_changed_rt() does not queue a push. If the boosted
task's mask excludes the idle CPU, neither path can move it.

In a four-vCPU non-PREEMPT_RT guest with RT_PUSH_IPI off, a FIFO-10 task
holding a PI mutex, boosted to FIFO 90 by a waiter on another CPU while
a FIFO-99 hog kept its CPU busy, stayed queued for the whole 1.5 s dwell.
A FIFO-60 spare was idle on a CPU in its mask. With RT_PUSH_IPI on, the
same setup resumed in under 6 ms.

Queue a push from prio_changed_rt() when a queued, non-current RT task
with more than one CPU has its priority raised. The push selects a target
from the task's cpus_mask, which is what the pulling CPU cannot do.
rt_mutex_setprio() calls __balance_callbacks() after prio_changed_rt(),
so the callback runs in the same rq->lock section as the existing
rt_queue_pull_task().

With no boost present the push finds no legal target.

Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Chaosheng Qu <chaosheng.qu.linux@gmail.com>
---
 kernel/sched/rt.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 85303add7..dc2775261 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2499,6 +2499,18 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
 		 */
 		if (p->prio < rq->donor->prio)
 			resched_curr(rq);
+
+		/*
+		 * A boost can create a migration opportunity that the pull
+		 * side cannot reach: pull_rt_task() on another CPU only picks
+		 * tasks whose affinity includes that CPU, so a task that gets
+		 * boosted while its CPUs are busy can stay queued until it
+		 * becomes current. Queue a push on this rq instead, which
+		 * selects a target from the task's own affinity.
+		 */
+		if (p->nr_cpus_allowed > 1 && p->prio < oldprio &&
+		    rq->rt.overloaded)
+			rt_queue_push_tasks(rq);
 	}
 }
 
-- 
2.43.0