[PATCH v2 sched_ext/for-7.3-fixes] sched_ext: Reenqueue IMMED tasks when their CPU picks idle

[PATCH v2 sched_ext/for-7.3-fixes] sched_ext: Reenqueue IMMED tasks when their CPU picks idle

[Andrea Righi]

Core scheduling can force a CPU idle while tasks are runnable on it:
when the CPU's highest priority task doesn't match the core cookie, or
when an IMMED task's own cookie doesn't match it, e.g. a tagged IMMED
task while the SMT sibling runs an untagged RT task. In both cases
rq->next_class is idle, so rq_is_open() reports the CPU as open and
IMMED tasks inserted into its local DSQ are parked there, instead of
going back to the BPF scheduler.

This was reproduced on an SMT pair, with the ext server runtime set to 0
on both siblings and an untagged SCHED_FIFO task spinning on one of
them, forcing the other idle in two ways: with a tagged SCHED_FIFO task
runnable on it, and with no higher class task, but a tagged IMMED task.
In both cases, the IMMED task inserted into the local DSQ of the
forced-idle sibling was never reenqueued and stayed parked for the
whole test time window.

Instead of predicting the core scheduling pick, react to it: a CPU
picking idle with IMMED tasks on its local DSQ means that they didn't
run immediately, whatever the reason. __scx_update_idle() is called on
every idle pick, both on a task to idle switch and on an idle to idle
re-pick, so schedule a local DSQ reenqueue from there, with a new
internal SCX_REENQ_IDLE_PICK flag that tells reenq_local() to skip
rq_is_open(), which would report the idle rq as open and keep the IMMED
task at the head. With this change the IMMED task is reenqueued right
away and the BPF scheduler runs it on another CPU.

Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED")
Cc: stable@vger.kernel.org # v7.1+
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
Changes in v2:
 - React to the idle pick instead of predicting it from rq_is_open(),
   which also covers an IMMED task whose own cookie forces the CPU idle
   (Tejun Heo)
 - Link to v1: https://lore.kernel.org/r/20261017124749.3608468-1-arighi@nvidia.com

 kernel/sched/ext/ext.c      |  2 +-
 kernel/sched/ext/idle.c     | 10 ++++++++++
 kernel/sched/ext/internal.h |  6 ++++++
 3 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 32b248a6d1777..7fc306ff54ac1 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -4591,7 +4591,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
 
 	if (WARN_ON_ONCE(reenq_flags & __SCX_REENQ_TSR_MASK))
 		reenq_flags &= ~__SCX_REENQ_TSR_MASK;
-	if (rq_is_open(rq, 0))
+	if (!(reenq_flags & SCX_REENQ_IDLE_PICK) && rq_is_open(rq, 0))
 		reenq_flags |= SCX_REENQ_TSR_RQ_OPEN;
 
 	/*
diff --git a/kernel/sched/ext/idle.c b/kernel/sched/ext/idle.c
index 28ed9e3270854..3d8d75ffbf9e1 100644
--- a/kernel/sched/ext/idle.c
+++ b/kernel/sched/ext/idle.c
@@ -804,6 +804,16 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)
 
 	lockdep_assert_rq_held(rq);
 
+	/*
+	 * Picking idle with IMMED tasks on the local DSQ means that they didn't
+	 * run immediately, e.g., core scheduling forced the CPU idle. Outside
+	 * of that, pick_task_scx() would have taken the local DSQ head. As
+	 * next_class is idle, the rq looks open to rq_is_open(), tell
+	 * reenq_local() to reenqueue the IMMED tasks anyway.
+	 */
+	if (idle && unlikely(rq->scx.nr_immed))
+		scx_schedule_reenq_local(rq, SCX_REENQ_IDLE_PICK);
+
 	/*
 	 * pick_task_idle() calls here only on an idle-to-idle re-pick and the
 	 * transitions call with @do_notify, so every reaching call updates the
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index 03f03204d1582..df8b013fcd09c 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -1819,6 +1819,12 @@ enum scx_reenq_flags {
 	/* internal: kernel-issued on cap revoke, not accepted from BPF */
 	SCX_REENQ_CAP_REVOKE	= 1LLU << 1,
 
+	/*
+	 * internal: kernel-issued when the rq picks idle with IMMED tasks on its
+	 * local DSQ, the rq isn't open for them regardless of rq_is_open().
+	 */
+	SCX_REENQ_IDLE_PICK	= 1LLU << 2,
+
 	__SCX_REENQ_FILTER_MASK	= 0xffffLLU,
 
 	__SCX_REENQ_USER_MASK	= SCX_REENQ_ANY,
-- 
2.56.0