[PATCH 4/5] sched/deadline: Don't charge core-sched forced idle to the DL servers

[Fernand Sieber <sieberf@amazon.com>]

A deferred DL server is charged the idle time of its rq, on the premise
that the time was available to the server's class and went unused. Core
scheduling breaks that premise: a CPU is forced idle when a SMT sibling
runs a task with an incompatible cookie, so that time was available to
no class at all.

update_curr_dl_se() then runs the server's runtime out while it is
throttled and calls replenish_dl_new_period(), pushing the activation a
full period out. dl_defer_idle does not help, as idle_rq() is false on a
forced-idle CPU: nr_running is not 0. Every period of forced idle pushes
the activation again, the zero-laxity timer never fires, and the class
the server reserves for gets nothing.

Reproduction on 2 cores x 2 threads with the default fair server
(50 ms / 1 s): a SCHED_FIFO spinner with a core-sched cookie on CPU0 and
an uncookied CFS spinner on CPU1 force-idle CPU1, and the CFS task gets
under 0.1% of a CPU instead of the reserved 5%. The ext server starves
sched_ext tasks the same way, and there a plain CFS spinner on the
sibling is enough.

Don't charge forced idle. A server still only loses runtime to real
execution or real idle, and one whose class has nothing to run still
stops through dl_defer_idle once the rq is idle for real; with core
scheduling off nothing changes. Getting the reserved bandwidth back also
needs the server's pick to win the core-wide selection, which the
preceding sched/core patches fix.

Fixes: a110a81c52a9 ("sched/deadline: Deferrable dl server")
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
 kernel/sched/idle.c  |  7 +++++++
 kernel/sched/sched.h | 15 +++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index f7dd151f77920..451b4846219d5 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -573,6 +573,13 @@ static void update_curr_idle(struct rq *rq)
 
 	se->exec_start = now;
 
+	/*
+	 * Forced idle was not time available to any class on this rq, so
+	 * don't let the deferred servers consume their runtime on it.
+	 */
+	if (forced_idle_rq(rq))
+		return;
+
 	dl_server_update_idle(&rq->fair_server, delta_exec);
 #ifdef CONFIG_SCHED_CLASS_EXT
 	dl_server_update_idle(&rq->ext_server, delta_exec);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a4df59c9c9b60..f3b22c4bd6782 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1525,6 +1525,16 @@ static inline bool sched_core_disabled(void)
 	return !static_branch_unlikely(&__sched_core_enabled);
 }
 
+/*
+ * True while core scheduling keeps @rq idle because a SMT sibling runs a task
+ * with an incompatible cookie, even though @rq has runnable tasks of its own.
+ */
+static inline bool forced_idle_rq(struct rq *rq)
+{
+	return sched_core_enabled(rq) && rq->core->core_forceidle_count &&
+	       rq->nr_running;
+}
+
 /*
  * Be careful with this function; not for general use. The return value isn't
  * stable unless you actually hold a relevant rq->__lock.
@@ -1659,6 +1669,11 @@ static inline bool sched_core_disabled(void)
 	return true;
 }
 
+static inline bool forced_idle_rq(struct rq *rq)
+{
+	return false;
+}
+
 static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 {
 	return &rq->__lock;
-- 
2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07