[PATCH] sched/core: keep offline cpus out of core-wide selections

[PATCH] sched/core: keep offline cpus out of core-wide selections

[Ahmed Abdelraoof]

From: Ahmed Abdelraoof <araoof@amazon.com>

During CPU hotplug in start_secondary(), the incoming CPU joins the
core-scheduling sibling sets by calling set_cpu_sibling_map() from
ap_starting(). Next, it runs notify_cpu_starting(), where
sched_core_cpu_starting() points the CPU's rq->core at the group's
leader. Finally, the CPU is set online via set_cpu_online().

This creates a narrow race leading to potentially silent corruptions: an
online CPU o performs a core-wide selection via pick_next_task() and
encounters an incoming CPU i that has joined the sibling set but has not
yet pointed its rq->core at the group's leader. o calls pick_task() on
i's runqueue and sets i's rq->core_pick while holding only o's rq lock,
which is not the lock serializing i's runqueue.

The same window has another ordering that leads to crashes. o can
complete the first walk of the sibling set before i joins it, leaving
i's rq->core_pick NULL. If set_cpu_sibling_map() then publishes i
before o starts the second walk, that walk visits i, reads the NULL
rq->core_pick and passes it to cookie_equals(), which dereferences
it, leading to crashes.

Skip offline CPUs during the initial core-wide pick. Use the resulting
non-NULL rq->core_pick values to identify participants in the later
passes as online mask can change between walks.

Opportunistically also, exclude offline CPUs from the single-pass
forced-idle accounting.

Fixes: 539f65125d20 ("sched: Add core wide task selection and scheduling")
Fixes: bc9ffef31bf5 ("sched/core: Simplify core-wide task selection")
Cc: stable@vger.kernel.org
Signed-off-by: Ahmed Abdelraoof <araoof@amazon.com>
---
 kernel/sched/core.c       | 12 ++++++++++++
 kernel/sched/core_sched.c |  2 +-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d7c3b43158de..f55f6a0befcc 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6413,6 +6413,9 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	 */
 	max = NULL;
 	for_each_cpu_wrap(i, smt_mask, cpu) {
+		if (!cpu_online(i))
+			continue;
+
 		rq_i = cpu_rq(i);
 
 		/*
@@ -6448,6 +6451,8 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	for_each_cpu(i, smt_mask) {
 		rq_i = cpu_rq(i);
 		p = rq_i->core_pick;
+		if (!p)
+			continue;
 
 		if (!cookie_equals(p, cookie)) {
 			p = NULL;
@@ -6760,6 +6765,13 @@ static inline void sched_core_cpu_dying(unsigned int cpu)
 
 	if (rq->core != rq)
 		rq->core = rq;
+
+	/*
+	 * Core-wide selections skip offline siblings and rely on a NULL
+	 * core_pick to tell them apart from participants.
+	 */
+	rq->core_pick = NULL;
+	rq->core_dl_server = NULL;
 }
 
 #else /* !CONFIG_SCHED_CORE: */
diff --git a/kernel/sched/core_sched.c b/kernel/sched/core_sched.c
index 43e0bde3038e..1efde88da76e 100644
--- a/kernel/sched/core_sched.c
+++ b/kernel/sched/core_sched.c
@@ -273,7 +273,7 @@ void __sched_core_account_forceidle(struct rq *rq)
 		delta = div_u64(delta, rq->core->core_forceidle_occupation);
 	}
 
-	for_each_cpu(i, smt_mask) {
+	for_each_cpu_and(i, smt_mask, cpu_online_mask) {
 		rq_i = cpu_rq(i);
 		p = rq_i->core_pick ?: rq_i->curr;
 
-- 
2.47.3