[PATCH] sched/core: keep offline cpus out of core-wide selections

[PATCH] sched/core: keep offline cpus out of core-wide selections

[Ahmed Abdelraoof]

From: Ahmed Abdelraoof <araoof@amazon.com>

During CPU hotplug in start_secondary(), the incoming CPU joins the
core-scheduling sibling sets by calling set_cpu_sibling_map() from
ap_starting(). Next, it runs notify_cpu_starting(), where
sched_core_cpu_starting() points the CPU's rq->core at the group's
leader. Finally, the CPU is set online via set_cpu_online().

This creates a narrow race leading to potentially silent corruptions: an
online CPU o performs a core-wide selection via pick_next_task() and
encounters an incoming CPU i that has joined the sibling set but has not
yet pointed its rq->core at the group's leader. o calls pick_task() on
i's runqueue and sets i's rq->core_pick while holding only o's rq lock,
which is not the lock serializing i's runqueue.

The same window has another ordering that leads to crashes. o can
complete the first walk of the sibling set before i joins it, leaving
i's rq->core_pick NULL. If set_cpu_sibling_map() then publishes i
before o starts the second walk, that walk visits i, reads the NULL
rq->core_pick and passes it to cookie_equals(), which dereferences
it, leading to crashes.

Skip offline CPUs during the initial core-wide pick. Use the resulting
non-NULL rq->core_pick values to identify participants in the later
passes as online mask can change between walks.

Opportunistically also, exclude offline CPUs from the single-pass
forced-idle accounting.

Fixes: 539f65125d20 ("sched: Add core wide task selection and scheduling")
Fixes: bc9ffef31bf5 ("sched/core: Simplify core-wide task selection")
Cc: stable@vger.kernel.org
Signed-off-by: Ahmed Abdelraoof <araoof@amazon.com>
---
 kernel/sched/core.c       | 12 ++++++++++++
 kernel/sched/core_sched.c |  2 +-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d7c3b43158de..f55f6a0befcc 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6413,6 +6413,9 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	 */
 	max = NULL;
 	for_each_cpu_wrap(i, smt_mask, cpu) {
+		if (!cpu_online(i))
+			continue;
+
 		rq_i = cpu_rq(i);
 
 		/*
@@ -6448,6 +6451,8 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	for_each_cpu(i, smt_mask) {
 		rq_i = cpu_rq(i);
 		p = rq_i->core_pick;
+		if (!p)
+			continue;
 
 		if (!cookie_equals(p, cookie)) {
 			p = NULL;
@@ -6760,6 +6765,13 @@ static inline void sched_core_cpu_dying(unsigned int cpu)
 
 	if (rq->core != rq)
 		rq->core = rq;
+
+	/*
+	 * Core-wide selections skip offline siblings and rely on a NULL
+	 * core_pick to tell them apart from participants.
+	 */
+	rq->core_pick = NULL;
+	rq->core_dl_server = NULL;
 }
 
 #else /* !CONFIG_SCHED_CORE: */
diff --git a/kernel/sched/core_sched.c b/kernel/sched/core_sched.c
index 43e0bde3038e..1efde88da76e 100644
--- a/kernel/sched/core_sched.c
+++ b/kernel/sched/core_sched.c
@@ -273,7 +273,7 @@ void __sched_core_account_forceidle(struct rq *rq)
 		delta = div_u64(delta, rq->core->core_forceidle_occupation);
 	}
 
-	for_each_cpu(i, smt_mask) {
+	for_each_cpu_and(i, smt_mask, cpu_online_mask) {
 		rq_i = cpu_rq(i);
 		p = rq_i->core_pick ?: rq_i->curr;
 
-- 
2.47.3

Re: [PATCH] sched/core: keep offline cpus out of core-wide selections

[Peter Zijlstra]

On Sun, Nov 01, 2026 at 10:12:18PM +0000, Ahmed Abdelraoof wrote:

>  kernel/sched/core.c       | 12 ++++++++++++
>  kernel/sched/core_sched.c |  2 +-
>  2 files changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index d7c3b43158de..f55f6a0befcc 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -6413,6 +6413,9 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
>  	 */
>  	max = NULL;
>  	for_each_cpu_wrap(i, smt_mask, cpu) {
> +		if (!cpu_online(i))
> +			continue;
> +
>  		rq_i = cpu_rq(i);
>  
>  		/*
> @@ -6448,6 +6451,8 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
>  	for_each_cpu(i, smt_mask) {
>  		rq_i = cpu_rq(i);
>  		p = rq_i->core_pick;
> +		if (!p)
> +			continue;

Can't you do what you do in the last hunk for all 3 cases? It seems
unfortunate to have 3 different ways to iterate in one patch.

>  
>  		if (!cookie_equals(p, cookie)) {
>  			p = NULL;
> @@ -6760,6 +6765,13 @@ static inline void sched_core_cpu_dying(unsigned int cpu)
>  
>  	if (rq->core != rq)
>  		rq->core = rq;
> +
> +	/*
> +	 * Core-wide selections skip offline siblings and rely on a NULL
> +	 * core_pick to tell them apart from participants.
> +	 */
> +	rq->core_pick = NULL;
> +	rq->core_dl_server = NULL;
>  }
>  
>  #else /* !CONFIG_SCHED_CORE: */
> diff --git a/kernel/sched/core_sched.c b/kernel/sched/core_sched.c
> index 43e0bde3038e..1efde88da76e 100644
> --- a/kernel/sched/core_sched.c
> +++ b/kernel/sched/core_sched.c
> @@ -273,7 +273,7 @@ void __sched_core_account_forceidle(struct rq *rq)
>  		delta = div_u64(delta, rq->core->core_forceidle_occupation);
>  	}
>  
> -	for_each_cpu(i, smt_mask) {
> +	for_each_cpu_and(i, smt_mask, cpu_online_mask) {
>  		rq_i = cpu_rq(i);
>  		p = rq_i->core_pick ?: rq_i->curr;
>  
> -- 
> 2.47.3
>