mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ibrahim Elshahed <lshahed@amazon.de>
To: Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Valentin Schneider <vschneid@redhat.com>,
	K Prateek Nayak <kprateek.nayak@amd.com>,
	Josh Don <joshdon@google.com>, <linux-kernel@vger.kernel.org>,
	<nh-open-source@amazon.com>, Fernand Sieber <sieberf@amazon.com>,
	Roman Kagan <rkagan@amazon.de>
Subject: [PATCH 2/2] sched/core: Don't skip a pending core-wide pick on idle re-entry
Date: Tue, 17 Nov 2026 14:27:03 +0000	[thread overview]
Message-ID: <20261117142704.6878-3-lshahed@amazon.de> (raw)
In-Reply-To: <20261117142704.6878-1-lshahed@amazon.de>

Since v6.12, with commit 3dcac251b066 ("sched/core: Introduce SM_IDLE
and an idle re-entry fast-path in __schedule()"), an idle CPU with an
empty runqueue skips pick_next_task() when it is rescheduled. Under
core scheduling that pick also consumes the pick a sibling's core-wide
selection left in rq->core_pick and ends or re-examines the core's
forced idle.

A core-wide selection chooses a task for an idle sibling and reschedules
it. If the chosen task is migrated away before the sibling schedules
and it was the sibling's only task, the sibling goes back to idle
through the fast path without consuming its pick. When that selection
also forced the other sibling idle, because its runnable tasks did not
match the chosen task's cookie, the core stays forced idle with nothing
running on it, and two things go wrong:

- Without the previous patch, the idle sibling's rq->core_pick keeps
  pointing at the migrated task, and with schedstats enabled the forced
  idle accounting keeps writing through it, even after the task and its
  cgroup are freed:

  BUG: unable to handle page fault for address: ffff8d6b205f7018
  #PF: supervisor write access in kernel mode
  #PF: error_code(0x0002) - not-present page
  RIP: 0010:__cgroup_account_cputime_field+0x3c/0x70
  Call Trace:
   __sched_core_account_forceidle+0xd8/0x140
   __schedule+0x1390/0x18d0
   schedule_idle+0x1a/0x30
   cpu_startup_entry+0x29/0x30
   start_secondary+0x11e/0x140

- The forced-idle sibling keeps waiting with runnable tasks until some
  unrelated event reschedules the core. Likewise, when the forced-idle
  CPU itself loses its last task, the reschedule that
  sched_core_dequeue() sends to end the forced idle also ends in the
  fast path.

Take the SM_IDLE fast path only when core scheduling has neither a
pick pending for this CPU nor forced idle to settle.

Fixes: 3dcac251b066 ("sched/core: Introduce SM_IDLE and an idle re-entry fast-path in __schedule()")
Cc: stable@vger.kernel.org
Co-developed-by: Roman Kagan <rkagan@amazon.de>
Signed-off-by: Roman Kagan <rkagan@amazon.de>
Signed-off-by: Ibrahim Elshahed <lshahed@amazon.de>
---
 kernel/sched/core.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b1718ba5a4..58b429bbe3 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -347,6 +347,17 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags)
 		resched_curr(rq);
 }
 
+/*
+ * An idle CPU with an empty runqueue must still call pick_next_task() while
+ * it has a pending core-wide pick or the core is forced idle. Only that pick
+ * consumes rq->core_pick and ends the forced idle.
+ */
+static inline bool sched_core_idle_needs_pick(struct rq *rq)
+{
+	return sched_core_enabled(rq) &&
+	       (rq->core_pick || rq->core->core_forceidle_count);
+}
+
 static int sched_task_is_throttled(struct task_struct *p, int cpu)
 {
 	if (p->sched_class->task_is_throttled)
@@ -550,6 +561,7 @@ void sched_core_put(void)
 static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
 static inline void
 sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }
+static inline bool sched_core_idle_needs_pick(struct rq *rq) { return false; }
 
 #endif /* !CONFIG_SCHED_CORE */
 
@@ -7285,8 +7297,13 @@ static void __sched notrace __schedule(int sched_mode)
 	 */
 	prev_state = READ_ONCE(prev->__state);
 	if (sched_mode == SM_IDLE) {
-		/* SCX must consult the BPF scheduler to tell if rq is empty */
-		if (!rq->nr_running && !scx_enabled()) {
+		/*
+		 * SCX must consult the BPF scheduler to tell if rq is empty,
+		 * and core scheduling may need pick_next_task() to consume
+		 * rq->core_pick or to end forced idle.
+		 */
+		if (!rq->nr_running && !scx_enabled() &&
+		    !sched_core_idle_needs_pick(rq)) {
 			next = prev;
 			rq->next_class = &idle_sched_class;
 			goto picked;
-- 
2.47.3


  parent reply	other threads:[~2026-11-17 14:27 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 14:27 [PATCH 0/2] sched/core: Fix stale core_pick and forced idle after SM_IDLE fast path Ibrahim Elshahed
2026-11-17 14:27 ` [PATCH 1/2] sched/core: Clear a stale rq->core_pick after dequeue Ibrahim Elshahed
2026-11-17 14:27 ` Ibrahim Elshahed [this message]
2026-11-17 14:27 ` [PATCH 0/2] sched/core: Fix stale core_pick and forced idle after SM_IDLE fast path Ibrahim Elshahed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117142704.6878-3-lshahed@amazon.de \
    --to=lshahed@amazon.de \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=joshdon@google.com \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=nh-open-source@amazon.com \
    --cc=peterz@infradead.org \
    --cc=rkagan@amazon.de \
    --cc=rostedt@goodmis.org \
    --cc=sieberf@amazon.com \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®