From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: Frederic Weisbecker <frederic@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Waiman Long <longman@redhat.com>, Michal Hocko <mhocko@suse.com>,
Tejun Heo <tj@kernel.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>,
linux-kernel@vger.kernel.org
Subject: [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
Date: Tue, 17 Nov 2026 09:48:17 -0500 [thread overview]
Message-ID: <20261117144817.265109-1-mathieu.desnoyers@efficios.com> (raw)
Hi,
With CONFIG_CPUSETS=n, a kernel booted with isolcpus=domain,0 hits the
BUG() in select_fallback_rq() during boot. CPU 0 is the boot CPU. I
found it while testing a SCHED_DEADLINE series [1]. I then reproduced
it on v7.4-rc2 alone. tip/master of 2026-11-17 and next-20261116 have
the same code. A proposed fix follows the scissors line.
Trigger
=======
Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
init runs, in every boot.
These boot fine:
- the same kernel with isolcpus=domain,1,2, or without isolcpus;
- a kernel with CONFIG_CPUSETS=y, with isolcpus=domain,0,1.
Log
===
kernel BUG at kernel/sched/core.c:3642!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.4.0-rc2-g7475ea8b5861 #1 PREEMPT(lazy)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:select_fallback_rq+0x1fd/0x350
Call Trace:
<TASK>
try_to_wake_up+0x299/0x9b0
create_worker+0x17d/0x200
workqueue_init+0x2bd/0x2f0
kernel_init_freeable+0x79/0x1f0
kernel_init+0x1a/0x130
ret_from_fork+0x22a/0x330
ret_from_fork_asm+0x1a/0x30
</TASK>
Kernel panic - not syncing: Fatal exception
Analysis
========
workqueue_init() runs before smp_init(), so only the boot CPU is
online. create_worker() binds a new unbound kworker to the CPUs of its
pool. Workqueue restricts unbound workers to the housekeeping CPUs. With
isolcpus=domain,0,1, those are CPUs 2 and 3. Neither is online yet.
The wakeup of the kworker then calls select_fallback_rq(). It finds no
allowed CPU online, and widens the mask of the task in two steps:
- cpuset_cpus_allowed_fallback(). With CONFIG_CPUSETS=y, it gives the
mask of the top cpuset, which holds every CPU. The kworker then runs
on CPU 0. With CONFIG_CPUSETS=n, it is a stub that returns false.
- task_cpu_fallback_mask(), which is
housekeeping_cpumask(HK_TYPE_DOMAIN). That is CPUs 2 and 3 again.
No step is left, and the loop reaches BUG().
The last step used to allow all CPUs. 3a5446612a3f ("sched,arm64:
Handle CPU isolation on last resort fallback rq selection"), in v6.14,
changed it to the nohz_full housekeeping CPUs. With isolcpus=domain
alone, those are all CPUs. bf5b0c27fad2 ("sched: Switch the fallback
task allowed cpumask to HK_TYPE_DOMAIN"), in v7.0, changed it to the
domain housekeeping CPUs. A kernel with bf5b0c27fad2 reverted boots
with isolcpus=domain,0,1 and CONFIG_CPUSETS=n.
Where to fix
============
The patch adds a last step before the BUG(): task_cpu_possible_mask(),
the mask the fallback used before 3a5446612a3f. It is reached only when
no housekeeping CPU is online. The kworker then runs on the boot CPU, as
with CONFIG_CPUSETS=y.
Another option is to have workqueue_init() leave the unbound workers
on the boot CPU until a housekeeping CPU is online. It would fix this
path only. Other early kthreads with a housekeeping mask could hit the
same BUG(). This is from reading.
Testing
=======
I ran v7.4-rc2 in VMs with 4 CPUs, built with CONFIG_CPUSETS=n:
- Without the patch: isolcpus=domain,0,1 hits the BUG() in 3 boots of
3, and isolcpus=domain,0 in 1 of 1. isolcpus=domain,1,2 and no
isolcpus boot.
- With bf5b0c27fad2 reverted: isolcpus=domain,0,1 boots, 3 of 3.
- With the patch: isolcpus=domain,0,1 boots, 3 of 3. So do
isolcpus=domain,0, isolcpus=domain,1,2 and no isolcpus.
After boot, the unbound kworkers have the same CPUs with the patch as
with CONFIG_CPUSETS=y: CPUs 2 and 3 for isolcpus=domain,0,1. No "no
longer affine" message is printed. The patch also builds with
CONFIG_CPUSETS=y. That kernel boots with isolcpus=domain,0,1. I did
not run other tests, nor other architectures.
The analysis and the patch were written by an LLM coding assistant
under my direction and review.
Thanks,
Mathieu
[1] https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
-- >8 --
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Subject: sched: Fall back to all possible CPUs when no housekeeping CPU is online
select_fallback_rq() picks a CPU for a task that has no allowed CPU
online. Its last steps widen the mask of the task: first to its
cpuset, then to task_cpu_fallback_mask(). Commit bf5b0c27fad2 ("sched:
Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN") made that
mask the domain housekeeping CPUs. The task then does not land on an
isolated CPU. If none of these CPUs is online, the next step is BUG().
That happens at boot when isolcpus=domain isolates the boot CPU.
workqueue_init() runs before the other CPUs come online. It wakes a new
unbound kworker, bound to the housekeeping CPUs. With CONFIG_CPUSETS=y,
the cpuset step gives the kworker every CPU, and it runs on the boot
CPU. With CONFIG_CPUSETS=n, that step does nothing, and the kernel
panics at boot.
Fix this by adding a last step before the BUG(): allow every CPU the
task can run on, as task_cpu_possible_mask() gives. This was the last
step before commit 3a5446612a3f ("sched,arm64: Handle CPU isolation on
last resort fallback rq selection"). It is reached only when no
housekeeping CPU is online. In testing, the kernel then boots. After
boot, the unbound kworkers have the housekeeping CPUs, as with
CONFIG_CPUSETS=y.
Fixes: bf5b0c27fad2 ("sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/core.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..556a27f14619 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3599,7 +3599,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
{
int nid = cpu_to_node(cpu);
const struct cpumask *nodemask = NULL;
- enum { cpuset, possible, fail } state = cpuset;
+ enum { cpuset, housekeeping, possible, fail } state = cpuset;
int dest_cpu;
/*
@@ -3630,12 +3630,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
switch (state) {
case cpuset:
if (cpuset_cpus_allowed_fallback(p)) {
- state = possible;
+ state = housekeeping;
break;
}
fallthrough;
- case possible:
+ case housekeeping:
set_cpus_allowed_force(p, task_cpu_fallback_mask(p));
+ state = possible;
+ break;
+ case possible:
+ /*
+ * No housekeeping CPU is online. This happens at boot
+ * when isolcpus=domain isolates the boot CPU. Allow
+ * every CPU, as the cpuset fallback does.
+ */
+ set_cpus_allowed_force(p, task_cpu_possible_mask(p));
state = fail;
break;
case fail:
next reply other threads:[~2026-11-17 14:48 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 14:48 Mathieu Desnoyers [this message]
2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117144817.265109-1-mathieu.desnoyers@efficios.com \
--to=mathieu.desnoyers@efficios.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=frederic@kernel.org \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mgorman@suse.de \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®