* [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
@ 2026-11-17 14:48 Mathieu Desnoyers
2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long
0 siblings, 2 replies; 3+ messages in thread
From: Mathieu Desnoyers @ 2026-11-17 14:48 UTC (permalink / raw)
To: Frederic Weisbecker, Peter Zijlstra, Ingo Molnar, Juri Lelli,
Vincent Guittot
Cc: Mathieu Desnoyers, Waiman Long, Michal Hocko, Tejun Heo,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, linux-kernel
Hi,
With CONFIG_CPUSETS=n, a kernel booted with isolcpus=domain,0 hits the
BUG() in select_fallback_rq() during boot. CPU 0 is the boot CPU. I
found it while testing a SCHED_DEADLINE series [1]. I then reproduced
it on v7.4-rc2 alone. tip/master of 2026-11-17 and next-20261116 have
the same code. A proposed fix follows the scissors line.
Trigger
=======
Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
init runs, in every boot.
These boot fine:
- the same kernel with isolcpus=domain,1,2, or without isolcpus;
- a kernel with CONFIG_CPUSETS=y, with isolcpus=domain,0,1.
Log
===
kernel BUG at kernel/sched/core.c:3642!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.4.0-rc2-g7475ea8b5861 #1 PREEMPT(lazy)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:select_fallback_rq+0x1fd/0x350
Call Trace:
<TASK>
try_to_wake_up+0x299/0x9b0
create_worker+0x17d/0x200
workqueue_init+0x2bd/0x2f0
kernel_init_freeable+0x79/0x1f0
kernel_init+0x1a/0x130
ret_from_fork+0x22a/0x330
ret_from_fork_asm+0x1a/0x30
</TASK>
Kernel panic - not syncing: Fatal exception
Analysis
========
workqueue_init() runs before smp_init(), so only the boot CPU is
online. create_worker() binds a new unbound kworker to the CPUs of its
pool. Workqueue restricts unbound workers to the housekeeping CPUs. With
isolcpus=domain,0,1, those are CPUs 2 and 3. Neither is online yet.
The wakeup of the kworker then calls select_fallback_rq(). It finds no
allowed CPU online, and widens the mask of the task in two steps:
- cpuset_cpus_allowed_fallback(). With CONFIG_CPUSETS=y, it gives the
mask of the top cpuset, which holds every CPU. The kworker then runs
on CPU 0. With CONFIG_CPUSETS=n, it is a stub that returns false.
- task_cpu_fallback_mask(), which is
housekeeping_cpumask(HK_TYPE_DOMAIN). That is CPUs 2 and 3 again.
No step is left, and the loop reaches BUG().
The last step used to allow all CPUs. 3a5446612a3f ("sched,arm64:
Handle CPU isolation on last resort fallback rq selection"), in v6.14,
changed it to the nohz_full housekeeping CPUs. With isolcpus=domain
alone, those are all CPUs. bf5b0c27fad2 ("sched: Switch the fallback
task allowed cpumask to HK_TYPE_DOMAIN"), in v7.0, changed it to the
domain housekeeping CPUs. A kernel with bf5b0c27fad2 reverted boots
with isolcpus=domain,0,1 and CONFIG_CPUSETS=n.
Where to fix
============
The patch adds a last step before the BUG(): task_cpu_possible_mask(),
the mask the fallback used before 3a5446612a3f. It is reached only when
no housekeeping CPU is online. The kworker then runs on the boot CPU, as
with CONFIG_CPUSETS=y.
Another option is to have workqueue_init() leave the unbound workers
on the boot CPU until a housekeeping CPU is online. It would fix this
path only. Other early kthreads with a housekeeping mask could hit the
same BUG(). This is from reading.
Testing
=======
I ran v7.4-rc2 in VMs with 4 CPUs, built with CONFIG_CPUSETS=n:
- Without the patch: isolcpus=domain,0,1 hits the BUG() in 3 boots of
3, and isolcpus=domain,0 in 1 of 1. isolcpus=domain,1,2 and no
isolcpus boot.
- With bf5b0c27fad2 reverted: isolcpus=domain,0,1 boots, 3 of 3.
- With the patch: isolcpus=domain,0,1 boots, 3 of 3. So do
isolcpus=domain,0, isolcpus=domain,1,2 and no isolcpus.
After boot, the unbound kworkers have the same CPUs with the patch as
with CONFIG_CPUSETS=y: CPUs 2 and 3 for isolcpus=domain,0,1. No "no
longer affine" message is printed. The patch also builds with
CONFIG_CPUSETS=y. That kernel boots with isolcpus=domain,0,1. I did
not run other tests, nor other architectures.
The analysis and the patch were written by an LLM coding assistant
under my direction and review.
Thanks,
Mathieu
[1] https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
-- >8 --
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Subject: sched: Fall back to all possible CPUs when no housekeeping CPU is online
select_fallback_rq() picks a CPU for a task that has no allowed CPU
online. Its last steps widen the mask of the task: first to its
cpuset, then to task_cpu_fallback_mask(). Commit bf5b0c27fad2 ("sched:
Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN") made that
mask the domain housekeeping CPUs. The task then does not land on an
isolated CPU. If none of these CPUs is online, the next step is BUG().
That happens at boot when isolcpus=domain isolates the boot CPU.
workqueue_init() runs before the other CPUs come online. It wakes a new
unbound kworker, bound to the housekeeping CPUs. With CONFIG_CPUSETS=y,
the cpuset step gives the kworker every CPU, and it runs on the boot
CPU. With CONFIG_CPUSETS=n, that step does nothing, and the kernel
panics at boot.
Fix this by adding a last step before the BUG(): allow every CPU the
task can run on, as task_cpu_possible_mask() gives. This was the last
step before commit 3a5446612a3f ("sched,arm64: Handle CPU isolation on
last resort fallback rq selection"). It is reached only when no
housekeeping CPU is online. In testing, the kernel then boots. After
boot, the unbound kworkers have the housekeeping CPUs, as with
CONFIG_CPUSETS=y.
Fixes: bf5b0c27fad2 ("sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/core.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..556a27f14619 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3599,7 +3599,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
{
int nid = cpu_to_node(cpu);
const struct cpumask *nodemask = NULL;
- enum { cpuset, possible, fail } state = cpuset;
+ enum { cpuset, housekeeping, possible, fail } state = cpuset;
int dest_cpu;
/*
@@ -3630,12 +3630,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
switch (state) {
case cpuset:
if (cpuset_cpus_allowed_fallback(p)) {
- state = possible;
+ state = housekeeping;
break;
}
fallthrough;
- case possible:
+ case housekeeping:
set_cpus_allowed_force(p, task_cpu_fallback_mask(p));
+ state = possible;
+ break;
+ case possible:
+ /*
+ * No housekeeping CPU is online. This happens at boot
+ * when isolcpus=domain isolates the boot CPU. Allow
+ * every CPU, as the cpuset fallback does.
+ */
+ set_cpus_allowed_force(p, task_cpu_possible_mask(p));
state = fail;
break;
case fail:
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
2026-11-17 14:48 [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n Mathieu Desnoyers
@ 2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long
1 sibling, 0 replies; 3+ messages in thread
From: Peter Zijlstra @ 2026-11-17 15:02 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Frederic Weisbecker, Ingo Molnar, Juri Lelli, Vincent Guittot,
Waiman Long, Michal Hocko, Tejun Heo, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, linux-kernel
On Tue, Nov 17, 2026 at 09:48:17AM -0500, Mathieu Desnoyers wrote:
> Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
> isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
> init runs, in every boot.
Arguably, marking the boot cpu as isolated is asking for it. Is this
really worth fixing? Or should we look for a better failure message?
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
2026-11-17 14:48 [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n Mathieu Desnoyers
2026-11-17 15:02 ` Peter Zijlstra
@ 2026-11-17 15:13 ` Waiman Long
1 sibling, 0 replies; 3+ messages in thread
From: Waiman Long @ 2026-11-17 15:13 UTC (permalink / raw)
To: Mathieu Desnoyers, Frederic Weisbecker, Peter Zijlstra,
Ingo Molnar, Juri Lelli, Vincent Guittot
Cc: Michal Hocko, Tejun Heo, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
linux-kernel
On 11/17/26 9:48 AM, Mathieu Desnoyers wrote:
> Hi,
>
> With CONFIG_CPUSETS=n, a kernel booted with isolcpus=domain,0 hits the
> BUG() in select_fallback_rq() during boot. CPU 0 is the boot CPU. I
> found it while testing a SCHED_DEADLINE series [1]. I then reproduced
> it on v7.4-rc2 alone. tip/master of 2026-11-17 and next-20261116 have
> the same code. A proposed fix follows the scissors line.
>
> Trigger
> =======
>
> Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
> isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
> init runs, in every boot.
>
> These boot fine:
> - the same kernel with isolcpus=domain,1,2, or without isolcpus;
> - a kernel with CONFIG_CPUSETS=y, with isolcpus=domain,0,1.
I am now thinking about disallowing the boot CPU from being domain
isolated. That will include using isolcpus=domain boot time isolation
parameter as well as in setting up isolated cpuset partition. Will that
be a simpler solution than fixing all these places where isolating the
boot CPU will cause various problems that need to be fixed?
Thanks,
Longman
> Log
> ===
>
> kernel BUG at kernel/sched/core.c:3642!
> Oops: invalid opcode: 0000 [#1] SMP NOPTI
> CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.4.0-rc2-g7475ea8b5861 #1 PREEMPT(lazy)
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> RIP: 0010:select_fallback_rq+0x1fd/0x350
> Call Trace:
> <TASK>
> try_to_wake_up+0x299/0x9b0
> create_worker+0x17d/0x200
> workqueue_init+0x2bd/0x2f0
> kernel_init_freeable+0x79/0x1f0
> kernel_init+0x1a/0x130
> ret_from_fork+0x22a/0x330
> ret_from_fork_asm+0x1a/0x30
> </TASK>
> Kernel panic - not syncing: Fatal exception
>
> Analysis
> ========
>
> workqueue_init() runs before smp_init(), so only the boot CPU is
> online. create_worker() binds a new unbound kworker to the CPUs of its
> pool. Workqueue restricts unbound workers to the housekeeping CPUs. With
> isolcpus=domain,0,1, those are CPUs 2 and 3. Neither is online yet.
>
> The wakeup of the kworker then calls select_fallback_rq(). It finds no
> allowed CPU online, and widens the mask of the task in two steps:
> - cpuset_cpus_allowed_fallback(). With CONFIG_CPUSETS=y, it gives the
> mask of the top cpuset, which holds every CPU. The kworker then runs
> on CPU 0. With CONFIG_CPUSETS=n, it is a stub that returns false.
> - task_cpu_fallback_mask(), which is
> housekeeping_cpumask(HK_TYPE_DOMAIN). That is CPUs 2 and 3 again.
>
> No step is left, and the loop reaches BUG().
>
> The last step used to allow all CPUs. 3a5446612a3f ("sched,arm64:
> Handle CPU isolation on last resort fallback rq selection"), in v6.14,
> changed it to the nohz_full housekeeping CPUs. With isolcpus=domain
> alone, those are all CPUs. bf5b0c27fad2 ("sched: Switch the fallback
> task allowed cpumask to HK_TYPE_DOMAIN"), in v7.0, changed it to the
> domain housekeeping CPUs. A kernel with bf5b0c27fad2 reverted boots
> with isolcpus=domain,0,1 and CONFIG_CPUSETS=n.
>
> Where to fix
> ============
>
> The patch adds a last step before the BUG(): task_cpu_possible_mask(),
> the mask the fallback used before 3a5446612a3f. It is reached only when
> no housekeeping CPU is online. The kworker then runs on the boot CPU, as
> with CONFIG_CPUSETS=y.
>
> Another option is to have workqueue_init() leave the unbound workers
> on the boot CPU until a housekeeping CPU is online. It would fix this
> path only. Other early kthreads with a housekeeping mask could hit the
> same BUG(). This is from reading.
>
> Testing
> =======
>
> I ran v7.4-rc2 in VMs with 4 CPUs, built with CONFIG_CPUSETS=n:
> - Without the patch: isolcpus=domain,0,1 hits the BUG() in 3 boots of
> 3, and isolcpus=domain,0 in 1 of 1. isolcpus=domain,1,2 and no
> isolcpus boot.
> - With bf5b0c27fad2 reverted: isolcpus=domain,0,1 boots, 3 of 3.
> - With the patch: isolcpus=domain,0,1 boots, 3 of 3. So do
> isolcpus=domain,0, isolcpus=domain,1,2 and no isolcpus.
>
> After boot, the unbound kworkers have the same CPUs with the patch as
> with CONFIG_CPUSETS=y: CPUs 2 and 3 for isolcpus=domain,0,1. No "no
> longer affine" message is printed. The patch also builds with
> CONFIG_CPUSETS=y. That kernel boots with isolcpus=domain,0,1. I did
> not run other tests, nor other architectures.
>
> The analysis and the patch were written by an LLM coding assistant
> under my direction and review.
>
> Thanks,
>
> Mathieu
>
> [1] https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
>
> -- >8 --
> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Subject: sched: Fall back to all possible CPUs when no housekeeping CPU is online
>
> select_fallback_rq() picks a CPU for a task that has no allowed CPU
> online. Its last steps widen the mask of the task: first to its
> cpuset, then to task_cpu_fallback_mask(). Commit bf5b0c27fad2 ("sched:
> Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN") made that
> mask the domain housekeeping CPUs. The task then does not land on an
> isolated CPU. If none of these CPUs is online, the next step is BUG().
>
> That happens at boot when isolcpus=domain isolates the boot CPU.
> workqueue_init() runs before the other CPUs come online. It wakes a new
> unbound kworker, bound to the housekeeping CPUs. With CONFIG_CPUSETS=y,
> the cpuset step gives the kworker every CPU, and it runs on the boot
> CPU. With CONFIG_CPUSETS=n, that step does nothing, and the kernel
> panics at boot.
>
> Fix this by adding a last step before the BUG(): allow every CPU the
> task can run on, as task_cpu_possible_mask() gives. This was the last
> step before commit 3a5446612a3f ("sched,arm64: Handle CPU isolation on
> last resort fallback rq selection"). It is reached only when no
> housekeeping CPU is online. In testing, the kernel then boots. After
> boot, the unbound kworkers have the housekeeping CPUs, as with
> CONFIG_CPUSETS=y.
>
> Fixes: bf5b0c27fad2 ("sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
> kernel/sched/core.c | 15 ++++++++++++---
> 1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 1a088d4576f5..556a27f14619 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3599,7 +3599,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
> {
> int nid = cpu_to_node(cpu);
> const struct cpumask *nodemask = NULL;
> - enum { cpuset, possible, fail } state = cpuset;
> + enum { cpuset, housekeeping, possible, fail } state = cpuset;
> int dest_cpu;
>
> /*
> @@ -3630,12 +3630,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
> switch (state) {
> case cpuset:
> if (cpuset_cpus_allowed_fallback(p)) {
> - state = possible;
> + state = housekeeping;
> break;
> }
> fallthrough;
> - case possible:
> + case housekeeping:
> set_cpus_allowed_force(p, task_cpu_fallback_mask(p));
> + state = possible;
> + break;
> + case possible:
> + /*
> + * No housekeeping CPU is online. This happens at boot
> + * when isolcpus=domain isolates the boot CPU. Allow
> + * every CPU, as the cpuset fallback does.
> + */
> + set_cpus_allowed_force(p, task_cpu_possible_mask(p));
> state = fail;
> break;
> case fail:
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-17 15:13 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 14:48 [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n Mathieu Desnoyers
2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®