From: Waiman Long <longman@redhat.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Frederic Weisbecker <frederic@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>
Cc: Michal Hocko <mhocko@suse.com>, Tejun Heo <tj@kernel.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>,
linux-kernel@vger.kernel.org
Subject: Re: [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
Date: Tue, 17 Nov 2026 10:13:02 -0500 [thread overview]
Message-ID: <dd335ee6-c879-4bee-9ea8-6d9e9216fb0e@redhat.com> (raw)
In-Reply-To: <20261117144817.265109-1-mathieu.desnoyers@efficios.com>
On 11/17/26 9:48 AM, Mathieu Desnoyers wrote:
> Hi,
>
> With CONFIG_CPUSETS=n, a kernel booted with isolcpus=domain,0 hits the
> BUG() in select_fallback_rq() during boot. CPU 0 is the boot CPU. I
> found it while testing a SCHED_DEADLINE series [1]. I then reproduced
> it on v7.4-rc2 alone. tip/master of 2026-11-17 and next-20261116 have
> the same code. A proposed fix follows the scissors line.
>
> Trigger
> =======
>
> Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
> isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
> init runs, in every boot.
>
> These boot fine:
> - the same kernel with isolcpus=domain,1,2, or without isolcpus;
> - a kernel with CONFIG_CPUSETS=y, with isolcpus=domain,0,1.
I am now thinking about disallowing the boot CPU from being domain
isolated. That will include using isolcpus=domain boot time isolation
parameter as well as in setting up isolated cpuset partition. Will that
be a simpler solution than fixing all these places where isolating the
boot CPU will cause various problems that need to be fixed?
Thanks,
Longman
> Log
> ===
>
> kernel BUG at kernel/sched/core.c:3642!
> Oops: invalid opcode: 0000 [#1] SMP NOPTI
> CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.4.0-rc2-g7475ea8b5861 #1 PREEMPT(lazy)
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> RIP: 0010:select_fallback_rq+0x1fd/0x350
> Call Trace:
> <TASK>
> try_to_wake_up+0x299/0x9b0
> create_worker+0x17d/0x200
> workqueue_init+0x2bd/0x2f0
> kernel_init_freeable+0x79/0x1f0
> kernel_init+0x1a/0x130
> ret_from_fork+0x22a/0x330
> ret_from_fork_asm+0x1a/0x30
> </TASK>
> Kernel panic - not syncing: Fatal exception
>
> Analysis
> ========
>
> workqueue_init() runs before smp_init(), so only the boot CPU is
> online. create_worker() binds a new unbound kworker to the CPUs of its
> pool. Workqueue restricts unbound workers to the housekeeping CPUs. With
> isolcpus=domain,0,1, those are CPUs 2 and 3. Neither is online yet.
>
> The wakeup of the kworker then calls select_fallback_rq(). It finds no
> allowed CPU online, and widens the mask of the task in two steps:
> - cpuset_cpus_allowed_fallback(). With CONFIG_CPUSETS=y, it gives the
> mask of the top cpuset, which holds every CPU. The kworker then runs
> on CPU 0. With CONFIG_CPUSETS=n, it is a stub that returns false.
> - task_cpu_fallback_mask(), which is
> housekeeping_cpumask(HK_TYPE_DOMAIN). That is CPUs 2 and 3 again.
>
> No step is left, and the loop reaches BUG().
>
> The last step used to allow all CPUs. 3a5446612a3f ("sched,arm64:
> Handle CPU isolation on last resort fallback rq selection"), in v6.14,
> changed it to the nohz_full housekeeping CPUs. With isolcpus=domain
> alone, those are all CPUs. bf5b0c27fad2 ("sched: Switch the fallback
> task allowed cpumask to HK_TYPE_DOMAIN"), in v7.0, changed it to the
> domain housekeeping CPUs. A kernel with bf5b0c27fad2 reverted boots
> with isolcpus=domain,0,1 and CONFIG_CPUSETS=n.
>
> Where to fix
> ============
>
> The patch adds a last step before the BUG(): task_cpu_possible_mask(),
> the mask the fallback used before 3a5446612a3f. It is reached only when
> no housekeeping CPU is online. The kworker then runs on the boot CPU, as
> with CONFIG_CPUSETS=y.
>
> Another option is to have workqueue_init() leave the unbound workers
> on the boot CPU until a housekeeping CPU is online. It would fix this
> path only. Other early kthreads with a housekeeping mask could hit the
> same BUG(). This is from reading.
>
> Testing
> =======
>
> I ran v7.4-rc2 in VMs with 4 CPUs, built with CONFIG_CPUSETS=n:
> - Without the patch: isolcpus=domain,0,1 hits the BUG() in 3 boots of
> 3, and isolcpus=domain,0 in 1 of 1. isolcpus=domain,1,2 and no
> isolcpus boot.
> - With bf5b0c27fad2 reverted: isolcpus=domain,0,1 boots, 3 of 3.
> - With the patch: isolcpus=domain,0,1 boots, 3 of 3. So do
> isolcpus=domain,0, isolcpus=domain,1,2 and no isolcpus.
>
> After boot, the unbound kworkers have the same CPUs with the patch as
> with CONFIG_CPUSETS=y: CPUs 2 and 3 for isolcpus=domain,0,1. No "no
> longer affine" message is printed. The patch also builds with
> CONFIG_CPUSETS=y. That kernel boots with isolcpus=domain,0,1. I did
> not run other tests, nor other architectures.
>
> The analysis and the patch were written by an LLM coding assistant
> under my direction and review.
>
> Thanks,
>
> Mathieu
>
> [1] https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
>
> -- >8 --
> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Subject: sched: Fall back to all possible CPUs when no housekeeping CPU is online
>
> select_fallback_rq() picks a CPU for a task that has no allowed CPU
> online. Its last steps widen the mask of the task: first to its
> cpuset, then to task_cpu_fallback_mask(). Commit bf5b0c27fad2 ("sched:
> Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN") made that
> mask the domain housekeeping CPUs. The task then does not land on an
> isolated CPU. If none of these CPUs is online, the next step is BUG().
>
> That happens at boot when isolcpus=domain isolates the boot CPU.
> workqueue_init() runs before the other CPUs come online. It wakes a new
> unbound kworker, bound to the housekeeping CPUs. With CONFIG_CPUSETS=y,
> the cpuset step gives the kworker every CPU, and it runs on the boot
> CPU. With CONFIG_CPUSETS=n, that step does nothing, and the kernel
> panics at boot.
>
> Fix this by adding a last step before the BUG(): allow every CPU the
> task can run on, as task_cpu_possible_mask() gives. This was the last
> step before commit 3a5446612a3f ("sched,arm64: Handle CPU isolation on
> last resort fallback rq selection"). It is reached only when no
> housekeeping CPU is online. In testing, the kernel then boots. After
> boot, the unbound kworkers have the housekeeping CPUs, as with
> CONFIG_CPUSETS=y.
>
> Fixes: bf5b0c27fad2 ("sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
> kernel/sched/core.c | 15 ++++++++++++---
> 1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 1a088d4576f5..556a27f14619 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3599,7 +3599,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
> {
> int nid = cpu_to_node(cpu);
> const struct cpumask *nodemask = NULL;
> - enum { cpuset, possible, fail } state = cpuset;
> + enum { cpuset, housekeeping, possible, fail } state = cpuset;
> int dest_cpu;
>
> /*
> @@ -3630,12 +3630,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
> switch (state) {
> case cpuset:
> if (cpuset_cpus_allowed_fallback(p)) {
> - state = possible;
> + state = housekeeping;
> break;
> }
> fallthrough;
> - case possible:
> + case housekeeping:
> set_cpus_allowed_force(p, task_cpu_fallback_mask(p));
> + state = possible;
> + break;
> + case possible:
> + /*
> + * No housekeeping CPU is online. This happens at boot
> + * when isolcpus=domain isolates the boot CPU. Allow
> + * every CPU, as the cpuset fallback does.
> + */
> + set_cpus_allowed_force(p, task_cpu_possible_mask(p));
> state = fail;
> break;
> case fail:
prev parent reply other threads:[~2026-11-17 15:13 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 14:48 Mathieu Desnoyers
2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dd335ee6-c879-4bee-9ea8-6d9e9216fb0e@redhat.com \
--to=longman@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=frederic@kernel.org \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mgorman@suse.de \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®