mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Waiman Long <longman@redhat.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Frederic Weisbecker <frederic@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>
Cc: Michal Hocko <mhocko@suse.com>, Tejun Heo <tj@kernel.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Valentin Schneider <vschneid@redhat.com>,
	K Prateek Nayak <kprateek.nayak@amd.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [BUG] sched: BUG() in select_fallback_rq() at boot with isolcpus=domain on the boot CPU and CONFIG_CPUSETS=n
Date: Tue, 17 Nov 2026 10:13:02 -0500	[thread overview]
Message-ID: <dd335ee6-c879-4bee-9ea8-6d9e9216fb0e@redhat.com> (raw)
In-Reply-To: <20261117144817.265109-1-mathieu.desnoyers@efficios.com>

On 11/17/26 9:48 AM, Mathieu Desnoyers wrote:
> Hi,
>
> With CONFIG_CPUSETS=n, a kernel booted with isolcpus=domain,0 hits the
> BUG() in select_fallback_rq() during boot. CPU 0 is the boot CPU. I
> found it while testing a SCHED_DEADLINE series [1]. I then reproduced
> it on v7.4-rc2 alone. tip/master of 2026-11-17 and next-20261116 have
> the same code. A proposed fix follows the scissors line.
>
> Trigger
> =======
>
> Build x86-64 with CONFIG_CPUSETS=n. Boot a VM of 4 CPUs with
> isolcpus=domain,0 or isolcpus=domain,0,1. The kernel panics before
> init runs, in every boot.
>
> These boot fine:
> - the same kernel with isolcpus=domain,1,2, or without isolcpus;
> - a kernel with CONFIG_CPUSETS=y, with isolcpus=domain,0,1.

I am now thinking about disallowing the boot CPU from being domain 
isolated. That will include using isolcpus=domain boot time isolation 
parameter as well as in setting up isolated cpuset partition. Will that 
be a simpler solution than fixing all these places where isolating the 
boot CPU will cause various problems that need to be fixed?

Thanks,
Longman

> Log
> ===
>
>    kernel BUG at kernel/sched/core.c:3642!
>    Oops: invalid opcode: 0000 [#1] SMP NOPTI
>    CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.4.0-rc2-g7475ea8b5861 #1 PREEMPT(lazy)
>    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>    RIP: 0010:select_fallback_rq+0x1fd/0x350
>    Call Trace:
>     <TASK>
>     try_to_wake_up+0x299/0x9b0
>     create_worker+0x17d/0x200
>     workqueue_init+0x2bd/0x2f0
>     kernel_init_freeable+0x79/0x1f0
>     kernel_init+0x1a/0x130
>     ret_from_fork+0x22a/0x330
>     ret_from_fork_asm+0x1a/0x30
>     </TASK>
>    Kernel panic - not syncing: Fatal exception
>
> Analysis
> ========
>
> workqueue_init() runs before smp_init(), so only the boot CPU is
> online. create_worker() binds a new unbound kworker to the CPUs of its
> pool. Workqueue restricts unbound workers to the housekeeping CPUs. With
> isolcpus=domain,0,1, those are CPUs 2 and 3. Neither is online yet.
>
> The wakeup of the kworker then calls select_fallback_rq(). It finds no
> allowed CPU online, and widens the mask of the task in two steps:
> - cpuset_cpus_allowed_fallback(). With CONFIG_CPUSETS=y, it gives the
>    mask of the top cpuset, which holds every CPU. The kworker then runs
>    on CPU 0. With CONFIG_CPUSETS=n, it is a stub that returns false.
> - task_cpu_fallback_mask(), which is
>    housekeeping_cpumask(HK_TYPE_DOMAIN). That is CPUs 2 and 3 again.
>
> No step is left, and the loop reaches BUG().
>
> The last step used to allow all CPUs. 3a5446612a3f ("sched,arm64:
> Handle CPU isolation on last resort fallback rq selection"), in v6.14,
> changed it to the nohz_full housekeeping CPUs. With isolcpus=domain
> alone, those are all CPUs. bf5b0c27fad2 ("sched: Switch the fallback
> task allowed cpumask to HK_TYPE_DOMAIN"), in v7.0, changed it to the
> domain housekeeping CPUs. A kernel with bf5b0c27fad2 reverted boots
> with isolcpus=domain,0,1 and CONFIG_CPUSETS=n.
>
> Where to fix
> ============
>
> The patch adds a last step before the BUG(): task_cpu_possible_mask(),
> the mask the fallback used before 3a5446612a3f. It is reached only when
> no housekeeping CPU is online. The kworker then runs on the boot CPU, as
> with CONFIG_CPUSETS=y.
>
> Another option is to have workqueue_init() leave the unbound workers
> on the boot CPU until a housekeeping CPU is online. It would fix this
> path only. Other early kthreads with a housekeeping mask could hit the
> same BUG(). This is from reading.
>
> Testing
> =======
>
> I ran v7.4-rc2 in VMs with 4 CPUs, built with CONFIG_CPUSETS=n:
> - Without the patch: isolcpus=domain,0,1 hits the BUG() in 3 boots of
>    3, and isolcpus=domain,0 in 1 of 1. isolcpus=domain,1,2 and no
>    isolcpus boot.
> - With bf5b0c27fad2 reverted: isolcpus=domain,0,1 boots, 3 of 3.
> - With the patch: isolcpus=domain,0,1 boots, 3 of 3. So do
>    isolcpus=domain,0, isolcpus=domain,1,2 and no isolcpus.
>
> After boot, the unbound kworkers have the same CPUs with the patch as
> with CONFIG_CPUSETS=y: CPUs 2 and 3 for isolcpus=domain,0,1. No "no
> longer affine" message is printed. The patch also builds with
> CONFIG_CPUSETS=y. That kernel boots with isolcpus=domain,0,1. I did
> not run other tests, nor other architectures.
>
> The analysis and the patch were written by an LLM coding assistant
> under my direction and review.
>
> Thanks,
>
> Mathieu
>
> [1] https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
>
> -- >8 --
> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Subject: sched: Fall back to all possible CPUs when no housekeeping CPU is online
>
> select_fallback_rq() picks a CPU for a task that has no allowed CPU
> online. Its last steps widen the mask of the task: first to its
> cpuset, then to task_cpu_fallback_mask(). Commit bf5b0c27fad2 ("sched:
> Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN") made that
> mask the domain housekeeping CPUs. The task then does not land on an
> isolated CPU. If none of these CPUs is online, the next step is BUG().
>
> That happens at boot when isolcpus=domain isolates the boot CPU.
> workqueue_init() runs before the other CPUs come online. It wakes a new
> unbound kworker, bound to the housekeeping CPUs. With CONFIG_CPUSETS=y,
> the cpuset step gives the kworker every CPU, and it runs on the boot
> CPU. With CONFIG_CPUSETS=n, that step does nothing, and the kernel
> panics at boot.
>
> Fix this by adding a last step before the BUG(): allow every CPU the
> task can run on, as task_cpu_possible_mask() gives. This was the last
> step before commit 3a5446612a3f ("sched,arm64: Handle CPU isolation on
> last resort fallback rq selection"). It is reached only when no
> housekeeping CPU is online. In testing, the kernel then boots. After
> boot, the unbound kworkers have the housekeeping CPUs, as with
> CONFIG_CPUSETS=y.
>
> Fixes: bf5b0c27fad2 ("sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN")
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>   kernel/sched/core.c | 15 ++++++++++++---
>   1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 1a088d4576f5..556a27f14619 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3599,7 +3599,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
>   {
>   	int nid = cpu_to_node(cpu);
>   	const struct cpumask *nodemask = NULL;
> -	enum { cpuset, possible, fail } state = cpuset;
> +	enum { cpuset, housekeeping, possible, fail } state = cpuset;
>   	int dest_cpu;
>
>   	/*
> @@ -3630,12 +3630,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
>   		switch (state) {
>   		case cpuset:
>   			if (cpuset_cpus_allowed_fallback(p)) {
> -				state = possible;
> +				state = housekeeping;
>   				break;
>   			}
>   			fallthrough;
> -		case possible:
> +		case housekeeping:
>   			set_cpus_allowed_force(p, task_cpu_fallback_mask(p));
> +			state = possible;
> +			break;
> +		case possible:
> +			/*
> +			 * No housekeeping CPU is online. This happens at boot
> +			 * when isolcpus=domain isolates the boot CPU. Allow
> +			 * every CPU, as the cpuset fallback does.
> +			 */
> +			set_cpus_allowed_force(p, task_cpu_possible_mask(p));
>   			state = fail;
>   			break;
>   		case fail:


      parent reply	other threads:[~2026-11-17 15:13 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 14:48 Mathieu Desnoyers
2026-11-17 15:02 ` Peter Zijlstra
2026-11-17 15:13 ` Waiman Long [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=dd335ee6-c879-4bee-9ea8-6d9e9216fb0e@redhat.com \
    --to=longman@redhat.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=frederic@kernel.org \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mgorman@suse.de \
    --cc=mhocko@suse.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=tj@kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®