From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C7E22253A0 for ; Sat, 22 Nov 2025 18:47:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763837238; cv=none; b=ZUa/Di4jxhrTkYSXqjYOu5IiAv/u+bTmDBDEffSPYSgpAy55tbo5gffX6SHtMyXeNB23DI69x5V0iG411eODr0Tbg34CjfwMpaO3mzjOhSLZoFoAQpNae3wRu/xiPh61eUMU25ggJrx5CBgj66g9YJITmF2533DhKQa+2lu8Fjk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763837238; c=relaxed/simple; bh=HhgAam5Vt7jzAat8XEnLjXOmqogU6rQu1+HAvltPBxE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=M5yliNLa9f2JISEpz6Yi3XE0WN/SoAxhiD8NeTKjw9oAlfoXfCJLS2Rn/itOluSJss23ZiUSwJUBdp8QspdsMs0zKrpaqN9kYNRgg5GLYkfd9tSJSevmsGF16jRpGGf4XuEhPdB52l4vL57CPHJXvA3bGC2NMl+wU2T2UbfIvsw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=E1z7tIA4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="E1z7tIA4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E0585C4CEF5; Sat, 22 Nov 2025 18:47:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763837237; bh=HhgAam5Vt7jzAat8XEnLjXOmqogU6rQu1+HAvltPBxE=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=E1z7tIA4RsUlUvTnoe8uqFVxnfmmTaKv8nh0YvQLnRdobBlOSPMT5GFnHL2sxp/Qa Qvuzq92ABRZBMI6O/+5j+YFoweHJhXZYfsvXXoWsTW8yCRLtNXVNI8WHsiTktrQ156 uM2kupUD6WJGzMHndS5F7wwRIWQmX/ty8/tQJlAfdFIst5ioT1cd2WniLjhFytxeMm nVqI+f5WLi0bY8hW5N9aDMJo/wqqX23KIMVlJNfdcZ07f3jZC4QsiBPXCVBAjouDnq mUTENgs2G2i8OyYrmmUPPM9ubdjh4dSKO+WfqLQbBazhj9nUsEAuiJsfdlk9cyPUI+ uZn1RsmUspR5A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 3D3A3CE0CFD; Sat, 22 Nov 2025 10:47:17 -0800 (PST) Date: Sat, 22 Nov 2025 10:47:17 -0800 From: "Paul E. McKenney" To: Marek Szyprowski Cc: Thomas Gleixner , LKML , Peter Zijlstra , Gabriele Monaco , Mathieu Desnoyers , Michael Jeanson , Jens Axboe , "Gautham R. Shenoy" , Florian Weimer , Tim Chen , Yury Norov , Shrikanth Hegde Subject: Re: [patch V5 09/20] cpumask: Cache num_possible_cpus() Message-ID: Reply-To: paulmck@kernel.org References: <20251119171016.815482037@linutronix.de> <20251119172549.578653738@linutronix.de> <89c7106e-a431-443a-9527-3d5fbce77fe1@samsung.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <89c7106e-a431-443a-9527-3d5fbce77fe1@samsung.com> On Fri, Nov 21, 2025 at 11:56:44PM +0100, Marek Szyprowski wrote: > Hi > > On 19.11.2025 18:27, Thomas Gleixner wrote: > > Reevaluating num_possible_cpus() over and over does not make sense. That > > becomes a constant after init as cpu_possible_mask is marked ro_after_init. > > > > Cache the value during initialization and provide that for consumption. > > > > Signed-off-by: Thomas Gleixner > > Reviewed-by: Yury Norov > > Reviewed-by: Mathieu Desnoyers > > Reviewed-by: Shrikanth Hegde > > This patch landed recently in linux-next as commit d0f23ccf6ba9 > ("cpumask: Cache num_possible_cpus()"). I found that it triggers the > following warning during boot on some of my test systems (namely > Raspberry Pi 3 and 4): I also bisected to this commit on an ARM Neoverse V2. It happens twice at about the same time, so the splats are interleaved, but they start with an almost-NULL pointer dereference during either TASKS01 or TRACE02 callback processing. Things go surprisingly normally after that, at least until a hang on shutdown. This sometimes happens quite late, as in *minutes* after boot. I do not see this on x86, nor on any rcutorture scenarios other than TASKS01 and TRACE02. These are unusual in that they are the only variants of the common-code tasks-RCU flavors that build with CONFIG_PROVE_LOCKING=y. My reproducer is: tools/testing/selftests/rcutorture/bin/torture.sh --duration 10 --do-none --do-rcutorture --configs-rcutorture "TRACE01 TRACE02" My current guess is that the snapshot is taken too early, though I would be more confident of that if it happened on TREE01, in which CPUs come online quite late. Thanx, Paul > kvm [1]: nv: 568 coarse grained trap handlers > kvm [1]: IPA Size Limit: 40 bits > ------------[ cut here ]------------ > WARNING: CPU: 1 PID: 1 at arch/arm64/kvm/vmid.c:183 > kvm_arm_vmid_alloc_init+0x98/0xac > Modules linked in: > CPU: 1 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.18.0-rc4+ #16195 PREEMPT > Hardware name: Raspberry Pi 3 Model B (DT) > pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : kvm_arm_vmid_alloc_init+0x98/0xac > lr : kvm_arm_vmid_alloc_init+0x1c/0xac > ... > Call trace: >  kvm_arm_vmid_alloc_init+0x98/0xac (P) >  kvm_arm_init+0x144/0x153c >  do_one_initcall+0x64/0x308 >  kernel_init_freeable+0x280/0x4fc >  kernel_init+0x20/0x1d8 >  ret_from_fork+0x10/0x20 > irq event stamp: 65606 > hardirqs last  enabled at (65605): [] > __up_console_sem+0x6c/0x80 > hardirqs last disabled at (65606): [] el1_brk64+0x20/0x60 > softirqs last  enabled at (65262): [] > handle_softirqs+0x4c4/0x4dc > softirqs last disabled at (65257): [] > __do_softirq+0x14/0x20 > ---[ end trace 0000000000000000 ]--- > kvm [1]: Hyp nVHE mode initialized successfully > > > Reverting it on top of linux-next fixes the issue. Let me know how can I > help debugging it. > > > > --- > > V4: Add comment why this is not marked __init .... > > V2: New patch > > --- > > include/linux/cpumask.h | 10 ++++++++-- > > kernel/cpu.c | 19 +++++++++++++++++++ > > 2 files changed, 27 insertions(+), 2 deletions(-) > > --- a/include/linux/cpumask.h > > +++ b/include/linux/cpumask.h > > @@ -126,6 +126,7 @@ extern struct cpumask __cpu_dying_mask; > > #define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) > > > > extern atomic_t __num_online_cpus; > > +extern unsigned int __num_possible_cpus; > > > > extern cpumask_t cpus_booted_once_mask; > > > > @@ -1152,13 +1153,13 @@ void init_cpu_possible(const struct cpum > > #define __assign_cpu(cpu, mask, val) \ > > __assign_bit(cpumask_check(cpu), cpumask_bits(mask), (val)) > > > > -#define set_cpu_possible(cpu, possible) assign_cpu((cpu), &__cpu_possible_mask, (possible)) > > #define set_cpu_enabled(cpu, enabled) assign_cpu((cpu), &__cpu_enabled_mask, (enabled)) > > #define set_cpu_present(cpu, present) assign_cpu((cpu), &__cpu_present_mask, (present)) > > #define set_cpu_active(cpu, active) assign_cpu((cpu), &__cpu_active_mask, (active)) > > #define set_cpu_dying(cpu, dying) assign_cpu((cpu), &__cpu_dying_mask, (dying)) > > > > void set_cpu_online(unsigned int cpu, bool online); > > +void set_cpu_possible(unsigned int cpu, bool possible); > > > > /** > > * to_cpumask - convert a NR_CPUS bitmap to a struct cpumask * > > @@ -1211,7 +1212,12 @@ static __always_inline unsigned int num_ > > { > > return raw_atomic_read(&__num_online_cpus); > > } > > -#define num_possible_cpus() cpumask_weight(cpu_possible_mask) > > + > > +static __always_inline unsigned int num_possible_cpus(void) > > +{ > > + return __num_possible_cpus; > > +} > > + > > #define num_enabled_cpus() cpumask_weight(cpu_enabled_mask) > > #define num_present_cpus() cpumask_weight(cpu_present_mask) > > #define num_active_cpus() cpumask_weight(cpu_active_mask) > > --- a/kernel/cpu.c > > +++ b/kernel/cpu.c > > @@ -3108,6 +3108,9 @@ EXPORT_SYMBOL(__cpu_dying_mask); > > atomic_t __num_online_cpus __read_mostly; > > EXPORT_SYMBOL(__num_online_cpus); > > > > +unsigned int __num_possible_cpus __ro_after_init = NR_CPUS; > > +EXPORT_SYMBOL(__num_possible_cpus); > > + > > void init_cpu_present(const struct cpumask *src) > > { > > cpumask_copy(&__cpu_present_mask, src); > > @@ -3116,6 +3119,7 @@ void init_cpu_present(const struct cpuma > > void init_cpu_possible(const struct cpumask *src) > > { > > cpumask_copy(&__cpu_possible_mask, src); > > + __num_possible_cpus = cpumask_weight(&__cpu_possible_mask); > > } > > > > void set_cpu_online(unsigned int cpu, bool online) > > @@ -3139,6 +3143,21 @@ void set_cpu_online(unsigned int cpu, bo > > } > > } > > > > +/* > > + * This should be marked __init, but there is a boatload of call sites > > + * which need to be fixed up to do so. Sigh... > > + */ > > +void set_cpu_possible(unsigned int cpu, bool possible) > > +{ > > + if (possible) { > > + if (!cpumask_test_and_set_cpu(cpu, &__cpu_possible_mask)) > > + __num_possible_cpus++; > > + } else { > > + if (cpumask_test_and_clear_cpu(cpu, &__cpu_possible_mask)) > > + __num_possible_cpus--; > > + } > > +} > > + > > /* > > * Activate the first processor. > > */ > > > > > > > Best regards > -- > Marek Szyprowski, PhD > Samsung R&D Institute Poland >