Re: [PATCH] riscv: smp: fix non-SPINWAIT secondary hart rendezvous for fw_dynamic platforms

[Conor Dooley <conor@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

On Wed, Jul 29, 2026 at 01:34:14AM -0700, Paul Sherman wrote:
> On platforms where firmware (e.g. fw_dynamic) releases all harts to the
> same Linux entry point simultaneously, the hart selected by OpenSBI as
> Domain0 Boot HART is already in SBI_HSM_STATE_STARTED when Linux later
> attempts to bring it online as a secondary CPU via SBI HART_START.
> OpenSBI correctly returns SBI_ERR_ALREADY_STARTED, but Linux has no
> recovery path: with CONFIG_RISCV_BOOT_SPINWAIT=n, there is no secondary
> wait path in _start_kernel for harts that entered Linux directly from
> firmware, and sbi_cpu_start() has no handler for SBI_ERR_ALREADY_STARTED.
> 
> This causes one CPU to be permanently dropped per boot. The missing CPU
> is always the OpenSBI Domain0 Boot HART, which varies between boots on
> Sophgo SG2042 (hart 1, 2, or 3), explaining the apparent 'moving victim'.
> 
> Fix with three cooperating changes:
> 
> 1. Initialize boot_cpu_hartid to INVALID_HARTID instead of relying on
>    BSS zero-initialization. Without this, boot_cpu_hartid aliases with
>    hart 0, causing hart 0 to always appear to win the boot CPU race
>    regardless of which hart actually stored its hartid first.
> 
> 2. Add a non-SPINWAIT secondary wait path in _start_kernel. When
>    firmware releases multiple harts to the same entry point, non-primary
>    harts divert into the existing spinwait rendezvous arrays (previously
>    used only by CONFIG_RISCV_BOOT_SPINWAIT) and wait for cpu_start() to
>    provide boot data before proceeding to secondary startup.
> 
> 3. Handle SBI_ERR_ALREADY_STARTED (-EALREADY) in sbi_cpu_start(). When
>    HART_START returns -EALREADY, the hart is already executing in Linux
>    and spinning in .Lwait_for_cpu_up_sbi. Write the spinwait rendezvous
>    arrays to release it into secondary startup, matching the approach
>    used by cpu_ops_spinwait.c.
> 
> The arrays __cpu_spinwait_stack_pointer and __cpu_spinwait_task_pointer
> are defined unconditionally in cpu_ops_spinwait.c but their extern
> declarations in head.h were guarded by CONFIG_RISCV_BOOT_SPINWAIT.
> Move the declarations outside the guard since the arrays are always
> present and now used by both boot paths.
> 
> Note: The NR_CPUS bound check mirrors the identical pattern in
> cpu_ops_spinwait.c:32 which guards the same arrays against out-of-range
> hartids on platforms with discontiguous hart numbering.
> 
> Link: https://lore.kernel.org/linux-riscv/20260727221508.5179-1-shermanpauldylan@gmail.com/

> Tested-on: Milk-V Pioneer (Sophgo SG2042, 64-hart RISC-V, 4-NUMA nodes,
>            128GB DDR4, OpenSBI v1.5, Linux v7.2-rc5)
> Result: boot_cpu_hartid correctly reflects Domain0 Boot HART, all 64
>         CPUs online in 2.7 seconds (was 63 CPUs, boot always on hart 0)

FWIW, neither of these are valid tags, just put this information in the
commit message proper.

> Signed-off-by: Paul Sherman <shermanpauldylan@gmail.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]