Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Jinjie Ruan <ruanjinjie@huawei.com>]

On 6/12/2026 11:45 PM, Michael Kelley wrote:
> From: Jinjie Ruan <ruanjinjie@huawei.com> Sent: Thursday, June 11, 2026 6:38 AM
>>
>> Support for parallel secondary CPU bringup is already utilized by x86,
>> MIPS, and RISC-V. This patch brings this capability to the arm64
>> architecture.
>>
>> Rework the global `secondary_data` accessed during early boot into
>> a per-CPU array. This array maps logical CPU IDs to MPIDR_EL1 values,
>> enabling the early boot code in head.S to resolve each secondary CPU's
>> logical ID concurrently.
>>
>> To fully enable HOTPLUG_PARALLEL, this patch implements:
>> 1) An arm64-specific arch_cpuhp_kick_ap_alive() handler.
>> 2) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
>>
>> Successfully tested on QEMU ARM64 virt machine (KVM on, 128 vCPUs).
>>
>> |     test kernel	   | secondary CPUs boot time |
>> |  ---------------------   |	--------------------  |
>> |   Without this patch     |		155.672	      |
>> |   cpuhp.parallel=0	   |		62.897	      |
>> |   cpuhp.parallel=1	   |		166.703	      |
> 
> The last two rows seem mixed up. I would expect parallel=0 to
> result in a longer boot time.

Hi, Michael,

The results are correct and not mixed up.

Compared to the original non‑HOTPLUG_PARALLEL approach, the advantage of
cpuhp.parallel=0 lies in its use of cpu_relax(`yield` on arm64) instead
of the wait_for_completion_timeout() mechanism (which may cause sleep
and context switching). This significantly reduces the overhead of VM
exits and context switches in a KVM guest, thereby cutting the secondary
CPU boot time by more than half.

Regarding cpuhp.parallel=1, I believe the reason it fails to optimize
boot time is that when a large number of CPUs issue the KICK_AP call
simultaneously, it results in severe lock contention within KVM, which
paradoxically slows down secondary CPU bringup. However, this needs
further investigation into the PSCI_CPU_ON code in KVM.

I'm testing these performance aspects on physical hardware, so the
results might be somewhat different because secondary CPU bringup
requires trapping into the ATF firmware.

Best regards,
Jinjie

> 
> Michael
> 
>>
>> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/Kconfig           |  1 +
>>  arch/arm64/include/asm/smp.h |  8 ++++++++
>>  arch/arm64/kernel/head.S     | 23 +++++++++++++++++++++++
>>  arch/arm64/kernel/smp.c      | 27 +++++++++++++++++++++++++++
>>  4 files changed, 59 insertions(+)
>>
> 
>