Re: [RFC] x86/smp: reject hot-added CPU with conflicting APIC ID?

Re: [RFC] x86/smp: reject hot-added CPU with conflicting APIC ID?

[Partha Satapathy]

Adding the relevant maintainers,
whom I inadvertently omitted from my initial message.

On 07-08-2026 16:12, Partha Satapathy wrote:
> From: Partha Sarathi Satapathy <partha.satapathy@oracle.com>
> 
> Hello,
> 
> We are investigating recurring guest hangs following vCPU hotplug on a KVM/QEMU
> system. We would appreciate guidance on whether x86 CPU bring-up should reject
> a hot-added CPU when its runtime LAPIC ID differs from its ACPI/firmware APIC
> ID, especially when that runtime ID conflicts with an existing CPU.
> 
> Hypervisor system:
> 
>   Kernel: 5.15.0-318.199.3.5.el9uek.x86_64
>   CPU(s):                      384
>     On-line CPU(s) list:       0-383
>   Vendor ID:                   AuthenticAMD
>     BIOS Vendor ID:            Advanced Micro Devices, Inc.
>     Model name:                AMD EPYC 9J25 96-Core Processor
>   NUMA node(s): 2
>   NUMA node0 CPU(s): 0-95,192-287
>   NUMA node1 CPU(s): 96-191,288-383
> 
> Guest system:
> 
>   Kernel: 5.15.0-318.199.3.5.el9uek.x86_64
>   Machine type: QEMU Standard PC (Q35)
>   Guest CPU model: AMD EPYC 9J25 96-Core Processor
>   Guest topology: 2 NUMA nodes, 2 sockets, 95 cores/socket, 2 threads/core
>   Maximum vCPUs: 380
> 
> We observe this issue regularly on the same hypervisor platform.
> The guest topology has maxcpus=380. A group of vCPUs is offlined and then
> hot-added.
> 
> For CPU 17, ACPI/firmware assigns APIC ID 0x011 on node 0. During hot-add,
> however, the CPU reads its local APIC ID as 0x111:
> 
>   [64830.401824] smpboot: Booting Node 0 Processor 17 APIC 0x11
>   [64830.402255] [Firmware Bug]: CPU17: APIC id mismatch.
>                    Firmware: 11 APIC: 111
> 
> CPU bring-up then continues through start_secondary() and
> set_cpu_sibling_map(). The runtime APIC ID causes CPU 17 to be decoded as
> socket/die 1, core 8, thread 0. This collides with CPU 35, which has APIC ID
> 0x111, and results in:
> 
>   sched: CPU #17's smt-sibling CPU #34 is not on the same node!
>   [node: 0 != 1]. Ignoring dependency.
> 
> The CPU is allowed to proceed to the CPUHP online path despite both the APIC-ID
> mismatch and invalid topology.
> 
> Approximately 57 seconds later, the guest stalls globally:
> 
>   - migration/stopper threads wait in multi_cpu_stop() /
>     stop_machine_yield();
>   - workqueues and RCU stall;
>   - CPU 0 has a pending call-single-data request for CPU 17;
>   - CPU 17 does not provide a valid crash-NMI register note;
>   - eventually panics the guest.
> 
> We have observed multiple instances of the same issue.
> 
> Reference dmesg :
> [64830.401824] smpboot: Booting Node 0 Processor 17 APIC 0x11
> [64830.401985] kvm-clock: cpu 17, msr 1071a01441, secondary cpu clock
> [64830.402255] [Firmware Bug]: CPU17: APIC id mismatch. Firmware: 11 APIC: 111
> [64830.404886] ------------[ cut here ]------------
> [64830.404887] sched: CPU #17's smt-sibling CPU #34 is not on the same node! [node: 0 != 1]. Ignoring dependency.
> [64830.404899] WARNING: CPU: 17 PID: 0 at arch/x86/kernel/smpboot.c:434 topology_sane.isra.0+0x7f/0xa3
> [64830.404965]  <TASK>
> [64830.404967]  set_cpu_sibling_map+0x2cd/0x63a
> [64830.404970]  start_secondary+0x5b/0x152
> [64830.404971]  secondary_startup_64_no_verify+0xc3/0x0
> [64830.404976]  </TASK>
> [64830.404976] ---[ end trace 956bce4479c9bf6d ]---
> [64830.405334] kvm-guest: setup async PF for cpu 17
> [64830.405337] kvm-guest: stealtime: cpu 17, msr 1197a74080
> [64830.405470] Will online and init hotplugged CPU: 17
> [64830.651019] smpboot: Booting Node 1 Processor 35 APIC 0x111
> [64830.651133] kvm-clock: cpu 35, msr 1071a018c1, secondary cpu clock
> [64887.589563] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 56s!
> 
> 
> Questions:
> 
> 1. Is it intentional for x86 CPU hot-add to continue after this ACPI/runtime
>    APIC-ID mismatch?
> 
> 2. Should CPU bring-up fail if the runtime APIC ID conflicts with an already
>    present CPU or creates invalid SMT/package/NUMA topology?
> 
> 3. Is there an existing mainline validation path or known KVM/QEMU issue for
>    this case that we should use instead?
> 
> We are separately investigating why QEMU/KVM presents the incorrect LAPIC ID.
> That may be a hypervisor/firmware defect. The kernel robustness question is
> whether Linux should permit CPU hot-add to continue when inconsistent firmware
> and runtime APIC identities result in a conflicting logical CPU topology.
> 
> The relevant UEK code path is identify_secondary_cpu(), including
> validate_apic_and_package_id() in arch/x86/kernel/cpu/common.c, followed by
> set_cpu_sibling_map() in arch/x86/kernel/smpboot.c. We would appreciate
> guidance on the corresponding current upstream validation path and the
> expected behavior for this condition.
> 
> Thanks,
> Partha