* [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon
@ 2025-06-12 7:29 K Prateek Nayak
2025-06-12 7:29 ` [PATCH 1/2] x86/cpu/topology: Always try cpu_parse_topology_ext() " K Prateek Nayak
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: K Prateek Nayak @ 2025-06-12 7:29 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel
Cc: H. Peter Anvin, Naveen rao, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit, K Prateek Nayak
When running an AMD guest on QEMU with > 255 cores, the following FW_BUG
was noticed with recent kernels:
[Firmware Bug]: CPU 512: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0200
Naveen, Sairaj debugged the cause to commit c749ce393b8f ("x86/cpu: Use
common topology code for AMD") where, after the rework, the initial
APICID was set using the CPUID leaf 0x8000001e EAX[31:0] as opposed to
the value from CPUID leaf 0xb EDX[31:0] previously.
This led us down a rabbit hole of XTOPOEXT vs TOPOEXT support, preferred
order of their parsing, and QEMU nuances like [1] where QEMU 0's out the
CPUID leaf 0x8000001e on CPUs where Core ID crosses 255 fearing a
Core ID collision in the 8 bit field which leads to the reported FW_BUG.
Following were major observations during the debug which the two
patches address respectively:
1. The support for CPUID leaf 0xb is independent of the TOPOEXT feature
and is rather linked to the x2APIC enablement. On baremetal, this has
not been a problem since TOPOEXT support (Fam 0x15 and above)
predates the support for CPUID leaf 0xb (Fam 0x17[Zen2] and above)
however, in virtualized environment, the support for x2APIC can be
enabled independent of topoext where QEMU expects the guest to parse
the topology and the APICID from CPUID leaf 0xb.
2. Since CPUID leaf 0x8000001e cannot represent Core ID without
collision for guests with > 255 cores, and QEMU 0's out the entire
leaf when Core ID crosses 255. Prefer initial APIC read from the
XTOPOEXT leaf before falling back to the APICID from 0x8000001e
which is still better than 8-bit APICID from leaf 0x1 EBX[31:24].
More details are enclosed in the commit logs.
Ideally, these changes should not affect baremetal AMD/Hygon platforms
as they have supported TOPOEXT long before the support for CPUID leaf
0xb and the extended CPUID leaf 0x80000026 (famous last words).
This series has been tested on baremetal Zen1 (contains topoext but not
0xb leaf), Zen3 (contains both topoext and 0xb leaf), and Zen4 (contains
topoext, 0xb leaf, and 0x80000026 leaf) servers with no changes
observed in "/sys/kernel/debug/x86/topo/" directory.
The series was also tested on 255 and 512 vCPU (each vCPU is an
individual core from QEMU topology being passed) EPYC-Genoa guest with
and without x2apic and topoext enabled and this series solves the FW_BUG
seen on guest with > 255 VCPUs. No changes observed in
"/sys/kernel/debug/x86/topo/" for all other cases without warning.
0xb leaf is provided unconditionally on these guests (with or without
topoext, even with x2apic disabled on guests with <= 255 vCPU).
Relevant bits of QEMU cmdline used during testing are as follows:
qemu-system-x86_64 \
-enable-kvm -m 32G -smp cpus=255,cores=255 \
-cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield,[-topoext] \
-machine q35,kernel_irqchip=split \
-global kvm-pit.lost_tick_policy=discard
...
References:
[1] https://github.com/qemu/qemu/commit/35ac5dfbcaa4b
Series is based on tip:x86/cpu at tag v6.15-rc6.
---
K Prateek Nayak (2):
x86/cpu/topology: Always try cpu_parse_topology_ext() on AMD/Hygon
x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
arch/x86/kernel/cpu/topology_amd.c | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
base-commit: 82f2b0b97b36ee3fcddf0f0780a9a0825d52fec3
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 1/2] x86/cpu/topology: Always try cpu_parse_topology_ext() on AMD/Hygon
2025-06-12 7:29 [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
@ 2025-06-12 7:29 ` K Prateek Nayak
2025-06-12 7:29 ` [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON K Prateek Nayak
2025-06-25 6:28 ` [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2 siblings, 0 replies; 7+ messages in thread
From: K Prateek Nayak @ 2025-06-12 7:29 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel
Cc: H. Peter Anvin, Naveen rao, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit, K Prateek Nayak
Support for parsing the topology on AMD/Hygon processors using CPUID
leaf 0xb was added in commit 3986a0a805e6 ("x86/CPU/AMD: Derive CPU
topology from CPUID function 0xB when available"). In an effort to keep
all the topology parsing bits in one place, this commit also introduced
a pseudo dependency on the TOPOEXT feature to parse the CPUID leaf 0xb.
TOPOEXT feature (CPUID 0x80000001 ECX[22]) advertises the support for
Cache Properties leaf 0x8000001d and the CPUID leaf 0x8000001e EAX for
"Extended APIC ID" however support for 0xb was introduced alongside the
x2APIC support not only on AMD [1], but also historically on x86 [2].
Similar to 0xb, the support for extended CPU topology leaf 0x80000026
too does not depend on the TOPOEXT feature. The support for these leaves
is expected to be confirmed by ensuring "leaf <= {extended_}cpuid_level"
and then parsing the level 0 of the respective leaf to confirm
ECX[15:8] (LevelType) is non-zero.
This has not been a problem on baremetal platforms since support for
TOPOEXT (Fam 0x15 and later) predates the support for CPUID leaf 0xb
(Fam 0x17[Zen2] and later), however, for AMD guests on QEMU, "x2apic"
feature can be enabled independent of the "topoext" feature where QEMU
expects topology and the initial APICID to be parsed using the CPUID
leaf 0xb (especially when number of cores > 255) which is populated
independent of the "topoext" feature flag.
Unconditionally call cpu_parse_topology_ext() on AMD and Hygon
processors to first parse the topology using the XTOPOEXT leaves before
using the TOPOEXT leaf.
Link: https://lore.kernel.org/lkml/1529686927-7665-1-git-send-email-suravee.suthikulpanit@amd.com/ [1]
Link: https://lore.kernel.org/lkml/20080818181435.523309000@linux-os.sc.intel.com/ [2]
Suggested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
arch/x86/kernel/cpu/topology_amd.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kernel/cpu/topology_amd.c b/arch/x86/kernel/cpu/topology_amd.c
index 03b3c9c3a45e..fec849fff82f 100644
--- a/arch/x86/kernel/cpu/topology_amd.c
+++ b/arch/x86/kernel/cpu/topology_amd.c
@@ -169,18 +169,14 @@ static void topoext_fixup(struct topo_scan *tscan)
static void parse_topology_amd(struct topo_scan *tscan)
{
- bool has_topoext = false;
-
/*
- * If the extended topology leaf 0x8000_001e is available
- * try to get SMT, CORE, TILE, and DIE shifts from extended
+ * Try to get SMT, CORE, TILE, and DIE shifts from extended
* CPUID leaf 0x8000_0026 on supported processors first. If
* extended CPUID leaf 0x8000_0026 is not supported, try to
* get SMT and CORE shift from leaf 0xb first, then try to
* get the CORE shift from leaf 0x8000_0008.
*/
- if (cpu_feature_enabled(X86_FEATURE_TOPOEXT))
- has_topoext = cpu_parse_topology_ext(tscan);
+ bool has_topoext = cpu_parse_topology_ext(tscan);
if (cpu_feature_enabled(X86_FEATURE_AMD_HETEROGENEOUS_CORES))
tscan->c->topo.cpu_type = cpuid_ebx(0x80000026);
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
2025-06-12 7:29 [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2025-06-12 7:29 ` [PATCH 1/2] x86/cpu/topology: Always try cpu_parse_topology_ext() " K Prateek Nayak
@ 2025-06-12 7:29 ` K Prateek Nayak
2025-06-18 4:00 ` Naveen N Rao
2025-06-25 6:28 ` [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2 siblings, 1 reply; 7+ messages in thread
From: K Prateek Nayak @ 2025-06-12 7:29 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel
Cc: H. Peter Anvin, Naveen rao, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit, K Prateek Nayak
Prior to the topology parsing rewrite and the switchover to the new
parsing logic for AMD processors in commit c749ce393b8f ("x86/cpu: Use
common topology code for AMD"), the "initial_apicid" on these platforms
was:
- First initialized to the LocalApicId from CPUID leaf 0x1 EBX[31:24].
- Then overwritten by the ExtendedLocalApicId in CPUID leaf 0xb
EDX[31:0] on processors that supported topoext.
With the new parsing flow introduced in commit f7fb3b2dd92c ("x86/cpu:
Provide an AMD/HYGON specific topology parser"), parse_8000_001e() now
unconditionally overwrites the "initial_apicid" already parsed during
cpu_parse_topology_ext().
Although this has not been a problem on baremetal platforms, on
virtualized AMD guests that feature more than 255 cores, QEMU 0's out
the CPUID leaf 0x8000001e on CPUs with "CoreID" > 255 to prevent
collision of these IDs in EBX[7:0] which can only represent a maximum of
255 cores [1].
This results in the following FW_BUG being logged when booting a guest
with more than 255 cores:
[Firmware Bug]: CPU 512: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0200
Rely on the APICID parsed during cpu_parse_topology_ext() from CPUID
leaf 0x80000026 or 0xb and only use the APICID from leaf 0x8000001e if
cpu_parse_topology_ext() failed (has_topoext is false).
On platforms that support the 0xb leaf (Zen2 or later, AMD guests on
QEMU) or the extended leaf 0x80000026 (Zen4 or later), the
"initial_apicid" is now set to the value parsed from EDX[31:0].
On older AMD/Hygon platforms that does not support the 0xb leaf but
supports the TOPOEXT extension (Fam 0x15, 0x16, 0x17[Zen1], and Hygon),
the current behavior is retained where "initial_apicid" is set using
the 0x8000001e leaf.
Link: https://github.com/qemu/qemu/commit/35ac5dfbcaa4b [1]
Debugged-by: Naveen N Rao (AMD) <naveen@kernel.org>
Debugged-by: Sairaj Kodilkar <sarunkod@amd.com>
Fixes: c749ce393b8f ("x86/cpu: Use common topology code for AMD")
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
arch/x86/kernel/cpu/topology_amd.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kernel/cpu/topology_amd.c b/arch/x86/kernel/cpu/topology_amd.c
index fec849fff82f..2ff6988e020a 100644
--- a/arch/x86/kernel/cpu/topology_amd.c
+++ b/arch/x86/kernel/cpu/topology_amd.c
@@ -80,7 +80,13 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_topoext)
cpuid_leaf(0x8000001e, &leaf);
- tscan->c->topo.initial_apicid = leaf.ext_apic_id;
+ /*
+ * Prefer initial_apicid parsed from CPUID leaf 0x8000026 or 0xb
+ * if available. Otherwise prefer the one from leaf 0x8000001e
+ * over 0x1.
+ */
+ if (!has_topoext)
+ tscan->c->topo.initial_apicid = leaf.ext_apic_id;
/*
* If leaf 0xb is available, then the domain shifts are set
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
2025-06-12 7:29 ` [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON K Prateek Nayak
@ 2025-06-18 4:00 ` Naveen N Rao
2025-06-18 5:26 ` K Prateek Nayak
0 siblings, 1 reply; 7+ messages in thread
From: Naveen N Rao @ 2025-06-18 4:00 UTC (permalink / raw)
To: K Prateek Nayak
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel, H. Peter Anvin, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit
On Thu, Jun 12, 2025 at 07:29:21AM +0000, K Prateek Nayak wrote:
> Prior to the topology parsing rewrite and the switchover to the new
> parsing logic for AMD processors in commit c749ce393b8f ("x86/cpu: Use
> common topology code for AMD"), the "initial_apicid" on these platforms
> was:
>
> - First initialized to the LocalApicId from CPUID leaf 0x1 EBX[31:24].
>
> - Then overwritten by the ExtendedLocalApicId in CPUID leaf 0xb
> EDX[31:0] on processors that supported topoext.
>
> With the new parsing flow introduced in commit f7fb3b2dd92c ("x86/cpu:
> Provide an AMD/HYGON specific topology parser"), parse_8000_001e() now
> unconditionally overwrites the "initial_apicid" already parsed during
> cpu_parse_topology_ext().
>
> Although this has not been a problem on baremetal platforms, on
> virtualized AMD guests that feature more than 255 cores, QEMU 0's out
> the CPUID leaf 0x8000001e on CPUs with "CoreID" > 255 to prevent
> collision of these IDs in EBX[7:0] which can only represent a maximum of
> 255 cores [1].
>
> This results in the following FW_BUG being logged when booting a guest
> with more than 255 cores:
>
> [Firmware Bug]: CPU 512: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0200
>
> Rely on the APICID parsed during cpu_parse_topology_ext() from CPUID
> leaf 0x80000026 or 0xb and only use the APICID from leaf 0x8000001e if
> cpu_parse_topology_ext() failed (has_topoext is false).
>
> On platforms that support the 0xb leaf (Zen2 or later, AMD guests on
> QEMU) or the extended leaf 0x80000026 (Zen4 or later), the
> "initial_apicid" is now set to the value parsed from EDX[31:0].
>
> On older AMD/Hygon platforms that does not support the 0xb leaf but
> supports the TOPOEXT extension (Fam 0x15, 0x16, 0x17[Zen1], and Hygon),
> the current behavior is retained where "initial_apicid" is set using
> the 0x8000001e leaf.
>
> Link: https://github.com/qemu/qemu/commit/35ac5dfbcaa4b [1]
> Debugged-by: Naveen N Rao (AMD) <naveen@kernel.org>
> Debugged-by: Sairaj Kodilkar <sarunkod@amd.com>
> Fixes: c749ce393b8f ("x86/cpu: Use common topology code for AMD")
> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
> ---
> arch/x86/kernel/cpu/topology_amd.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kernel/cpu/topology_amd.c b/arch/x86/kernel/cpu/topology_amd.c
> index fec849fff82f..2ff6988e020a 100644
> --- a/arch/x86/kernel/cpu/topology_amd.c
> +++ b/arch/x86/kernel/cpu/topology_amd.c
> @@ -80,7 +80,13 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_topoext)
>
> cpuid_leaf(0x8000001e, &leaf);
>
> - tscan->c->topo.initial_apicid = leaf.ext_apic_id;
> + /*
> + * Prefer initial_apicid parsed from CPUID leaf 0x8000026 or 0xb
> + * if available. Otherwise prefer the one from leaf 0x8000001e
> + * over 0x1.
> + */
> + if (!has_topoext)
> + tscan->c->topo.initial_apicid = leaf.ext_apic_id;
My understanding is that the rest of this function continues to work
properly with an all-zero return value from CPUID leaf 0x8000001e - data
from topoext is preferred where available, and other sources are used
for llc_id.
And, this fixes the firmware bug warning seen during a guest boot in
qemu for me, so for this patch:
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Thanks,
Naveen
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
2025-06-18 4:00 ` Naveen N Rao
@ 2025-06-18 5:26 ` K Prateek Nayak
0 siblings, 0 replies; 7+ messages in thread
From: K Prateek Nayak @ 2025-06-18 5:26 UTC (permalink / raw)
To: Naveen N Rao
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel, H. Peter Anvin, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit
Hello Naveen,
On 6/18/2025 9:30 AM, Naveen N Rao wrote:
>> diff --git a/arch/x86/kernel/cpu/topology_amd.c b/arch/x86/kernel/cpu/topology_amd.c
>> index fec849fff82f..2ff6988e020a 100644
>> --- a/arch/x86/kernel/cpu/topology_amd.c
>> +++ b/arch/x86/kernel/cpu/topology_amd.c
>> @@ -80,7 +80,13 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_topoext)
>>
>> cpuid_leaf(0x8000001e, &leaf);
>>
>> - tscan->c->topo.initial_apicid = leaf.ext_apic_id;
>> + /*
>> + * Prefer initial_apicid parsed from CPUID leaf 0x8000026 or 0xb
>> + * if available. Otherwise prefer the one from leaf 0x8000001e
>> + * over 0x1.
>> + */
>> + if (!has_topoext)
>> + tscan->c->topo.initial_apicid = leaf.ext_apic_id;
>
> My understanding is that the rest of this function continues to work
> properly with an all-zero return value from CPUID leaf 0x8000001e - data
> from topoext is preferred where available, and other sources are used
> for llc_id.
Yes. QEMU just errs on the side of caution for CoreID collision in
0x8000001e and "NumSharingCache" in 0x8000001d EAX[25:14] can still
represent 2^12 (4096) CPUs in each cache domain which is larger than
the total number of vCPUs possible currently I believe.
> And, this fixes the firmware bug warning seen during a guest boot in
> qemu for me, so for this patch:
> Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Thank you for debugging the issue and testing :)
--
Thanks and Regards,
Prateek
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon
2025-06-12 7:29 [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2025-06-12 7:29 ` [PATCH 1/2] x86/cpu/topology: Always try cpu_parse_topology_ext() " K Prateek Nayak
2025-06-12 7:29 ` [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON K Prateek Nayak
@ 2025-06-25 6:28 ` K Prateek Nayak
2025-07-02 8:27 ` K Prateek Nayak
2 siblings, 1 reply; 7+ messages in thread
From: K Prateek Nayak @ 2025-06-25 6:28 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel
Cc: H. Peter Anvin, Naveen rao, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit
On 6/12/2025 12:59 PM, K Prateek Nayak wrote:
> When running an AMD guest on QEMU with > 255 cores, the following FW_BUG
> was noticed with recent kernels:
>
> [Firmware Bug]: CPU 512: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0200
>
> Naveen, Sairaj debugged the cause to commit c749ce393b8f ("x86/cpu: Use
> common topology code for AMD") where, after the rework, the initial
> APICID was set using the CPUID leaf 0x8000001e EAX[31:0] as opposed to
> the value from CPUID leaf 0xb EDX[31:0] previously.
>
> This led us down a rabbit hole of XTOPOEXT vs TOPOEXT support, preferred
> order of their parsing, and QEMU nuances like [1] where QEMU 0's out the
> CPUID leaf 0x8000001e on CPUs where Core ID crosses 255 fearing a
> Core ID collision in the 8 bit field which leads to the reported FW_BUG.
>
> Following were major observations during the debug which the two
> patches address respectively:
>
> 1. The support for CPUID leaf 0xb is independent of the TOPOEXT feature
> and is rather linked to the x2APIC enablement. On baremetal, this has
> not been a problem since TOPOEXT support (Fam 0x15 and above)
> predates the support for CPUID leaf 0xb (Fam 0x17[Zen2] and above)
> however, in virtualized environment, the support for x2APIC can be
> enabled independent of topoext where QEMU expects the guest to parse
> the topology and the APICID from CPUID leaf 0xb.
>
> 2. Since CPUID leaf 0x8000001e cannot represent Core ID without
> collision for guests with > 255 cores, and QEMU 0's out the entire
> leaf when Core ID crosses 255. Prefer initial APIC read from the
> XTOPOEXT leaf before falling back to the APICID from 0x8000001e
> which is still better than 8-bit APICID from leaf 0x1 EBX[31:24].
>
> More details are enclosed in the commit logs.
>
> Ideally, these changes should not affect baremetal AMD/Hygon platforms
> as they have supported TOPOEXT long before the support for CPUID leaf
> 0xb and the extended CPUID leaf 0x80000026 (famous last words).
>
> This series has been tested on baremetal Zen1 (contains topoext but not
> 0xb leaf), Zen3 (contains both topoext and 0xb leaf), and Zen4 (contains
> topoext, 0xb leaf, and 0x80000026 leaf) servers with no changes
> observed in "/sys/kernel/debug/x86/topo/" directory.
>
> The series was also tested on 255 and 512 vCPU (each vCPU is an
> individual core from QEMU topology being passed) EPYC-Genoa guest with
> and without x2apic and topoext enabled and this series solves the FW_BUG
> seen on guest with > 255 VCPUs. No changes observed in
> "/sys/kernel/debug/x86/topo/" for all other cases without warning.
> 0xb leaf is provided unconditionally on these guests (with or without
> topoext, even with x2apic disabled on guests with <= 255 vCPU).
>
> Relevant bits of QEMU cmdline used during testing are as follows:
>
> qemu-system-x86_64 \
> -enable-kvm -m 32G -smp cpus=255,cores=255 \
> -cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield,[-topoext] \
> -machine q35,kernel_irqchip=split \
> -global kvm-pit.lost_tick_policy=discard
> ...
>
> References:
>
> [1] https://github.com/qemu/qemu/commit/35ac5dfbcaa4b
>
> Series is based on tip:x86/cpu at tag v6.15-rc6.
>
> ---
> K Prateek Nayak (2):
> x86/cpu/topology: Always try cpu_parse_topology_ext() on AMD/Hygon
> x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
>
> arch/x86/kernel/cpu/topology_amd.c | 16 +++++++++-------
> 1 file changed, 9 insertions(+), 7 deletions(-)
>
>
> base-commit: 82f2b0b97b36ee3fcddf0f0780a9a0825d52fec3
Gentle ping!
--
Thanks and Regards,
Prateek
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon
2025-06-25 6:28 ` [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
@ 2025-07-02 8:27 ` K Prateek Nayak
0 siblings, 0 replies; 7+ messages in thread
From: K Prateek Nayak @ 2025-07-02 8:27 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
linux-kernel
Cc: H. Peter Anvin, Naveen rao, Sairaj Kodilkar, Mario Limonciello,
Pawan Gupta, Gautham R. Shenoy, Babu Moger,
Suravee Suthikulpanit
On 6/25/2025 11:58 AM, K Prateek Nayak wrote:
> On 6/12/2025 12:59 PM, K Prateek Nayak wrote:
>> When running an AMD guest on QEMU with > 255 cores, the following FW_BUG
>> was noticed with recent kernels:
>>
>> [Firmware Bug]: CPU 512: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0200
>>
>> Naveen, Sairaj debugged the cause to commit c749ce393b8f ("x86/cpu: Use
>> common topology code for AMD") where, after the rework, the initial
>> APICID was set using the CPUID leaf 0x8000001e EAX[31:0] as opposed to
>> the value from CPUID leaf 0xb EDX[31:0] previously.
>>
>> This led us down a rabbit hole of XTOPOEXT vs TOPOEXT support, preferred
>> order of their parsing, and QEMU nuances like [1] where QEMU 0's out the
>> CPUID leaf 0x8000001e on CPUs where Core ID crosses 255 fearing a
>> Core ID collision in the 8 bit field which leads to the reported FW_BUG.
>>
>> Following were major observations during the debug which the two
>> patches address respectively:
>>
>> 1. The support for CPUID leaf 0xb is independent of the TOPOEXT feature
>> and is rather linked to the x2APIC enablement. On baremetal, this has
>> not been a problem since TOPOEXT support (Fam 0x15 and above)
>> predates the support for CPUID leaf 0xb (Fam 0x17[Zen2] and above)
>> however, in virtualized environment, the support for x2APIC can be
>> enabled independent of topoext where QEMU expects the guest to parse
>> the topology and the APICID from CPUID leaf 0xb.
>>
>> 2. Since CPUID leaf 0x8000001e cannot represent Core ID without
>> collision for guests with > 255 cores, and QEMU 0's out the entire
>> leaf when Core ID crosses 255. Prefer initial APIC read from the
>> XTOPOEXT leaf before falling back to the APICID from 0x8000001e
>> which is still better than 8-bit APICID from leaf 0x1 EBX[31:24].
>>
>> More details are enclosed in the commit logs.
>>
>> Ideally, these changes should not affect baremetal AMD/Hygon platforms
>> as they have supported TOPOEXT long before the support for CPUID leaf
>> 0xb and the extended CPUID leaf 0x80000026 (famous last words).
>>
>> This series has been tested on baremetal Zen1 (contains topoext but not
>> 0xb leaf), Zen3 (contains both topoext and 0xb leaf), and Zen4 (contains
>> topoext, 0xb leaf, and 0x80000026 leaf) servers with no changes
>> observed in "/sys/kernel/debug/x86/topo/" directory.
>>
>> The series was also tested on 255 and 512 vCPU (each vCPU is an
>> individual core from QEMU topology being passed) EPYC-Genoa guest with
>> and without x2apic and topoext enabled and this series solves the FW_BUG
>> seen on guest with > 255 VCPUs. No changes observed in
>> "/sys/kernel/debug/x86/topo/" for all other cases without warning.
>> 0xb leaf is provided unconditionally on these guests (with or without
>> topoext, even with x2apic disabled on guests with <= 255 vCPU).
>>
>> Relevant bits of QEMU cmdline used during testing are as follows:
>>
>> qemu-system-x86_64 \
>> -enable-kvm -m 32G -smp cpus=255,cores=255 \
>> -cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield,[-topoext] \
>> -machine q35,kernel_irqchip=split \
>> -global kvm-pit.lost_tick_policy=discard
>> ...
>>
>> References:
>>
>> [1] https://github.com/qemu/qemu/commit/35ac5dfbcaa4b
>>
>> Series is based on tip:x86/cpu at tag v6.15-rc6.
>>
>> ---
>> K Prateek Nayak (2):
>> x86/cpu/topology: Always try cpu_parse_topology_ext() on AMD/Hygon
>> x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON
>>
>> arch/x86/kernel/cpu/topology_amd.c | 16 +++++++++-------
>> 1 file changed, 9 insertions(+), 7 deletions(-)
>>
>>
>> base-commit: 82f2b0b97b36ee3fcddf0f0780a9a0825d52fec3
>
> Gentle ping!
Gentle ping! But a wee bit less gentle compared to last time :)
--
Thanks and Regards,
Prateek
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-07-02 8:27 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-06-12 7:29 [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2025-06-12 7:29 ` [PATCH 1/2] x86/cpu/topology: Always try cpu_parse_topology_ext() " K Prateek Nayak
2025-06-12 7:29 ` [PATCH 2/2] x86/cpu/topology: Use initial APICID from XTOPOEXT on AMD/HYGON K Prateek Nayak
2025-06-18 4:00 ` Naveen N Rao
2025-06-18 5:26 ` K Prateek Nayak
2025-06-25 6:28 ` [PATCH 0/2] x86/cpu/topology: Work around the nuances of virtualization on AMD/Hygon K Prateek Nayak
2025-07-02 8:27 ` K Prateek Nayak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®