Re: [PATCH v3 8/8] PCI/quirks: Avoid certain address on Genoa systems

[Mario Limonciello <mario.limonciello@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 4409 bytes --]



On 10/13/26 08:09, Ilpo Järvinen wrote:
> While testing the resource placement changes, tests hit a case where
> igb fails to probe when BAR 0 is placed at 0x9c000000:
> 
>     90000000-9cffffff : PCI Bus 0000:a0
>    -  90000000-902fffff : PCI Bus 0000:a1
>    -    90000000-900fffff : 0000:a1:00.0
>    -      90000000-900fffff : igb
>    -    90100000-901fffff : 0000:a1:00.0
>    -    90200000-90203fff : 0000:a1:00.0
>    -      90200000-90203fff : igb
>    +  9be00000-9c0fffff : PCI Bus 0000:a1
>    +    9be00000-9befffff : 0000:a1:00.0
>    +    9bf00000-9bf03fff : 0000:a1:00.0
>    +    9c000000-9c0fffff : 0000:a1:00.0
>       9c100000-9c17ffff : amd_iommu
>       9c180000-9c1803ff : IOAPIC 8
> 
>    -       Region 0: Memory at 90000000 (32-bit, non-prefetchable) [size=1M]
>    -       Region 3: Memory at 90200000 (32-bit, non-prefetchable) [size=16K]
>    -       Expansion ROM at 90100000 [disabled] [size=1M]
>    +       Region 0: Memory at 9c000000 (32-bit, non-prefetchable) [size=1M]
>    +       Region 3: Memory at 9bf00000 (32-bit, non-prefetchable) [size=16K]
>    +       Expansion ROM at 9be00000 [disabled] [size=1M]
> 
> igb 0000:a1:00.0 0000:a1:00.0 (uninitialized): PCIe link lost
> ------------[ cut here ]------------
> igb: Failed to read reg 0x18!
> WARNING: drivers/net/ethernet/intel/igb/igb_main.c:724 at igb_rd32.cold+0x3c/0x4f [igb], CPU#32: kworker/32:1/706
> ...
>   igb_get_invariants_82575+0xff/0xf00 [igb]
>   igb_probe+0x3c8/0x1190 [igb]
>   local_pci_probe+0x3b/0x80
> 
> It turns out there is a 64kB iomem black hole at 9c000000 that returns
> ~0 and this is where igb's BAR 0 resides. If another BAR of the same
> card is placed into that address, it is similarly black holed.
> 
> Mark the problematic 64kB range reserved using a quirk bound to the
> bridge found in the problematic system.
> 
> Closes: https://bugzilla.kernel.org/show_bug.cgi?id=222074
> Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> ---
> Cc: Borislav Petkov <bp@alien8.de>
> 
> As an alternative to this, I've also tested reverting the commit
> 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") and
> that also resolves the issue by retaining "9c000000-9cffffff : Reserved"
> entry in the resource tree.
> 
> ---
>   arch/x86/pci/fixup.c | 36 ++++++++++++++++++++++++++++++++++++
>   1 file changed, 36 insertions(+)
> 
> diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
> index b301c6c8df75..cc463bf492b6 100644
> --- a/arch/x86/pci/fixup.c
> +++ b/arch/x86/pci/fixup.c
> @@ -822,6 +822,42 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1571, pci_amd_enable_64bit_bar);
>   DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);
>   DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar);
>   
> +/*
> + * TODO: Remove when/if root cause is found.
> + *
> + * Genoa appears to have a 64kB iomem black hole starting at 0x9c000000
> + * address for which all reads return ~0 blocking an overlapping BAR from
> + * working.
> + *
> + * Work around the problem by reserving the space prior to making any iomem
> + * allocations that could overlap with the black hole.
> + */
> +static struct resource genoa_black_hole = DEFINE_RES_MEM_NAMED(0x9c000000,
> +							       SZ_64K,
> +							       "reserved");
> +
> +static void genoa_iomem_black_hole(struct pci_dev *dev)
> +{
> +	struct resource *r;
> +
> +	pci_bus_for_each_resource(dev->bus, r) {
> +		if (!r || !r->flags || !resource_assigned(r))
> +			continue;
> +
> +		if (!__resource_contains_unbound(r, &genoa_black_hole))
> +			continue;
> +
> +		if (resource_assigned(&genoa_black_hole))
> +			pci_dbg(dev, "iomem black hole workaround already applied\n");
> +		else if (!insert_resource(r, &genoa_black_hole))
> +			pci_info(dev, "iomem black hole workaround enabled\n");
> +		else
> +			pci_dbg(dev, "iomem black hole workaround add failed\n");
> +		return;
> +	}
> +}
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0x14ab, genoa_iomem_black_hole);
> +
>   #define RS690_LOWER_TOP_OF_DRAM2	0x30
>   #define RS690_LOWER_TOP_OF_DRAM2_VALID	0x1
>   #define RS690_UPPER_TOP_OF_DRAM2	0x31

I guess it's good we know the root cause is 07eab0901ede. If we're going 
down the path of making holes in that we may well put the hole in 
efi_remove_e820_mmio() like this (attached).


[-- Attachment #2: 0001-efi-x86-Preserve-Genoa-MMIO-guard-regions-in-E820.patch --]
[-- Type: text/x-patch, Size: 3785 bytes --]

From 17d564bd829934c9627f6ff5e1cc10f301115406 Mon Sep 17 00:00:00 2001
From: Mario Limonciello <mario.limonciello@amd.com>
Date: Tue, 13 Oct 2026 11:54:33 -0500
Subject: [PATCH] efi/x86: Preserve Genoa MMIO guard regions in E820

Commit 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
removes every EFI MMIO descriptor of at least 256KB from the E820 map.

This makes PCI host bridge apertures described as EFI MMIO available
for BAR assignment, but assumes that all large descriptors are
allocatable.

On an AMD EPYC 9354 system, firmware supplies a 16MB EFI MMIO descriptor
at the end of each PCI root window.  The first 64KB at 0x9c000000 is not
described by ACPI and reads as all ones.  Removing the descriptor from
E820 allows PCI to assign an igb BAR there, which makes the device
unusable.  The IOMMU and IOAPIC regions farther into the descriptor are
reserved separately.

For Genoa, retain the first 64KB of each matching 16MB descriptor in
E820 while removing the remainder.  This keeps the inaccessible prefix
out of PCI allocation without hard-coding its physical address,
covers every root window, and preserves the BAR space made available
by 07eab0901ede.

Fixes: 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=222074
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 arch/x86/platform/efi/efi.c | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
index 0c39adb96b912..676dca172e351 100644
--- a/arch/x86/platform/efi/efi.c
+++ b/arch/x86/platform/efi/efi.c
@@ -44,10 +44,12 @@
 #include <linux/io.h>
 #include <linux/reboot.h>
 #include <linux/bcd.h>
+#include <linux/sizes.h>
 
 #include <asm/setup.h>
 #include <asm/efi.h>
 #include <asm/e820/api.h>
+#include <asm/processor.h>
 #include <asm/time.h>
 #include <asm/tlbflush.h>
 #include <asm/x86_init.h>
@@ -322,7 +324,7 @@ static void __init efi_clean_memmap(void)
 static void __init efi_remove_e820_mmio(void)
 {
 	efi_memory_desc_t *md;
-	u64 size, start, end;
+	u64 size, start, end, remove_start, remove_size;
 	int i = 0;
 
 	for_each_efi_memory_desc(md) {
@@ -330,10 +332,34 @@ static void __init efi_remove_e820_mmio(void)
 			size = md->num_pages << EFI_PAGE_SHIFT;
 			start = md->phys_addr;
 			end = start + size - 1;
-			if (size >= 256*1024) {
+			remove_start = start;
+			remove_size = size;
+
+			/*
+			 * Genoa firmware places a 16MB MMIO descriptor at the end
+			 * of every PCI root window.  The first 64KB of one such
+			 * descriptor is an undocumented hardware hole, while
+			 * resources such as the IOMMU and IOAPIC are described
+			 * separately farther into the descriptor.  Keep the prefix
+			 * of each such descriptor out of PCI allocation, but make
+			 * the rest available.
+			 */
+			if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD &&
+			    boot_cpu_data.x86 == 0x19 &&
+			    boot_cpu_data.x86_model >= 0x10 &&
+			    boot_cpu_data.x86_model <= 0x1f &&
+			    size == SZ_16M) {
+				pr_info("Preserve mem%02u: MMIO range=[0x%08llx-0x%08llx] (64KB) in e820 map\n",
+					i, start, start + SZ_64K - 1);
+				remove_start += SZ_64K;
+				remove_size -= SZ_64K;
+			}
+
+			if (size >= SZ_256K) {
 				pr_info("Remove mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluMB) from e820 map\n",
-					i, start, end, size >> 20);
-				e820__range_remove(start, size, E820_TYPE_RESERVED);
+					i, remove_start, end, remove_size >> 20);
+				e820__range_remove(remove_start, remove_size,
+						   E820_TYPE_RESERVED);
 			} else {
 				pr_info("Not removing mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluKB) from e820 map\n",
 					i, start, end, size >> 10);
-- 
2.43.0