* [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang @ 2026-08-21 21:11 Andreas Juch 2026-08-22 5:49 ` Joerg Roedel 0 siblings, 1 reply; 6+ messages in thread From: Andreas Juch @ 2026-08-21 21:11 UTC (permalink / raw) To: iommu Cc: Joerg Roedel, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Hello, I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. The older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 identified the following commit as the first bad commit: dc266f6c4e2628576ab33042b6b2a12480220026 iommu/amd: Fix premature break in init_iommu_one() The visible symptom is a black screen immediately after systemd-boot. The machine also remains unreachable over SSH and leaves no persistent journal entry for the failed boot. A reset or power cycle is required. I reverted that single commit, built the kernel again and the machine booted, so I'm quite sure that I found the right commit. System information: ASRockRack B550D4-4L BIOS P1.10 (06/28/2021) I'm no expert at all, but my research found that the following information might be relevant: journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD AmdTable 00000001 AMD 00000001) Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem 0xbb062000-0xbb0620cf] Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 Aug 21 22:24:17 kernel: iommu: Default domain type: Translated Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy mode Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, 0x0): PreF GA GA_vAPIC Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD .| 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 |AmdTable....AMD | 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 .........| 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 |..H...@...P.....| 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 |................| 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 |C...............| 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 |H.......H.......| 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 |H.........X...@.| 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 |..P.............| 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 |ZJ)".w..........| 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 |........C.......| 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 |........H.......| 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 |H.......H.......| 000000d0 The relevant part (after decoding with `iasl -d ivrs.dat`): [090h 0144 008h] EFR Image : 058F77EF22294A5A I'll provide any required details and test patches! Thanks for your work, it's very appreciated! Andreas ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang 2026-08-21 21:11 [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang Andreas Juch @ 2026-08-22 5:49 ` Joerg Roedel 2026-08-22 10:17 ` Vasant Hegde 0 siblings, 1 reply; 6+ messages in thread From: Joerg Roedel @ 2026-08-22 5:49 UTC (permalink / raw) To: Andreas Juch, Vasant Hegde Cc: iommu, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Adding Vasant. On Fri, Aug 21, 2026 at 11:11:07PM +0200, Andreas Juch wrote: > Hello, > > I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. > The > older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 > identified > the following commit as the first bad commit: > > dc266f6c4e2628576ab33042b6b2a12480220026 > iommu/amd: Fix premature break in init_iommu_one() > > The visible symptom is a black screen immediately after systemd-boot. The > machine also remains unreachable over SSH and leaves no persistent journal > entry > for the failed boot. A reset or power cycle is required. > > I reverted that single commit, built the kernel again and the machine > booted, so > I'm quite sure that I found the right commit. > > System information: > > ASRockRack B550D4-4L > BIOS P1.10 (06/28/2021) > > I'm no expert at all, but my research found that the following information > might > be relevant: > > journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' > Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD > AmdTable 00000001 AMD 00000001) > Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem > 0xbb062000-0xbb0620cf] > Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 > Aug 21 22:24:17 kernel: iommu: Default domain type: Translated > Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy > mode > Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 > Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 > Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 > Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 > Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 > Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 > Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 > Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 > Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 > Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 > Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 > Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 > Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 > Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 > Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 > Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 > Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 > Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, > 0x0): PreF GA GA_vAPIC > Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled > > And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: > > 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD > .| > 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 > |AmdTable....AMD | > 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 > .........| > 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 > |..H...@...P.....| > 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 > |................| > 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 > |C...............| > 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 > |H.......H.......| > 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 > |H.........X...@.| > 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 > |..P.............| > 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 > |ZJ)".w..........| > 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 > |........C.......| > 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 > |........H.......| > 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 > |H.......H.......| > 000000d0 > > The relevant part (after decoding with `iasl -d ivrs.dat`): > > [090h 0144 008h] EFR Image : 058F77EF22294A5A > > I'll provide any required details and test patches! > > Thanks for your work, it's very appreciated! > > Andreas > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang 2026-08-22 5:49 ` Joerg Roedel @ 2026-08-22 10:17 ` Vasant Hegde 2026-08-22 13:32 ` Andreas Juch 0 siblings, 1 reply; 6+ messages in thread From: Vasant Hegde @ 2026-08-22 10:17 UTC (permalink / raw) To: Joerg Roedel, Andreas Juch Cc: iommu, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Andreas, On 8/22/2026 11:19 AM, Joerg Roedel wrote: > Adding Vasant. > > On Fri, Aug 21, 2026 at 11:11:07PM +0200, Andreas Juch wrote: >> Hello, >> >> I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. >> The >> older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 >> identified >> the following commit as the first bad commit: >> >> dc266f6c4e2628576ab33042b6b2a12480220026 >> iommu/amd: Fix premature break in init_iommu_one() >> >> The visible symptom is a black screen immediately after systemd-boot. The >> machine also remains unreachable over SSH and leaves no persistent journal >> entry >> for the failed boot. A reset or power cycle is required. Interesting. I never imagined above fix will break some system! I assumed below dmesg is from good boot (without above patch) and did the analysis (code walk with below IVRS data): - EFR advertised by IVRS is different from actual EFR (read by IOMMU MMIO register). Its worse because IVRS advertised few features like GT/PPR which is not supported by IOMMU. - Without this patch we would have skipped early_iommu_features_init() so global EFR is zero and then init_iommu_one_late() set the EFR properly based on actual IOMMU capability. Only with this log, its hard to tell where exactly system failed. My guess is driver enabled features which are not supported (like GT/PPR) or we setup 32bit interrupt remapping table. I believe above patch exposed an existing bug with some other commit. Ideally we should properly fix late_iommu_features_init() and use actual EFRs. But for now lets fix the regression. Can you please try below fix? -Vasant ---<--- diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 69170146d442..7a5b78beb352 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -1930,19 +1930,20 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h, else iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET; + if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { + pr_warn_once("Host Address Translation is not supported.\n"); + amd_iommu_hatdis = true; + } + /* XT and GAM require GA mode. */ if ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0) { amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY; + break; } else { if (h->efr_reg & BIT(IOMMU_EFR_XTSUP_SHIFT)) amd_iommu_xt_mode = IRQ_REMAP_X2APIC_MODE; } - if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { - pr_warn_once("Host Address Translation is not supported.\n"); - amd_iommu_hatdis = true; - } - early_iommu_features_init(iommu, h); break; >> >> I reverted that single commit, built the kernel again and the machine >> booted, so >> I'm quite sure that I found the right commit. >> >> System information: >> >> ASRockRack B550D4-4L >> BIOS P1.10 (06/28/2021) >> >> I'm no expert at all, but my research found that the following information >> might >> be relevant: >> >> journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' >> Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD >> AmdTable 00000001 AMD 00000001) >> Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem >> 0xbb062000-0xbb0620cf] >> Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 >> Aug 21 22:24:17 kernel: iommu: Default domain type: Translated >> Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy >> mode >> Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 >> Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 >> Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 >> Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 >> Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 >> Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 >> Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 >> Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 >> Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 >> Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 >> Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 >> Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 >> Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 >> Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 >> Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 >> Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 >> Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 >> Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, >> 0x0): PreF GA GA_vAPIC >> Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled >> >> And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: >> >> 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD >> .| >> 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 >> |AmdTable....AMD | >> 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 >> .........| >> 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 >> |..H...@...P.....| >> 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 >> |................| >> 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 >> |C...............| >> 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 >> |H.......H.......| >> 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 >> |H.........X...@.| >> 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 >> |..P.............| >> 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 >> |ZJ)".w..........| >> 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 >> |........C.......| >> 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 >> |........H.......| >> 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 >> |H.......H.......| >> 000000d0 >> >> The relevant part (after decoding with `iasl -d ivrs.dat`): >> >> [090h 0144 008h] EFR Image : 058F77EF22294A5A >> >> I'll provide any required details and test patches! >> >> Thanks for your work, it's very appreciated! >> >> Andreas >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang 2026-08-22 10:17 ` Vasant Hegde @ 2026-08-22 13:32 ` Andreas Juch 2026-08-24 5:52 ` Vasant Hegde 0 siblings, 1 reply; 6+ messages in thread From: Andreas Juch @ 2026-08-22 13:32 UTC (permalink / raw) To: Vasant Hegde, Joerg Roedel Cc: iommu, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Hello Vasant, On 8/22/26 12:17, Vasant Hegde wrote: > Andreas, > > On 8/22/2026 11:19 AM, Joerg Roedel wrote: >> Adding Vasant. >> >> On Fri, Aug 21, 2026 at 11:11:07PM +0200, Andreas Juch wrote: >>> Hello, >>> >>> I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. >>> The >>> older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 >>> identified >>> the following commit as the first bad commit: >>> >>> dc266f6c4e2628576ab33042b6b2a12480220026 >>> iommu/amd: Fix premature break in init_iommu_one() >>> >>> The visible symptom is a black screen immediately after systemd-boot. The >>> machine also remains unreachable over SSH and leaves no persistent journal >>> entry >>> for the failed boot. A reset or power cycle is required. > Interesting. I never imagined above fix will break some system! > > I assumed below dmesg is from good boot (without above patch) and did the > analysis (code walk with below IVRS data): Yes, that's correct, it's from the .40 kernel without commit dc266f6c. > - EFR advertised by IVRS is different from actual EFR (read by IOMMU MMIO > register). Its worse because IVRS advertised few features like GT/PPR which is > not supported by IOMMU. > - Without this patch we would have skipped early_iommu_features_init() so > global EFR is zero and then init_iommu_one_late() set the EFR properly based on > actual IOMMU capability. > > Only with this log, its hard to tell where exactly system failed. My guess is > driver enabled features which are not supported (like GT/PPR) or we setup 32bit > interrupt remapping table. > > I believe above patch exposed an existing bug with some other commit. > > > Ideally we should properly fix late_iommu_features_init() and use actual EFRs. > But for now lets fix the regression. Can you please try below fix? > > -Vasant Thank you for investigating! I created a new branch from the v6.18.40 tag and applied your patch. The resulting kernel booted without any issues! If you need any further information or testing (for a proper fix later on), I'd be happy to test patches again! Thanks again! Andreas > > ---<--- > > diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c > index 69170146d442..7a5b78beb352 100644 > --- a/drivers/iommu/amd/init.c > +++ b/drivers/iommu/amd/init.c > @@ -1930,19 +1930,20 @@ static int __init init_iommu_one(struct amd_iommu > *iommu, struct ivhd_header *h, > else > iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET; > > + if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { > + pr_warn_once("Host Address Translation is not supported.\n"); > + amd_iommu_hatdis = true; > + } > + > /* XT and GAM require GA mode. */ > if ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0) { > amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY; > + break; > } else { > if (h->efr_reg & BIT(IOMMU_EFR_XTSUP_SHIFT)) > amd_iommu_xt_mode = IRQ_REMAP_X2APIC_MODE; > } > > - if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { > - pr_warn_once("Host Address Translation is not supported.\n"); > - amd_iommu_hatdis = true; > - } > - > early_iommu_features_init(iommu, h); > > break; > > > >>> I reverted that single commit, built the kernel again and the machine >>> booted, so >>> I'm quite sure that I found the right commit. >>> >>> System information: >>> >>> ASRockRack B550D4-4L >>> BIOS P1.10 (06/28/2021) >>> >>> I'm no expert at all, but my research found that the following information >>> might >>> be relevant: >>> >>> journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' >>> Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD >>> AmdTable 00000001 AMD 00000001) >>> Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem >>> 0xbb062000-0xbb0620cf] >>> Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 >>> Aug 21 22:24:17 kernel: iommu: Default domain type: Translated >>> Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy >>> mode >>> Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 >>> Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 >>> Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 >>> Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 >>> Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 >>> Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 >>> Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 >>> Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 >>> Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 >>> Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 >>> Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 >>> Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 >>> Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 >>> Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 >>> Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 >>> Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 >>> Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, >>> 0x0): PreF GA GA_vAPIC >>> Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled >>> >>> And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: >>> >>> 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD >>> .| >>> 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 >>> |AmdTable....AMD | >>> 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 >>> .........| >>> 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 >>> |..H...@...P.....| >>> 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 >>> |................| >>> 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 >>> |C...............| >>> 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 >>> |H.......H.......| >>> 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 >>> |H.........X...@.| >>> 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 >>> |..P.............| >>> 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 >>> |ZJ)".w..........| >>> 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 >>> |........C.......| >>> 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 >>> |........H.......| >>> 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 >>> |H.......H.......| >>> 000000d0 >>> >>> The relevant part (after decoding with `iasl -d ivrs.dat`): >>> >>> [090h 0144 008h] EFR Image : 058F77EF22294A5A >>> >>> I'll provide any required details and test patches! >>> >>> Thanks for your work, it's very appreciated! >>> >>> Andreas >>> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang 2026-08-22 13:32 ` Andreas Juch @ 2026-08-24 5:52 ` Vasant Hegde 2026-08-24 21:00 ` Andreas Juch 0 siblings, 1 reply; 6+ messages in thread From: Vasant Hegde @ 2026-08-24 5:52 UTC (permalink / raw) To: Andreas Juch, Joerg Roedel Cc: iommu, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Hl Andreas, On 8/22/2026 7:02 PM, Andreas Juch wrote: > [Some people who received this message don't often get email from > andreas@juch.cc. Learn why this is important at https://aka.ms/ > LearnAboutSenderIdentification ] > > Hello Vasant, > > On 8/22/26 12:17, Vasant Hegde wrote: >> Andreas, >> >> On 8/22/2026 11:19 AM, Joerg Roedel wrote: >>> Adding Vasant. >>> >>> On Fri, Aug 21, 2026 at 11:11:07PM +0200, Andreas Juch wrote: >>>> Hello, >>>> >>>> I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. >>>> The >>>> older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 >>>> identified >>>> the following commit as the first bad commit: >>>> >>>> dc266f6c4e2628576ab33042b6b2a12480220026 >>>> iommu/amd: Fix premature break in init_iommu_one() >>>> >>>> The visible symptom is a black screen immediately after systemd-boot. The >>>> machine also remains unreachable over SSH and leaves no persistent journal >>>> entry >>>> for the failed boot. A reset or power cycle is required. >> Interesting. I never imagined above fix will break some system! >> >> I assumed below dmesg is from good boot (without above patch) and did the >> analysis (code walk with below IVRS data): > > Yes, that's correct, it's from the .40 kernel without commit dc266f6c. > >> - EFR advertised by IVRS is different from actual EFR (read by IOMMU MMIO >> register). Its worse because IVRS advertised few features like GT/PPR which is >> not supported by IOMMU. >> - Without this patch we would have skipped early_iommu_features_init() so >> global EFR is zero and then init_iommu_one_late() set the EFR properly based on >> actual IOMMU capability. >> >> Only with this log, its hard to tell where exactly system failed. My guess is >> driver enabled features which are not supported (like GT/PPR) or we setup 32bit >> interrupt remapping table. >> >> I believe above patch exposed an existing bug with some other commit. >> >> >> Ideally we should properly fix late_iommu_features_init() and use actual EFRs. >> But for now lets fix the regression. Can you please try below fix? >> >> -Vasant > > Thank you for investigating! I created a new branch from the v6.18.40 > tag and > applied your patch. The resulting kernel booted without any issues! Thanks a lot. This is like another band-aid fix, but I think it's good enough for a stable backport. In my view, we should prioritize the IOMMU MMIO-advertised EFR over the one reported via IVRS. However, that would require much more extensive changes and isn't a good candidate for stable. > > If you need any further information or testing (for a proper fix later > on), I'd > be happy to test patches again! Can you help me to test below patch (just apply this patch and remove previously provided fix patch). I want to make sure that prioritizing MMIO EFR doesn't break things. -Vasant ---<--- commit aa5f50f1ee66ebf7a219906a4df6aa2b7cae9717 Author: Vasant Hegde <vasant.hegde@amd.com> Date: Mon Aug 24 11:12:27 2026 +0530 iommu/amd: Prioritize IOMMU MMIO advertised EFR over IVRS Signed-off-by: Vasant Hegde <vasant.hegde@amd.com> diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 69170146d442..0172db87b1df 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -2153,9 +2153,11 @@ static void __init late_iommu_features_init(struct amd_iommu *iommu) if (features != amd_iommu_efr || features2 != amd_iommu_efr2) { pr_warn(FW_WARN - "EFR mismatch. Use IVHD EFR (%#llx : %#llx), EFR2 (%#llx : %#llx).\n", + "EFR mismatch. Use MMIO EFR (%#llx : %#llx), EFR2 (%#llx : %#llx).\n", features, amd_iommu_efr, features2, amd_iommu_efr2); + amd_iommu_efr = features; + amd_iommu_efr2 = features2; } } > > Thanks again! > > Andreas > >> >> ---<--- >> >> diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c >> index 69170146d442..7a5b78beb352 100644 >> --- a/drivers/iommu/amd/init.c >> +++ b/drivers/iommu/amd/init.c >> @@ -1930,19 +1930,20 @@ static int __init init_iommu_one(struct amd_iommu >> *iommu, struct ivhd_header *h, >> else >> iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET; >> >> + if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { >> + pr_warn_once("Host Address Translation is not supported. >> \n"); >> + amd_iommu_hatdis = true; >> + } >> + >> /* XT and GAM require GA mode. */ >> if ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0) { >> amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY; >> + break; >> } else { >> if (h->efr_reg & BIT(IOMMU_EFR_XTSUP_SHIFT)) >> amd_iommu_xt_mode = IRQ_REMAP_X2APIC_MODE; >> } >> >> - if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { >> - pr_warn_once("Host Address Translation is not supported. >> \n"); >> - amd_iommu_hatdis = true; >> - } >> - >> early_iommu_features_init(iommu, h); >> >> break; >> >> >> >>>> I reverted that single commit, built the kernel again and the machine >>>> booted, so >>>> I'm quite sure that I found the right commit. >>>> >>>> System information: >>>> >>>> ASRockRack B550D4-4L >>>> BIOS P1.10 (06/28/2021) >>>> >>>> I'm no expert at all, but my research found that the following information >>>> might >>>> be relevant: >>>> >>>> journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' >>>> Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD >>>> AmdTable 00000001 AMD 00000001) >>>> Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem >>>> 0xbb062000-0xbb0620cf] >>>> Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 >>>> Aug 21 22:24:17 kernel: iommu: Default domain type: Translated >>>> Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy >>>> mode >>>> Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 >>>> Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 >>>> Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 >>>> Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 >>>> Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 >>>> Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 >>>> Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 >>>> Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 >>>> Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 >>>> Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 >>>> Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 >>>> Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 >>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 >>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 >>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 >>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 >>>> Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, >>>> 0x0): PreF GA GA_vAPIC >>>> Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled >>>> >>>> And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: >>>> >>>> 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD >>>> .| >>>> 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 >>>> |AmdTable....AMD | >>>> 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 >>>> .........| >>>> 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 >>>> |..H...@...P.....| >>>> 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 >>>> |................| >>>> 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 >>>> |C...............| >>>> 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 >>>> |H.......H.......| >>>> 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 >>>> |H.........X...@.| >>>> 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 >>>> |..P.............| >>>> 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 >>>> |ZJ)".w..........| >>>> 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 >>>> |........C.......| >>>> 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 >>>> |........H.......| >>>> 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 >>>> |H.......H.......| >>>> 000000d0 >>>> >>>> The relevant part (after decoding with `iasl -d ivrs.dat`): >>>> >>>> [090h 0144 008h] EFR Image : 058F77EF22294A5A >>>> >>>> I'll provide any required details and test patches! >>>> >>>> Thanks for your work, it's very appreciated! >>>> >>>> Andreas >>>> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang 2026-08-24 5:52 ` Vasant Hegde @ 2026-08-24 21:00 ` Andreas Juch 0 siblings, 0 replies; 6+ messages in thread From: Andreas Juch @ 2026-08-24 21:00 UTC (permalink / raw) To: Vasant Hegde, Joerg Roedel Cc: iommu, Suravee Suthikulpanit, Will Deacon, Robin Murphy, Robert Richter, linux-kernel, regressions Hi Vasant, On 8/24/26 07:52, Vasant Hegde wrote: > Hl Andreas, > > > On 8/22/2026 7:02 PM, Andreas Juch wrote: >> [Some people who received this message don't often get email from >> andreas@juch.cc. Learn why this is important at https://aka.ms/ >> LearnAboutSenderIdentification ] >> >> Hello Vasant, >> >> On 8/22/26 12:17, Vasant Hegde wrote: >>> Andreas, >>> >>> On 8/22/2026 11:19 AM, Joerg Roedel wrote: >>>> Adding Vasant. >>>> >>>> On Fri, Aug 21, 2026 at 11:11:07PM +0200, Andreas Juch wrote: >>>>> Hello, >>>>> >>>>> I noticed that one of my NixOS sytems failed to boot with kernel 6.18.40. >>>>> The >>>>> older 6.18.39 booted fine. A full git bisect of v6.18.39..v6.18.40 >>>>> identified >>>>> the following commit as the first bad commit: >>>>> >>>>> dc266f6c4e2628576ab33042b6b2a12480220026 >>>>> iommu/amd: Fix premature break in init_iommu_one() >>>>> >>>>> The visible symptom is a black screen immediately after systemd-boot. The >>>>> machine also remains unreachable over SSH and leaves no persistent journal >>>>> entry >>>>> for the failed boot. A reset or power cycle is required. >>> Interesting. I never imagined above fix will break some system! >>> >>> I assumed below dmesg is from good boot (without above patch) and did the >>> analysis (code walk with below IVRS data): >> Yes, that's correct, it's from the .40 kernel without commit dc266f6c. >> >>> - EFR advertised by IVRS is different from actual EFR (read by IOMMU MMIO >>> register). Its worse because IVRS advertised few features like GT/PPR which is >>> not supported by IOMMU. >>> - Without this patch we would have skipped early_iommu_features_init() so >>> global EFR is zero and then init_iommu_one_late() set the EFR properly based on >>> actual IOMMU capability. >>> >>> Only with this log, its hard to tell where exactly system failed. My guess is >>> driver enabled features which are not supported (like GT/PPR) or we setup 32bit >>> interrupt remapping table. >>> >>> I believe above patch exposed an existing bug with some other commit. >>> >>> >>> Ideally we should properly fix late_iommu_features_init() and use actual EFRs. >>> But for now lets fix the regression. Can you please try below fix? >>> >>> -Vasant >> Thank you for investigating! I created a new branch from the v6.18.40 >> tag and >> applied your patch. The resulting kernel booted without any issues! > > Thanks a lot. This is like another band-aid fix, but I think it's good enough > for a stable backport. In my view, we should prioritize the IOMMU > MMIO-advertised EFR over the one reported via IVRS. However, that would require > much more extensive changes and isn't a good candidate for stable. > > >> If you need any further information or testing (for a proper fix later >> on), I'd >> be happy to test patches again! > Can you help me to test below patch (just apply this patch and remove previously > provided fix patch). I want to make sure that prioritizing MMIO EFR doesn't > break things. I compiled the kernel with just your patch on top of the 6.18.40 tag and it didn't boot. Please let me know how I can provide better diagnostics. I tried to get serial output over IPMI SoL, but never got anything useful back. Andreas > -Vasant > > ---<--- > commit aa5f50f1ee66ebf7a219906a4df6aa2b7cae9717 > Author: Vasant Hegde <vasant.hegde@amd.com> > Date: Mon Aug 24 11:12:27 2026 +0530 > > iommu/amd: Prioritize IOMMU MMIO advertised EFR over IVRS > > Signed-off-by: Vasant Hegde <vasant.hegde@amd.com> > > diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c > index 69170146d442..0172db87b1df 100644 > --- a/drivers/iommu/amd/init.c > +++ b/drivers/iommu/amd/init.c > @@ -2153,9 +2153,11 @@ static void __init late_iommu_features_init(struct > amd_iommu *iommu) > if (features != amd_iommu_efr || > features2 != amd_iommu_efr2) { > pr_warn(FW_WARN > - "EFR mismatch. Use IVHD EFR (%#llx : %#llx), EFR2 (%#llx : %#llx).\n", > + "EFR mismatch. Use MMIO EFR (%#llx : %#llx), EFR2 (%#llx : %#llx).\n", > features, amd_iommu_efr, > features2, amd_iommu_efr2); > + amd_iommu_efr = features; > + amd_iommu_efr2 = features2; > } > } > > > > > > >> Thanks again! >> >> Andreas >> >>> ---<--- >>> >>> diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c >>> index 69170146d442..7a5b78beb352 100644 >>> --- a/drivers/iommu/amd/init.c >>> +++ b/drivers/iommu/amd/init.c >>> @@ -1930,19 +1930,20 @@ static int __init init_iommu_one(struct amd_iommu >>> *iommu, struct ivhd_header *h, >>> else >>> iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET; >>> >>> + if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { >>> + pr_warn_once("Host Address Translation is not supported. >>> \n"); >>> + amd_iommu_hatdis = true; >>> + } >>> + >>> /* XT and GAM require GA mode. */ >>> if ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0) { >>> amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY; >>> + break; >>> } else { >>> if (h->efr_reg & BIT(IOMMU_EFR_XTSUP_SHIFT)) >>> amd_iommu_xt_mode = IRQ_REMAP_X2APIC_MODE; >>> } >>> >>> - if (h->efr_attr & BIT(IOMMU_IVHD_ATTR_HATDIS_SHIFT)) { >>> - pr_warn_once("Host Address Translation is not supported. >>> \n"); >>> - amd_iommu_hatdis = true; >>> - } >>> - >>> early_iommu_features_init(iommu, h); >>> >>> break; >>> >>> >>> >>>>> I reverted that single commit, built the kernel again and the machine >>>>> booted, so >>>>> I'm quite sure that I found the right commit. >>>>> >>>>> System information: >>>>> >>>>> ASRockRack B550D4-4L >>>>> BIOS P1.10 (06/28/2021) >>>>> >>>>> I'm no expert at all, but my research found that the following information >>>>> might >>>>> be relevant: >>>>> >>>>> journalctl -W -b -k --no-pager | rg -i 'AMD-Vi|AMD IOMMU|IOMMU|IVRS|IVHD' >>>>> Aug 21 22:24:17 kernel: ACPI: IVRS 0x00000000BB062000 0000D0 (v02 AMD >>>>> AmdTable 00000001 AMD 00000001) >>>>> Aug 21 22:24:17 kernel: ACPI: Reserving IVRS table memory at [mem >>>>> 0xbb062000-0xbb0620cf] >>>>> Aug 21 22:24:17 kernel: AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0 >>>>> Aug 21 22:24:17 kernel: iommu: Default domain type: Translated >>>>> Aug 21 22:24:17 kernel: iommu: DMA domain TLB invalidation policy: lazy >>>>> mode >>>>> Aug 21 22:24:17 kernel: pci 0000:00:01.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:01.2: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:02.0: Adding to iommu group 1 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:03.0: Adding to iommu group 2 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:04.0: Adding to iommu group 3 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:05.0: Adding to iommu group 4 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:07.0: Adding to iommu group 5 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:07.1: Adding to iommu group 6 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:08.0: Adding to iommu group 7 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:08.1: Adding to iommu group 8 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:14.0: Adding to iommu group 9 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:14.3: Adding to iommu group 9 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.0: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.1: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.2: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.3: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.4: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.5: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.6: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:00:18.7: Adding to iommu group 10 >>>>> Aug 21 22:24:17 kernel: pci 0000:02:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:02:00.1: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:02:00.2: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:01.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:02.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:03.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:04.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:08.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:20:09.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:21:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:22:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:23:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:24:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:25:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:29:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:2a:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:2b:00.0: Adding to iommu group 0 >>>>> Aug 21 22:24:17 kernel: pci 0000:2c:00.0: Adding to iommu group 11 >>>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.0: Adding to iommu group 12 >>>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.1: Adding to iommu group 13 >>>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.3: Adding to iommu group 14 >>>>> Aug 21 22:24:17 kernel: pci 0000:2d:00.4: Adding to iommu group 15 >>>>> Aug 21 22:24:17 kernel: AMD-Vi: Extended features (0x58f77e022294881, >>>>> 0x0): PreF GA GA_vAPIC >>>>> Aug 21 22:24:17 kernel: AMD-Vi: Interrupt remapping enabled >>>>> >>>>> And the output of the decoded IVRS table, captured by `acpidump -n IVRS -b`: >>>>> >>>>> 00000000 49 56 52 53 d0 00 00 00 02 e2 41 4d 44 20 20 00 |IVRS......AMD >>>>> .| >>>>> 00000010 41 6d 64 54 61 62 6c 65 01 00 00 00 41 4d 44 20 >>>>> |AmdTable....AMD | >>>>> 00000020 01 00 00 00 41 30 20 00 00 00 00 00 00 00 00 00 |....A0 >>>>> .........| >>>>> 00000030 10 b0 48 00 02 00 40 00 00 00 50 fd 00 00 00 00 >>>>> |..H...@...P.....| >>>>> 00000040 00 00 00 00 2e 8f 04 80 03 08 00 00 04 fe ff 00 >>>>> |................| >>>>> 00000050 43 00 ff 00 00 a5 00 00 04 ff ff 00 00 00 00 00 >>>>> |C...............| >>>>> 00000060 48 00 00 00 00 a0 00 02 48 00 00 d7 0d a0 00 01 >>>>> |H.......H.......| >>>>> 00000070 48 00 00 00 0e 01 00 01 11 b0 58 00 02 00 40 00 >>>>> |H.........X...@.| >>>>> 00000080 00 00 50 fd 00 00 00 00 00 00 00 00 00 02 04 00 >>>>> |..P.............| >>>>> 00000090 5a 4a 29 22 ef 77 8f 05 00 00 00 00 00 00 00 00 >>>>> |ZJ)".w..........| >>>>> 000000a0 03 08 00 00 04 fe ff 00 43 00 ff 00 00 a5 00 00 >>>>> |........C.......| >>>>> 000000b0 04 ff ff 00 00 00 00 00 48 00 00 00 00 a0 00 02 >>>>> |........H.......| >>>>> 000000c0 48 00 00 d7 0d a0 00 01 48 00 00 00 0e 01 00 01 >>>>> |H.......H.......| >>>>> 000000d0 >>>>> >>>>> The relevant part (after decoding with `iasl -d ivrs.dat`): >>>>> >>>>> [090h 0144 008h] EFR Image : 058F77EF22294A5A >>>>> >>>>> I'll provide any required details and test patches! >>>>> >>>>> Thanks for your work, it's very appreciated! >>>>> >>>>> Andreas >>>>> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-08-24 21:00 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-08-21 21:11 [REGRESSION] iommu/amd: "Fix premature break in init_iommu_one()" causes early boot hang Andreas Juch 2026-08-22 5:49 ` Joerg Roedel 2026-08-22 10:17 ` Vasant Hegde 2026-08-22 13:32 ` Andreas Juch 2026-08-24 5:52 ` Vasant Hegde 2026-08-24 21:00 ` Andreas Juch
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®