[PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[Breno Leitao]

I've found that kdump does not work on some ARM host, such as AWS
Graviton metal, and the SMMU is what seems to kills it.

The capture kernel dies about a second after it resets the SMMU, part
way through PCI enumeration, and takes the machine with it.

There is no panic and no oops: the console stops mid-line and the box
reboots, leaving no vmcore, no log and nothing on disk. Crashes on this
hardware mostly produce no dump at all.

This is my theory, from an "outsider view":

1) A crash kexec does not run .device_shutdown(), so the capture
   kernel boots on a machine whose devices are still running.

2) They keep translating through the stream table the crashed kernel
   programmed, which sits in memory the capture kernel never touches.

3) arm_smmu_device_reset() installs a fresh table with no entry for
   those StreamIDs, so their in-flight DMA raises C_BAD_STREAMID. (seen
   when I cleared CR0_EVTQEN)

4) When the platform reports hardware errors to firmware first,
   firmware answers by resetting the machine.

If my  kdump initrd has no driver for the two ENA NICs, so nothing
resets them and they keep DMAing for as long as the capture kernel runs.

Things that will not crash/interrupt kdump:

   a) Get the ENA driver NIC into the kdump. This is costly.
   b) Removing ENA driver (or PCI device) before the crash

Debugging further, I  found that faults are invisible by default,
because the capture kernel clears CR0_EVTQEN. With the event queue left
enabled, one device accounts for all of them:

  [   36.191293] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting...
  [   36.418009] event: C_BAD_STREAMID client: (unassigned sid) sid: 0x32d00 ssid: 0x0
  [   36.706345] event: C_BAD_STREAMID client: (unassigned sid) sid: 0x32d00 ssid: 0x0
  [   37.313703] pci 0003:02:02.2: Adding

with the platform resetting mid-line. sid 0x32d00 is 0003:2d:00.0, one of
ENA NICs.

My lovely robot and I came with two patches that solved the problem,
from a practical perspective and I want to share what has been tested.

Patch 1 carries the previous kernel's L1 descriptors into the new KDUMP table, so
those streams keep translating.

Patch 2 stops the core discarding that again: iommu_dma_init() already
enables deferred attach for a capture kernel, but the core only honours
it for drivers implementing .is_attach_deferred, which until now meant
amd and intel. Without it iommu_setup_default_domain() attaches
a default domain at probe and the fault returns as F_TRANSLATION.

The x86 IOMMU drivers solve the same/similar problem the same way; see
commit 38e5f33ee3596 ("iommu/amd: Reuse device table for kdump").

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (2):
      iommu/arm-smmu-v3: inherit the previous kernel's stream table in kdump
      iommu/arm-smmu-v3: defer attach in kdump so inherited entries survive probe

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 104 ++++++++++++++++++++++++----
 1 file changed, 91 insertions(+), 13 deletions(-)
---
base-commit: f2bfbc3554ca6919484030729424b9dee2942d24
change-id: 20260911-smmu_fix_aws-95f486d8ee5d

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH 1/2] iommu/arm-smmu-v3: inherit the previous kernel's stream table in kdump

[Breno Leitao]

A crash kexec does not run device_shutdown(), so the capture kernel boots
on a machine whose devices are still running. Whatever the crashed kernel
left DMAing keeps translating through the stream table it programmed,
because that table sits in memory the capture kernel never touches.

arm_smmu_device_reset() installs a fresh stream table instead, and that
table has no entry for those StreamIDs. On an Amazon EC2 m8g.metal-24xl
the two ENA NICs have no driver in the kdump initrd, so nothing resets
them and they keep DMAing for as long as the capture kernel runs:

  [   36.192904] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting...
  [   36.491369] arm-smmu-v3 arm-smmu-v3.0.auto: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x32d00 ssid: 0x0

StreamID 0x32d00 is 0003:2d:00.0, one of those NICs. The platform reports
hardware errors to firmware first:

  GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.

so firmware answers the fault by resetting the instance, and no dump is
written.

Read the previous stream table out of STRTAB_BASE before the SMMU is
disabled and copy its L1 descriptors into the new table.

Those descriptors still point at the previous kernel's L2 tables, so
arm_smmu_init_l2_strtab() maps one when a StreamID behind it is first
used rather than allocating a replacement.

Inheriting is skipped unless the SMMU was enabled, its geometry matches
what this driver would program, and it is coherent, since the descriptors
are written with no cache maintenance.

DMA that keeps translating also keeps landing in memory the dump then
records, so a vmcore can contain buffers that changed after the crash.
Intel and AMD accept the same trade.

On its own this does not keep the inherited entries alive. The core
attaches a default domain as it probes each device, which replaces them.
The next patch holds the inherited entry until a driver maps DMA.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 98 +++++++++++++++++++++++++----
 1 file changed, 85 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 5732f3ba0122d..51a809400d56a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1998,16 +1998,42 @@ static void arm_smmu_init_initial_stes(struct arm_smmu_ste *strtab,
 	}
 }
 
+/*
+ * Inherit L1 descriptor from previous kernel
+ */
+static struct arm_smmu_strtab_l2 *
+arm_smmu_inherit_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
+{
+	struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+	u64 l1d;
+
+	l1d = le64_to_cpu(cfg->l2.l1tab[arm_smmu_strtab_l1_idx(sid)].l2ptr);
+	if (!FIELD_GET(STRTAB_L1_DESC_SPAN, l1d))
+		return NULL;
+
+	return devm_memremap(smmu->dev, l1d & STRTAB_L1_DESC_L2PTR_MASK,
+			     sizeof(struct arm_smmu_strtab_l2), MEMREMAP_WB);
+}
+
 static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
 {
 	dma_addr_t l2ptr_dma;
 	struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
 	struct arm_smmu_strtab_l2 **l2table;
+	struct arm_smmu_strtab_l2 *inherited;
 
 	l2table = &cfg->l2.l2ptrs[arm_smmu_strtab_l1_idx(sid)];
 	if (*l2table)
 		return 0;
 
+	if (is_kdump_kernel()) {
+		inherited = arm_smmu_inherit_l2_strtab(smmu, sid);
+		if (!IS_ERR_OR_NULL(inherited)) {
+			*l2table = inherited;
+			return 0;
+		}
+	}
+
 	*l2table = dmam_alloc_coherent(smmu->dev, sizeof(**l2table),
 				       &l2ptr_dma, GFP_KERNEL);
 	if (!*l2table) {
@@ -4526,6 +4552,58 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
 				       PRIQ_ENT_DWORDS, "priq");
 }
 
+static u32 arm_smmu_strtab_base_cfg(struct arm_smmu_device *smmu)
+{
+	struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+
+	if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)
+		return FIELD_PREP(STRTAB_BASE_CFG_FMT,
+				  STRTAB_BASE_CFG_FMT_2LVL) |
+		       FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE,
+				  ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) |
+		       FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT);
+
+	return FIELD_PREP(STRTAB_BASE_CFG_FMT, STRTAB_BASE_CFG_FMT_LINEAR) |
+	       FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits);
+}
+
+static void arm_smmu_inherit_strtab(struct arm_smmu_device *smmu, u32 l1size)
+{
+	struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+	void *old;
+
+	if (!is_kdump_kernel())
+		return;
+
+	if (!(readl_relaxed(smmu->base + ARM_SMMU_CR0) & CR0_SMMUEN))
+		return;
+
+	/* Descriptors are written here without cache maintenance. */
+	if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY))
+		return;
+
+	/*
+	 * Only take the table over if it has the geometry this driver is about
+	 * to program, since the descriptors are copied into a table of that
+	 * shape. Comparing against the value arm_smmu_write_strtab() would
+	 * write covers format, size and split at once.
+	 */
+	if (readl_relaxed(smmu->base + ARM_SMMU_STRTAB_BASE_CFG) !=
+	    arm_smmu_strtab_base_cfg(smmu))
+		return;
+
+	old = memremap(readq_relaxed(smmu->base + ARM_SMMU_STRTAB_BASE) &
+		       STRTAB_BASE_ADDR_MASK, l1size, MEMREMAP_WB);
+	if (!old) {
+		dev_warn(smmu->dev, "failed to map previous stream table\n");
+		return;
+	}
+
+	memcpy(cfg->l2.l1tab, old, l1size);
+	memunmap(old);
+	dev_info(smmu->dev, "inherited stream table from previous kernel\n");
+}
+
 static int arm_smmu_init_strtab_2lvl(struct arm_smmu_device *smmu)
 {
 	u32 l1size;
@@ -4556,6 +4634,8 @@ static int arm_smmu_init_strtab_2lvl(struct arm_smmu_device *smmu)
 	if (!cfg->l2.l2ptrs)
 		return -ENOMEM;
 
+	arm_smmu_inherit_strtab(smmu, l1size);
+
 	return 0;
 }
 
@@ -4821,24 +4901,16 @@ static void arm_smmu_write_strtab(struct arm_smmu_device *smmu)
 {
 	struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
 	dma_addr_t dma;
-	u32 reg;
 
-	if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
-		reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
-				 STRTAB_BASE_CFG_FMT_2LVL) |
-		      FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE,
-				 ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) |
-		      FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT);
+	if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)
 		dma = cfg->l2.l1_dma;
-	} else {
-		reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
-				 STRTAB_BASE_CFG_FMT_LINEAR) |
-		      FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits);
+	else
 		dma = cfg->linear.ste_dma;
-	}
+
 	writeq_relaxed((dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_RA,
 		       smmu->base + ARM_SMMU_STRTAB_BASE);
-	writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
+	writel_relaxed(arm_smmu_strtab_base_cfg(smmu),
+		       smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
 }
 
 static int arm_smmu_device_reset(struct arm_smmu_device *smmu)

-- 
2.53.0-Meta

[PATCH 2/2] iommu/arm-smmu-v3: defer attach in kdump so inherited entries survive probe

[Breno Leitao]

An inherited stream table keeps the crashed kernel's devices translating
only until the core probes them. iommu_setup_default_domain() then attaches
a default domain, replacing the inherited entry with one that maps nothing,
and the fault returns as F_TRANSLATION:

  [   37.824064] pci 0003:2d:00.0: Adding to iommu group 62
  [   38.035698] arm-smmu-v3 arm-smmu-v3.0.auto: event: F_TRANSLATION client: 0003:2d:00.0 iova: 0xfffe2965

The core already avoids this. iommu_dma_init() enables deferred attach in a
capture kernel, but only honours it for drivers implementing
.is_attach_deferred, which so far means amd and intel.

Implement it for SMMU, so we keep the inherited table until the next
DMA.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 51a809400d56a..9c8b344c75ab8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -4400,6 +4400,11 @@ static int arm_smmu_def_domain_type(struct device *dev)
 	return 0;
 }
 
+static bool arm_smmu_is_attach_deferred(struct device *dev)
+{
+	return is_kdump_kernel();
+}
+
 static const struct iommu_ops arm_smmu_ops = {
 	.identity_domain	= &arm_smmu_identity_domain,
 	.blocked_domain		= &arm_smmu_blocked_domain,
@@ -4415,6 +4420,7 @@ static const struct iommu_ops arm_smmu_ops = {
 	.get_resv_regions	= arm_smmu_get_resv_regions,
 	.page_response		= arm_smmu_page_response,
 	.def_domain_type	= arm_smmu_def_domain_type,
+	.is_attach_deferred	= arm_smmu_is_attach_deferred,
 	.get_viommu_size	= arm_smmu_get_viommu_size,
 	.viommu_init		= arm_vsmmu_init,
 	.user_pasid_table	= 1,

-- 
2.53.0-Meta

Re: [PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[Breno Leitao]

On Fri, Sep 11, 2026 at 07:54:43AM -0700, Breno Leitao wrote:
> I've found that kdump does not work on some ARM host, such as AWS
> Graviton metal, and the SMMU is what seems to kills it.
>
> The capture kernel dies about a second after it resets the SMMU, part
> way through PCI enumeration, and takes the machine with it.
>
> There is no panic and no oops: the console stops mid-line and the box
> reboots, leaving no vmcore, no log and nothing on disk. Crashes on this
> hardware mostly produce no dump at all.
>
> This is my theory, from an "outsider view":

This should really have been tagged RFC — apologies for missing that.

Re: [PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[Nicolin Chen]

On Fri, Sep 11, 2026 at 07:54:43AM -0700, Breno Leitao wrote:
> My lovely robot and I came with two patches that solved the problem,
> from a practical perspective and I want to share what has been tested.
> 
> Patch 1 carries the previous kernel's L1 descriptors into the new KDUMP table, so
> those streams keep translating.
> 
> Patch 2 stops the core discarding that again: iommu_dma_init() already
> enables deferred attach for a capture kernel, but the core only honours
> it for drivers implementing .is_attach_deferred, which until now meant
> amd and intel. Without it iommu_setup_default_domain() attaches
> a default domain at probe and the fault returns as F_TRANSLATION.
> 
> The x86 IOMMU drivers solve the same/similar problem the same way; see
> commit 38e5f33ee3596 ("iommu/amd: Reuse device table for kdump").

This looks like my bigger series:
https://lore.kernel.org/linux-iommu/cover.1788130528.git.nicolinc@nvidia.com/

Would you please help check/test?

Thanks
Nicolin

Re: [PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[Breno Leitao]

On Fri, Sep 11, 2026 at 03:31:28PM -0700, Nicolin Chen wrote:
> On Fri, Sep 11, 2026 at 07:54:43AM -0700, Breno Leitao wrote:
> > My lovely robot and I came with two patches that solved the problem,
> > from a practical perspective and I want to share what has been tested.
> > 
> > Patch 1 carries the previous kernel's L1 descriptors into the new KDUMP table, so
> > those streams keep translating.
> > 
> > Patch 2 stops the core discarding that again: iommu_dma_init() already
> > enables deferred attach for a capture kernel, but the core only honours
> > it for drivers implementing .is_attach_deferred, which until now meant
> > amd and intel. Without it iommu_setup_default_domain() attaches
> > a default domain at probe and the fault returns as F_TRANSLATION.
> > 
> > The x86 IOMMU drivers solve the same/similar problem the same way; see
> > commit 38e5f33ee3596 ("iommu/amd: Reuse device table for kdump").
> 
> This looks like my bigger series:
> https://lore.kernel.org/linux-iommu/cover.1788130528.git.nicolinc@nvidia.com/
> 
> Would you please help check/test?

Tested, and it fixes the problem on Meta kdump kernel. Thanks for the message.
I am dropping my two patches in favour of your series.

Test it on AWS Graviton EC2 metal, 96 cores, two ENA NICs at
0003:2c:00.0 and 0003:2d:00.0. When the kernel crashes the NICs keep
bus-mastering, and the capture kernel builds a stream table with no L1
descriptor for the ENA's StreamID 0x32d00, so that DMA raises
C_BAD_STREAMID. 

Patch 11 is what makes the difference for us. "SMMU currently enabled!
Resetting..." opens every failed capture boot we have on record, and it
does not appear at all with your series applied.

Tested-by: Breno Leitao <leitao@kernel.org>