From: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
To: <linux-kernel@vger.kernel.org>, <iommu@lists.linux.dev>,
<joro@8bytes.org>, <jgg@nvidia.com>
Cc: <yi.l.liu@intel.com>, <kevin.tian@intel.com>,
<nicolinc@nvidia.com>, <vasant.hegde@amd.com>,
<jon.grimm@amd.com>, <santosh.shukla@amd.com>, <Sairaj.K@amd.com>,
<jay.chen@amd.com>, <Ming.Shu@amd.com>, <SooJin.Tan@amd.com>,
<chrisl@kernel.org>, <wvw@google.com>, <wnliu@google.com>,
<dantuluris@google.com>, <kpsingh@google.com>,
<alejandro.j.jimenez@oracle.com>, <joao.m.martins@oracle.com>,
<guanghuifeng@linux.alibaba.com>,
Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: [PATCH v6 13/24] iommu/amd: Program IOMMU DTE with the private IPA domain
Date: Wed, 30 Sep 2026 09:23:33 +0000 [thread overview]
Message-ID: <20260930092344.5616-14-suravee.suthikulpanit@amd.com> (raw)
In-Reply-To: <20260930092344.5616-1-suravee.suthikulpanit@amd.com>
The IOMMU PCI function does not go through the normal device attach
path, so it has no iommu_dev_data. Add amd_iommu_alloc_dev_data()
and amd_iommu_free_dev_data() for a synthetic object that is not
interned in pci_seg->dev_data_xa. Program that DTE with the private
IPA domain's v1 page table so the IOMMU can DMA to its own vIOMMU
backing store.
Leave the synthetic DTE off pdom->dev_list and with no struct
device so clone_aliases() is skipped; it is only iommu->devid.
Rewrite it from amd_iommu_change_top() when the IPA table grows
so later high maps remain reachable.
Do this after viommu_private_space_init() so viommu_pdom exists.
On teardown, clear the DTE only while CONTROL_CMDBUF_EN is set,
then free the object. After disable_iommus() the INV_DTE flush
would time out, so skip it and still free the object. Failed
self DTE allocation unwinds through amd_viommu_uninit().
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
drivers/iommu/amd/amd_iommu.h | 4 ++
drivers/iommu/amd/amd_iommu_types.h | 2 +
drivers/iommu/amd/init.c | 7 +-
drivers/iommu/amd/iommu.c | 102 +++++++++++++++++++++++++---
drivers/iommu/amd/viommu.c | 57 ++++++++++++++++
5 files changed, 161 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 9cc5da745a73..6ac64524208b 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -50,6 +50,10 @@ extern u8 amd_iommu_hpt_vasize;
extern unsigned long amd_iommu_pgsize_bitmap;
extern bool amd_iommu_hatdis;
+struct iommu_dev_data *amd_iommu_alloc_dev_data(u16 devid);
+void amd_iommu_free_dev_data(struct amd_iommu *iommu,
+ struct iommu_dev_data *dev_data);
+
/* Protection domain ops */
void amd_iommu_init_identity_domain(void);
struct protection_domain *protection_domain_alloc(void);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index d02f49a39a90..4b48ec8bacd6 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -814,6 +814,8 @@ struct amd_iommu {
struct ida gid_ida; /* guest IDs for this IOMMU */
/* HW vIOMMU support */
+ /* Synthetic IOMMU-self DTE; not in pci_seg->dev_data_xa or pdom->dev_list */
+ struct iommu_dev_data *viommu_dev_data;
struct protection_domain *viommu_pdom;
void *viommu_priv_region[VIOMMU_PRIV_SUBREGION_CNT];
};
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index e73139ba7d91..96299f10b2e8 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1809,9 +1809,10 @@ static void __init free_iommu_one(struct amd_iommu *iommu)
free_sysfs(iommu);
/*
* Tear down vIOMMU before buffers and MMIO are released.
- * IOTLB invalidation runs only while CONTROL_CMDBUF_EN is
- * set (amd_viommu_init() unwind). disable_iommus() clears
- * that bit first on the boot-failure path.
+ * IOTLB invalidation and the self-DTE clear run only while
+ * CONTROL_CMDBUF_EN is set (amd_viommu_init() unwind).
+ * disable_iommus() clears that bit first on the boot-failure
+ * path.
*/
amd_viommu_uninit(iommu);
free_iommu_buffers(iommu);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index f9b49df4b90c..edeaea48d434 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -217,7 +217,12 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
struct dev_table_entry *new)
{
update_dte256(iommu, dev_data, new);
- clone_aliases(iommu, dev_data->dev);
+ /*
+ * Synthetic DTEs (vIOMMU self, translate-device-id) leave
+ * dev_data->dev NULL so clone_aliases() is skipped.
+ */
+ if (dev_data->dev)
+ clone_aliases(iommu, dev_data->dev);
device_flush_dte(iommu, dev_data);
iommu_completion_wait(iommu);
}
@@ -391,14 +396,13 @@ static struct amd_iommu *rlookup_amd_iommu(struct device *dev)
}
/*
- * Allocate an immortal per-devid object stored in pci_seg->dev_data_xa.
- * These are never erased: amd_iommu_release_device() keeps them for
- * replug, and IRQ/DTE paths look them up locklessly via xa_load().
+ * Allocate a synthetic DTE object that is not inserted into
+ * pci_seg->dev_data_xa. Lifetime is owned by the caller (vIOMMU
+ * self DTE or translate-device-id).
*/
-static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
+struct iommu_dev_data *amd_iommu_alloc_dev_data(u16 devid)
{
- struct iommu_dev_data *dev_data, *old;
- struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
+ struct iommu_dev_data *dev_data;
dev_data = kzalloc_obj(*dev_data);
if (!dev_data)
@@ -408,6 +412,22 @@ static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
spin_lock_init(&dev_data->dte_lock);
dev_data->devid = devid;
ratelimit_default_init(&dev_data->rs);
+ return dev_data;
+}
+
+/*
+ * Allocate an immortal per-devid object stored in pci_seg->dev_data_xa.
+ * These are never erased: amd_iommu_release_device() keeps them for
+ * replug, and IRQ/DTE paths look them up locklessly via xa_load().
+ */
+static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
+{
+ struct iommu_dev_data *dev_data, *old;
+ struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
+
+ dev_data = amd_iommu_alloc_dev_data(devid);
+ if (!dev_data)
+ return NULL;
old = xa_cmpxchg(&pci_seg->dev_data_xa, devid, NULL, dev_data,
GFP_KERNEL);
@@ -428,6 +448,22 @@ struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid)
return xa_load(&iommu->pci_seg->dev_data_xa, devid);
}
+void amd_iommu_free_dev_data(struct amd_iommu *iommu,
+ struct iommu_dev_data *dev_data)
+{
+ if (!dev_data)
+ return;
+
+ /*
+ * PCI/alias objects in the xarray are immortal. Never kfree
+ * those; synthetics must not be stored there.
+ */
+ if (WARN_ON_ONCE(search_dev_data(iommu, dev_data->devid) == dev_data))
+ return;
+
+ kfree(dev_data);
+}
+
static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data)
{
struct dev_table_entry new;
@@ -1833,7 +1869,11 @@ static int device_flush_dte(struct amd_iommu *iommu, struct iommu_dev_data *dev_
u16 alias;
int ret;
- if (dev_is_pci(dev_data->dev))
+ /*
+ * Synthetic DTEs leave dev_data->dev NULL; flush iommu->devid
+ * rather than walking PCI DMA aliases.
+ */
+ if (dev_data->dev && dev_is_pci(dev_data->dev))
pdev = to_pci_dev(dev_data->dev);
if (pdev)
@@ -2777,6 +2817,51 @@ static spinlock_t *amd_iommu_get_top_lock(struct pt_iommu *iommupt)
return &pdom->lock;
}
+#if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
+/*
+ * The vIOMMU private IPA domain programs a synthetic DTE for the
+ * IOMMU's own requester ID so hardware can DMA to backing store.
+ * That object is not on pdom->dev_list: it is not an IOMMU-API
+ * attach, and walkers such as rlookup and clone_aliases assume a
+ * real struct device.
+ *
+ * amd_iommu_change_top() therefore misses it. Early maps fit under
+ * the initial page-table top (PT_FEAT_DYNAMIC_TOP). Later DevID and
+ * DomID maps at high IPA call increase_top(); the old root stays
+ * live as a child of the new one, but the self DTE still holds the
+ * old MODE and would not translate those IOVAs.
+ *
+ * Walk iommu_array (from amd_iommu_pdom_bind_iommu()) and rewrite
+ * iommu->viommu_dev_data when this domain is that IOMMU's private
+ * IPA table. set_dte_entry() skips clone_aliases() because the
+ * synthetic DTE has no struct device.
+ */
+static void update_viommu_self_dte(struct protection_domain *pdom,
+ phys_addr_t top_paddr,
+ unsigned int top_level)
+{
+ struct pdom_iommu_info *pdom_iommu_info;
+ unsigned long i;
+
+ lockdep_assert_held(&pdom->lock);
+
+ xa_for_each(&pdom->iommu_array, i, pdom_iommu_info) {
+ struct amd_iommu *iommu = pdom_iommu_info->iommu;
+
+ if (iommu->viommu_pdom != pdom || !iommu->viommu_dev_data)
+ continue;
+ set_dte_entry(iommu, iommu->viommu_dev_data, top_paddr,
+ top_level);
+ }
+}
+#else
+static inline void update_viommu_self_dte(struct protection_domain *pdom,
+ phys_addr_t top_paddr,
+ unsigned int top_level)
+{
+}
+#endif
+
/*
* Update all HW references to the domain with a new pgtable configuration.
*/
@@ -2799,6 +2884,7 @@ static void amd_iommu_change_top(struct pt_iommu *iommu_table,
device_flush_dte(iommu, dev_data);
}
+ update_viommu_self_dte(pdom, top_paddr, top_level);
domain_flush_complete(pdom);
}
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index 89d6dc520b2c..e9554a3e9591 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -40,11 +40,64 @@ static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
}
}
+
+static bool viommu_cmdbuf_enabled(struct amd_iommu *iommu);
+
+static void viommu_free_self_dev_data(struct amd_iommu *iommu)
+{
+ struct iommu_dev_data *dev_data = iommu->viommu_dev_data;
+ struct dev_table_entry new = {};
+
+ if (!dev_data)
+ return;
+
+ /*
+ * INV_DTE needs the command buffer. Skip it after
+ * disable_iommus() clears CONTROL_CMDBUF_EN. The init
+ * unwind still flushes while the buffer is enabled.
+ */
+ if (viommu_cmdbuf_enabled(iommu)) {
+ amd_iommu_make_clear_dte(iommu, dev_data->devid, &new);
+ amd_iommu_update_dte(iommu, dev_data, &new);
+ }
+ amd_iommu_free_dev_data(iommu, dev_data);
+ iommu->viommu_dev_data = NULL;
+}
+
+static int viommu_alloc_self_dev_data(struct amd_iommu *iommu)
+{
+ struct protection_domain *pdom = iommu->viommu_pdom;
+ struct pt_iommu_amdv1_hw_info pt_info;
+ struct iommu_dev_data *dev_data;
+ struct dev_table_entry new = {};
+
+ dev_data = amd_iommu_alloc_dev_data(iommu->devid);
+ if (!dev_data) {
+ pr_err("%s: Failed to allocate dev_data\n", __func__);
+ return -ENOMEM;
+ }
+ /*
+ * Synthetic DTE for iommu->devid only: no struct device, so
+ * amd_iommu_update_dte() skips clone_aliases(), and not on
+ * pdom->dev_list (not an IOMMU-API attach).
+ */
+ dev_data->dev = NULL;
+ dev_data->domain = pdom;
+ iommu->viommu_dev_data = dev_data;
+
+ amd_iommu_make_clear_dte(iommu, iommu->devid, &new);
+ pt_iommu_amdv1_hw_info(&pdom->amdv1, &pt_info);
+ amd_iommu_set_dte_v1(dev_data, pdom, pdom->id, &pt_info, &new);
+ amd_iommu_update_dte(iommu, dev_data, &new);
+ return 0;
+}
+
static void viommu_private_space_uninit(struct amd_iommu *iommu);
void __init amd_viommu_uninit(struct amd_iommu *iommu)
{
iommu->flags &= ~AMD_IOMMU_FLAG_VIOMMU_EN;
+ viommu_free_self_dev_data(iommu);
viommu_private_space_uninit(iommu);
amd_viommu_vf_vfcntl_unmap(iommu);
}
@@ -319,6 +372,10 @@ int __init amd_viommu_init(struct amd_iommu *iommu)
if (ret)
goto err;
+ ret = viommu_alloc_self_dev_data(iommu);
+ if (ret)
+ goto err;
+
iommu->flags |= AMD_IOMMU_FLAG_VIOMMU_EN;
return 0;
err:
--
2.34.1
next prev parent reply other threads:[~2026-09-30 9:25 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-30 9:23 [PATCH v6 00/24] iommu/amd: Introduce AMD Hardware-accelerated Virtualized IOMMU (vIOMMU) Support Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 01/24] iommu/amd: Introduce vIOMMU-specific events and event Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 02/24] iommu/amd: Introduce EVENT_TYPE_GUEST_EVENT_FAULT Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 03/24] iommu/amd: Detect and initialize AMD vIOMMU feature Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 04/24] iommu/amd: Introduce IOMMUFD vIOMMU support for AMD Suravee Suthikulpanit
2026-09-30 21:44 ` Nicolin Chen
2026-09-30 9:23 ` [PATCH v6 05/24] iommu/amd: Allocate Guest IDs for IOMMUFD vIOMMU instances Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 06/24] iommu/amd: Map vIOMMU VF and VF Control MMIO BARs Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 07/24] iommu/amd: Add support for AMD vIOMMU VF MMIO region Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 08/24] iommu/amd: Introduce Reset vMMIO Command Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 09/24] iommu/amd: Introduce and map vIOMMU private IPA region Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 10/24] iommu/amd: Pass iommu to device_flush_dte() Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 11/24] iommu/amd: Pass iommu and devid to amd_iommu_make_clear_dte() Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 12/24] iommu/amd: Store per-segment iommu_dev_data in an xarray Suravee Suthikulpanit
2026-09-30 9:23 ` Suravee Suthikulpanit [this message]
2026-09-30 9:23 ` [PATCH v6 14/24] iommu/amd: Add per-VM private IPA alloc/map helpers Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 15/24] iommu/amd: Add helper functions to manage DevID / DomID mapping tables Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 16/24] iommu/amd: Add IOMMUFD vDevice and DevID mapping Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 17/24] iommu/amd: Program nested DTE and DomID map on attach Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 18/24] iommu/amd: Init and clear vIOMMU DevID and DomID maps Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 19/24] iommu/amd: Add per-segment translate device ID pool Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 20/24] iommu/amd: Reserve translate-device-id for PCI requestor aliases Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 21/24] iommu/amd: Add translation DTE and VFctrl TransDevID helpers Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 22/24] iommu/amd: Add translate-device-id alloc/free with vIOMMU owner Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 23/24] iommu/amd: Assign per-vIOMMU translate device ID Suravee Suthikulpanit
2026-09-30 9:23 ` [PATCH v6 24/24] iommu/amd: Relocate vIOMMU translate-device-id on PCI reserve Suravee Suthikulpanit
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260930092344.5616-14-suravee.suthikulpanit@amd.com \
--to=suravee.suthikulpanit@amd.com \
--cc=Ming.Shu@amd.com \
--cc=Sairaj.K@amd.com \
--cc=SooJin.Tan@amd.com \
--cc=alejandro.j.jimenez@oracle.com \
--cc=chrisl@kernel.org \
--cc=dantuluris@google.com \
--cc=guanghuifeng@linux.alibaba.com \
--cc=iommu@lists.linux.dev \
--cc=jay.chen@amd.com \
--cc=jgg@nvidia.com \
--cc=joao.m.martins@oracle.com \
--cc=jon.grimm@amd.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=kpsingh@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nicolinc@nvidia.com \
--cc=santosh.shukla@amd.com \
--cc=vasant.hegde@amd.com \
--cc=wnliu@google.com \
--cc=wvw@google.com \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®