[PATCH v6 00/24] iommu/amd: Introduce AMD Hardware-accelerated Virtualized IOMMU (vIOMMU) Support

[PATCH v6 00/24] iommu/amd: Introduce AMD Hardware-accelerated Virtualized IOMMU (vIOMMU) Support

[Suravee Suthikulpanit]

OVERVIEW
========

AMD IOMMU introduces the vIOMMU feature, which provides partial hardware
acceleration when implementing Guest IOMMUs. This feature provides
acceleration for guest Command Buffer, Event Log, and PPR Log. This
eliminates the CPU overhead needed for the supporting HV intercepts and
reduces the latency of these operations.

When a guest attempts to access guest IOMMU MMIO registers with offsets
between 8KB and 12KB (i.e. the 3rd 4K region) such as the Command Buffer,
Event Log and PPR Log head and tail pointer registers, this is serviced
directly by the IOMMU. When the IOMMU accesses a Command Buffer, PPR Log
or a COMPLETION_WAIT store location in memory, it directly accesses guest
physical memory. The HV/VMM continues to trap and emulate the IOMMU
configuration MMIO registers between 0KB and 4KB (i.e. the 1st 4K
region), which are primarily used during initialization.

Additionally, the HV must initialize the vIOMMU feature, map MMIO resources
between the VMs and the IOMMU, manage additional supporting data structures
in memory (e.g. GPA->SPA translation DTE, Device ID and Domain ID mapping
tables), and allocate/map vIOMMU Private Address region used as backing
storage memory for the IOMMU. Support for new IOMMU command and events
specifically for vIOMMU are also added.

Guest IOMMUs are IOMMUs exposed to VMs with additional support from VMM
(QEMU) to generate guest ACPI IVRS table and define guest PCI topology for
IOMMU and pass-through VFIO devices, which are not covered by this series.

For more detail, please see the vIOMMU section of the AMD IOMMU
Specification[1].

ABOUT SERIES V6
===============

This is version 6 of the AMD HW-vIOMMU series [2]. It is implemented on top
of the IOMMUFD vIOMMU, vDevice, and nested-domain framework in Linux
v7.3.0-rc3 (base 704340f1cd0d).

Note: This series is a partial implementation of AMD hardware-accelerated
vIOMMU. Subsequent series will add hardware-queue and extended interrupt
remapping support, which are needed to fully support AMD IOMMU
virtualization in the guest VM. The fully supported version is available
in a GitHub repository [3].

The series is organized into the following subsets:

  Patch 1-3   : Events and vIOMMU feature detect/init
  Patch 4-8   : Introduce IOMMUFD vIOMMU support and VF MMIO setup
  Patch 9-14  : Introduce and map vIOMMU Private Address (IPA) region
  Patch 15-18 : IOMMUFD vDevice, DevID/DomID maps, and nested attach
  Patch 19-24 : Translate-device-ID pool, per-vIOMMU translation DTE, and
                PCI-reserve relocation

CHANGES FROM V5
===============

V5: (https://lore.kernel.org/linux-iommu/20260914184750.222939-1-suravee.suthikulpanit@amd.com/)

IOMMUFD vIOMMU and VF MMIO (patches 4-8):
  * Patch 6: In iommu_unmap_mmio_space(), release the host MMIO
    reservation only when mmio_base is set. iommu_map_mmio_space()
    already drops the reservation when ioremap() fails, so a failed
    map is not released twice.

IPA / DTE infrastructure (patches 9-14):
  * Patch 13: Clear the self DTE only while CONTROL_CMDBUF_EN is
    set. disable_iommus() clears that bit before teardown, and
    INV_DTE would time out. dev_data is still freed.

IOMMUFD vDevice and DevID/DomID maps (patches 15-18):
  * Patch 16: Program QueueID (bits 45:30) in the DevID table entry.
    Callers pass 0. QueueID is not supported yet.
  * Patch 17: Drop amd_viommu_domain_id_update(). Callers hold
    vfctrl_lock and use amd_viommu_domain_id_update_locked().
  * Patch 18: Prefill each gdev_id with hdev_id 0 and QueueID 0.

Translate device ID (patches 19-24):
  * Patch 19: Reserve the IOMMU function devid from
    init_iommu_one_late(). Clearing rlookup keeps that function off
    probe_device(), while the vIOMMU self DTE still occupies the slot.
  * Patch 23: Destroy trans_devid_lock on init error and on vIOMMU
    destroy, after the last unlock, the same way as vfctrl_lock.

[1] IOMMU Specification: https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
[2] Series v6 tree: https://github.com/AMDESE/linux-iommu/tree/linux-7.3.0-rc3-amd-viommu_upstream_v6
[3] Fully supported tree (work-in-progress): https://github.com/AMDESE/linux-iommu/tree/wip/v7.3.0-rc3-viommu_20260930

Thank you,
Suravee

Suravee Suthikulpanit (24):
  iommu/amd: Introduce vIOMMU-specific events and event
  iommu/amd: Introduce EVENT_TYPE_GUEST_EVENT_FAULT
  iommu/amd: Detect and initialize AMD vIOMMU feature
  iommu/amd: Introduce IOMMUFD vIOMMU support for AMD
  iommu/amd: Allocate Guest IDs for IOMMUFD vIOMMU instances
  iommu/amd: Map vIOMMU VF and VF Control MMIO BARs
  iommu/amd: Add support for AMD vIOMMU VF MMIO region
  iommu/amd: Introduce Reset vMMIO Command
  iommu/amd: Introduce and map vIOMMU private IPA region
  iommu/amd: Pass iommu to device_flush_dte()
  iommu/amd: Pass iommu and devid to amd_iommu_make_clear_dte()
  iommu/amd: Store per-segment iommu_dev_data in an xarray
  iommu/amd: Program IOMMU DTE with the private IPA domain
  iommu/amd: Add per-VM private IPA alloc/map helpers
  iommu/amd: Add helper functions to manage DevID / DomID mapping tables
  iommu/amd: Add IOMMUFD vDevice and DevID mapping
  iommu/amd: Program nested DTE and DomID map on attach
  iommu/amd: Init and clear vIOMMU DevID and DomID maps
  iommu/amd: Add per-segment translate device ID pool
  iommu/amd: Reserve translate-device-id for PCI requestor aliases
  iommu/amd: Add translation DTE and VFctrl TransDevID helpers
  iommu/amd: Add translate-device-id alloc/free with vIOMMU owner
  iommu/amd: Assign per-vIOMMU translate device ID
  iommu/amd: Relocate vIOMMU translate-device-id on PCI reserve

 drivers/iommu/amd/Makefile          |   2 +-
 drivers/iommu/amd/amd_iommu.h       |  34 +-
 drivers/iommu/amd/amd_iommu_types.h | 125 ++++-
 drivers/iommu/amd/amd_viommu.h      |  90 ++++
 drivers/iommu/amd/init.c            |  84 +++-
 drivers/iommu/amd/iommu.c           | 496 +++++++++++++++++---
 drivers/iommu/amd/iommufd.c         | 185 +++++++-
 drivers/iommu/amd/nested.c          |  76 ++-
 drivers/iommu/amd/trans_devid.c     | 376 +++++++++++++++
 drivers/iommu/amd/viommu.c          | 698 ++++++++++++++++++++++++++++
 include/uapi/linux/iommufd.h        |  10 +
 11 files changed, 2090 insertions(+), 86 deletions(-)
 create mode 100644 drivers/iommu/amd/amd_viommu.h
 create mode 100644 drivers/iommu/amd/trans_devid.c
 create mode 100644 drivers/iommu/amd/viommu.c

-- 
2.34.1

[PATCH v6 01/24] iommu/amd: Introduce vIOMMU-specific events and event

[Suravee Suthikulpanit]

Adding support for new vIOMMU events:
  * vIOMMU Hardware Error event

Also, adding support for the additional vIOMMU related flags
in existing events.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  5 ++++
 drivers/iommu/amd/iommu.c           | 37 +++++++++++++++++++----------
 2 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 8241ef922519..f26a9059a356 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -138,6 +138,8 @@
 #define EVENT_TYPE_IOTLB_INV_TO	0x7
 #define EVENT_TYPE_INV_DEV_REQ	0x8
 #define EVENT_TYPE_INV_PPR_REQ	0x9
+#define EVENT_TYPE_VIOMMU_HW_ERR	0xc
+
 #define EVENT_TYPE_RMP_FAULT	0xd
 #define EVENT_TYPE_RMP_HW_ERR	0xe
 #define EVENT_DEVID_MASK	0xffff
@@ -251,6 +253,9 @@
 #define EVTLOG_SIZE_MAX		SZ_512K /* 32K entries */
 #define EVTLOG_LEN_MASK_MAX	(0xFULL << EVTLOG_SIZE_SHIFT)
 
+/* Constants for vIOMMU event fields */
+#define EVENT_VFLAGS_MASK		GENMASK_ULL(31, 27)
+
 /* Constants for PPR Log handling */
 #define PPRLOG_ENTRY_SIZE	0x10
 #define PPRLOG_SIZE_SHIFT	56
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 56262f6b1f70..43a5690721af 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -859,7 +859,7 @@ static void amd_iommu_report_rmp_fault(struct amd_iommu *iommu, volatile u32 *ev
 
 static void amd_iommu_report_page_fault(struct amd_iommu *iommu,
 					u16 devid, u16 domain_id,
-					u64 address, int flags)
+					u64 address, int flags, u8 vflags)
 {
 	struct iommu_dev_data *dev_data = NULL;
 	struct pci_dev *pdev;
@@ -894,13 +894,13 @@ static void amd_iommu_report_page_fault(struct amd_iommu *iommu,
 		}
 
 		if (__ratelimit(&dev_data->rs)) {
-			pci_err(pdev, "Event logged [IO_PAGE_FAULT domain=0x%04x address=0x%llx flags=0x%04x]\n",
-				domain_id, address, flags);
+			pci_err(pdev, "Event logged [IO_PAGE_FAULT domain=0x%04x address=%#llx flags=0x%04x vflags=%#x]\n",
+				domain_id, address, flags, vflags);
 		}
 	} else {
-		pr_err_ratelimited("Event logged [IO_PAGE_FAULT device=%04x:%02x:%02x.%x domain=0x%04x address=0x%llx flags=0x%04x]\n",
+		pr_err_ratelimited("Event logged [IO_PAGE_FAULT device=%04x:%02x:%02x.%x domain=0x%04x address=%#llx flags=0x%04x vflags=%#x]\n",
 			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
-			domain_id, address, flags);
+			domain_id, address, flags, vflags);
 	}
 
 out:
@@ -950,6 +950,7 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 	int count = 0;
 	u64 address, ctrl;
 	u32 pasid;
+	u8 vflags;
 
 retry:
 	type    = (event[1] >> EVENT_TYPE_SHIFT)  & EVENT_TYPE_MASK;
@@ -970,29 +971,30 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 		goto retry;
 	}
 
+	vflags = FIELD_GET(EVENT_VFLAGS_MASK, event[0]);
+
 	if (type == EVENT_TYPE_IO_FAULT) {
-		amd_iommu_report_page_fault(iommu, devid, pasid, address, flags);
+		amd_iommu_report_page_fault(iommu, devid, pasid, address, flags, vflags);
 		return;
 	}
 
 	switch (type) {
 	case EVENT_TYPE_ILL_DEV:
-		dev_err(dev, "Event logged [ILLEGAL_DEV_TABLE_ENTRY device=%04x:%02x:%02x.%x pasid=0x%05x address=0x%llx flags=0x%04x]\n",
+		dev_err(dev, "Event logged [ILLEGAL_DEV_TABLE_ENTRY device=%04x:%02x:%02x.%x pasid=0x%05x address=%#llx flags=0x%04x vflags=%#x]\n",
 			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
-			pasid, address, flags);
+			pasid, address, flags, vflags);
 		dev_err(dev, "Control Reg : 0x%llx\n", ctrl);
 		dump_dte_entry(iommu, devid);
 		break;
 	case EVENT_TYPE_DEV_TAB_ERR:
-		dev_err(dev, "Event logged [DEV_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x "
-			"address=0x%llx flags=0x%04x]\n",
+		dev_err(dev, "Event logged [DEV_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x address=%#llx flags=0x%04x vflags=%#x]\n",
 			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
-			address, flags);
+			address, flags, vflags);
 		break;
 	case EVENT_TYPE_PAGE_TAB_ERR:
-		dev_err(dev, "Event logged [PAGE_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x pasid=0x%04x address=0x%llx flags=0x%04x]\n",
+		dev_err(dev, "Event logged [PAGE_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x pasid=0x%04x address=%#llx flags=0x%04x vflags=%#x]\n",
 			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
-			pasid, address, flags);
+			pasid, address, flags, vflags);
 		break;
 	case EVENT_TYPE_ILL_CMD:
 		dev_err(dev, "Event logged [ILLEGAL_COMMAND_ERROR address=0x%llx]\n", address);
@@ -1021,6 +1023,15 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 	case EVENT_TYPE_INV_PPR_REQ:
 		amd_iommu_report_ppr_err(iommu, event, devid, address, flags);
 		break;
+	case EVENT_TYPE_VIOMMU_HW_ERR:
+	{
+		u16 gid = event[0] & 0xFFFF;
+		u8 src = (event[0] >> 16) & 0x3;
+
+		dev_err(dev, "Event logged [VIOMMU_HW_ERR gid=%#x address=%#llx src=%#x flags=0x%04x vflags=%#x]\n",
+			gid, address, src, flags, vflags);
+		break;
+	}
 	default:
 		dev_err(dev, "Event logged [UNKNOWN event[0]=0x%08x event[1]=0x%08x event[2]=0x%08x event[3]=0x%08x\n",
 			event[0], event[1], event[2], event[3]);
-- 
2.34.1

[PATCH v6 02/24] iommu/amd: Introduce EVENT_TYPE_GUEST_EVENT_FAULT

[Suravee Suthikulpanit]

INSERT_GUEST_EVENT with non-zero reserved bits logs
GUEST_EVENT_FAULT followed by the original guest event.
Consume both event-log slots, wrapping via
evt_buf+(head+EVTLOG_ENTRY_SIZE)%size, and dump the guest
payload raw so it is not decoded as a host event.

Wait until occupancy is at least two entries, not merely
until live tail is not the pair offset. Re-read the
event-log tail each poll iteration so a return of 2 cannot
advance head past a stale cached tail and walk empty slots
with the type-0 erratum retry.

Retry a type-0 guest slot up to LOOP_TIMEOUT, the same way
erratum 732 waits for the primary slot. Return 2 only when
that slot is already in [head, live_tail). Return 1 if the
pair never appears in tail. Rate-limit the guest-triggered
logs.

Log an invalid guest-event when the pair slot is still
missing or type 0 after that wait, instead of stalling
the event log.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |   2 +
 drivers/iommu/amd/iommu.c           | 116 +++++++++++++++++++++++++---
 2 files changed, 107 insertions(+), 11 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index f26a9059a356..704b4c8acdc5 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -138,6 +138,7 @@
 #define EVENT_TYPE_IOTLB_INV_TO	0x7
 #define EVENT_TYPE_INV_DEV_REQ	0x8
 #define EVENT_TYPE_INV_PPR_REQ	0x9
+#define EVENT_TYPE_GUEST_EVENT_FAULT	0xb
 #define EVENT_TYPE_VIOMMU_HW_ERR	0xc
 
 #define EVENT_TYPE_RMP_FAULT	0xd
@@ -255,6 +256,7 @@
 
 /* Constants for vIOMMU event fields */
 #define EVENT_VFLAGS_MASK		GENMASK_ULL(31, 27)
+#define EVENT_GID_MASK			GENMASK(15, 0)
 
 /* Constants for PPR Log handling */
 #define PPRLOG_ENTRY_SIZE	0x10
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 43a5690721af..2df81e29519e 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -942,8 +942,18 @@ static void amd_iommu_report_ppr_err(struct amd_iommu *iommu, volatile u32 *even
 	pci_dev_put(pdev);
 }
 
-static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
+static u32 evtlog_used(u32 head, u32 tail)
 {
+	return (tail + amd_iommu_evtlog_size - head) % amd_iommu_evtlog_size;
+}
+
+/*
+ * Decode one event-log slot (or a GUEST_EVENT_FAULT pair).
+ * Returns the number of slots consumed.
+ */
+static unsigned int iommu_print_event(struct amd_iommu *iommu, u32 head)
+{
+	void *__evt = iommu->evt_buf + head;
 	struct device *dev = iommu->iommu.dev;
 	int type, devid, flags;
 	volatile u32 *event = __evt;
@@ -965,7 +975,7 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 		/* Did we hit the erratum? */
 		if (++count == LOOP_TIMEOUT) {
 			pr_err("No event written to event log\n");
-			return;
+			return 1;
 		}
 		udelay(1);
 		goto retry;
@@ -975,7 +985,7 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 
 	if (type == EVENT_TYPE_IO_FAULT) {
 		amd_iommu_report_page_fault(iommu, devid, pasid, address, flags, vflags);
-		return;
+		return 1;
 	}
 
 	switch (type) {
@@ -1023,9 +1033,90 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 	case EVENT_TYPE_INV_PPR_REQ:
 		amd_iommu_report_ppr_err(iommu, event, devid, address, flags);
 		break;
+	case EVENT_TYPE_GUEST_EVENT_FAULT:
+	{
+		u16 gid = FIELD_GET(EVENT_GID_MASK, event[1]);
+		u32 next = (head + EVTLOG_ENTRY_SIZE) % amd_iommu_evtlog_size;
+		u32 *guest;
+		u32 live_tail, w0, w1, w2, w3;
+		int guest_type, wait;
+
+		dev_err_ratelimited(dev, "Event logged [GUEST_EVENT_FAULT gid=%#x flags=0x%04x vflags=%#x]\n",
+				    gid, flags, vflags);
+
+		/*
+		 * GUEST_EVENT_FAULT is paired with the original guest
+		 * event in the next log slot (possibly wrapped). The
+		 * poll-time tail can predate that slot. Wait until
+		 * occupancy is at least two entries, then retry a
+		 * type-0 pair slot the same way erratum 732 waits
+		 * for DMA visibility. Return 2 only when that slot
+		 * is already in [head, live_tail). iommu_poll_events()
+		 * re-reads tail each iteration so head cannot skip
+		 * past tail.
+		 */
+		wait = 0;
+		for (;;) {
+			live_tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);
+			if (evtlog_used(head, live_tail) >=
+			    2 * EVTLOG_ENTRY_SIZE)
+				break;
+			if (++wait == LOOP_TIMEOUT) {
+				dev_err_ratelimited(dev, "Event logged [GUEST_EVENT_FAULT invalid guest-event (does not exist)]\n");
+				if (!amd_iommu_snp_en)
+					memset(__evt, 0, EVTLOG_ENTRY_SIZE);
+				return 1;
+			}
+			udelay(1);
+		}
+
+		guest = (u32 *)(iommu->evt_buf + next);
+		wait = 0;
+		for (;;) {
+			w1 = READ_ONCE(guest[1]);
+			guest_type = (w1 >> EVENT_TYPE_SHIFT) & EVENT_TYPE_MASK;
+			if (guest_type)
+				break;
+			if (++wait == LOOP_TIMEOUT) {
+				w0 = READ_ONCE(guest[0]);
+				w2 = READ_ONCE(guest[2]);
+				w3 = READ_ONCE(guest[3]);
+				dev_err_ratelimited(dev, "Event logged [GUEST_EVENT_FAULT invalid guest-event %08x %08x %08x %08x]\n",
+						    w0, w1, w2, w3);
+				if (!amd_iommu_snp_en) {
+					memset(__evt, 0, EVTLOG_ENTRY_SIZE);
+					memset(guest, 0, EVTLOG_ENTRY_SIZE);
+				}
+				/* Occupancy already includes this slot.
+				 * Skip it so a late DMA fill is not
+				 * decoded as a host event.
+				 */
+				return 2;
+			}
+			udelay(1);
+		}
+
+		w0 = READ_ONCE(guest[0]);
+		w2 = READ_ONCE(guest[2]);
+		w3 = READ_ONCE(guest[3]);
+		dev_err_ratelimited(dev, "Event logged [GUEST_EVENT_FAULT guest-event %08x %08x %08x %08x]\n",
+				    w0, w1, w2, w3);
+
+		/*
+		 * Erratum 732: clear both pair slots so a stale guest
+		 * event is not decoded on the next poll. The second
+		 * slot may wrap to the start of the ring. Skip on SNP;
+		 * that buffer is not writable.
+		 */
+		if (!amd_iommu_snp_en) {
+			memset(__evt, 0, EVTLOG_ENTRY_SIZE);
+			memset(guest, 0, EVTLOG_ENTRY_SIZE);
+		}
+		return 2;
+	}
 	case EVENT_TYPE_VIOMMU_HW_ERR:
 	{
-		u16 gid = event[0] & 0xFFFF;
+		u16 gid = FIELD_GET(EVENT_GID_MASK, event[0]);
 		u8 src = (event[0] >> 16) & 0x3;
 
 		dev_err(dev, "Event logged [VIOMMU_HW_ERR gid=%#x address=%#llx src=%#x flags=0x%04x vflags=%#x]\n",
@@ -1044,24 +1135,27 @@ static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
 	 * SNP enabled system.
 	 */
 	if (!amd_iommu_snp_en)
-		memset(__evt, 0, 4 * sizeof(u32));
+		memset(__evt, 0, EVTLOG_ENTRY_SIZE);
+
+	return 1;
 }
 
 static void iommu_poll_events(struct amd_iommu *iommu)
 {
 	u32 head, tail;
+	unsigned int n;
 
 	head = readl(iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
-	tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);
-
-	while (head != tail) {
-		iommu_print_event(iommu, iommu->evt_buf + head);
+	for (;;) {
+		tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);
+		if (head == tail)
+			break;
+		n = iommu_print_event(iommu, head);
 
 		/* Update head pointer of hardware ring-buffer */
-		head = (head + EVTLOG_ENTRY_SIZE) % amd_iommu_evtlog_size;
+		head = (head + n * EVTLOG_ENTRY_SIZE) % amd_iommu_evtlog_size;
 		writel(head, iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
 	}
-
 }
 
 #ifdef CONFIG_IRQ_REMAP
-- 
2.34.1

[PATCH v6 03/24] iommu/amd: Detect and initialize AMD vIOMMU feature

[Suravee Suthikulpanit]

The feature is advertised w/ EFR[VIOMMUSup]. Please see the AMD IOMMU
specification[1] for more detail.

Gate amd_viommu_init() on this IOMMU's EFR[VIOMMUSup]
(iommu->features) at the PCI init call site. Do not use
check_feature()/amd_iommu_efr; that global mask is the AND of
every IOMMU. Do not fail other IOMMUs when one instance lacks
the bit or init fails.

Set AMD_IOMMU_FLAG_VIOMMU_EN when amd_viommu_init() succeeds.
Log PCI names of IOMMUs that have that flag set.

Forward-declare struct amd_iommu in amd_viommu.h so the header
is self-contained.

When CONFIG_AMD_IOMMU_IOMMUFD is off, the amd_viommu_init() stub
returns 0 so a missing build is not logged as an init failure on
EFR[VIOMMUSup] hardware.

[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/Makefile          |  2 +-
 drivers/iommu/amd/amd_iommu.h       |  5 +++++
 drivers/iommu/amd/amd_iommu_types.h |  2 ++
 drivers/iommu/amd/amd_viommu.h      | 29 ++++++++++++++++++++++++
 drivers/iommu/amd/init.c            | 34 +++++++++++++++++++++++++++++
 drivers/iommu/amd/viommu.c          | 25 +++++++++++++++++++++
 6 files changed, 96 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/amd/amd_viommu.h
 create mode 100644 drivers/iommu/amd/viommu.c

diff --git a/drivers/iommu/amd/Makefile b/drivers/iommu/amd/Makefile
index 94b8ef2acb18..e1e824b9c7b0 100644
--- a/drivers/iommu/amd/Makefile
+++ b/drivers/iommu/amd/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-y += iommu.o init.o quirks.o ppr.o pasid.o
-obj-$(CONFIG_AMD_IOMMU_IOMMUFD) += iommufd.o nested.o
+obj-$(CONFIG_AMD_IOMMU_IOMMUFD) += iommufd.o nested.o viommu.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += debugfs.o
diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 71113e860859..b2ae17e3df34 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -112,6 +112,11 @@ static inline bool check_feature(u64 mask)
 	return (amd_iommu_efr & mask);
 }
 
+static inline bool amd_iommu_viommu_enabled(struct amd_iommu *iommu)
+{
+	return iommu && (iommu->flags & AMD_IOMMU_FLAG_VIOMMU_EN);
+}
+
 static inline bool check_feature2(u64 mask)
 {
 	return (amd_iommu_efr2 & mask);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 704b4c8acdc5..688cd9240790 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -91,6 +91,7 @@
 #define FEATURE_HASUP		BIT_ULL(49)
 #define FEATURE_EPHSUP		BIT_ULL(50)
 #define FEATURE_HDSUP		BIT_ULL(52)
+#define FEATURE_VIOMMU		BIT_ULL(55)
 #define FEATURE_SNP		BIT_ULL(63)
 
 
@@ -465,6 +466,7 @@ struct irq_domain;
 struct amd_irte_ops;
 
 #define AMD_IOMMU_FLAG_TRANS_PRE_ENABLED      (1 << 0)
+#define AMD_IOMMU_FLAG_VIOMMU_EN              (1 << 1)
 
 struct gcr3_tbl_info {
 	u64	*gcr3_tbl;	/* Guest CR3 table */
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
new file mode 100644
index 000000000000..864e7b283cff
--- /dev/null
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 Advanced Micro Devices, Inc.
+ */
+
+#ifndef AMD_VIOMMU_H
+#define AMD_VIOMMU_H
+
+struct amd_iommu;
+
+#if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
+
+int amd_viommu_init(struct amd_iommu *iommu);
+
+#else
+
+/*
+ * CONFIG_AMD_IOMMU_IOMMUFD is off: host vIOMMU is not compiled in.
+ * Succeed as a no-op so iommu_init_pci() does not pr_err on
+ * EFR[VIOMMUSup] hardware.
+ */
+static inline int amd_viommu_init(struct amd_iommu *iommu)
+{
+	return 0;
+}
+
+#endif /* CONFIG_AMD_IOMMU_IOMMUFD */
+
+#endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 8a410d4aa370..c80872ce0b60 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -34,6 +34,7 @@
 #include <linux/crash_dump.h>
 
 #include "amd_iommu.h"
+#include "amd_viommu.h"
 #include "../irq_remapping.h"
 #include "../iommu-pages.h"
 
@@ -2196,6 +2197,19 @@ static int __init iommu_init_pci(struct amd_iommu *iommu)
 	if (check_feature(FEATURE_PPR) && amd_iommu_alloc_ppr_log(iommu))
 		return -ENOMEM;
 
+	/*
+	 * vIOMMU is advertised in this IOMMU's EFR[VIOMMUSup]
+	 * (iommu->features). Do not use check_feature()/amd_iommu_efr;
+	 * that global mask is the AND of every IOMMU and can hide the
+	 * bit on mixed systems.
+	 */
+	if (iommu->features & FEATURE_VIOMMU) {
+		ret = amd_viommu_init(iommu);
+		if (ret)
+			pr_err("Failed to initialize vIOMMU on %s\n",
+			       pci_name(iommu->dev));
+	}
+
 	if (iommu->cap & (1UL << IOMMU_CAP_NPCACHE)) {
 		pr_info("Using strict mode due to virtualization\n");
 		iommu_set_dma_strict();
@@ -2264,6 +2278,25 @@ static int __init iommu_init_pci(struct amd_iommu *iommu)
 	return pci_enable_device(iommu->dev);
 }
 
+static void print_viommu_info(void)
+{
+	struct amd_iommu *iommu;
+	const char *sep = "";
+
+	for_each_iommu(iommu) {
+		if (!amd_iommu_viommu_enabled(iommu))
+			continue;
+		if (!sep[0])
+			pr_info("vIOMMU enabled: ");
+		else
+			pr_cont("%s", sep);
+		pr_cont("%s", pci_name(iommu->dev));
+		sep = ", ";
+	}
+	if (sep[0])
+		pr_cont("\n");
+}
+
 static void print_iommu_info(void)
 {
 	int i;
@@ -2301,6 +2334,7 @@ static void print_iommu_info(void)
 		pr_info("V2 page table enabled (Paging mode : %d level)\n",
 			amd_iommu_gpt_level);
 	}
+	print_viommu_info();
 }
 
 static int __init amd_iommu_init_pci(void)
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
new file mode 100644
index 000000000000..53f3302f7cc6
--- /dev/null
+++ b/drivers/iommu/amd/viommu.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 Advanced Micro Devices, Inc.
+ */
+
+#define pr_fmt(fmt)     "AMD-Vi: " fmt
+#define dev_fmt(fmt)    pr_fmt(fmt)
+
+#include <linux/iommu.h>
+#include <linux/iommufd.h>
+#include <uapi/linux/iommufd.h>
+
+#include <asm/iommu.h>
+#include <asm/set_memory.h>
+
+#include "iommufd.h"
+#include "amd_iommu.h"
+#include "amd_iommu_types.h"
+#include "amd_viommu.h"
+
+int __init amd_viommu_init(struct amd_iommu *iommu)
+{
+	iommu->flags |= AMD_IOMMU_FLAG_VIOMMU_EN;
+	return 0;
+}
-- 
2.34.1

[PATCH v6 04/24] iommu/amd: Introduce IOMMUFD vIOMMU support for AMD

[Suravee Suthikulpanit]

Introduce a new enum iommu_viommu_type (IOMMU_VIOMMU_TYPE_AMD) for AMD
vIOMMU along with the struct iommu_viommu_amd, which is used to initialize
IOMMUFD vIOMMU instance when calling struct iommu_ops.viommu_init().

Also, hook up struct iomufd_viomu_ops.alloc_domain_nested to connect
nested domain allocation with AMD vIOMMU implementation.

Only report a non-zero viommu size when the type is AMD and
amd_iommu_viommu_enabled() is true for the device's IOMMU.

Additional initialization will be added in subsequent patches.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/iommufd.c  | 10 ++++++++++
 include/uapi/linux/iommufd.h | 10 ++++++++++
 2 files changed, 20 insertions(+)

diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 52300b867c1f..e8e7807f2a33 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -34,6 +34,15 @@ void *amd_iommufd_hw_info(struct device *dev, u32 *length, enum iommu_hw_info_ty
 
 size_t amd_iommufd_get_viommu_size(struct device *dev, enum iommu_viommu_type viommu_type)
 {
+	struct amd_iommu *iommu;
+
+	if (viommu_type != IOMMU_VIOMMU_TYPE_AMD)
+		return 0;
+
+	iommu = get_amd_iommu_from_dev(dev);
+	if (!amd_iommu_viommu_enabled(iommu))
+		return 0;
+
 	return VIOMMU_STRUCT_SIZE(struct amd_iommu_viommu, core);
 }
 
@@ -73,5 +82,6 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
  * struct iommufd_viommu_ops - vIOMMU specific operations
  */
 static const struct iommufd_viommu_ops amd_viommu_ops = {
+	.alloc_domain_nested = amd_iommu_alloc_domain_nested,
 	.destroy = amd_iommufd_viommu_destroy,
 };
diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h
index 206fa667c782..e1080b997e8f 100644
--- a/include/uapi/linux/iommufd.h
+++ b/include/uapi/linux/iommufd.h
@@ -1095,6 +1095,7 @@ struct iommu_fault_alloc {
  * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
  * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
  *                                    SMMUv3) enabled ARM SMMUv3 type
+ * @IOMMU_VIOMMU_TYPE_AMD: AMD HW-vIOMMU type
  */
 enum iommu_viommu_type {
 	IOMMU_VIOMMU_TYPE_DEFAULT = 0,
@@ -1105,6 +1106,7 @@ enum iommu_viommu_type {
 	 *   VMM must wire the HYP_OWN bit to 0 in guest VINTF_CONFIG register
 	 */
 	IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+	IOMMU_VIOMMU_TYPE_AMD = 3,
 };
 
 /**
@@ -1123,6 +1125,14 @@ struct iommu_viommu_tegra241_cmdqv {
 	__aligned_u64 out_vintf_mmap_length;
 };
 
+/**
+ * struct iommu_viommu_amd - AMD vIOMMU Interface (IOMMU_VIOMMU_TYPE_AMD)
+ * @out_vfmmio_mmap_offset: (out) mmap offset for vIOMMU VF-MMIO
+ */
+struct iommu_viommu_amd {
+	__aligned_u64 out_vfmmio_mmap_offset;
+};
+
 /**
  * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
  * @size: sizeof(struct iommu_viommu_alloc)
-- 
2.34.1

Re: [PATCH v6 04/24] iommu/amd: Introduce IOMMUFD vIOMMU support for AMD

[Nicolin Chen]

On Wed, Sep 30, 2026 at 09:23:24AM +0000, Suravee Suthikulpanit wrote:
> Introduce a new enum iommu_viommu_type (IOMMU_VIOMMU_TYPE_AMD) for AMD
> vIOMMU along with the struct iommu_viommu_amd, which is used to initialize
> IOMMUFD vIOMMU instance when calling struct iommu_ops.viommu_init().
> 
> Also, hook up struct iomufd_viomu_ops.alloc_domain_nested to connect
> nested domain allocation with AMD vIOMMU implementation.
> 
> Only report a non-zero viommu size when the type is AMD and
> amd_iommu_viommu_enabled() is true for the device's IOMMU.
> 
> Additional initialization will be added in subsequent patches.
> 
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>

Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>

[PATCH v6 05/24] iommu/amd: Allocate Guest IDs for IOMMUFD vIOMMU instances

[Suravee Suthikulpanit]

Hardware vIOMMU uses a 16-bit Guest ID (GID) per guest IOMMU to
index driver and hardware state. Allocate one GID per IOMMUFD vIOMMU
from a per-amd_iommu IDA (unique within that IOMMU; a VM behind
multiple IOMMUs may hold more than one GID).

Add amd_iommu_gid_alloc() and amd_iommu_gid_free(), store the ID in
amd_iommu_viommu::gid, and call them from amd_iommufd_viommu_init()
and destroy. Initialize gid_ida in init_iommu_one() and destroy it
in free_iommu_one() with the IOMMU object.

Reject amd_iommufd_viommu_init() with -EOPNOTSUPP when the IOMMU
does not have AMD_IOMMU_FLAG_VIOMMU_EN.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |  4 ++++
 drivers/iommu/amd/amd_iommu_types.h |  7 +++++++
 drivers/iommu/amd/init.c            |  2 ++
 drivers/iommu/amd/iommu.c           | 16 ++++++++++++++++
 drivers/iommu/amd/iommufd.c         | 14 ++++++++++++++
 5 files changed, 43 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index b2ae17e3df34..f6cf412a9820 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -227,4 +227,8 @@ amd_iommu_make_clear_dte(struct iommu_dev_data *dev_data, struct dev_table_entry
 struct iommu_domain *
 amd_iommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags,
 			      const struct iommu_user_data *user_data);
+
+/* Guest ID for vIOMMU */
+int amd_iommu_gid_alloc(struct amd_iommu *iommu);
+void amd_iommu_gid_free(struct amd_iommu *iommu, int gid);
 #endif /* AMD_IOMMU_H */
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 688cd9240790..8c38775ff5c1 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -21,6 +21,7 @@
 #include <linux/iommufd.h>
 #include <linux/irqreturn.h>
 #include <linux/generic_pt/iommu.h>
+#include <linux/idr.h>
 
 #include <uapi/linux/iommufd.h>
 
@@ -404,6 +405,9 @@
 
 #define MAX_DOMAIN_ID 65536
 
+/* Guest ID bits 0-14; bit 15 is reserved for secure vIOMMU. */
+#define VIOMMU_MAX_GID		0x7FFF
+
 /* Timeout stuff */
 #define LOOP_TIMEOUT		100000
 #define MMIO_STATUS_TIMEOUT	2000000
@@ -501,6 +505,7 @@ struct amd_iommu_viommu {
 	struct iommufd_viommu core;
 	struct protection_domain *parent; /* nest parent domain for this viommu */
 	struct list_head pdom_list;	  /* For protection_domain->viommu_list */
+	u16 gid;			  /* Guest ID for the vIOMMU */
 
 	/*
 	 * Per-vIOMMU guest domain ID to host domain ID mapping.
@@ -760,6 +765,8 @@ struct amd_iommu {
 	/* IOPF support */
 	struct iopf_queue *iopf_queue;
 	unsigned char iopfq_name[32];
+
+	struct ida gid_ida;		 /* guest IDs for this IOMMU */
 };
 
 static inline struct amd_iommu *dev_to_amd_iommu(struct device *dev)
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index c80872ce0b60..7b4cb473019d 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1797,6 +1797,7 @@ static void __init free_iommu_one(struct amd_iommu *iommu)
 	free_ga_log(iommu);
 	iommu_unmap_mmio_space(iommu);
 	amd_iommu_iopf_uninit(iommu);
+	ida_destroy(&iommu->gid_ida);
 }
 
 static void __init free_iommu_all(void)
@@ -1885,6 +1886,7 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
 
 	raw_spin_lock_init(&iommu->lock);
 	iommu->cmd_sem_val = 0;
+	ida_init(&iommu->gid_ida);
 
 	/* Add IOMMU to internal data structures */
 	list_add_tail(&iommu->list, &amd_iommu_list);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 2df81e29519e..d3378754cf87 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -257,6 +257,22 @@ static inline bool pdom_is_sva_capable(struct protection_domain *pdom)
 	return pdom_is_v2_pgtbl_mode(pdom) || pdom_is_in_pt_mode(pdom);
 }
 
+int amd_iommu_gid_alloc(struct amd_iommu *iommu)
+{
+	int ret = ida_alloc_range(&iommu->gid_ida, 1, VIOMMU_MAX_GID, GFP_KERNEL);
+
+	if (ret >= 0)
+		pr_debug("%s: iommu devid=%#x, gid=%u\n", __func__, iommu->devid, ret);
+
+	return ret;
+}
+
+void amd_iommu_gid_free(struct amd_iommu *iommu, int gid)
+{
+	pr_debug("%s: iommu devid=%#x, gid=%u\n", __func__, iommu->devid, gid);
+	ida_free(&iommu->gid_ida, gid);
+}
+
 static inline int get_acpihid_device_id(struct device *dev,
 					struct acpihid_map_entry **entry)
 {
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index e8e7807f2a33..69a0e61bf83d 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -49,13 +49,23 @@ size_t amd_iommufd_get_viommu_size(struct device *dev, enum iommu_viommu_type vi
 int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *parent,
 			    const struct iommu_user_data *user_data)
 {
+	int ret;
 	unsigned long flags;
 	struct protection_domain *pdom = to_pdomain(parent);
 	struct amd_iommu_viommu *aviommu = container_of(viommu, struct amd_iommu_viommu, core);
+	struct amd_iommu *iommu = container_of(viommu->iommu_dev, struct amd_iommu, iommu);
+
+	if (!amd_iommu_viommu_enabled(iommu))
+		return -EOPNOTSUPP;
 
 	xa_init_flags(&aviommu->gdomid_array, XA_FLAGS_ALLOC1);
 	aviommu->parent = pdom;
 
+	ret = amd_iommu_gid_alloc(iommu);
+	if (ret < 0)
+		return ret;
+	aviommu->gid = ret;
+
 	viommu->ops = &amd_viommu_ops;
 
 	spin_lock_irqsave(&pdom->lock, flags);
@@ -70,11 +80,15 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 	unsigned long flags;
 	struct amd_iommu_viommu *aviommu = container_of(viommu, struct amd_iommu_viommu, core);
 	struct protection_domain *pdom = aviommu->parent;
+	struct amd_iommu *iommu = container_of(viommu->iommu_dev, struct amd_iommu, iommu);
+
+	pr_debug("%s: gid=%#x, iommu devid=%#x\n", __func__, aviommu->gid, iommu->devid);
 
 	spin_lock_irqsave(&pdom->lock, flags);
 	list_del(&aviommu->pdom_list);
 	spin_unlock_irqrestore(&pdom->lock, flags);
 	xa_destroy(&aviommu->gdomid_array);
+	amd_iommu_gid_free(iommu, aviommu->gid);
 }
 
 /*
-- 
2.34.1

[PATCH v6 06/24] iommu/amd: Map vIOMMU VF and VF Control MMIO BARs

[Suravee Suthikulpanit]

Enable hardware vIOMMU on an IOMMU by locating its PCI vendor-specific
capability (VSC), reading the VF and VF Control BAR addresses, and
mapping them for host access (256MB VF, 4MB VF Control).

VF Control covers the first 4K of guest IOMMU MMIO (control registers,
trapped by QEMU). VF MMIO covers the third 4K (virtualized by the
IOMMU). Per-guest bases use the Guest ID from the previous patch.

Reject a VSC VF or VF Control BAR that is disabled or decodes to
physical address zero before mapping MMIO.

Set AMD_IOMMU_FLAG_VIOMMU_EN after the VF BARs are mapped. Export
MMIO map helpers. Call amd_viommu_uninit() from IOMMU teardown
before free_iommu_buffers() so MMIO and the command buffer remain.

Release the reserved MMIO region if ioremap() fails so VF/VFCTRL map
errors do not leak request_mem_region() for the life of the machine.

Skip the host release in iommu_unmap_mmio_space() when mmio_base is
NULL, so a failed ioremap() is not released twice.

Signed-off-by: Vasant Hegde <vasant.hegde@amd.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |  2 +
 drivers/iommu/amd/amd_iommu_types.h | 31 +++++++++
 drivers/iommu/amd/amd_viommu.h      |  6 ++
 drivers/iommu/amd/init.c            | 25 ++++++--
 drivers/iommu/amd/viommu.c          | 98 +++++++++++++++++++++++++++++
 5 files changed, 157 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index f6cf412a9820..51e4364e8564 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -28,6 +28,8 @@ void amd_iommu_set_rlookup_table(struct amd_iommu *iommu, u16 devid);
 void iommu_feature_enable(struct amd_iommu *iommu, u8 bit);
 void *__init iommu_alloc_4k_pages(struct amd_iommu *iommu,
 				  gfp_t gfp, size_t size);
+u8 __iomem * __init iommu_map_mmio_space(u64 address, u64 end);
+void __init iommu_unmap_mmio_space(struct amd_iommu *iommu);
 
 #ifdef CONFIG_AMD_IOMMU_DEBUGFS
 void amd_iommu_debugfs_setup(void);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 8c38775ff5c1..c2257d48e222 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -40,6 +40,12 @@
 #define MMIO_RANGE_OFFSET	0x0c
 #define MMIO_MISC_OFFSET	0x10
 
+/* vIOMMU Capability offsets (from IOMMU Capability Header) */
+#define MMIO_VSC_VF_BAR_LO_OFFSET	0x08
+#define MMIO_VSC_VF_BAR_HI_OFFSET	0x0c
+#define MMIO_VSC_VF_CNTL_BAR_LO_OFFSET	0x10
+#define MMIO_VSC_VF_CNTL_BAR_HI_OFFSET	0x14
+
 /* Used offsets into the MMIO space */
 #define MMIO_DEV_TABLE_OFFSET   0x0000
 #define MMIO_CMD_BUF_OFFSET     0x0008
@@ -464,6 +470,20 @@ extern bool amdr_ivrs_remap_support;
 #define for_each_ivhd_dte_flags(entry) \
 	list_for_each_entry((entry), &amd_ivhd_dev_flags_list, list)
 
+/* VIOMMU stuff */
+#define VIOMMU_VF_MMIO_ENTRY_SIZE		4096
+#define VIOMMU_VFCTRL_MMIO_ENTRY_SIZE		64
+
+/* Host ioremap/request_mem_region sizes for VF / VF_CNTL BARs */
+#define VIOMMU_VF_MMIO_MAP_SIZE		0x10000000UL
+#define VIOMMU_VF_CNTL_MMIO_MAP_SIZE	0x400000UL
+
+#define VIOMMU_VF_MMIO_BASE(iommu, guestId) \
+	(iommu->vf_base + (guestId * VIOMMU_VF_MMIO_ENTRY_SIZE))
+
+#define VIOMMU_VFCTRL_MMIO_BASE(iommu, guestId) \
+	(iommu->vfctrl_base + (guestId * VIOMMU_VFCTRL_MMIO_ENTRY_SIZE))
+
 struct amd_iommu;
 struct iommu_domain;
 struct irq_domain;
@@ -678,6 +698,17 @@ struct amd_iommu {
 	 */
 	u16 cap_ptr;
 
+	/*
+	 * VF MMIO base physical address. This is needed to calculate/pass
+	 * per guest VF MMIO address (3rd 4K of IOMMU MMIO space)
+	 */
+	u64 vf_base_phys;
+	u64 vf_cntl_phys;
+
+	/* virtual addresses of vIOMMU VF/VF_CNTL BAR */
+	u8 __iomem *vf_base;
+	u8 __iomem *vfctrl_base;
+
 	/* pci domain of this IOMMU */
 	struct amd_iommu_pci_seg *pci_seg;
 
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 864e7b283cff..e5fbea666e01 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -12,6 +12,8 @@ struct amd_iommu;
 
 int amd_viommu_init(struct amd_iommu *iommu);
 
+void __init amd_viommu_uninit(struct amd_iommu *iommu);
+
 #else
 
 /*
@@ -24,6 +26,10 @@ static inline int amd_viommu_init(struct amd_iommu *iommu)
 	return 0;
 }
 
+static inline void amd_viommu_uninit(struct amd_iommu *iommu)
+{
+}
+
 #endif /* CONFIG_AMD_IOMMU_IOMMUFD */
 
 #endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 7b4cb473019d..5be422591bac 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -459,8 +459,10 @@ static void iommu_disable(struct amd_iommu *iommu)
  * mapping and unmapping functions for the IOMMU MMIO space. Each AMD IOMMU in
  * the system has one.
  */
-static u8 __iomem * __init iommu_map_mmio_space(u64 address, u64 end)
+u8 __iomem * __init iommu_map_mmio_space(u64 address, u64 end)
 {
+	u8 __iomem *base;
+
 	if (!request_mem_region(address, end, "amd_iommu")) {
 		pr_err("Can not reserve memory region %llx-%llx for mmio\n",
 			address, end);
@@ -468,13 +470,24 @@ static u8 __iomem * __init iommu_map_mmio_space(u64 address, u64 end)
 		return NULL;
 	}
 
-	return (u8 __iomem *)ioremap(address, end);
+	base = ioremap(address, end);
+	if (!base)
+		release_mem_region(address, end);
+
+	return base;
 }
 
-static void __init iommu_unmap_mmio_space(struct amd_iommu *iommu)
+void __init iommu_unmap_mmio_space(struct amd_iommu *iommu)
 {
-	if (iommu->mmio_base)
-		iounmap(iommu->mmio_base);
+	/*
+	 * iommu_map_mmio_space() drops the reservation when ioremap()
+	 * fails, so release only a mapping that was installed.
+	 */
+	if (!iommu->mmio_base)
+		return;
+
+	iounmap(iommu->mmio_base);
+	iommu->mmio_base = NULL;
 	release_mem_region(iommu->mmio_phys, iommu->mmio_phys_end);
 }
 
@@ -1792,6 +1805,8 @@ static void __init free_sysfs(struct amd_iommu *iommu)
 static void __init free_iommu_one(struct amd_iommu *iommu)
 {
 	free_sysfs(iommu);
+	/* Tear down vIOMMU before IOMMU buffers and MMIO are released. */
+	amd_viommu_uninit(iommu);
 	free_iommu_buffers(iommu);
 	amd_iommu_free_ppr_log(iommu);
 	free_ga_log(iommu);
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index 53f3302f7cc6..7e6c4412ff20 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -18,8 +18,106 @@
 #include "amd_iommu_types.h"
 #include "amd_viommu.h"
 
+static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
+{
+	if (iommu->vfctrl_base) {
+		iounmap(iommu->vfctrl_base);
+		iommu->vfctrl_base = NULL;
+	}
+	if (iommu->vf_cntl_phys) {
+		release_mem_region(iommu->vf_cntl_phys, VIOMMU_VF_CNTL_MMIO_MAP_SIZE);
+		iommu->vf_cntl_phys = 0;
+	}
+
+	if (iommu->vf_base) {
+		iounmap(iommu->vf_base);
+		iommu->vf_base = NULL;
+	}
+	if (iommu->vf_base_phys) {
+		release_mem_region(iommu->vf_base_phys, VIOMMU_VF_MMIO_MAP_SIZE);
+		iommu->vf_base_phys = 0;
+	}
+}
+
+void __init amd_viommu_uninit(struct amd_iommu *iommu)
+{
+	iommu->flags &= ~AMD_IOMMU_FLAG_VIOMMU_EN;
+	amd_viommu_vf_vfcntl_unmap(iommu);
+}
+
+static int __init viommu_vf_vfcntl_init(struct amd_iommu *iommu)
+{
+	u32 lo, hi;
+	u16 vsc;
+	u64 vf_phys, vf_cntl_phys;
+
+	vsc = pci_find_capability(iommu->dev, PCI_CAP_ID_VNDR);
+	if (!vsc)
+		return -ENODEV;
+
+	pr_debug("device:%s, vsc offset:%04x\n",
+		 pci_name(iommu->dev), vsc);
+
+	/* Setting up VF and VF_CNTL MMIOs */
+	pci_read_config_dword(iommu->dev, vsc + MMIO_VSC_VF_BAR_LO_OFFSET, &lo);
+	pci_read_config_dword(iommu->dev, vsc + MMIO_VSC_VF_BAR_HI_OFFSET, &hi);
+	vf_phys = hi;
+	vf_phys = (vf_phys << 32) | lo;
+	if (!(vf_phys & 1)) {
+		pr_err(FW_BUG "vf_phys disabled\n");
+		return -EINVAL;
+	}
+	vf_phys &= ~1ULL;
+	if (!vf_phys) {
+		pr_err(FW_BUG "vf_phys address is zero\n");
+		return -EINVAL;
+	}
+
+	pci_read_config_dword(iommu->dev, vsc + MMIO_VSC_VF_CNTL_BAR_LO_OFFSET, &lo);
+	pci_read_config_dword(iommu->dev, vsc + MMIO_VSC_VF_CNTL_BAR_HI_OFFSET, &hi);
+	vf_cntl_phys = hi;
+	vf_cntl_phys = (vf_cntl_phys << 32) | lo;
+	if (!(vf_cntl_phys & 1)) {
+		pr_err(FW_BUG "vf_cntl_phys disabled\n");
+		return -EINVAL;
+	}
+	vf_cntl_phys &= ~1ULL;
+	if (!vf_cntl_phys) {
+		pr_err(FW_BUG "vf_cntl_phys address is zero\n");
+		return -EINVAL;
+	}
+
+	/* Mapping 256MB of VF and 4MB of VF_CNTL BARs */
+	iommu->vf_base = iommu_map_mmio_space(vf_phys, VIOMMU_VF_MMIO_MAP_SIZE);
+	if (!iommu->vf_base) {
+		pr_err("Can't reserve vf_base\n");
+		goto err_out;
+	}
+	iommu->vf_base_phys = vf_phys;
+
+	iommu->vfctrl_base = iommu_map_mmio_space(vf_cntl_phys, VIOMMU_VF_CNTL_MMIO_MAP_SIZE);
+	if (!iommu->vfctrl_base) {
+		pr_err("Can't reserve vfctrl_base\n");
+		goto err_out;
+	}
+	iommu->vf_cntl_phys = vf_cntl_phys;
+
+	pr_debug("%s: IOMMU device:%s, vf_base:%#llx, vfctrl_base:%#llx\n",
+		 __func__, pci_name(iommu->dev), vf_phys, vf_cntl_phys);
+	return 0;
+err_out:
+	amd_viommu_vf_vfcntl_unmap(iommu);
+	return -ENOMEM;
+}
+
 int __init amd_viommu_init(struct amd_iommu *iommu)
 {
+	int ret;
+
+	ret = viommu_vf_vfcntl_init(iommu);
+	if (ret)
+		return ret;
+
 	iommu->flags |= AMD_IOMMU_FLAG_VIOMMU_EN;
 	return 0;
 }
-- 
2.34.1

[PATCH v6 07/24] iommu/amd: Add support for AMD vIOMMU VF MMIO region

[Suravee Suthikulpanit]

The AMD vIOMMU virtualizes guest MMIO registers at the 3rd 4K region.
This is achieved using the iommufd_viommu_alloc_mmap().

Copy struct iommu_viommu_amd from and to userspace so the VF-MMIO
mmap offset is validated and returned on vIOMMU alloc.

Return -EINVAL if the VF-MMIO page_base is zero so mmap is not
registered at physical address 0.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Co-developed-by: Vasant Hegde <Vasant.Hegde@amd.com>
Signed-off-by: Vasant Hegde <Vasant.Hegde@amd.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  3 +++
 drivers/iommu/amd/amd_viommu.h      |  7 +++++
 drivers/iommu/amd/iommufd.c         | 41 ++++++++++++++++++++++++++++-
 drivers/iommu/amd/viommu.c          | 12 +++++++++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index c2257d48e222..3aaaf69a1e7d 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -532,6 +532,9 @@ struct amd_iommu_viommu {
 	 * Indexed by guest domain ID.
 	 */
 	struct xarray gdomid_array;
+
+	/* Offset for mmap() of guest VF MMIO; set after iommufd_viommu_alloc_mmap(). */
+	unsigned long vfmmio_mmap_offset;
 };
 
 /*
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index e5fbea666e01..276f2126290f 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -14,6 +14,8 @@ int amd_viommu_init(struct amd_iommu *iommu);
 
 void __init amd_viommu_uninit(struct amd_iommu *iommu);
 
+u64 amd_viommu_get_vfmmio_addr(struct amd_iommu *iommu, u16 gid);
+
 #else
 
 /*
@@ -30,6 +32,11 @@ static inline void amd_viommu_uninit(struct amd_iommu *iommu)
 {
 }
 
+static inline u64 amd_viommu_get_vfmmio_addr(struct amd_iommu *iommu, u16 gid)
+{
+	return 0;
+}
+
 #endif /* CONFIG_AMD_IOMMU_IOMMUFD */
 
 #endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 69a0e61bf83d..7f358e57efde 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -7,6 +7,7 @@
 
 #include "iommufd.h"
 #include "amd_iommu.h"
+#include "amd_viommu.h"
 #include "amd_iommu_types.h"
 
 static const struct iommufd_viommu_ops amd_viommu_ops;
@@ -51,6 +52,8 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 {
 	int ret;
 	unsigned long flags;
+	phys_addr_t page_base;
+	struct iommu_viommu_amd data = {};
 	struct protection_domain *pdom = to_pdomain(parent);
 	struct amd_iommu_viommu *aviommu = container_of(viommu, struct amd_iommu_viommu, core);
 	struct amd_iommu *iommu = container_of(viommu->iommu_dev, struct amd_iommu, iommu);
@@ -58,14 +61,43 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	if (!amd_iommu_viommu_enabled(iommu))
 		return -EOPNOTSUPP;
 
+	if (!user_data)
+		return -EINVAL;
+
+	ret = iommu_copy_struct_from_user(&data, user_data,
+					  IOMMU_VIOMMU_TYPE_AMD,
+					  out_vfmmio_mmap_offset);
+	if (ret)
+		return ret;
+
 	xa_init_flags(&aviommu->gdomid_array, XA_FLAGS_ALLOC1);
 	aviommu->parent = pdom;
 
 	ret = amd_iommu_gid_alloc(iommu);
 	if (ret < 0)
-		return ret;
+		goto err_gid;
 	aviommu->gid = ret;
 
+	page_base = amd_viommu_get_vfmmio_addr(iommu, aviommu->gid);
+	if (!page_base) {
+		ret = -EINVAL;
+		goto err_mmap;
+	}
+
+	ret = iommufd_viommu_alloc_mmap(&aviommu->core,
+					page_base, SZ_4K,
+					&aviommu->vfmmio_mmap_offset);
+	if (ret)
+		goto err_mmap;
+
+	data.out_vfmmio_mmap_offset = aviommu->vfmmio_mmap_offset;
+
+	ret = iommu_copy_struct_to_user(user_data, &data,
+					IOMMU_VIOMMU_TYPE_AMD,
+					out_vfmmio_mmap_offset);
+	if (ret)
+		goto err_init;
+
 	viommu->ops = &amd_viommu_ops;
 
 	spin_lock_irqsave(&pdom->lock, flags);
@@ -73,6 +105,12 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	spin_unlock_irqrestore(&pdom->lock, flags);
 
 	return 0;
+err_init:
+	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
+err_mmap:
+	amd_iommu_gid_free(iommu, aviommu->gid);
+err_gid:
+	return ret;
 }
 
 static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
@@ -88,6 +126,7 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 	list_del(&aviommu->pdom_list);
 	spin_unlock_irqrestore(&pdom->lock, flags);
 	xa_destroy(&aviommu->gdomid_array);
+	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
 	amd_iommu_gid_free(iommu, aviommu->gid);
 }
 
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index 7e6c4412ff20..c9207ed47fb6 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -110,6 +110,18 @@ static int __init viommu_vf_vfcntl_init(struct amd_iommu *iommu)
 	return -ENOMEM;
 }
 
+/*
+ * Returns VF MMIO BAR offset for the given guest ID which will be
+ * mapped to guest vIOMMU 3rd 4K MMIO address
+ */
+u64 amd_viommu_get_vfmmio_addr(struct amd_iommu *iommu, u16 gid)
+{
+	if (WARN_ON(!iommu->vf_base_phys || gid > VIOMMU_MAX_GID))
+		return 0;
+
+	return iommu->vf_base_phys + gid * VIOMMU_VF_MMIO_ENTRY_SIZE;
+}
+
 int __init amd_viommu_init(struct amd_iommu *iommu)
 {
 	int ret;
-- 
2.34.1

[PATCH v6 08/24] iommu/amd: Introduce Reset vMMIO Command

[Suravee Suthikulpanit]

Introduce new IOMMU commands for vIOMMU to reset
virtualized MMIO registers of a particular guest.

Always program RESET_MMIO_ALL_FLAG and RESET_MMIO_VCMD_FLAG.
Propagate iommu_completion_wait() errors to the caller.

Reviewed-by: Weinan Liu <wnliu@google.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |  1 +
 drivers/iommu/amd/amd_iommu_types.h |  3 +++
 drivers/iommu/amd/iommu.c           | 21 +++++++++++++++++++++
 drivers/iommu/amd/iommufd.c         |  5 +++++
 4 files changed, 30 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 51e4364e8564..8c298e090889 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -233,4 +233,5 @@ amd_iommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags,
 /* Guest ID for vIOMMU */
 int amd_iommu_gid_alloc(struct amd_iommu *iommu);
 void amd_iommu_gid_free(struct amd_iommu *iommu, int gid);
+int amd_iommu_reset_vmmio(struct amd_iommu *iommu, u16 gid);
 #endif /* AMD_IOMMU_H */
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 3aaaf69a1e7d..9eb9a0c190de 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -209,6 +209,9 @@
 #define CMD_INV_IRT		0x05
 #define CMD_COMPLETE_PPR	0x07
 #define CMD_INV_ALL		0x08
+#define CMD_RESET_VMMIO		0x0A
+#define RESET_MMIO_ALL_FLAG	BIT(28)
+#define RESET_MMIO_VCMD_FLAG	BIT(31)
 
 #define CMD_COMPL_WAIT_STORE_MASK	0x01
 #define CMD_COMPL_WAIT_INT_MASK		0x02
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index d3378754cf87..ae44050fe86a 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -1524,6 +1524,13 @@ static void build_inv_irt(struct iommu_cmd *cmd, u16 devid)
 	CMD_SET_TYPE(cmd, CMD_INV_IRT);
 }
 
+static void build_reset_vmmio(struct iommu_cmd *cmd, u16 gid)
+{
+	memset(cmd, 0, sizeof(*cmd));
+	cmd->data[0] = gid | RESET_MMIO_ALL_FLAG | RESET_MMIO_VCMD_FLAG;
+	CMD_SET_TYPE(cmd, CMD_RESET_VMMIO);
+}
+
 /*
  * Writes the command to the IOMMUs command buffer and informs the
  * hardware about the new command.
@@ -1777,6 +1784,20 @@ void amd_iommu_flush_all_caches(struct amd_iommu *iommu)
 	}
 }
 
+int amd_iommu_reset_vmmio(struct amd_iommu *iommu, u16 gid)
+{
+	int ret;
+	struct iommu_cmd cmd;
+
+	build_reset_vmmio(&cmd, gid);
+
+	ret = iommu_queue_command(iommu, &cmd);
+	if (ret)
+		return ret;
+
+	return iommu_completion_wait(iommu);
+}
+
 /*
  * Command send function for flushing on-device TLB
  */
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 7f358e57efde..2e34491e3cf7 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -92,6 +92,11 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 
 	data.out_vfmmio_mmap_offset = aviommu->vfmmio_mmap_offset;
 
+	/* Reset vIOMMU MMIOs to initialize the vIOMMU */
+	ret = amd_iommu_reset_vmmio(iommu, aviommu->gid);
+	if (ret)
+		goto err_init;
+
 	ret = iommu_copy_struct_to_user(user_data, &data,
 					IOMMU_VIOMMU_TYPE_AMD,
 					out_vfmmio_mmap_offset);
-- 
2.34.1

[PATCH v6 09/24] iommu/amd: Introduce and map vIOMMU private IPA region

[Suravee Suthikulpanit]

AMD vIOMMU introduces the IOMMU Private Address (IPA) region for
guest-side IOMMU virtualization data structures.

Introduce a per-IOMMU v1 paging domain in viommu_pdom, allocate
8MB of backing memory as four 2MB subregions, map them into the
domain, and add viommu_private_space_init() /
viommu_private_space_uninit() as a matched pair.

Register the owning IOMMU in viommu_pdom->iommu_array via
amd_iommu_pdom_bind_iommu() so unmap uses the standard
domain_flush_pages_v1() IOTLB invalidation path. On bare metal
iommu_map() does not flush (amd_iommu_iotlb_sync_map() returns
early unless amd_iommu_np_cache is set).

Call viommu_private_space_uninit() from amd_viommu_uninit() so
boot failure and free_iommu_one() release the 8MB IPA mapping.
Failed amd_viommu_init() after VF mapping goes through uninit.

viommu_priv_alloc_map() takes a gfp for backing pages and
iommu_map() page tables. Boot uses GFP_KERNEL.
iommu_alloc_pages_node_sz() zeros them. On set_memory_uc()
or iommu_map() failure, restore WB or leak rather than returning
UC-mapped memory to the allocator. iommu_map() already unmaps
any partial IOVA.

Skip IOTLB invalidation when CONTROL_CMDBUF_EN is clear so boot
failure after disable_iommus() does not wait on a dead command
buffer.

set_memory_uc() on each 2MB-aligned subregion splits the x86
direct map to 4KB PTEs. set_memory_wb() on free does not
re-coalesce them, so the direct map stays fragmented (8MB per
vIOMMU-capable IOMMU) for the life of the machine.

The private IPA domain skips iommu_domain_init(), so set
IOMMU_DOMAIN_UNMANAGED. set_dte_entry() only programs a v1 table
when type has __IOMMU_DOMAIN_PAGING; type 0 would WARN and install
a cleared DTE when increase_top() rewrites the self DTE.

For more info, see section vIOMMU Private Address Space of the
IOMMU specification [1].

[1] https://docs.amd.com/v/u/en-US/48882_3.10_PUB

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |   5 +
 drivers/iommu/amd/amd_iommu_types.h |  11 ++
 drivers/iommu/amd/init.c            |   7 +-
 drivers/iommu/amd/iommu.c           |  13 +-
 drivers/iommu/amd/viommu.c          | 192 ++++++++++++++++++++++++++++
 5 files changed, 225 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 8c298e090889..a0d5d7e34020 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -53,6 +53,8 @@ extern bool amd_iommu_hatdis;
 /* Protection domain ops */
 void amd_iommu_init_identity_domain(void);
 struct protection_domain *protection_domain_alloc(void);
+struct iommu_domain *amd_iommu_domain_alloc_paging_v1(struct device *dev,
+						      u32 flags);
 struct iommu_domain *amd_iommu_domain_alloc_sva(struct device *dev,
 						struct mm_struct *mm);
 void amd_iommu_domain_free(struct iommu_domain *dom);
@@ -94,6 +96,9 @@ void amd_iommu_domain_flush_pages(struct protection_domain *domain,
 void amd_iommu_dev_flush_pasid_pages(struct iommu_dev_data *dev_data,
 				     ioasid_t pasid, u64 address, u64 last);
 
+int amd_iommu_pdom_bind_iommu(struct amd_iommu *iommu, struct protection_domain *pdom);
+void amd_iommu_pdom_unbind_iommu(struct amd_iommu *iommu, struct protection_domain *pdom);
+
 #ifdef CONFIG_IRQ_REMAP
 int amd_iommu_create_irq_domain(struct amd_iommu *iommu);
 #else
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 9eb9a0c190de..9a90b8fe0fe3 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -417,6 +417,13 @@
 /* Guest ID bits 0-14; bit 15 is reserved for secure vIOMMU. */
 #define VIOMMU_MAX_GID		0x7FFF
 
+/*
+ * Total IOMMU private region is 8MB (4 x 2MB-subregion)
+ */
+#define VIOMMU_PRIV_REGION_BASE		(0)
+#define VIOMMU_PRIV_SUBREGION_CNT	(4)
+#define VIOMMU_PRIV_SUBREGION_SIZE	(0x200000)  /* 2MB */
+
 /* Timeout stuff */
 #define LOOP_TIMEOUT		100000
 #define MMIO_STATUS_TIMEOUT	2000000
@@ -804,6 +811,10 @@ struct amd_iommu {
 	unsigned char iopfq_name[32];
 
 	struct ida gid_ida;		 /* guest IDs for this IOMMU */
+
+	/* HW vIOMMU support */
+	struct protection_domain *viommu_pdom;
+	void *viommu_priv_region[VIOMMU_PRIV_SUBREGION_CNT];
 };
 
 static inline struct amd_iommu *dev_to_amd_iommu(struct device *dev)
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 5be422591bac..d34f991dee24 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1805,7 +1805,12 @@ static void __init free_sysfs(struct amd_iommu *iommu)
 static void __init free_iommu_one(struct amd_iommu *iommu)
 {
 	free_sysfs(iommu);
-	/* Tear down vIOMMU before IOMMU buffers and MMIO are released. */
+	/*
+	 * Tear down vIOMMU before buffers and MMIO are released.
+	 * IOTLB invalidation runs only while CONTROL_CMDBUF_EN is
+	 * set (amd_viommu_init() unwind). disable_iommus() clears
+	 * that bit first on the boot-failure path.
+	 */
 	amd_viommu_uninit(iommu);
 	free_iommu_buffers(iommu);
 	amd_iommu_free_ppr_log(iommu);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index ae44050fe86a..dfb9268f3c4e 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2509,6 +2509,16 @@ static void pdom_detach_iommu(struct amd_iommu *iommu,
 	spin_unlock_irqrestore(&pdom->lock, flags);
 }
 
+int amd_iommu_pdom_bind_iommu(struct amd_iommu *iommu, struct protection_domain *pdom)
+{
+	return pdom_attach_iommu(iommu, pdom);
+}
+
+void amd_iommu_pdom_unbind_iommu(struct amd_iommu *iommu, struct protection_domain *pdom)
+{
+	pdom_detach_iommu(iommu, pdom);
+}
+
 /*
  * If a device is not yet associated with a domain, this function makes the
  * device visible in the domain
@@ -2858,8 +2868,7 @@ static const struct iommu_dirty_ops amdv1_dirty_ops = {
 	.set_dirty_tracking = amd_iommu_set_dirty_tracking,
 };
 
-static struct iommu_domain *amd_iommu_domain_alloc_paging_v1(struct device *dev,
-							     u32 flags)
+struct iommu_domain *amd_iommu_domain_alloc_paging_v1(struct device *dev, u32 flags)
 {
 	struct pt_iommu_amdv1_cfg cfg = {};
 	struct protection_domain *domain;
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index c9207ed47fb6..89d6dc520b2c 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -17,6 +17,7 @@
 #include "amd_iommu.h"
 #include "amd_iommu_types.h"
 #include "amd_viommu.h"
+#include "../iommu-pages.h"
 
 static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 {
@@ -39,9 +40,12 @@ static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 	}
 }
 
+static void viommu_private_space_uninit(struct amd_iommu *iommu);
+
 void __init amd_viommu_uninit(struct amd_iommu *iommu)
 {
 	iommu->flags &= ~AMD_IOMMU_FLAG_VIOMMU_EN;
+	viommu_private_space_uninit(iommu);
 	amd_viommu_vf_vfcntl_unmap(iommu);
 }
 
@@ -110,6 +114,187 @@ static int __init viommu_vf_vfcntl_init(struct amd_iommu *iommu)
 	return -ENOMEM;
 }
 
+/*
+ * Restore WB and free @va. If WB restore fails, leak the pages so
+ * UC-MINUS direct-map PTEs and the PAT memtype reservation are not
+ * returned to the buddy allocator.
+ */
+static void viommu_priv_wb_free(void *va, size_t size)
+{
+	if (WARN_ON_ONCE(set_memory_wb((unsigned long)va,
+				       size >> PAGE_SHIFT)))
+		return;
+	iommu_free_pages(va);
+}
+
+/*
+ * Allocate backing pages, mark UC, and map at @iova in viommu_pdom.
+ * *@out_va is NULL on any failure. @gfp is used for the backing
+ * folio and iommu_map() page tables.
+ */
+static int viommu_priv_alloc_map(struct amd_iommu *iommu, u64 iova, size_t size,
+				 gfp_t gfp, void **out_va)
+{
+	int ret;
+	void *va;
+	int nid = iommu && iommu->dev ? dev_to_node(&iommu->dev->dev) : NUMA_NO_NODE;
+
+	*out_va = NULL;
+
+	if (!iommu || !iommu->viommu_pdom)
+		return -EINVAL;
+
+	va = iommu_alloc_pages_node_sz(nid, gfp, size);
+	if (!va)
+		return -ENOMEM;
+
+	/*
+	 * IOMMU spec: backing storage must be UC. set_memory_uc()
+	 * splits covering 2MB direct-map pages to 4K; set_memory_wb()
+	 * does not re-coalesce them.
+	 */
+	ret = set_memory_uc((unsigned long)va, size >> PAGE_SHIFT);
+	if (ret)
+		goto err_uc;
+
+	ret = iommu_map(&iommu->viommu_pdom->domain, iova,
+			iommu_virt_to_phys(va), size,
+			IOMMU_READ | IOMMU_WRITE, gfp);
+	if (ret)
+		goto err_uc;
+
+	*out_va = va;
+	return 0;
+
+err_uc:
+	viommu_priv_wb_free(va, size);
+	return ret;
+}
+
+static bool viommu_cmdbuf_enabled(struct amd_iommu *iommu)
+{
+	if (!iommu->mmio_base)
+		return false;
+	return readq(iommu->mmio_base + MMIO_CONTROL_OFFSET) &
+	       BIT_ULL(CONTROL_CMDBUF_EN);
+}
+
+/*
+ * Unmap @iova if the command buffer is still enabled, restore WB, and
+ * free @cpu_va. Skip IOTLB invalidation after disable_iommus().
+ */
+static void viommu_priv_unmap_free(struct amd_iommu *iommu, u64 iova, size_t size,
+				   void *cpu_va)
+{
+	size_t unmapped;
+
+	if (!cpu_va)
+		return;
+	if (!iommu || !iommu->viommu_pdom)
+		return;
+
+	if (viommu_cmdbuf_enabled(iommu)) {
+		unmapped = iommu_unmap(&iommu->viommu_pdom->domain, iova, size);
+		WARN_ON(unmapped != size);
+	}
+
+	viommu_priv_wb_free(cpu_va, size);
+}
+
+static void *alloc_private_subregion(struct amd_iommu *iommu, u64 base, size_t size)
+{
+	void *region = NULL;
+	int ret;
+
+	ret = viommu_priv_alloc_map(iommu, base, size, GFP_KERNEL, &region);
+	if (ret)
+		return NULL;
+
+	pr_debug("%s: base=%#llx, size=%#lx, subregion=%#llx(%#llx)\n",
+		 __func__, base, size, (unsigned long long)region, iommu_virt_to_phys(region));
+
+	return region;
+}
+
+static void viommu_private_space_uninit(struct amd_iommu *iommu)
+{
+	int i;
+	u64 base;
+	struct protection_domain *pdom;
+	struct iommu_domain *dom;
+
+	pdom = iommu->viommu_pdom;
+	if (!pdom)
+		return;
+
+	for (i = 0; i < VIOMMU_PRIV_SUBREGION_CNT; i++) {
+		if (!iommu->viommu_priv_region[i])
+			continue;
+		base = VIOMMU_PRIV_REGION_BASE + (i * VIOMMU_PRIV_SUBREGION_SIZE);
+		viommu_priv_unmap_free(iommu, base, VIOMMU_PRIV_SUBREGION_SIZE,
+					     iommu->viommu_priv_region[i]);
+		iommu->viommu_priv_region[i] = NULL;
+	}
+
+	dom = &pdom->domain;
+	amd_iommu_pdom_unbind_iommu(iommu, pdom);
+	amd_iommu_domain_free(dom);
+	iommu->viommu_pdom = NULL;
+}
+
+static int viommu_private_space_init(struct amd_iommu *iommu)
+{
+	int i, ret;
+	u64 base;
+	struct iommu_domain *dom;
+	struct protection_domain *pdom;
+	struct pt_iommu_amdv1_hw_info pt_info;
+
+	dom = amd_iommu_domain_alloc_paging_v1(&iommu->dev->dev, 0);
+	if (IS_ERR(dom)) {
+		pr_err("%s: Failed to initialize private space\n", __func__);
+		return PTR_ERR(dom);
+	}
+
+	/*
+	 * Skipped iommu_domain_init(). set_dte_entry() only programs
+	 * v1 when type has __IOMMU_DOMAIN_PAGING.
+	 */
+	dom->type = IOMMU_DOMAIN_UNMANAGED;
+
+	pdom = to_pdomain(dom);
+	iommu->viommu_pdom = pdom;
+
+	ret = amd_iommu_pdom_bind_iommu(iommu, pdom);
+	if (ret) {
+		amd_iommu_domain_free(dom);
+		iommu->viommu_pdom = NULL;
+		return ret;
+	}
+
+	/*
+	 * Each private region requires to 8MB of memory to be allocated
+	 * and mapped. Split the region into 4 x 2MB-subregion.
+	 */
+	for (i = 0; i < VIOMMU_PRIV_SUBREGION_CNT; i++) {
+		base = VIOMMU_PRIV_REGION_BASE + (i * VIOMMU_PRIV_SUBREGION_SIZE);
+		iommu->viommu_priv_region[i] = alloc_private_subregion(iommu, base,
+								       VIOMMU_PRIV_SUBREGION_SIZE);
+		if (!iommu->viommu_priv_region[i]) {
+			pr_err("%s: Failed to allocate vIOMMU private subregion %d\n", __func__, i);
+			viommu_private_space_uninit(iommu);
+			return -ENOMEM;
+		}
+	}
+
+	pt_iommu_amdv1_hw_info(&pdom->amdv1, &pt_info);
+	pr_debug("%s: devid=%#x, pte_root=%#llx\n",
+		 __func__, iommu->devid,
+		 (unsigned long long)pt_info.host_pt_root);
+
+	return 0;
+}
+
 /*
  * Returns VF MMIO BAR offset for the given guest ID which will be
  * mapped to guest vIOMMU 3rd 4K MMIO address
@@ -130,6 +315,13 @@ int __init amd_viommu_init(struct amd_iommu *iommu)
 	if (ret)
 		return ret;
 
+	ret = viommu_private_space_init(iommu);
+	if (ret)
+		goto err;
+
 	iommu->flags |= AMD_IOMMU_FLAG_VIOMMU_EN;
 	return 0;
+err:
+	amd_viommu_uninit(iommu);
+	return ret;
 }
-- 
2.34.1

[PATCH v6 10/24] iommu/amd: Pass iommu to device_flush_dte()

[Suravee Suthikulpanit]

Take struct amd_iommu explicitly so DTE invalidation is issued on the
same IOMMU instance used to program the DTE. Also, flush each device in
amd_iommu_change_top() immediately after updating its DTE.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/iommu.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index dfb9268f3c4e..a45044e1d423 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -76,7 +76,7 @@ static void set_dte_entry(struct amd_iommu *iommu,
 			  struct iommu_dev_data *dev_data,
 			  phys_addr_t top_paddr, unsigned int top_level);
 
-static int device_flush_dte(struct iommu_dev_data *dev_data);
+static int device_flush_dte(struct amd_iommu *iommu, struct iommu_dev_data *dev_data);
 
 static void amd_iommu_change_top(struct pt_iommu *iommu_table,
 				 phys_addr_t top_paddr, unsigned int top_level);
@@ -218,7 +218,7 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
 {
 	update_dte256(iommu, dev_data, new);
 	clone_aliases(iommu, dev_data->dev);
-	device_flush_dte(dev_data);
+	device_flush_dte(iommu, dev_data);
 	iommu_completion_wait(iommu);
 }
 
@@ -1824,9 +1824,8 @@ static int device_flush_dte_alias(struct pci_dev *pdev, u16 alias, void *data)
 /*
  * Command send function for invalidating a device table entry
  */
-static int device_flush_dte(struct iommu_dev_data *dev_data)
+static int device_flush_dte(struct amd_iommu *iommu, struct iommu_dev_data *dev_data)
 {
-	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
 	struct pci_dev *pdev = NULL;
 	struct amd_iommu_pci_seg *pci_seg;
 	u16 alias;
@@ -2795,11 +2794,9 @@ static void amd_iommu_change_top(struct pt_iommu *iommu_table,
 		/* Update the HW references with the new level and top ptr */
 		set_dte_entry(iommu, dev_data, top_paddr, top_level);
 		clone_aliases(iommu, dev_data->dev);
+		device_flush_dte(iommu, dev_data);
 	}
 
-	list_for_each_entry(dev_data, &pdom->dev_list, list)
-		device_flush_dte(dev_data);
-
 	domain_flush_complete(pdom);
 }
 
@@ -3233,7 +3230,7 @@ static int amd_iommu_set_dirty_tracking(struct iommu_domain *domain,
 		spin_unlock(&dev_data->dte_lock);
 
 		/* Flush device DTE */
-		device_flush_dte(dev_data);
+		device_flush_dte(iommu, dev_data);
 		domain_flush = true;
 	}
 
-- 
2.34.1

[PATCH v6 11/24] iommu/amd: Pass iommu and devid to amd_iommu_make_clear_dte()

[Suravee Suthikulpanit]

Take struct amd_iommu and device ID explicitly instead of deriving
the IOMMU from struct iommu_dev_data.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h | 6 +++---
 drivers/iommu/amd/iommu.c     | 4 ++--
 drivers/iommu/amd/nested.c    | 2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index a0d5d7e34020..9cc5da745a73 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -210,10 +210,10 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
 			  struct dev_table_entry *new);
 
 static inline void
-amd_iommu_make_clear_dte(struct iommu_dev_data *dev_data, struct dev_table_entry *new)
+amd_iommu_make_clear_dte(struct amd_iommu *iommu, u16 devid,
+			 struct dev_table_entry *new)
 {
 	struct dev_table_entry *initial_dte;
-	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev_data->dev);
 
 	/* All existing DTE must have V bit set */
 	new->data128[0] = DTE_FLAG_V;
@@ -223,7 +223,7 @@ amd_iommu_make_clear_dte(struct iommu_dev_data *dev_data, struct dev_table_entry
 	 * Restore cached persistent DTE bits, which can be set by information
 	 * in IVRS table. See set_dev_entry_from_acpi().
 	 */
-	initial_dte = amd_iommu_get_ivhd_dte_flags(iommu->pci_seg->id, dev_data->devid);
+	initial_dte = amd_iommu_get_ivhd_dte_flags(iommu->pci_seg->id, devid);
 	if (initial_dte) {
 		new->data128[0] |= initial_dte->data128[0];
 		new->data128[1] |= initial_dte->data128[1];
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index a45044e1d423..d4de87ded457 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2347,7 +2347,7 @@ static void set_dte_entry(struct amd_iommu *iommu,
 	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
 	struct dev_table_entry *dte = &get_dev_table(iommu)[dev_data->devid];
 
-	amd_iommu_make_clear_dte(dev_data, &new);
+	amd_iommu_make_clear_dte(iommu, dev_data->devid, &new);
 
 	old_domid = READ_ONCE(dte->data[1]) & DTE_DOMID_MASK;
 	if (gcr3_info->gcr3_tbl)
@@ -2379,7 +2379,7 @@ static void clear_dte_entry(struct amd_iommu *iommu, struct iommu_dev_data *dev_
 {
 	struct dev_table_entry new = {};
 
-	amd_iommu_make_clear_dte(dev_data, &new);
+	amd_iommu_make_clear_dte(iommu, dev_data->devid, &new);
 	amd_iommu_update_dte(iommu, dev_data, &new);
 }
 
diff --git a/drivers/iommu/amd/nested.c b/drivers/iommu/amd/nested.c
index f1c7987fc585..8bb0d1596c6c 100644
--- a/drivers/iommu/amd/nested.c
+++ b/drivers/iommu/amd/nested.c
@@ -205,7 +205,7 @@ static void set_dte_nested(struct amd_iommu *iommu, struct iommu_domain *dom,
 		return;
 
 	parent = ndom->viommu->parent;
-	amd_iommu_make_clear_dte(dev_data, new);
+	amd_iommu_make_clear_dte(iommu, dev_data->devid, new);
 
 	/* Retrieve the current pagetable info via the IOMMU PT API. */
 	pt_iommu_amdv1_hw_info(&parent->amdv1, &pt_info);
-- 
2.34.1

[PATCH v6 12/24] iommu/amd: Store per-segment iommu_dev_data in an xarray

[Suravee Suthikulpanit]

Replace the per-segment llist of iommu_dev_data with an xarray keyed
by devid so lookup is O(1). These objects stay immortal for the life
of the PCI segment: amd_iommu_release_device() keeps them for replug,
and IRQ/DTE paths look them up locklessly via xa_load().

Same-devid intern on xa_cmpxchg() covers replug and racing
find_dev_data(). Aliases still get separate keys. Intern remains
local to iommu.c; only the lookup table changes.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  6 ++---
 drivers/iommu/amd/init.c            |  4 +++-
 drivers/iommu/amd/iommu.c           | 34 +++++++++++++++--------------
 3 files changed, 24 insertions(+), 20 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 9a90b8fe0fe3..d02f49a39a90 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -22,6 +22,7 @@
 #include <linux/irqreturn.h>
 #include <linux/generic_pt/iommu.h>
 #include <linux/idr.h>
+#include <linux/xarray.h>
 
 #include <uapi/linux/iommufd.h>
 
@@ -605,8 +606,8 @@ struct amd_iommu_pci_seg {
 	/* List with all PCI segments in the system */
 	struct list_head list;
 
-	/* List of all available dev_data structures */
-	struct llist_head dev_data_list;
+	/* Immortal iommu_dev_data objects keyed by devid */
+	struct xarray dev_data_xa;
 
 	/* PCI segment number */
 	u16 id;
@@ -861,7 +862,6 @@ struct iommu_dev_data {
 	spinlock_t dte_lock;              /* DTE lock for 256-bit access */
 
 	struct list_head list;		  /* For domain->dev_list */
-	struct llist_node dev_data_list;  /* For global dev_data_list */
 	struct protection_domain *domain; /* Domain the device is bound to */
 	struct gcr3_tbl_info gcr3_info;   /* Per-device GCR3 table */
 	struct device *dev;
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index d34f991dee24..e73139ba7d91 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1744,7 +1744,7 @@ static struct amd_iommu_pci_seg *__init alloc_pci_segment(u16 id,
 		    SZ_4K);
 
 	pci_seg->id = id;
-	init_llist_head(&pci_seg->dev_data_list);
+	xa_init(&pci_seg->dev_data_xa);
 	INIT_LIST_HEAD(&pci_seg->unity_map);
 	list_add_tail(&pci_seg->list, &amd_iommu_pci_seg_list);
 
@@ -1763,6 +1763,7 @@ static struct amd_iommu_pci_seg *__init alloc_pci_segment(u16 id,
 	free_dev_table(pci_seg);
 err_free_pci_seg:
 	list_del(&pci_seg->list);
+	xa_destroy(&pci_seg->dev_data_xa);
 	kfree(pci_seg);
 	return NULL;
 }
@@ -1786,6 +1787,7 @@ static void __init free_pci_segments(void)
 
 	for_each_pci_segment_safe(pci_seg, next) {
 		list_del(&pci_seg->list);
+		xa_destroy(&pci_seg->dev_data_xa);
 		free_irq_lookup_table(pci_seg);
 		free_rlookup_table(pci_seg);
 		free_alias_table(pci_seg);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index d4de87ded457..f9b49df4b90c 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -390,9 +390,14 @@ static struct amd_iommu *rlookup_amd_iommu(struct device *dev)
 	return __rlookup_amd_iommu(seg, PCI_SBDF_TO_DEVID(devid));
 }
 
+/*
+ * Allocate an immortal per-devid object stored in pci_seg->dev_data_xa.
+ * These are never erased: amd_iommu_release_device() keeps them for
+ * replug, and IRQ/DTE paths look them up locklessly via xa_load().
+ */
 static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
 {
-	struct iommu_dev_data *dev_data;
+	struct iommu_dev_data *dev_data, *old;
 	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
 
 	dev_data = kzalloc_obj(*dev_data);
@@ -404,26 +409,23 @@ static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
 	dev_data->devid = devid;
 	ratelimit_default_init(&dev_data->rs);
 
-	llist_add(&dev_data->dev_data_list, &pci_seg->dev_data_list);
+	old = xa_cmpxchg(&pci_seg->dev_data_xa, devid, NULL, dev_data,
+			 GFP_KERNEL);
+	if (xa_is_err(old)) {
+		kfree(dev_data);
+		return NULL;
+	}
+	if (old) {
+		kfree(dev_data);
+		return old;
+	}
+
 	return dev_data;
 }
 
 struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid)
 {
-	struct iommu_dev_data *dev_data;
-	struct llist_node *node;
-	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
-
-	if (llist_empty(&pci_seg->dev_data_list))
-		return NULL;
-
-	node = pci_seg->dev_data_list.first;
-	llist_for_each_entry(dev_data, node, dev_data_list) {
-		if (dev_data->devid == devid)
-			return dev_data;
-	}
-
-	return NULL;
+	return xa_load(&iommu->pci_seg->dev_data_xa, devid);
 }
 
 static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data)
-- 
2.34.1

[PATCH v6 13/24] iommu/amd: Program IOMMU DTE with the private IPA domain

[Suravee Suthikulpanit]

The IOMMU PCI function does not go through the normal device attach
path, so it has no iommu_dev_data. Add amd_iommu_alloc_dev_data()
and amd_iommu_free_dev_data() for a synthetic object that is not
interned in pci_seg->dev_data_xa. Program that DTE with the private
IPA domain's v1 page table so the IOMMU can DMA to its own vIOMMU
backing store.

Leave the synthetic DTE off pdom->dev_list and with no struct
device so clone_aliases() is skipped; it is only iommu->devid.
Rewrite it from amd_iommu_change_top() when the IPA table grows
so later high maps remain reachable.

Do this after viommu_private_space_init() so viommu_pdom exists.
On teardown, clear the DTE only while CONTROL_CMDBUF_EN is set,
then free the object. After disable_iommus() the INV_DTE flush
would time out, so skip it and still free the object. Failed
self DTE allocation unwinds through amd_viommu_uninit().

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |   4 ++
 drivers/iommu/amd/amd_iommu_types.h |   2 +
 drivers/iommu/amd/init.c            |   7 +-
 drivers/iommu/amd/iommu.c           | 102 +++++++++++++++++++++++++---
 drivers/iommu/amd/viommu.c          |  57 ++++++++++++++++
 5 files changed, 161 insertions(+), 11 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 9cc5da745a73..6ac64524208b 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -50,6 +50,10 @@ extern u8 amd_iommu_hpt_vasize;
 extern unsigned long amd_iommu_pgsize_bitmap;
 extern bool amd_iommu_hatdis;
 
+struct iommu_dev_data *amd_iommu_alloc_dev_data(u16 devid);
+void amd_iommu_free_dev_data(struct amd_iommu *iommu,
+			     struct iommu_dev_data *dev_data);
+
 /* Protection domain ops */
 void amd_iommu_init_identity_domain(void);
 struct protection_domain *protection_domain_alloc(void);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index d02f49a39a90..4b48ec8bacd6 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -814,6 +814,8 @@ struct amd_iommu {
 	struct ida gid_ida;		 /* guest IDs for this IOMMU */
 
 	/* HW vIOMMU support */
+	/* Synthetic IOMMU-self DTE; not in pci_seg->dev_data_xa or pdom->dev_list */
+	struct iommu_dev_data *viommu_dev_data;
 	struct protection_domain *viommu_pdom;
 	void *viommu_priv_region[VIOMMU_PRIV_SUBREGION_CNT];
 };
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index e73139ba7d91..96299f10b2e8 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1809,9 +1809,10 @@ static void __init free_iommu_one(struct amd_iommu *iommu)
 	free_sysfs(iommu);
 	/*
 	 * Tear down vIOMMU before buffers and MMIO are released.
-	 * IOTLB invalidation runs only while CONTROL_CMDBUF_EN is
-	 * set (amd_viommu_init() unwind). disable_iommus() clears
-	 * that bit first on the boot-failure path.
+	 * IOTLB invalidation and the self-DTE clear run only while
+	 * CONTROL_CMDBUF_EN is set (amd_viommu_init() unwind).
+	 * disable_iommus() clears that bit first on the boot-failure
+	 * path.
 	 */
 	amd_viommu_uninit(iommu);
 	free_iommu_buffers(iommu);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index f9b49df4b90c..edeaea48d434 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -217,7 +217,12 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
 			     struct dev_table_entry *new)
 {
 	update_dte256(iommu, dev_data, new);
-	clone_aliases(iommu, dev_data->dev);
+	/*
+	 * Synthetic DTEs (vIOMMU self, translate-device-id) leave
+	 * dev_data->dev NULL so clone_aliases() is skipped.
+	 */
+	if (dev_data->dev)
+		clone_aliases(iommu, dev_data->dev);
 	device_flush_dte(iommu, dev_data);
 	iommu_completion_wait(iommu);
 }
@@ -391,14 +396,13 @@ static struct amd_iommu *rlookup_amd_iommu(struct device *dev)
 }
 
 /*
- * Allocate an immortal per-devid object stored in pci_seg->dev_data_xa.
- * These are never erased: amd_iommu_release_device() keeps them for
- * replug, and IRQ/DTE paths look them up locklessly via xa_load().
+ * Allocate a synthetic DTE object that is not inserted into
+ * pci_seg->dev_data_xa. Lifetime is owned by the caller (vIOMMU
+ * self DTE or translate-device-id).
  */
-static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
+struct iommu_dev_data *amd_iommu_alloc_dev_data(u16 devid)
 {
-	struct iommu_dev_data *dev_data, *old;
-	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
+	struct iommu_dev_data *dev_data;
 
 	dev_data = kzalloc_obj(*dev_data);
 	if (!dev_data)
@@ -408,6 +412,22 @@ static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
 	spin_lock_init(&dev_data->dte_lock);
 	dev_data->devid = devid;
 	ratelimit_default_init(&dev_data->rs);
+	return dev_data;
+}
+
+/*
+ * Allocate an immortal per-devid object stored in pci_seg->dev_data_xa.
+ * These are never erased: amd_iommu_release_device() keeps them for
+ * replug, and IRQ/DTE paths look them up locklessly via xa_load().
+ */
+static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
+{
+	struct iommu_dev_data *dev_data, *old;
+	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
+
+	dev_data = amd_iommu_alloc_dev_data(devid);
+	if (!dev_data)
+		return NULL;
 
 	old = xa_cmpxchg(&pci_seg->dev_data_xa, devid, NULL, dev_data,
 			 GFP_KERNEL);
@@ -428,6 +448,22 @@ struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid)
 	return xa_load(&iommu->pci_seg->dev_data_xa, devid);
 }
 
+void amd_iommu_free_dev_data(struct amd_iommu *iommu,
+			     struct iommu_dev_data *dev_data)
+{
+	if (!dev_data)
+		return;
+
+	/*
+	 * PCI/alias objects in the xarray are immortal. Never kfree
+	 * those; synthetics must not be stored there.
+	 */
+	if (WARN_ON_ONCE(search_dev_data(iommu, dev_data->devid) == dev_data))
+		return;
+
+	kfree(dev_data);
+}
+
 static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data)
 {
 	struct dev_table_entry new;
@@ -1833,7 +1869,11 @@ static int device_flush_dte(struct amd_iommu *iommu, struct iommu_dev_data *dev_
 	u16 alias;
 	int ret;
 
-	if (dev_is_pci(dev_data->dev))
+	/*
+	 * Synthetic DTEs leave dev_data->dev NULL; flush iommu->devid
+	 * rather than walking PCI DMA aliases.
+	 */
+	if (dev_data->dev && dev_is_pci(dev_data->dev))
 		pdev = to_pci_dev(dev_data->dev);
 
 	if (pdev)
@@ -2777,6 +2817,51 @@ static spinlock_t *amd_iommu_get_top_lock(struct pt_iommu *iommupt)
 	return &pdom->lock;
 }
 
+#if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
+/*
+ * The vIOMMU private IPA domain programs a synthetic DTE for the
+ * IOMMU's own requester ID so hardware can DMA to backing store.
+ * That object is not on pdom->dev_list: it is not an IOMMU-API
+ * attach, and walkers such as rlookup and clone_aliases assume a
+ * real struct device.
+ *
+ * amd_iommu_change_top() therefore misses it. Early maps fit under
+ * the initial page-table top (PT_FEAT_DYNAMIC_TOP). Later DevID and
+ * DomID maps at high IPA call increase_top(); the old root stays
+ * live as a child of the new one, but the self DTE still holds the
+ * old MODE and would not translate those IOVAs.
+ *
+ * Walk iommu_array (from amd_iommu_pdom_bind_iommu()) and rewrite
+ * iommu->viommu_dev_data when this domain is that IOMMU's private
+ * IPA table. set_dte_entry() skips clone_aliases() because the
+ * synthetic DTE has no struct device.
+ */
+static void update_viommu_self_dte(struct protection_domain *pdom,
+				   phys_addr_t top_paddr,
+				   unsigned int top_level)
+{
+	struct pdom_iommu_info *pdom_iommu_info;
+	unsigned long i;
+
+	lockdep_assert_held(&pdom->lock);
+
+	xa_for_each(&pdom->iommu_array, i, pdom_iommu_info) {
+		struct amd_iommu *iommu = pdom_iommu_info->iommu;
+
+		if (iommu->viommu_pdom != pdom || !iommu->viommu_dev_data)
+			continue;
+		set_dte_entry(iommu, iommu->viommu_dev_data, top_paddr,
+			      top_level);
+	}
+}
+#else
+static inline void update_viommu_self_dte(struct protection_domain *pdom,
+					  phys_addr_t top_paddr,
+					  unsigned int top_level)
+{
+}
+#endif
+
 /*
  * Update all HW references to the domain with a new pgtable configuration.
  */
@@ -2799,6 +2884,7 @@ static void amd_iommu_change_top(struct pt_iommu *iommu_table,
 		device_flush_dte(iommu, dev_data);
 	}
 
+	update_viommu_self_dte(pdom, top_paddr, top_level);
 	domain_flush_complete(pdom);
 }
 
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index 89d6dc520b2c..e9554a3e9591 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -40,11 +40,64 @@ static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 	}
 }
 
+
+static bool viommu_cmdbuf_enabled(struct amd_iommu *iommu);
+
+static void viommu_free_self_dev_data(struct amd_iommu *iommu)
+{
+	struct iommu_dev_data *dev_data = iommu->viommu_dev_data;
+	struct dev_table_entry new = {};
+
+	if (!dev_data)
+		return;
+
+	/*
+	 * INV_DTE needs the command buffer. Skip it after
+	 * disable_iommus() clears CONTROL_CMDBUF_EN. The init
+	 * unwind still flushes while the buffer is enabled.
+	 */
+	if (viommu_cmdbuf_enabled(iommu)) {
+		amd_iommu_make_clear_dte(iommu, dev_data->devid, &new);
+		amd_iommu_update_dte(iommu, dev_data, &new);
+	}
+	amd_iommu_free_dev_data(iommu, dev_data);
+	iommu->viommu_dev_data = NULL;
+}
+
+static int viommu_alloc_self_dev_data(struct amd_iommu *iommu)
+{
+	struct protection_domain *pdom = iommu->viommu_pdom;
+	struct pt_iommu_amdv1_hw_info pt_info;
+	struct iommu_dev_data *dev_data;
+	struct dev_table_entry new = {};
+
+	dev_data = amd_iommu_alloc_dev_data(iommu->devid);
+	if (!dev_data) {
+		pr_err("%s: Failed to allocate dev_data\n", __func__);
+		return -ENOMEM;
+	}
+	/*
+	 * Synthetic DTE for iommu->devid only: no struct device, so
+	 * amd_iommu_update_dte() skips clone_aliases(), and not on
+	 * pdom->dev_list (not an IOMMU-API attach).
+	 */
+	dev_data->dev = NULL;
+	dev_data->domain = pdom;
+	iommu->viommu_dev_data = dev_data;
+
+	amd_iommu_make_clear_dte(iommu, iommu->devid, &new);
+	pt_iommu_amdv1_hw_info(&pdom->amdv1, &pt_info);
+	amd_iommu_set_dte_v1(dev_data, pdom, pdom->id, &pt_info, &new);
+	amd_iommu_update_dte(iommu, dev_data, &new);
+	return 0;
+}
+
 static void viommu_private_space_uninit(struct amd_iommu *iommu);
 
 void __init amd_viommu_uninit(struct amd_iommu *iommu)
 {
 	iommu->flags &= ~AMD_IOMMU_FLAG_VIOMMU_EN;
+	viommu_free_self_dev_data(iommu);
 	viommu_private_space_uninit(iommu);
 	amd_viommu_vf_vfcntl_unmap(iommu);
 }
@@ -319,6 +372,10 @@ int __init amd_viommu_init(struct amd_iommu *iommu)
 	if (ret)
 		goto err;
 
+	ret = viommu_alloc_self_dev_data(iommu);
+	if (ret)
+		goto err;
+
 	iommu->flags |= AMD_IOMMU_FLAG_VIOMMU_EN;
 	return 0;
 err:
-- 
2.34.1

[PATCH v6 14/24] iommu/amd: Add per-VM private IPA alloc/map helpers

[Suravee Suthikulpanit]

Guest device ID and guest domain ID tables use dedicated slots in
the vIOMMU private address (IPA) region, indexed by guest ID (GID).

Add alloc_private_vm_region() and free_private_vm_region() to
allocate backing pages, map them through viommu_pdom, and tear down
with an unmap flush on VM destroy.

Charge GFP_KERNEL_ACCOUNT so the DevID/DomID pages count against
the caller's memcg. iommufd already charges the vIOMMU object.

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/viommu.c | 39 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index e9554a3e9591..d80f545c0648 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -382,3 +382,42 @@ int __init amd_viommu_init(struct amd_iommu *iommu)
 	amd_viommu_uninit(iommu);
 	return ret;
 }
+
+static int __maybe_unused alloc_private_vm_region(struct amd_iommu *iommu, u64 **entry,
+						 u64 base, size_t size, u16 gid)
+{
+	int ret;
+	void *va = NULL;
+	u64 addr = base + (gid * size);
+
+	/* Charge the caller; boot IPA stays GFP_KERNEL. */
+	ret = viommu_priv_alloc_map(iommu, addr, size, GFP_KERNEL_ACCOUNT, &va);
+	if (ret) {
+		*entry = NULL;
+		return ret;
+	}
+
+	*entry = (u64 *)va;
+
+	pr_debug("%s: entry=%#llx(%#llx), addr=%#llx, size=%#lx\n", __func__,
+		 (unsigned long long)*entry, iommu_virt_to_phys(*entry), addr, size);
+
+	return 0;
+}
+
+static void __maybe_unused free_private_vm_region(struct amd_iommu *iommu, u64 **entry,
+						  u64 base, size_t size, u16 gid)
+{
+	u64 addr = base + (gid * size);
+
+	if (!iommu || !iommu->viommu_pdom || !*entry)
+		return;
+
+	pr_debug("%s: entry=%#llx(%#llx), base=%#llx, addr=%#llx, size=%#lx\n",
+		 __func__, (unsigned long long)*entry,
+		 iommu_virt_to_phys(*entry), base, addr, size);
+
+	viommu_priv_unmap_free(iommu, addr, size, *entry);
+
+	*entry = NULL;
+}
-- 
2.34.1

[PATCH v6 15/24] iommu/amd: Add helper functions to manage DevID / DomID mapping tables

[Suravee Suthikulpanit]

Introduce amd_viommu_init_one() and amd_viommu_uninit_one().
These functions are called during IOMMUFD vIOMMU initialize and destroy.
Currently, it manages the IPA mapping for Device ID and Domain ID mapping
tables.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  3 ++
 drivers/iommu/amd/amd_viommu.h      |  4 ++
 drivers/iommu/amd/iommu.c           |  1 +
 drivers/iommu/amd/iommufd.c         |  9 +++-
 drivers/iommu/amd/viommu.c          | 64 +++++++++++++++++++++++++++++
 5 files changed, 80 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 4b48ec8bacd6..fc8e1378c7ed 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -544,6 +544,9 @@ struct amd_iommu_viommu {
 	 */
 	struct xarray gdomid_array;
 
+	u64 *devid_table;
+	u64 *domid_table;
+
 	/* Offset for mmap() of guest VF MMIO; set after iommufd_viommu_alloc_mmap(). */
 	unsigned long vfmmio_mmap_offset;
 };
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 276f2126290f..580cad0f9f5a 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -16,6 +16,10 @@ void __init amd_viommu_uninit(struct amd_iommu *iommu);
 
 u64 amd_viommu_get_vfmmio_addr(struct amd_iommu *iommu, u16 gid);
 
+int amd_viommu_init_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu);
+
+void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu);
+
 #else
 
 /*
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index edeaea48d434..b7073b0ca8b0 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -43,6 +43,7 @@
 #include <linux/generic_pt/iommu.h>
 
 #include "amd_iommu.h"
+#include "amd_viommu.h"
 #include "iommufd.h"
 #include "../irq_remapping.h"
 #include "../iommu-pages.h"
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 2e34491e3cf7..97de07d3ad10 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -97,11 +97,15 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	if (ret)
 		goto err_init;
 
+	ret = amd_viommu_init_one(iommu, aviommu);
+	if (ret)
+		goto err_init;
+
 	ret = iommu_copy_struct_to_user(user_data, &data,
 					IOMMU_VIOMMU_TYPE_AMD,
 					out_vfmmio_mmap_offset);
 	if (ret)
-		goto err_init;
+		goto err_init_one;
 
 	viommu->ops = &amd_viommu_ops;
 
@@ -110,6 +114,8 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	spin_unlock_irqrestore(&pdom->lock, flags);
 
 	return 0;
+err_init_one:
+	amd_viommu_uninit_one(iommu, aviommu);
 err_init:
 	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
 err_mmap:
@@ -132,6 +138,7 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 	spin_unlock_irqrestore(&pdom->lock, flags);
 	xa_destroy(&aviommu->gdomid_array);
 	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
+	amd_viommu_uninit_one(iommu, aviommu);
 	amd_iommu_gid_free(iommu, aviommu->gid);
 }
 
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index d80f545c0648..b160c70f3804 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -19,6 +19,32 @@
 #include "amd_viommu.h"
 #include "../iommu-pages.h"
 
+/*
+ * Per-vIOMMU DevID / DomID tables share iommu->viommu_pdom:
+ *   IOVA = base + gid * entry_size, gid in [1, VIOMMU_MAX_GID].
+ * The 64GB gap between these bases is 0x10000 1MB DevID slots.
+ * VIOMMU_MAX_GID is 0x7FFF, so a max-gid DevID window ends at
+ * 0x1800000000 and cannot meet VIOMMU_DOMID_MAPPING_BASE.
+ *
+ * VIOMMU_MAX_GDEVID / VIOMMU_MAX_GDOMID are 16-bit guest DevID
+ * and DomID table indices (hardware mapping-table width), not GID.
+ */
+#define VIOMMU_MAX_GDEVID		0xFFFF
+#define VIOMMU_DEVID_MAPPING_BASE	0x1000000000ULL
+#define VIOMMU_DEVID_MAPPING_ENTRY_SIZE	(1 << 20)
+
+#define VIOMMU_MAX_GDOMID		0xFFFF
+#define VIOMMU_DOMID_MAPPING_BASE	0x2000000000ULL
+#define VIOMMU_DOMID_MAPPING_ENTRY_SIZE	(1 << 19)
+
+static_assert(VIOMMU_DEVID_MAPPING_BASE +
+	      (VIOMMU_MAX_GID + 1ULL) * VIOMMU_DEVID_MAPPING_ENTRY_SIZE <=
+	      VIOMMU_DOMID_MAPPING_BASE);
+static_assert(VIOMMU_DEVID_MAPPING_ENTRY_SIZE >=
+	      (VIOMMU_MAX_GDEVID + 1ULL) * sizeof(u64));
+static_assert(VIOMMU_DOMID_MAPPING_ENTRY_SIZE >=
+	      (VIOMMU_MAX_GDOMID + 1ULL) * sizeof(u64));
+
 static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 {
 	if (iommu->vfctrl_base) {
@@ -421,3 +447,41 @@ static void __maybe_unused free_private_vm_region(struct amd_iommu *iommu, u64 *
 
 	*entry = NULL;
 }
+
+void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *aviommu)
+{
+	pr_debug("%s: gid=%u\n", __func__, aviommu->gid);
+
+	free_private_vm_region(iommu, &aviommu->devid_table,
+			       VIOMMU_DEVID_MAPPING_BASE,
+			       VIOMMU_DEVID_MAPPING_ENTRY_SIZE,
+			       aviommu->gid);
+	free_private_vm_region(iommu, &aviommu->domid_table,
+			       VIOMMU_DOMID_MAPPING_BASE,
+			       VIOMMU_DOMID_MAPPING_ENTRY_SIZE,
+			       aviommu->gid);
+}
+
+int amd_viommu_init_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu)
+{
+	int ret;
+
+	ret = alloc_private_vm_region(iommu, &viommu->devid_table,
+				      VIOMMU_DEVID_MAPPING_BASE,
+				      VIOMMU_DEVID_MAPPING_ENTRY_SIZE,
+				      viommu->gid);
+	if (ret)
+		goto err_out;
+
+	ret = alloc_private_vm_region(iommu, &viommu->domid_table,
+				      VIOMMU_DOMID_MAPPING_BASE,
+				      VIOMMU_DOMID_MAPPING_ENTRY_SIZE,
+				      viommu->gid);
+	if (ret)
+		goto err_out;
+
+	return 0;
+err_out:
+	amd_viommu_uninit_one(iommu, viommu);
+	return ret;
+}
-- 
2.34.1

[PATCH v6 16/24] iommu/amd: Add IOMMUFD vDevice and DevID mapping

[Suravee Suthikulpanit]

IOMMUFD vDevice allocation supplies the guest device ID (virt_id).
The AMD vIOMMU Device ID mapping table translates that gdev_id to the
host device ID for virtualised IOMMU commands.

Add vdevice_init, which validates a PCI device and virt_id, then
programs the mapping through VFCTRL via amd_viommu_set_device_mapping().
Get the host struct device via iommufd_vdevice_to_device(). Set
vdev->destroy to restore an idle mapping when the vDevice is removed
while the vIOMMU is still alive.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h | 12 +++++
 drivers/iommu/amd/amd_viommu.h      | 10 ++++
 drivers/iommu/amd/iommufd.c         | 52 +++++++++++++++++++
 drivers/iommu/amd/viommu.c          | 79 ++++++++++++++++++++++++++++-
 4 files changed, 152 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index fc8e1378c7ed..07eaeeb72e38 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -547,6 +547,14 @@ struct amd_iommu_viommu {
 	u64 *devid_table;
 	u64 *domid_table;
 
+	/*
+	 * Serializes VFCTRL Guest DID map CONTROL0/CONTROL1 doorbells
+	 * (DevID and DomID table updates) for this gid. Those registers
+	 * are shared by every vDevice on this vIOMMU; igroup locks are
+	 * not. Do not nest with trans_devid_lock.
+	 */
+	struct mutex vfctrl_lock;
+
 	/* Offset for mmap() of guest VF MMIO; set after iommufd_viommu_alloc_mmap(). */
 	unsigned long vfmmio_mmap_offset;
 };
@@ -1113,6 +1121,10 @@ struct amd_irte_ops {
 	void (*clear_allocated)(struct irq_remap_table *, int);
 };
 
+struct amd_iommu_vdevice {
+	struct iommufd_vdevice core;
+};
+
 #ifdef CONFIG_IRQ_REMAP
 extern struct amd_irte_ops irte_32_ops;
 extern struct amd_irte_ops irte_128_ops;
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 580cad0f9f5a..1022ae20c170 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -8,6 +8,9 @@
 
 struct amd_iommu;
 
+/* 16-bit guest DevID index; not VIOMMU_MAX_GID. */
+#define VIOMMU_MAX_GDEVID	0xFFFF
+
 #if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
 
 int amd_viommu_init(struct amd_iommu *iommu);
@@ -20,6 +23,8 @@ int amd_viommu_init_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu
 
 void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu);
 
+void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
+				   u16 hdev_id, u16 gdev_id);
 #else
 
 /*
@@ -41,6 +46,11 @@ static inline u64 amd_viommu_get_vfmmio_addr(struct amd_iommu *iommu, u16 gid)
 	return 0;
 }
 
+static inline void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
+						 u16 hdev_id, u16 gdev_id)
+{
+}
+
 #endif /* CONFIG_AMD_IOMMU_IOMMUFD */
 
 #endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 97de07d3ad10..5760b183ce62 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -70,6 +70,7 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	if (ret)
 		return ret;
 
+	mutex_init(&aviommu->vfctrl_lock);
 	xa_init_flags(&aviommu->gdomid_array, XA_FLAGS_ALLOC1);
 	aviommu->parent = pdom;
 
@@ -121,6 +122,7 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 err_mmap:
 	amd_iommu_gid_free(iommu, aviommu->gid);
 err_gid:
+	mutex_destroy(&aviommu->vfctrl_lock);
 	return ret;
 }
 
@@ -140,6 +142,54 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
 	amd_viommu_uninit_one(iommu, aviommu);
 	amd_iommu_gid_free(iommu, aviommu->gid);
+	mutex_destroy(&aviommu->vfctrl_lock);
+}
+
+/*
+ * Called from drivers/iommu/iommufd/viommu.c: iommufd_vdevice_abort()
+ *
+ * Restore an idle DevID map entry for this virt_id. Every DevID mapping
+ * table slot must have V=1 for both active and non-active devices. A
+ * non-valid entry raises INVALID_DEVICE_REQUEST when the IOMMU processes
+ * INVALIDATE_IOMMU_ALL. Use host device ID 0: that RID is the IOMMU PCI
+ * function and does not normally host an endpoint.
+ */
+static void _amd_viommu_vdevice_destroy(struct iommufd_vdevice *vdev)
+{
+	struct iommufd_viommu *viommu = vdev->viommu;
+	struct amd_iommu_viommu *aviommu =
+		container_of(viommu, struct amd_iommu_viommu, core);
+
+	amd_viommu_set_device_mapping(aviommu, 0, (u16)vdev->virt_id);
+}
+
+/*
+ * Called from drivers/iommu/iommufd/viommu.c: iommufd_vdevice_alloc_ioctl()
+ */
+static int _amd_viommu_vdevice_init(struct iommufd_vdevice *vdev)
+{
+	struct device *dev = iommufd_vdevice_to_device(vdev);
+	struct pci_dev *pdev;
+	struct iommufd_viommu *viommu = vdev->viommu;
+	struct amd_iommu_viommu *aviommu = container_of(viommu, struct amd_iommu_viommu, core);
+
+	if (!dev_is_pci(dev))
+		return -EINVAL;
+	pdev = to_pci_dev(dev);
+
+	if (vdev->virt_id > VIOMMU_MAX_GDEVID) {
+		pr_err("%s: Invalid virtual ID (virt_id=%#llx)\n", __func__, vdev->virt_id);
+		return -EINVAL;
+	}
+
+	pr_debug("%s: gid=%#x, hdev_id=%#x, gdev_id=%#llx\n", __func__,
+		 aviommu->gid, pci_dev_id(pdev), vdev->virt_id);
+
+	amd_viommu_set_device_mapping(aviommu, pci_dev_id(pdev),
+				      (u16)vdev->virt_id);
+	vdev->destroy = _amd_viommu_vdevice_destroy;
+
+	return 0;
 }
 
 /*
@@ -149,4 +199,6 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 static const struct iommufd_viommu_ops amd_viommu_ops = {
 	.alloc_domain_nested = amd_iommu_alloc_domain_nested,
 	.destroy = amd_iommufd_viommu_destroy,
+	.vdevice_size = VDEVICE_STRUCT_SIZE(struct amd_iommu_vdevice, core),
+	.vdevice_init = _amd_viommu_vdevice_init,
 };
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index b160c70f3804..e263d63afe3d 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -6,6 +6,7 @@
 #define pr_fmt(fmt)     "AMD-Vi: " fmt
 #define dev_fmt(fmt)    pr_fmt(fmt)
 
+#include <linux/delay.h>
 #include <linux/iommu.h>
 #include <linux/iommufd.h>
 #include <uapi/linux/iommufd.h>
@@ -29,7 +30,6 @@
  * VIOMMU_MAX_GDEVID / VIOMMU_MAX_GDOMID are 16-bit guest DevID
  * and DomID table indices (hardware mapping-table width), not GID.
  */
-#define VIOMMU_MAX_GDEVID		0xFFFF
 #define VIOMMU_DEVID_MAPPING_BASE	0x1000000000ULL
 #define VIOMMU_DEVID_MAPPING_ENTRY_SIZE	(1 << 20)
 
@@ -45,6 +45,8 @@ static_assert(VIOMMU_DEVID_MAPPING_ENTRY_SIZE >=
 static_assert(VIOMMU_DOMID_MAPPING_ENTRY_SIZE >=
 	      (VIOMMU_MAX_GDOMID + 1ULL) * sizeof(u64));
 
+#define VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET	0x00
+
 static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 {
 	if (iommu->vfctrl_base) {
@@ -448,6 +450,81 @@ static void __maybe_unused free_private_vm_region(struct amd_iommu *iommu, u64 *
 	*entry = NULL;
 }
 
+#define DEVID_ENTRY_GDEVID_MASK		GENMASK_ULL(61, 46)
+#define DEVID_ENTRY_QUEUEID_MASK	GENMASK_ULL(45, 30)
+#define DEVID_ENTRY_HDEVID_MASK		GENMASK_ULL(29, 14)
+#define DEVID_ENTRY_WRITE		BIT_ULL(63)
+#define DEVID_ENTRY_VALID		BIT_ULL(0)
+
+/*
+ * CONTROL0/CONTROL1 bit 63 (WRITE) is a doorbell: hardware copies one
+ * entry into the IPA-backed table and clears WRITE when that DMA write
+ * retires. This is not a command-buffer operation;
+ * iommu_completion_wait() does not drain it. A readq() of CONTROL_EN
+ * is also unrelated. Poll WRITE=0 before ringing a new doorbell and
+ * after each write so the table update is retired before the caller
+ * proceeds (DTE commit, unmap/free).
+ */
+static void vfctrl_wait_write_clear(struct amd_iommu *iommu, u16 gid,
+				    unsigned int offset)
+{
+	u8 __iomem *reg = VIOMMU_VFCTRL_MMIO_BASE(iommu, gid) + offset;
+	int i;
+
+	for (i = 0; i < LOOP_TIMEOUT; i++) {
+		if (!(readq(reg) & DEVID_ENTRY_WRITE))
+			return;
+		udelay(1);
+	}
+	pr_err("%s: WRITE timeout gid=%#x offset=%#x val=%#llx\n",
+	       __func__, gid, offset, readq(reg));
+}
+
+/*
+ * Guest DevID table update via VFCTRL CONTROL0.
+ *
+ * TODO: The driver does not support QueueID yet,
+ *       so current callers pass 0. This will be
+ *       enhanced in the future.
+ */
+static void set_device_mapping_unlocked(struct amd_iommu *iommu, u16 gid,
+					u16 hdev_id, u16 gdev_id, u16 queue_id)
+{
+	u64 val;
+	u8 __iomem *vfctrl = VIOMMU_VFCTRL_MMIO_BASE(iommu, gid);
+
+	val = FIELD_PREP(DEVID_ENTRY_GDEVID_MASK, gdev_id) |
+	      FIELD_PREP(DEVID_ENTRY_QUEUEID_MASK, queue_id) |
+	      FIELD_PREP(DEVID_ENTRY_HDEVID_MASK, hdev_id) |
+	      DEVID_ENTRY_WRITE | DEVID_ENTRY_VALID;
+
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET);
+	writeq(val, vfctrl + VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET);
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET);
+}
+
+/*
+ * Program the DevID via VFCTRL CONTROL0. Waits for WRITE to clear so
+ * the IPA table DMA has retired before return. Serialized by
+ * aviommu->vfctrl_lock: CONTROL0 is shared by all vDevices on this gid.
+ */
+void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
+				   u16 hdev_id, u16 gdev_id)
+{
+	struct amd_iommu *iommu =
+		container_of(aviommu->core.iommu_dev, struct amd_iommu, iommu);
+
+	pr_debug("%s: iommu_devid=%#x, gid=%#x, hdev_id=%#x, gdev_id=%#x\n",
+		 __func__, pci_dev_id(iommu->dev), aviommu->gid, hdev_id,
+		 gdev_id);
+
+	mutex_lock(&aviommu->vfctrl_lock);
+	set_device_mapping_unlocked(iommu, aviommu->gid, hdev_id, gdev_id, 0);
+	mutex_unlock(&aviommu->vfctrl_lock);
+}
+
 void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *aviommu)
 {
 	pr_debug("%s: gid=%u\n", __func__, aviommu->gid);
-- 
2.34.1

[PATCH v6 17/24] iommu/amd: Program nested DTE and DomID map on attach

[Suravee Suthikulpanit]

On nested attach, the host DTE must enable vIOMMU and carry the guest
device and guest IDs from the vDevice. Look up gdev_id with
iommufd_viommu_get_vdev_id() and fail the attach if that lookup fails.

Program the Domain ID mapping table through VFCTRL so hardware can
translate gdom_id to hdom_id. Write CONTROL1 before the DTE is live
with DTE_VIOMMU_EN so guest DMA cannot resolve against the idle
nest-parent map.

DomID CONTROL1 uses the same per-vIOMMU vfctrl_lock as DevID CONTROL0.
Hold that lock across CONTROL1 and the DTE commit, and on last-ref
nested_domain_free() take it before gdomid_array so idle restore
cannot clobber a concurrent alloc+attach.

CONTROL1 WRITE uses the same doorbell as CONTROL0. Poll WRITE until
it clears; iommu_completion_wait() does not drain the table DMA.

On the last nested_domain_free() for a gdom_id, restore the idle
DomID map (nest parent, V=1) before returning hdom_id to the IDA.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h | 10 ++++
 drivers/iommu/amd/amd_viommu.h      |  8 ++++
 drivers/iommu/amd/nested.c          | 74 +++++++++++++++++++++++++----
 drivers/iommu/amd/viommu.c          | 41 ++++++++++++++++
 4 files changed, 125 insertions(+), 8 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 07eaeeb72e38..b5d1b23791da 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -382,6 +382,11 @@
 #define DTE_GPT_LEVEL_SHIFT	54
 #define DTE_GPT_LEVEL_MASK	GENMASK_ULL(55, 54)
 
+/* vIOMMU bit fields */
+#define DTE_VIOMMU_EN_SHIFT		15
+#define DTE_VIOMMU_GDEVICEID_MASK	GENMASK_ULL(31, 16)
+#define DTE_VIOMMU_GUESTID_MASK		GENMASK_ULL(47, 32)
+
 #define GCR3_VALID		0x01ULL
 
 /* DTE[128:179] | DTE[184:191] */
@@ -552,6 +557,11 @@ struct amd_iommu_viommu {
 	 * (DevID and DomID table updates) for this gid. Those registers
 	 * are shared by every vDevice on this vIOMMU; igroup locks are
 	 * not. Do not nest with trans_devid_lock.
+	 *
+	 * nested_domain_free() last-ref takes this before
+	 * gdomid_array's xa_lock so CONTROL1 idle restore cannot race
+	 * a concurrent alloc+attach. Attach holds it across CONTROL1
+	 * and the DTE commit.
 	 */
 	struct mutex vfctrl_lock;
 
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 1022ae20c170..30227a5e4bfa 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -23,6 +23,9 @@ int amd_viommu_init_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu
 
 void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu);
 
+void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *aviommu,
+					u16 hdom_id, u16 gdom_id);
+
 void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
 				   u16 hdev_id, u16 gdev_id);
 #else
@@ -51,6 +54,11 @@ static inline void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviomm
 {
 }
 
+static inline void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *aviommu,
+						      u16 hdom_id, u16 gdom_id)
+{
+}
+
 #endif /* CONFIG_AMD_IOMMU_IOMMUFD */
 
 #endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/nested.c b/drivers/iommu/amd/nested.c
index 8bb0d1596c6c..ecdcecdc0bb7 100644
--- a/drivers/iommu/amd/nested.c
+++ b/drivers/iommu/amd/nested.c
@@ -10,6 +10,7 @@
 #include <uapi/linux/iommufd.h>
 
 #include "amd_iommu.h"
+#include "amd_viommu.h"
 
 static const struct iommu_domain_ops nested_domain_ops;
 
@@ -188,13 +189,16 @@ amd_iommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags,
 	return ERR_PTR(ret);
 }
 
-static void set_dte_nested(struct amd_iommu *iommu, struct iommu_domain *dom,
-			   struct iommu_dev_data *dev_data, struct dev_table_entry *new)
+static int set_dte_nested(struct amd_iommu *iommu, struct iommu_domain *dom,
+			  struct iommu_dev_data *dev_data, struct dev_table_entry *new)
 {
+	int ret;
+	u16 gid;
 	struct protection_domain *parent;
 	struct nested_domain *ndom = to_ndomain(dom);
 	struct iommu_hwpt_amd_guest *gdte = &ndom->gdte;
 	struct pt_iommu_amdv1_hw_info pt_info;
+	unsigned long gdev_id;
 
 	/*
 	 * The nest parent domain is attached during the call to the
@@ -202,9 +206,15 @@ static void set_dte_nested(struct amd_iommu *iommu, struct iommu_domain *dom,
 	 * of the struct amd_iommu_viommu.parent.
 	 */
 	if (WARN_ON(!ndom->viommu || !ndom->viommu->parent))
-		return;
+		return -EINVAL;
 
+	gid = ndom->viommu->gid;
 	parent = ndom->viommu->parent;
+
+	ret = iommufd_viommu_get_vdev_id(&ndom->viommu->core, dev_data->dev, &gdev_id);
+	if (ret)
+		return ret;
+
 	amd_iommu_make_clear_dte(iommu, dev_data->devid, new);
 
 	/* Retrieve the current pagetable info via the IOMMU PT API. */
@@ -232,12 +242,19 @@ static void set_dte_nested(struct amd_iommu *iommu, struct iommu_domain *dom,
 
 	/* Guest paging mode */
 	new->data[2] |= gdte->dte[2] & DTE_GPT_LEVEL_MASK;
+
+	new->data[3] |= 1ULL << DTE_VIOMMU_EN_SHIFT;
+	new->data[3] |= FIELD_PREP(DTE_VIOMMU_GDEVICEID_MASK, gdev_id);
+	new->data[3] |= FIELD_PREP(DTE_VIOMMU_GUESTID_MASK, gid);
+
+	return 0;
 }
 
 static int nested_attach_device(struct iommu_domain *dom, struct device *dev,
 				struct iommu_domain *old)
 {
 	struct dev_table_entry new = {0};
+	struct nested_domain *ndom = to_ndomain(dom);
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
 	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
 	int ret = 0;
@@ -251,10 +268,23 @@ static int nested_attach_device(struct iommu_domain *dom, struct device *dev,
 
 	mutex_lock(&dev_data->mutex);
 
-	set_dte_nested(iommu, dom, dev_data, &new);
+	ret = set_dte_nested(iommu, dom, dev_data, &new);
+	if (ret)
+		goto out_err;
 
+	/*
+	 * Program gdom_id -> hdom_id before the DTE is live with
+	 * DTE_VIOMMU_EN. Hold vfctrl_lock across both so a concurrent
+	 * last-ref free cannot restore the idle map in between.
+	 */
+	mutex_lock(&ndom->viommu->vfctrl_lock);
+	amd_viommu_domain_id_update_locked(ndom->viommu,
+					   ndom->gdom_info->hdom_id,
+					   ndom->gdom_id);
 	amd_iommu_update_dte(iommu, dev_data, &new);
+	mutex_unlock(&ndom->viommu->vfctrl_lock);
 
+out_err:
 	mutex_unlock(&dev_data->mutex);
 
 	return ret;
@@ -267,10 +297,18 @@ static void nested_domain_free(struct iommu_domain *dom)
 	struct nested_domain *ndom __free(kfree) = to_ndomain(dom);
 	struct amd_iommu_viommu *aviommu = ndom->viommu;
 
+	/*
+	 * vfctrl_lock then gdomid_array: keep the slot empty in the
+	 * xarray until CONTROL1 is idle so a concurrent alloc+attach
+	 * for this gdom_id cannot publish H2 and then lose it to this
+	 * restore. Attach takes the same lock around CONTROL1 + DTE.
+	 */
+	mutex_lock(&aviommu->vfctrl_lock);
 	xa_lock_irqsave(&aviommu->gdomid_array, irqflags);
 
 	if (!refcount_dec_and_test(&ndom->gdom_info->users)) {
 		xa_unlock_irqrestore(&aviommu->gdomid_array, irqflags);
+		mutex_unlock(&aviommu->vfctrl_lock);
 		return;
 	}
 
@@ -282,14 +320,34 @@ static void nested_domain_free(struct iommu_domain *dom)
 			    ndom->gdom_info, NULL, GFP_ATOMIC);
 
 	xa_unlock_irqrestore(&aviommu->gdomid_array, irqflags);
-	if (WARN_ON(!curr || xa_err(curr)))
+	if (WARN_ON(!curr || xa_err(curr))) {
+		mutex_unlock(&aviommu->vfctrl_lock);
 		return;
+	}
 
-	/* success */
 	pr_debug("%s: Free gdom_id=%#x, hdom_id=%#x\n",
-		__func__, ndom->gdom_id, curr->hdom_id);
+		 __func__, ndom->gdom_id, curr->hdom_id);
+
+	/*
+	 * Restore the idle DomID map before releasing hdom_id. Every
+	 * slot stays V=1; unknown mapping raises an event. Idle is the
+	 * nest parent CONTROL1 map, same as vIOMMU init prefill.
+	 * amd_iommu_reset_vmmio() only resets guest MMIO.
+	 *
+	 * Guest INVALIDATE_IOMMU_ALL walks the HW table, not
+	 * gdomid_array. The pdom_ids IDA is global, so a stale V=1
+	 * entry can invalidate a recycled hdom_id that now belongs to
+	 * another domain.
+	 *
+	 * Detach does not clear this slot: hdom_id stays allocated
+	 * until this last free, so a lingering gdom_id -> hdom_id
+	 * map cannot point at a recycled ID.
+	 */
+	amd_viommu_domain_id_update_locked(aviommu, aviommu->parent->id,
+					   ndom->gdom_id);
+	mutex_unlock(&aviommu->vfctrl_lock);
 
-	amd_iommu_pdom_id_free(ndom->gdom_info->hdom_id);
+	amd_iommu_pdom_id_free(curr->hdom_id);
 	kfree(curr);
 }
 
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index e263d63afe3d..e2275d6b7442 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -46,6 +46,7 @@ static_assert(VIOMMU_DOMID_MAPPING_ENTRY_SIZE >=
 	      (VIOMMU_MAX_GDOMID + 1ULL) * sizeof(u64));
 
 #define VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET	0x00
+#define VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL1_OFFSET	0x08
 
 static void __init amd_viommu_vf_vfcntl_unmap(struct amd_iommu *iommu)
 {
@@ -525,6 +526,46 @@ void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
 	mutex_unlock(&aviommu->vfctrl_lock);
 }
 
+#define DOMID_ENTRY_GDOMID_MASK	GENMASK_ULL(61, 46)
+#define DOMID_ENTRY_HDOMID_MASK	GENMASK_ULL(29, 14)
+#define DOMID_ENTRY_VALID		BIT_ULL(0)
+#define DOMID_ENTRY_WRITE		BIT_ULL(63)
+
+/*
+ * Guest DomID table update via VFCTRL CONTROL1. Same WRITE-bit
+ * doorbell as CONTROL0.
+ */
+static void domain_id_update_unlocked(struct amd_iommu *iommu, u16 gid,
+				      u16 hdom_id, u16 gdom_id)
+{
+	u64 val;
+	u8 __iomem *vfctrl = VIOMMU_VFCTRL_MMIO_BASE(iommu, gid);
+
+	val = FIELD_PREP(DOMID_ENTRY_GDOMID_MASK, gdom_id) |
+	      FIELD_PREP(DOMID_ENTRY_HDOMID_MASK, hdom_id) |
+	      DOMID_ENTRY_WRITE | DOMID_ENTRY_VALID;
+
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL1_OFFSET);
+	writeq(val, vfctrl + VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL1_OFFSET);
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL1_OFFSET);
+}
+
+/*
+ * Program a gdom_id -> hdom_id entry through VFCTRL CONTROL1.
+ * Caller must hold aviommu->vfctrl_lock.
+ */
+void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *aviommu,
+					u16 hdom_id, u16 gdom_id)
+{
+	struct amd_iommu *iommu =
+		container_of(aviommu->core.iommu_dev, struct amd_iommu, iommu);
+
+	lockdep_assert_held(&aviommu->vfctrl_lock);
+	domain_id_update_unlocked(iommu, aviommu->gid, hdom_id, gdom_id);
+}
+
 void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *aviommu)
 {
 	pr_debug("%s: gid=%u\n", __func__, aviommu->gid);
-- 
2.34.1

[PATCH v6 18/24] iommu/amd: Init and clear vIOMMU DevID and DomID maps

[Suravee Suthikulpanit]

Before using a vIOMMU instance, prefill the mapping tables so hardware
never sees an unknown guest ID. Map every gdom_id to the nest parent
domain ID, and write every gdev_id with hdev_id 0. Init cannot be
scoped to IDs in use; INVALIDATE_IOMMU_ALL walks the HW table.

Yield vfctrl_lock every 256 doorbells so the 128K MMIO loop can
resched. Attach cannot run until viommu->ops is set after init_one().

On destroy, drain CONTROL0/CONTROL1 WRITE and unmap. Do not rewrite
the full tables: that would start 128K DMA writes into pages about
to be freed. vDevice destroy and nested_domain_free already restored
idle entries.

Skip VFCTRL access when CONTROL_CMDBUF_EN is already clear, matching
viommu_priv_unmap_free() on the same uninit_one() path.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/viommu.c | 90 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index e2275d6b7442..2f299a16d113 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -7,6 +7,7 @@
 #define dev_fmt(fmt)    pr_fmt(fmt)
 
 #include <linux/delay.h>
+#include <linux/sched.h>
 #include <linux/iommu.h>
 #include <linux/iommufd.h>
 #include <uapi/linux/iommufd.h>
@@ -457,6 +458,8 @@ static void __maybe_unused free_private_vm_region(struct amd_iommu *iommu, u64 *
 #define DEVID_ENTRY_WRITE		BIT_ULL(63)
 #define DEVID_ENTRY_VALID		BIT_ULL(0)
 
+#define VIOMMU_VFCTRL_PREFILL_RESCHED	256
+
 /*
  * CONTROL0/CONTROL1 bit 63 (WRITE) is a doorbell: hardware copies one
  * entry into the IPA-backed table and clears WRITE when that DMA write
@@ -566,10 +569,95 @@ void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *aviommu,
 	domain_id_update_unlocked(iommu, aviommu->gid, hdom_id, gdom_id);
 }
 
+static bool viommu_vfctrl_ready(struct amd_iommu *iommu)
+{
+	return iommu->vfctrl_base && viommu_cmdbuf_enabled(iommu);
+}
+
+/*
+ * Drain outstanding CONTROL0/CONTROL1 table DMA before the IPA pages
+ * are unmapped and freed. Do not rewrite 0..0xFFFF: vDevice destroy
+ * and nested_domain_free already restored idle entries, and a full
+ * walk would start 128K new DMA writes into pages about to be freed.
+ *
+ * Skip after disable_iommus(): CMDBUF_EN is already clear, matching
+ * viommu_priv_unmap_free() which skips IOTLB invalidation.
+ */
+static void viommu_drain_mapping(struct amd_iommu *iommu,
+				 struct amd_iommu_viommu *aviommu)
+{
+	u16 gid = aviommu->gid;
+
+	if (!viommu_vfctrl_ready(iommu))
+		return;
+
+	mutex_lock(&aviommu->vfctrl_lock);
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL0_OFFSET);
+	vfctrl_wait_write_clear(iommu, gid,
+			VIOMMU_VFCTRL_GUEST_DID_MAP_CONTROL1_OFFSET);
+	mutex_unlock(&aviommu->vfctrl_lock);
+}
+
+static void viommu_prefill_resched(struct amd_iommu_viommu *aviommu, int i)
+{
+	lockdep_assert_held(&aviommu->vfctrl_lock);
+
+	if ((i & (VIOMMU_VFCTRL_PREFILL_RESCHED - 1)) !=
+	    VIOMMU_VFCTRL_PREFILL_RESCHED - 1)
+		return;
+
+	mutex_unlock(&aviommu->vfctrl_lock);
+	cond_resched();
+	mutex_lock(&aviommu->vfctrl_lock);
+}
+
+/*
+ * Prefill every guest ID so hardware never sees an unknown mapping.
+ * Map all gdom_id to the nest parent domain ID, and all gdev_id with
+ * hdev_id 0 and QueueID 0. Cannot be scoped to IDs in use: none exist
+ * yet, and INVALIDATE_IOMMU_ALL walks the HW table.
+ *
+ * Drop vfctrl_lock every VIOMMU_VFCTRL_PREFILL_RESCHED doorbells so
+ * the 128K MMIO loop can yield. Safe on init: vDevice/nested attach
+ * cannot run until viommu->ops is published after init_one().
+ */
+static void viommu_prefill_mapping(struct amd_iommu *iommu,
+				   struct amd_iommu_viommu *aviommu)
+{
+	int i;
+	u16 gid = aviommu->gid;
+
+	if (!viommu_vfctrl_ready(iommu))
+		return;
+
+	mutex_lock(&aviommu->vfctrl_lock);
+	if (aviommu->domid_table) {
+		for (i = 0; i <= VIOMMU_MAX_GDOMID; i++) {
+			domain_id_update_unlocked(iommu, gid,
+						  aviommu->parent->id, i);
+			viommu_prefill_resched(aviommu, i);
+		}
+	}
+
+	if (aviommu->devid_table) {
+		for (i = 0; i <= VIOMMU_MAX_GDEVID; i++) {
+			set_device_mapping_unlocked(iommu, gid, 0, i, 0);
+			viommu_prefill_resched(aviommu, i);
+		}
+	}
+	mutex_unlock(&aviommu->vfctrl_lock);
+}
+
 void amd_viommu_uninit_one(struct amd_iommu *iommu, struct amd_iommu_viommu *aviommu)
 {
 	pr_debug("%s: gid=%u\n", __func__, aviommu->gid);
 
+	/*
+	 * Wait until CONTROL0/CONTROL1 table DMA has retired before
+	 * the backing pages are unmapped and freed.
+	 */
+	viommu_drain_mapping(iommu, aviommu);
 	free_private_vm_region(iommu, &aviommu->devid_table,
 			       VIOMMU_DEVID_MAPPING_BASE,
 			       VIOMMU_DEVID_MAPPING_ENTRY_SIZE,
@@ -598,6 +686,8 @@ int amd_viommu_init_one(struct amd_iommu *iommu, struct amd_iommu_viommu *viommu
 	if (ret)
 		goto err_out;
 
+	viommu_prefill_mapping(iommu, viommu);
+
 	return 0;
 err_out:
 	amd_viommu_uninit_one(iommu, viommu);
-- 
2.34.1

[PATCH v6 19/24] iommu/amd: Add per-segment translate device ID pool

[Suravee Suthikulpanit]

A translate device ID (TransDevID) is a host device-table index used
by hardware vIOMMU. VFCTRL guest-misc names the slot; that DTE holds
the nest-parent v1 table for GPA->SPA walks of guest IOMMU traffic.

Reserve PCI requestor IDs in a per-segment pool so they are not
used as vIOMMU translation DTEs. The device table is per-segment,
so the pool is too.

Add trans_devid.c with pci_seg init/fini and
amd_iommu_trans_devid_reserve() from probe_device(). Keep
reservations for the pci_seg lifetime, including after
amd_iommu_release_device().

The IOMMU function never reaches probe_device(): rlookup is cleared
at iommu->devid, while the self DTE still occupies that slot. Reserve
it in init_iommu_one_late() for the same pci_seg lifetime.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/Makefile          |  2 +-
 drivers/iommu/amd/amd_iommu_types.h | 10 ++++
 drivers/iommu/amd/amd_viommu.h      | 20 +++++++
 drivers/iommu/amd/init.c            | 13 +++++
 drivers/iommu/amd/iommu.c           | 22 +++++++
 drivers/iommu/amd/trans_devid.c     | 90 +++++++++++++++++++++++++++++
 drivers/iommu/amd/viommu.c          |  3 +
 7 files changed, 159 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/amd/trans_devid.c

diff --git a/drivers/iommu/amd/Makefile b/drivers/iommu/amd/Makefile
index e1e824b9c7b0..12c3fe83e4ce 100644
--- a/drivers/iommu/amd/Makefile
+++ b/drivers/iommu/amd/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-y += iommu.o init.o quirks.o ppr.o pasid.o
-obj-$(CONFIG_AMD_IOMMU_IOMMUFD) += iommufd.o nested.o viommu.o
+obj-$(CONFIG_AMD_IOMMU_IOMMUFD) += iommufd.o nested.o viommu.o trans_devid.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += debugfs.o
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index b5d1b23791da..6b3a43952797 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -681,6 +681,16 @@ struct amd_iommu_pci_seg {
 	 * parsing time.
 	 */
 	struct list_head unity_map;
+
+#ifdef CONFIG_AMD_IOMMU_IOMMUFD
+	/*
+	 * Per-segment translate-device-id pool indexed by id.  Entries are:
+	 * absent (FREE), xa value TRANS_DEVID_RESERVED, or a pointer to the
+	 * owning struct amd_iommu_viommu (ALLOCATED).
+	 */
+	struct mutex trans_devid_mutex;
+	struct xarray trans_devid_xa;
+#endif
 };
 
 /*
diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 30227a5e4bfa..772d99391b6c 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -28,6 +28,10 @@ void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *aviommu,
 
 void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
 				   u16 hdev_id, u16 gdev_id);
+
+void amd_iommu_pci_seg_trans_devid_init(struct amd_iommu_pci_seg *pci_seg);
+void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg);
+int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id);
 #else
 
 /*
@@ -59,6 +63,22 @@ static inline void amd_viommu_domain_id_update_locked(struct amd_iommu_viommu *a
 {
 }
 
+static inline void
+amd_iommu_pci_seg_trans_devid_init(struct amd_iommu_pci_seg *pci_seg)
+{
+}
+
+static inline int
+amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id)
+{
+	return 0;
+}
+
+static inline void
+amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg)
+{
+}
+
 #endif /* CONFIG_AMD_IOMMU_IOMMUFD */
 
 #endif /* AMD_VIOMMU_H */
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 96299f10b2e8..59a06c3014fe 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1747,6 +1747,7 @@ static struct amd_iommu_pci_seg *__init alloc_pci_segment(u16 id,
 	xa_init(&pci_seg->dev_data_xa);
 	INIT_LIST_HEAD(&pci_seg->unity_map);
 	list_add_tail(&pci_seg->list, &amd_iommu_pci_seg_list);
+	amd_iommu_pci_seg_trans_devid_init(pci_seg);
 
 	if (alloc_dev_table(pci_seg))
 		goto err_free_pci_seg;
@@ -1763,6 +1764,7 @@ static struct amd_iommu_pci_seg *__init alloc_pci_segment(u16 id,
 	free_dev_table(pci_seg);
 err_free_pci_seg:
 	list_del(&pci_seg->list);
+	amd_iommu_pci_seg_trans_devid_fini(pci_seg);
 	xa_destroy(&pci_seg->dev_data_xa);
 	kfree(pci_seg);
 	return NULL;
@@ -1787,6 +1789,7 @@ static void __init free_pci_segments(void)
 
 	for_each_pci_segment_safe(pci_seg, next) {
 		list_del(&pci_seg->list);
+		amd_iommu_pci_seg_trans_devid_fini(pci_seg);
 		xa_destroy(&pci_seg->dev_data_xa);
 		free_irq_lookup_table(pci_seg);
 		free_rlookup_table(pci_seg);
@@ -2006,8 +2009,18 @@ static int __init init_iommu_one_late(struct amd_iommu *iommu)
 	/*
 	 * Make sure IOMMU is not considered to translate itself. The IVRS
 	 * table tells us so, but this is a lie!
+	 *
+	 * Clearing rlookup keeps this function off amd_iommu_probe_device(),
+	 * so its devid is never reserved there. The vIOMMU self DTE is
+	 * still programmed at iommu->devid. Reserve it here so translate-id
+	 * alloc does not treat the empty xarray slot as free.
 	 */
 	iommu->pci_seg->rlookup_table[iommu->devid] = NULL;
+	ret = amd_iommu_trans_devid_reserve(iommu->pci_seg, iommu->devid);
+	if (ret) {
+		pr_err("Failed to reserve IOMMU devid %#x\n", iommu->devid);
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index b7073b0ca8b0..17716a506f93 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2738,6 +2738,25 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 	if (dev_is_pci(dev))
 		pci_prepare_ats(to_pci_dev(dev), PAGE_SHIFT);
 
+#if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
+	/*
+	 * Occupy this RID in the per-segment translate-device-id pool
+	 * so vIOMMU alloc cannot reuse it. Probe is the RID lifetime
+	 * for devices that reach it. The IOMMU function is reserved
+	 * from init_iommu_one_late() instead: rlookup is cleared, so
+	 * probe never runs, while the self DTE still uses iommu->devid.
+	 * The reservation is not released on unplug. If reserve fails,
+	 * the IOMMU core frees dev->iommu and a later re-probe retries.
+	 */
+	ret = amd_iommu_trans_devid_reserve(iommu->pci_seg, dev_data->devid);
+	if (ret) {
+		pr_err("%s: Failed to reserve device id %#x\n", __func__,
+		       dev_data->devid);
+		iommu_dev = ERR_PTR(ret);
+		goto out_err;
+	}
+#endif
+
 out_err:
 	return iommu_dev;
 }
@@ -2751,6 +2770,9 @@ static void amd_iommu_release_device(struct device *dev)
 	/*
 	 * We keep dev_data around for unplugged devices and reuse it when the
 	 * device is re-plugged - not doing so would introduce a ton of races.
+	 * Translate-device-id reservations stay as well: the RID remains
+	 * occupied for the pci_seg lifetime so later alloc cannot reuse a
+	 * DTE index that still belongs to this function.
 	 */
 }
 
diff --git a/drivers/iommu/amd/trans_devid.c b/drivers/iommu/amd/trans_devid.c
new file mode 100644
index 000000000000..e0b637138260
--- /dev/null
+++ b/drivers/iommu/amd/trans_devid.c
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 Advanced Micro Devices, Inc.
+ *
+ * AMD vIOMMU translate-device-id management.
+ *
+ * The pool is per PCI segment because the AMD IOMMU device table is
+ * per-segment.  Each id must be allocated from unused slots in that
+ * segment.  It is used to program the vIOMMU VF Control register to
+ * specify the DTE used to contain the GPA->SPA mapping (v1 page table).
+ */
+
+#include <linux/kernel.h>
+#include <linux/xarray.h>
+
+#include "amd_iommu.h"
+#include "amd_viommu.h"
+
+enum trans_devid_state {
+	TRANS_DEVID_FREE = 0,
+	TRANS_DEVID_RESERVED,
+};
+
+static inline bool trans_devid_xa_is_reserved(void *entry)
+{
+	return entry && xa_is_value(entry) &&
+	       xa_to_value(entry) == TRANS_DEVID_RESERVED;
+}
+
+static inline void *trans_devid_xa_mk_reserved(void)
+{
+	return xa_mk_value(TRANS_DEVID_RESERVED);
+}
+
+static int trans_devid_xa_install_reserved_locked(struct amd_iommu_pci_seg *pci_seg,
+						  u16 id)
+{
+	void *old;
+
+	old = xa_store(&pci_seg->trans_devid_xa, id,
+		       trans_devid_xa_mk_reserved(), GFP_KERNEL);
+	if (xa_is_err(old))
+		return xa_err(old);
+	WARN_ON_ONCE(old);
+	return 0;
+}
+
+void amd_iommu_pci_seg_trans_devid_init(struct amd_iommu_pci_seg *pci_seg)
+{
+	mutex_init(&pci_seg->trans_devid_mutex);
+	xa_init(&pci_seg->trans_devid_xa);
+}
+
+void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg)
+{
+	xa_destroy(&pci_seg->trans_devid_xa);
+	mutex_destroy(&pci_seg->trans_devid_mutex);
+}
+
+/*
+ * amd_iommu_trans_devid_reserve - occupy @id so it is never returned by alloc
+ *
+ * Reservation is done when probing the device (see amd_iommu_probe_device())
+ * and for the IOMMU function in init_iommu_one_late(). That function never
+ * probes: rlookup_table[iommu->devid] is cleared, but the vIOMMU self DTE
+ * still occupies the slot.
+ * It is not released from amd_iommu_release_device(); the slot stays
+ * reserved for the pci_seg lifetime so replug cannot race later alloc.
+ *
+ * Return: 0 on success.  A second reserve of an already-reserved @id succeeds.
+ */
+int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id)
+{
+	void *entry;
+	int ret = 0;
+
+	mutex_lock(&pci_seg->trans_devid_mutex);
+	entry = xa_load(&pci_seg->trans_devid_xa, id);
+	if (trans_devid_xa_is_reserved(entry))
+		goto unlock;
+
+	ret = trans_devid_xa_install_reserved_locked(pci_seg, id);
+unlock:
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+
+	if (!ret)
+		pr_debug("%s: Reserved trans_devid %#x (seg %#x)\n", __func__, id,
+			 pci_seg->id);
+	return ret;
+}
diff --git a/drivers/iommu/amd/viommu.c b/drivers/iommu/amd/viommu.c
index 2f299a16d113..ff5394e6a4c7 100644
--- a/drivers/iommu/amd/viommu.c
+++ b/drivers/iommu/amd/viommu.c
@@ -110,6 +110,9 @@ static int viommu_alloc_self_dev_data(struct amd_iommu *iommu)
 	 * Synthetic DTE for iommu->devid only: no struct device, so
 	 * amd_iommu_update_dte() skips clone_aliases(), and not on
 	 * pdom->dev_list (not an IOMMU-API attach).
+	 *
+	 * init_iommu_one_late() reserves this devid. rlookup is NULL, so
+	 * probe never runs, and alloc must not reuse the self DTE slot.
 	 */
 	dev_data->dev = NULL;
 	dev_data->domain = pdom;
-- 
2.34.1

[PATCH v6 20/24] iommu/amd: Reserve translate-device-id for PCI requestor aliases

[Suravee Suthikulpanit]

The per-segment translate-device-id pool tracks which device-table
indices are reserved for real PCI functions versus available for
dynamic allocation to vIOMMU translation DTEs.
amd_iommu_probe_device() already calls amd_iommu_trans_devid_reserve()
for the struct device's own PCI BDF so that id cannot later be returned
by the allocator.

That is not sufficient on its own. The AMD IOMMU driver programs
identical DMA translation device-table entries (DTEs) for every
requestor ID that can issue DMA on behalf of the same PCI function: the
IVRS alias from alias_table[] when it is not covered by the PCI
DMA-alias walk (different bus than the device), and every alias visited
by pci_for_each_dma_alias(). Those alternate BDFs are not separate
struct device probe targets, so they never received a trans_devid
reservation and could in principle collide with a dynamically allocated
trans_devid.

Introduce amd_iommu_trans_devid_reserve_pci_aliases() in trans_devid.c
and invoke it from amd_iommu_probe_device() immediately after the
primary amd_iommu_trans_devid_reserve() succeeds. For PCI devices the
helper reserves the IVRS alias when it differs from the device BDF,
then walks pci_for_each_dma_alias() and reserves each alias BDF.
Repeated probe and overlap with the primary BDF in the PCI walk are
handled by the existing idempotency of amd_iommu_trans_devid_reserve()
(a second reserve of an already-reserved id succeeds).

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_viommu.h  |  2 ++
 drivers/iommu/amd/iommu.c       |  8 ++++++
 drivers/iommu/amd/trans_devid.c | 45 +++++++++++++++++++++++++++++++++
 3 files changed, 55 insertions(+)

diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 772d99391b6c..44a342608d43 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -32,6 +32,8 @@ void amd_viommu_set_device_mapping(struct amd_iommu_viommu *aviommu,
 void amd_iommu_pci_seg_trans_devid_init(struct amd_iommu_pci_seg *pci_seg);
 void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg);
 int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id);
+int amd_iommu_trans_devid_reserve_pci_aliases(struct amd_iommu *iommu,
+					      struct device *dev);
 #else
 
 /*
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 17716a506f93..65550b070a12 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2755,6 +2755,14 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 		iommu_dev = ERR_PTR(ret);
 		goto out_err;
 	}
+
+	ret = amd_iommu_trans_devid_reserve_pci_aliases(iommu, dev);
+	if (ret) {
+		pr_err("%s: Failed to reserve translate devid for alias of %#x (err %d)\n",
+		       __func__, dev_data->devid, ret);
+		iommu_dev = ERR_PTR(ret);
+		goto out_err;
+	}
 #endif
 
 out_err:
diff --git a/drivers/iommu/amd/trans_devid.c b/drivers/iommu/amd/trans_devid.c
index e0b637138260..f51a332fbc72 100644
--- a/drivers/iommu/amd/trans_devid.c
+++ b/drivers/iommu/amd/trans_devid.c
@@ -11,6 +11,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/pci.h>
 #include <linux/xarray.h>
 
 #include "amd_iommu.h"
@@ -88,3 +89,47 @@ int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id)
 			 pci_seg->id);
 	return ret;
 }
+
+static int reserve_trans_devid_each_dma_alias(struct pci_dev *pdev, u16 alias,
+					      void *data)
+{
+	struct amd_iommu_pci_seg *pci_seg = data;
+
+	(void)pdev;
+	return amd_iommu_trans_devid_reserve(pci_seg, alias);
+}
+
+/**
+ * amd_iommu_trans_devid_reserve_pci_aliases - reserve translate-device-ids for
+ * PCI DMA aliases and for the IVRS alias when it is not walked as a PCI DMA
+ * alias (different bus). Idempotent for repeated probe; see
+ * amd_iommu_trans_devid_reserve().
+ *
+ * Return: 0 on success or if @dev is not PCI; otherwise an errno from
+ * amd_iommu_trans_devid_reserve() or pci_for_each_dma_alias().
+ */
+int amd_iommu_trans_devid_reserve_pci_aliases(struct amd_iommu *iommu,
+					      struct device *dev)
+{
+	struct pci_dev *pdev;
+	struct amd_iommu_pci_seg *pci_seg;
+	u16 devid, ivrs_alias;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return 0;
+
+	pdev = to_pci_dev(dev);
+	pci_seg = iommu->pci_seg;
+	devid = pci_dev_id(pdev);
+
+	ivrs_alias = pci_seg->alias_table[devid];
+	if (ivrs_alias != devid) {
+		ret = amd_iommu_trans_devid_reserve(pci_seg, ivrs_alias);
+		if (ret)
+			return ret;
+	}
+
+	return pci_for_each_dma_alias(pdev, reserve_trans_devid_each_dma_alias,
+				      pci_seg);
+}
-- 
2.34.1

[PATCH v6 21/24] iommu/amd: Add translation DTE and VFctrl TransDevID helpers

[Suravee Suthikulpanit]

vIOMMU GPA->SPA walks use a per-vIOMMU TransDevID: VFCTRL guest-misc
names the slot, and that host DTE points at the nest-parent v1 table.

Add amd_iommu_set_translate_dte() and amd_iommu_clear_translate_dte()
to program that DTE, owned on amd_iommu_viommu.trans_dev_data rather
than pci_seg->dev_data_xa. Rewrite it from amd_iommu_change_top() via
pdom->viommu_list when the nest-parent table grows.

Add amd_iommu_update_vfctrl_mmio_translate_devid() to publish the
TransDevID in VFCTRL guest-misc MMIO.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |   7 ++
 drivers/iommu/amd/amd_iommu_types.h |   4 +
 drivers/iommu/amd/iommu.c           | 153 ++++++++++++++++++++++++----
 3 files changed, 142 insertions(+), 22 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 6ac64524208b..4aede6f6d489 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -213,6 +213,13 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
 			  struct iommu_dev_data *dev_data,
 			  struct dev_table_entry *new);
 
+int amd_iommu_set_translate_dte(struct iommufd_viommu *viommu);
+void amd_iommu_clear_translate_dte(struct amd_iommu *iommu,
+				   struct iommu_dev_data *dev_data,
+				   u16 trans_devid);
+void amd_iommu_update_vfctrl_mmio_translate_devid(struct amd_iommu *iommu,
+						  u16 gid, u32 trans_devid);
+
 static inline void
 amd_iommu_make_clear_dte(struct amd_iommu *iommu, u16 devid,
 			 struct dev_table_entry *new)
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 6b3a43952797..5f84f25e3f37 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -489,6 +489,7 @@ extern bool amdr_ivrs_remap_support;
 /* VIOMMU stuff */
 #define VIOMMU_VF_MMIO_ENTRY_SIZE		4096
 #define VIOMMU_VFCTRL_MMIO_ENTRY_SIZE		64
+#define VIOMMU_VFCTRL_GUEST_MISC_CONTROL_OFFSET	0x10
 
 /* Host ioremap/request_mem_region sizes for VF / VF_CNTL BARs */
 #define VIOMMU_VF_MMIO_MAP_SIZE		0x10000000UL
@@ -551,6 +552,9 @@ struct amd_iommu_viommu {
 
 	u64 *devid_table;
 	u64 *domid_table;
+	u16 trans_devid;
+	/* Synthetic DTE; not stored in pci_seg->dev_data_xa */
+	struct iommu_dev_data *trans_dev_data;
 
 	/*
 	 * Serializes VFCTRL Guest DID map CONTROL0/CONTROL1 doorbells
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 65550b070a12..8943863105b6 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -148,11 +148,11 @@ static void write_dte_lower128(struct dev_table_entry *ptr, struct dev_table_ent
  * So, only consider control bits related to DMA when updating the entry.
  */
 static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_data,
-			  struct dev_table_entry *new)
+			  u16 devid, struct dev_table_entry *new)
 {
 	unsigned long flags;
 	struct dev_table_entry *dev_table = get_dev_table(iommu);
-	struct dev_table_entry *ptr = &dev_table[dev_data->devid];
+	struct dev_table_entry *ptr = &dev_table[devid];
 
 	spin_lock_irqsave(&dev_data->dte_lock, flags);
 
@@ -160,12 +160,12 @@ static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_da
 		/* Existing DTE is not valid. */
 		write_dte_upper128(ptr, new);
 		write_dte_lower128(ptr, new);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 	} else if (!(new->data[0] & DTE_FLAG_V)) {
 		/* Existing DTE is valid. New DTE is not valid.  */
 		write_dte_lower128(ptr, new);
 		write_dte_upper128(ptr, new);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 	} else if (!FIELD_GET(DTE_FLAG_GV, ptr->data[0])) {
 		/*
 		 * Both DTEs are valid.
@@ -173,7 +173,7 @@ static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_da
 		 */
 		write_dte_upper128(ptr, new);
 		write_dte_lower128(ptr, new);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 	} else if (!FIELD_GET(DTE_FLAG_GV, new->data[0])) {
 		/*
 		 * Both DTEs are valid.
@@ -182,7 +182,7 @@ static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_da
 		 */
 		write_dte_lower128(ptr, new);
 		write_dte_upper128(ptr, new);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 	} else if (FIELD_GET(DTE_GPT_LEVEL_MASK, ptr->data[2]) !=
 		   FIELD_GET(DTE_GPT_LEVEL_MASK, new->data[2])) {
 		/*
@@ -195,12 +195,12 @@ static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_da
 
 		/* First disable DTE */
 		write_dte_lower128(ptr, &clear);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 
 		/* Then update DTE */
 		write_dte_upper128(ptr, new);
 		write_dte_lower128(ptr, new);
-		iommu_flush_dte_sync(iommu, dev_data->devid);
+		iommu_flush_dte_sync(iommu, devid);
 	} else {
 		/*
 		 * Both DTEs are valid and have guest page table,
@@ -217,7 +217,7 @@ void amd_iommu_update_dte(struct amd_iommu *iommu,
 			     struct iommu_dev_data *dev_data,
 			     struct dev_table_entry *new)
 {
-	update_dte256(iommu, dev_data, new);
+	update_dte256(iommu, dev_data, dev_data->devid, new);
 	/*
 	 * Synthetic DTEs (vIOMMU self, translate-device-id) leave
 	 * dev_data->dev NULL so clone_aliases() is skipped.
@@ -497,7 +497,7 @@ static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data)
 		ret = -EINVAL;
 		goto out;
 	}
-	update_dte256(iommu, alias_data, &new);
+	update_dte256(iommu, alias_data, alias, &new);
 
 	amd_iommu_set_rlookup_table(iommu, alias);
 out:
@@ -2850,28 +2850,37 @@ static spinlock_t *amd_iommu_get_top_lock(struct pt_iommu *iommupt)
 
 #if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
 /*
- * The vIOMMU private IPA domain programs a synthetic DTE for the
- * IOMMU's own requester ID so hardware can DMA to backing store.
- * That object is not on pdom->dev_list: it is not an IOMMU-API
- * attach, and walkers such as rlookup and clone_aliases assume a
- * real struct device.
+ * Synthetic DTEs are not on pdom->dev_list: they are not IOMMU-API
+ * attaches, and walkers such as rlookup and clone_aliases assume a
+ * real struct device. amd_iommu_change_top() therefore misses them.
+ *
+ * 1. The vIOMMU private IPA domain programs a DTE for the IOMMU's
+ *    own requester ID (iommu->viommu_dev_data) so hardware can DMA
+ *    to backing store. Early maps fit under the initial page-table
+ *    top (PT_FEAT_DYNAMIC_TOP). Later DevID and DomID maps at high
+ *    IPA call increase_top(); the old root stays live as a child of
+ *    the new one, but the self DTE still holds the old MODE and
+ *    would not translate those IOVAs.
  *
- * amd_iommu_change_top() therefore misses it. Early maps fit under
- * the initial page-table top (PT_FEAT_DYNAMIC_TOP). Later DevID and
- * DomID maps at high IPA call increase_top(); the old root stays
- * live as a child of the new one, but the self DTE still holds the
- * old MODE and would not translate those IOVAs.
+ * 2. Each vIOMMU's translate-device-id DTE (aviommu->trans_dev_data)
+ *    points at the nest-parent v1 table for GPA->SPA. That parent is
+ *    also DYNAMIC_TOP. Guest maps of high GPA call increase_top() on
+ *    the nest parent; the translate DTE would otherwise keep the old
+ *    MODE and host_pt_root.
  *
  * Walk iommu_array (from amd_iommu_pdom_bind_iommu()) and rewrite
  * iommu->viommu_dev_data when this domain is that IOMMU's private
- * IPA table. set_dte_entry() skips clone_aliases() because the
- * synthetic DTE has no struct device.
+ * IPA table. Walk viommu_list and rewrite each trans_dev_data when
+ * this domain is a nest parent. set_dte_entry() skips clone_aliases()
+ * because those objects have no struct device. Use the vIOMMU's
+ * IOMMU, not rlookup.
  */
 static void update_viommu_self_dte(struct protection_domain *pdom,
 				   phys_addr_t top_paddr,
 				   unsigned int top_level)
 {
 	struct pdom_iommu_info *pdom_iommu_info;
+	struct amd_iommu_viommu *aviommu;
 	unsigned long i;
 
 	lockdep_assert_held(&pdom->lock);
@@ -2884,6 +2893,17 @@ static void update_viommu_self_dte(struct protection_domain *pdom,
 		set_dte_entry(iommu, iommu->viommu_dev_data, top_paddr,
 			      top_level);
 	}
+
+	list_for_each_entry(aviommu, &pdom->viommu_list, pdom_list) {
+		struct amd_iommu *iommu;
+
+		if (!aviommu->trans_dev_data)
+			continue;
+		iommu = container_of(aviommu->core.iommu_dev,
+				     struct amd_iommu, iommu);
+		set_dte_entry(iommu, aviommu->trans_dev_data, top_paddr,
+			      top_level);
+	}
 }
 #else
 static inline void update_viommu_self_dte(struct protection_domain *pdom,
@@ -3463,6 +3483,95 @@ static bool amd_iommu_enforce_cache_coherency(struct iommu_domain *domain)
 	return true;
 }
 
+#if IS_ENABLED(CONFIG_AMD_IOMMU_IOMMUFD)
+
+void amd_iommu_update_vfctrl_mmio_translate_devid(struct amd_iommu *iommu,
+						  u16 gid, u32 devid)
+{
+	writeq((devid & 0xFFFFULL) << 16,
+	       VIOMMU_VFCTRL_MMIO_BASE(iommu, gid) +
+	       VIOMMU_VFCTRL_GUEST_MISC_CONTROL_OFFSET);
+}
+
+int amd_iommu_set_translate_dte(struct iommufd_viommu *viommu)
+{
+	struct amd_iommu_viommu *aviommu =
+		container_of(viommu, struct amd_iommu_viommu, core);
+	struct amd_iommu *iommu =
+		container_of(viommu->iommu_dev, struct amd_iommu, iommu);
+	struct protection_domain *pdom = aviommu->parent;
+	u16 gid = aviommu->gid;
+	u16 trans_devid = aviommu->trans_devid;
+	struct iommu_dev_data *trans_dev_data;
+	unsigned long flags;
+
+	if (!aviommu->trans_dev_data) {
+		aviommu->trans_dev_data =
+			amd_iommu_alloc_dev_data(trans_devid);
+		if (!aviommu->trans_dev_data) {
+			pr_err("%s: Failed to allocate dev_data for translate-device-id %#x\n",
+			       __func__, trans_devid);
+			return -ENOMEM;
+		}
+	}
+
+	trans_dev_data = aviommu->trans_dev_data;
+	trans_dev_data->dev = NULL;
+	trans_dev_data->domain = pdom;
+	/*
+	 * Leave off pdom->dev_list: this is not an IOMMU-API attach.
+	 * Hold pdom->lock so change_top() cannot grow the nest-parent
+	 * top between the sample and the DTE commit. Publish devid
+	 * under that lock; it is the live TransDevID for
+	 * update_viommu_self_dte(). If this vIOMMU is not yet on
+	 * viommu_list, add it before dropping the lock so a concurrent
+	 * increase_top() cannot miss the new DTE.
+	 */
+
+	spin_lock_irqsave(&pdom->lock, flags);
+	trans_dev_data->devid = trans_devid;
+	set_dte_entry(iommu, trans_dev_data, 0, 0);
+	if (list_empty(&aviommu->pdom_list))
+		list_add(&aviommu->pdom_list, &pdom->viommu_list);
+	spin_unlock_irqrestore(&pdom->lock, flags);
+
+	pr_debug("%s: gid=%#x, iommu_devid=%#x, devid=%#x\n",
+		 __func__, gid, iommu->devid, trans_devid);
+	return 0;
+}
+
+static void amd_iommu_update_dte_slot(struct amd_iommu *iommu,
+				      struct iommu_dev_data *dev_data,
+				      u16 devid, struct dev_table_entry *new)
+{
+	update_dte256(iommu, dev_data, devid, new);
+	if (dev_data->dev)
+		clone_aliases(iommu, dev_data->dev);
+	iommu_flush_dte(iommu, devid);
+	iommu_completion_wait(iommu);
+}
+
+void amd_iommu_clear_translate_dte(struct amd_iommu *iommu,
+				   struct iommu_dev_data *dev_data,
+				   u16 trans_devid)
+{
+	struct dev_table_entry new = {};
+
+	if (!dev_data)
+		return;
+
+	pr_debug("%s: iommu_devid=%#x, trans_devid=%#x\n",
+		 __func__, iommu->devid, trans_devid);
+
+	/*
+	 * Clear @trans_devid without swapping the live
+	 * trans_dev_data->devid. change_top() indexes that field.
+	 */
+	amd_iommu_make_clear_dte(iommu, trans_devid, &new);
+	amd_iommu_update_dte_slot(iommu, dev_data, trans_devid, &new);
+}
+#endif /* CONFIG_AMD_IOMMU_IOMMUFD */
+
 const struct iommu_ops amd_iommu_ops = {
 	.capable = amd_iommu_capable,
 	.hw_info = amd_iommufd_hw_info,
-- 
2.34.1

[PATCH v6 22/24] iommu/amd: Add translate-device-id alloc/free with vIOMMU owner

[Suravee Suthikulpanit]

Extend the per-segment translate-device-id pool to record the owning
struct amd_iommu_viommu in the xarray.  Add amd_iommu_trans_devid_alloc()
and amd_iommu_trans_devid_free() to allocate ids from last_bdf down
to 0 so each id indexes the device and alias tables, then return them
on teardown.  Free loads the xarray slot and erases it only when the
owner matches @aviommu.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_viommu.h  |   4 ++
 drivers/iommu/amd/trans_devid.c | 105 ++++++++++++++++++++++++++++++--
 2 files changed, 105 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/amd/amd_viommu.h b/drivers/iommu/amd/amd_viommu.h
index 44a342608d43..3cf140d6439a 100644
--- a/drivers/iommu/amd/amd_viommu.h
+++ b/drivers/iommu/amd/amd_viommu.h
@@ -34,6 +34,10 @@ void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg);
 int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id);
 int amd_iommu_trans_devid_reserve_pci_aliases(struct amd_iommu *iommu,
 					      struct device *dev);
+int amd_iommu_trans_devid_alloc(struct amd_iommu_pci_seg *pci_seg,
+				struct amd_iommu_viommu *aviommu);
+void amd_iommu_trans_devid_free(struct amd_iommu_pci_seg *pci_seg, u16 id,
+				struct amd_iommu_viommu *aviommu);
 #else
 
 /*
diff --git a/drivers/iommu/amd/trans_devid.c b/drivers/iommu/amd/trans_devid.c
index f51a332fbc72..a63ee5626912 100644
--- a/drivers/iommu/amd/trans_devid.c
+++ b/drivers/iommu/amd/trans_devid.c
@@ -28,24 +28,55 @@ static inline bool trans_devid_xa_is_reserved(void *entry)
 	       xa_to_value(entry) == TRANS_DEVID_RESERVED;
 }
 
+static inline struct amd_iommu_viommu *trans_devid_xa_owner(void *entry)
+{
+	if (!entry || xa_is_value(entry))
+		return NULL;
+	return entry;
+}
+
 static inline void *trans_devid_xa_mk_reserved(void)
 {
 	return xa_mk_value(TRANS_DEVID_RESERVED);
 }
 
-static int trans_devid_xa_install_reserved_locked(struct amd_iommu_pci_seg *pci_seg,
-						  u16 id)
+static int trans_devid_find_free_locked(struct amd_iommu_pci_seg *pci_seg)
+{
+	int id;
+
+	for (id = pci_seg->last_bdf; id >= 0; id--) {
+		if (!xa_load(&pci_seg->trans_devid_xa, id))
+			return id;
+	}
+	return -ENOSPC;
+}
+
+static int trans_devid_xa_install_locked(struct amd_iommu_pci_seg *pci_seg,
+					 u16 id, void *entry)
 {
 	void *old;
 
-	old = xa_store(&pci_seg->trans_devid_xa, id,
-		       trans_devid_xa_mk_reserved(), GFP_KERNEL);
+	old = xa_store(&pci_seg->trans_devid_xa, id, entry, GFP_KERNEL);
 	if (xa_is_err(old))
 		return xa_err(old);
 	WARN_ON_ONCE(old);
 	return 0;
 }
 
+static int trans_devid_xa_install_allocated_locked(struct amd_iommu_pci_seg *pci_seg,
+						   u16 id,
+						   struct amd_iommu_viommu *aviommu)
+{
+	return trans_devid_xa_install_locked(pci_seg, id, aviommu);
+}
+
+static int trans_devid_xa_install_reserved_locked(struct amd_iommu_pci_seg *pci_seg,
+						u16 id)
+{
+	return trans_devid_xa_install_locked(pci_seg, id,
+					      trans_devid_xa_mk_reserved());
+}
+
 void amd_iommu_pci_seg_trans_devid_init(struct amd_iommu_pci_seg *pci_seg)
 {
 	mutex_init(&pci_seg->trans_devid_mutex);
@@ -133,3 +164,69 @@ int amd_iommu_trans_devid_reserve_pci_aliases(struct amd_iommu *iommu,
 	return pci_for_each_dma_alias(pdev, reserve_trans_devid_each_dma_alias,
 				      pci_seg);
 }
+
+/**
+ * amd_iommu_trans_devid_alloc - allocate a translate-device-id for @pci_seg
+ *
+ * The trans_devid is allocated from pci_seg->last_bdf down to 0 so the
+ * id is always a valid device-table and alias-table index.  PCI devices
+ * are generally enumerated from the low end of the bus range, so high
+ * ids are likely unused.
+ *
+ * Each vIOMMU receives its own translate-device-id from the per-segment pool.
+ * @aviommu is stored in the xarray as the slot owner.
+ *
+ * Return: allocated id on success, negative errno on failure.
+ */
+int amd_iommu_trans_devid_alloc(struct amd_iommu_pci_seg *pci_seg,
+				struct amd_iommu_viommu *aviommu)
+{
+	int id, ret;
+
+	mutex_lock(&pci_seg->trans_devid_mutex);
+	id = trans_devid_find_free_locked(pci_seg);
+	if (id < 0) {
+		ret = id;
+		goto unlock;
+	}
+
+	ret = trans_devid_xa_install_allocated_locked(pci_seg, id, aviommu);
+	if (ret)
+		goto unlock;
+
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+	pr_debug("%s: Allocated trans_devid %#x (seg %#x)\n", __func__, id,
+		 pci_seg->id);
+	return id;
+
+unlock:
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+	if (ret == -ENOSPC)
+		pr_err("%s: No free trans_devid found (seg %#x)\n", __func__,
+		       pci_seg->id);
+	return ret;
+}
+
+/**
+ * amd_iommu_trans_devid_free - return @id to the per-segment pool
+ *
+ * Erase @id only if the xarray owner is @aviommu. A reserved or foreign
+ * slot is left unchanged.
+ *
+ * Caller must hold @aviommu->trans_devid_lock if racing with relocation.
+ */
+void amd_iommu_trans_devid_free(struct amd_iommu_pci_seg *pci_seg, u16 id,
+				struct amd_iommu_viommu *aviommu)
+{
+	void *entry;
+
+	mutex_lock(&pci_seg->trans_devid_mutex);
+	entry = xa_load(&pci_seg->trans_devid_xa, id);
+	if (WARN_ON_ONCE(trans_devid_xa_owner(entry) != aviommu))
+		goto out;
+	xa_erase(&pci_seg->trans_devid_xa, id);
+	pr_debug("%s: Freed trans_devid %#x (seg %#x)\n", __func__, id,
+		 pci_seg->id);
+out:
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+}
-- 
2.34.1

[PATCH v6 23/24] iommu/amd: Assign per-vIOMMU translate device ID

[Suravee Suthikulpanit]

Allocate one translate-device-id per IOMMUFD vIOMMU instance from the
per-segment pool on init.  Reset MMIO first, then program translation
DTE and VFctrl TransDevID under trans_devid_lock so PCI-probe relocate
cannot interleave.  Clear both on init error and destroy, and drop the
id before amd_viommu_uninit_one().

Initialize pdom_list before set_translate_dte() so list_empty() is
valid. Drop the late list_add; set_translate_dte() already publishes
the vIOMMU on viommu_list under pdom->lock. list_del_init() on init
error if that add ran.

Free the synthetic translate-device-id DTE object on init error and
destroy; it is not stored in the per-segment xarray.

Add per-vIOMMU trans_devid_lock to serialize DTE and VFctrl TransDevID
updates on init and teardown.  VFctrl TransDevID is not written from
amd_viommu_uninit_one().

Destroy trans_devid_lock on init error and on vIOMMU destroy, after
the last unlock, the same way as vfctrl_lock.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h | 10 +++++
 drivers/iommu/amd/iommufd.c         | 68 +++++++++++++++++++++++++----
 2 files changed, 69 insertions(+), 9 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 5f84f25e3f37..c748bab27509 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -556,6 +556,16 @@ struct amd_iommu_viommu {
 	/* Synthetic DTE; not stored in pci_seg->dev_data_xa */
 	struct iommu_dev_data *trans_dev_data;
 
+	/*
+	 * Serializes this vIOMMU's translation DTE and VFctrl TransDevID
+	 * updates on init and teardown. VFctrl TransDevID is not written
+	 * from amd_viommu_uninit_one().
+	 *
+	 * Nested lock order: trans_devid_lock, then
+	 * pci_seg->trans_devid_mutex.
+	 */
+	struct mutex trans_devid_lock;
+
 	/*
 	 * Serializes VFCTRL Guest DID map CONTROL0/CONTROL1 doorbells
 	 * (DevID and DomID table updates) for this gid. Those registers
diff --git a/drivers/iommu/amd/iommufd.c b/drivers/iommu/amd/iommufd.c
index 5760b183ce62..1f7485131e63 100644
--- a/drivers/iommu/amd/iommufd.c
+++ b/drivers/iommu/amd/iommufd.c
@@ -53,6 +53,9 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	int ret;
 	unsigned long flags;
 	phys_addr_t page_base;
+	bool trans_devid_allocated = false;
+	bool trans_dte_set = false;
+	bool uninit_on_err = false;
 	struct iommu_viommu_amd data = {};
 	struct protection_domain *pdom = to_pdomain(parent);
 	struct amd_iommu_viommu *aviommu = container_of(viommu, struct amd_iommu_viommu, core);
@@ -70,7 +73,9 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 	if (ret)
 		return ret;
 
+	mutex_init(&aviommu->trans_devid_lock);
 	mutex_init(&aviommu->vfctrl_lock);
+	INIT_LIST_HEAD(&aviommu->pdom_list);
 	xa_init_flags(&aviommu->gdomid_array, XA_FLAGS_ALLOC1);
 	aviommu->parent = pdom;
 
@@ -93,35 +98,68 @@ int amd_iommufd_viommu_init(struct iommufd_viommu *viommu, struct iommu_domain *
 
 	data.out_vfmmio_mmap_offset = aviommu->vfmmio_mmap_offset;
 
-	/* Reset vIOMMU MMIOs to initialize the vIOMMU */
+	/* Reset vIOMMU MMIOs before the trans_devid is visible in the pool. */
 	ret = amd_iommu_reset_vmmio(iommu, aviommu->gid);
 	if (ret)
-		goto err_init;
+		goto err_trans_devid;
+
+	mutex_lock(&aviommu->trans_devid_lock);
+	ret = amd_iommu_trans_devid_alloc(iommu->pci_seg, aviommu);
+	if (ret < 0) {
+		mutex_unlock(&aviommu->trans_devid_lock);
+		goto err_trans_devid;
+	}
+	trans_devid_allocated = true;
+	aviommu->trans_devid = ret;
+
+	ret = amd_iommu_set_translate_dte(viommu);
+	if (ret)
+		goto err_init_unlock;
+	trans_dte_set = true;
+	amd_iommu_update_vfctrl_mmio_translate_devid(iommu, aviommu->gid,
+						     aviommu->trans_devid);
+	mutex_unlock(&aviommu->trans_devid_lock);
 
 	ret = amd_viommu_init_one(iommu, aviommu);
 	if (ret)
 		goto err_init;
+	uninit_on_err = true;
 
 	ret = iommu_copy_struct_to_user(user_data, &data,
 					IOMMU_VIOMMU_TYPE_AMD,
 					out_vfmmio_mmap_offset);
 	if (ret)
-		goto err_init_one;
+		goto err_init;
 
 	viommu->ops = &amd_viommu_ops;
 
-	spin_lock_irqsave(&pdom->lock, flags);
-	list_add(&aviommu->pdom_list, &pdom->viommu_list);
-	spin_unlock_irqrestore(&pdom->lock, flags);
-
 	return 0;
-err_init_one:
-	amd_viommu_uninit_one(iommu, aviommu);
 err_init:
+	mutex_lock(&aviommu->trans_devid_lock);
+err_init_unlock:
+	if (trans_dte_set) {
+		spin_lock_irqsave(&pdom->lock, flags);
+		if (!list_empty(&aviommu->pdom_list))
+			list_del_init(&aviommu->pdom_list);
+		spin_unlock_irqrestore(&pdom->lock, flags);
+		amd_iommu_update_vfctrl_mmio_translate_devid(iommu, aviommu->gid, 0);
+		amd_iommu_clear_translate_dte(iommu, aviommu->trans_dev_data,
+					      aviommu->trans_devid);
+		amd_iommu_free_dev_data(iommu, aviommu->trans_dev_data);
+		aviommu->trans_dev_data = NULL;
+	}
+	if (trans_devid_allocated)
+		amd_iommu_trans_devid_free(iommu->pci_seg, aviommu->trans_devid,
+					   aviommu);
+	mutex_unlock(&aviommu->trans_devid_lock);
+	if (uninit_on_err)
+		amd_viommu_uninit_one(iommu, aviommu);
+err_trans_devid:
 	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
 err_mmap:
 	amd_iommu_gid_free(iommu, aviommu->gid);
 err_gid:
+	mutex_destroy(&aviommu->trans_devid_lock);
 	mutex_destroy(&aviommu->vfctrl_lock);
 	return ret;
 }
@@ -140,8 +178,20 @@ static void amd_iommufd_viommu_destroy(struct iommufd_viommu *viommu)
 	spin_unlock_irqrestore(&pdom->lock, flags);
 	xa_destroy(&aviommu->gdomid_array);
 	iommufd_viommu_destroy_mmap(&aviommu->core, aviommu->vfmmio_mmap_offset);
+
+	mutex_lock(&aviommu->trans_devid_lock);
+	amd_iommu_update_vfctrl_mmio_translate_devid(iommu, aviommu->gid, 0);
+	amd_iommu_clear_translate_dte(iommu, aviommu->trans_dev_data,
+				      aviommu->trans_devid);
+	amd_iommu_free_dev_data(iommu, aviommu->trans_dev_data);
+	aviommu->trans_dev_data = NULL;
+	amd_iommu_trans_devid_free(iommu->pci_seg, aviommu->trans_devid,
+				   aviommu);
+	mutex_unlock(&aviommu->trans_devid_lock);
+
 	amd_viommu_uninit_one(iommu, aviommu);
 	amd_iommu_gid_free(iommu, aviommu->gid);
+	mutex_destroy(&aviommu->trans_devid_lock);
 	mutex_destroy(&aviommu->vfctrl_lock);
 }
 
-- 
2.34.1

[PATCH v6 24/24] iommu/amd: Relocate vIOMMU translate-device-id on PCI reserve

[Suravee Suthikulpanit]

When PCI probe reserves a BDF already allocated to a vIOMMU, relocate
that vIOMMU to a newly allocated translate-device-id before marking the
BDF reserved for the device. Reserve holds the segment mutex and
trylocks trans_devid_lock so the vIOMMU pointer stays valid without
reversing the nested lock order. Reprogram translation DTE and VFctrl
under trans_devid_lock; roll back the pool on failure.

Mark from_id reserved with a raw xa_store so ALLOCATED->RESERVED does
not WARN on the non-NULL owner. On MMIO failure, restore from_id the
same way (RESERVED->owner) instead of install_allocated.

Reuse the vIOMMU's trans_dev_data object. After programming the new
slot, clear the old DTE without freeing the object.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  10 +-
 drivers/iommu/amd/trans_devid.c     | 146 +++++++++++++++++++++++++++-
 2 files changed, 152 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index c748bab27509..347c6954439d 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -558,11 +558,15 @@ struct amd_iommu_viommu {
 
 	/*
 	 * Serializes this vIOMMU's translation DTE and VFctrl TransDevID
-	 * updates on init and teardown. VFctrl TransDevID is not written
-	 * from amd_viommu_uninit_one().
+	 * updates on init and teardown against PCI-probe relocation, which
+	 * may reprogram the same registers to a newly allocated id.
+	 * VFctrl TransDevID is not written from amd_viommu_uninit_one().
 	 *
 	 * Nested lock order: trans_devid_lock, then
-	 * pci_seg->trans_devid_mutex.
+	 * pci_seg->trans_devid_mutex (same as destroy). Relocate drops
+	 * the segment mutex around the MMIO updates. Reserve holds the
+	 * segment mutex and mutex_trylock()s trans_devid_lock so the
+	 * aviommu pointer stays valid without reversing lock order.
 	 */
 	struct mutex trans_devid_lock;
 
diff --git a/drivers/iommu/amd/trans_devid.c b/drivers/iommu/amd/trans_devid.c
index a63ee5626912..e844afe6da77 100644
--- a/drivers/iommu/amd/trans_devid.c
+++ b/drivers/iommu/amd/trans_devid.c
@@ -12,6 +12,7 @@
 
 #include <linux/kernel.h>
 #include <linux/pci.h>
+#include <linux/sched.h>
 #include <linux/xarray.h>
 
 #include "amd_iommu.h"
@@ -89,6 +90,118 @@ void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg)
 	mutex_destroy(&pci_seg->trans_devid_mutex);
 }
 
+/**
+ * trans_devid_do_relocate - move vIOMMU translation DTE from @old_id to @new_id
+ *
+ * Caller holds @aviommu->trans_devid_lock.  Pool xarray already records @new_id
+ * as allocated to @aviommu and @old_id as reserved.
+ *
+ * amd_iommu_set_translate_dte() cannot fail here: trans_dev_data is
+ * allocated in the same trans_devid_lock section as the xarray owner
+ * and is only cleared with trans_devid_free(). A later failure after
+ * the new DTE is programmed must clear that DTE before the pool drops
+ * @new_id.
+ */
+static int trans_devid_do_relocate(struct amd_iommu_viommu *aviommu,
+				   u16 old_id, u16 new_id)
+{
+	struct iommufd_viommu *viommu = &aviommu->core;
+	struct amd_iommu *iommu =
+		container_of(viommu->iommu_dev, struct amd_iommu, iommu);
+	int ret;
+
+	aviommu->trans_devid = new_id;
+
+	ret = amd_iommu_set_translate_dte(viommu);
+	if (ret)
+		goto err_restore_id;
+
+	amd_iommu_update_vfctrl_mmio_translate_devid(iommu, aviommu->gid, new_id);
+
+	amd_iommu_clear_translate_dte(iommu, aviommu->trans_dev_data, old_id);
+
+	return 0;
+
+err_restore_id:
+	aviommu->trans_devid = old_id;
+	return ret;
+}
+
+/**
+ * trans_devid_relocate - move an allocated id to a new slot and reserve @from_id
+ *
+ * Called when PCI probe needs a BDF that a vIOMMU already owns.  Updates the
+ * per-segment pool, then reprograms DTE and VFctrl on the owning vIOMMU.
+ *
+ * Locking: caller holds @aviommu->trans_devid_lock and
+ * pci_seg->trans_devid_mutex.  Hardware steps run with the viommu lock
+ * held and the segment mutex dropped.
+ */
+static int trans_devid_relocate(struct amd_iommu_pci_seg *pci_seg, u16 from_id,
+				struct amd_iommu_viommu *aviommu)
+{
+	void *old;
+	int new_id, ret;
+
+	if (trans_devid_xa_owner(xa_load(&pci_seg->trans_devid_xa, from_id)) !=
+	    aviommu) {
+		ret = -ENOENT;
+		goto unlock_seg;
+	}
+
+	if (aviommu->trans_devid != from_id) {
+		ret = -EINVAL;
+		goto unlock_seg;
+	}
+
+	new_id = trans_devid_find_free_locked(pci_seg);
+	if (new_id < 0) {
+		ret = new_id;
+		goto unlock_seg;
+	}
+
+	ret = trans_devid_xa_install_allocated_locked(pci_seg, new_id, aviommu);
+	if (ret)
+		goto unlock_seg;
+
+	/*
+	 * Replace ALLOCATED with RESERVED in place. install_reserved
+	 * WARNs on a non-NULL old entry; do not erase first or a failed
+	 * store would leave from_id FREE while hardware still uses it.
+	 */
+	old = xa_store(&pci_seg->trans_devid_xa, from_id,
+		       trans_devid_xa_mk_reserved(), GFP_KERNEL);
+	if (xa_is_err(old)) {
+		xa_erase(&pci_seg->trans_devid_xa, new_id);
+		ret = xa_err(old);
+		goto unlock_seg;
+	}
+	WARN_ON_ONCE(trans_devid_xa_owner(old) != aviommu);
+
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+
+	ret = trans_devid_do_relocate(aviommu, from_id, new_id);
+	if (ret) {
+		mutex_lock(&pci_seg->trans_devid_mutex);
+		xa_erase(&pci_seg->trans_devid_xa, new_id);
+		old = xa_store(&pci_seg->trans_devid_xa, from_id, aviommu,
+			       GFP_KERNEL);
+		if (xa_is_err(old))
+			ret = xa_err(old);
+		else
+			WARN_ON_ONCE(!trans_devid_xa_is_reserved(old));
+		mutex_unlock(&pci_seg->trans_devid_mutex);
+	}
+
+	mutex_unlock(&aviommu->trans_devid_lock);
+	return ret;
+
+unlock_seg:
+	mutex_unlock(&pci_seg->trans_devid_mutex);
+	mutex_unlock(&aviommu->trans_devid_lock);
+	return ret;
+}
+
 /*
  * amd_iommu_trans_devid_reserve - occupy @id so it is never returned by alloc
  *
@@ -99,18 +212,49 @@ void amd_iommu_pci_seg_trans_devid_fini(struct amd_iommu_pci_seg *pci_seg)
  * It is not released from amd_iommu_release_device(); the slot stays
  * reserved for the pci_seg lifetime so replug cannot race later alloc.
  *
- * Return: 0 on success.  A second reserve of an already-reserved @id succeeds.
+ * When @id is allocated to a vIOMMU (e.g. after PCI hot-plug), the driver relocates
+ * that vIOMMU to a newly allocated translate-device-id and reserves @id for the PCI
+ * function.
+ *
+ * Return: 0 on success, %-ENOSPC if relocation cannot find a free id, or another
+ * errno from relocation.  A second reserve of an already-reserved @id succeeds.
  */
 int amd_iommu_trans_devid_reserve(struct amd_iommu_pci_seg *pci_seg, u16 id)
 {
 	void *entry;
+	struct amd_iommu_viommu *aviommu;
 	int ret = 0;
 
+retry:
 	mutex_lock(&pci_seg->trans_devid_mutex);
 	entry = xa_load(&pci_seg->trans_devid_xa, id);
 	if (trans_devid_xa_is_reserved(entry))
 		goto unlock;
 
+	aviommu = trans_devid_xa_owner(entry);
+	if (aviommu) {
+		/*
+		 * Pin aviommu with trylock while the mutex still covers
+		 * the xarray pointer. A blocking lock here would invert
+		 * the nested order (ABBA with destroy). Dropping the
+		 * mutex first would UAF if destroy frees aviommu.
+		 *
+		 * No retry bound. cond_resched() avoids a soft lockup;
+		 * progress is this owner dropping trans_devid_lock or
+		 * freeing the slot.
+		 */
+		if (!mutex_trylock(&aviommu->trans_devid_lock)) {
+			mutex_unlock(&pci_seg->trans_devid_mutex);
+			cond_resched();
+			goto retry;
+		}
+		ret = trans_devid_relocate(pci_seg, id, aviommu);
+		if (!ret)
+			pr_debug("%s: Reserved trans_devid %#x after relocation (seg %#x)\n",
+				 __func__, id, pci_seg->id);
+		return ret;
+	}
+
 	ret = trans_devid_xa_install_reserved_locked(pci_seg, id);
 unlock:
 	mutex_unlock(&pci_seg->trans_devid_mutex);
-- 
2.34.1