[PATCH v5 00/24] iommu/amd: Introduce AMD Hardware-accelerated Virtualized IOMMU (vIOMMU) Support

[Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>]

OVERVIEW
========

AMD IOMMU introduces the vIOMMU feature, which provides partial hardware
acceleration when implementing Guest IOMMUs. This feature provides
acceleration for guest Command Buffer, Event Log, and PPR Log. This
eliminates the CPU overhead needed for the supporting HV intercepts and
reduces the latency of these operations.

When a guest attempts to access guest IOMMU MMIO registers with offsets
between 8KB and 12KB (i.e. the 3rd 4K region) such as the Command Buffer,
Event Log and PPR Log head and tail pointer registers, this is serviced
directly by the IOMMU. When the IOMMU accesses a Command Buffer, PPR Log
or a COMPLETION_WAIT store location in memory, it directly accesses guest
physical memory. The HV/VMM continues to trap and emulate the IOMMU
configuration MMIO registers between 0KB and 4KB (i.e. the 1st 4K
region), which are primarily used during initialization.

Additionally, the HV must initialize the vIOMMU feature, map MMIO resources
between the VMs and the IOMMU, manage additional supporting data structures
in memory (e.g. GPA->SPA translation DTE, Device ID and Domain ID mapping
tables), and allocate/map vIOMMU Private Address region used as backing
storage memory for the IOMMU. Support for new IOMMU command and events
specifically for vIOMMU are also added.

Guest IOMMUs are IOMMUs exposed to VMs with additional support from VMM
(QEMU) to generate guest ACPI IVRS table and define guest PCI topology for
IOMMU and pass-through VFIO devices, which are not covered by this series.

For more detail, please see the vIOMMU section of the AMD IOMMU
Specification[1].

ABOUT SERIES V5
===============

This is version 5 of the AMD HW-vIOMMU series [2]. It is implemented on top
of the IOMMUFD vIOMMU, vDevice, and nested-domain framework in Linux
v7.3.0-rc3 (base 704340f1cd0d).

Note: This series is a partial implementation of AMD hardware-accelerated
vIOMMU. Subsequent series will add hardware-queue and extended interrupt
remapping support, which are needed to fully support AMD IOMMU
virtualization in the guest VM. The fully supported version is available
in a GitHub repository [3].

The series is organized into the following subsets:

  Patch 1-3   : Events and vIOMMU feature detect/init
  Patch 4-8   : Introduce IOMMUFD vIOMMU support and VF MMIO setup
  Patch 9-14  : Introduce and map vIOMMU Private Address (IPA) region
  Patch 15-18 : IOMMUFD vDevice, DevID/DomID maps, and nested attach
  Patch 19-24 : Translate-device-ID pool, per-vIOMMU translation DTE, and
                PCI-reserve relocation

CHANGES FROM V4
===============

V4: (https://lore.kernel.org/linux-iommu/20260727132913.22475-1-suravee.suthikulpanit@amd.com/)

Rebase / series scope:
  * Rebased onto 704340f1cd0d (Linux v7.3.0-rc3 plus x86 urgent
    for v7.3-rc4).
  * Drop "Make amd_iommu_completion_wait() non-static"; the helper is
    already available, and VFCTRL CONTROL0/CONTROL1 doorbells are not
    command-buffer operations.
  * Drop "Export amd_iommu_alloc_dev_data() helper".
  * Add EVENT_TYPE_GUEST_EVENT_FAULT as patch 2.
  * Replace the vDevice mapping helpers (v4 16-18) with vDevice+DevID
    mapping, nested DTE+DomID attach, and DevID/DomID table prefill.

Events and init (patches 1-3):
  * Patch 2: INSERT_GUEST_EVENT with reserved bits logs GUEST_EVENT_FAULT
    then the original guest event; consume both event-log slots.
    Wait until occupancy is at least two entries; re-read the event-log
    tail each poll iteration so a return of 2 cannot walk empty type-0
    slots. Return 2 only when the pair is in [head, live_tail); return
    1 if the pair never appears. Rate-limit guest-triggered logs.
  * Patch 3: Gate amd_viommu_init() on this IOMMU's EFR[VIOMMUSup]. Set
    AMD_IOMMU_FLAG_VIOMMU_EN on success. When CONFIG_AMD_IOMMU_IOMMUFD
    is off, the stub returns 0 so a missing build is not logged as an
    init failure.

IOMMUFD vIOMMU and VF MMIO (patches 4-8):
  * Patch 4: Report a non-zero viommu size only for AMD when
    amd_iommu_viommu_enabled() is true.
  * Patch 5: Initialize gid_ida with the IOMMU object; reject vIOMMU
    init without AMD_IOMMU_FLAG_VIOMMU_EN.
  * Patch 6: Reject a disabled or zero VSC VF/VFCTRL BAR; set
    VIOMMU_EN after the BARs are mapped; uninit before
    free_iommu_buffers(); release the reserved MMIO region if ioremap()
    fails.
  * Patch 7: Return -EINVAL if the VF-MMIO page_base is zero.
  * Patch 8: Always program RESET_MMIO_ALL_FLAG and RESET_MMIO_VCMD_FLAG;
    propagate completion_wait errors.

IPA / DTE infrastructure (patches 9-14):
  * Patch 9: Tear down the 8MB IPA mapping from amd_viommu_uninit();
    pass GFP for backing pages so failed init can unwind. The private
    IPA domain skips iommu_domain_init(), so set IOMMU_DOMAIN_UNMANAGED
    so set_dte_entry() programs v1 when increase_top() rewrites the
    self DTE.
  * Patch 12: Store per-segment iommu_dev_data in an xarray instead of
    exporting amd_iommu_alloc_dev_data().
  * Patch 13: Program the IOMMU's own DTE with the private IPA domain
    (off pdom->dev_list); rewrite it from amd_iommu_change_top() when
    the table grows.
  * Patch 14: Charge DevID/DomID backing with GFP_KERNEL_ACCOUNT.

IOMMUFD vDevice and DevID/DomID maps (patches 15-18):
  * Patch 16: vdevice_init programs DevID via VFCTRL. Idle entries keep
    V=1 with host device ID 0. Serialize CONTROL0 with vfctrl_lock.
    Destroy restores the idle mapping. Poll CONTROL0 WRITE until it
    clears; iommu_completion_wait() is not a barrier.
  * Patch 17: Program nested DTE and DomID map on attach. Last
    nested_domain_free() for a gdom_id restores the idle DomID map
    (nest parent, V=1) before freeing hdom_id. Poll CONTROL1 WRITE
    the same way as CONTROL0.
  * Patch 18: Prefill DevID/DomID tables on init only; yield
    vfctrl_lock every 256 doorbells. Destroy drains WRITE then unmaps
    and does not rewrite 0..0xFFFF. Skip VFCTRL when CONTROL_CMDBUF_EN
    is already clear.

Translate device ID (patches 19-24):
  * Patch 19: Initialize the pool when pci_seg is allocated; reserved
    RIDs stay reserved for the segment lifetime (including after
    release_device).
  * Patch 21: Own trans_dev_data on the vIOMMU. Sample the nest-parent
    top and commit the DTE under pdom->lock; publish on viommu_list
    before dropping that lock. Keep trans_dev_data->devid as the live
    TransDevID; clear programs an explicit slot id.
  * Patch 23: trans_devid_lock; do not write VFctrl TransDevID from
    amd_viommu_uninit_one(); own the synthetic DTE on the vIOMMU, not
    in the per-segment xarray. INIT_LIST_HEAD(pdom_list) before
    set_translate_dte(); drop the late list_add; list_del_init() on
    init error.
  * Patch 24: Mark from_id reserved with a raw xa_store so
    ALLOCATED->RESERVED does not WARN. On MMIO failure, restore
    from_id the same way. Reuse the vIOMMU trans_dev_data object.

[1] IOMMU Specification: https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
[2] Series v5 tree: https://github.com/AMDESE/linux-iommu/tree/linux-7.3.0-rc3-amd-viommu_upstream_v5
[3] Fully supported tree (work-in-progress): https://github.com/AMDESE/linux-iommu/tree/wip/v7.3.0-rc3-viommu_20260915

Thank you,
Suravee

Suravee Suthikulpanit (24):
  iommu/amd: Introduce vIOMMU-specific events and event
  iommu/amd: Introduce EVENT_TYPE_GUEST_EVENT_FAULT
  iommu/amd: Detect and initialize AMD vIOMMU feature
  iommu/amd: Introduce IOMMUFD vIOMMU support for AMD
  iommu/amd: Allocate Guest IDs for IOMMUFD vIOMMU instances
  iommu/amd: Map vIOMMU VF and VF Control MMIO BARs
  iommu/amd: Add support for AMD vIOMMU VF MMIO region
  iommu/amd: Introduce Reset vMMIO Command
  iommu/amd: Introduce and map vIOMMU private IPA region
  iommu/amd: Pass iommu to device_flush_dte()
  iommu/amd: Pass iommu and devid to amd_iommu_make_clear_dte()
  iommu/amd: Store per-segment iommu_dev_data in an xarray
  iommu/amd: Program IOMMU DTE with the private IPA domain
  iommu/amd: Add per-VM private IPA alloc/map helpers
  iommu/amd: Add helper functions to manage DevID / DomID mapping tables
  iommu/amd: Add IOMMUFD vDevice and DevID mapping
  iommu/amd: Program nested DTE and DomID map on attach
  iommu/amd: Init and clear vIOMMU DevID and DomID maps
  iommu/amd: Add per-segment translate device ID pool
  iommu/amd: Reserve translate-device-id for PCI requestor aliases
  iommu/amd: Add translation DTE and VFctrl TransDevID helpers
  iommu/amd: Add translate-device-id alloc/free with vIOMMU owner
  iommu/amd: Assign per-vIOMMU translate device ID
  iommu/amd: Relocate vIOMMU translate-device-id on PCI reserve

 drivers/iommu/amd/Makefile          |   2 +-
 drivers/iommu/amd/amd_iommu.h       |  34 +-
 drivers/iommu/amd/amd_iommu_types.h | 125 ++++-
 drivers/iommu/amd/amd_viommu.h      |  91 ++++
 drivers/iommu/amd/init.c            |  62 ++-
 drivers/iommu/amd/iommu.c           | 486 +++++++++++++++++---
 drivers/iommu/amd/iommufd.c         | 183 +++++++-
 drivers/iommu/amd/nested.c          |  76 ++-
 drivers/iommu/amd/trans_devid.c     | 373 +++++++++++++++
 drivers/iommu/amd/viommu.c          | 688 ++++++++++++++++++++++++++++
 include/uapi/linux/iommufd.h        |  10 +
 11 files changed, 2046 insertions(+), 84 deletions(-)
 create mode 100644 drivers/iommu/amd/amd_viommu.h
 create mode 100644 drivers/iommu/amd/trans_devid.c
 create mode 100644 drivers/iommu/amd/viommu.c

-- 
2.34.1