[PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jose Ignacio Tornos Martinez]

This series enables Qualcomm ath11k/ath12k WiFi devices to work in VM
passthrough scenarios, addressing a long-standing issue where these
devices fail to initialize when passed through to VMs via VFIO.

Qualcomm ath11k/ath12k WiFi devices have been broken in VM passthrough
since they were introduced. The devices use an embedded interrupt
controller that requires physical host MSI addresses programmed to
device registers, but in VMs the driver only sees virtualized guest
addresses. This causes firmware initialization to fail with errors like
"BHI offset: 0xffffffff is out of range".

Multiple attempts have been made to solve this over the past 2 years:

March 2024: Workaround using module parameters [1]
  - Required manually copying MSI values from host to VM
  - Rejected by Johannes Berg as too manual, suggested VMM solution

August 2024: QEMU + kernel solution by Alex Williamson [2]
  - Kernel disables MSI virtualization, QEMU intercepts config writes
  - Uses brute force pattern matching to infer MSI data writes
  - Stuck as RFC - no progress in 8+ months

No action from Qualcomm: Despite multiple reports, no vendor solution

The solution in this series provides a clean, kernel-only solution that
addresses the previous concerns.

Architecture:
1. qcom-vfio-pci variant driver caches physical host MSI values and
   exposes them via extended config space with magic signature "QMSI"
   In this way, device-specific hardware quirks belong only in kernel
   drivers, not userspace, independent of the tools used.
   This follows the established VFIO variant driver pattern used by
   other drivers like mlx5-vfio-pci (netdev tree) and nvgrace-gpu (platform
   tree).
2. VM drivers (ath11k/ath12k) automatically discover and use cached values
3. PCIe link recovery handles VM timing variations

Thoroughly tested: 3 VMs across 2 WiFi generations, all devices successfully
initialized and are ready for use.
- VM1: ath11k (WCN6855)
- VM2: ath11k (WCN6855)
- VM3: ath12k (WCN7850)

[1] https://lore.kernel.org/all/20240322104912.94811-1-jtornosm@redhat.com/
[2] https://lore.kernel.org/kvm/20240812170014.1583783-1-alex.williamson@redhat.com/
[3] Original problem report:
    https://lore.kernel.org/all/fc6bd06f-d52b-4dee-ab1b-4bb845cc0b95@quicinc.com/

Jose Ignacio Tornos Martinez (7):
  PCI: Add pci_find_free_ext_cap_offset() helper
  vfio: Add qcom_vfio.h header for MSI cache protocol
  vfio/pci: Add qcom-vfio-pci variant driver
  ath11k: Add PCIe link recovery retry for VMs
  ath11k: Use VFIO MSI cache when available
  ath12k: Add PCIe link recovery retry for VMs
  ath12k: Use VFIO MSI cache when available
--
2.53.0

[PATCH 1/7] PCI: Add pci_find_free_ext_cap_offset() helper

[Jose Ignacio Tornos Martinez]

Add helper function to find free space in PCI extended configuration
space. This is needed by VFIO variant drivers that need to use unused
extended config space for device-specific purposes, such as passing
metadata to VMs.

The function scans the extended capability chain and returns an offset
to a gap of at least the requested size. This allows drivers to safely
use extended config space without conflicting with existing capabilities.

Use case: The qcom-vfio-pci variant driver uses this to find space for
caching host MSI addresses that need to be passed to Qualcomm device
firmware in VMs.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/pci/pci.c   | 54 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  3 +++
 2 files changed, 57 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8f7cfcc00090..7fa261140457 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -552,6 +552,60 @@ u16 pci_find_ext_capability(struct pci_dev *dev, int cap)
 }
 EXPORT_SYMBOL_GPL(pci_find_ext_capability);
 
+/**
+ * pci_find_free_ext_cap_offset - Find free space in extended config
+ * @dev: PCI device to query
+ * @size: Minimum size needed (in bytes)
+ *
+ * Scans the extended capability chain to find a gap of at least @size bytes
+ * in the extended configuration space (0x100-0xFFF). This can be used by
+ * drivers that need to use unused extended config space for device-specific
+ * purposes (e.g., VFIO quirks for VM passthrough).
+ *
+ * Returns the offset to free space, or 0 if no suitable gap found.
+ */
+int pci_find_free_ext_cap_offset(struct pci_dev *dev, size_t size)
+{
+	int pos = PCI_CFG_SPACE_SIZE;
+	int last_cap_start = 0;
+	int free_offset;
+	u32 header;
+	int next;
+
+	if (!pci_is_pcie(dev))
+		return 0;
+
+	while (pos >= PCI_CFG_SPACE_SIZE) {
+		if (pci_read_config_dword(dev, pos, &header))
+			break;
+
+		if (header == 0 || header == 0xffffffff)
+			break;
+
+		last_cap_start = pos;
+		next = PCI_EXT_CAP_NEXT(header);
+		if (!next)
+			break;
+		pos = next;
+	}
+
+	/*
+	 * Use space after the last capability. We align to the next capability
+	 * boundary (typically 4-byte aligned). Start at a safe offset past the
+	 * last capability - capabilities vary in size, so use a conservative
+	 * offset. Most extended capabilities are <= 64 bytes.
+	 */
+	if (last_cap_start) {
+		free_offset = last_cap_start + 64;
+		free_offset = ALIGN(free_offset, 4);
+		if (PCI_CFG_SPACE_EXP_SIZE - free_offset >= size)
+			return free_offset;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(pci_find_free_ext_cap_offset);
+
 /**
  * pci_get_dsn - Read and return the 8-byte Device Serial Number
  * @dev: PCI device to query
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2c4454583c11..5d1ccbb2ed29 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1287,6 +1287,7 @@ u8 pci_find_ht_capability(struct pci_dev *dev, int ht_cap);
 u8 pci_find_next_ht_capability(struct pci_dev *dev, u8 pos, int ht_cap);
 u16 pci_find_ext_capability(struct pci_dev *dev, int cap);
 u16 pci_find_next_ext_capability(struct pci_dev *dev, u16 pos, int cap);
+int pci_find_free_ext_cap_offset(struct pci_dev *dev, size_t size);
 struct pci_bus *pci_find_next_bus(const struct pci_bus *from);
 u16 pci_find_vsec_capability(struct pci_dev *dev, u16 vendor, int cap);
 u16 pci_find_dvsec_capability(struct pci_dev *dev, u16 vendor, u16 dvsec);
@@ -2160,6 +2161,8 @@ static inline u8 pci_find_next_capability(struct pci_dev *dev, u8 post, int cap)
 { return 0; }
 static inline u16 pci_find_ext_capability(struct pci_dev *dev, int cap)
 { return 0; }
+static inline int pci_find_free_ext_cap_offset(struct pci_dev *dev, size_t size)
+{ return 0; }
 
 static inline u64 pci_get_dsn(struct pci_dev *dev)
 { return 0; }
-- 
2.53.0

[PATCH 2/7] vfio: Add qcom_vfio.h header for MSI cache protocol

[Jose Ignacio Tornos Martinez]

Add shared header defining the communication protocol between
qcom-vfio-pci variant driver and Qualcomm device drivers (ath11k,
ath12k) for passing physical host MSI addresses to VMs.

Qualcomm device firmware requires physical host MSI addresses rather
than guest IOVA addresses. The qcom-vfio-pci driver caches host MSI
values in extended PCI config space with a magic signature "QMSI" for
VM discovery.

Protocol:
 1. Host: qcom-vfio-pci finds free space in extended config (0x100-0xFFF)
 2. Host: Writes magic signature "QMSI" (0x49534D51) followed by MSI data
 3. VM: Device driver searches for "QMSI" signature to locate cached MSI
 4. VM: Reads MSI values and programs device firmware

This avoids offset mismatch when QEMU filters certain capabilities,
causing host and VM to see different extended capability chains.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 include/linux/qcom_vfio.h | 88 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)
 create mode 100644 include/linux/qcom_vfio.h

diff --git a/include/linux/qcom_vfio.h b/include/linux/qcom_vfio.h
new file mode 100644
index 000000000000..c59c81936e84
--- /dev/null
+++ b/include/linux/qcom_vfio.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* SPDX-FileCopyrightText: Copyright Red Hat */
+
+#ifndef _QCOM_VFIO_H_
+#define _QCOM_VFIO_H_
+
+#include <linux/pci.h>
+
+/*
+ * MSI Address Passthrough Quirk
+ *
+ * Qualcomm device firmware requires physical host MSI addresses rather than
+ * guest IOVA addresses. The qcom-vfio-pci variant driver caches host MSI
+ * values in extended PCI config space with a magic signature for VM
+ * discovery.
+ *
+ * Protocol:
+ *  1. Host: qcom-vfio-pci finds free space in extended config (0x100-0xFFF)
+ *  2. Host: Writes magic signature "QMSI" (0x49534D51) followed by MSI data
+ *  3. VM: Device driver searches for "QMSI" signature to locate cached MSI
+ *  4. VM: Reads MSI values and programs device firmware
+ *
+ * This avoids offset mismatch when QEMU filters certain capabilities, causing
+ * host and VM to see different extended capability chains.
+ *
+ * Layout in extended config space:
+ *   +0x00: Magic signature "QMSI" (0x49534D51)
+ *   +0x04: MSI address low 32 bits
+ *   +0x08: MSI address high 32 bits
+ *   +0x0C: MSI data (16 bits)
+ *
+ * Applicable to:
+ *  - ath11k WiFi (PCI ID 0x17cb:0x1103)
+ *  - ath12k WiFi (PCI ID 0x17cb:0x1107)
+ */
+
+/* Magic signature "QMSI" to identify cached MSI data */
+#define QCOM_VFIO_MSI_MAGIC		0x49534D51
+
+/* Space needed for magic + MSI passthrough data */
+#define QCOM_VFIO_MSI_SPACE_SIZE	16
+
+/* Relative offsets from base discovered by qcom_vfio_find_msi_offset() */
+#define QCOM_VFIO_MSI_MAGIC_OFFSET	0x00	/* Magic signature "QMSI" */
+#define QCOM_VFIO_MSI_ADDR_LO_OFFSET	0x04	/* Host MSI address low 32 bits */
+#define QCOM_VFIO_MSI_ADDR_HI_OFFSET	0x08	/* Host MSI address high 32 bits */
+#define QCOM_VFIO_MSI_DATA_OFFSET	0x0C	/* Host MSI data (16 bits) */
+
+/**
+ * qcom_vfio_find_msi_offset - Find free space for host to cache MSI
+ * @pdev: PCI device
+ *
+ * Host-side function: Finds free space in extended config to write MSI cache.
+ * Uses PCI core utility to scan capability chain.
+ *
+ * Returns: Offset in extended config space (>= 0x100), or 0 if no space found
+ */
+static inline int qcom_vfio_find_msi_offset(struct pci_dev *pdev)
+{
+	return pci_find_free_ext_cap_offset(pdev, QCOM_VFIO_MSI_SPACE_SIZE);
+}
+
+/**
+ * qcom_vfio_find_msi_cache - Locate cached MSI data by magic signature
+ * @pdev: PCI device
+ *
+ * VM-side function: Searches extended config space for "QMSI" magic signature
+ * to locate host-cached MSI values. This avoids offset mismatch when QEMU
+ * filters certain capabilities.
+ *
+ * Returns: Offset of MSI data (after magic), or 0 if not found
+ */
+static inline int qcom_vfio_find_msi_cache(struct pci_dev *pdev)
+{
+	u32 magic;
+	int offset;
+
+	for (offset = 0x100; offset <= 0xf00; offset += 4) {
+		if (pci_read_config_dword(pdev, offset, &magic))
+			continue;
+		if (magic == QCOM_VFIO_MSI_MAGIC)
+			return offset;
+	}
+
+	return 0;
+}
+
+#endif /* _QCOM_VFIO_H_ */
-- 
2.53.0

[PATCH 3/7] vfio/pci: Add qcom-vfio-pci variant driver

[Jose Ignacio Tornos Martinez]

Add VFIO variant driver for Qualcomm PCIe devices that require
MSI address passthrough for VM operation.

Qualcomm ath11k and ath12k WiFi devices have embedded interrupt
controllers that require physical host MSI addresses programmed to
device registers. In VMs, the driver only sees virtualized guest
addresses, causing firmware initialization to fail.

This variant driver:
1. Caches physical host MSI values after allocation
2. Writes them to extended config space with magic signature "QMSI"
3. VM drivers discover and use these values automatically

The driver follows the VFIO variant driver pattern used by mlx5-vfio-pci
and nvgrace-gpu, providing device-specific quirks within the kernel
without requiring QEMU modifications.

Note: the VM must have IOMMU interrupt remapping and caching mode
enabled so that MSI writes from the device using cached host physical
addresses are properly routed through the IOMMU interrupt remapping
table to the VM.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/qcom/Kconfig         |  22 ++
 drivers/vfio/pci/qcom/Makefile        |   3 +
 drivers/vfio/pci/qcom/main.c          | 272 ++++++++++++++++++++++++++
 drivers/vfio/pci/qcom/msi_quirk.c     | 163 ++++++++++++++++
 drivers/vfio/pci/qcom/qcom_vfio_pci.h |  36 ++++
 7 files changed, 500 insertions(+)
 create mode 100644 drivers/vfio/pci/qcom/Kconfig
 create mode 100644 drivers/vfio/pci/qcom/Makefile
 create mode 100644 drivers/vfio/pci/qcom/main.c
 create mode 100644 drivers/vfio/pci/qcom/msi_quirk.c
 create mode 100644 drivers/vfio/pci/qcom/qcom_vfio_pci.h

diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..237a29443bf5 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -72,6 +72,8 @@ source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
 
 source "drivers/vfio/pci/qat/Kconfig"
 
+source "drivers/vfio/pci/qcom/Kconfig"
+
 source "drivers/vfio/pci/xe/Kconfig"
 
 endmenu
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..63db19102850 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -23,4 +23,6 @@ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
 
 obj-$(CONFIG_QAT_VFIO_PCI) += qat/
 
+obj-$(CONFIG_QCOM_VFIO_PCI) += qcom/
+
 obj-$(CONFIG_XE_VFIO_PCI) += xe/
diff --git a/drivers/vfio/pci/qcom/Kconfig b/drivers/vfio/pci/qcom/Kconfig
new file mode 100644
index 000000000000..8e4c170d01a0
--- /dev/null
+++ b/drivers/vfio/pci/qcom/Kconfig
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config QCOM_VFIO_PCI
+	tristate "VFIO support for Qualcomm PCI devices"
+	depends on PCI
+	select VFIO_PCI_CORE
+	help
+	  This provides VFIO support for Qualcomm PCI devices with device-specific
+	  quirks. Currently supports:
+	  - Qualcomm ath11k WiFi (Device ID 0x1103)
+	  - Qualcomm ath12k WiFi (Device ID 0x1107)
+
+	  These devices require special handling for MSI address/data passthrough
+	  when used with VFIO. The firmware expects physical host MSI addresses
+	  rather than guest IOVA addresses.
+
+	  The VM must have IOMMU interrupt remapping and caching mode enabled.
+	  For example, with libvirt/QEMU on Intel, add to the domain XML:
+	    <iommu model="intel">
+	      <driver intremap="on" caching_mode="on"/>
+	    </iommu>
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/qcom/Makefile b/drivers/vfio/pci/qcom/Makefile
new file mode 100644
index 000000000000..dc48d87d1ef6
--- /dev/null
+++ b/drivers/vfio/pci/qcom/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_QCOM_VFIO_PCI) += qcom-vfio-pci.o
+qcom-vfio-pci-y := main.o msi_quirk.o
diff --git a/drivers/vfio/pci/qcom/main.c b/drivers/vfio/pci/qcom/main.c
new file mode 100644
index 000000000000..0abfdd5faf80
--- /dev/null
+++ b/drivers/vfio/pci/qcom/main.c
@@ -0,0 +1,272 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// SPDX-FileCopyrightText: Copyright Red Hat
+
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+#include <linux/vfio.h>
+#include <linux/vfio_pci_core.h>
+#include <uapi/linux/vfio.h>
+
+#include "qcom_vfio_pci.h"
+
+static struct qcom_vfio_pci_device *to_qcom_vdev(struct vfio_pci_core_device *vdev)
+{
+	return container_of(vdev, struct qcom_vfio_pci_device, vdev);
+}
+
+/*
+ * Intercept config space read operations to return cached host MSI values.
+ * Call core VFIO if not intercepted.
+ */
+static ssize_t qcom_vfio_pci_read(struct vfio_device *core_vdev, char __user *buf,
+				  size_t count, loff_t *ppos)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	struct qcom_vfio_pci_device *qdev = to_qcom_vdev(vdev);
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	loff_t offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	ssize_t ret;
+
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX) {
+		ret = qcom_vfio_pci_config_rw(qdev, buf, count, offset, false);
+		if (ret != -ENOTTY) {
+			if (ret > 0)
+				*ppos += ret;
+			return ret;
+		}
+	}
+
+	return vfio_pci_core_read(core_vdev, buf, count, ppos);
+}
+
+static int qcom_vfio_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qcom_vfio_pci_device *qdev =
+		container_of(core_vdev, struct qcom_vfio_pci_device, vdev.vdev);
+	struct pci_dev *pdev = to_pci_dev(core_vdev->dev);
+	int ret;
+
+	ret = vfio_pci_core_init_dev(core_vdev);
+	if (ret)
+		return ret;
+
+	qdev->msi_offset = qcom_vfio_find_msi_offset(pdev);
+	if (!qdev->msi_offset) {
+		pci_err(pdev, "No free space in extended config for MSI quirk\n");
+		pci_err(pdev, "qcom-vfio-pci cannot function without MSI passthrough, failing probe\n");
+		vfio_pci_core_release_dev(core_vdev);
+		return -ENODEV;
+	}
+	pci_info(pdev, "MSI passthrough quirk using offset 0x%x\n", qdev->msi_offset);
+
+	qdev->msi_cached = false;
+	qdev->msi_addr_lo = 0;
+	qdev->msi_addr_hi = 0;
+	qdev->msi_data = 0;
+
+	return 0;
+}
+
+static const struct vfio_pci_device_ops qcom_vfio_pci_device_ops = {
+	/* No device-specific operations needed */
+};
+
+static int qcom_vfio_pci_open_device(struct vfio_device *core_vdev)
+{
+	struct qcom_vfio_pci_device *qdev =
+		container_of(core_vdev, struct qcom_vfio_pci_device, vdev.vdev);
+	struct vfio_pci_core_device *vdev = &qdev->vdev;
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+	return 0;
+}
+
+/*
+ * Wrap ioctl to intercept MSI allocation and cache host MSI values.
+ * This is called after QEMU allocates MSI via VFIO_DEVICE_SET_IRQS.
+ */
+static long qcom_vfio_pci_ioctl(struct vfio_device *core_vdev,
+				unsigned int cmd, unsigned long arg)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	struct qcom_vfio_pci_device *qdev = to_qcom_vdev(vdev);
+	struct vfio_irq_set hdr;
+	long ret;
+
+	ret = vfio_pci_core_ioctl(core_vdev, cmd, arg);
+
+	/*
+	 * If SET_IRQS ioctl succeeded for MSI and MSI wasn't cached yet,
+	 * cache the MSI descriptor values.
+	 * Only intercept MSI index (not INTx, MSI-X, ERR, or REQ).
+	 */
+	if (ret == 0 && cmd == VFIO_DEVICE_SET_IRQS) {
+		unsigned long minsz = offsetofend(struct vfio_irq_set, count);
+
+		if (copy_from_user(&hdr, (void __user *)arg, minsz))
+			return ret;
+
+		if (hdr.index == VFIO_PCI_MSI_IRQ_INDEX) {
+			pci_dbg(qdev->vdev.pdev,
+				 "SET_IRQS: index=%u flags=0x%x start=%u count=%u cached=%d\n",
+				 hdr.index, hdr.flags, hdr.start, hdr.count, qdev->msi_cached);
+
+			if (!qdev->msi_cached && (hdr.flags & VFIO_IRQ_SET_ACTION_TRIGGER))
+				qcom_cache_and_write_msi(qdev);
+		}
+	}
+
+	return ret;
+}
+
+static const struct vfio_device_ops qcom_vfio_pci_ops = {
+	.name		= "qcom-vfio-pci",
+	.init		= qcom_vfio_pci_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= qcom_vfio_pci_open_device,
+	.close_device	= vfio_pci_core_close_device,
+	.ioctl		= qcom_vfio_pci_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= qcom_vfio_pci_read,
+	.write		= vfio_pci_core_write,
+	.mmap		= vfio_pci_core_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.match_token_uuid = vfio_pci_core_match_token_uuid,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static int qcom_vfio_pci_probe(struct pci_dev *pdev,
+			       const struct pci_device_id *id)
+{
+	struct qcom_vfio_pci_device *qdev;
+	int ret;
+
+	qdev = vfio_alloc_device(qcom_vfio_pci_device, vdev.vdev,
+				 &pdev->dev, &qcom_vfio_pci_ops);
+	if (IS_ERR(qdev))
+		return PTR_ERR(qdev);
+
+	dev_set_drvdata(&pdev->dev, &qdev->vdev);
+	qdev->vdev.pci_ops = &qcom_vfio_pci_device_ops;
+	ret = vfio_pci_core_register_device(&qdev->vdev);
+	if (ret)
+		goto out_put_vdev;
+
+	return 0;
+
+out_put_vdev:
+	vfio_put_device(&qdev->vdev.vdev);
+	return ret;
+}
+
+static void qcom_vfio_pci_remove(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+	struct qcom_vfio_pci_device *qdev = to_qcom_vdev(vdev);
+
+	vfio_pci_core_unregister_device(&qdev->vdev);
+	vfio_put_device(&qdev->vdev.vdev);
+}
+
+static const struct pci_device_id qcom_vfio_pci_table[] = {
+	/* Qualcomm ath11k WiFi */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_QCOM, 0x1103) },
+	/* Qualcomm ath12k WiFi */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_QCOM, 0x1107) },
+	{ }
+};
+
+MODULE_DEVICE_TABLE(pci, qcom_vfio_pci_table);
+
+static struct pci_driver qcom_vfio_pci_driver = {
+	.name			= "qcom-vfio-pci",
+	.id_table		= qcom_vfio_pci_table,
+	.probe			= qcom_vfio_pci_probe,
+	.remove			= qcom_vfio_pci_remove,
+	.driver_managed_dma	= true,
+};
+
+static char ids[1024] __initdata;
+module_param_string(ids, ids, sizeof(ids), 0);
+MODULE_PARM_DESC(ids, "Initial PCI IDs to add to the driver, format is \"vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]\" and multiple comma separated entries can be specified");
+
+static void __init qcom_vfio_pci_fill_ids(void)
+{
+	char *p, *id;
+	int rc;
+
+	if (ids[0] == '\0')
+		return;
+
+	p = ids;
+	while ((id = strsep(&p, ","))) {
+		unsigned int vendor, device, subvendor = PCI_ANY_ID,
+			subdevice = PCI_ANY_ID, class = 0, class_mask = 0;
+		int fields;
+
+		if (!strlen(id))
+			continue;
+
+		fields = sscanf(id, "%x:%x:%x:%x:%x:%x",
+				&vendor, &device, &subvendor, &subdevice,
+				&class, &class_mask);
+
+		if (fields < 2) {
+			pr_warn("qcom-vfio-pci: invalid id string \"%s\"\n", id);
+			continue;
+		}
+
+		rc = pci_add_dynid(&qcom_vfio_pci_driver, vendor, device,
+				   subvendor, subdevice, class, class_mask, 0);
+		if (rc)
+			pr_warn("qcom-vfio-pci: failed to add dynamic id [%04x:%04x[%04x:%04x]] class %#08x/%08x (%d)\n",
+				vendor, device, subvendor, subdevice,
+				class, class_mask, rc);
+		else
+			pr_info("qcom-vfio-pci: add [%04x:%04x[%04x:%04x]] class %#08x/%08x\n",
+				vendor, device, subvendor, subdevice,
+				class, class_mask);
+	}
+}
+
+static int __init qcom_vfio_pci_init(void)
+{
+	int ret;
+
+	ret = pci_register_driver(&qcom_vfio_pci_driver);
+	if (ret)
+		return ret;
+
+	qcom_vfio_pci_fill_ids();
+
+	return 0;
+}
+
+static void __exit qcom_vfio_pci_exit(void)
+{
+	pci_unregister_driver(&qcom_vfio_pci_driver);
+}
+
+module_init(qcom_vfio_pci_init);
+module_exit(qcom_vfio_pci_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Jose Ignacio Tornos Martinez <jtornosm@redhat.com>");
+MODULE_DESCRIPTION("Qualcomm VFIO PCI - Device-Specific VFIO PCI driver for Qualcomm devices");
diff --git a/drivers/vfio/pci/qcom/msi_quirk.c b/drivers/vfio/pci/qcom/msi_quirk.c
new file mode 100644
index 000000000000..583673d699d5
--- /dev/null
+++ b/drivers/vfio/pci/qcom/msi_quirk.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// SPDX-FileCopyrightText: Copyright Red Hat
+
+#include <linux/kernel.h>
+#include <linux/msi.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+#include <linux/vfio_pci_core.h>
+
+#include "qcom_vfio_pci.h"
+
+/**
+ * qcom_cache_and_write_msi - Cache host MSI address and data
+ * @qdev: Qualcomm VFIO device
+ *
+ * Called after QEMU allocates MSI via SET_IRQS ioctl to cache the physical
+ * host MSI addresses that will be returned to the VM driver.
+ */
+void qcom_cache_and_write_msi(struct qcom_vfio_pci_device *qdev)
+{
+	struct pci_dev *pdev = qdev->vdev.pdev;
+	struct device *dev = &pdev->dev;
+	struct msi_desc *desc;
+	u32 addr_lo, addr_hi;
+	u16 data;
+
+	if (qdev->msi_cached)
+		return;
+
+	if (!qdev->msi_offset)
+		return;
+
+	__msi_lock_descs(dev);
+	desc = msi_first_desc(dev, MSI_DESC_ASSOCIATED);
+	if (!desc) {
+		__msi_unlock_descs(dev);
+		pci_warn(pdev, "MSI descriptor not found\n");
+		return;
+	}
+
+	addr_lo = desc->msg.address_lo;
+	addr_hi = desc->msg.address_hi;
+	data = desc->msg.data;
+	__msi_unlock_descs(dev);
+
+	pci_dbg(pdev, "MSI descriptor - addr_lo=0x%x addr_hi=0x%x data=0x%x\n",
+		 addr_lo, addr_hi, data);
+
+	/*
+	 * Validate: address must be valid. Data can be 0 with interrupt remapping.
+	 * When Intel VT-d interrupt remapping is enabled, the MSI data field
+	 * contains an index into the interrupt remapping table, which can be 0.
+	 * We cache it anyway since that's what the VM driver will program.
+	 */
+	if (!addr_lo) {
+		pci_warn(pdev, "MSI address invalid, skipping cache\n");
+		return;
+	}
+	qdev->msi_addr_lo = addr_lo;
+	qdev->msi_addr_hi = addr_hi;
+	qdev->msi_data = data;
+	qdev->msi_cached = true;
+
+	pci_write_config_dword(pdev, qdev->msi_offset + QCOM_VFIO_MSI_MAGIC_OFFSET,
+			       QCOM_VFIO_MSI_MAGIC);
+	pci_write_config_dword(pdev, qdev->msi_offset + QCOM_VFIO_MSI_ADDR_LO_OFFSET,
+			       qdev->msi_addr_lo);
+	pci_write_config_dword(pdev, qdev->msi_offset + QCOM_VFIO_MSI_ADDR_HI_OFFSET,
+			       qdev->msi_addr_hi);
+	pci_write_config_word(pdev, qdev->msi_offset + QCOM_VFIO_MSI_DATA_OFFSET,
+			      qdev->msi_data);
+
+	pci_info(pdev, "Qualcomm WiFi MSI cached at offset 0x%x - addr_lo=0x%x addr_hi=0x%x data=0x%x (magic=QMSI)\n",
+		 qdev->msi_offset, qdev->msi_addr_lo, qdev->msi_addr_hi, qdev->msi_data);
+}
+
+/**
+ * qcom_intercept_msi_read - Intercept MSI config reads and return cached values
+ * @qdev: Qualcomm VFIO device
+ * @pos: Config space offset
+ * @buf: User buffer
+ * @count: Read size
+ *
+ * Returns: Number of bytes read, or negative error
+ */
+static ssize_t qcom_intercept_msi_read(struct qcom_vfio_pci_device *qdev,
+				       loff_t pos, char __user *buf, size_t count)
+{
+	u32 tmp_val = 0;
+	int msi_magic, msi_addr_lo, msi_addr_hi, msi_data;
+
+	if (!qdev->msi_offset)
+		return -ENOTTY;
+
+	msi_magic = qdev->msi_offset + QCOM_VFIO_MSI_MAGIC_OFFSET;
+	msi_addr_lo = qdev->msi_offset + QCOM_VFIO_MSI_ADDR_LO_OFFSET;
+	msi_addr_hi = qdev->msi_offset + QCOM_VFIO_MSI_ADDR_HI_OFFSET;
+	msi_data = qdev->msi_offset + QCOM_VFIO_MSI_DATA_OFFSET;
+
+	if (pos == msi_magic && count == sizeof(u32)) {
+		if (qdev->msi_cached) {
+			tmp_val = QCOM_VFIO_MSI_MAGIC;
+			pci_dbg(qdev->vdev.pdev, "Intercepting MAGIC read at 0x%llx, returning 0x%x (QMSI)\n",
+				 pos, tmp_val);
+		} else {
+			return -ENOTTY;
+		}
+	} else if (qdev->msi_cached) {
+		if (pos == msi_addr_lo && count == sizeof(u32)) {
+			tmp_val = qdev->msi_addr_lo;
+			pci_dbg(qdev->vdev.pdev, "Intercepting ADDR_LO read at 0x%llx, returning 0x%x\n",
+				 pos, tmp_val);
+		} else if (pos == msi_addr_hi && count == sizeof(u32)) {
+			tmp_val = qdev->msi_addr_hi;
+			pci_dbg(qdev->vdev.pdev, "Intercepting ADDR_HI read at 0x%llx, returning 0x%x\n",
+				 pos, tmp_val);
+		} else if (pos == msi_data && count == sizeof(u16)) {
+			tmp_val = qdev->msi_data;
+			pci_dbg(qdev->vdev.pdev, "Intercepting MSI_DATA read at 0x%llx, returning 0x%x\n",
+				 pos, tmp_val);
+		} else {
+			return -ENOTTY;
+		}
+	} else {
+		return -ENOTTY;
+	}
+
+	if (copy_to_user(buf, &tmp_val, count))
+		return -EFAULT;
+
+	return count;
+}
+
+/**
+ * qcom_vfio_pci_config_rw - Config space read/write with MSI interception
+ * @qdev: Qualcomm VFIO device
+ * @buf: User buffer
+ * @count: Size
+ * @offset: Config space offset (0-0xFFF)
+ * @iswrite: true for write, false for read
+ *
+ * Returns: Number of bytes transferred, or -ENOTTY if not intercepted
+ */
+ssize_t qcom_vfio_pci_config_rw(struct qcom_vfio_pci_device *qdev,
+				char __user *buf, size_t count,
+				loff_t offset, bool iswrite)
+{
+	int msi_range_start, msi_range_end;
+
+	if (iswrite)
+		return -ENOTTY;
+
+	if (!qdev->msi_offset)
+		return -ENOTTY;
+
+	msi_range_start = qdev->msi_offset;
+	msi_range_end = qdev->msi_offset + QCOM_VFIO_MSI_SPACE_SIZE;
+
+	if (offset >= msi_range_start && offset < msi_range_end)
+		return qcom_intercept_msi_read(qdev, offset, buf, count);
+
+	return -ENOTTY;
+}
diff --git a/drivers/vfio/pci/qcom/qcom_vfio_pci.h b/drivers/vfio/pci/qcom/qcom_vfio_pci.h
new file mode 100644
index 000000000000..de37fd94b74b
--- /dev/null
+++ b/drivers/vfio/pci/qcom/qcom_vfio_pci.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* SPDX-FileCopyrightText: Copyright Red Hat */
+
+#ifndef _QCOM_VFIO_PCI_H_
+#define _QCOM_VFIO_PCI_H_
+
+#include <linux/vfio_pci_core.h>
+#include <linux/qcom_vfio.h>
+
+/**
+ * struct qcom_vfio_pci_device - Qualcomm VFIO PCI device with quirks
+ * @vdev: Core VFIO PCI device
+ * @msi_offset: Base offset in extended config for MSI passthrough (0 if disabled)
+ * @msi_cached: Whether host MSI address/data has been cached
+ * @msi_addr_lo: Cached host MSI address (low 32 bits)
+ * @msi_addr_hi: Cached host MSI address (high 32 bits)
+ * @msi_data: Cached host MSI data value
+ */
+struct qcom_vfio_pci_device {
+	struct vfio_pci_core_device vdev;
+
+	/* MSI quirk state */
+	int msi_offset;
+	bool msi_cached;
+	u32 msi_addr_lo;
+	u32 msi_addr_hi;
+	u16 msi_data;
+};
+
+/* MSI quirk functions */
+ssize_t qcom_vfio_pci_config_rw(struct qcom_vfio_pci_device *qdev,
+				char __user *buf, size_t count,
+				loff_t offset, bool iswrite);
+void qcom_cache_and_write_msi(struct qcom_vfio_pci_device *qdev);
+
+#endif /* _QCOM_VFIO_PCI_H_ */
-- 
2.54.0

[PATCH 4/7] ath11k: add PCIe link recovery retry

[Jose Ignacio Tornos Martinez]

When ath11k devices are passed through to VMs via PCIe passthrough,
the PCIe link may not stabilize immediately after global reset due to
virtualization layer timing variations. This causes MHI initialization
to fail with "BHI offset: 0xffffffff is out of range" errors because
the device is read before it's ready.

Currently, ath11k_pci_soc_global_reset() just warns if the link is
down but continues anyway, leading to inevitable MHI failures.

This patch adds:
1. A retry loop (up to 5000ms) to wait for the PCIe link to recover
   after global reset (similar to the existing ath12k LTSSM retry logic)
2. Proper error propagation to prevent continuing with a dead link

This is especially important for VM environments where timing can vary
significantly from bare metal.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/net/wireless/ath/ath11k/pci.c | 39 ++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 7 deletions(-)

diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
index 7114eca8810d..4fc6aaf036c1 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -190,8 +190,10 @@ static void ath11k_pci_restore_window(struct ath11k_base *ab)
 	spin_unlock_bh(&ab_pci->window_lock);
 }
 
-static void ath11k_pci_soc_global_reset(struct ath11k_base *ab)
+static int ath11k_pci_soc_global_reset(struct ath11k_base *ab)
 {
+	unsigned long timeout;
+	bool link_recovered = false;
 	u32 val, delay;
 
 	val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
@@ -211,14 +213,29 @@ static void ath11k_pci_soc_global_reset(struct ath11k_base *ab)
 
 	mdelay(delay);
 
-	val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
-	if (val == 0xffffffff)
-		ath11k_warn(ab, "link down error during global reset\n");
+	/* Wait for PCIe link to recover, especially important in VM environments
+	 * where timing can vary significantly from bare metal.
+	 */
+	timeout = jiffies + msecs_to_jiffies(5000);
+	while (time_before(jiffies, timeout)) {
+		val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
+		if (val != 0xffffffff) {
+			link_recovered = true;
+			break;
+		}
+		mdelay(20);
+	}
+	if (!link_recovered) {
+		ath11k_err(ab, "PCIe link failed to recover - device may need power cycle\n");
+		return -ETIMEDOUT;
+	}
 
 	/* Restore window register as its content is cleared during
 	 * hardware global reset, such that it aligns with host cache.
 	 */
 	ath11k_pci_restore_window(ab);
+
+	return 0;
 }
 
 static void ath11k_pci_clear_dbg_registers(struct ath11k_base *ab)
@@ -373,8 +390,10 @@ static void ath11k_pci_force_wake(struct ath11k_base *ab)
 	mdelay(5);
 }
 
-static void ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
+static int ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
 {
+	int ret;
+
 	mdelay(100);
 
 	if (power_on) {
@@ -387,8 +406,12 @@ static void ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
 
 	ath11k_mhi_clear_vector(ab);
 	ath11k_pci_clear_dbg_registers(ab);
-	ath11k_pci_soc_global_reset(ab);
+	ret = ath11k_pci_soc_global_reset(ab);
+	if (ret)
+		return ret;
 	ath11k_mhi_set_mhictrl_reset(ab);
+
+	return 0;
 }
 
 static void ath11k_pci_init_qmi_ce_config(struct ath11k_base *ab)
@@ -818,7 +841,9 @@ static int ath11k_pci_power_up(struct ath11k_base *ab)
 
 	ab_pci->register_window = 0;
 	clear_bit(ATH11K_FLAG_DEVICE_INIT_DONE, &ab->dev_flags);
-	ath11k_pci_sw_reset(ab_pci->ab, true);
+	ret = ath11k_pci_sw_reset(ab_pci->ab, true);
+	if (ret)
+		return ret;
 
 	/* Disable ASPM during firmware download due to problems switching
 	 * to AMSS state.
-- 
2.53.0

[PATCH 5/7] ath11k: Use VFIO MSI cache when available

[Jose Ignacio Tornos Martinez]

The Qualcomm ath11k firmware requires physical host MSI addresses for
interrupt configuration but cannot handle IOVA-based MSI addressing in
virtualized environments.

When the device is passed through to a VM via VFIO:
- The guest kernel allocates MSI vectors and sees IOVA addresses
- The ath11k firmware expects physical host MSI addresses
- Providing IOVA to the firmware causes interrupt delivery to fail
- Result: device is non-functional in VMs

The qcom-vfio-pci variant driver (introduced in earlier patches) solves
this by caching the physical host MSI values and exposing them through
PCI extended config space with a "QMSI" magic signature for discovery.

Search for the QMSI magic signature in extended config space. If found,
the device is running on a VM and we can read the cached host MSI values
from that location. If the VFIO cache is not present (i.e. on baremetal),
fall back to standard PCI MSI configuration.

This change is transparent, arch independent and requires no user
configuration when used with the qcom-vfio-pci driver.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/net/wireless/ath/ath11k/pci.c | 33 +++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
index c372676b097f..99a3a709831c 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -10,6 +10,7 @@
 #include <linux/of.h>
 #include <linux/time.h>
 #include <linux/vmalloc.h>
+#include <linux/qcom_vfio.h>
 
 #include "pci.h"
 #include "core.h"
@@ -463,8 +464,11 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci)
 	struct ath11k_base *ab = ab_pci->ab;
 	const struct ath11k_msi_config *msi_config = ab->pci.msi.config;
 	struct pci_dev *pci_dev = ab_pci->pdev;
+	u32 msi_addr_lo = 0, msi_addr_hi = 0;
 	struct msi_desc *msi_desc;
+	u16 msi_data = 0;
 	int num_vectors;
+	int msi_offset;
 	int ret;
 
 	num_vectors = pci_alloc_irq_vectors(pci_dev,
@@ -488,6 +492,35 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci)
 	}
 	ath11k_info(ab, "MSI vectors: %d\n", num_vectors);
 
+	/*
+	 * Try to get host MSI address from VFIO cache.
+	 * When running on a VM, the ath11k firmware requires physical host
+	 * MSI addresses and cannot handle guest IOVA addresses.
+	 * qcom-vfio-pci writes host MSI values to extended config space with
+	 * a magic signature "QMSI" for discovery. Search for it.
+	 */
+	msi_offset = qcom_vfio_find_msi_cache(pci_dev);
+	if (msi_offset) {
+		pci_read_config_dword(pci_dev,
+				      msi_offset + QCOM_VFIO_MSI_ADDR_LO_OFFSET,
+				      &msi_addr_lo);
+		pci_read_config_dword(pci_dev,
+				      msi_offset + QCOM_VFIO_MSI_ADDR_HI_OFFSET,
+				      &msi_addr_hi);
+		pci_read_config_word(pci_dev,
+				     msi_offset + QCOM_VFIO_MSI_DATA_OFFSET,
+				     &msi_data);
+
+		if (msi_addr_lo && msi_addr_lo != 0xFFFFFFFF) {
+			ab->pci.msi.addr_lo = msi_addr_lo;
+			ab->pci.msi.addr_hi = msi_addr_hi;
+			ab->pci.msi.ep_base_data = msi_data;
+			ath11k_info(ab, "VFIO MSI cache at offset 0x%x: addr_lo=0x%x addr_hi=0x%x data=%d\n",
+				    msi_offset, msi_addr_lo, msi_addr_hi, msi_data);
+			return 0;
+		}
+	}
+
 	ath11k_pci_msi_disable(ab_pci);
 
 	msi_desc = irq_get_msi_desc(ab_pci->pdev->irq);
-- 
2.55.0

[PATCH 6/7] ath12k: add PCIe link recovery retry

[Jose Ignacio Tornos Martinez]

When ath12k devices are passed through to VMs via PCIe passthrough,
the PCIe link may not stabilize immediately after global reset due to
virtualization layer timing variations. This causes MHI initialization
to fail with "BHI offset: 0xffffffff is out of range" errors because
the device is read before it's ready.

Currently, ath12k_pci_soc_global_reset() just warns if the link is
down but continues anyway, leading to inevitable MHI failures.

This patch adds:
1. A retry loop (up to 5000ms) to wait for the PCIe link to recover
   after global reset (similar to the existing LTSSM retry logic)
2. Proper error propagation to prevent continuing with a dead link

This is especially important for VM environments where timing can vary
significantly from bare metal.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/net/wireless/ath/ath12k/pci.c | 39 ++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 7 deletions(-)

diff --git a/drivers/net/wireless/ath/ath12k/pci.c b/drivers/net/wireless/ath/ath12k/pci.c
index af0e882fd0b5..65712486d312 100644
--- a/drivers/net/wireless/ath/ath12k/pci.c
+++ b/drivers/net/wireless/ath/ath12k/pci.c
@@ -181,8 +181,10 @@ static void ath12k_pci_restore_window(struct ath12k_base *ab)
 	spin_unlock_bh(&ab_pci->window_lock);
 }
 
-static void ath12k_pci_soc_global_reset(struct ath12k_base *ab)
+static int ath12k_pci_soc_global_reset(struct ath12k_base *ab)
 {
+	unsigned long timeout;
+	bool link_recovered = false;
 	u32 val, delay;
 
 	val = ath12k_pci_read32(ab, PCIE_SOC_GLOBAL_RESET);
@@ -206,14 +208,29 @@ static void ath12k_pci_soc_global_reset(struct ath12k_base *ab)
 
 	mdelay(delay);
 
-	val = ath12k_pci_read32(ab, PCIE_SOC_GLOBAL_RESET);
-	if (val == 0xffffffff)
-		ath12k_warn(ab, "link down error during global reset\n");
+	/* Wait for PCIe link to recover, especially important in VM environments
+	 * where timing can vary significantly from bare metal.
+	 */
+	timeout = jiffies + msecs_to_jiffies(5000);
+	while (time_before(jiffies, timeout)) {
+		val = ath12k_pci_read32(ab, PCIE_SOC_GLOBAL_RESET);
+		if (val != 0xffffffff) {
+			link_recovered = true;
+			break;
+		}
+		mdelay(20);
+	}
+	if (!link_recovered) {
+		ath12k_err(ab, "PCIe link failed to recover - device may need power cycle\n");
+		return -ETIMEDOUT;
+	}
 
 	/* Restore window register as its content is cleared during
 	 * hardware global reset, such that it aligns with host cache.
 	 */
 	ath12k_pci_restore_window(ab);
+
+	return 0;
 }
 
 static void ath12k_pci_clear_dbg_registers(struct ath12k_base *ab)
@@ -299,8 +316,10 @@ static void ath12k_pci_force_wake(struct ath12k_base *ab)
 	mdelay(5);
 }
 
-static void ath12k_pci_sw_reset(struct ath12k_base *ab, bool power_on)
+static int ath12k_pci_sw_reset(struct ath12k_base *ab, bool power_on)
 {
+	int ret;
+
 	if (power_on) {
 		ath12k_pci_enable_ltssm(ab);
 		ath12k_pci_clear_all_intrs(ab);
@@ -309,8 +328,12 @@ static void ath12k_pci_sw_reset(struct ath12k_base *ab, bool power_on)
 
 	ath12k_mhi_clear_vector(ab);
 	ath12k_pci_clear_dbg_registers(ab);
-	ath12k_pci_soc_global_reset(ab);
+	ret = ath12k_pci_soc_global_reset(ab);
+	if (ret)
+		return ret;
 	ath12k_mhi_set_mhictrl_reset(ab);
+
+	return 0;
 }
 
 static void ath12k_pci_free_ce_irq(struct ath12k_base *ab, int num_ce)
@@ -1452,7 +1475,9 @@ int ath12k_pci_power_up(struct ath12k_base *ab)
 
 	ab_pci->register_window = 0;
 	clear_bit(ATH12K_PCI_FLAG_INIT_DONE, &ab_pci->flags);
-	ath12k_pci_sw_reset(ab_pci->ab, true);
+	ret = ath12k_pci_sw_reset(ab_pci->ab, true);
+	if (ret)
+		return ret;
 
 	/* Disable ASPM during firmware download due to problems switching
 	 * to AMSS state.
-- 
2.55.0

[PATCH 7/7] ath12k: Use VFIO MSI cache when available

[Jose Ignacio Tornos Martinez]

The Qualcomm ath12k firmware requires physical host MSI addresses for
interrupt configuration but cannot handle IOVA-based MSI addressing in
virtualized environments.

When the device is passed through to a VM via VFIO:
- The guest kernel allocates MSI vectors and sees IOVA addresses
- The ath12k firmware expects physical host MSI addresses
- Providing IOVA to the firmware causes interrupt delivery to fail
- Result: device is non-functional in VMs

The qcom-vfio-pci variant driver (introduced in earlier patches) solves
this by caching the physical host MSI values and exposing them through
PCI extended config space with a "QMSI" magic signature for discovery.

Search for the QMSI magic signature in extended config space. If found,
the device is running on a VM and we can read the cached host MSI values
from that location. If the VFIO cache is not present (i.e. on baremetal),
fall back to standard PCI MSI configuration.

This change is transparent, arch independent and requires no user
configuration when used with the qcom-vfio-pci driver.

Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
---
 drivers/net/wireless/ath/ath12k/pci.c | 43 +++++++++++++++++++++++++++
 drivers/net/wireless/ath/ath12k/pci.h |  6 ++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/net/wireless/ath/ath12k/pci.c b/drivers/net/wireless/ath/ath12k/pci.c
index 65712486d312..832d3cf652d9 100644
--- a/drivers/net/wireless/ath/ath12k/pci.c
+++ b/drivers/net/wireless/ath/ath12k/pci.c
@@ -8,6 +8,7 @@
 #include <linux/interrupt.h>
 #include <linux/msi.h>
 #include <linux/pci.h>
+#include <linux/qcom_vfio.h>
 #include <linux/time.h>
 #include <linux/vmalloc.h>
 
@@ -783,8 +784,11 @@ static int ath12k_pci_msi_alloc(struct ath12k_pci *ab_pci)
 {
 	struct ath12k_base *ab = ab_pci->ab;
 	const struct ath12k_msi_config *msi_config = ab_pci->msi_config;
+	u32 msi_addr_lo = 0, msi_addr_hi = 0;
 	struct msi_desc *msi_desc;
+	u16 msi_data = 0;
 	int num_vectors;
+	int msi_offset;
 	int ret;
 
 	num_vectors = pci_alloc_irq_vectors(ab_pci->pdev,
@@ -812,6 +816,39 @@ static int ath12k_pci_msi_alloc(struct ath12k_pci *ab_pci)
 
 	ath12k_info(ab, "MSI vectors: %d\n", num_vectors);
 
+	/*
+	 * Try to get host MSI address from VFIO cache.
+	 * When running on a VM, the ath12k firmware requires physical host
+	 * MSI addresses and cannot handle guest IOVA addresses.
+	 * qcom-vfio-pci writes host MSI values to extended config space with
+	 * a magic signature "QMSI" for discovery. Search for it.
+	 */
+	msi_offset = qcom_vfio_find_msi_cache(ab_pci->pdev);
+	if (msi_offset) {
+		pci_read_config_dword(ab_pci->pdev,
+				      msi_offset + QCOM_VFIO_MSI_ADDR_LO_OFFSET,
+				      &msi_addr_lo);
+		pci_read_config_dword(ab_pci->pdev,
+				      msi_offset + QCOM_VFIO_MSI_ADDR_HI_OFFSET,
+				      &msi_addr_hi);
+		pci_read_config_word(ab_pci->pdev,
+				     msi_offset + QCOM_VFIO_MSI_DATA_OFFSET,
+				     &msi_data);
+
+		if (msi_addr_lo && msi_addr_lo != 0xFFFFFFFF) {
+			ab_pci->msi_addr_hi = msi_addr_hi;
+			ab_pci->msi_addr_lo = msi_addr_lo;
+			ab_pci->msi_ep_base_data = msi_data;
+			ab_pci->msi_addr_cached = true;
+
+			ath12k_info(ab,
+				    "using host MSI from VFIO at offset 0x%x: addr_lo=0x%x addr_hi=0x%x data=%d\n",
+				    msi_offset, msi_addr_lo, msi_addr_hi, msi_data);
+			return 0;
+		}
+	}
+	ab_pci->msi_addr_cached = false;
+
 	ath12k_pci_msi_disable(ab_pci);
 
 	msi_desc = irq_get_msi_desc(ab_pci->pdev->irq);
@@ -1083,6 +1120,12 @@ void ath12k_pci_get_msi_address(struct ath12k_base *ab, u32 *msi_addr_lo,
 	struct ath12k_pci *ab_pci = ath12k_pci_priv(ab);
 	struct pci_dev *pci_dev = to_pci_dev(ab->dev);
 
+	if (ab_pci->msi_addr_cached) {
+		*msi_addr_hi = ab_pci->msi_addr_hi;
+		*msi_addr_lo = ab_pci->msi_addr_lo;
+		return;
+	}
+
 	pci_read_config_dword(pci_dev, pci_dev->msi_cap + PCI_MSI_ADDRESS_LO,
 			      msi_addr_lo);
 
diff --git a/drivers/net/wireless/ath/ath12k/pci.h b/drivers/net/wireless/ath/ath12k/pci.h
index 0e0e2020c6ae..ee3696d40dfb 100644
--- a/drivers/net/wireless/ath/ath12k/pci.h
+++ b/drivers/net/wireless/ath/ath12k/pci.h
@@ -129,6 +129,12 @@ struct ath12k_pci {
 	/* enum ath12k_pci_flags */
 	unsigned long flags;
 	u16 link_ctl;
+
+	/* Cached host MSI address for firmware configuration in VMs */
+	u32 msi_addr_hi;
+	u32 msi_addr_lo;
+	bool msi_addr_cached;
+
 	unsigned long irq_flags;
 	const struct ath12k_pci_ops *pci_ops;
 	u32 qmi_instance;
-- 
2.55.0

Re: [PATCH 3/7] vfio/pci: Add qcom-vfio-pci variant driver

[Jason Gunthorpe]

On Wed, Sep 30, 2026 at 04:08:29PM +0200, Jose Ignacio Tornos Martinez wrote:
> Add VFIO variant driver for Qualcomm PCIe devices that require
> MSI address passthrough for VM operation.
> 
> Qualcomm ath11k and ath12k WiFi devices have embedded interrupt
> controllers that require physical host MSI addresses programmed to
> device registers. In VMs, the driver only sees virtualized guest
> addresses, causing firmware initialization to fail.
> 
> This variant driver:
> 1. Caches physical host MSI values after allocation
> 2. Writes them to extended config space with magic signature "QMSI"
> 3. VM drivers discover and use these values automatically

You should probably explain a little be more here

1) Linux VM driver fills up the normal MSI-X table
2) HW has some non-MSI-X table registers
3) Linux VM driver pokes into the interrupt layer and extracts
   one of the MSX-X table entries addr/data pair
4) Linux VM driver now programs that copied addr/data pair into #2

This is, of course, all wrong. These days it should be using one of
our mechanisms to allow devices to have their own private MSI
registers.

I forget if this is the right way for PCI, but the driver can call

  platform_device_msi_init_and_alloc_irqs()

And directly program the MSI registers with their own special
interrupts vectors, no copying from MSI-X.

If the driver is fixed to work like this, as it should be, then it
fully breaks the scheme you propose here. That's not good.

The problem here is not really a qcom problem, and treating it as a
qcom quirk is why it keeps being stuck, IMHO.

The real issue is that this device MSI scheme does not work in VMs at
all. It does not work because the VM IRQ design requires the VM to
trap and modify all the MSI addr/data pairs at the register write.

The technically clean solution is to redo the VMMs so they don't
require that, ie use interrupt remapping so the VM's view of the
addr/data pair matches physical. That's super hard and will probably
never happen.

But! Now that we have these device MSI domains I wonder if there is
some half option to provide a hypercall so the device MSI domains can
call out to the hypervisor to get the true physical addr/data pair to
program?

This is fundamentally an irq layer issue in Linux, not a qcom one.

Jason

Re: [PATCH 3/7] vfio/pci: Add qcom-vfio-pci variant driver

[Jose Ignacio Tornos Martinez]

Hi Jason,

Thank you for the review.

You're right that the commit message should better explain the current
driver behavior. The flow today is:

1. Linux allocates MSI-X vectors (standard PCI MSI-X table)
2. The Qualcomm firmware has its own interrupt controller with
   private registers that need to be programmed with MSI addr/data
3. The ath11k/ath12k driver extracts the addr/data from the MSI
   descriptor and programs the firmware's private registers
4. In a VM, the driver only sees virtual (IOVA) addr/data, but
   the firmware's private writes bypass the IOMMU, so they need
   the physical host values

I'll improve the commit message.

Regarding platform_device_msi_init_and_alloc_irqs(), I agree that
moving the driver to device MSI domains would be cleaner on the
driver side. However, even with that change, I think the VM problem
remains exactly the same: the device MSI domain in the VM would
allocate virtual addr/data pairs, and the firmware's private interrupt
controller still needs the physical host values. The problem is not
how the driver allocates MSI vectors, but that the firmware writes
to its own registers bypassing the IOMMU.

Also, the VFIO variant driver scheme would not fully break if the
driver moved to device MSI domains. The variant driver caches host
MSI values from the MSI descriptor, regardless of the allocation
method. The hook point may need adaptation, but the fundamental
approach (host caches physical values, VM reads them) remains valid.

Regarding the hypercall idea for device MSI domains, I think that
would be an interesting direction worth exploring as a generic
solution, but that's a multi-subsystem effort that will take time to
design and land. If there is any existing work or design discussion
in this direction, I'd be happy to look into it and contribute.
In the meantime, the VFIO variant driver provides an immediate path
for users with deployed hardware, using an established kernel
pattern. For now, it could coexist and be replaced later on when
everything is ready.

I completely agree that this could theoretically affect any device
with private MSI registers. However, in practice, I'm only aware of
this specific behavior in Qualcomm ath11k/ath12k firmware, where the
device's interrupt controller requires the host physical MSI
addresses to be programmed directly. I haven't seen this in other
devices, which is why I think a device-specific approach seems
appropriate, and the VFIO variant driver pattern fits well for this,
following the same pattern as mlx5-vfio-pci and nvgrace-gpu for
their own device-specific needs. I'd be happy to incorporate any
ideas or suggestions to improve the approach, make it acceptable,
and allow finally to use these devices from VMs.

It's also worth noting that the PCI reset support for these devices
has already landed in linux-next (commit 290153d46d1a "PCI: Add
device-specific reset for Qualcomm devices") [1], solving the
device reset path for VFIO passthrough. This MSI series is the
remaining piece, with both in place, these Qualcomm WiFi devices
would be fully operational in VMs for the first time.

[1] https://lore.kernel.org/all/20260721081301.205374-1-jtornosm@redhat.com/

Thanks

Best regards
Jose Ignacio

Re: [PATCH 3/7] vfio/pci: Add qcom-vfio-pci variant driver

[Jason Gunthorpe]

On Thu, Oct 01, 2026 at 09:19:45AM +0200, Jose Ignacio Tornos Martinez wrote:

> Regarding platform_device_msi_init_and_alloc_irqs(), I agree that
> moving the driver to device MSI domains would be cleaner on the
> driver side. However, even with that change, I think the VM problem
> remains exactly the same: the device MSI domain in the VM would
> allocate virtual addr/data pairs, and the firmware's private interrupt
> controller still needs the physical host values.

Yes, I'm not saying it fixes the problem..

> Also, the VFIO variant driver scheme would not fully break if the
> driver moved to device MSI domains. The variant driver caches host
> MSI values from the MSI descriptor, regardless of the allocation
> method. The hook point may need adaptation, but the fundamental
> approach (host caches physical values, VM reads them) remains valid.

It breaks because VFIO will only cache *actual* MSI-X table entries,
it does not have visibility into anything that
platform_device_msi_init_and_alloc_irqs()

The only reason this works for you at all is because the driver
hackily copies the vectors from the real MSI-X table so it can look in
the "cache" to find their true physical versions.

Which is my general objection, I would like to see the driver to use
platform_device_msi_init_and_alloc_irqs() and don't want to get stuck
unable to do that because it would break this.

> Regarding the hypercall idea for device MSI domains, I think that
> would be an interesting direction worth exploring as a generic
> solution, but that's a multi-subsystem effort that will take time to
> design and land.

Yes, but it does actually solve the problem in all its forms..

> I completely agree that this could theoretically affect any device
> with private MSI registers. However, in practice, I'm only aware of
> this specific behavior in Qualcomm ath11k/ath12k firmware, where the
> device's interrupt controller requires the host physical MSI
> addresses to be programmed directly. 

Everyone else seems to know this stuff doesn't work for
virtualization and doesn't try to do something like that.

> It's also worth noting that the PCI reset support for these devices
> has already landed in linux-next (commit 290153d46d1a "PCI: Add
> device-specific reset for Qualcomm devices") [1], solving the
> device reset path for VFIO passthrough. This MSI series is the
> remaining piece, with both in place, these Qualcomm WiFi devices
> would be fully operational in VMs for the first time.

Why do people care so much about this? I always thought it was a bit
odd?

Jason

Re: [PATCH 3/7] vfio/pci: Add qcom-vfio-pci variant driver

[Jose Ignacio Tornos Martinez]

Hi Jason,

Thanks for the follow-up.

> It breaks because VFIO will only cache *actual* MSI-X table entries,
> it does not have visibility into anything that
> platform_device_msi_init_and_alloc_irqs()

You're right, the current hook is on SET_IRQS which is tied to standard
PCI MSI allocation. If the driver moved to device MSI domains, this specific
hook would stop working.

However, and I was referring to this, the variant driver is a
self-contained module, it doesn't prevent anyone from moving
ath11k/ath12k to device MSI domains. If that work happens, the
variant driver can be adapted to hook the new allocation path, or
replaced entirely.

> The only reason this works for you at all is because the driver
> hackily copies the vectors from the real MSI-X table so it can look in
> the "cache" to find their true physical versions.
>
> Which is my general objection, I would like to see the driver to use
> platform_device_msi_init_and_alloc_irqs() and don't want to get stuck
> unable to do that because it would break this.

I understand the concern, but I haven't seen any active work on
moving ath11k/ath12k to device MSI domains, and the VM problem would
remain exactly the same after that move (as you acknowledged). It would
mean waiting for a refactoring that has no active work or timeline yet,
while users remain without a working solution.

> Yes, but it does actually solve the problem in all its forms..

From my research, there is currently no existing work on device
MSI domain virtualization in KVM. Building it would require
coordinated changes across multiple subsystems (KVM, QEMU, VFIO,
IRQ core, guest drivers, arch code) and several maintainer trees,
and there is no KVM hypercall that bridges to VFIO, so the
architecture would need to be designed from scratch.

In the meantime, the variant driver provides an immediate solution
using an established kernel pattern, for users with deployed hardware.
It can coexist and be replaced when a generic mechanism arrives.

That said, I'm open to restructuring the current approach as a more
generic VFIO mechanism for exposing host-side device metadata to VM
guests, where MSI address caching would be one quirk type rather than
a Qualcomm-specific driver. This way the framework would be reusable
for any device with similar needs. Would that direction be more
acceptable as an intermediate step?

> Everyone else seems to know this stuff doesn't work for
> virtualization and doesn't try to do something like that.

The firmware is what it is, and the hardware is deployed. The kernel
regularly supports devices with non-ideal firmware designs through
quirks and device-specific drivers, and this is what I am trying to
do with the VFIO variant driver pattern (as it was previously done
for other drivers).

> Why do people care so much about this? I always thought it was a bit
> odd?

WiFi passthrough to VMs is useful for network function virtualization,
security isolation (running the wireless stack in a separate VM),
edge computing, and development/testing environments. These are real
use cases with real users who have this hardware deployed today.
Qualcomm wireless cards are widely deployed, perform well, and could be
actively used in these scenarios.

Thanks

Best regards
Jose Ignacio

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jose Ignacio Tornos Martinez]

Hi,

Since no further feedback was received after the last discussion,
I proceeded with v2 addressing review findings (Sashiko):
https://lore.kernel.org/all/20261011074514.515894-1-jtornosm@redhat.com/

I remain open to suggestions and happy to iterate further if needed
to get this acceptable.

Thanks

Best regards
Jose Ignacio

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jason Gunthorpe]

On Sun, Oct 11, 2026 at 09:54:25AM +0200, Jose Ignacio Tornos Martinez wrote:
> Hi,
> 
> Since no further feedback was received after the last discussion,
> I proceeded with v2 addressing review findings (Sashiko):
> https://lore.kernel.org/all/20261011074514.515894-1-jtornosm@redhat.com/
> 
> I remain open to suggestions and happy to iterate further if needed
> to get this acceptable.

IMHO this is not OK, building a new crazy hypervisor interface that
only works with the broken driver design we have today is not OK. This
is not some "quirk", it is a permanent new ABI between VMM and kernel
that has real compatibility implications.

Go tell an AI to convert it to
platform_device_msi_init_and_alloc_irqs() and then solve for that so
at least we are not getting trapped into a completely broken ABI by
all the hacks here.

The whole "MSI Caching" concept is fully wrong to be an ABI to a VM.

Jason

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jose Ignacio Tornos Martinez]

Hi Jason,

I understand the concern about compatibility implications, but to
clarify, my intention is not to create an ABI or a "MSI caching
concept". It is simply passing the necessary information from the
host driver to the VM driver so the device can work. The mechanism
uses the existing VFIO variant driver pattern, but I'm open to any
other mechanism that achieves the same goal.

I can explore converting ath11k/ath12k to
platform_device_msi_init_and_alloc_irqs(). However, if I understood
correctly, the VM problem remains identical after that change — the
firmware still needs physical host addresses, and device MSI domains
in the VM would still provide virtual ones.

I'm happy to do this refactoring if it opens a path to get these
devices working in VMs. What VM mechanism would you accept for
exposing physical MSI values through device MSI domains?

Jeff is escalating to Qualcomm engineering again (thank you for
that), but users with deployed hardware need something working in
a reasonable timeframe.

Thanks

Best regards
Jose Ignacio

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jason Gunthorpe]

On Tue, Oct 13, 2026 at 03:43:13PM +0200, Jose Ignacio Tornos Martinez wrote:
> Hi Jason,
> 
> I understand the concern about compatibility implications, but to
> clarify, my intention is not to create an ABI or a "MSI caching
> concept". 

It very much is an ABI, you defined a new ABI in the config space a
layout between guest and host that we have to maintain forever.

Just for a hacky solution, it is not so nice.

> I can explore converting ath11k/ath12k to
> platform_device_msi_init_and_alloc_irqs(). However, if I understood
> correctly, the VM problem remains identical after that change — the
> firmware still needs physical host addresses, and device MSI domains
> in the VM would still provide virtual ones.

Yes, but it removes the solution of copying data from the existing
MSI-X table from consideration, which is the primary hacky thing here.

> I'm happy to do this refactoring if it opens a path to get these
> devices working in VMs. What VM mechanism would you accept for
> exposing physical MSI values through device MSI domains?

Once you have a MSI domain you just need a driver wired into it that
can request and get the physical addresses from the hypervisor. Then
when that domain is in use the guest will always see true physical.

That part is easy enough, it already exists and you don't need any
driver changes.

The bit that isn't clear is how you get the domain in the first
place. I suppose something like a virtio-msi is a good starting point,
but I don't know enough about ACPI/etc to imagine how to wire it into
the flow.

IIRC hyperv does this, so maybe there is something to follow there..

> Jeff is escalating to Qualcomm engineering again (thank you for
> that), but users with deployed hardware need something working in a
> reasonable timeframe.

Yes, but it has already been years, and Qualcomm doesn't seem to care
about these uses cases, so I'm not sure I see the urgency to do a bad
ABI hack..

Jason

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jose Ignacio Tornos Martinez]

Thank you for the direction on device MSI domains. The Hyper-V
reference is difficult for me to follow for this case. Could
you point me to some existing KVM/VFIO equivalent, or similar
example for my use case? Any pointers to relevant code,
documentation, or previous discussions on this topic would be
very helpful to get started in the right direction.

Also, where would this need to be implemented? Kernel only
because it is already supported in QEMU in a general way, or
kernel and QEMU for some specifics?

Best regards
Jose Ignacio

Re: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Jason Gunthorpe]

On Tue, Oct 13, 2026 at 05:42:25PM +0200, Jose Ignacio Tornos Martinez wrote:
> Thank you for the direction on device MSI domains. The Hyper-V
> reference is difficult for me to follow for this case. Could
> you point me to some existing KVM/VFIO equivalent, or similar
> example for my use case? Any pointers to relevant code,
> documentation, or previous discussions on this topic would be
> very helpful to get started in the right direction.
> 
> Also, where would this need to be implemented? Kernel only
> because it is already supported in QEMU in a general way, or
> kernel and QEMU for some specifics?

I spent some time with AI to research this and drafted up some slop
for you that outlines a bunch of ideas and explains how some of the
bits work today. Not saying this is something to follow but it should
be a good starting point to come up with a good idea.

It is quite interesting that amd virtual interrupt remapping in the
iommu is very close to being able to solve the problem without any
guest changes. Maybe that is the right thread to pull on, if AMD can
be fixed fully then maybe there is an option to negotiate something
within the Intel spec, or build a similar idea with a 'virtio
interrupt remapping'

Nevertheless this looks reasonably tractable to me, assuming you have
some strong AI to help understand the whole picture and baroque
hardware details it is even almost approachable..

Jason

Host-managed MSI: the Hyper-V flow and a virtio proposal
========================================================

Goal: make device-specific MSI storage work for devices assigned to KVM guests.
Such storage is outside the PCI MSI/MSI-X capability. In Linux it is driven
through a per-device MSI domain, today via
platform_device_msi_init_and_alloc_irqs(). A PCI device needs a PCI counterpart
of that helper, which does not exist yet and must be written. The VMM cannot
trap this storage, so the guest must write host-valid MSI messages into it
directly, as Hyper-V guests do for MSI-X. The guest obtains those messages, and
routes the interrupts, through a paravirtual (virtio) interface.

Sections 1–4 describe how Hyper-V does this through the Linux IRQ domain
hierarchy. Section 5 maps that onto the KVM/VFIO host stack and assesses the
virtio proposal. Section 6 looks at an alternative with no paravirtual
interface: the host mirrors the guest's vIOMMU interrupt remapping (or vITS)
entries 1:1.

0. Why host-managed MSI exists at all
-------------------------------------

On a normal VFIO/KVM system the VMM traps guest writes to a passthrough device's
MSI/MSI-X registers. The guest writes a virtual message (a guest APIC address
and a guest vector). The VMM then programs the physical device and the physical
interrupt remapping (IR) table on the guest's behalf. The guest never sees a
host-valid message.

Hyper-V does not do that. In the vPCI and root-partition cases, the guest writes
the device's MSI capability and MSI-X table directly. Writes are not trapped and
emulated (see the comment above hv_pci_restore_msi_state()). The IOMMU and its
interrupt remapping table are owned by the hypervisor. The value the guest
writes must therefore already be a host-valid, remapped message, and only the
host can produce one. Hyper-V has the guest ask the host for each message.

Every design decision below follows from that premise:

  The guest chooses a destination (vCPU + vector, or a guest interrupt line).
  The host allocates a remapping entry for (source device → destination) and
  returns the MSI address/data that selects it. A Linux IRQ domain hands that
  pair to the MSI core, which writes it into the device using the normal PCI MSI
  writer.

For device-specific storage on KVM there is no trap to remove: the storage was
never trappable. Adopting this premise is the only way it can work in a guest.
Section 5.2 shows that the KVM half (stable host-composed messages, IRTE-only
retargeting, posting) already exists. The missing pieces are host IRQs without
host-side storage in VFIO, and an MSI parent in the guest that serves both the
PCI MSI-X domain and per-device MSI domains.

1. Separate the destination from the message
--------------------------------------------

  -------------------------------------------------------------------------------
  Identity                Meaning
  ----------------------- -------------------------------------------------------
  Linux IRQ number (virq) The guest kernel's software handle. Never visible to
                          the host.

  Destination             What the guest wants to receive: on x86 a vCPU set +
                          IDT vector; on Hyper-V ARM64 a GIC SPI number

  Source identity         The requester whose MSI writes the host will accept for
                          this mapping

  Host mapping            Host state binding (source, message) → destination; on
                          hardware, an IR table entry

  MSI address/data        What is written into the device; on IR hardware this
                          effectively selects the host mapping
  -------------------------------------------------------------------------------

In a hierarchical domain, one Linux IRQ has an irq_data at each level, linked by
parent_data, and each level keeps private state in chip_data. For host-managed
MSI:

-   the parent (receive) domain allocates the guest destination;
-   the host-managed MSI parent domain obtains the message from the host and
    owns the mapping's lifetime;
-   the device MSI domain writes the message into the device.

The guest must treat the returned pair as host-defined. It must not reconstruct
an APIC-format address from the CPU number, substitute its own vector into the
data, or assume an ITS doorbell. The only structure the guest may rely on is
what the protocol explicitly defines. Hyper-V's one example is conventional
multi-message MSI, where the data of vector n is base data + n (section 4).

2. The common Linux path: compose, then write
---------------------------------------------

Root-partition hierarchy on x86:

    PCI device MSI/MSI-X domain     writes device registers / MSI-X table
                 |
    Hyper-V MSI parent domain       obtains message from host, owns mapping
                 |
    x86 vector domain               allocates guest CPU + vector

Allocation walks down the hierarchy to reserve receive resources. Activation
walks parents first, then the MSI device domain's msi_domain_activate() does:

    msi_domain_activate(device_domain, irq_data)
      +-- irq_chip_compose_msi_msg(irq_data, msg)
      |     walks irq_data -> parent_data to the first level with
      |     .irq_compose_msi_msg  -> Hyper-V callback returns host pair
      +-- irq_chip_write_msi_msg(irq_data, msg)
            -> pci_msi_domain_write_msg() -> __pci_write_msi_msg()
                 MSI:   config-space address/data registers
                 MSI-X: the entry in the device's MSI-X table

Notes:

-   The Hyper-V compose callback never touches device registers; the write
    belongs to the PCI MSI layer.
-   __pci_write_msi_msg() caches the message in msi_desc->msg. If the device is
    not in D0 or is disconnected, it skips the hardware write.
-   irq_compose_msi_msg returns void. msi_domain_activate() and
    msi_domain_set_affinity() start from a zeroed msg. If the provider fails, it
    simply leaves (or sets) 0/0, and a zero message is written to the device.
    Nothing reports the failure to the caller.
-   msi_domain_set_affinity() asks the parent to change destination. Unless the
    parent returns IRQ_SET_MASK_OK_DONE, it composes and writes again. Every
    affinity change therefore re-runs the Hyper-V compose callback.
-   Both Hyper-V x86 chips set IRQCHIP_MOVE_DEFERRED. On x86 the affinity change
    is applied from the interrupt's own handler context (irq_move_masked_irq()).
    Any host interaction done in compose therefore runs in hard-IRQ context with
    the descriptor lock held.

3. Hyper-V root partition: direct hypercalls
--------------------------------------------

Code: arch/x86/hyperv/irqdomain.c.

Here Linux is the root partition. It sees the physical PCI topology, but the
hypervisor owns the IOMMU.

Installing the domain
~~~~~~~~~~~~~~~~~~~~~

hv_init.c sets x86_init.irqs.create_pci_msi_domain = hv_create_pci_msi_domain
when running as root. That creates a single system-wide MSI parent domain over
x86_vector_domain. hv_msi_domain_alloc() only allocates parent vectors and
installs hv_pci_msi_controller. It does no host work; a TODO says the mapping
should move there.

Supported flags are generic + MSI-X. Conventional multi-message MSI is not
supported, and each mapping has vector_count = 1, edge-triggered.

Mapping: all work happens in compose
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    hv_irq_compose_msi_msg(irqd, msg)
      if irqd->chip_data:                  # already mapped (affinity change)
          HVCALL_UNMAP_DEVICE_INTERRUPT(old entry); kfree
      entry = kzalloc(GFP_ATOMIC)
      hv_map_msi_interrupt()
         source = physical segment/BDF, after walking DMA aliases;
                  PCI-X bridges add a bus-range "shadow"
         cpu    = first CPU of the effective affinity (single CPU)
         vector = parent vector (irqd_cfg()->vector)
         HVCALL_MAP_DEVICE_INTERRUPT  -> hv_interrupt_entry
      irqd->chip_data = entry
      msg = { address_hi = 0,
              address_lo = entry.msi_entry.address,
              data       = entry.msi_entry.data }

The hypercall returns the 32-bit address and 32-bit data. Linux copies them
verbatim. address_hi = 0 is a property of this interface, not a guest encoding
choice.

Affinity change
~~~~~~~~~~~~~~~

The hypervisor does not let the root partition change the vector or move the
interrupt to VPs outside the originally mapped set. For that reason, root does
not use RETARGET. Every affinity change is unmap, then map, followed by a device
rewrite with the possibly new pair.

This has consequences:

-   Between the unmap and the device rewrite, the device holds a message that
    refers to a released mapping.
-   If the map fails, compose returns with chip_data == NULL and a zero msg, and
    the device is programmed with 0/0. The IRQ is now dead and nothing reports
    it.
-   All of this runs in the deferred-move path, in hard-IRQ context (section 2).

Teardown
~~~~~~~~

hv_msi_domain_free() → hv_teardown_msi_irq() issues
HVCALL_UNMAP_DEVICE_INTERRUPT with the stored entry. The mapping is a host
resource; forgetting the pair in the guest leaks it.

4. Hyper-V guest vPCI: CREATE over VMBus, RETARGET by hypercall
---------------------------------------------------------------

Code: drivers/pci/controller/pci-hyperv.c.

Here Linux is an ordinary guest with passthrough devices exposed on a synthetic
PCI bus. Each vPCI bus is a VMBus device with its own channel.

Two control paths
~~~~~~~~~~~~~~~~~

This split is the most important thing to understand before designing a virtio
version:

CREATE (PCI_CREATE_INTERRUPT_MESSAGE{,2,3})

-   Transport: VMBus request/response
-   Called from: hv_compose_msi_msg()
-   Context: IRQ descriptor lock held; polls the channel

DELETE (PCI_DELETE_INTERRUPT_MESSAGE)

-   Transport: VMBus, fire-and-forget
-   Called from: compose (replace) and hv_msi_free()

RETARGET (x86 only)

-   Transport: HVCALL_RETARGET_INTERRUPT, synchronous hypercall
-   Called from: hv_irq_unmask()
-   Context: atomic

Slow, rare operations use the message transport. The operation that must
complete inside an IRQ chip callback uses a synchronous trap. VMBus is not used
for that.

Domain setup
~~~~~~~~~~~~

hv_pcie_init_irq_domain() creates one MSI parent domain per vPCI bus, and
dev_set_msi_domain() binds it to that bus's host bridge. Device-to-domain
selection is therefore by topology: every device on that synthetic bus uses that
domain. Source identity is likewise implicit. The host already knows which
physical function sits behind each slot (wslot) on that channel.

Parent receive domain:

-   x86: x86_vector_domain. Destination = vCPU(s) + IDT vector.
-   ARM64: a private hv_vpci_arm64 domain over the GIC. It hands out SPIs
    64–1019 from a bitmap in power-of-two aligned blocks. Destination = SPI
    number, sent to the host as the "vector". CPU routing of that SPI is done by
    the guest's virtual GIC distributor, which Hyper-V already emulates. So
    hv_arch_irq_unmask() is empty on ARM64, and no retarget is ever needed.

CREATE
~~~~~~

hv_compose_msi_msg():

1.  For multi-MSI with existing chip_data, return the cached pair (no host
    call).
2.  If a previous int_desc exists (single MSI / MSI-X), send DELETE for it.
3.  Build a v1/v2/v3 CREATE request: slot, vector, vector count, and one "dummy"
    vCPU (first online CPU of the affinity; round-robin for multi-MSI).
4.  Send it with vmbus_sendpacket_getid(). Disable the channel tasklet. Spin
    calling hv_pci_onchannelcallback() with udelay(100) until the completion
    fires, the device disappears (vendor ID 0xFFFF), or the channel is torn
    down.
5.  Store the returned tran_int_desc in chip_data. Return
    address_hi:lo = int_desc.address (64-bit) and data = int_desc.data.

On any failure it returns a zero message.

The vCPU in CREATE is not cosmetic. Hyper-V picks the physical CPU that services
the interrupt based on it, and (as of the comment in the driver) RETARGET does
not reselect the pCPU. The vector in CREATE is the real vector for single
MSI/MSI-X, but a dummy 32 for multi-MSI.

RETARGET at unmask (x86)
~~~~~~~~~~~~~~~~~~~~~~~~

hv_irq_unmask() → hv_arch_irq_unmask() → hv_irq_retarget_interrupt(). This
issues HVCALL_RETARGET_INTERRUPT with:

-   device id: built from the vPCI bus instance GUID + PCI function;
-   the existing message (address_lo, data) used as the key that identifies the
    mapping;
-   the new vector and the full effective-affinity vCPU set.

The message is unchanged; the host updates its IR entry behind it. Note that
Hyper-V uses the address/data pair itself as the mapping handle. There is no
separate handle.

Nested root (Linux is root under a nested hypervisor) calls
hv_map_msi_interrupt(data, NULL) instead and discards the returned entry.

Affinity in practice (x86)
~~~~~~~~~~~~~~~~~~~~~~~~~~

An affinity change goes through msi_domain_set_affinity() → recompose. For
single MSI/MSI-X that means DELETE + CREATE over VMBus, polled in hard-IRQ
context, then a device rewrite, then RETARGET on the next unmask. Multi-MSI
skips the recompose (cached) and relies on RETARGET alone. All vectors of a
multi-MSI device therefore share whatever pCPU the host picked at CREATE.

Multi-message MSI
~~~~~~~~~~~~~~~~~

Conventional MSI has one address and a base data value; the device ORs in the
vector index. The host allocates an aligned block for nvec_used vectors during
the first IRQ's CREATE. Later IRQs derive data = base + (irq - first_irq)
without asking the host. This works only because the host guarantees block
semantics. Independent per-vector pairs cannot be advertised as multi-MSI.

Free and resume
~~~~~~~~~~~~~~~

hv_msi_free() sends DELETE for the stored descriptor. On resume, a cached MSI-X
table replay is useless because the host's IR entries are gone.
hv_pci_restore_msi_state() therefore calls hv_compose_msi_msg() for every
descriptor to re-CREATE.

What the Hyper-V code itself admits is wrong
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   Both providers have a TODO to move host allocation out of compose and into
    .alloc.
-   Failure in compose silently programs 0/0.
-   vPCI CREATE runs a polled, unbounded message round-trip under IRQ locks,
    including from the deferred affinity-move path.

5. Applying this to KVM/VFIO: device MSI storage in a guest
-----------------------------------------------------------

5.1 The functional problem
~~~~~~~~~~~~~~~~~~~~~~~~~~

Many devices keep MSI address/data in device-specific storage: a context table
in device memory, or a field in a queue context that lives in DMA'd host memory.
The PCI MSI/MSI-X capability is not involved. (The PCI/IMS proposal used to
cover this; the current tree does it with per-device MSI domains, see below.) A
guest driver for an assigned device has to write a message into that storage,
and:

-   the VMM cannot trap it. The format is device-specific, and the storage may
    be in guest RAM that the device reads by DMA;
-   a guest-format message (guest APIC ID + guest vector) is wrong. On a host
    with interrupt remapping, the device must emit a remappable-format message
    that selects a host IRTE belonging to this device.

So device-specific MSI storage cannot work in a KVM guest today. The only fix is
the Hyper-V model: the guest asks the host for a host-valid message and writes
it itself. This is "Model B": a host-provided message that the guest programs
into the storage.

The generic Linux side already exists.
platform_device_msi_init_and_alloc_irqs(dev, nvec, write_msi_msg) creates a
DOMAIN_BUS_DEVICE_MSI domain on dev, parented on dev->msi.domain, and allocates
nvec interrupts. The parent composes each message and the core hands it to the
driver's write_msi_msg(desc, msg), which stores it wherever the device wants it.
ARM ITS/MBI and the irq-msi-lib parents (e.g. RISC-V IMSIC) accept this bus
token. The x86 parents (x86_vector_msi_parent_ops, DMAR, AMDVI) accept only
DOMAIN_BUS_PCI_DEVICE_MSI{,X}, which is one of the gaps this work fills.

Two properties of the helper shape the design:

-   The platform helper is the wrong API for a PCI device. It is built for
    devices whose only MSI domain is the device MSI domain. It always uses
    MSI_DEFAULT_DOMAIN, and MSI_MAX_DEVICE_IRQDOMAINS is 1. A PCI function's own
    MSI/MSI-X domain already occupies that slot. A PCI device therefore needs a
    new pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) (and a
    matching pci_device_msi_free_irqs_all()). Neither exists in the tree yet;
    both must be written as part of this work. The new helper uses the same
    DOMAIN_BUS_DEVICE_MSI template and driver write callback, but places the
    domain in a second per-device domain slot that the core must add, so it
    coexists with MSI-X on the same struct device. Both domains are then
    parented on the function's dev->msi.domain, and the source identity is
    simply the function itself.
-   The device MSI chip has no mask of its own (irq_chip_mask_parent), no
    affinity callback, and no feature flags. Masking and retargeting are the
    parent's job, which suits a host-managed parent.

5.2 What the KVM/VFIO host already provides
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    VFIO_DEVICE_SET_IRQS(MSI-X, index i, eventfd)
       host: pci_msix_alloc_irq_at(i) / request_irq(vfio_msihandler)
       host IR domain composes a remappable msg (IRTE index [+ subhandle])
       host writes that msg into the physical MSI-X entry i
       irq_bypass_register_producer(eventfd, host_irq)
    KVM_SET_GSI_ROUTING(gsi -> guest-format msg) + KVM_IRQFD(eventfd, gsi)
       kvm_pi_update_irte(): post the IRTE to the vCPU PI descriptor + guest vector,
       or leave it remapped to a host vector and let KVM inject

Properties to reuse:

-   The host message is stable for the life of the host IRQ.
    intel_ir_compose_msi_msg() returns the cached ir_data->msi_entry. Host
    affinity changes (intel_ir_set_affinity(), amd_ir_set_affinity()) rewrite
    only the IRTE and return IRQ_SET_MASK_OK_DONE. Posting changes
    (irq_set_vcpu_affinity()) also rewrite only the IRTE.
-   Guest retargeting is host-only. It is a KVM routing update followed by an
    IRTE rewrite. The device is never touched.

The KVM half (irqfd, GSI routing, posting) works on any host IRQ that has an IR
parent. It does not care whether the host IRQ came from MSI-X or from somewhere
else. The missing piece is entirely on the VFIO/host-IRQ side. VFIO can only
create host IRQs for MSI-X table indices. A device MSI interrupt needs a host
IRQ that has an IRTE but no host-side storage.

5.3 Security: what makes guest-written messages safe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The guest controls the device's DMA. It can already make the device write any
value to 0xFEEx_xxxx with its requester ID. VFIO's isolation therefore already
rests on interrupt remapping:

-   Intel: source-ID-checked IRTEs (set_msi_sid()), and compatibility-format
    MSIs blocked;
-   AMD: per-device remap tables.

Letting the guest write host-valid messages into device storage adds no new
requirement, provided the whole requester ID belongs to that one guest. The
guest can only reach IRTEs whose source-ID check passes for its own RID, and all
of those are its own.

This breaks for shared-RID assignment (SIOV/ADIs). Several guests' ADIs emit
MSIs with the same RID, and IR checks only the RID, not the PASID. A guest could
put another guest's IRTE handle into its own storage entry and inject interrupts
into that guest. That is why SIOV designs kept this storage host-programmed.
Guest-written messages are safe only for whole-function assignment (a PF or a VF
with its own RID). The proposal must state this scope, and the host must refuse
CREATE for devices whose RID is shared.

5.4 Protocol
~~~~~~~~~~~~

  ----------------------------------------------------------------------------------
  Op           Request                     Response                    Transport
  ------------ --------------------------- --------------------------- -------------
  CREATE       device, domain (MSI-X /     per vector: handle, address virtqueue;
               device MSI), index, count,  (64-bit), data (32-bit);    can sleep
               storage address/data widths count granted               

  RETARGET     handle, guest-format MSI    status; host message        synchronous
               message composed by the     unchanged                   trapped
               guest's parent vector                                   register
               domain                                                  

  DESTROY      handle                      status; no further delivery virtqueue;
                                                                       can sleep

  (host →      config-change notification  all handles stale; guest    config
  guest)                                   must re-CREATE and rewrite  interrupt
  INVALIDATE                               storage                     
  ----------------------------------------------------------------------------------

Design points:

-   The message is stable and destination-independent. The host IR already
    guarantees this (5.2). CREATE therefore carries no destination and can run
    from the guest's .alloc, which can sleep and fail cleanly. Compose copies
    the cached pair. Affinity changes never touch device storage. That matters
    for device MSI in particular: writing a queue context from hard-IRQ context
    is often impossible, and the device MSI chip has no affinity callback of its
    own anyway.
-   The RETARGET destination is the guest-format message. The guest's virtio-msi
    level composes it by calling the parent x86_vector_domain (APIC ID, vector,
    extended destination ID). The VMM puts it unchanged into KVM routing, so no
    new destination encoding is needed.
-   RETARGET is synchronous. It runs from irq_set_affinity (hard-IRQ context
    with IRQCHIP_MOVE_DEFERRED) and from activation. Use a write to a trapped
    virtio config/MMIO register and read back status. The vCPU waits in the exit
    until the VMM has updated routing. A virtqueue cannot provide this. Hyper-V
    uses the same split: VMBus for CREATE, a hypercall for RETARGET.
-   The device MSI index is only a name. The host does not interpret the guest's
    msi_index or storage slot. It only scopes handles, so that DESTROY and
    INVALIDATE are well defined. (MSI-X differs, see 5.6(c).)
-   Width negotiation. The x86 remappable message has a 32-bit address_lo
    carrying the IRTE index, address_hi = 0, and data carrying the subhandle.
    Device storage may be narrower. The guest declares the widths it has, and
    the host fails CREATE if its message cannot fit.

5.5 Host-side work (VFIO + IR)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1.  Storage-less host IRQs. A VFIO device needs a way to allocate host IRQs with
    an IR parent but no host writer. The same device MSI mechanism fits: VFIO
    calls the new (not yet written) pci_device_msi_init_and_alloc_irqs() on the
    assigned function, under its existing IR parent, with a write callback that
    only records the composed message. The host cannot mask storage it doesn't
    own; masking stays at the IR/eventfd level and the guest owns device-side
    masking. This needs:
    -   writing pci_device_msi_init_and_alloc_irqs() and adding the second
        per-device domain slot (5.1), shared with the guest side,
    -   the x86 vector, DMAR and AMDVI parents accepting DOMAIN_BUS_DEVICE_MSI
        (x86_init_dev_msi_info(), x86_msi_prepare()), with an allocation type
        that programs the PCI function's source ID (set_msi_sid()) as for MSI-X,
    -   the host refusing it without isolated MSI.
2.  New VFIO uAPI. "Allocate N storage-less interrupts on this device, attach
    eventfds, return each host message." This could be a new IRQ index or a new
    ioctl. It needs a quota, and the irq bypass producer is registered exactly
    as for MSI-X, so KVM posting works unchanged.
3.  Scalability. Device-specific storage exists largely to go beyond MSI-X's
    2048 vectors. Each guest interrupt here costs:
    -   a host IRQ and a host vector, because the IR domain allocates a
        vector-domain parent even when posted. That is about 200 per host CPU.
    -   a KVM GSI route. KVM_MAX_IRQ_ROUTES is 4096, and KVM_SET_GSI_ROUTING
        replaces the whole table, with expedited SRCU, on every RETARGET.
    -   an IRTE. Intel has 64K per IOMMU, shared host-wide; AMD has a
        512/2048-entry per-device table.

    Thousands of device MSI interrupts per guest need posted-only IRTEs without
    a host vector, and an incremental single-route KVM update (or a KVM-side
    RETARGET). Neither exists today. Without them the design tops out at roughly
    MSI-X scale, which defeats much of the point.

5.6 Guest-side work
~~~~~~~~~~~~~~~~~~~

(a) Device storage through a device MSI domain. Once the new helper exists
(5.1), the device driver calls
pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) alongside its
normal pci_alloc_irq_vectors() MSI-X allocation. The callback receives the
msi_desc and uses desc->msi_index to find the queue/context to write. The domain
is a fixed-size block allocated up front and is torn down with
pci_device_msi_free_irqs_all(). Apart from the new PCI entry point and the
second slot, this is the existing platform MSI mechanism unchanged.

(b) The virtio-msi domain is the device's MSI parent.
msi_create_device_irq_domain() parents every per-device domain on
dev->msi.domain, so the function's MSI-X domain and its device MSI domain share
one parent. For a served device that parent must be the virtio-msi domain. It
supplies its own init_dev_msi_info accepting DOMAIN_BUS_PCI_DEVICE_MSIX and
DOMAIN_BUS_DEVICE_MSI (the x86 helper rejects both a non-x86 real parent and the
device MSI token), its own msi_prepare that tags CREATE with the domain type,
and irq_mask/irq_unmask, because the device MSI chip forwards masking to the
parent. MSI-X is then also guest-written. VFIO's host writes to entry i write
the same value, and the VMM must stop emulating the MSI-X table for that device.
This is consistent and simpler than splitting MSI-X and device MSI across two
models.

(c) MSI-X entries must be bound to the host index. The host masks, restores and
refreshes MSI-X entry i (in vfio_msi_set_vector_signal() and
pci_restore_msi_state()). CREATE for MSI-X therefore names index i, and the
guest must write that message only into entry i. Ownership must be defined:

-   address/data: the guest, with the host only rewriting the identical value;
-   the per-vector mask bit: the guest, once set-up completes.

Device MSI has no host writer, so there is no conflict there.

(d) Domain binding on KVM. Hyper-V gives each passthrough device its own
synthetic bus, so dev_set_msi_domain() on the bridge selects the provider. A q35
guest mixes emulated devices, which keep normal MSI, with assigned ones, and x86
has no firmware mechanism such as DT msi-map or ACPI IORT to choose an MSI
parent per device. Put the assigned devices behind a dedicated host bridge (e.g.
pxb-pcie) and bind virtio-msi to that bridge. The device MSI domain lives on the
same function, so it gets the same parent.

(e) Probe ordering and bootstrap. The virtio-msi control device takes its own
interrupts from ordinary emulated MSI, not from its own domain. Its domain must
exist before any served device allocates vectors. A dedicated bridge enumerated
after virtio-msi probes, or -EPROBE_DEFER on the domain's absence, handles that.

(f) Callbacks.

    struct virtio_msi_irq {
            u32 handle;
            struct msi_msg msg;     /* host message, stable until INVALIDATE */
    };

-   .alloc: allocate parent vectors, then CREATE (sleeping). On failure, unwind
    and return the error, so pci_alloc_irq_vectors() or
    pci_device_msi_init_and_alloc_irqs() fails cleanly. Hyper-V instead programs
    0/0.
-   .activate: the parent activates first. Compose the guest-format message from
    the parent and RETARGET.
-   .irq_set_affinity: call the parent, compose its new guest-format message,
    RETARGET, and return IRQ_SET_MASK_OK_DONE so the device storage is never
    rewritten. If RETARGET fails, restore the parent and return an error.
-   .irq_compose_msi_msg: *msg = m->msg.
-   .free: DESTROY (sleeping) after the core has shut the IRQ down.

5.7 Migration and invalidation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The destination host allocates different IRTEs, so every message the guest has
written becomes invalid. For MSI-X the guest can rewrite from cached
descriptors. For device MSI the storage is device-specific, possibly in guest
RAM. Only the driver's write_msi_msg callback can update it, and it must happen
before the device raises interrupts on the destination. Options:

-   INVALIDATE + re-CREATE + rewrite. The core walks each served domain's
    descriptors, recomposes and calls irq_chip_write_msi_msg(), which reaches
    the driver callback. That is a new requirement on device MSI drivers: the
    callback must work outside the normal activation path, and the driver must
    tolerate it while queues are live or quiesced.
-   Host preserves the messages. This is impractical with Intel's global IRTE
    table. It may be feasible with AMD's per-device remap table, because indices
    are per device and the destination's table is empty.

Device-state migration (VFIO migration v2) must also be ordered with this: the
device must not resume before its stored messages are valid. The same path
handles host suspend/resume and control-device reset.

5.8 Points the proposal must settle
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1.  Scope: whole-function assignment only (5.3), x86 only. ARM64 with a vITS has
    an architectural path (doorbell mapping plus guest EventIDs installed in the
    physical ITT) and does not need a paravirtual message allocator.
2.  Host-side scale (5.5.3): without vector-less posted IRTEs and incremental
    KVM routing, device MSI in a guest is limited to roughly MSI-X scale.
3.  The write_msi_msg callback contract for migration and invalidation (5.7).
4.  Domain binding: a dedicated bridge (5.6(d)).
5.  Whether RETARGET stays a VMM register (simple, the same cost as today's
    trapped MSI-X write) or becomes a KVM-side operation (scales, new uAPI).

5.9 Prototype order
~~~~~~~~~~~~~~~~~~~

1.  Host: storage-less IR-backed IRQs in VFIO plus message readback. Verify that
    posting through kvm_pi_update_irte works on them.
2.  Guest: virtio-msi parent with MSI-X only (index-bound, guest-written, VMM
    table trap disabled). Verify that the written value equals the host's
    composed message, and that affinity changes cause RETARGET without any
    device rewrite.
3.  Device MSI: add the second per-device domain slot and write
    pci_device_msi_init_and_alloc_irqs(), then allocate a device MSI domain next
    to MSI-X on a test device, or on an emulated device with its own message
    storage in QEMU backed by a real VFIO function.
4.  Failure injection (CREATE/RETARGET), CPU hot-unplug, FLR, guest driver
    unbind with live interrupts.
5.  INVALIDATE/rewrite, then live migration.
6.  Scale tests at thousands of device MSI vectors per guest.

6. Alternative: mirror the guest's interrupt remapping 1:1
----------------------------------------------------------

6.1 The idea
~~~~~~~~~~~~

Sections 4–5 make the host choose the message and teach the guest to ask for it.
The alternative turns this around: the guest chooses the message with its own,
unmodified interrupt remapping driver, and the host makes the physical remapping
hardware match.

With a vIOMMU that does interrupt remapping (VT-d IR, AMD IR), or a vITS on ARM,
a guest driver's device MSI domain is parented on the guest's own remapping
domain. The message it composes is already in remappable form: an IRTE index
(x86) or an EventID (ITS). It names an entry in a guest-owned table, not a
destination. If the host installs, at the same index, a physical entry for the
same source device that delivers to the destination the guest's entry names,
then the guest-written message works on real hardware unchanged. The device
storage needs no trap and no host-provided message, so virtio-msi is
unnecessary.

The destination translation already exists. KVM's posted-interrupt and GICv4
paths turn a guest-format destination into a physical IRTE or vLPI mapping
(5.2). The new requirement is that the host index is chosen by the guest rather
than by the host allocator.

    guest driver:  device MSI domain -> guest IR domain (vIOMMU) -> vector domain
                   composes {IRTE index i}, writes it into device storage
    guest IR:      writes vIRTE[i] = (vBDF, APIC, vector); invalidates index i
    VMM:           traps the invalidation, reads vIRTE[i], maps vBDF -> VFIO device
    host:          allocates a storage-less host IRQ at IR index i for that RID,
                   KVM routing gsi -> (APIC, vector), irqfd, posting as today
    device:        emits {index i} with its RID -> physical IRTE[i] -> vCPU

Retargeting needs nothing new. A guest affinity change rewrites vIRTE[i] and
invalidates it. The VMM updates the KVM route, and kvm_pi_update_irte() rewrites
the physical IRTE. The index, and so the device storage, never changes. That is
the stability property section 5.4 relied on, obtained here without a protocol.

What the host needs in every variant:

-   Storage-less host IRQs (5.5.1), with the extra ability to allocate at a
    caller-specified remapping index or EventID. Today alloc_irte() (Intel),
    alloc_irq_index() (AMD) and its_alloc_device_irq() (ITS) all pick the first
    free slot.
-   A uAPI to request that index, e.g. VFIO "allocate a storage-less IRQ on this
    device at remapping index i", with failure reported if i is not available.
-   A guest that can parent a device MSI domain on its vIOMMU IR domain. That is
    the same x86 gap as on the host (DMAR/AMDVI accepting
    DOMAIN_BUS_DEVICE_MSI), plus the new pci_device_msi_init_and_alloc_irqs()
    (5.1). No paravirtual driver is needed, apart from VT-d's index allocation
    callback (6.2).

Whether the scheme is viable depends entirely on whether the guest's index space
can be made to coincide with a host index space that the host can dedicate to
this device. That differs sharply by architecture.

6.2 Intel VT-d: one global index space per IOMMU
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

How the VMM learns the guest's entry. The guest writes IRTEs into a table in its
own memory, then issues an interrupt entry cache (IEC) invalidation through the
queued-invalidation interface (modify_irte() → qi_flush_iec(), index-selective).
QEMU already traps this. vtd_iec_notify_all() fans it out to IEC notifiers, and
today kvm_update_msi_routes_all() uses it to refresh KVM routes for trapped
MSI/MSI-X. A mirror would add a notifier that, for index i, reads vIRTE[i], uses
its SID (Linux always programs source-ID verification, set_msi_sid()) to find
the VFIO device, and allocates or updates the host IRQ at index i. The guest's
invalidation waits on a QI wait descriptor, so the host update completes before
the guest goes on to write device storage. Ordering is correct for free.

The problem: index collisions. A VT-d message carries a 16-bit handle into the
IOMMU's single interrupt remapping table. There is no per-device or per-domain
table and no offset. The physical table on that IOMMU is shared by the host's
own interrupts, by every other assigned device behind it, and by every other
guest. The guest's IR driver allocates from index 0 upward
(bitmap_find_free_region()), exactly as the host's does and as every other
guest's does. Guest index i will often already be taken on the host.

VT-d gives the vIOMMU no way to tell the guest "that index is busy". IEC
invalidation cannot fail. The best the VMM can do is leave the interrupt dead
and perhaps log a fault, which is the silent-failure mode section 4 criticised
in Hyper-V.

The fix: a paravirtual index allocation callback. The guest must not pick the
index itself. It asks the host. The vIOMMU advertises a PV capability, and the
guest's alloc_irte() calls it instead of searching its own bitmap:

    guest alloc_irte(SID, count)
       -> PV ALLOC(SID, count)         trapped vIOMMU register (or hypercall)
       host: find a free, count-aligned block on the physical IOMMU behind the
             device with that SID; reserve it for that RID; return index i
       guest marks i..i+count-1 used in its bitmap, writes vIRTE[i], flushes IEC
       -> existing IEC trap: VMM reads vIRTE[i], programs physical IRTE[i]
    guest clear_entries(i)
       -> PV FREE(i)                   host releases the reservation

-   Only allocation is paravirtual. The IRTE format, the remappable message, IEC
    invalidation, retargeting and posting all stay architectural VT-d, and they
    reuse the QEMU trap points above. This is far thinner than virtio-msi: there
    is no host-composed message, no CREATE/RETARGET protocol and no new guest
    IRQ domain. The change is a hook in the guest's Intel IR driver.
-   Allocation can fail cleanly. The PV call returns an error, so alloc_irte()
    fails and the IRQ allocation unwinds. That is the failure path IEC
    invalidation lacks.
-   It must be synchronous and cheap. alloc_irte() runs under irq_2_ir_lock (a
    raw spinlock) during IRQ domain allocation. A trapped vIOMMU register write
    with status readback fits that context; a virtqueue does not. Alternatively
    the guest calls out before taking the lock.
-   All guest allocations go through it. Indices for emulated devices need no
    physical IRTE, but they must not collide with indices the host later hands
    out for assigned devices in the same vIR table. Letting the host hand out
    every index keeps the guest's table and the reservations consistent. For an
    emulated SID the host just allocates from a per-VM bitmap without touching
    hardware.
-   MSI-X can use the same path. If the host IRQ for MSI-X entry k is allocated
    at the guest-obtained index, the guest-written MSI-X message equals the
    host's, as in the AMD case (6.3).

Alternatives considered, all worse:

-   One mirroring guest per physical IOMMU. The host allocates its own IRTEs
    from the top of the table, and one guest gets the low indices. This is
    deterministic but unscalable, and it depends on the platform's DRHD
    topology.
-   Pre-reserve a static window. Boot the vIOMMU with IR pre-enabled and every
    vIRTE outside a host-reserved window marked present, so that the guest's
    iommu_load_old_irte() avoids them. That path runs only in a kdump kernel, so
    it still needs a guest change, and it fixes the window size at boot.
-   Host IRTE window per guest, plus a fixed offset. Not possible: the device
    emits the guest's index verbatim, and VT-d has no per-source index offset.

Smaller issues:

-   Multi-vector blocks: alloc_irte() aligns count to a power of two and uses
    the handle mask in invalidations. The host must reserve the same aligned
    block.

-   The guest may use a global IEC invalidation (also issued when IRTA is
    written). The VMM then has to rescan up to 64K vIRTEs.

-   Scale is unchanged from 5.5.3: each mirrored entry is still a host IRQ, a
    host vector and a KVM route.

-   Migration: the indices are guest state (in vIRTEs and in device storage).
    The destination host must reserve the same indices on its physical IOMMU,
    and with a global table they may already be in use there. Fixing that needs
    either a per-VM index range coordinated across the migration pool, or a host
    → guest "reallocate" event that makes the guest rewrite device storage. The
    latter is the 5.7 problem again.

Verdict: VT-d needs one PV hook, an index allocation callback in the guest's IR
driver, because its handle space is global. With that hook, the rest of the
mirroring scheme is architectural and reuses existing QEMU and KVM paths.
Migration remains the weak point.

6.3 AMD-Vi: per-device index space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

AMD remaps through a per-DeviceID interrupt table referenced from the DTE (512
entries, or 2048 with MAX_IRQS_PER_TABLE_2K). The message's data field carries
the index, and the IOMMU looks it up in the table belonging to the requester's
DeviceID. QEMU's AMD vIOMMU likewise gives the guest a per-device table and
traps INVALIDATE_INTERRUPT_TABLE (AMDVI_CMD_INVAL_INTR_TABLE), calling the same
IEC notifier chain.

For whole-function assignment (5.3) the physical table belongs to that one
device. No other device, guest or host consumer shares it. The guest's index i
and the host's index i live in the same private namespace, so the collision
problem from 6.2 does not arise for device MSI.

The one sharer is the host itself, for the same device's MSI-X: VFIO allocates
host MSI-X IRTEs in the same per-device table, at indices chosen by
alloc_irq_index(). The guest's IR driver also allocates its MSI-X vIRTEs in its
per-device table. Two consistent options:

-   Mirror everything. When the guest writes a remappable MSI-X message (data =
    vIRTE index i), VFIO allocates the host MSI-X IRQ at IRTE i too. The
    host-composed message then equals the guest-written one, and MSI-X table
    trapping is no longer needed for correctness either.
-   Partition. The host keeps MSI-X in a range the guest's allocator will not
    use. The guest allocator, like Intel's, gives no such guarantee without a
    hint, so this is the weaker option.

Caveats:

-   The table is per DeviceID, which is the alias after DMA aliasing. A function
    behind a PCIe-to-PCI bridge shares its table with its siblings, so only
    functions with their own DeviceID qualify. This is the same condition as
    5.3.
-   Capacity is 512 or 2048 entries per device. That is enough for many
    device-MSI uses, though well short of the "thousands per device" goal.
-   The AMD vIOMMU in QEMU must emulate interrupt remapping completely enough
    for the guest to use it (including x2APIC and the table-length encoding),
    and the guest must enable IR on it.

Verdict: AMD-Vi fits the mirroring model well. The host work is index-pinned,
storage-less host IRQs plus a VMM notifier. No guest PV interface is needed.

6.4 ARM GICv3/v4 ITS: per-device EventID space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ITS is architecturally the closest match. A device's MSI is a write of an
EventID to the ITS doorbell (GITS_TRANSLATER). The DeviceID comes from the bus
(RID → StreamID → DeviceID through IORT / msi-map), not from the message.
Translation is per DeviceID: a device table entry points to that device's ITT,
indexed by EventID. EventIDs are therefore a private namespace per device, like
AMD-Vi and unlike VT-d.

The guest drives a vITS. It issues MAPD (virtual DeviceID → ITT) and MAPTI
(DeviceID, EventID → vLPI, collection), then writes the EventID into device
storage. For that write to reach the guest on real hardware:

1.  EventID. The host ITT for the physical DeviceID must contain the guest's
    EventID e, mapped to a host LPI, and with GICv4 forwarded (VMAPTI) to the
    guest's vLPI. The host ITS allocates events with its_alloc_device_irq(), the
    first free region of lpi_map. It would need to allocate at a requested
    EventID. The obvious rule for the device MSI domain is EventID = msi_index,
    which VFIO sets to e.
2.  ITT sizing. its_msi_prepare() reuses an existing its_device for the DeviceID
    ("shared") and keeps the ITT size from first creation. If MSI-X created the
    ITT with 32 events, a device MSI domain on the same DeviceID cannot place
    events beyond that. The ITT has to be sized for the whole guest-visible
    EventID range at the first allocation, or be grown.
3.  Doorbell address. The device writes to the guest's vITS doorbell address.
    That is a GPA (no vSMMU), or an IOVA the guest's stage 1 maps to that GPA
    (nested vSMMU). Host stage 2 must map that GPA to the physical
    GITS_TRANSLATER page of the ITS serving the device. Today the host maps the
    physical doorbell at the reserved SW_MSI window (MSI_IOVA_BASE) that the
    host itself chose (iommufd_sw_msi), and the guest's vITS doorbell sits
    elsewhere. One of two things is needed: either the VMM places the vITS
    doorbell where the host maps the physical one, or there is a new uAPI to map
    the physical doorbell at a VMM-chosen IPA. iommufd has neither today.
4.  Learning the guest's events. With the in-kernel vITS, MAPTI is handled by
    KVM (vgic_its_cmd_handle_mapi()), not QEMU. KVM already resolves (DeviceID,
    EventID) → vLPI and calls its_map_vlpi() when an irqfd route is attached
    (kvm_vgic_v4_set_forwarding()). But it does that lazily: forwarding happens
    at route time, and MAPTI only creates the vLPI. Two options:
    -   The VMM pre-allocates host events 0..N-1 for the device, with routes for
        each (DeviceID, e). This is a quota-sized upfront cost. KVM then
        forwards whichever ones the guest maps.
    -   KVM (or a KVM → VMM exit on MAPTI/DISCARD for assigned DeviceIDs) drives
        host event allocation on demand. That is new KVM ↔ VFIO plumbing.
5.  Retarget is MOVI/VMOVI, already handled by KVM for forwarded vLPIs. The
    EventID, and so device storage, never changes.

Security matches 5.3. The guest can only name EventIDs in its own device's ITT,
and the DeviceID is fixed by the hardware path. A shared DeviceID (aliasing, or
SIOV-style sharing) breaks it the same way.

Verdict: the ITS model supports mirroring without collisions. The work lies in
the host ITS driver (allocating at an EventID, ITT sizing), in the doorbell
mapping uAPI, and in how KVM/VMM learn guest events. As 5.8 noted, this is also
the native ARM answer that never needed a message allocator.

6.5 Comparison with virtio-msi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  --------------------------------------------------------------------------------------
              virtio-msi (5) Mirror on VT-d       Mirror on AMD-Vi  Mirror on ITS
  ----------- -------------- -------------------- ----------------- --------------------
  Guest PV    virtio-msi IRQ index alloc/free     none              none
  interface   domain +       callback in the IR                     
              protocol       driver                                 

  Who picks   host           host, via PV         guest             guest
  the index                  callback                               

  Collision   none           none: host allocates none: per-device  none: per-device ITT
  risk                       in the global table  table             

  Failure     CREATE status  PV ALLOC status      none, but cannot  none, but cannot
  reporting                                       collide           collide
  to guest                                                          

  Retarget    trapped        existing IEC trap →  existing trap →   existing KVM vITS
              register       KVM route            KVM route         

  Main new    storage-less   PV alloc/free,       index-pinned      EventID-pinned
  host work   IRQs, protocol index-pinned IRTEs,  IRTEs, QEMU       alloc, ITT sizing,
                             QEMU notifier        notifier          doorbell mapping,
                                                                    event discovery

  Migration   INVALIDATE +   destination must     destination       destination
              guest rewrite  reproduce indices:   reproduces        reproduces
              (5.7)          fragile with a       per-device        per-device EventIDs:
                             global table, even   indices: feasible feasible
                             with PV alloc                          
  --------------------------------------------------------------------------------------

Migration deserves emphasis. With mirroring, the guest's indices are part of the
guest state, held in guest memory and in device storage that only the guest
driver can rewrite. The destination host must recreate exactly those indices.
That works for per-device namespaces (AMD-Vi, ITS), because the destination
device's table is empty. With VT-d's global table it fails whenever the
destination IOMMU already uses one of them. This is the mirror image of 5.7,
where virtio-msi needs the guest to rewrite storage instead.

Overall: 1:1 mirroring removes the need for virtio-msi where the remapping index
space is per device: AMD-Vi and the ARM ITS. On Intel VT-d the global handle
space needs one PV hook, an index allocation callback in the guest's IR driver.
That is still much smaller than virtio-msi, because the message format,
retargeting and posting stay architectural. Migration on VT-d remains fragile
without a coordinated per-VM index range, a guest rewrite path, or a hardware
change (per-device or per-domain IR tables).

Source reading map
------------------

arch/x86/hyperv/hv_init.c

-   Functions: create_pci_msi_domain hook
-   What they establish: Root-partition provider selection

arch/x86/hyperv/irqdomain.c

-   Functions: hv_irq_compose_msi_msg, hv_map_interrupt,
    hv_build_devid_type_pci, entry_to_msi_msg, hv_teardown_msi_irq
-   What they establish: Hypercall output becomes the message; unmap+map on move

drivers/pci/controller/pci-hyperv.c

-   Functions: hv_pcie_init_irq_domain, hv_compose_msi_msg,
    hv_irq_retarget_interrupt, hv_arch_irq_unmask, hv_msi_free,
    hv_pci_vec_irq_domain_alloc (ARM64), hv_pci_restore_msi_state
-   What they establish: VMBus CREATE, hypercall RETARGET, per-bus binding, SPI
    model, resume

kernel/irq/chip.c

-   Functions: irq_chip_compose_msi_msg
-   What they establish: Finding the composer in the hierarchy

kernel/irq/msi.c

-   Functions: msi_domain_activate, msi_domain_set_affinity
-   What they establish: Compose → write; recompose on move unless OK_DONE

kernel/irq/migration.c

-   Functions: irq_move_masked_irq
-   What they establish: Deferred moves run in interrupt context

drivers/pci/msi/irqdomain.c

-   Functions: pci_msi_domain_write_msg
-   What they establish: Device domain write callback

drivers/pci/msi/msi.c

-   Functions: __pci_write_msi_msg
-   What they establish: Config-space / MSI-X table programming

drivers/vfio/pci/vfio_pci_intrs.c

-   Functions: vfio_msi_set_vector_signal, vfio_msi_alloc_irq
-   What they establish: Host IRQ per MSI-X index, message refresh, irq bypass
    producer

drivers/iommu/intel/irq_remapping.c

-   Functions: intel_ir_compose_msi_msg, fill_msi_msg, intel_ir_set_affinity,
    intel_ir_set_vcpu_affinity, set_msi_sid
-   What they establish: Stable remappable message; IRTE-only retarget;
    source-ID isolation

drivers/iommu/amd/iommu.c

-   Functions: amd_ir_set_affinity, amd_ir_set_vcpu_affinity
-   What they establish: Same properties on AMD

drivers/base/platform-msi.c

-   Functions: platform_device_msi_init_and_alloc_irqs, platform_msi_template,
    platform_device_msi_free_irqs_all
-   What they establish: Device MSI domain with a driver write callback; masking
    forwarded to the parent. Model for the new PCI helper

kernel/irq/msi.c, include/linux/msi_api.h

-   Functions: msi_create_device_irq_domain, msi_domain_ids
-   What they establish: Per-device domain parented on dev->msi.domain; only one
    slot per device today

drivers/irqchip/irq-msi-lib.c

-   Functions: msi_lib_init_dev_msi_info
-   What they establish: How non-x86 parents accept DOMAIN_BUS_DEVICE_MSI

arch/x86/kernel/apic/msi.c

-   Functions: x86_init_dev_msi_info, x86_msi_prepare
-   What they establish: x86 parents accept only PCI MSI/MSI-X device domains

arch/x86/kvm/irq.c

-   Functions: kvm_pi_update_irte, kvm_arch_update_irqfd_routing
-   What they establish: Guest routing change → posted IRTE update

drivers/iommu/intel/irq_remapping.c

-   Functions: alloc_irte, modify_irte, qi_flush_iec, iommu_load_old_irte,
    fill_msi_msg
-   What they establish: Global first-fit IRTE allocation; IEC invalidation per
    index; pre-enabled IR copy (kdump only)

drivers/iommu/amd/iommu.c

-   Functions: alloc_irq_index, irq_remapping_alloc
-   What they establish: Per-DeviceID interrupt table, 512/2048 entries

drivers/irqchip/irq-gic-v3-its.c

-   Functions: its_msi_prepare, its_alloc_device_irq, its_map_vlpi
-   What they establish: Per-DeviceID ITT, first-fit EventIDs, ITT size fixed at
    first use, GICv4 forwarding

arch/arm64/kvm/vgic/vgic-its.c, arch/arm64/kvm/vgic/vgic-v4.c

-   Functions: vgic_its_cmd_handle_mapi, kvm_vgic_v4_set_forwarding
-   What they establish: In-kernel vITS commands; vLPI forwarding set up at
    irqfd route time

QEMU hw/i386/intel_iommu.c, hw/i386/amd_iommu.c, target/i386/kvm/kvm.c

-   Functions: vtd_iec_notify_all, AMDVI_CMD_INVAL_INTR_TABLE,
    kvm_update_msi_routes_all
-   What they establish: VMM trap points for guest IRTE changes

Re: [PATCH 4/7] ath11k: add PCIe link recovery retry

[Bjorn Helgaas]

On Wed, Sep 30, 2026 at 04:08:30PM +0200, Jose Ignacio Tornos Martinez wrote:
> When ath11k devices are passed through to VMs via PCIe passthrough,
> the PCIe link may not stabilize immediately after global reset due to
> virtualization layer timing variations. This causes MHI initialization
> to fail with "BHI offset: 0xffffffff is out of range" errors because
> the device is read before it's ready.
> 
> Currently, ath11k_pci_soc_global_reset() just warns if the link is
> down but continues anyway, leading to inevitable MHI failures.
> 
> This patch adds:
> 1. A retry loop (up to 5000ms) to wait for the PCIe link to recover
>    after global reset (similar to the existing ath12k LTSSM retry logic)
> 2. Proper error propagation to prevent continuing with a dead link
> 
> This is especially important for VM environments where timing can vary
> significantly from bare metal.
> 
> Signed-off-by: Jose Ignacio Tornos Martinez <jtornosm@redhat.com>
> ---
>  drivers/net/wireless/ath/ath11k/pci.c | 39 ++++++++++++++++++++++-----
>  1 file changed, 32 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
> index 7114eca8810d..4fc6aaf036c1 100644
> --- a/drivers/net/wireless/ath/ath11k/pci.c
> +++ b/drivers/net/wireless/ath/ath11k/pci.c
> @@ -190,8 +190,10 @@ static void ath11k_pci_restore_window(struct ath11k_base *ab)
>  	spin_unlock_bh(&ab_pci->window_lock);
>  }
>  
> -static void ath11k_pci_soc_global_reset(struct ath11k_base *ab)
> +static int ath11k_pci_soc_global_reset(struct ath11k_base *ab)
>  {
> +	unsigned long timeout;
> +	bool link_recovered = false;
>  	u32 val, delay;
>  
>  	val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
> @@ -211,14 +213,29 @@ static void ath11k_pci_soc_global_reset(struct ath11k_base *ab)
>  
>  	mdelay(delay);
>  
> -	val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
> -	if (val == 0xffffffff)
> -		ath11k_warn(ab, "link down error during global reset\n");
> +	/* Wait for PCIe link to recover, especially important in VM environments
> +	 * where timing can vary significantly from bare metal.
> +	 */
> +	timeout = jiffies + msecs_to_jiffies(5000);
> +	while (time_before(jiffies, timeout)) {
> +		val = ath11k_pcic_read32(ab, PCIE_SOC_GLOBAL_RESET);
> +		if (val != 0xffffffff) {
> +			link_recovered = true;
> +			break;
> +		}
> +		mdelay(20);
> +	}
> +	if (!link_recovered) {
> +		ath11k_err(ab, "PCIe link failed to recover - device may need power cycle\n");
> +		return -ETIMEDOUT;
> +	}

Link down is a PCIe reset of the endpoint, which should clear all
PCIe-specific state, e.g., BAR values.  Is that restored somewhere?

>  	/* Restore window register as its content is cleared during
>  	 * hardware global reset, such that it aligns with host cache.
>  	 */
>  	ath11k_pci_restore_window(ab);
> +
> +	return 0;
>  }
>  
>  static void ath11k_pci_clear_dbg_registers(struct ath11k_base *ab)
> @@ -373,8 +390,10 @@ static void ath11k_pci_force_wake(struct ath11k_base *ab)
>  	mdelay(5);
>  }
>  
> -static void ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
> +static int ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
>  {
> +	int ret;
> +
>  	mdelay(100);
>  
>  	if (power_on) {
> @@ -387,8 +406,12 @@ static void ath11k_pci_sw_reset(struct ath11k_base *ab, bool power_on)
>  
>  	ath11k_mhi_clear_vector(ab);
>  	ath11k_pci_clear_dbg_registers(ab);
> -	ath11k_pci_soc_global_reset(ab);
> +	ret = ath11k_pci_soc_global_reset(ab);
> +	if (ret)
> +		return ret;
>  	ath11k_mhi_set_mhictrl_reset(ab);
> +
> +	return 0;
>  }
>  
>  static void ath11k_pci_init_qmi_ce_config(struct ath11k_base *ab)
> @@ -818,7 +841,9 @@ static int ath11k_pci_power_up(struct ath11k_base *ab)
>  
>  	ab_pci->register_window = 0;
>  	clear_bit(ATH11K_FLAG_DEVICE_INIT_DONE, &ab->dev_flags);
> -	ath11k_pci_sw_reset(ab_pci->ab, true);
> +	ret = ath11k_pci_sw_reset(ab_pci->ab, true);
> +	if (ret)
> +		return ret;
>  
>  	/* Disable ASPM during firmware download due to problems switching
>  	 * to AMSS state.
> -- 
> 2.53.0
> 

RE: [PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs

[Michael Kelley]

From: Jason Gunthorpe <jgg@ziepe.ca> Sent: Wednesday, October 14, 2026 8:01 AM
> 
> On Tue, Oct 13, 2026 at 05:42:25PM +0200, Jose Ignacio Tornos Martinez wrote:
> > Thank you for the direction on device MSI domains. The Hyper-V
> > reference is difficult for me to follow for this case. Could
> > you point me to some existing KVM/VFIO equivalent, or similar
> > example for my use case? Any pointers to relevant code,
> > documentation, or previous discussions on this topic would be
> > very helpful to get started in the right direction.
> >
> > Also, where would this need to be implemented? Kernel only
> > because it is already supported in QEMU in a general way, or
> > kernel and QEMU for some specifics?
> 
> I spent some time with AI to research this and drafted up some slop
> for you that outlines a bunch of ideas and explains how some of the
> bits work today. Not saying this is something to follow but it should
> be a good starting point to come up with a good idea.

This is a nice summary of what Hyper-V does. It looks right to me
based on my experience with that driver (pci-hyperv.c) over the
years.

I wrote the documentation in Documentation/virt/hyperv/vpci.rst.
With your permission, I'd like to adapt some of your material below
and add it to vpci.rst, primarily to provide some broader context
on the "why" and the implications of the overall approach. I'm
not sure how soon that will be, but it will go on my "to do" list.

Michael

> 
> It is quite interesting that amd virtual interrupt remapping in the
> iommu is very close to being able to solve the problem without any
> guest changes. Maybe that is the right thread to pull on, if AMD can
> be fixed fully then maybe there is an option to negotiate something
> within the Intel spec, or build a similar idea with a 'virtio
> interrupt remapping'
> 
> Nevertheless this looks reasonably tractable to me, assuming you have
> some strong AI to help understand the whole picture and baroque
> hardware details it is even almost approachable..
> 
> Jason
> 
> Host-managed MSI: the Hyper-V flow and a virtio proposal
> ========================================================
> 
> Goal: make device-specific MSI storage work for devices assigned to KVM guests.
> Such storage is outside the PCI MSI/MSI-X capability. In Linux it is driven
> through a per-device MSI domain, today via
> platform_device_msi_init_and_alloc_irqs(). A PCI device needs a PCI counterpart
> of that helper, which does not exist yet and must be written. The VMM cannot
> trap this storage, so the guest must write host-valid MSI messages into it
> directly, as Hyper-V guests do for MSI-X. The guest obtains those messages, and
> routes the interrupts, through a paravirtual (virtio) interface.
> 
> Sections 1–4 describe how Hyper-V does this through the Linux IRQ domain
> hierarchy. Section 5 maps that onto the KVM/VFIO host stack and assesses the
> virtio proposal. Section 6 looks at an alternative with no paravirtual
> interface: the host mirrors the guest's vIOMMU interrupt remapping (or vITS)
> entries 1:1.
> 
> 0. Why host-managed MSI exists at all
> -------------------------------------
> 
> On a normal VFIO/KVM system the VMM traps guest writes to a passthrough device's
> MSI/MSI-X registers. The guest writes a virtual message (a guest APIC address
> and a guest vector). The VMM then programs the physical device and the physical
> interrupt remapping (IR) table on the guest's behalf. The guest never sees a
> host-valid message.
> 
> Hyper-V does not do that. In the vPCI and root-partition cases, the guest writes
> the device's MSI capability and MSI-X table directly. Writes are not trapped and
> emulated (see the comment above hv_pci_restore_msi_state()). The IOMMU and its
> interrupt remapping table are owned by the hypervisor. The value the guest
> writes must therefore already be a host-valid, remapped message, and only the
> host can produce one. Hyper-V has the guest ask the host for each message.
> 
> Every design decision below follows from that premise:
> 
>   The guest chooses a destination (vCPU + vector, or a guest interrupt line).
>   The host allocates a remapping entry for (source device → destination) and
>   returns the MSI address/data that selects it. A Linux IRQ domain hands that
>   pair to the MSI core, which writes it into the device using the normal PCI MSI
>   writer.
> 
> For device-specific storage on KVM there is no trap to remove: the storage was
> never trappable. Adopting this premise is the only way it can work in a guest.
> Section 5.2 shows that the KVM half (stable host-composed messages, IRTE-only
> retargeting, posting) already exists. The missing pieces are host IRQs without
> host-side storage in VFIO, and an MSI parent in the guest that serves both the
> PCI MSI-X domain and per-device MSI domains.
> 
> 1. Separate the destination from the message
> --------------------------------------------
> 
>   -------------------------------------------------------------------------------
>   Identity                Meaning
>   ----------------------- -------------------------------------------------------
>   Linux IRQ number (virq) The guest kernel's software handle. Never visible to
>                           the host.
> 
>   Destination             What the guest wants to receive: on x86 a vCPU set +
>                           IDT vector; on Hyper-V ARM64 a GIC SPI number
> 
>   Source identity         The requester whose MSI writes the host will accept for
>                           this mapping
> 
>   Host mapping            Host state binding (source, message) → destination; on
>                           hardware, an IR table entry
> 
>   MSI address/data        What is written into the device; on IR hardware this
>                           effectively selects the host mapping
>   -------------------------------------------------------------------------------
> 
> In a hierarchical domain, one Linux IRQ has an irq_data at each level, linked by
> parent_data, and each level keeps private state in chip_data. For host-managed
> MSI:
> 
> -   the parent (receive) domain allocates the guest destination;
> -   the host-managed MSI parent domain obtains the message from the host and
>     owns the mapping's lifetime;
> -   the device MSI domain writes the message into the device.
> 
> The guest must treat the returned pair as host-defined. It must not reconstruct
> an APIC-format address from the CPU number, substitute its own vector into the
> data, or assume an ITS doorbell. The only structure the guest may rely on is
> what the protocol explicitly defines. Hyper-V's one example is conventional
> multi-message MSI, where the data of vector n is base data + n (section 4).
> 
> 2. The common Linux path: compose, then write
> ---------------------------------------------
> 
> Root-partition hierarchy on x86:
> 
>     PCI device MSI/MSI-X domain     writes device registers / MSI-X table
>                  |
>     Hyper-V MSI parent domain       obtains message from host, owns mapping
>                  |
>     x86 vector domain               allocates guest CPU + vector
> 
> Allocation walks down the hierarchy to reserve receive resources. Activation
> walks parents first, then the MSI device domain's msi_domain_activate() does:
> 
>     msi_domain_activate(device_domain, irq_data)
>       +-- irq_chip_compose_msi_msg(irq_data, msg)
>       |     walks irq_data -> parent_data to the first level with
>       |     .irq_compose_msi_msg  -> Hyper-V callback returns host pair
>       +-- irq_chip_write_msi_msg(irq_data, msg)
>             -> pci_msi_domain_write_msg() -> __pci_write_msi_msg()
>                  MSI:   config-space address/data registers
>                  MSI-X: the entry in the device's MSI-X table
> 
> Notes:
> 
> -   The Hyper-V compose callback never touches device registers; the write
>     belongs to the PCI MSI layer.
> -   __pci_write_msi_msg() caches the message in msi_desc->msg. If the device is
>     not in D0 or is disconnected, it skips the hardware write.
> -   irq_compose_msi_msg returns void. msi_domain_activate() and
>     msi_domain_set_affinity() start from a zeroed msg. If the provider fails, it
>     simply leaves (or sets) 0/0, and a zero message is written to the device.
>     Nothing reports the failure to the caller.
> -   msi_domain_set_affinity() asks the parent to change destination. Unless the
>     parent returns IRQ_SET_MASK_OK_DONE, it composes and writes again. Every
>     affinity change therefore re-runs the Hyper-V compose callback.
> -   Both Hyper-V x86 chips set IRQCHIP_MOVE_DEFERRED. On x86 the affinity change
>     is applied from the interrupt's own handler context (irq_move_masked_irq()).
>     Any host interaction done in compose therefore runs in hard-IRQ context with
>     the descriptor lock held.
> 
> 3. Hyper-V root partition: direct hypercalls
> --------------------------------------------
> 
> Code: arch/x86/hyperv/irqdomain.c.
> 
> Here Linux is the root partition. It sees the physical PCI topology, but the
> hypervisor owns the IOMMU.
> 
> Installing the domain
> ~~~~~~~~~~~~~~~~~~~~~
> 
> hv_init.c sets x86_init.irqs.create_pci_msi_domain = hv_create_pci_msi_domain
> when running as root. That creates a single system-wide MSI parent domain over
> x86_vector_domain. hv_msi_domain_alloc() only allocates parent vectors and
> installs hv_pci_msi_controller. It does no host work; a TODO says the mapping
> should move there.
> 
> Supported flags are generic + MSI-X. Conventional multi-message MSI is not
> supported, and each mapping has vector_count = 1, edge-triggered.
> 
> Mapping: all work happens in compose
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>     hv_irq_compose_msi_msg(irqd, msg)
>       if irqd->chip_data:                  # already mapped (affinity change)
>           HVCALL_UNMAP_DEVICE_INTERRUPT(old entry); kfree
>       entry = kzalloc(GFP_ATOMIC)
>       hv_map_msi_interrupt()
>          source = physical segment/BDF, after walking DMA aliases;
>                   PCI-X bridges add a bus-range "shadow"
>          cpu    = first CPU of the effective affinity (single CPU)
>          vector = parent vector (irqd_cfg()->vector)
>          HVCALL_MAP_DEVICE_INTERRUPT  -> hv_interrupt_entry
>       irqd->chip_data = entry
>       msg = { address_hi = 0,
>               address_lo = entry.msi_entry.address,
>               data       = entry.msi_entry.data }
> 
> The hypercall returns the 32-bit address and 32-bit data. Linux copies them
> verbatim. address_hi = 0 is a property of this interface, not a guest encoding
> choice.
> 
> Affinity change
> ~~~~~~~~~~~~~~~
> 
> The hypervisor does not let the root partition change the vector or move the
> interrupt to VPs outside the originally mapped set. For that reason, root does
> not use RETARGET. Every affinity change is unmap, then map, followed by a device
> rewrite with the possibly new pair.
> 
> This has consequences:
> 
> -   Between the unmap and the device rewrite, the device holds a message that
>     refers to a released mapping.
> -   If the map fails, compose returns with chip_data == NULL and a zero msg, and
>     the device is programmed with 0/0. The IRQ is now dead and nothing reports
>     it.
> -   All of this runs in the deferred-move path, in hard-IRQ context (section 2).
> 
> Teardown
> ~~~~~~~~
> 
> hv_msi_domain_free() → hv_teardown_msi_irq() issues
> HVCALL_UNMAP_DEVICE_INTERRUPT with the stored entry. The mapping is a host
> resource; forgetting the pair in the guest leaks it.
> 
> 4. Hyper-V guest vPCI: CREATE over VMBus, RETARGET by hypercall
> ---------------------------------------------------------------
> 
> Code: drivers/pci/controller/pci-hyperv.c.
> 
> Here Linux is an ordinary guest with passthrough devices exposed on a synthetic
> PCI bus. Each vPCI bus is a VMBus device with its own channel.
> 
> Two control paths
> ~~~~~~~~~~~~~~~~~
> 
> This split is the most important thing to understand before designing a virtio
> version:
> 
> CREATE (PCI_CREATE_INTERRUPT_MESSAGE{,2,3})
> 
> -   Transport: VMBus request/response
> -   Called from: hv_compose_msi_msg()
> -   Context: IRQ descriptor lock held; polls the channel
> 
> DELETE (PCI_DELETE_INTERRUPT_MESSAGE)
> 
> -   Transport: VMBus, fire-and-forget
> -   Called from: compose (replace) and hv_msi_free()
> 
> RETARGET (x86 only)
> 
> -   Transport: HVCALL_RETARGET_INTERRUPT, synchronous hypercall
> -   Called from: hv_irq_unmask()
> -   Context: atomic
> 
> Slow, rare operations use the message transport. The operation that must
> complete inside an IRQ chip callback uses a synchronous trap. VMBus is not used
> for that.
> 
> Domain setup
> ~~~~~~~~~~~~
> 
> hv_pcie_init_irq_domain() creates one MSI parent domain per vPCI bus, and
> dev_set_msi_domain() binds it to that bus's host bridge. Device-to-domain
> selection is therefore by topology: every device on that synthetic bus uses that
> domain. Source identity is likewise implicit. The host already knows which
> physical function sits behind each slot (wslot) on that channel.
> 
> Parent receive domain:
> 
> -   x86: x86_vector_domain. Destination = vCPU(s) + IDT vector.
> -   ARM64: a private hv_vpci_arm64 domain over the GIC. It hands out SPIs
>     64–1019 from a bitmap in power-of-two aligned blocks. Destination = SPI
>     number, sent to the host as the "vector". CPU routing of that SPI is done by
>     the guest's virtual GIC distributor, which Hyper-V already emulates. So
>     hv_arch_irq_unmask() is empty on ARM64, and no retarget is ever needed.
> 
> CREATE
> ~~~~~~
> 
> hv_compose_msi_msg():
> 
> 1.  For multi-MSI with existing chip_data, return the cached pair (no host
>     call).
> 2.  If a previous int_desc exists (single MSI / MSI-X), send DELETE for it.
> 3.  Build a v1/v2/v3 CREATE request: slot, vector, vector count, and one "dummy"
>     vCPU (first online CPU of the affinity; round-robin for multi-MSI).
> 4.  Send it with vmbus_sendpacket_getid(). Disable the channel tasklet. Spin
>     calling hv_pci_onchannelcallback() with udelay(100) until the completion
>     fires, the device disappears (vendor ID 0xFFFF), or the channel is torn
>     down.
> 5.  Store the returned tran_int_desc in chip_data. Return
>     address_hi:lo = int_desc.address (64-bit) and data = int_desc.data.
> 
> On any failure it returns a zero message.
> 
> The vCPU in CREATE is not cosmetic. Hyper-V picks the physical CPU that services
> the interrupt based on it, and (as of the comment in the driver) RETARGET does
> not reselect the pCPU. The vector in CREATE is the real vector for single
> MSI/MSI-X, but a dummy 32 for multi-MSI.
> 
> RETARGET at unmask (x86)
> ~~~~~~~~~~~~~~~~~~~~~~~~
> 
> hv_irq_unmask() → hv_arch_irq_unmask() → hv_irq_retarget_interrupt(). This
> issues HVCALL_RETARGET_INTERRUPT with:
> 
> -   device id: built from the vPCI bus instance GUID + PCI function;
> -   the existing message (address_lo, data) used as the key that identifies the
>     mapping;
> -   the new vector and the full effective-affinity vCPU set.
> 
> The message is unchanged; the host updates its IR entry behind it. Note that
> Hyper-V uses the address/data pair itself as the mapping handle. There is no
> separate handle.
> 
> Nested root (Linux is root under a nested hypervisor) calls
> hv_map_msi_interrupt(data, NULL) instead and discards the returned entry.
> 
> Affinity in practice (x86)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> An affinity change goes through msi_domain_set_affinity() → recompose. For
> single MSI/MSI-X that means DELETE + CREATE over VMBus, polled in hard-IRQ
> context, then a device rewrite, then RETARGET on the next unmask. Multi-MSI
> skips the recompose (cached) and relies on RETARGET alone. All vectors of a
> multi-MSI device therefore share whatever pCPU the host picked at CREATE.
> 
> Multi-message MSI
> ~~~~~~~~~~~~~~~~~
> 
> Conventional MSI has one address and a base data value; the device ORs in the
> vector index. The host allocates an aligned block for nvec_used vectors during
> the first IRQ's CREATE. Later IRQs derive data = base + (irq - first_irq)
> without asking the host. This works only because the host guarantees block
> semantics. Independent per-vector pairs cannot be advertised as multi-MSI.
> 
> Free and resume
> ~~~~~~~~~~~~~~~
> 
> hv_msi_free() sends DELETE for the stored descriptor. On resume, a cached MSI-X
> table replay is useless because the host's IR entries are gone.
> hv_pci_restore_msi_state() therefore calls hv_compose_msi_msg() for every
> descriptor to re-CREATE.
> 
> What the Hyper-V code itself admits is wrong
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> -   Both providers have a TODO to move host allocation out of compose and into
>     .alloc.
> -   Failure in compose silently programs 0/0.
> -   vPCI CREATE runs a polled, unbounded message round-trip under IRQ locks,
>     including from the deferred affinity-move path.
> 
> 5. Applying this to KVM/VFIO: device MSI storage in a guest
> -----------------------------------------------------------
> 
> 5.1 The functional problem
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Many devices keep MSI address/data in device-specific storage: a context table
> in device memory, or a field in a queue context that lives in DMA'd host memory.
> The PCI MSI/MSI-X capability is not involved. (The PCI/IMS proposal used to
> cover this; the current tree does it with per-device MSI domains, see below.) A
> guest driver for an assigned device has to write a message into that storage,
> and:
> 
> -   the VMM cannot trap it. The format is device-specific, and the storage may
>     be in guest RAM that the device reads by DMA;
> -   a guest-format message (guest APIC ID + guest vector) is wrong. On a host
>     with interrupt remapping, the device must emit a remappable-format message
>     that selects a host IRTE belonging to this device.
> 
> So device-specific MSI storage cannot work in a KVM guest today. The only fix is
> the Hyper-V model: the guest asks the host for a host-valid message and writes
> it itself. This is "Model B": a host-provided message that the guest programs
> into the storage.
> 
> The generic Linux side already exists.
> platform_device_msi_init_and_alloc_irqs(dev, nvec, write_msi_msg) creates a
> DOMAIN_BUS_DEVICE_MSI domain on dev, parented on dev->msi.domain, and
> allocates
> nvec interrupts. The parent composes each message and the core hands it to the
> driver's write_msi_msg(desc, msg), which stores it wherever the device wants it.
> ARM ITS/MBI and the irq-msi-lib parents (e.g. RISC-V IMSIC) accept this bus
> token. The x86 parents (x86_vector_msi_parent_ops, DMAR, AMDVI) accept only
> DOMAIN_BUS_PCI_DEVICE_MSI{,X}, which is one of the gaps this work fills.
> 
> Two properties of the helper shape the design:
> 
> -   The platform helper is the wrong API for a PCI device. It is built for
>     devices whose only MSI domain is the device MSI domain. It always uses
>     MSI_DEFAULT_DOMAIN, and MSI_MAX_DEVICE_IRQDOMAINS is 1. A PCI function's
> own
>     MSI/MSI-X domain already occupies that slot. A PCI device therefore needs a
>     new pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) (and a
>     matching pci_device_msi_free_irqs_all()). Neither exists in the tree yet;
>     both must be written as part of this work. The new helper uses the same
>     DOMAIN_BUS_DEVICE_MSI template and driver write callback, but places the
>     domain in a second per-device domain slot that the core must add, so it
>     coexists with MSI-X on the same struct device. Both domains are then
>     parented on the function's dev->msi.domain, and the source identity is
>     simply the function itself.
> -   The device MSI chip has no mask of its own (irq_chip_mask_parent), no
>     affinity callback, and no feature flags. Masking and retargeting are the
>     parent's job, which suits a host-managed parent.
> 
> 5.2 What the KVM/VFIO host already provides
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>     VFIO_DEVICE_SET_IRQS(MSI-X, index i, eventfd)
>        host: pci_msix_alloc_irq_at(i) / request_irq(vfio_msihandler)
>        host IR domain composes a remappable msg (IRTE index [+ subhandle])
>        host writes that msg into the physical MSI-X entry i
>        irq_bypass_register_producer(eventfd, host_irq)
>     KVM_SET_GSI_ROUTING(gsi -> guest-format msg) + KVM_IRQFD(eventfd, gsi)
>        kvm_pi_update_irte(): post the IRTE to the vCPU PI descriptor + guest vector,
>        or leave it remapped to a host vector and let KVM inject
> 
> Properties to reuse:
> 
> -   The host message is stable for the life of the host IRQ.
>     intel_ir_compose_msi_msg() returns the cached ir_data->msi_entry. Host
>     affinity changes (intel_ir_set_affinity(), amd_ir_set_affinity()) rewrite
>     only the IRTE and return IRQ_SET_MASK_OK_DONE. Posting changes
>     (irq_set_vcpu_affinity()) also rewrite only the IRTE.
> -   Guest retargeting is host-only. It is a KVM routing update followed by an
>     IRTE rewrite. The device is never touched.
> 
> The KVM half (irqfd, GSI routing, posting) works on any host IRQ that has an IR
> parent. It does not care whether the host IRQ came from MSI-X or from somewhere
> else. The missing piece is entirely on the VFIO/host-IRQ side. VFIO can only
> create host IRQs for MSI-X table indices. A device MSI interrupt needs a host
> IRQ that has an IRTE but no host-side storage.
> 
> 5.3 Security: what makes guest-written messages safe
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The guest controls the device's DMA. It can already make the device write any
> value to 0xFEEx_xxxx with its requester ID. VFIO's isolation therefore already
> rests on interrupt remapping:
> 
> -   Intel: source-ID-checked IRTEs (set_msi_sid()), and compatibility-format
>     MSIs blocked;
> -   AMD: per-device remap tables.
> 
> Letting the guest write host-valid messages into device storage adds no new
> requirement, provided the whole requester ID belongs to that one guest. The
> guest can only reach IRTEs whose source-ID check passes for its own RID, and all
> of those are its own.
> 
> This breaks for shared-RID assignment (SIOV/ADIs). Several guests' ADIs emit
> MSIs with the same RID, and IR checks only the RID, not the PASID. A guest could
> put another guest's IRTE handle into its own storage entry and inject interrupts
> into that guest. That is why SIOV designs kept this storage host-programmed.
> Guest-written messages are safe only for whole-function assignment (a PF or a VF
> with its own RID). The proposal must state this scope, and the host must refuse
> CREATE for devices whose RID is shared.
> 
> 5.4 Protocol
> ~~~~~~~~~~~~
> 
>   ----------------------------------------------------------------------------------
>   Op           Request                     Response                    Transport
>   ------------ --------------------------- --------------------------- -------------
>   CREATE       device, domain (MSI-X /     per vector: handle, address virtqueue;
>                device MSI), index, count,  (64-bit), data (32-bit);    can sleep
>                storage address/data widths count granted
> 
>   RETARGET     handle, guest-format MSI    status; host message        synchronous
>                message composed by the     unchanged                   trapped
>                guest's parent vector                                   register
>                domain
> 
>   DESTROY      handle                      status; no further delivery virtqueue;
>                                                                        can sleep
> 
>   (host →      config-change notification  all handles stale; guest    config
>   guest)                                   must re-CREATE and rewrite  interrupt
>   INVALIDATE                               storage
>   ----------------------------------------------------------------------------------
> 
> Design points:
> 
> -   The message is stable and destination-independent. The host IR already
>     guarantees this (5.2). CREATE therefore carries no destination and can run
>     from the guest's .alloc, which can sleep and fail cleanly. Compose copies
>     the cached pair. Affinity changes never touch device storage. That matters
>     for device MSI in particular: writing a queue context from hard-IRQ context
>     is often impossible, and the device MSI chip has no affinity callback of its
>     own anyway.
> -   The RETARGET destination is the guest-format message. The guest's virtio-msi
>     level composes it by calling the parent x86_vector_domain (APIC ID, vector,
>     extended destination ID). The VMM puts it unchanged into KVM routing, so no
>     new destination encoding is needed.
> -   RETARGET is synchronous. It runs from irq_set_affinity (hard-IRQ context
>     with IRQCHIP_MOVE_DEFERRED) and from activation. Use a write to a trapped
>     virtio config/MMIO register and read back status. The vCPU waits in the exit
>     until the VMM has updated routing. A virtqueue cannot provide this. Hyper-V
>     uses the same split: VMBus for CREATE, a hypercall for RETARGET.
> -   The device MSI index is only a name. The host does not interpret the guest's
>     msi_index or storage slot. It only scopes handles, so that DESTROY and
>     INVALIDATE are well defined. (MSI-X differs, see 5.6(c).)
> -   Width negotiation. The x86 remappable message has a 32-bit address_lo
>     carrying the IRTE index, address_hi = 0, and data carrying the subhandle.
>     Device storage may be narrower. The guest declares the widths it has, and
>     the host fails CREATE if its message cannot fit.
> 
> 5.5 Host-side work (VFIO + IR)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 1.  Storage-less host IRQs. A VFIO device needs a way to allocate host IRQs with
>     an IR parent but no host writer. The same device MSI mechanism fits: VFIO
>     calls the new (not yet written) pci_device_msi_init_and_alloc_irqs() on the
>     assigned function, under its existing IR parent, with a write callback that
>     only records the composed message. The host cannot mask storage it doesn't
>     own; masking stays at the IR/eventfd level and the guest owns device-side
>     masking. This needs:
>     -   writing pci_device_msi_init_and_alloc_irqs() and adding the second
>         per-device domain slot (5.1), shared with the guest side,
>     -   the x86 vector, DMAR and AMDVI parents accepting DOMAIN_BUS_DEVICE_MSI
>         (x86_init_dev_msi_info(), x86_msi_prepare()), with an allocation type
>         that programs the PCI function's source ID (set_msi_sid()) as for MSI-X,
>     -   the host refusing it without isolated MSI.
> 2.  New VFIO uAPI. "Allocate N storage-less interrupts on this device, attach
>     eventfds, return each host message." This could be a new IRQ index or a new
>     ioctl. It needs a quota, and the irq bypass producer is registered exactly
>     as for MSI-X, so KVM posting works unchanged.
> 3.  Scalability. Device-specific storage exists largely to go beyond MSI-X's
>     2048 vectors. Each guest interrupt here costs:
>     -   a host IRQ and a host vector, because the IR domain allocates a
>         vector-domain parent even when posted. That is about 200 per host CPU.
>     -   a KVM GSI route. KVM_MAX_IRQ_ROUTES is 4096, and KVM_SET_GSI_ROUTING
>         replaces the whole table, with expedited SRCU, on every RETARGET.
>     -   an IRTE. Intel has 64K per IOMMU, shared host-wide; AMD has a
>         512/2048-entry per-device table.
> 
>     Thousands of device MSI interrupts per guest need posted-only IRTEs without
>     a host vector, and an incremental single-route KVM update (or a KVM-side
>     RETARGET). Neither exists today. Without them the design tops out at roughly
>     MSI-X scale, which defeats much of the point.
> 
> 5.6 Guest-side work
> ~~~~~~~~~~~~~~~~~~~
> 
> (a) Device storage through a device MSI domain. Once the new helper exists
> (5.1), the device driver calls
> pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) alongside its
> normal pci_alloc_irq_vectors() MSI-X allocation. The callback receives the
> msi_desc and uses desc->msi_index to find the queue/context to write. The domain
> is a fixed-size block allocated up front and is torn down with
> pci_device_msi_free_irqs_all(). Apart from the new PCI entry point and the
> second slot, this is the existing platform MSI mechanism unchanged.
> 
> (b) The virtio-msi domain is the device's MSI parent.
> msi_create_device_irq_domain() parents every per-device domain on
> dev->msi.domain, so the function's MSI-X domain and its device MSI domain share
> one parent. For a served device that parent must be the virtio-msi domain. It
> supplies its own init_dev_msi_info accepting DOMAIN_BUS_PCI_DEVICE_MSIX and
> DOMAIN_BUS_DEVICE_MSI (the x86 helper rejects both a non-x86 real parent and the
> device MSI token), its own msi_prepare that tags CREATE with the domain type,
> and irq_mask/irq_unmask, because the device MSI chip forwards masking to the
> parent. MSI-X is then also guest-written. VFIO's host writes to entry i write
> the same value, and the VMM must stop emulating the MSI-X table for that device.
> This is consistent and simpler than splitting MSI-X and device MSI across two
> models.
> 
> (c) MSI-X entries must be bound to the host index. The host masks, restores and
> refreshes MSI-X entry i (in vfio_msi_set_vector_signal() and
> pci_restore_msi_state()). CREATE for MSI-X therefore names index i, and the
> guest must write that message only into entry i. Ownership must be defined:
> 
> -   address/data: the guest, with the host only rewriting the identical value;
> -   the per-vector mask bit: the guest, once set-up completes.
> 
> Device MSI has no host writer, so there is no conflict there.
> 
> (d) Domain binding on KVM. Hyper-V gives each passthrough device its own
> synthetic bus, so dev_set_msi_domain() on the bridge selects the provider. A q35
> guest mixes emulated devices, which keep normal MSI, with assigned ones, and x86
> has no firmware mechanism such as DT msi-map or ACPI IORT to choose an MSI
> parent per device. Put the assigned devices behind a dedicated host bridge (e.g.
> pxb-pcie) and bind virtio-msi to that bridge. The device MSI domain lives on the
> same function, so it gets the same parent.
> 
> (e) Probe ordering and bootstrap. The virtio-msi control device takes its own
> interrupts from ordinary emulated MSI, not from its own domain. Its domain must
> exist before any served device allocates vectors. A dedicated bridge enumerated
> after virtio-msi probes, or -EPROBE_DEFER on the domain's absence, handles that.
> 
> (f) Callbacks.
> 
>     struct virtio_msi_irq {
>             u32 handle;
>             struct msi_msg msg;     /* host message, stable until INVALIDATE */
>     };
> 
> -   .alloc: allocate parent vectors, then CREATE (sleeping). On failure, unwind
>     and return the error, so pci_alloc_irq_vectors() or
>     pci_device_msi_init_and_alloc_irqs() fails cleanly. Hyper-V instead programs
>     0/0.
> -   .activate: the parent activates first. Compose the guest-format message from
>     the parent and RETARGET.
> -   .irq_set_affinity: call the parent, compose its new guest-format message,
>     RETARGET, and return IRQ_SET_MASK_OK_DONE so the device storage is never
>     rewritten. If RETARGET fails, restore the parent and return an error.
> -   .irq_compose_msi_msg: *msg = m->msg.
> -   .free: DESTROY (sleeping) after the core has shut the IRQ down.
> 
> 5.7 Migration and invalidation
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The destination host allocates different IRTEs, so every message the guest has
> written becomes invalid. For MSI-X the guest can rewrite from cached
> descriptors. For device MSI the storage is device-specific, possibly in guest
> RAM. Only the driver's write_msi_msg callback can update it, and it must happen
> before the device raises interrupts on the destination. Options:
> 
> -   INVALIDATE + re-CREATE + rewrite. The core walks each served domain's
>     descriptors, recomposes and calls irq_chip_write_msi_msg(), which reaches
>     the driver callback. That is a new requirement on device MSI drivers: the
>     callback must work outside the normal activation path, and the driver must
>     tolerate it while queues are live or quiesced.
> -   Host preserves the messages. This is impractical with Intel's global IRTE
>     table. It may be feasible with AMD's per-device remap table, because indices
>     are per device and the destination's table is empty.
> 
> Device-state migration (VFIO migration v2) must also be ordered with this: the
> device must not resume before its stored messages are valid. The same path
> handles host suspend/resume and control-device reset.
> 
> 5.8 Points the proposal must settle
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 1.  Scope: whole-function assignment only (5.3), x86 only. ARM64 with a vITS has
>     an architectural path (doorbell mapping plus guest EventIDs installed in the
>     physical ITT) and does not need a paravirtual message allocator.
> 2.  Host-side scale (5.5.3): without vector-less posted IRTEs and incremental
>     KVM routing, device MSI in a guest is limited to roughly MSI-X scale.
> 3.  The write_msi_msg callback contract for migration and invalidation (5.7).
> 4.  Domain binding: a dedicated bridge (5.6(d)).
> 5.  Whether RETARGET stays a VMM register (simple, the same cost as today's
>     trapped MSI-X write) or becomes a KVM-side operation (scales, new uAPI).
> 
> 5.9 Prototype order
> ~~~~~~~~~~~~~~~~~~~
> 
> 1.  Host: storage-less IR-backed IRQs in VFIO plus message readback. Verify that
>     posting through kvm_pi_update_irte works on them.
> 2.  Guest: virtio-msi parent with MSI-X only (index-bound, guest-written, VMM
>     table trap disabled). Verify that the written value equals the host's
>     composed message, and that affinity changes cause RETARGET without any
>     device rewrite.
> 3.  Device MSI: add the second per-device domain slot and write
>     pci_device_msi_init_and_alloc_irqs(), then allocate a device MSI domain next
>     to MSI-X on a test device, or on an emulated device with its own message
>     storage in QEMU backed by a real VFIO function.
> 4.  Failure injection (CREATE/RETARGET), CPU hot-unplug, FLR, guest driver
>     unbind with live interrupts.
> 5.  INVALIDATE/rewrite, then live migration.
> 6.  Scale tests at thousands of device MSI vectors per guest.
> 
> 6. Alternative: mirror the guest's interrupt remapping 1:1
> ----------------------------------------------------------
> 
> 6.1 The idea
> ~~~~~~~~~~~~
> 
> Sections 4–5 make the host choose the message and teach the guest to ask for it.
> The alternative turns this around: the guest chooses the message with its own,
> unmodified interrupt remapping driver, and the host makes the physical remapping
> hardware match.
> 
> With a vIOMMU that does interrupt remapping (VT-d IR, AMD IR), or a vITS on ARM,
> a guest driver's device MSI domain is parented on the guest's own remapping
> domain. The message it composes is already in remappable form: an IRTE index
> (x86) or an EventID (ITS). It names an entry in a guest-owned table, not a
> destination. If the host installs, at the same index, a physical entry for the
> same source device that delivers to the destination the guest's entry names,
> then the guest-written message works on real hardware unchanged. The device
> storage needs no trap and no host-provided message, so virtio-msi is
> unnecessary.
> 
> The destination translation already exists. KVM's posted-interrupt and GICv4
> paths turn a guest-format destination into a physical IRTE or vLPI mapping
> (5.2). The new requirement is that the host index is chosen by the guest rather
> than by the host allocator.
> 
>     guest driver:  device MSI domain -> guest IR domain (vIOMMU) -> vector domain
>                    composes {IRTE index i}, writes it into device storage
>     guest IR:      writes vIRTE[i] = (vBDF, APIC, vector); invalidates index i
>     VMM:           traps the invalidation, reads vIRTE[i], maps vBDF -> VFIO device
>     host:          allocates a storage-less host IRQ at IR index i for that RID,
>                    KVM routing gsi -> (APIC, vector), irqfd, posting as today
>     device:        emits {index i} with its RID -> physical IRTE[i] -> vCPU
> 
> Retargeting needs nothing new. A guest affinity change rewrites vIRTE[i] and
> invalidates it. The VMM updates the KVM route, and kvm_pi_update_irte() rewrites
> the physical IRTE. The index, and so the device storage, never changes. That is
> the stability property section 5.4 relied on, obtained here without a protocol.
> 
> What the host needs in every variant:
> 
> -   Storage-less host IRQs (5.5.1), with the extra ability to allocate at a
>     caller-specified remapping index or EventID. Today alloc_irte() (Intel),
>     alloc_irq_index() (AMD) and its_alloc_device_irq() (ITS) all pick the first
>     free slot.
> -   A uAPI to request that index, e.g. VFIO "allocate a storage-less IRQ on this
>     device at remapping index i", with failure reported if i is not available.
> -   A guest that can parent a device MSI domain on its vIOMMU IR domain. That is
>     the same x86 gap as on the host (DMAR/AMDVI accepting
>     DOMAIN_BUS_DEVICE_MSI), plus the new pci_device_msi_init_and_alloc_irqs()
>     (5.1). No paravirtual driver is needed, apart from VT-d's index allocation
>     callback (6.2).
> 
> Whether the scheme is viable depends entirely on whether the guest's index space
> can be made to coincide with a host index space that the host can dedicate to
> this device. That differs sharply by architecture.
> 
> 6.2 Intel VT-d: one global index space per IOMMU
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> How the VMM learns the guest's entry. The guest writes IRTEs into a table in its
> own memory, then issues an interrupt entry cache (IEC) invalidation through the
> queued-invalidation interface (modify_irte() → qi_flush_iec(), index-selective).
> QEMU already traps this. vtd_iec_notify_all() fans it out to IEC notifiers, and
> today kvm_update_msi_routes_all() uses it to refresh KVM routes for trapped
> MSI/MSI-X. A mirror would add a notifier that, for index i, reads vIRTE[i], uses
> its SID (Linux always programs source-ID verification, set_msi_sid()) to find
> the VFIO device, and allocates or updates the host IRQ at index i. The guest's
> invalidation waits on a QI wait descriptor, so the host update completes before
> the guest goes on to write device storage. Ordering is correct for free.
> 
> The problem: index collisions. A VT-d message carries a 16-bit handle into the
> IOMMU's single interrupt remapping table. There is no per-device or per-domain
> table and no offset. The physical table on that IOMMU is shared by the host's
> own interrupts, by every other assigned device behind it, and by every other
> guest. The guest's IR driver allocates from index 0 upward
> (bitmap_find_free_region()), exactly as the host's does and as every other
> guest's does. Guest index i will often already be taken on the host.
> 
> VT-d gives the vIOMMU no way to tell the guest "that index is busy". IEC
> invalidation cannot fail. The best the VMM can do is leave the interrupt dead
> and perhaps log a fault, which is the silent-failure mode section 4 criticised
> in Hyper-V.
> 
> The fix: a paravirtual index allocation callback. The guest must not pick the
> index itself. It asks the host. The vIOMMU advertises a PV capability, and the
> guest's alloc_irte() calls it instead of searching its own bitmap:
> 
>     guest alloc_irte(SID, count)
>        -> PV ALLOC(SID, count)         trapped vIOMMU register (or hypercall)
>        host: find a free, count-aligned block on the physical IOMMU behind the
>              device with that SID; reserve it for that RID; return index i
>        guest marks i..i+count-1 used in its bitmap, writes vIRTE[i], flushes IEC
>        -> existing IEC trap: VMM reads vIRTE[i], programs physical IRTE[i]
>     guest clear_entries(i)
>        -> PV FREE(i)                   host releases the reservation
> 
> -   Only allocation is paravirtual. The IRTE format, the remappable message, IEC
>     invalidation, retargeting and posting all stay architectural VT-d, and they
>     reuse the QEMU trap points above. This is far thinner than virtio-msi: there
>     is no host-composed message, no CREATE/RETARGET protocol and no new guest
>     IRQ domain. The change is a hook in the guest's Intel IR driver.
> -   Allocation can fail cleanly. The PV call returns an error, so alloc_irte()
>     fails and the IRQ allocation unwinds. That is the failure path IEC
>     invalidation lacks.
> -   It must be synchronous and cheap. alloc_irte() runs under irq_2_ir_lock (a
>     raw spinlock) during IRQ domain allocation. A trapped vIOMMU register write
>     with status readback fits that context; a virtqueue does not. Alternatively
>     the guest calls out before taking the lock.
> -   All guest allocations go through it. Indices for emulated devices need no
>     physical IRTE, but they must not collide with indices the host later hands
>     out for assigned devices in the same vIR table. Letting the host hand out
>     every index keeps the guest's table and the reservations consistent. For an
>     emulated SID the host just allocates from a per-VM bitmap without touching
>     hardware.
> -   MSI-X can use the same path. If the host IRQ for MSI-X entry k is allocated
>     at the guest-obtained index, the guest-written MSI-X message equals the
>     host's, as in the AMD case (6.3).
> 
> Alternatives considered, all worse:
> 
> -   One mirroring guest per physical IOMMU. The host allocates its own IRTEs
>     from the top of the table, and one guest gets the low indices. This is
>     deterministic but unscalable, and it depends on the platform's DRHD
>     topology.
> -   Pre-reserve a static window. Boot the vIOMMU with IR pre-enabled and every
>     vIRTE outside a host-reserved window marked present, so that the guest's
>     iommu_load_old_irte() avoids them. That path runs only in a kdump kernel, so
>     it still needs a guest change, and it fixes the window size at boot.
> -   Host IRTE window per guest, plus a fixed offset. Not possible: the device
>     emits the guest's index verbatim, and VT-d has no per-source index offset.
> 
> Smaller issues:
> 
> -   Multi-vector blocks: alloc_irte() aligns count to a power of two and uses
>     the handle mask in invalidations. The host must reserve the same aligned
>     block.
> 
> -   The guest may use a global IEC invalidation (also issued when IRTA is
>     written). The VMM then has to rescan up to 64K vIRTEs.
> 
> -   Scale is unchanged from 5.5.3: each mirrored entry is still a host IRQ, a
>     host vector and a KVM route.
> 
> -   Migration: the indices are guest state (in vIRTEs and in device storage).
>     The destination host must reserve the same indices on its physical IOMMU,
>     and with a global table they may already be in use there. Fixing that needs
>     either a per-VM index range coordinated across the migration pool, or a host
>     → guest "reallocate" event that makes the guest rewrite device storage. The
>     latter is the 5.7 problem again.
> 
> Verdict: VT-d needs one PV hook, an index allocation callback in the guest's IR
> driver, because its handle space is global. With that hook, the rest of the
> mirroring scheme is architectural and reuses existing QEMU and KVM paths.
> Migration remains the weak point.
> 
> 6.3 AMD-Vi: per-device index space
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> AMD remaps through a per-DeviceID interrupt table referenced from the DTE (512
> entries, or 2048 with MAX_IRQS_PER_TABLE_2K). The message's data field carries
> the index, and the IOMMU looks it up in the table belonging to the requester's
> DeviceID. QEMU's AMD vIOMMU likewise gives the guest a per-device table and
> traps INVALIDATE_INTERRUPT_TABLE (AMDVI_CMD_INVAL_INTR_TABLE), calling the
> same
> IEC notifier chain.
> 
> For whole-function assignment (5.3) the physical table belongs to that one
> device. No other device, guest or host consumer shares it. The guest's index i
> and the host's index i live in the same private namespace, so the collision
> problem from 6.2 does not arise for device MSI.
> 
> The one sharer is the host itself, for the same device's MSI-X: VFIO allocates
> host MSI-X IRTEs in the same per-device table, at indices chosen by
> alloc_irq_index(). The guest's IR driver also allocates its MSI-X vIRTEs in its
> per-device table. Two consistent options:
> 
> -   Mirror everything. When the guest writes a remappable MSI-X message (data =
>     vIRTE index i), VFIO allocates the host MSI-X IRQ at IRTE i too. The
>     host-composed message then equals the guest-written one, and MSI-X table
>     trapping is no longer needed for correctness either.
> -   Partition. The host keeps MSI-X in a range the guest's allocator will not
>     use. The guest allocator, like Intel's, gives no such guarantee without a
>     hint, so this is the weaker option.
> 
> Caveats:
> 
> -   The table is per DeviceID, which is the alias after DMA aliasing. A function
>     behind a PCIe-to-PCI bridge shares its table with its siblings, so only
>     functions with their own DeviceID qualify. This is the same condition as
>     5.3.
> -   Capacity is 512 or 2048 entries per device. That is enough for many
>     device-MSI uses, though well short of the "thousands per device" goal.
> -   The AMD vIOMMU in QEMU must emulate interrupt remapping completely enough
>     for the guest to use it (including x2APIC and the table-length encoding),
>     and the guest must enable IR on it.
> 
> Verdict: AMD-Vi fits the mirroring model well. The host work is index-pinned,
> storage-less host IRQs plus a VMM notifier. No guest PV interface is needed.
> 
> 6.4 ARM GICv3/v4 ITS: per-device EventID space
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The ITS is architecturally the closest match. A device's MSI is a write of an
> EventID to the ITS doorbell (GITS_TRANSLATER). The DeviceID comes from the bus
> (RID → StreamID → DeviceID through IORT / msi-map), not from the message.
> Translation is per DeviceID: a device table entry points to that device's ITT,
> indexed by EventID. EventIDs are therefore a private namespace per device, like
> AMD-Vi and unlike VT-d.
> 
> The guest drives a vITS. It issues MAPD (virtual DeviceID → ITT) and MAPTI
> (DeviceID, EventID → vLPI, collection), then writes the EventID into device
> storage. For that write to reach the guest on real hardware:
> 
> 1.  EventID. The host ITT for the physical DeviceID must contain the guest's
>     EventID e, mapped to a host LPI, and with GICv4 forwarded (VMAPTI) to the
>     guest's vLPI. The host ITS allocates events with its_alloc_device_irq(), the
>     first free region of lpi_map. It would need to allocate at a requested
>     EventID. The obvious rule for the device MSI domain is EventID = msi_index,
>     which VFIO sets to e.
> 2.  ITT sizing. its_msi_prepare() reuses an existing its_device for the DeviceID
>     ("shared") and keeps the ITT size from first creation. If MSI-X created the
>     ITT with 32 events, a device MSI domain on the same DeviceID cannot place
>     events beyond that. The ITT has to be sized for the whole guest-visible
>     EventID range at the first allocation, or be grown.
> 3.  Doorbell address. The device writes to the guest's vITS doorbell address.
>     That is a GPA (no vSMMU), or an IOVA the guest's stage 1 maps to that GPA
>     (nested vSMMU). Host stage 2 must map that GPA to the physical
>     GITS_TRANSLATER page of the ITS serving the device. Today the host maps the
>     physical doorbell at the reserved SW_MSI window (MSI_IOVA_BASE) that the
>     host itself chose (iommufd_sw_msi), and the guest's vITS doorbell sits
>     elsewhere. One of two things is needed: either the VMM places the vITS
>     doorbell where the host maps the physical one, or there is a new uAPI to map
>     the physical doorbell at a VMM-chosen IPA. iommufd has neither today.
> 4.  Learning the guest's events. With the in-kernel vITS, MAPTI is handled by
>     KVM (vgic_its_cmd_handle_mapi()), not QEMU. KVM already resolves (DeviceID,
>     EventID) → vLPI and calls its_map_vlpi() when an irqfd route is attached
>     (kvm_vgic_v4_set_forwarding()). But it does that lazily: forwarding happens
>     at route time, and MAPTI only creates the vLPI. Two options:
>     -   The VMM pre-allocates host events 0..N-1 for the device, with routes for
>         each (DeviceID, e). This is a quota-sized upfront cost. KVM then
>         forwards whichever ones the guest maps.
>     -   KVM (or a KVM → VMM exit on MAPTI/DISCARD for assigned DeviceIDs) drives
>         host event allocation on demand. That is new KVM ↔ VFIO plumbing.
> 5.  Retarget is MOVI/VMOVI, already handled by KVM for forwarded vLPIs. The
>     EventID, and so device storage, never changes.
> 
> Security matches 5.3. The guest can only name EventIDs in its own device's ITT,
> and the DeviceID is fixed by the hardware path. A shared DeviceID (aliasing, or
> SIOV-style sharing) breaks it the same way.
> 
> Verdict: the ITS model supports mirroring without collisions. The work lies in
> the host ITS driver (allocating at an EventID, ITT sizing), in the doorbell
> mapping uAPI, and in how KVM/VMM learn guest events. As 5.8 noted, this is also
> the native ARM answer that never needed a message allocator.
> 
> 6.5 Comparison with virtio-msi
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>   --------------------------------------------------------------------------------------
>               virtio-msi (5) Mirror on VT-d       Mirror on AMD-Vi  Mirror on ITS
>   ----------- -------------- -------------------- ----------------- --------------------
>   Guest PV    virtio-msi IRQ index alloc/free     none              none
>   interface   domain +       callback in the IR
>               protocol       driver
> 
>   Who picks   host           host, via PV         guest             guest
>   the index                  callback
> 
>   Collision   none           none: host allocates none: per-device  none: per-device ITT
>   risk                       in the global table  table
> 
>   Failure     CREATE status  PV ALLOC status      none, but cannot  none, but cannot
>   reporting                                       collide           collide
>   to guest
> 
>   Retarget    trapped        existing IEC trap →  existing trap →   existing KVM vITS
>               register       KVM route            KVM route
> 
>   Main new    storage-less   PV alloc/free,       index-pinned      EventID-pinned
>   host work   IRQs, protocol index-pinned IRTEs,  IRTEs, QEMU       alloc, ITT sizing,
>                              QEMU notifier        notifier          doorbell mapping,
>                                                                     event discovery
> 
>   Migration   INVALIDATE +   destination must     destination       destination
>               guest rewrite  reproduce indices:   reproduces        reproduces
>               (5.7)          fragile with a       per-device        per-device EventIDs:
>                              global table, even   indices: feasible feasible
>                              with PV alloc
>   --------------------------------------------------------------------------------------
> 
> Migration deserves emphasis. With mirroring, the guest's indices are part of the
> guest state, held in guest memory and in device storage that only the guest
> driver can rewrite. The destination host must recreate exactly those indices.
> That works for per-device namespaces (AMD-Vi, ITS), because the destination
> device's table is empty. With VT-d's global table it fails whenever the
> destination IOMMU already uses one of them. This is the mirror image of 5.7,
> where virtio-msi needs the guest to rewrite storage instead.
> 
> Overall: 1:1 mirroring removes the need for virtio-msi where the remapping index
> space is per device: AMD-Vi and the ARM ITS. On Intel VT-d the global handle
> space needs one PV hook, an index allocation callback in the guest's IR driver.
> That is still much smaller than virtio-msi, because the message format,
> retargeting and posting stay architectural. Migration on VT-d remains fragile
> without a coordinated per-VM index range, a guest rewrite path, or a hardware
> change (per-device or per-domain IR tables).
> 
> Source reading map
> ------------------
> 
> arch/x86/hyperv/hv_init.c
> 
> -   Functions: create_pci_msi_domain hook
> -   What they establish: Root-partition provider selection
> 
> arch/x86/hyperv/irqdomain.c
> 
> -   Functions: hv_irq_compose_msi_msg, hv_map_interrupt,
>     hv_build_devid_type_pci, entry_to_msi_msg, hv_teardown_msi_irq
> -   What they establish: Hypercall output becomes the message; unmap+map on move
> 
> drivers/pci/controller/pci-hyperv.c
> 
> -   Functions: hv_pcie_init_irq_domain, hv_compose_msi_msg,
>     hv_irq_retarget_interrupt, hv_arch_irq_unmask, hv_msi_free,
>     hv_pci_vec_irq_domain_alloc (ARM64), hv_pci_restore_msi_state
> -   What they establish: VMBus CREATE, hypercall RETARGET, per-bus binding, SPI
>     model, resume
> 
> kernel/irq/chip.c
> 
> -   Functions: irq_chip_compose_msi_msg
> -   What they establish: Finding the composer in the hierarchy
> 
> kernel/irq/msi.c
> 
> -   Functions: msi_domain_activate, msi_domain_set_affinity
> -   What they establish: Compose → write; recompose on move unless OK_DONE
> 
> kernel/irq/migration.c
> 
> -   Functions: irq_move_masked_irq
> -   What they establish: Deferred moves run in interrupt context
> 
> drivers/pci/msi/irqdomain.c
> 
> -   Functions: pci_msi_domain_write_msg
> -   What they establish: Device domain write callback
> 
> drivers/pci/msi/msi.c
> 
> -   Functions: __pci_write_msi_msg
> -   What they establish: Config-space / MSI-X table programming
> 
> drivers/vfio/pci/vfio_pci_intrs.c
> 
> -   Functions: vfio_msi_set_vector_signal, vfio_msi_alloc_irq
> -   What they establish: Host IRQ per MSI-X index, message refresh, irq bypass
>     producer
> 
> drivers/iommu/intel/irq_remapping.c
> 
> -   Functions: intel_ir_compose_msi_msg, fill_msi_msg, intel_ir_set_affinity,
>     intel_ir_set_vcpu_affinity, set_msi_sid
> -   What they establish: Stable remappable message; IRTE-only retarget;
>     source-ID isolation
> 
> drivers/iommu/amd/iommu.c
> 
> -   Functions: amd_ir_set_affinity, amd_ir_set_vcpu_affinity
> -   What they establish: Same properties on AMD
> 
> drivers/base/platform-msi.c
> 
> -   Functions: platform_device_msi_init_and_alloc_irqs, platform_msi_template,
>     platform_device_msi_free_irqs_all
> -   What they establish: Device MSI domain with a driver write callback; masking
>     forwarded to the parent. Model for the new PCI helper
> 
> kernel/irq/msi.c, include/linux/msi_api.h
> 
> -   Functions: msi_create_device_irq_domain, msi_domain_ids
> -   What they establish: Per-device domain parented on dev->msi.domain; only one
>     slot per device today
> 
> drivers/irqchip/irq-msi-lib.c
> 
> -   Functions: msi_lib_init_dev_msi_info
> -   What they establish: How non-x86 parents accept DOMAIN_BUS_DEVICE_MSI
> 
> arch/x86/kernel/apic/msi.c
> 
> -   Functions: x86_init_dev_msi_info, x86_msi_prepare
> -   What they establish: x86 parents accept only PCI MSI/MSI-X device domains
> 
> arch/x86/kvm/irq.c
> 
> -   Functions: kvm_pi_update_irte, kvm_arch_update_irqfd_routing
> -   What they establish: Guest routing change → posted IRTE update
> 
> drivers/iommu/intel/irq_remapping.c
> 
> -   Functions: alloc_irte, modify_irte, qi_flush_iec, iommu_load_old_irte,
>     fill_msi_msg
> -   What they establish: Global first-fit IRTE allocation; IEC invalidation per
>     index; pre-enabled IR copy (kdump only)
> 
> drivers/iommu/amd/iommu.c
> 
> -   Functions: alloc_irq_index, irq_remapping_alloc
> -   What they establish: Per-DeviceID interrupt table, 512/2048 entries
> 
> drivers/irqchip/irq-gic-v3-its.c
> 
> -   Functions: its_msi_prepare, its_alloc_device_irq, its_map_vlpi
> -   What they establish: Per-DeviceID ITT, first-fit EventIDs, ITT size fixed at
>     first use, GICv4 forwarding
> 
> arch/arm64/kvm/vgic/vgic-its.c, arch/arm64/kvm/vgic/vgic-v4.c
> 
> -   Functions: vgic_its_cmd_handle_mapi, kvm_vgic_v4_set_forwarding
> -   What they establish: In-kernel vITS commands; vLPI forwarding set up at
>     irqfd route time
> 
> QEMU hw/i386/intel_iommu.c, hw/i386/amd_iommu.c, target/i386/kvm/kvm.c
> 
> -   Functions: vtd_iec_notify_all, AMDVI_CMD_INVAL_INTR_TABLE,
>     kvm_update_msi_routes_all
> -   What they establish: VMM trap points for guest IRTE changes