[PATCH v1 0/4] Implement IOMMU SVA page fault processing error notifiers

[PATCH v1 0/4] Implement IOMMU SVA page fault processing error notifiers

[Sergey Temerkhanov]

This series introduces a facility of notifying endpoint device
drivers of failing IOMMU page faults occurring for specific
Requester IDs.

According to the PCIe spec chapter "Page Request Group Response Message",
in case of Page Request processing failure (e.g., detecting attempt to
access pages with insufficient permissions, like writing to RO pages)
the system responds to the requester with a message containing a non-zero
response code. The requester device is then supposed to process that
response in implementation dependent way.

However, some endpoint hardware supporting SVA does not provide insight
on failed IO page faults, being limited to generic error reporting
(via interrupts and/or few register bits).

These series contain changes that provide endpoint device drivers
with the ability to retrieve information about page fault processing
errors (the most relevant being PASID and IOVA PFN).

Endpoint device drivers are able to set a callback functions
which will be called when a failing page fault (such as accessing pages
with mismatching attributes - e.g. writing to RO pages) occurs.

Normal processing path (i.e., swapping in or allocating backing physical
memory for non-present pages) is unaffected by these changes.

The endpoint driver code can then take appropriate actions, such as
fault reporting and recovery, sending signals to affected processes etc,
using the fault context provided in the fault parameter structure.


Sergey Temerkhanov (4):
  iommu: Pass the Requester ID in the fault parameter structure
  iommu: Add rid_notifier array to the dev_iommu structure
  iommu: Implement iommu_set_rid_fault_notifier()
  iommu: Notify requesters of IOMMU fault failures

 drivers/iommu/intel/prq.c  |  1 +
 drivers/iommu/io-pgfault.c | 53 ++++++++++++++++++++++++++++++++++++++
 drivers/iommu/iommu-priv.h | 11 ++++++++
 drivers/iommu/iommu-sva.c  | 19 +++++++++++++-
 drivers/iommu/iommu.c      | 21 +++++++++++++++
 include/linux/iommu.h      | 19 ++++++++++++++
 6 files changed, 123 insertions(+), 1 deletion(-)

-- 
2.43.0

[PATCH v1 1/4] iommu: Pass the Requester ID in the fault parameter structure

[Sergey Temerkhanov]

Pass the Requester ID in the fault parameter structure to facilitate
requester notifications on IOMMU faults.

Signed-off-by: Sergey Temerkhanov <sergey.temerkhanov@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
 drivers/iommu/intel/prq.c | 1 +
 include/linux/iommu.h     | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/drivers/iommu/intel/prq.c b/drivers/iommu/intel/prq.c
index 52570e42a14c..d88385adc3cb 100644
--- a/drivers/iommu/intel/prq.c
+++ b/drivers/iommu/intel/prq.c
@@ -181,6 +181,7 @@ static void intel_prq_report(struct intel_iommu *iommu, struct device *dev,
 	/* Fill in event data for device specific processing */
 	event.fault.type = IOMMU_FAULT_PAGE_REQ;
 	event.fault.prm.addr = (u64)desc->addr << VTD_PAGE_SHIFT;
+	event.fault.prm.rid = desc->rid;
 	event.fault.prm.pasid = desc->pasid;
 	event.fault.prm.grpid = desc->prg_index;
 	event.fault.prm.perm = prq_to_iommu_prot(desc);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 156732807994..4ddb63a3f2cf 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -65,6 +65,7 @@ enum iommu_fault_type {
  *         When IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID is set, the page response
  *         must have the same PASID value as the page request. When it is clear,
  *         the page response should not have a PASID.
+ * @rid: Requester ID
  * @pasid: Process Address Space ID
  * @grpid: Page Request Group Index
  * @perm: requested page permissions (IOMMU_FAULT_PERM_* values)
@@ -76,6 +77,7 @@ struct iommu_fault_page_request {
 #define IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE	(1 << 1)
 #define IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID	(1 << 2)
 	u32	flags;
+	u32	rid;
 	u32	pasid;
 	u32	grpid;
 	u32	perm;
-- 
2.43.0

[PATCH v1 2/4] iommu: Add rid_notifier array to the dev_iommu structure

[Sergey Temerkhanov]

Add rid_notifier array to struct dev_iommu to facilitate
requester notification of IOMMU faults.

Signed-off-by: Sergey Temerkhanov <sergey.temerkhanov@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
 drivers/iommu/iommu.c | 9 +++++++++
 include/linux/iommu.h | 4 ++++
 2 files changed, 13 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index a4b606c591da..3789c16806a1 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -354,6 +354,7 @@ static struct dev_iommu *dev_iommu_get(struct device *dev)
 		return NULL;
 
 	mutex_init(&param->lock);
+	xa_init(&param->rid_notifiers);
 	dev->iommu = param;
 	return param;
 }
@@ -361,8 +362,16 @@ static struct dev_iommu *dev_iommu_get(struct device *dev)
 void dev_iommu_free(struct device *dev)
 {
 	struct dev_iommu *param = dev->iommu;
+	struct rid_notifier *notifier;
+	unsigned long rid;
 
 	dev->iommu = NULL;
+
+	xa_for_each(&param->rid_notifiers, rid, notifier)
+		kfree(notifier);
+
+	xa_destroy(&param->rid_notifiers);
+
 	if (param->fwspec) {
 		fwnode_handle_put(param->fwspec->iommu_fwnode);
 		kfree(param->fwspec);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 4ddb63a3f2cf..2062623fdf32 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -795,6 +795,8 @@ struct iommu_fault_param {
  * @pci_32bit_workaround: Limit DMA allocations to 32-bit IOVAs
  * @require_direct: device requires IOMMU_RESV_DIRECT regions
  * @shadow_on_flush: IOTLB flushes are used to sync shadow tables
+ * @rid_notifiers: holds pointers and data to the notifier functions for
+ *                   specific RIDs
  *
  * TODO: migrate other per device data pointers under iommu_dev_data, e.g.
  *	struct iommu_group	*iommu_group;
@@ -810,6 +812,8 @@ struct dev_iommu {
 	u32				pci_32bit_workaround:1;
 	u32				require_direct:1;
 	u32				shadow_on_flush:1;
+
+	struct xarray			rid_notifiers;
 };
 
 int iommu_device_register(struct iommu_device *iommu,
-- 
2.43.0

[PATCH v1 3/4] iommu: Implement iommu_set_rid_fault_notifier()

[Sergey Temerkhanov]

Introduce the iommu_set_rid_fault_notifier() function which sets
an IOMMU fault notification callback for a specified endpoint device.
PCI endpoint device support is implemented in this patch, with possibility
of extension.

This change will provide endpoint device drivers with means of retrieving
details of IOMMU page faults occurring during operation and taking
device-specific actions.

Sample code using this facility:

static int sample_iommu_fault_notify(struct device *dev,
				     struct iommu_fault *fault,
                                     enum iommu_page_response_code,
				     void *data)
{
       printk("Fault data: rid: %#x, pasid: %u, ...  addr: %#llx\n",
              fault->prm.rid, fault->prm.pasid, ...
	      fault->prm.addr);
}

...

       ret = iommu_set_rid_fault_notifier(&pdev->dev,
					  sample_iommu_fault_notify,
					  data);
       if (ret)
               ...

Signed-off-by: Sergey Temerkhanov <sergey.temerkhanov@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
 drivers/iommu/io-pgfault.c | 53 ++++++++++++++++++++++++++++++++++++++
 drivers/iommu/iommu-priv.h | 11 ++++++++
 drivers/iommu/iommu.c      | 12 +++++++++
 include/linux/iommu.h      | 17 ++++++++++--
 4 files changed, 91 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c
index 8b5926c1452e..8e50c5f03da2 100644
--- a/drivers/iommu/io-pgfault.c
+++ b/drivers/iommu/io-pgfault.c
@@ -5,6 +5,7 @@
  * Copyright (C) 2020 ARM Ltd.
  */
 
+#include <linux/cleanup.h>
 #include <linux/iommu.h>
 #include <linux/list.h>
 #include <linux/sched/mm.h>
@@ -547,3 +548,55 @@ void iopf_queue_free(struct iopf_queue *queue)
 	kfree(queue);
 }
 EXPORT_SYMBOL_GPL(iopf_queue_free);
+
+/**
+ * iommu_set_rid_fault_notifier() - Set a Requester ID fault notifier
+ * @dev: the requester device
+ * @notifier: notifier function pointer (NULL unsets the notifier)
+ * @data: data to pass to the notifier function as a parameter
+ *
+ * Set or remove a device Requester ID based IOMMU fault failure
+ * notifier function.
+ *
+ * Return: 0 on success, or an error code.
+ */
+int iommu_set_rid_fault_notifier(struct device *dev,
+				 iommu_fault_rid_notifier_t notifier,
+				 void *data)
+{
+	struct iommu_rid_notifier *entry, *old;
+	struct dev_iommu *param = dev->iommu;
+	u32 rid;
+	int ret;
+
+	if (!param || !param->fault_param)
+		return -EINVAL;
+
+	rid = iommu_get_dev_rid(dev);
+
+	if (rid == IOMMU_INVALID_RID)
+		return -EINVAL;
+
+	guard(mutex)(&param->lock);
+
+	entry = kmalloc(sizeof(struct iommu_rid_notifier), GFP_KERNEL);
+	if (!entry)
+		return -ENOMEM;
+
+	entry->notifier = notifier;
+	entry->data = data;
+	entry->dev = dev;
+
+	old = xa_store(&param->rid_notifiers, rid, entry, GFP_KERNEL);
+
+	ret = xa_err(old);
+	if (ret) {
+		kfree(entry);
+		return ret;
+	}
+
+	kfree(old);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_set_rid_fault_notifier);
diff --git a/drivers/iommu/iommu-priv.h b/drivers/iommu/iommu-priv.h
index e236b932e766..71e187bc024b 100644
--- a/drivers/iommu/iommu-priv.h
+++ b/drivers/iommu/iommu-priv.h
@@ -63,4 +63,15 @@ static inline int iommufd_sw_msi(struct iommu_domain *domain,
 int iommu_replace_device_pasid(struct iommu_domain *domain,
 			       struct device *dev, ioasid_t pasid,
 			       struct iommu_attach_handle *handle);
+
+struct iommu_rid_notifier {
+	iommu_fault_rid_notifier_t notifier;
+	struct device *dev;
+	void *data;
+};
+
+#define IOMMU_INVALID_RID	U32_MAX
+
+u32 iommu_get_dev_rid(struct device *dev);
+
 #endif /* __LINUX_IOMMU_PRIV_H */
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 3789c16806a1..c34c602dfb8a 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -3845,3 +3845,15 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr)
 	return ret;
 }
 #endif /* CONFIG_IRQ_MSI_IOMMU */
+
+u32 iommu_get_dev_rid(struct device *dev)
+{
+	struct pci_dev *pdev;
+
+	if (!dev_is_pci(dev))
+		return IOMMU_INVALID_RID;
+
+	pdev = to_pci_dev(dev);
+
+	return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 2062623fdf32..3a3f92222e27 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -161,6 +161,8 @@ struct iopf_queue {
 
 typedef int (*iommu_fault_handler_t)(struct iommu_domain *,
 			struct device *, unsigned long, int, void *);
+typedef void (*iommu_fault_rid_notifier_t)(struct device *, struct iommu_fault *,
+					   enum iommu_page_response_code, void *);
 
 struct iommu_domain_geometry {
 	dma_addr_t aperture_start; /* First address that can be mapped    */
@@ -803,6 +805,7 @@ struct iommu_fault_param {
  */
 struct dev_iommu {
 	struct mutex lock;
+	struct xarray			rid_notifiers;
 	struct iommu_fault_param __rcu	*fault_param;
 	struct iommu_fwspec		*fwspec;
 	struct iommu_device		*iommu_dev;
@@ -812,8 +815,6 @@ struct dev_iommu {
 	u32				pci_32bit_workaround:1;
 	u32				require_direct:1;
 	u32				shadow_on_flush:1;
-
-	struct xarray			rid_notifiers;
 };
 
 int iommu_device_register(struct iommu_device *iommu,
@@ -931,6 +932,10 @@ void iommu_set_dma_strict(void);
 extern int report_iommu_fault(struct iommu_domain *domain, struct device *dev,
 			      unsigned long iova, int flags);
 
+extern int iommu_set_rid_fault_notifier(struct device *dev,
+					iommu_fault_rid_notifier_t notifier,
+					void *data);
+
 static inline void iommu_flush_iotlb_all(struct iommu_domain *domain)
 {
 	if (domain->ops->flush_iotlb_all)
@@ -1480,6 +1485,14 @@ static inline int iommu_dma_prepare_msi(struct msi_desc *desc,
 {
 	return 0;
 }
+
+static inline int iommu_set_rid_fault_notifier(struct device *dev,
+					       iommu_fault_rid_notifier_t notifier,
+					       void *data)
+{
+	return 0;
+}
+
 #endif /* CONFIG_IOMMU_API */
 #endif /* CONFIG_IRQ_MSI_IOMMU */
 
-- 
2.43.0

[PATCH v1 4/4] iommu: Notify requesters of IOMMU fault failures

[Sergey Temerkhanov]

Call the notifier callbacks installed by the device drivers when
failing IOMMU page faults occur during the SVA mode operation.

Signed-off-by: Sergey Temerkhanov <sergey.temerkhanov@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
 drivers/iommu/iommu-sva.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
index 1a51cfd82808..18b6d9b02899 100644
--- a/drivers/iommu/iommu-sva.c
+++ b/drivers/iommu/iommu-sva.c
@@ -256,11 +256,19 @@ iommu_sva_handle_mm(struct iommu_fault *fault, struct mm_struct *mm)
 	return status;
 }
 
+static struct iommu_rid_notifier *iommu_get_rid_notifier(struct device *dev, u32 rid)
+{
+	struct dev_iommu *param = dev->iommu;
+
+	return xa_load(&param->rid_notifiers, rid);
+}
+
 static void iommu_sva_handle_iopf(struct work_struct *work)
 {
 	struct iopf_fault *iopf;
 	struct iopf_group *group;
 	enum iommu_page_response_code status = IOMMU_PAGE_RESP_SUCCESS;
+	struct iommu_rid_notifier *rid_notifier;
 
 	group = container_of(work, struct iopf_group, work);
 	list_for_each_entry(iopf, &group->faults, list) {
@@ -268,8 +276,17 @@ static void iommu_sva_handle_iopf(struct work_struct *work)
 		 * For the moment, errors are sticky: don't handle subsequent
 		 * faults in the group if there is an error.
 		 */
-		if (status != IOMMU_PAGE_RESP_SUCCESS)
+		if (status != IOMMU_PAGE_RESP_SUCCESS) {
+			/* Notify the requester of a failure. */
+			rid_notifier = iommu_get_rid_notifier(group->fault_param->dev,
+							      iopf->fault.prm.rid);
+
+			if (rid_notifier && rid_notifier->notifier)
+				rid_notifier->notifier(rid_notifier->dev, &iopf->fault,
+						       status, rid_notifier->data);
+
 			break;
+		}
 
 		status = iommu_sva_handle_mm(&iopf->fault,
 					     group->attach_handle->domain->mm);
-- 
2.43.0

Re: [PATCH v1 0/4] Implement IOMMU SVA page fault processing error notifiers

[Jason Gunthorpe]

On Thu, Jul 10, 2025 at 01:42:11PM +0000, Sergey Temerkhanov wrote:
> These series contain changes that provide endpoint device drivers
> with the ability to retrieve information about page fault processing
> errors (the most relevant being PASID and IOVA PFN).

You need to come with a driver using this as well.

Jason

Re: [PATCH v1 4/4] iommu: Notify requesters of IOMMU fault failures

[Jason Gunthorpe]

On Thu, Jul 10, 2025 at 01:42:15PM +0000, Sergey Temerkhanov wrote:
> Call the notifier callbacks installed by the device drivers when
> failing IOMMU page faults occur during the SVA mode operation.
> 
> Signed-off-by: Sergey Temerkhanov <sergey.temerkhanov@intel.com>
> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> ---
>  drivers/iommu/iommu-sva.c | 19 ++++++++++++++++++-
>  1 file changed, 18 insertions(+), 1 deletion(-)

It doesn't seem to me like any of this is really needed..

Look at how iommufd already associates faults back to its own struct
device structure side-structure.

I think all you want for this problem is a way to hook the fault
callback chain on a domain, and that should be local to the domain not
global to the device..

Jason

RE: [PATCH v1 0/4] Implement IOMMU SVA page fault processing error notifiers

[Temerkhanov, Sergey]

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> 
> You need to come with a driver using this as well.
> 

There is an OOT driver which would use this facility do report and handle possible invalid accesses (e.g. mismatched VMA permissions) during the SVA operation
https://www.intel.com/content/www/us/en/download/858019/intel-media-transcode-accelerator.html

Regards,
Sergey

RE: [PATCH v1 4/4] iommu: Notify requesters of IOMMU fault failures

[Temerkhanov, Sergey]

> Look at how iommufd already associates faults back to its own struct device
> structure side-structure.
> 
> I think all you want for this problem is a way to hook the fault callback chain
> on a domain, and that should be local to the domain not global to the device..
> 

Moving this to a domain sounds reasonable, however the main goal is to reuse the kernel
SVA IOPF handling code to the maximum extent and only provide "feedback" when
iommu_sva_handle_iopf() returns an error for some reason.

Regards,
Sergey

Re: [PATCH v1 0/4] Implement IOMMU SVA page fault processing error notifiers

[Jason Gunthorpe]

On Wed, Jul 16, 2025 at 11:07:50AM +0000, Temerkhanov, Sergey wrote:
> > -----Original Message-----
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > 
> > You need to come with a driver using this as well.
> > 
> 
> There is an OOT driver which would use this facility do report and
> handle possible invalid accesses (e.g. mismatched VMA permissions)
> during the SVA operation

Upstream a basic driver first then ask for iommu changes to enhance
it.

This is normal stuff, we don't change the kernel to accomodate OOT
drivers.

Jason