[PATCH v2 1/1] iommu/vt-d: Move PCI PASID enablement to probe path

[PATCH v2 1/1] iommu/vt-d: Move PCI PASID enablement to probe path

[Lu Baolu]

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is
attached to the device and disabled when the device transitions to block
translation mode. This approach is inappropriate as PCI PASID is a device
feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the
release path.

Suggested-by: Yi Liu <yi.l.liu@intel.com>
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Yi Liu <yi.l.liu@intel.com>
---
 drivers/iommu/intel/iommu.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)

Change log:
v2:
 - Move pasid enablement the last step in intel_iommu_probe_device().
 - Clear pasid enabled flags after pci_disable_pasid().

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 9ff8b83c19a3..43ad1f551581 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
 		return;
 
 	pdev = to_pci_dev(info->dev);
-
-	/* The PCIe spec, in its wisdom, declares that the behaviour of
-	   the device if you enable PASID support after ATS support is
-	   undefined. So always enable PASID support on devices which
-	   have it, even if we can't yet know if we're ever going to
-	   use it. */
-	if (info->pasid_supported && !pci_enable_pasid(pdev, info->pasid_supported & ~1))
-		info->pasid_enabled = 1;
-
 	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
 	    !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) {
 		info->ats_enabled = 1;
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct device_domain_info *info)
 		info->ats_enabled = 0;
 		domain_update_iotlb(info->domain);
 	}
-
-	if (info->pasid_enabled) {
-		pci_disable_pasid(pdev);
-		info->pasid_enabled = 0;
-	}
 }
 
 static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4112,6 +4098,16 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
 
 	intel_iommu_debugfs_create_dev(info);
 
+	/*
+	 * The PCIe spec, in its wisdom, declares that the behaviour of the
+	 * device is undefined if you enable PASID support after ATS support.
+	 * So always enable PASID support on devices which have it, even if
+	 * we can't yet know if we're ever going to use it.
+	 */
+	if (info->pasid_supported &&
+	    !pci_enable_pasid(pdev, info->pasid_supported & ~1))
+		info->pasid_enabled = 1;
+
 	return &iommu->iommu;
 free_table:
 	intel_pasid_free_table(dev);
@@ -4128,6 +4124,11 @@ static void intel_iommu_release_device(struct device *dev)
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
 
+	if (info->pasid_enabled) {
+		pci_disable_pasid(to_pci_dev(dev));
+		info->pasid_enabled = 0;
+	}
+
 	mutex_lock(&iommu->iopf_lock);
 	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
 		device_rbtree_remove(info);
-- 
2.34.1

Re: [PATCH v2 1/1] iommu/vt-d: Move PCI PASID enablement to probe path

[Yi Liu]

On 2024/8/19 13:18, Lu Baolu wrote:
> Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is
> attached to the device and disabled when the device transitions to block
> translation mode. This approach is inappropriate as PCI PASID is a device
> feature independent of the type of the attached domain.
> 
> Enable PCI PASID during the IOMMU device probe and disables it during the
> release path.
> 
> Suggested-by: Yi Liu <yi.l.liu@intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> Reviewed-by: Yi Liu <yi.l.liu@intel.com>

Applied this patch to 6.11-rc4 and assigned a PASID-capable device to VM. I
can see the PASID cap is enabled before and after VM is boot. In the
before, this cap would be disabled when it is attached to blocking domain.
Hence,

Tested-by: Yi Liu <yi.l.liu@intel.com>

> ---
>   drivers/iommu/intel/iommu.c | 29 +++++++++++++++--------------
>   1 file changed, 15 insertions(+), 14 deletions(-)
> 
> Change log:
> v2:
>   - Move pasid enablement the last step in intel_iommu_probe_device().
>   - Clear pasid enabled flags after pci_disable_pasid().
> 
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index 9ff8b83c19a3..43ad1f551581 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
>   		return;
>   
>   	pdev = to_pci_dev(info->dev);
> -
> -	/* The PCIe spec, in its wisdom, declares that the behaviour of
> -	   the device if you enable PASID support after ATS support is
> -	   undefined. So always enable PASID support on devices which
> -	   have it, even if we can't yet know if we're ever going to
> -	   use it. */
> -	if (info->pasid_supported && !pci_enable_pasid(pdev, info->pasid_supported & ~1))
> -		info->pasid_enabled = 1;
> -
>   	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
>   	    !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) {
>   		info->ats_enabled = 1;
> @@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct device_domain_info *info)
>   		info->ats_enabled = 0;
>   		domain_update_iotlb(info->domain);
>   	}
> -
> -	if (info->pasid_enabled) {
> -		pci_disable_pasid(pdev);
> -		info->pasid_enabled = 0;
> -	}
>   }
>   
>   static void intel_flush_iotlb_all(struct iommu_domain *domain)
> @@ -4112,6 +4098,16 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
>   
>   	intel_iommu_debugfs_create_dev(info);
>   
> +	/*
> +	 * The PCIe spec, in its wisdom, declares that the behaviour of the
> +	 * device is undefined if you enable PASID support after ATS support.
> +	 * So always enable PASID support on devices which have it, even if
> +	 * we can't yet know if we're ever going to use it.
> +	 */
> +	if (info->pasid_supported &&
> +	    !pci_enable_pasid(pdev, info->pasid_supported & ~1))
> +		info->pasid_enabled = 1;
> +
>   	return &iommu->iommu;
>   free_table:
>   	intel_pasid_free_table(dev);
> @@ -4128,6 +4124,11 @@ static void intel_iommu_release_device(struct device *dev)
>   	struct device_domain_info *info = dev_iommu_priv_get(dev);
>   	struct intel_iommu *iommu = info->iommu;
>   
> +	if (info->pasid_enabled) {
> +		pci_disable_pasid(to_pci_dev(dev));
> +		info->pasid_enabled = 0;
> +	}
> +
>   	mutex_lock(&iommu->iopf_lock);
>   	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
>   		device_rbtree_remove(info);

-- 
Regards,
Yi Liu

Re: [PATCH v2 1/1] iommu/vt-d: Move PCI PASID enablement to probe path

[Jason Gunthorpe]

On Mon, Aug 19, 2024 at 01:18:05PM +0800, Lu Baolu wrote:
> Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is
> attached to the device and disabled when the device transitions to block
> translation mode. This approach is inappropriate as PCI PASID is a device
> feature independent of the type of the attached domain.
> 
> Enable PCI PASID during the IOMMU device probe and disables it during the
> release path.
> 
> Suggested-by: Yi Liu <yi.l.liu@intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> Reviewed-by: Yi Liu <yi.l.liu@intel.com>
> ---
>  drivers/iommu/intel/iommu.c | 29 +++++++++++++++--------------
>  1 file changed, 15 insertions(+), 14 deletions(-)
> 
> Change log:
> v2:
>  - Move pasid enablement the last step in intel_iommu_probe_device().
>  - Clear pasid enabled flags after pci_disable_pasid().

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason

Re: [PATCH v2 1/1] iommu/vt-d: Move PCI PASID enablement to probe path

[Baolu Lu]

On 8/19/24 1:18 PM, Lu Baolu wrote:
> Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is
> attached to the device and disabled when the device transitions to block
> translation mode. This approach is inappropriate as PCI PASID is a device
> feature independent of the type of the attached domain.
> 
> Enable PCI PASID during the IOMMU device probe and disables it during the
> release path.
> 
> Suggested-by: Yi Liu<yi.l.liu@intel.com>
> Signed-off-by: Lu Baolu<baolu.lu@linux.intel.com>
> Reviewed-by: Yi Liu<yi.l.liu@intel.com>
> ---
>   drivers/iommu/intel/iommu.c | 29 +++++++++++++++--------------
>   1 file changed, 15 insertions(+), 14 deletions(-)

Queued for v6.12-rc1.

Thanks,
baolu