[RESEND PATCH] cdx: fix driver managed dma support

[RESEND PATCH] cdx: fix driver managed dma support

[Nipun Gupta]

The devices on cdx could be bound to drivers with the device
DMA managed by kernel drivers or user-space applications.
As multiple devices can be placed in the same IOMMU group, the
DMA on these devices must either be entirely under kernel control
or userspace control. Fix the CDX bus driver to acknowlege the
driver_managed_dma flag and call the appropriate iommu APIs.

Fixes: 2959ab247061 ("cdx: add the cdx bus driver")
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Reported-by: Alex Williamson <alex.williamson@redhat.com>
Closes: https://lore.kernel.org/lkml/20230524134831.28dc97e2.alex.williamson@redhat.com/
---

I have resend this patch to include fixes, closes and
reported by tags. We request that this fix be incorporated
into the 6.4 release, along with the inclusion of CDX bus
driver support.
Original patch was sent at:
https://lore.kernel.org/lkml/20230531114759.6173-1-nipun.gupta@amd.com/

 drivers/cdx/cdx.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/cdx/cdx.c b/drivers/cdx/cdx.c
index dc74fae02a78..42bbe5ed370d 100644
--- a/drivers/cdx/cdx.c
+++ b/drivers/cdx/cdx.c
@@ -63,6 +63,8 @@
 #include <linux/mm.h>
 #include <linux/xarray.h>
 #include <linux/cdx/cdx_bus.h>
+#include <linux/iommu.h>
+#include <linux/dma-map-ops.h>
 #include "cdx.h"
 
 /* Default DMA mask for devices on a CDX bus */
@@ -267,6 +269,7 @@ static void cdx_shutdown(struct device *dev)
 
 static int cdx_dma_configure(struct device *dev)
 {
+	struct cdx_driver *cdx_drv = to_cdx_driver(dev->driver);
 	struct cdx_device *cdx_dev = to_cdx_device(dev);
 	u32 input_id = cdx_dev->req_id;
 	int ret;
@@ -277,9 +280,23 @@ static int cdx_dma_configure(struct device *dev)
 		return ret;
 	}
 
+	if (!ret && !cdx_drv->driver_managed_dma) {
+		ret = iommu_device_use_default_domain(dev);
+		if (ret)
+			arch_teardown_dma_ops(dev);
+	}
+
 	return 0;
 }
 
+static void cdx_dma_cleanup(struct device *dev)
+{
+	struct cdx_driver *cdx_drv = to_cdx_driver(dev->driver);
+
+	if (!cdx_drv->driver_managed_dma)
+		iommu_device_unuse_default_domain(dev);
+}
+
 /* show configuration fields */
 #define cdx_config_attr(field, format_string)	\
 static ssize_t	\
@@ -503,6 +520,7 @@ struct bus_type cdx_bus_type = {
 	.remove		= cdx_remove,
 	.shutdown	= cdx_shutdown,
 	.dma_configure	= cdx_dma_configure,
+	.dma_cleanup	= cdx_dma_cleanup,
 	.bus_groups	= cdx_bus_groups,
 	.dev_groups	= cdx_dev_groups,
 };
-- 
2.17.1

Re: [RESEND PATCH] cdx: fix driver managed dma support

[Greg KH]

On Mon, Jun 05, 2023 at 06:40:09PM +0530, Nipun Gupta wrote:
> The devices on cdx could be bound to drivers with the device
> DMA managed by kernel drivers or user-space applications.
> As multiple devices can be placed in the same IOMMU group, the
> DMA on these devices must either be entirely under kernel control
> or userspace control. Fix the CDX bus driver to acknowlege the
> driver_managed_dma flag and call the appropriate iommu APIs.
> 
> Fixes: 2959ab247061 ("cdx: add the cdx bus driver")
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Reported-by: Alex Williamson <alex.williamson@redhat.com>
> Closes: https://lore.kernel.org/lkml/20230524134831.28dc97e2.alex.williamson@redhat.com/
> ---
> 
> I have resend this patch to include fixes, closes and
> reported by tags. We request that this fix be incorporated
> into the 6.4 release, along with the inclusion of CDX bus
> driver support.

What do you mean by "inclusion of CDX bus driver support"?  What patches
exactly?

confused,

greg k-h

Re: [RESEND PATCH] cdx: fix driver managed dma support

[Nipun Gupta]

On 6/5/2023 8:52 PM, Greg KH wrote:
> 
> On Mon, Jun 05, 2023 at 06:40:09PM +0530, Nipun Gupta wrote:
>> The devices on cdx could be bound to drivers with the device
>> DMA managed by kernel drivers or user-space applications.
>> As multiple devices can be placed in the same IOMMU group, the
>> DMA on these devices must either be entirely under kernel control
>> or userspace control. Fix the CDX bus driver to acknowlege the
>> driver_managed_dma flag and call the appropriate iommu APIs.
>>
>> Fixes: 2959ab247061 ("cdx: add the cdx bus driver")
>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>> Reported-by: Alex Williamson <alex.williamson@redhat.com>
>> Closes: https://lore.kernel.org/lkml/20230524134831.28dc97e2.alex.williamson@redhat.com/
>> ---
>>
>> I have resend this patch to include fixes, closes and
>> reported by tags. We request that this fix be incorporated
>> into the 6.4 release, along with the inclusion of CDX bus
>> driver support.
> 
> What do you mean by "inclusion of CDX bus driver support"?  What patches
> exactly?

Sorry for the confusion. I mean that CDX bus support would be coming up 
in 6.4 release and this patch (cdx: fix driver managed dma support) is a 
fix for the driver managed dma support in the bus. So if this patch can 
be part of 6.4 release (i.e. added in upcoming rc release/s for 6.4)?

Thanks,
Nipun

> 
> confused,
> 
> greg k-h

RE: [RESEND PATCH] cdx: fix driver managed dma support

[Agarwal, Nikhil]

Reviewed-by: Nikhil Agarwal <nikhil.agarwal@amd.com>

> -----Original Message-----
> From: Nipun Gupta <nipun.gupta@amd.com>
> Sent: Monday, June 5, 2023 6:40 PM
> To: gregkh@linuxfoundation.org; alex.williamson@redhat.com; jgg@ziepe.ca;
> linux-kernel@vger.kernel.org
> Cc: git (AMD-Xilinx) <git@amd.com>; Anand, Harpreet
> <harpreet.anand@amd.com>; Jansen Van Vuuren, Pieter <pieter.jansen-van-
> vuuren@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; Simek,
> Michal <michal.simek@amd.com>; Gupta, Nipun <Nipun.Gupta@amd.com>
> Subject: [RESEND PATCH] cdx: fix driver managed dma support
> 
> The devices on cdx could be bound to drivers with the device DMA managed
> by kernel drivers or user-space applications.
> As multiple devices can be placed in the same IOMMU group, the DMA on
> these devices must either be entirely under kernel control or userspace
> control. Fix the CDX bus driver to acknowlege the driver_managed_dma flag
> and call the appropriate iommu APIs.
> 
> Fixes: 2959ab247061 ("cdx: add the cdx bus driver")
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Reported-by: Alex Williamson <alex.williamson@redhat.com>
> Closes:
> https://lore.kernel.org/lkml/20230524134831.28dc97e2.alex.williamson@re
> dhat.com/
> ---
> 
> I have resend this patch to include fixes, closes and reported by tags. We
> request that this fix be incorporated into the 6.4 release, along with the
> inclusion of CDX bus driver support.
> Original patch was sent at:
> https://lore.kernel.org/lkml/20230531114759.6173-1-
> nipun.gupta@amd.com/
> 
>  drivers/cdx/cdx.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/drivers/cdx/cdx.c b/drivers/cdx/cdx.c index
> dc74fae02a78..42bbe5ed370d 100644
> --- a/drivers/cdx/cdx.c
> +++ b/drivers/cdx/cdx.c
> @@ -63,6 +63,8 @@
>  #include <linux/mm.h>
>  #include <linux/xarray.h>
>  #include <linux/cdx/cdx_bus.h>
> +#include <linux/iommu.h>
> +#include <linux/dma-map-ops.h>
>  #include "cdx.h"
> 
>  /* Default DMA mask for devices on a CDX bus */ @@ -267,6 +269,7 @@
> static void cdx_shutdown(struct device *dev)
> 
>  static int cdx_dma_configure(struct device *dev)  {
> +	struct cdx_driver *cdx_drv = to_cdx_driver(dev->driver);
>  	struct cdx_device *cdx_dev = to_cdx_device(dev);
>  	u32 input_id = cdx_dev->req_id;
>  	int ret;
> @@ -277,9 +280,23 @@ static int cdx_dma_configure(struct device *dev)
>  		return ret;
>  	}
> 
> +	if (!ret && !cdx_drv->driver_managed_dma) {
> +		ret = iommu_device_use_default_domain(dev);
> +		if (ret)
> +			arch_teardown_dma_ops(dev);
> +	}
> +
>  	return 0;
>  }
> 
> +static void cdx_dma_cleanup(struct device *dev) {
> +	struct cdx_driver *cdx_drv = to_cdx_driver(dev->driver);
> +
> +	if (!cdx_drv->driver_managed_dma)
> +		iommu_device_unuse_default_domain(dev);
> +}
> +
>  /* show configuration fields */
>  #define cdx_config_attr(field, format_string)	\
>  static ssize_t	\
> @@ -503,6 +520,7 @@ struct bus_type cdx_bus_type = {
>  	.remove		= cdx_remove,
>  	.shutdown	= cdx_shutdown,
>  	.dma_configure	= cdx_dma_configure,
> +	.dma_cleanup	= cdx_dma_cleanup,
>  	.bus_groups	= cdx_bus_groups,
>  	.dev_groups	= cdx_dev_groups,
>  };
> --
> 2.17.1