[PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device

[PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device

[Koichiro Den]

Hi,

Some NTB implementations are backed by a "virtual" PCI device, while the
actual DMA mapping context (IOMMU domain) belongs to a different device.

One example is vNTB, where the NTB device is represented as a virtual
PCI endpoint function, but DMA operations must be performed against the
EPC parent device, which owns the IOMMU context.

Currently, vNTB on an IOMMU-backed EPC does not work because the
"virtual" PCI device is used for DMA API operations, while the actual
IOMMU domain belongs to the EPC parent device. This series fixes that.

Note that there are two related but separable DMA-mapping issues in this
area:

  1). Selecting the correct struct device for DMA API operations used for
      MW translation setup and coherent buffer allocation.

  2). The existing RX-side dmaengine path in ntb_transport, where the MW
      source buffer may later be re-mapped against a per-queue dmaengine
      device ("double mapping"). The current physically-contiguous
      allocation workaround is about that existing issue.

This series is only about (1). It does not attempt to solve the larger
initialization-order / per-queue dmaengine issue behind (2), nor does it
change the current behavior of that path. In particular, this is not a
series to eliminate or redesign the existing "double mapping" behavior.

This series introduces an optional .get_dma_dev() callback in struct
ntb_dev_ops, together with a helper ntb_get_dma_dev(). If the callback
is not implemented, the helper falls back to the current default
behavior.  Drivers that implement .get_dma_dev() must return a non-NULL
struct device.

  - Patch 1/3: Add .get_dma_dev() to struct ntb_dev_ops and provide
               ntb_get_dma_dev().

  - Patch 2/3: Switch ntb_transport MW buffer allocation/free paths to use
               ntb_get_dma_dev().

  - Patch 3/3: Implement .get_dma_dev() for vNTB by returning the EPC
               parent device.

As suggested by Dave at:
https://lore.kernel.org/all/17cdfa38-16d5-49cc-ac61-20b606a4308b@intel.com/
I expect patch 1 and 2 to get NTB acks, patch 3 to get PCI EP ack, and
the full series to be taken through the PCI EP tree.

Best regards,
Koichiro

---
Changes since v1:
  - Reworked the series so that the actual user (vNTB) is included in
    the same series.
  - Clarified in the cover letter that this series only addresses the
    DMA-mapping device selection for MW translation, and does not
    attempt to solve the existing RX-side dmaengine "double mapping"
    issue.


Koichiro Den (3):
  NTB: core: Add .get_dma_dev() callback to ntb_dev_ops
  NTB: ntb_transport: Use ntb_get_dma_dev() for DMA buffers
  PCI: endpoint: pci-epf-vntb: Implement .get_dma_dev()

 drivers/ntb/ntb_transport.c                   | 14 +++++------
 drivers/pci/endpoint/functions/pci-epf-vntb.c |  9 ++++++++
 include/linux/ntb.h                           | 23 +++++++++++++++++++
 3 files changed, 39 insertions(+), 7 deletions(-)

-- 
2.51.0

[PATCH v2 1/3] NTB: core: Add .get_dma_dev() callback to ntb_dev_ops

[Koichiro Den]

Some NTB implementations are backed by a PCI function that is not the
right struct device to use with DMA API helpers (e.g. due to IOMMU
topology, or because the NTB device is virtual).

Add an optional .get_dma_dev() callback to struct ntb_dev_ops and
provide a helper, ntb_get_dma_dev(), so NTB clients can use the
appropriate struct device for DMA allocations and mappings.

If the callback is not implemented, ntb_get_dma_dev() returns the
current default (ntb->dev.parent). Drivers that implement .get_dma_dev()
must return a non-NULL device.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 include/linux/ntb.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index 8ff9d663096b..99209f957eb9 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -256,6 +256,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
  * @msg_clear_mask:	See ntb_msg_clear_mask().
  * @msg_read:		See ntb_msg_read().
  * @peer_msg_write:	See ntb_peer_msg_write().
+ * @get_dma_dev:	See ntb_get_dma_dev().
  */
 struct ntb_dev_ops {
 	int (*port_number)(struct ntb_dev *ntb);
@@ -329,6 +330,7 @@ struct ntb_dev_ops {
 	int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
 	u32 (*msg_read)(struct ntb_dev *ntb, int *pidx, int midx);
 	int (*peer_msg_write)(struct ntb_dev *ntb, int pidx, int midx, u32 msg);
+	struct device *(*get_dma_dev)(struct ntb_dev *ntb);
 };
 
 static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -391,6 +393,8 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
 		/* !ops->msg_clear_mask == !ops->msg_count	&& */
 		!ops->msg_read == !ops->msg_count		&&
 		!ops->peer_msg_write == !ops->msg_count		&&
+
+		/* ops->get_dma_dev is optional */
 		1;
 }
 
@@ -1563,6 +1567,25 @@ static inline int ntb_peer_msg_write(struct ntb_dev *ntb, int pidx, int midx,
 	return ntb->ops->peer_msg_write(ntb, pidx, midx, msg);
 }
 
+/**
+ * ntb_get_dma_dev() - get the device to use for DMA allocations/mappings
+ * @ntb:	NTB device context.
+ *
+ * Return a struct device suitable for DMA API allocations and mappings.
+ * This is typically the parent of the NTB device, but may be overridden by a
+ * driver by implementing .get_dma_dev().
+ * Drivers that implement .get_dma_dev() must return a non-NULL pointer.
+ *
+ * Return: device pointer to use for DMA operations.
+ */
+static inline struct device *ntb_get_dma_dev(struct ntb_dev *ntb)
+{
+	if (!ntb->ops->get_dma_dev)
+		return ntb->dev.parent;
+
+	return ntb->ops->get_dma_dev(ntb);
+}
+
 /**
  * ntb_peer_resource_idx() - get a resource index for a given peer idx
  * @ntb:	NTB device context.
-- 
2.51.0

[PATCH v2 2/3] NTB: ntb_transport: Use ntb_get_dma_dev() for DMA buffers

[Koichiro Den]

ntb_transport currently uses ndev->pdev->dev for coherent allocations
and frees.

Switch the coherent buffer allocation/free paths to use
ntb_get_dma_dev(), so ntb_transport can work with NTB implementations
where the NTB PCI function is not the right device to use for DMA
mappings.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/ntb/ntb_transport.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index 78e02fe6caba..a67cc26e47b9 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -759,13 +759,13 @@ static void ntb_transport_msi_desc_changed(void *data)
 static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
 {
 	struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
-	struct pci_dev *pdev = nt->ndev->pdev;
+	struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
 
 	if (!mw->virt_addr)
 		return;
 
 	ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
-	dma_free_coherent(&pdev->dev, mw->alloc_size,
+	dma_free_coherent(dma_dev, mw->alloc_size,
 			  mw->alloc_addr, mw->dma_addr);
 	mw->xlat_size = 0;
 	mw->buff_size = 0;
@@ -835,7 +835,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
 		      resource_size_t size)
 {
 	struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
-	struct pci_dev *pdev = nt->ndev->pdev;
+	struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
 	size_t xlat_size, buff_size;
 	resource_size_t xlat_align;
 	resource_size_t xlat_align_size;
@@ -864,12 +864,12 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
 	mw->buff_size = buff_size;
 	mw->alloc_size = buff_size;
 
-	rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+	rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
 	if (rc) {
 		mw->alloc_size *= 2;
-		rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+		rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
 		if (rc) {
-			dev_err(&pdev->dev,
+			dev_err(dma_dev,
 				"Unable to alloc aligned MW buff\n");
 			mw->xlat_size = 0;
 			mw->buff_size = 0;
@@ -882,7 +882,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
 	rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
 			      mw->xlat_size);
 	if (rc) {
-		dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
+		dev_err(dma_dev, "Unable to set mw%d translation", num_mw);
 		ntb_free_mw(nt, num_mw);
 		return -EIO;
 	}
-- 
2.51.0

[PATCH v2 3/3] PCI: endpoint: pci-epf-vntb: Implement .get_dma_dev()

[Koichiro Den]

When vNTB is used as a PCI endpoint function, the NTB device is backed
by a virtual PCI function. For DMA API allocations and mappings, NTB
clients must use the device that is associated with the IOMMU domain.

Implement ntb_dev_ops->get_dma_dev() for pci-epf-vntb and return the EPC
parent device.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 20a400e83439..e5433404f573 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -1436,6 +1436,14 @@ static int vntb_epf_link_disable(struct ntb_dev *ntb)
 	return 0;
 }
 
+static struct device *vntb_epf_get_dma_dev(struct ntb_dev *ndev)
+{
+	struct epf_ntb *ntb = ntb_ndev(ndev);
+	struct pci_epc *epc = ntb->epf->epc;
+
+	return epc->dev.parent;
+}
+
 static const struct ntb_dev_ops vntb_epf_ops = {
 	.mw_count		= vntb_epf_mw_count,
 	.spad_count		= vntb_epf_spad_count,
@@ -1457,6 +1465,7 @@ static const struct ntb_dev_ops vntb_epf_ops = {
 	.db_clear_mask		= vntb_epf_db_clear_mask,
 	.db_clear		= vntb_epf_db_clear,
 	.link_disable		= vntb_epf_link_disable,
+	.get_dma_dev		= vntb_epf_get_dma_dev,
 };
 
 static int pci_vntb_probe(struct pci_dev *pdev, const struct pci_device_id *id)
-- 
2.51.0

Re: [PATCH v2 3/3] PCI: endpoint: pci-epf-vntb: Implement .get_dma_dev()

[Dave Jiang]

On 3/5/26 8:14 PM, Koichiro Den wrote:
> When vNTB is used as a PCI endpoint function, the NTB device is backed
> by a virtual PCI function. For DMA API allocations and mappings, NTB
> clients must use the device that is associated with the IOMMU domain.
> 
> Implement ntb_dev_ops->get_dma_dev() for pci-epf-vntb and return the EPC
> parent device.
> 
> Suggested-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index 20a400e83439..e5433404f573 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -1436,6 +1436,14 @@ static int vntb_epf_link_disable(struct ntb_dev *ntb)
>  	return 0;
>  }
>  
> +static struct device *vntb_epf_get_dma_dev(struct ntb_dev *ndev)
> +{
> +	struct epf_ntb *ntb = ntb_ndev(ndev);
> +	struct pci_epc *epc = ntb->epf->epc;
> +
> +	return epc->dev.parent;
> +}
> +
>  static const struct ntb_dev_ops vntb_epf_ops = {
>  	.mw_count		= vntb_epf_mw_count,
>  	.spad_count		= vntb_epf_spad_count,
> @@ -1457,6 +1465,7 @@ static const struct ntb_dev_ops vntb_epf_ops = {
>  	.db_clear_mask		= vntb_epf_db_clear_mask,
>  	.db_clear		= vntb_epf_db_clear,
>  	.link_disable		= vntb_epf_link_disable,
> +	.get_dma_dev		= vntb_epf_get_dma_dev,
>  };
>  
>  static int pci_vntb_probe(struct pci_dev *pdev, const struct pci_device_id *id)

Re: [PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device

[Manivannan Sadhasivam]

On Fri, 06 Mar 2026 12:14:40 +0900, Koichiro Den wrote:
> Some NTB implementations are backed by a "virtual" PCI device, while the
> actual DMA mapping context (IOMMU domain) belongs to a different device.
> 
> One example is vNTB, where the NTB device is represented as a virtual
> PCI endpoint function, but DMA operations must be performed against the
> EPC parent device, which owns the IOMMU context.
> 
> [...]

Applied, thanks!

[1/3] NTB: core: Add .get_dma_dev() callback to ntb_dev_ops
      commit: af43a3a0c0ad524a69a7921efd0a04d50c03090b
[2/3] NTB: ntb_transport: Use ntb_get_dma_dev() for DMA buffers
      commit: c2fbbb8b341c58d24c7b114fae4cadd55a4660f1
[3/3] PCI: endpoint: pci-epf-vntb: Implement .get_dma_dev()
      commit: 185596ad93f545bcab2fea0dad6420c0c3cb386f

Best regards,
-- 
Manivannan Sadhasivam <mani@kernel.org>