[PATCH v10 00/10] PCI: endpoint: Remote DMA support via vNTB

[Koichiro Den <den@valinux.co.jp>]

Hi,

This series lets host Linux use EPC-local DMA read (RC-to-EP) channels
through vNTB. I also tried a separate PCI DMA EPF, which reached v7 [1],
but came back to this vNTB-embedded approach after discussion.

An EPC-local DMA engine can use host DMA addresses (IOVAs when an IOMMU
is enabled) as PCI addresses to reach any host buffer mapped for the
endpoint. Letting the RC side drive its read channels also gives the RC
a direct path into endpoint-local buffers.

A major motivation is direct TX in ntb_transport. Both sides could use
the same dmaengine-based TX path (ie. symmetric), sending directly to RX
buffer DMA addresses supplied by the peer. This would avoid MW bounce
buffers and per-transfer MW remapping for payloads in both directions.

This assumes both sides trust each other. The RC side still programs DMA
over PCIe MMIO, so this is expected to pay off for large data transfers.

DesignWare eDMA is the only supported dmaengine backend for now. Other
EPC-local DMA engines can be added if there is a need and they have a
suitable programming interface.

This series does not include a consumer for the dmaengine channels exposed
to RC Linux. Native HDMA support is also left out to keep the series from
growing too large. I plan to send a follow-up series,
"PCI: endpoint: Support native HDMA for vNTB DMA",
once this series is acked or merged.

Setting pci-epf-vntb's 'dma_bar' configfs attribute enables DMA export.
The default is -1, which leaves it disabled. pci-epf-ntb only switches to
the shared control layout header and is functionally unchanged.

Based on v7.4-rc1. No external patch dependencies.

v10 uses the previously unused IRQ slot 1 for DMA, so enabling DMA no
longer takes a doorbell vector. EP behavior and the wire format are
unchanged.

Patch 9 needs another review. Using slot 1 makes the changes to
ntb_epf_init_isr() much smaller than in v9.


Proposed merge plan
===================

Patches 1-7 are for the PCI tree, and patches 8-10 for the NTB tree.
The NTB patches need the shared header as updated by patches 4-6.


Compatibility matrix
====================

"Old" means v7.4-rc1 without this series, and "new" means with it
applied. The table describes expected behavior.

  EP                              | Old host     | New host
  --------------------------------+--------------+----------
  Old                             | OK*          | OK*
  New (dma_bar = -1)              | OK*          | OK*
  New (dedicated DMA BAR)         | OK*          | OK
  New (shared MW/DMA BAR)         | OK**         | OK

  *  NTB works without DMA.
  ** NTB works without DMA. An "Old" host treats the whole shared BAR
     as an MW, so the EP falls back to that layout at LINK_UP and the
     DMA ranges are no longer mapped.


Testing with v10
================

The existing MW-based ntb_netdev path worked with DMA export disabled
(dma_bar = -1) and enabled with the following EP-side BAR settings:

* Dedicated DMA BAR configuration on RK3588 Rock 5B:

  ctrl_bar = 0
  mw1_bar  = 2
  db_bar   = 4
  dma_bar  = 5

* MW-shared DMA BAR configuration on R-Car S4 Spider:

  ctrl_bar = 0
  mw1_bar  = 2
  dma_bar  = 2
  db_bar   = 4

The ntb_netdev tests covered all matrix cases above involving a new
kernel (i.e. everything except the Old x Old pair).

Since this series includes no DMA consumer, the tests above are
regression tests for the existing MW path. With an out-of-tree NTB
transport that uses the exported DMA channels, I also confirmed that the
slot 1 DMA IRQ works.


Best regards,
Koichiro
---
Changes in v10:
  - Share IRQ slot 1 across DMA channels instead of taking a doorbell
    vector, and remove the spare-vector fallback. (Sashiko)
  - Add Frank's R-b tag for patch 6 and drop it from patch 9 after
    changing IRQ allocation.
  - Retest the compatibility matrix on R-Car S4 and RK3588.
  - Link to v9: https://lore.kernel.org/r/20261110082459.1798763-1-den@valinux.co.jp/

Changes in v9:
  - Rebase on v7.4-rc1. All prerequisites are now upstream.
  - Fix shared-MW size and alignment handling, and validate the shared
    BAR at bind time. (Sashiko)
  - Reject SETUP_DMA_BAR when DMA export is disabled. (Sashiko)
  - Retest the compatibility matrix on R-Car S4 and RK3588.
  - Link to v8: https://lore.kernel.org/r/20261022082230.3164834-1-den@valinux.co.jp/

Changes in v8:
  - Fall back to exposing the whole shared BAR as an MW for hosts
    without MW/DMA boundary support (ie. "Old" Host), instead of
    rejecting LINK_UP. (Frank)
  - Link to v7: https://lore.kernel.org/r/20261021042229.2280937-1-den@valinux.co.jp/

Changes in v7:
  - Split and reorder the patches for the PCI and NTB trees. (Frank)
  - Reject LINK_UP from hosts without MW/DMA boundary support when
    a BAR is shared. (Frank)
  - Link to v6: https://lore.kernel.org/r/20261020054948.1218370-1-den@valinux.co.jp/

Changes in v6:
  - Add a prep patch that moves the control region layout to
    include/linux/pci-epf-ntb.h, shared by pci-epf-ntb, pci-epf-vntb and
    ntb_hw_epf. (Frank)

For earlier changes, see v5:
https://lore.kernel.org/r/20261009045949.1109871-1-den@valinux.co.jp/

[1] Superseded approach, PCI DMA EPF, v7:
    https://lore.kernel.org/r/20260813063757.3131865-1-den@valinux.co.jp/


Koichiro Den (10):
  PCI: endpoint: Add DMA auxiliary resource metadata
  PCI: dwc: Expose endpoint DMA resources
  PCI: endpoint: pci-epf-vntb: Move epf_ntb_is_bar_used() up
  PCI: endpoint: Add pci-epf-ntb.h for the shared NTB control layout
  PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels
  PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR
  Documentation: PCI: endpoint: Document vNTB DMA export
  NTB: ntb_hw_epf: Use the shared NTB control layout
  NTB: ntb_hw_epf: Discover vNTB-embedded DMA
  NTB: ntb_hw_epf: Support DMA and MW sharing a BAR

 Documentation/PCI/endpoint/pci-vntb-howto.rst |  33 +-
 MAINTAINERS                                   |   1 +
 drivers/ntb/hw/epf/ntb_hw_epf.c               | 396 +++++++--
 .../pci/controller/dwc/pcie-designware-ep.c   | 107 ++-
 drivers/pci/endpoint/functions/pci-epf-ntb.c  |  85 +-
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 840 +++++++++++++++---
 include/linux/pci-epc.h                       |  34 +-
 include/linux/pci-epf-ntb.h                   |  90 ++
 8 files changed, 1311 insertions(+), 275 deletions(-)
 create mode 100644 include/linux/pci-epf-ntb.h


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.54.0