[PATCH net-next v1 00/11] xsk: Zero copy through page-pool memory providers

["Björn Töpel" <bjorn@kernel.org>]

Hi!

AF_XDP zero copy needs a second RX allocator in every driver. This
series makes UMEM a page-pool memory provider instead: a page_pool
driver opts in, and keeps its refill, recycling and DMA sync code.
fbnic is the first user. Patches 1-7 are core, 8-11 fbnic.

Since the RFC [RFC], net_iov is a first class netmem, the core owns
AF_XDP setup, and a refill contract replaces the NAPI kicks. The fixes
are out.

Changes since the RFC:

 - A readable net_iov has an address. netmem_address() and
   skb_frag_address() find it through struct net_iov_area, which now
   holds niov_shift. xdp_frag_address() is gone. (Mina, Stanislav)
 - PP_FLAG_ALLOW_UNREADABLE_NETMEM keeps its meaning. A pool takes a
   readable provider only with the new PP_FLAG_MEM_PROVIDER. (Mina)
 - The core installs the provider at bind, through the queue API, as
   for devmem and io_uring. A driver sets
   netdev_queue_mgmt_ops.xsk_page_pool, and checks chunk size,
   headroom and multi-buffer mode in XDP_VALIDATE_XSK_POOL. MTU and
   XDP program changes of a bound device get the same check.
   (Stanislav)
 - No NAPI kicks. Drivers report the end of each refill with
   page_pool_refill_done(), and the core publishes the queue's NAPI
   for provider and socket wakeups, so no ndo_xsk_wakeup. (Mina)
 - Page pools declare the RX buffers they supply, header or payload.
   A per-queue generation tells new pools from old ones in the restart
   check.
 - MP_CAP_RETAIN says whether the stack may keep provider buffers.
   The stack copies the others on XDP_PASS, frame conversion and
   non-XSKMAP redirects. Frame copies are linear, so multi-buffer
   frames to devmap and cpumap are dropped.
 - xdp_buff does not grow. The head netmem and shared info live in
   driver memory, reached through a union with txq.
 - The buffer pool allocates xdp_buff_xsk only for copy mode and
   driver zero copy: 128 bytes less per chunk. (Stanislav)
 - With XDP_USE_NEED_WAKEUP the FILL flag stays set. Setting it only
   when starved races with userspace.
 - fbnic is split in four: XDP redirect, netmem heads and XDP_TX,
   unsplittable buffers, zero copy.
 - The RFC's fixes are out. The fbnic one is upstream, and the skb
   head sizing is Clement's fix [1]. Other fixes found on the way go
   to net separately. (Stanislav, Maciej)

Unchanged: aligned chunks only (4 KiB on fbnic), and generic XDP and
CPUMAP cannot feed a provider socket.

fbnic, one queue, 66-byte packets: XSK RX 21.2 Mpps (17.0 with epoll),
XSK TX 28.9 Mpps, XDP_DROP costs 1.9%, 30 instructions per packet,
against net-next. Tested on fbnic hardware and the fbnic QEMU model.

Still open from the RFC: how should userspace learn the chunk
constraints before bind?

netkit queue leases and bnxt come later.


Björn

[RFC] https://lore.kernel.org/netdev/20261002190018.696925-1-bjorn@kernel.org/
[1] https://lore.kernel.org/bpf/20260629072300.1664622-1-clecigne@google.com/

Björn Töpel (11):
  net: Get skb fragment addresses through netmem
  net: Add memory provider capabilities
  xdp: Copy buffers with restricted lifetime
  xsk: Allocate UMEM chunks through page_pool
  xsk: Install UMEM page pool providers in the core
  page_pool: Coordinate refill with memory providers
  xsk: Deliver provider chunks without copying
  eth: fbnic: Support XDP redirect
  eth: fbnic: Use netmem for RX headers and XDP_TX
  eth: fbnic: Support RX buffers it cannot split
  eth: fbnic: Support AF_XDP page pool zero copy

 Documentation/networking/af_xdp.rst           |  64 +-
 Documentation/networking/netmem.rst           |  15 +
 Documentation/networking/xsk-tx-metadata.rst  |   3 +-
 .../net/ethernet/meta/fbnic/fbnic_netdev.c    |  22 +-
 drivers/net/ethernet/meta/fbnic/fbnic_txrx.c  | 601 ++++++++++++---
 drivers/net/ethernet/meta/fbnic/fbnic_txrx.h  |  28 +
 drivers/net/netkit.c                          |   3 +-
 include/linux/netdevice.h                     |   8 +
 include/linux/skbuff.h                        |  22 +-
 include/net/netdev_queues.h                   |  12 +
 include/net/netdev_rx_queue.h                 |   8 +
 include/net/netmem.h                          |  41 +-
 include/net/page_pool/helpers.h               |  38 +-
 include/net/page_pool/memory_provider.h       |  21 +
 include/net/page_pool/types.h                 |  33 +-
 include/net/xdp.h                             |  94 ++-
 include/net/xdp_sock.h                        |   4 +
 include/net/xdp_sock_drv.h                    |  45 ++
 include/net/xsk_buff_pool.h                   |   3 +
 io_uring/zcrx.c                               |   2 +
 net/core/dev.c                                |  30 +-
 net/core/dev.h                                |   8 +
 net/core/devmem.c                             |   9 +-
 net/core/devmem.h                             |   4 +-
 net/core/filter.c                             |  20 +-
 net/core/netdev_config.c                      |   2 +
 net/core/netdev_rx_queue.c                    | 143 +++-
 net/core/page_pool.c                          | 100 ++-
 net/core/page_pool_user.c                     |  25 +-
 net/core/xdp.c                                |  78 +-
 net/ethtool/rings.c                           |   7 +-
 net/xdp/Kconfig                               |   1 +
 net/xdp/Makefile                              |   2 +-
 net/xdp/xsk.c                                 |  86 ++-
 net/xdp/xsk.h                                 |  15 +
 net/xdp/xsk_buff_pool.c                       |  72 +-
 net/xdp/xsk_pp.c                              | 697 ++++++++++++++++++
 net/xdp/xsk_queue.h                           |  18 +-
 38 files changed, 2159 insertions(+), 225 deletions(-)
 create mode 100644 net/xdp/xsk_pp.c


base-commit: 2d916a8c307c31b433318509bf0311d52d509f09
-- 
2.56.0