mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout
@ 2026-11-11 22:47 Stanislav Fomichev
  2026-11-11 22:47 ` [PATCH net-next v4 1/5] bnxt_en: let the stack segment GSO skbs with too many frags Stanislav Fomichev
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Stanislav Fomichev @ 2026-11-11 22:47 UTC (permalink / raw)
  To: netdev
  Cc: davem, edumazet, kuba, pabeni, michael.chan, pavan.chebbi,
	andrew+netdev, edumazet, horms, shuah, almasrymina, sdf, leitao,
	dtatulea, asml.silence, bobbyeshleman, daniel, razor,
	linux-kernel, linux-kselftest

TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
68KB, which makes BIG TCP ineffective for devmem TX.

This series picks the TX net_iov size from the dmabuf sg table instead:
the largest power of 2, up to 2MB, which aligns every DMA segment. With
a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
no UAPI change since TX addresses are dmabuf offsets.

This doesn't make skbs larger. TCP still sizes them with size_goal.
Only the frags get larger, and a single frag can now cover the whole
skb.

An skb stays under 64KB unless BIG TCP is enabled. With BIG TCP,
MSG_ZEROCOPY from huge pages can already create frags of 64KB or
more. So drivers already have to handle them:
- mlx5 has a 31-bit length per data segment
- gve splits frags into 16KB descriptors
- fbnic doesn't enable BIG TCP
- bnxt has a 16-bit BD length. It is broken for such page frags today
  and will be fixed separately.

Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
kernel on the sender only. 1 flow, MTU 4200, receiver tcp_rmem capped
at 16MB so it doesn't drop, median of 5 x 60s runs:

                              before    after
  frags per skb (sender)       16-17      1-2
  throughput (Gb/s)            208.6    211.2
  sender sendmsg CPU (sys)     40.8%    19.5%   (-52%)

The receiver's CPU is the bottleneck, so the gain is CPU, not
throughput. More of the transmit runs from the ACK path, so the
sender's softirq grows (52% -> 59%), but sys + sirq still drops by 16%.

kperf output (sender side) of runs near the median, before:

  client: == Source <sender>
  client:   Tx213.826 Gbps (1603901849600 bytes in 60007723 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60007723 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:             0     0 633      2    0 15556608 7657
  client:   net CPU 49: usr: 0.01% sys: 0.10% idle:44.13% iow: 0.00% irq: 3.21% sirq:52.53%
  client:   app CPU 50: usr: 0.05% sys:41.55% idle:53.88% iow: 0.00% irq: 1.40% sirq: 3.10%

after:

  client: == Source <sender>
  client:   Tx211.242 Gbps (1584440279040 bytes in 60004741 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60004741 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:             0     0 556      3    0 15499264 7650
  client:   net CPU 13: usr: 0.01% sys: 0.08% idle:37.75% iow: 0.00% irq: 3.87% sirq:58.26%
  client:   app CPU 14: usr: 0.06% sys:19.48% idle:77.23% iow: 0.00% irq: 1.19% sirq: 2.02%

The bnxt patch was tested on two 50Gb/s bnxt hosts with
CONFIG_MAX_SKB_FRAGS=45. With the default of 17 its check compiles
out. ncdevmem sending 1KB chunks builds skbs with up to 45 frags. The
stack segments them and the data arrives intact.

Patches:
- 1: bnxt: segment GSO skbs with too many frags in the stack
- 2-3: devmem TX prep (truesize, niov_shift in the TX lookup)
- 4: pick the TX net_iov size from the sg table
- 5: selftest for TX from a hugepage-backed dmabuf

v4:
- drop the per-device frag size limit and the bnxt, fbnic and mlx5
  patches. Drivers already have to handle large page frags (Jakub).

v3: https://lore.kernel.org/netdev/20261105222058.944293-1-sdf@fomichev.me/
- default the netmem TX frag size limit to PAGE_SIZE, opt in bnxt,
  fbnic and mlx5 (Mina)

v2: https://lore.kernel.org/netdev/20261103235152.1904727-1-sdf@fomichev.me/
- reword the TX truesize commit message (Mina)
- explain why the TX net_iov size ignores the device limit (Mina)
- use 1 flow results in the cover letter (Mina)

v1: https://lore.kernel.org/netdev/20261102191438.1059771-1-sdf@fomichev.me/

Stanislav Fomichev (5):
  bnxt_en: let the stack segment GSO skbs with too many frags
  net: devmem: charge the frag length to TX skb truesize
  net: devmem: use niov_shift in net_devmem_get_niov_at()
  net: devmem: size TX net_iovs from the dmabuf sg table
  selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  5 +++
 net/core/datagram.c                           |  2 +-
 net/core/devmem.c                             | 36 +++++++++++++---
 net/core/netdev-genl.c                        |  2 +-
 .../selftests/drivers/net/hw/devmem.py        | 10 ++++-
 .../selftests/drivers/net/hw/devmem_lib.py    | 43 +++++++++++++++++--
 .../selftests/drivers/net/hw/ncdevmem.c       |  8 +++-
 .../selftests/drivers/net/hw/nk_devmem.py     | 11 ++++-
 8 files changed, 101 insertions(+), 16 deletions(-)

-- 
2.53.0-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-11 22:47 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-11 22:47 [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 1/5] bnxt_en: let the stack segment GSO skbs with too many frags Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 2/5] net: devmem: charge the frag length to TX skb truesize Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 3/5] net: devmem: use niov_shift in net_devmem_get_niov_at() Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 4/5] net: devmem: size TX net_iovs from the dmabuf sg table Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 5/5] selftests: drv-net: devmem: TX from a hugepage-backed dmabuf Stanislav Fomichev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®