Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 11/12, Stanislav Fomichev wrote:
> On 11/12, Paolo Abeni wrote:
> > Hi,
> > 
> > A few (late) questions/comments to check I'm not missing relevant
> > points,
> > 
> > On 11/11/26 23:47, Stanislav Fomichev wrote:
> > > TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> > > A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> > > 68KB, which makes BIG TCP ineffective for devmem TX.
> > 
> > ... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
> > ~180K...
> > > This series picks the TX net_iov size from the dmabuf sg table instead:
> > > the largest power of 2, up to 2MB, which aligns every DMA segment. With
> > > a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> > > no UAPI change since TX addresses are dmabuf offsets.
> > 
> > ... still this option should be more effective because the limit would be
> > higher and the skb would use less cachelines for the shared_info,
> > 
> > I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
> > complex and if you could do that, too.
> 
> Originally I did not want to include bigtcp to keep it apples to apples:
> same tso size, just 16 frags in skb vs 1 (in ideal case).
> 
> I just tried running mlx5 with 45 frags configured and it fails to start with
> the following error:
> mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.
> 
> (fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)
> 
> What I also tried just now is `gso_max_size 524280` (on top of this series)
> and it ups single flow throughput from 200Gbps to 240Gbps (presumably
> by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
> the sender softirq..
> 
> I hope this gives you more data, not sure running with 20 frags is worth
> it (vs 16)? LMK if you disagree or want me to run something else.

Fun fact, also setting gro_max_size on the receiver gets line rate over
single flow:

I   client.c: Connected 1:cpu 39 | 1:cpu 118
client: == Source 2401:db00:255a:4a:bace::b
client:   Tx389.448 Gbps (2921248849920 bytes in 60008026 usec)
client:   Rx  0.000 Gbps (0 bytes in 60008026 usec)
client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
client:             2   125  86      2    0 15300608 10142
client:   net CPU 39: usr: 0.03% sys: 0.06% idle:67.39% iow: 0.00% irq: 5.69% sirq:26.81%
client:   app CPU 40: usr: 0.05% sys: 2.91% idle:95.67% iow: 0.00% irq: 0.21% sirq: 1.14%
client: == Target 2401:db00:255a:44:bace::b
client:   Tx  0.000 Gbps (0 bytes in 60009306 usec)
client:   Rx389.438 Gbps (2921238435008 bytes in 60009306 usec)
client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
client:             0     0 210    122    0  999424   10
client:   net CPU118: usr: 0.00% sys: 0.00% idle: 1.60% iow: 0.00% irq: 0.33% sirq:98.05%
client:   app CPU119: usr: 2.75% sys:77.46% idle:18.95% iow: 0.00% irq: 0.47% sirq: 0.34%

4200MTU, 32K rx-buf-len, this patch series, gso/gro_mas_size of 524280,
still the default number of frags (enough on rx to fill 512K GRO with 32K
chunks).