From: Paolo Abeni <pabeni@redhat.com>
To: Stanislav Fomichev <sdf.kernel@gmail.com>
Cc: netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com,
kuba@kernel.org, michael.chan@broadcom.com,
pavan.chebbi@broadcom.com, andrew+netdev@lunn.ch,
edumazet@kernel.org, horms@kernel.org, shuah@kernel.org,
almasrymina@google.com, sdf@fomichev.me, leitao@debian.org,
dtatulea@nvidia.com, asml.silence@gmail.com,
bobbyeshleman@meta.com, daniel@iogearbox.net,
razor@blackwall.org, linux-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org
Subject: Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout
Date: Tue, 17 Nov 2026 11:23:25 +0100 [thread overview]
Message-ID: <09156786-c9c6-47e3-8674-816b80300a3f@redhat.com> (raw)
In-Reply-To: <avZZ-6GGciPC0Lti@devvm16445.scu0.facebook.com>
On 11/13/26 01:41, Stanislav Fomichev wrote:
> On 11/12, Stanislav Fomichev wrote:
>> On 11/12, Paolo Abeni wrote:
>>> I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
>>> complex and if you could do that, too.
>>
>> Originally I did not want to include bigtcp to keep it apples to apples:
>> same tso size, just 16 frags in skb vs 1 (in ideal case).
>>
>> I just tried running mlx5 with 45 frags configured and it fails to start with
>> the following error:
>> mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.
>>
>> (fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)
>>
>> What I also tried just now is `gso_max_size 524280` (on top of this series)
>> and it ups single flow throughput from 200Gbps to 240Gbps (presumably
>> by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
>> the sender softirq..
>>
>> I hope this gives you more data, not sure running with 20 frags is worth
>> it (vs 16)? LMK if you disagree or want me to run something else.
Thanks, the above is quite exhaustive :)
> Fun fact, also setting gro_max_size on the receiver gets line rate over
> single flow:
>
> I client.c: Connected 1:cpu 39 | 1:cpu 118
> client: == Source 2401:db00:255a:4a:bace::b
> client: Tx389.448 Gbps (2921248849920 bytes in 60008026 usec)
> client: Rx 0.000 Gbps (0 bytes in 60008026 usec)
> client: TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> client: 2 125 86 2 0 15300608 10142
> client: net CPU 39: usr: 0.03% sys: 0.06% idle:67.39% iow: 0.00% irq: 5.69% sirq:26.81%
> client: app CPU 40: usr: 0.05% sys: 2.91% idle:95.67% iow: 0.00% irq: 0.21% sirq: 1.14%
> client: == Target 2401:db00:255a:44:bace::b
> client: Tx 0.000 Gbps (0 bytes in 60009306 usec)
> client: Rx389.438 Gbps (2921238435008 bytes in 60009306 usec)
> client: TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> client: 0 0 210 122 0 999424 10
> client: net CPU118: usr: 0.00% sys: 0.00% idle: 1.60% iow: 0.00% irq: 0.33% sirq:98.05%
> client: app CPU119: usr: 2.75% sys:77.46% idle:18.95% iow: 0.00% irq: 0.47% sirq: 0.34%
>
> 4200MTU, 32K rx-buf-len, this patch series, gso/gro_mas_size of 524280,
> still the default number of frags (enough on rx to fill 512K GRO with 32K
> chunks).
That looks just great!
FTR, my understanding is that Jakub wants to say his last word here, so:
Acked-by: Paolo Abeni <pabeni@redhat.com>
next prev parent reply other threads:[~2026-11-17 10:23 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 22:47 Stanislav Fomichev
2026-11-11 22:47 ` [PATCH net-next v4 1/5] bnxt_en: let the stack segment GSO skbs with too many frags Stanislav Fomichev
2026-11-12 22:48 ` netdev-bot+sashiko
2026-11-11 22:47 ` [PATCH net-next v4 2/5] net: devmem: charge the frag length to TX skb truesize Stanislav Fomichev
2026-11-12 22:48 ` netdev-bot+sashiko
2026-11-11 22:47 ` [PATCH net-next v4 3/5] net: devmem: use niov_shift in net_devmem_get_niov_at() Stanislav Fomichev
2026-11-13 12:05 ` Breno Leitao
2026-11-11 22:47 ` [PATCH net-next v4 4/5] net: devmem: size TX net_iovs from the dmabuf sg table Stanislav Fomichev
2026-11-12 22:49 ` netdev-bot+sashiko
2026-11-13 12:10 ` Breno Leitao
2026-11-13 12:20 ` Breno Leitao
2026-11-11 22:47 ` [PATCH net-next v4 5/5] selftests: drv-net: devmem: TX from a hugepage-backed dmabuf Stanislav Fomichev
2026-11-12 22:49 ` netdev-bot+sashiko
2026-11-13 12:21 ` Breno Leitao
2026-11-12 8:25 ` [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout Paolo Abeni
2026-11-12 23:00 ` Stanislav Fomichev
2026-11-13 0:41 ` Stanislav Fomichev
2026-11-13 2:41 ` Mina Almasry
2026-11-13 17:02 ` Stanislav Fomichev
2026-11-17 10:23 ` Paolo Abeni [this message]
2026-11-18 1:18 ` Jakub Kicinski
2026-11-13 14:20 ` Nikolay Aleksandrov
2026-11-18 1:20 ` patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=09156786-c9c6-47e3-8674-816b80300a3f@redhat.com \
--to=pabeni@redhat.com \
--cc=almasrymina@google.com \
--cc=andrew+netdev@lunn.ch \
--cc=asml.silence@gmail.com \
--cc=bobbyeshleman@meta.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dtatulea@nvidia.com \
--cc=edumazet@google.com \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=michael.chan@broadcom.com \
--cc=netdev@vger.kernel.org \
--cc=pavan.chebbi@broadcom.com \
--cc=razor@blackwall.org \
--cc=sdf.kernel@gmail.com \
--cc=sdf@fomichev.me \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®