From: "Jonas Köppeler" <j.koeppeler@tu-berlin.de>
To: Simon Schippers <simon.schippers@tu-dortmund.de>,
<hawk@kernel.org>, <netdev@vger.kernel.org>
Cc: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Stanislav Fomichev <sdf@fomichev.me>,
<linux-kernel@vger.kernel.org>, <bpf@vger.kernel.org>
Subject: Re: [PATCH net-next v6 5/5] veth: time-based BQL completion coalescing via ethtool tx-usecs
Date: Wed, 3 Jun 2026 10:28:07 +0200 [thread overview]
Message-ID: <866cd367-5d53-46c3-b2f8-b0989a1d462d@tu-berlin.de> (raw)
In-Reply-To: <53dc682c-f3df-485a-91f6-74212554cd10@tu-dortmund.de>
On 6/2/26 17:37, Simon Schippers wrote:
> On 6/2/26 09:24, Jonas Köppeler wrote:
>> On 6/1/26 6:16 PM, Simon Schippers wrote:
>>
>>> On 6/1/26 16:03, Jonas Köppeler wrote:
>>>> On 6/1/26 2:00 PM, Simon Schippers wrote:
>>>>> On 5/28/26 09:46, Jonas Köppeler wrote:
>>>>>> On 5/27/26 3:54 PM, hawk@kernel.org wrote:
>>>>>>> From: Simon Schippers <simon.schippers@tu-dortmund.de>
>>>>>>>
>>>>>>> Per-packet BQL completion forces DQL to converge on limit=2, causing
>>>>>>> excessive NAPI scheduling overhead and qdisc requeues.
>>>>>>>
>>>>>>> Accumulate BQL completions and flush them when a configurable time
>>>>>>> threshold is exceeded, letting DQL discover a limit that bounds actual
>>>>>>> queuing delay to the configured interval. Coalescing state persists
>>>>>>> across NAPI polls in struct veth_rq so completions can accumulate
>>>>>>> beyond a single budget=64 cycle.
>>>>>>>
>>>>>>> Add ethtool tx-usecs support for runtime tuning. Default is 100 us;
>>>>>>> setting tx-usecs to 0 disables coalescing and falls back to per-packet
>>>>>>> completion.
>>>>>>>
>>>>>>> ethtool -C <veth-dev> tx-usecs 500 # 500us coalescing
>>>>>>> ethtool -C <veth-dev> tx-usecs 0 # per-packet (no coalescing)
>>>>>>>
>>>>>>> Co-developed-by: Jesper Dangaard Brouer <hawk@kernel.org>
>>>>>>> Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
>>>>>>> Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
>>>>>> Tested-by: Jonas Köppeler<j.koeppeler@tu-berlin.de>
>>>>>>
>>>>> Thanks for your testing!
>>>>>
>>>>> However, I have issues reproducing.
>>>>> I run bare metal (without virtme) with v6 + your pktgen patch
>>>>> and I am on the branch pktgen-and-benchmark, commit
>>>>> "results: add veth-bql measurements":
>>>>>
>>>>> 1. ping fails with 100% packet loss ~20% of the times with --pktgen.
>>>>> When this happens the avg ping of this run is mistakenly set
>>>>> to 0.0 ms, which distorts the results.
>>>>> I fixed it locally by rerunning when this happens.
>>>>>
>>>>> 2. pktgen runs with > 3 Mpps even with --nrules 10000, see log below.
>>>>> I see that this is because of qdisc drops.
>>>>> I also tried pfifo and sfq but with the same result.
>>>>> I spent quite some time on it but I do not know a fix.
>>>>>
>>>>> Do you have an idea?
>>>>> Thanks!
>>>> Hi,
>>>> yes there are some changes missing in the test script.
>>>> I have pushed it now, sorry. This should fix 1.
>>> I pulled it and ran...
>>>
>>> sudo ./veth_bql_sweep.sh --runs 1 --pktgen --duration 20 --qdisc fq_codel --no-bpftrace
>>>
>>> ... but still 8/32=1/4 of the pings are zero, I do not see
>>> a pattern.
>>>
>>>
>>> I grabbed the logs from /tmp and this is what a failing
>>> ping looks like:
>>>
>>> PING 10.99.0.2 (10.99.0.2) 56(84) bytes of data.
>>>
>>> --- 10.99.0.2 ping statistics ---
>>> 97 packets transmitted, 0 received, 100% packet loss, time 19967ms
>>>
>>>
>>> Feels like a race or something..
>>> Can you reproduce with the exact command?
>>> I think you need --runs 1, else it just averages over multiple
>>> runs.
>> Sorry, no I could not reproduce this. I used the exact same
>> command as you did, and I am using net-next/main + v6 patches.
>> I have 0% ping loss across all tests. Does the ping loss
>> happen regardless of the qdisc?
>>
> Yes, it happens for each qdisc I tested.
> As a fix I changed the script to rerun if this happens.
I was able to reproduce it and I think it is because you are lacking
support of the qdiscs, and thus you will fallback to pfifo, this
explains why it occasionally will drop all packets.
>
> With that I ran the benchmark and also created a script to have
> the result as an ASCII table.
> I think it would make sense to include something like this in
> the commit message.
>
> Throughput (pps)
> ==================================================
> nrules | 0us | 100us | 1000us | 10000us || stock
> -------+-------+-------+--------+---------++------
> 0 | 1.65M | 1.75M | 1.74M | 1.74M || 1.73M
> 100 | 684K | 755K | 730K | 728K || 744K
> 1000 | 119K | 126K | 126K | 125K || 126K
> 10000 | 13K | 12K | 13K | 13K || 13K
>
>
> Ping RTT ms (avg)
> ==================================================
> nrules | 0us | 100us | 1000us | 10000us || stock
> -------+-------+-------+--------+---------++------
> 0 | 0.016 | 0.138 | 0.137 | 0.135 || 0.133
> 100 | 0.029 | 0.185 | 0.310 | 0.315 || 0.310
> 1000 | 0.137 | 0.321 | 1.66 | 1.81 || 1.78
> 10000 | 1.22 | 1.87 | 3.02 | 16.0 || 17.2
Yes, good idea. I think we should report p99 + max_ping and not the avg.
>
>>>> Regarding 2.: do not look at the pktgen output, in the
>>>> new version you will see something like "goodput",
>>>> which is the number you should look for.
>>>> Pktgen will report at what speed it enqueued packets in
>>>> the qdisc.
>>> Exactly. Now it works. Had a single outlier but apart from that
>>> everything is fine.
>>>
>>> Thanks,
>>> Simon
>>>
>>>> Let me know if it worked.
>>>> Best,
>>>> Jonas
>>>>
next prev parent reply other threads:[~2026-06-03 8:28 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20260527135418.1166665-1-hawk@kernel.org>
2026-05-27 13:54 ` [PATCH net-next v6 1/5] net: add dev->bql flag to allow BQL sysfs for IFF_NO_QUEUE devices hawk
2026-05-27 13:54 ` [PATCH net-next v6 2/5] veth: implement Byte Queue Limits (BQL) for latency reduction hawk
2026-05-28 7:45 ` Jonas Köppeler
2026-06-04 8:19 ` Paolo Abeni
2026-06-10 12:21 ` Jesper Dangaard Brouer
2026-05-27 13:54 ` [PATCH net-next v6 3/5] veth: add tx_timeout watchdog as BQL safety net hawk
2026-06-04 8:24 ` Paolo Abeni
2026-06-10 12:37 ` Jesper Dangaard Brouer
2026-05-27 13:54 ` [PATCH net-next v6 4/5] net: sched: add timeout count to NETDEV WATCHDOG message hawk
2026-06-04 8:30 ` Paolo Abeni
2026-06-12 9:39 ` Jesper Dangaard Brouer
2026-05-27 13:54 ` [PATCH net-next v6 5/5] veth: time-based BQL completion coalescing via ethtool tx-usecs hawk
2026-05-28 7:46 ` Jonas Köppeler
2026-06-01 12:00 ` Simon Schippers
2026-06-01 14:03 ` Jonas Köppeler
2026-06-01 16:16 ` Simon Schippers
2026-06-02 7:24 ` Jonas Köppeler
2026-06-02 15:37 ` Simon Schippers
2026-06-03 8:28 ` Jonas Köppeler [this message]
2026-05-29 14:51 ` Simon Schippers
2026-06-04 8:21 ` Paolo Abeni
2026-06-08 10:38 ` Simon Schippers
2026-06-08 13:04 ` Jesper Dangaard Brouer
2026-06-08 13:13 ` Jonas Köppeler
2026-06-08 14:21 ` Simon Schippers
2026-06-09 13:59 ` Jesper Dangaard Brouer
2026-06-09 15:08 ` Simon Schippers
2026-06-10 7:04 ` Simon Schippers
2026-06-10 10:15 ` Jesper Dangaard Brouer
2026-06-10 12:00 ` Simon Schippers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=866cd367-5d53-46c3-b2f8-b0989a1d462d@tu-berlin.de \
--to=j.koeppeler@tu-berlin.de \
--cc=andrew+netdev@lunn.ch \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hawk@kernel.org \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sdf@fomichev.me \
--cc=simon.schippers@tu-dortmund.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®