mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Simon Schippers <simon.schippers@tu-dortmund.de>
To: netdev@vger.kernel.org
Cc: "Andrew Lunn" <andrew+netdev@lunn.ch>,
	"David S . Miller" <davem@davemloft.net>,
	"Eric Dumazet" <edumazet@kernel.org>,
	"Jakub Kicinski" <kuba@kernel.org>,
	"Paolo Abeni" <pabeni@redhat.com>,
	"Jesper Dangaard Brouer" <hawk@kernel.org>,
	"Toshiaki Makita" <toshiaki.makita1@gmail.com>,
	"Jonas Köppeler" <j.koeppeler@tu-berlin.de>,
	"Alexei Starovoitov" <ast@kernel.org>,
	"Daniel Borkmann" <daniel@iogearbox.net>,
	"John Fastabend" <john.fastabend@gmail.com>,
	"Stanislav Fomichev" <sdf@fomichev.me>,
	simon.schippers@tu-dortmund.de, bpf@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH net 1/2] veth: wake a stopped peer txq beyond real_num_tx_queues
Date: Fri, 13 Nov 2026 11:46:47 +0100	[thread overview]
Message-ID: <20261113104648.225983-2-simon.schippers@tu-dortmund.de> (raw)
In-Reply-To: <20261113104648.225983-1-simon.schippers@tu-dortmund.de>

Since commit dc82a33297fc ("veth: apply qdisc backpressure on full
ptr_ring to reduce TX drops"), only the peer's veth_poll() wakes a txq
that veth_xmit() stopped on a full ptr_ring. veth_poll() skips txqs at
or above the peer's real_num_tx_queues.

If the peer shrinks its TX channels while such a txq is stopped, nothing
wakes it. netif_set_real_num_tx_queues() keeps its state, so once the
channels are grown back, its first skb is requeued forever. With a single
root qdisc this blocks all transmission of the device.

Look the txq up below num_tx_queues instead. Waking a txq that is not in
use is harmless.

Reproduced in virtme-ng with a UDP flood from veth0 into veth1, which has
GRO, while switching veth0 between one and two TX channels: without this
patch veth0 stopped transmitting within 11 cycles in 3 of 3 runs, with
it veth0 kept transmitting through 30 cycles in 3 of 3 runs.

Fixes: dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---

Notes:
    Reproducer (needs >= 4 CPUs, no debug hook needed). Run as root:
    
      ./repro-veth-chan-shrink-txq-stall.sh 30
    
    repro-veth-chan-shrink-txq-stall.sh:
    
      #!/bin/bash
      # SPDX-License-Identifier: GPL-2.0
      #
      # Reproducer: a veth txq stopped on a full ptr_ring stays stopped after the
      # device shrinks its TX channels and grows them back.
      #
      # veth0 (init netns, 2 txqs, pfifo qdisc, UDP flood) -> veth1 (netns rx, GRO
      # on, threaded NAPI). The ring fills, veth_xmit() returns NETDEV_TX_BUSY and
      # stops a veth0 txq until veth1's veth_poll() wakes it. veth0 is switched to
      # one TX channel and back to two. If txq 1 is stopped at the shrink,
      # veth_poll() skips it as it is beyond real_num_tx_queues. Once it is back in
      # use, its first skb is requeued forever and blocks the qdisc.
      #
      # Needs at least 4 CPUs. No debug hook needed.
      #
      # Usage: repro-veth-chan-shrink-txq-stall.sh [cycles]
      #
      # RULES (default 1000) nft rules in netns rx slow the consumer down.
      #
      # Exits 1 if veth0 got stuck, 0 if it survived all cycles, 2 if the ring never
      # filled.
    
      ITER=${1:-30}
      RULES=${RULES:-1000}
    
      cleanup() {
      	kill $(jobs -p) 2>/dev/null
      	wait 2>/dev/null
      	ip link del veth0 2>/dev/null
      	ip netns del rx 2>/dev/null
      }
      trap cleanup EXIT
    
      sent() { tc -s qdisc show dev veth0 | awk '/Sent/ { print $4; exit }'; }
      requeues() { tc -s qdisc show dev veth0 | sed -n 's/.*requeues \([0-9]*\).*/\1/p' | head -1; }
      rx() { ip netns exec rx "$@"; }
    
      ip netns add rx
      ip link add veth0 numtxqueues 2 numrxqueues 2 type veth \
      	peer name veth1 numtxqueues 2 numrxqueues 2 netns rx
      ip addr add 10.0.0.1/24 dev veth0
      rx ip addr add 10.0.0.2/24 dev veth1
      ip link set veth0 up
      rx ip link set veth1 up
      tc qdisc replace dev veth0 root pfifo limit 1000
      # veth only takes skbs to the ring without TSO, unless the peer has UDP GRO
      ethtool -K veth0 tso off
      rx ethtool -K veth1 gro on
      rx sh -c 'echo 1 > /sys/class/net/veth1/threaded'
      for p in $(pgrep napi/veth1); do taskset -pc 3 "$p" >/dev/null; done
    
      # Slow down the consumer so that the 256-entry rings fill up
      rx nft add table inet slow
      rx nft add chain inet slow in '{ type filter hook input priority 0; }'
      for i in $(seq 1 "$RULES"); do
      	echo "add rule inet slow in meta mark $i drop"
      done | rx nft -f -
      ping -c 1 -W 1 10.0.0.2 >/dev/null
    
      # One socket per flooder, so that they hash onto both txqs. Unlike pktgen they
      # keep sending while set_channels turns the carrier off.
      for f in 1 2 3 4; do
      	taskset -c $(((f - 1) % 3)) python3 -c '
      import socket
      s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
      s.setblocking(False)
      while True:
          try:
              s.sendto(b"x" * 18, ("10.0.0.2", 9))
          except OSError:
              pass' &
      done
      sleep 2
    
      echo "kernel $(uname -r), cycles $ITER"
      echo "after 2s flood: sent $(sent) pkts, requeues $(requeues)"
      if [ "$(requeues)" = 0 ]; then
      	echo "ring never filled, raise RULES"
      	exit 2
      fi
    
      for i in $(seq 1 "$ITER"); do
      	ethtool -L veth0 tx 1
      	ethtool -L veth0 tx 2
      	# give linkwatch time to bring the qdisc back after the carrier toggle
      	sleep 2
      	a=$(sent); sleep 1; b=$(sent)
      	if [ $((b - a)) -lt 1000 ]; then
      		echo "STUCK after $i TX shrink/grow cycles: sent $a -> $b in 1s"
      		sleep 3
      		echo "3s later: sent $b -> $(sent), $(ip -br link show veth0)"
      		exit 1
      	fi
      done
    
      echo "OK: veth0 kept sending through $ITER TX shrink/grow cycles," \
           "sent $(sent) pkts, requeues $(requeues)"
      exit 0

 drivers/net/veth.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index a61842b2585b..62a60a033675 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -984,7 +984,7 @@ static int veth_poll(struct napi_struct *napi, int budget)
 
 	/* NAPI functions as RCU section */
 	peer_dev = rcu_dereference_check(priv->peer, rcu_read_lock_bh_held());
-	peer_txq = (peer_dev && queue_idx < peer_dev->real_num_tx_queues) ?
+	peer_txq = (peer_dev && queue_idx < peer_dev->num_tx_queues) ?
 		   netdev_get_tx_queue(peer_dev, queue_idx) : NULL;
 
 	xdp_set_return_frame_no_direct();
-- 
2.43.0


  reply	other threads:[~2026-11-13 10:47 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13 10:46 [PATCH net 0/2] veth: fix txqs that are never woken after backpressure Simon Schippers
2026-11-13 10:46 ` Simon Schippers [this message]
2026-11-17 18:16   ` [PATCH net 1/2] veth: wake a stopped peer txq beyond real_num_tx_queues Tianyi Gao
2026-11-17 18:53     ` Simon Schippers
2026-11-13 10:46 ` [PATCH net 2/2] veth: wake the peer txq when deleting NAPI Simon Schippers
2026-11-17 18:16   ` Tianyi Gao

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113104648.225983-2-simon.schippers@tu-dortmund.de \
    --to=simon.schippers@tu-dortmund.de \
    --cc=andrew+netdev@lunn.ch \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=hawk@kernel.org \
    --cc=j.koeppeler@tu-berlin.de \
    --cc=john.fastabend@gmail.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=sdf@fomichev.me \
    --cc=toshiaki.makita1@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®