[PATCH net 0/2] gre: merged GRE-over-FOU/GUE packets are dropped (bug report, possible fix)

[PATCH net 0/2] gre: merged GRE-over-FOU/GUE packets are dropped (bug report, possible fix)

[Brad Fitzpatrick via B4 Relay]

I hit this by accident while working on something else: bulk TCP through
GRE-over-FOU tunnels between some EC2 VMs crawled at about one segment per
round trip, and turning GRO off on the receiving NIC made it go away. I'm
not a kernel networking person, so I went to debug it with an LLM coding
agent, and it found what's below.

Please consider this series a bug report at best. Feel free to ignore it,
or to fix it some other way if this isn't the right fix; I may well have
this wrong. Empirically, though, patch 1 fixes the packet loss we were
seeing, both on our VMs and in the reproducer in patch 2.

What seems to be going on: when GRO merges GRE packets carried over UDP by
FOU or GUE, udp_gro_complete() sets SKB_GSO_UDP_TUNNEL, then
gre_gro_complete() assigns gso_type = SKB_GSO_GRE, which clears that bit.
The merged packet then can't be segmented where it needs to be:

  - at the encapsulation socket, since commit 3d010c8031e3 ("udp: do not
    accept non-tunnel GSO skbs landing in a tunnel") in v6.9, where
    udp_unexpected_gso() catches it and it's dropped and counted in
    UdpInErrors;
  - when forwarded out a device that can't segment it itself, since FOU
    gained GRO in commit afe93325bc02 ("fou: Add GRO support"), where
    udp4_ufo_fragment() fails on it.

Either way, only packets GRO didn't merge get through. Patch 1 ORs in
SKB_GSO_GRE instead, as ipip_gro_complete() and sit_gro_complete() do for
their GSO bit.

Patch 2 adds a selftest, fou_gre_gro.sh, that sends bulk TCP through GRE
over FOU and over GUE, each over IPv4 and IPv6, with GRO on at the
receiving veth, and checks that the receiver counts no UDP errors and that
GRO really merged packets. Without patch 1 all four cases fail; with it
they pass.

Tested under virtme-ng (QEMU/KVM, Ubuntu userspace), before and after, on
net's main (af32da41b032) and on v7.3-rc6, plus v6.8 for the forwarding
case. gre_gso.sh and fou_mcast_encap.sh give the same results before and
after. net/ipv4/ builds with no warnings under W=1 with both allmodconfig
and allyesconfig, before and after; I didn't build the rest of those
configs.

Disclosure, per Documentation/process/generated-content.rst: the agent was
Claude Code (Anthropic). Over a long session, I described the slow
traffic; it captured packets, traced the drops to udp_queue_rcv_skb() and
then to gre_gro_complete(), wrote the one-line change, the reproducer and
selftest, and original versions of the commit messages, and ran the tests.
I've reviewed the code: the new selftest entirely and confidently, and the
one-line operator fix seems at least plausible, given that the old code
doesn't match similar GRO-completing code in the net directory. But I'm no
expert in the net code or the life of GRO handling, so review cautiously.
Both patches carry Assisted-by: LLM.

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
---
Brad Fitzpatrick (2):
      gre: keep the UDP tunnel GSO type when completing GRO
      selftests: net: add a test for GRO of GRE over FOU and GUE

 net/ipv4/gre_offload.c                     |   2 +-
 tools/testing/selftests/net/Makefile       |   1 +
 tools/testing/selftests/net/fou_gre_gro.sh | 164 +++++++++++++++++++++++++++++
 3 files changed, 166 insertions(+), 1 deletion(-)
---
base-commit: af32da41b0327b9c6a37856ba82b6760d6c8d10e
change-id: 20261011-bradfitz-gre-fou-gro-b4-770f33e4e8af

Best regards,
--  
Brad Fitzpatrick <bradfitz@tailscale.com>

[PATCH net 1/2] gre: keep the UDP tunnel GSO type when completing GRO

[Brad Fitzpatrick via B4 Relay]

From: Brad Fitzpatrick <bradfitz@tailscale.com>

gre_gro_complete() assigns SKB_GSO_GRE to gso_type rather than OR-ing
it in. When GRE is carried over UDP by FOU or GUE, udp_gro_complete()
has already set SKB_GSO_UDP_TUNNEL{,_CSUM} before calling into the FOU
or GUE gro_complete handler and then gre_gro_complete(), so that bit is
lost.

That has broken forwarding of such packets since FOU gained GRO: if a
host with a FOU or GUE socket on the tunnel's port merges transit
packets and then forwards them out a device that can't segment them
itself, udp4_ufo_fragment() finds neither tunnel bit and fails, and
the packets are dropped. Since commit 3d010c8031e3 ("udp: do not accept
non-tunnel GSO skbs landing in a tunnel"), local receive is broken too:
udp_unexpected_gso() considers the merged skb, which reaches the
encapsulation socket without either UDP tunnel bit, unexpected and
hands it to udp_rcv_segment(). Segmenting fails, so the packet is
freed and counted in UdpInErrors. Either way, only packets GRO didn't
merge get through, so TCP over FOU or GUE GRE tunnels slows to about
one segment per round trip whenever the receiving device has GRO
enabled.

OR in SKB_GSO_GRE instead, as ipip_gro_complete() and
sit_gro_complete() do for SKB_GSO_IPXIP4. For plain GRE, gso_type is
still zero at this point unless the driver set it, so nothing changes
there in the usual case.

The assignment dates to commit bf5a755f5e91 ("net-gre-gro: Add GRE
support to the GRO stack"), but nothing set gso_type before it until
FOU GRO came along, so the Fixes tag points there.

Tested under virtme-ng with bulk TCP over a GRE tunnel between two
network namespaces joined by a veth, with GRO on the receiving veth and
segmentation off on the sending one, so GRO sees wire-sized packets.
32 MiB took:

                  before                  after
  FOU over IPv4   26.0s, 9658 UDP errors  0.28s, 0 errors
  GUE over IPv4   26.1s, 9686 UDP errors  0.39s, 0 errors
  FOU over IPv6   26.4s, 9860 UDP errors  0.16s, 0 errors
  plain GRE       0.17s                   0.15s

Forwarding the same FOU traffic through a third namespace with its own
FOU socket on the port, GRO on its ingress and segmentation off on its
egress, took 26.0s with 4826 egress drops before and 0.31s with none
after. On v6.8, before 3d010c8031e3, local receive was fine but
forwarding failed the same way (25.7s, 4764 drops).

With GRO off on the receiver, all took under 0.2s both before and
after. The gre_gso.sh and fou_mcast_encap.sh selftests give the same
results before and after.

Fixes: afe93325bc02 ("fou: Add GRO support")
Assisted-by: LLM
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
---
 net/ipv4/gre_offload.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/ipv4/gre_offload.c b/net/ipv4/gre_offload.c
index 5028c72d494a..c8129782d39d 100644
--- a/net/ipv4/gre_offload.c
+++ b/net/ipv4/gre_offload.c
@@ -242,7 +242,7 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff)
 	__be16 type;
 
 	skb->encapsulation = 1;
-	skb_shinfo(skb)->gso_type = SKB_GSO_GRE;
+	skb_shinfo(skb)->gso_type |= SKB_GSO_GRE;
 
 	type = greh->protocol;
 	if (greh->flags & GRE_KEY)

-- 
2.43.0

Re: [PATCH net 1/2] gre: keep the UDP tunnel GSO type when completing GRO

[Eric Dumazet]

Le dim. 11 oct. 2026 à 03:41, Brad Fitzpatrick via B4 Relay
<devnull+bradfitz.tailscale.com@kernel.org> a écrit :
>
> From: Brad Fitzpatrick <bradfitz@tailscale.com>
>
> gre_gro_complete() assigns SKB_GSO_GRE to gso_type rather than OR-ing
> it in. When GRE is carried over UDP by FOU or GUE, udp_gro_complete()
> has already set SKB_GSO_UDP_TUNNEL{,_CSUM} before calling into the FOU
> or GUE gro_complete handler and then gre_gro_complete(), so that bit is
> lost.
>
> That has broken forwarding of such packets since FOU gained GRO: if a
> host with a FOU or GUE socket on the tunnel's port merges transit
> packets and then forwards them out a device that can't segment them
> itself, udp4_ufo_fragment() finds neither tunnel bit and fails, and
> the packets are dropped. Since commit 3d010c8031e3 ("udp: do not accept
> non-tunnel GSO skbs landing in a tunnel"), local receive is broken too:
> udp_unexpected_gso() considers the merged skb, which reaches the
> encapsulation socket without either UDP tunnel bit, unexpected and
> hands it to udp_rcv_segment(). Segmenting fails, so the packet is
> freed and counted in UdpInErrors. Either way, only packets GRO didn't
> merge get through, so TCP over FOU or GUE GRE tunnels slows to about
> one segment per round trip whenever the receiving device has GRO
> enabled.
>
> OR in SKB_GSO_GRE instead, as ipip_gro_complete() and
> sit_gro_complete() do for SKB_GSO_IPXIP4. For plain GRE, gso_type is
> still zero at this point unless the driver set it, so nothing changes
> there in the usual case.
>
> The assignment dates to commit bf5a755f5e91 ("net-gre-gro: Add GRE
> support to the GRO stack"), but nothing set gso_type before it until
> FOU GRO came along, so the Fixes tag points there.
>
> Tested under virtme-ng with bulk TCP over a GRE tunnel between two
> network namespaces joined by a veth, with GRO on the receiving veth and
> segmentation off on the sending one, so GRO sees wire-sized packets.
> 32 MiB took:
>
>                   before                  after
>   FOU over IPv4   26.0s, 9658 UDP errors  0.28s, 0 errors
>   GUE over IPv4   26.1s, 9686 UDP errors  0.39s, 0 errors
>   FOU over IPv6   26.4s, 9860 UDP errors  0.16s, 0 errors
>   plain GRE       0.17s                   0.15s
>
> Forwarding the same FOU traffic through a third namespace with its own
> FOU socket on the port, GRO on its ingress and segmentation off on its
> egress, took 26.0s with 4826 egress drops before and 0.31s with none
> after. On v6.8, before 3d010c8031e3, local receive was fine but
> forwarding failed the same way (25.7s, 4764 drops).
>
> With GRO off on the receiver, all took under 0.2s both before and
> after. The gre_gso.sh and fou_mcast_encap.sh selftests give the same
> results before and after.
>
> Fixes: afe93325bc02 ("fou: Add GRO support")
> Assisted-by: LLM
> Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
> ---

Nice catch, thanks.

Reviewed-by: Eric Dumazet <edumazet@kernel.org>

[PATCH net 2/2] selftests: net: add a test for GRO of GRE over FOU and GUE

[Brad Fitzpatrick via B4 Relay]

From: Brad Fitzpatrick <bradfitz@tailscale.com>

Send bulk TCP through GRE tunnels carried over UDP by FOU and by GUE,
each over IPv4 and IPv6, with GRO on at the receiving veth and
segmentation off at the sender so the receiver gets wire-sized packets
to merge. Check that the receiver counts no UDP errors.

Verified under virtme-ng: without the previous patch, all four cases
fail, with the encapsulation socket counting every merged packet in
UdpInErrors or Udp6InErrors. With it, they pass in a few seconds.

Assisted-by: LLM
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
---
 tools/testing/selftests/net/Makefile       |   1 +
 tools/testing/selftests/net/fou_gre_gro.sh | 164 +++++++++++++++++++++++++++++
 2 files changed, 165 insertions(+)

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 590b33e16d9f..55101ed41c52 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -42,6 +42,7 @@ TEST_PROGS := \
 	fib_rule_tests.sh \
 	fib_tests.sh \
 	fin_ack_lat.sh \
+	fou_gre_gro.sh \
 	fou_mcast_encap.sh \
 	fq_band_pktlimit.sh \
 	gre_gso.sh \
diff --git a/tools/testing/selftests/net/fou_gre_gro.sh b/tools/testing/selftests/net/fou_gre_gro.sh
new file mode 100755
index 000000000000..21ed50f647ae
--- /dev/null
+++ b/tools/testing/selftests/net/fou_gre_gro.sh
@@ -0,0 +1,164 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Test that GRE packets carried over UDP (FOU or GUE) get through when the
+# receiver merges them with GRO.
+#
+# Send bulk TCP through such tunnels with GRO on at the receiver, and
+# segmentation off at the sender so the receiver gets wire-sized packets
+# to merge, and check that the receiver counts no UDP errors (UdpInErrors
+# or Udp6InErrors). Also check that GRO really merged packets, by comparing
+# the UDP datagrams the receiver delivered with the packets its veth
+# received, so that a setup where GRO doesn't engage skips rather than
+# passes.
+
+source lib.sh
+
+NSEND=""
+NRECV=""
+
+FOU_PORT=5555
+TCP_PORT=9000
+SIZE_MB=4
+
+TUN_S=192.168.99.1
+TUN_R=192.168.99.2
+
+cleanup() {
+	cleanup_all_ns
+}
+
+trap cleanup EXIT
+
+# setup_tunnel creates the namespaces, the veth between them and a GRE
+# tunnel over UDP encapsulation $1 (fou or gue) over IPv$2. It returns
+# $ksft_skip if the kernel or iproute2 can't set it up.
+setup_tunnel() {
+	local encap=$1
+	local family=$2
+	local type=gre
+	local addr_s=10.0.0.1
+	local addr_r=10.0.0.2
+	local fou_args="ipproto 47"
+
+	if [ "$encap" = gue ]; then
+		fou_args=gue
+	fi
+	if [ "$family" = 6 ]; then
+		type=ip6gre
+		addr_s=2001:db8::1
+		addr_r=2001:db8::2
+		fou_args="$fou_args -6"
+	fi
+
+	setup_ns NSEND NRECV || return "$ksft_skip"
+
+	ip link add veth_s netns "$NSEND" type veth \
+		peer name veth_r netns "$NRECV" || return "$ksft_skip"
+	ip -n "$NSEND" addr add 10.0.0.1/24 dev veth_s
+	ip -n "$NRECV" addr add 10.0.0.2/24 dev veth_r
+	ip -n "$NSEND" addr add 2001:db8::1/64 dev veth_s nodad
+	ip -n "$NRECV" addr add 2001:db8::2/64 dev veth_r nodad
+	ip -n "$NSEND" link set veth_s up
+	ip -n "$NRECV" link set veth_r up
+
+	ip netns exec "$NRECV" ethtool -K veth_r gro on || return "$ksft_skip"
+	ip netns exec "$NSEND" ethtool -K veth_s tso off gso off \
+		tx-gre-segmentation off tx-gre-csum-segmentation off \
+		tx-udp_tnl-segmentation off tx-udp_tnl-csum-segmentation off ||
+		return "$ksft_skip"
+
+	# shellcheck disable=SC2086 # $fou_args is several arguments
+	ip netns exec "$NSEND" ip fou add port "$FOU_PORT" $fou_args ||
+		return "$ksft_skip"
+	# shellcheck disable=SC2086
+	ip netns exec "$NRECV" ip fou add port "$FOU_PORT" $fou_args ||
+		return "$ksft_skip"
+	ip -n "$NSEND" link add gre_t type "$type" \
+		local "$addr_s" remote "$addr_r" \
+		encap "$encap" encap-sport auto encap-dport "$FOU_PORT" ||
+		return "$ksft_skip"
+	ip -n "$NRECV" link add gre_t type "$type" \
+		local "$addr_r" remote "$addr_s" \
+		encap "$encap" encap-sport auto encap-dport "$FOU_PORT" ||
+		return "$ksft_skip"
+
+	ip -n "$NSEND" addr add "$TUN_S/30" dev gre_t
+	ip -n "$NRECV" addr add "$TUN_R/30" dev gre_t
+	ip -n "$NSEND" link set gre_t up
+	ip -n "$NRECV" link set gre_t up
+}
+
+# recv_counter prints the sum of the receiver's nstat counters "$@".
+recv_counter() {
+	ip netns exec "$NRECV" nstat -az "$@" |
+		awk '!/^#/ { n += $2 } END { print n + 0 }'
+}
+
+run_test() {
+	local encap=$1
+	local family=$2
+	local name="GRO of GRE in ${encap^^} over IPv$family"
+	local datagrams
+	local packets
+	local errors
+	local pid
+	local rc
+
+	RET=0
+
+	if ! setup_tunnel "$encap" "$family"; then
+		log_test_skip "$name"
+		cleanup_all_ns
+		return
+	fi
+
+	ip netns exec "$NRECV" socat -u \
+		TCP4-LISTEN:"$TCP_PORT",bind="$TUN_R",reuseaddr OPEN:/dev/null &
+	pid=$!
+	wait_local_port_listen "$NRECV" "$TCP_PORT" tcp
+
+	# The timeout only bounds failing runs; passing ones take under a second.
+	dd if=/dev/zero bs=1M count="$SIZE_MB" 2>/dev/null |
+		ip netns exec "$NSEND" timeout 20 \
+			socat -u STDIN TCP4:"$TUN_R":"$TCP_PORT"
+	rc=$?
+
+	# Check the UDP error count rather than the duration, which depends
+	# on the machine.
+	errors=$(recv_counter UdpInErrors Udp6InErrors)
+	check_err "$errors" "$errors UDP errors at the receiver"
+	check_err "$rc" "transfer failed"
+
+	# Without errors, a pass only means something if GRO merged packets:
+	# then the encapsulation socket sees far fewer datagrams than the
+	# veth received packets.
+	datagrams=$(recv_counter UdpInDatagrams Udp6InDatagrams)
+	packets=$(ip netns exec "$NRECV" \
+		cat /sys/class/net/veth_r/statistics/rx_packets)
+	if [ "$RET" -eq 0 ] && [ $((datagrams * 2)) -gt "$packets" ]; then
+		ret_set_ksft_status "$ksft_skip" \
+			"GRO didn't merge: $datagrams UDP datagrams from $packets packets"
+	fi
+
+	log_test "$name"
+	kill "$pid" 2>/dev/null
+	wait "$pid" 2>/dev/null
+	cleanup_all_ns
+}
+
+require_command socat
+require_command ethtool
+require_command nstat
+
+modprobe -q fou
+modprobe -q ip_gre
+modprobe -q fou6
+modprobe -q ip6_gre
+
+run_test fou 4
+run_test gue 4
+run_test fou 6
+run_test gue 6
+
+exit "$EXIT_STATUS"

-- 
2.43.0

Re: [PATCH net 0/2] gre: merged GRE-over-FOU/GUE packets are dropped (bug report, possible fix)

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Sun, 11 Oct 2026 01:41:26 +0000 you wrote:
> I hit this by accident while working on something else: bulk TCP through
> GRE-over-FOU tunnels between some EC2 VMs crawled at about one segment per
> round trip, and turning GRO off on the receiving NIC made it go away. I'm
> not a kernel networking person, so I went to debug it with an LLM coding
> agent, and it found what's below.
> 
> Please consider this series a bug report at best. Feel free to ignore it,
> or to fix it some other way if this isn't the right fix; I may well have
> this wrong. Empirically, though, patch 1 fixes the packet loss we were
> seeing, both on our VMs and in the reproducer in patch 2.
> 
> [...]

Here is the summary with links:
  - [net,1/2] gre: keep the UDP tunnel GSO type when completing GRO
    https://git.kernel.org/netdev/net/c/718f364a20ec
  - [net,2/2] selftests: net: add a test for GRO of GRE over FOU and GUE
    https://git.kernel.org/netdev/net/c/d0eb9632fd98

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html