[BUG] vrf: attaching a packet tap to a VRF device corrupts skb->csum for packets from L3 ingress devices (Udp6InCsumErrors, packets dropped)

[BUG] vrf: attaching a packet tap to a VRF device corrupts skb->csum for packets from L3 ingress devices (Udp6InCsumErrors, packets dropped)

[Stefano Sasso]

Hi all,

Up front, so you know what you are dealing with: **I am not a kernel
programmer — I am a networking guy.** Please bear with me if I get the
terminology or the conventions wrong, and feel free to tell me bluntly if this
is not the right place or the right format. I also used an AI assistant to help
me navigate the kernel sources and narrow this down, so please treat my source
analysis below as a hypothesis rather than as an authoritative statement.

What I do have is a **small, deterministic, self-contained reproducer** that
triggers the problem 100% of the time on a stock kernel, plus a control test
that isolates the trigger condition. Those parts I am confident about, because I
ran them.

## Summary

Attaching a packet tap (tcpdump/dumpcap, i.e. anything that populates
`vrf_dev->ptype_all`) to a VRF device causes incoming UDP datagrams to be
dropped with `Udp6InCsumErrors`, when:

- the packet arrives on an ingress device with no `header_ops`
  (ARPHRD_NONE: tun, ip6tnl, gre, wireguard...) that is enslaved to
that VRF, and
- the skb is in `CHECKSUM_COMPLETE` state by the time it reaches
  `vrf_ip6_rcv()` (in my case because conntrack software-verified the checksum
  in PREROUTING).

Stop the capture and the packets are delivered again. Start it and they are
dropped again. Capturing on the enslaved ingress device instead of on the VRF
device is harmless.

IPv4 should be affected identically, since `vrf_ip_rcv()` uses the same helper.
I only tested IPv6.

## Real-world impact

I hit this in production: a SIP proxy bound into a VRF stopped receiving large
(fragmented) REGISTER messages over IPv6/UDP, but only while we were running a
packet capture on the VRF device to debug an unrelated issue. The capture we
took to diagnose the problem was the cause of the problem.

In the capture, the reassembled datagram as seen on the VRF device has
**source MAC == destination MAC == the VRF device MAC**, which is what put me
on the trail of `vrf_prepare_mac_header()`.

## Suspected cause

`drivers/net/vrf.c`, `vrf_prepare_mac_header()`:

```c
__skb_push(skb, ETH_HLEN);
eth = (struct ethhdr *)skb->data;

skb_reset_mac_header(skb);
skb_reset_mac_len(skb);

ether_addr_copy(eth->h_dest, vrf_dev->dev_addr);
ether_addr_copy(eth->h_source, vrf_dev->dev_addr);
eth->h_proto = htons(proto);

skb->protocol = eth->h_proto;
skb->pkt_type = PACKET_HOST;

skb_postpush_rcsum(skb, skb->data, ETH_HLEN);

skb_pull_inline(skb, ETH_HLEN);
```

The synthetic 14-byte Ethernet header is folded into `skb->csum` by
`skb_postpush_rcsum()`, then removed from the packet by `skb_pull_inline()` —
but there is no matching `skb_postpull_rcsum()`. For a `CHECKSUM_COMPLETE` skb,
`skb->csum` is left permanently off by `csum_partial(fake_eth, ETH_HLEN)`, and
`udpv6_rcv()` subsequently rejects the datagram.

This code is only reached from `vrf_ip6_rcv()` / `vrf_ip_rcv()` inside

```c
if (!list_empty(&vrf_dev->ptype_all)) {
```

which is exactly why the behaviour depends on whether a sniffer is attached.

`vrf_add_mac_header_if_unset()` short-circuits when the ingress device has a
real link layer (`dev_has_header(orig_dev)`), so a veth-enslaved port never
shows the problem — only L3 devices such as tun do.

Naively, the fix looks like it should be to make the accounting symmetric:
either drop the `skb_postpush_rcsum()` call, since the push/pull pair leaves the
packet unchanged, or add the matching `skb_postpull_rcsum()` before
`skb_pull_inline()`. I do not feel qualified to say which is correct, or whether
there are other callers/paths that rely on the current behaviour, so I am not
sending a patch.

## Why the packet is already reassembled when the VRF taps it

`l3mdev_ip6_rcv()` is called from `ip6_rcv_finish()`, i.e. after
`NF_INET_PRE_ROUTING`, so `nf_defrag_ipv6` (PREROUTING, prio -400) has already
reassembled the fragments. That is why the ingress device shows two fragments
while the VRF device shows one full datagram.

## Why the skb is CHECKSUM_COMPLETE

A tun device hands the stack a `CHECKSUM_NONE` skb, for which
`skb_postpush_rcsum()` is a no-op. In my setup the promotion is done by
conntrack: `nf_conntrack_udp_packet()` -> `nf_checksum()` -> `nf_ip6_checksum()`
-> `__skb_checksum_complete()`, which sets `CHECKSUM_COMPLETE` and stores
`skb->csum`. This also runs at PREROUTING, before the VRF rx handler, and only
when `net.netfilter.nf_conntrack_checksum=1` (the default).

Test 4 of the reproducer sets `net.netfilter.nf_conntrack_checksum=0` and the
problem disappears while the capture stays attached — which I believe confirms
that the corruption is in `skb->csum` and not in the packet data.

## Reproducer

Tested on `6.12.0-211.49.1.el10_2.x86_64` (Red Hat Enterprise Linux 10). It
needs root, `iproute2`, `nftables`, `tcpdump` and python3. No external network,
no second host — everything happens in one network namespace.

Topology:

```
[netns kbug]
  vrf0                 (table 100)
    └── tun0           IFF_TUN, ARPHRD_NONE, 2001:db8:2::1/64
```

`inject.py` opens `/dev/net/tun`, attaches to `tun0`, and writes two hand-built
IPv6 fragments of a single 1640-byte UDP datagram (correct UDP checksum) into
the fd. `receiver.py` is a UDP socket bound to `[2001:db8:2::1]:5060` with
`SO_BINDTODEVICE=vrf0`.

An nftables `ct state` rule in PREROUTING activates `nf_defrag_ipv6` and
conntrack:

```
add table inet ctstate
add chain inet ctstate prerouting { type filter hook prerouting
priority -150; policy accept; }
add rule inet ctstate prerouting ct state new,established,related,invalid accept
```

Results:

```
== Test 1 — no tap                                  PASS — delivered
(Udp6InCsumErrors +0)
== Test 2 — tcpdump on vrf0                         FAIL — dropped
(Udp6InCsumErrors +1)
== Test 3 — tcpdump on tun0 (ingress)               PASS — delivered
(Udp6InCsumErrors +0)
== Test 4 — tcpdump on vrf0, nf_conntrack_checksum=0 PASS — delivered
(Udp6InCsumErrors +0)
```

The full set of scripts (`setup.sh`, `inject.py`, `receiver.py`,
`reproduce.sh`, `teardown.sh`) is available here:

  https://gist.github.com/ssasso/c089227b116b0fa6a954a18d85639d67

They are short and have no dependencies beyond the above. I can inline them in a
follow-up mail if you prefer not to follow links.

## Workarounds I am using meanwhile

- Capture on the enslaved interface instead of on the VRF device.
- `sysctl -w net.netfilter.nf_conntrack_checksum=0` in the affected netns.

Happy to run any test, apply any debug patch, or collect any additional data you
need — just tell me exactly what to run.

Thanks for your time,
Stefano

Re: [BUG] vrf: attaching a packet tap to a VRF device corrupts skb->csum for packets from L3 ingress devices (Udp6InCsumErrors, packets dropped)

[Stefano Sasso]

Hello all,
  I would like to follow up on this, since it seems I got no responses.

> Attaching a packet tap (tcpdump/dumpcap, i.e. anything that populates
> `vrf_dev->ptype_all`) to a VRF device causes incoming UDP datagrams to be
> dropped with `Udp6InCsumErrors`, when:
>
> - the packet arrives on an ingress device with no `header_ops`
>   (ARPHRD_NONE: tun, ip6tnl, gre, wireguard...) that is enslaved to
> that VRF, and
> - the skb is in `CHECKSUM_COMPLETE` state by the time it reaches
>   `vrf_ip6_rcv()` (in my case because conntrack software-verified the checksum
>   in PREROUTING).
>
> Stop the capture and the packets are delivered again. Start it and they are
> dropped again. Capturing on the enslaved ingress device instead of on the VRF
> device is harmless.
>
> IPv4 should be affected identically, since `vrf_ip_rcv()` uses the same helper.
> I only tested IPv6.

Please let me know if there's anything else I can or need to do - or
if the previous mail is enough and now I only need to hope for a patch
:)

thanks again,
Stefano

Re: [BUG] vrf: attaching a packet tap to a VRF device corrupts skb->csum for packets from L3 ingress devices (Udp6InCsumErrors, packets dropped)

[David Ahern]

Adding Andrea who modified the suspected code at fault, and Ido.

I do not have time to look into this problem for at least the next 2
weeks. Maybe one of them can look into it.


On 9/5/26 6:59 AM, Stefano Sasso wrote:
> Hi all,
> 
> Up front, so you know what you are dealing with: **I am not a kernel
> programmer — I am a networking guy.** Please bear with me if I get the
> terminology or the conventions wrong, and feel free to tell me bluntly if this
> is not the right place or the right format. I also used an AI assistant to help
> me navigate the kernel sources and narrow this down, so please treat my source
> analysis below as a hypothesis rather than as an authoritative statement.
> 
> What I do have is a **small, deterministic, self-contained reproducer** that
> triggers the problem 100% of the time on a stock kernel, plus a control test
> that isolates the trigger condition. Those parts I am confident about, because I
> ran them.
> 
> ## Summary
> 
> Attaching a packet tap (tcpdump/dumpcap, i.e. anything that populates
> `vrf_dev->ptype_all`) to a VRF device causes incoming UDP datagrams to be
> dropped with `Udp6InCsumErrors`, when:
> 
> - the packet arrives on an ingress device with no `header_ops`
>   (ARPHRD_NONE: tun, ip6tnl, gre, wireguard...) that is enslaved to
> that VRF, and
> - the skb is in `CHECKSUM_COMPLETE` state by the time it reaches
>   `vrf_ip6_rcv()` (in my case because conntrack software-verified the checksum
>   in PREROUTING).
> 
> Stop the capture and the packets are delivered again. Start it and they are
> dropped again. Capturing on the enslaved ingress device instead of on the VRF
> device is harmless.
> 
> IPv4 should be affected identically, since `vrf_ip_rcv()` uses the same helper.
> I only tested IPv6.
> 
> ## Real-world impact
> 
> I hit this in production: a SIP proxy bound into a VRF stopped receiving large
> (fragmented) REGISTER messages over IPv6/UDP, but only while we were running a
> packet capture on the VRF device to debug an unrelated issue. The capture we
> took to diagnose the problem was the cause of the problem.
> 
> In the capture, the reassembled datagram as seen on the VRF device has
> **source MAC == destination MAC == the VRF device MAC**, which is what put me
> on the trail of `vrf_prepare_mac_header()`.
> 
> ## Suspected cause
> 
> `drivers/net/vrf.c`, `vrf_prepare_mac_header()`:
> 
> ```c
> __skb_push(skb, ETH_HLEN);
> eth = (struct ethhdr *)skb->data;
> 
> skb_reset_mac_header(skb);
> skb_reset_mac_len(skb);
> 
> ether_addr_copy(eth->h_dest, vrf_dev->dev_addr);
> ether_addr_copy(eth->h_source, vrf_dev->dev_addr);
> eth->h_proto = htons(proto);
> 
> skb->protocol = eth->h_proto;
> skb->pkt_type = PACKET_HOST;
> 
> skb_postpush_rcsum(skb, skb->data, ETH_HLEN);
> 
> skb_pull_inline(skb, ETH_HLEN);
> ```
> 
> The synthetic 14-byte Ethernet header is folded into `skb->csum` by
> `skb_postpush_rcsum()`, then removed from the packet by `skb_pull_inline()` —
> but there is no matching `skb_postpull_rcsum()`. For a `CHECKSUM_COMPLETE` skb,
> `skb->csum` is left permanently off by `csum_partial(fake_eth, ETH_HLEN)`, and
> `udpv6_rcv()` subsequently rejects the datagram.
> 
> This code is only reached from `vrf_ip6_rcv()` / `vrf_ip_rcv()` inside
> 
> ```c
> if (!list_empty(&vrf_dev->ptype_all)) {
> ```
> 
> which is exactly why the behaviour depends on whether a sniffer is attached.
> 
> `vrf_add_mac_header_if_unset()` short-circuits when the ingress device has a
> real link layer (`dev_has_header(orig_dev)`), so a veth-enslaved port never
> shows the problem — only L3 devices such as tun do.
> 
> Naively, the fix looks like it should be to make the accounting symmetric:
> either drop the `skb_postpush_rcsum()` call, since the push/pull pair leaves the
> packet unchanged, or add the matching `skb_postpull_rcsum()` before
> `skb_pull_inline()`. I do not feel qualified to say which is correct, or whether
> there are other callers/paths that rely on the current behaviour, so I am not
> sending a patch.
> 
> ## Why the packet is already reassembled when the VRF taps it
> 
> `l3mdev_ip6_rcv()` is called from `ip6_rcv_finish()`, i.e. after
> `NF_INET_PRE_ROUTING`, so `nf_defrag_ipv6` (PREROUTING, prio -400) has already
> reassembled the fragments. That is why the ingress device shows two fragments
> while the VRF device shows one full datagram.
> 
> ## Why the skb is CHECKSUM_COMPLETE
> 
> A tun device hands the stack a `CHECKSUM_NONE` skb, for which
> `skb_postpush_rcsum()` is a no-op. In my setup the promotion is done by
> conntrack: `nf_conntrack_udp_packet()` -> `nf_checksum()` -> `nf_ip6_checksum()`
> -> `__skb_checksum_complete()`, which sets `CHECKSUM_COMPLETE` and stores
> `skb->csum`. This also runs at PREROUTING, before the VRF rx handler, and only
> when `net.netfilter.nf_conntrack_checksum=1` (the default).
> 
> Test 4 of the reproducer sets `net.netfilter.nf_conntrack_checksum=0` and the
> problem disappears while the capture stays attached — which I believe confirms
> that the corruption is in `skb->csum` and not in the packet data.
> 
> ## Reproducer
> 
> Tested on `6.12.0-211.49.1.el10_2.x86_64` (Red Hat Enterprise Linux 10). It
> needs root, `iproute2`, `nftables`, `tcpdump` and python3. No external network,
> no second host — everything happens in one network namespace.
> 
> Topology:
> 
> ```
> [netns kbug]
>   vrf0                 (table 100)
>     └── tun0           IFF_TUN, ARPHRD_NONE, 2001:db8:2::1/64
> ```
> 
> `inject.py` opens `/dev/net/tun`, attaches to `tun0`, and writes two hand-built
> IPv6 fragments of a single 1640-byte UDP datagram (correct UDP checksum) into
> the fd. `receiver.py` is a UDP socket bound to `[2001:db8:2::1]:5060` with
> `SO_BINDTODEVICE=vrf0`.
> 
> An nftables `ct state` rule in PREROUTING activates `nf_defrag_ipv6` and
> conntrack:
> 
> ```
> add table inet ctstate
> add chain inet ctstate prerouting { type filter hook prerouting
> priority -150; policy accept; }
> add rule inet ctstate prerouting ct state new,established,related,invalid accept
> ```
> 
> Results:
> 
> ```
> == Test 1 — no tap                                  PASS — delivered
> (Udp6InCsumErrors +0)
> == Test 2 — tcpdump on vrf0                         FAIL — dropped
> (Udp6InCsumErrors +1)
> == Test 3 — tcpdump on tun0 (ingress)               PASS — delivered
> (Udp6InCsumErrors +0)
> == Test 4 — tcpdump on vrf0, nf_conntrack_checksum=0 PASS — delivered
> (Udp6InCsumErrors +0)
> ```
> 
> The full set of scripts (`setup.sh`, `inject.py`, `receiver.py`,
> `reproduce.sh`, `teardown.sh`) is available here:
> 
>   https://gist.github.com/ssasso/c089227b116b0fa6a954a18d85639d67
> 
> They are short and have no dependencies beyond the above. I can inline them in a
> follow-up mail if you prefer not to follow links.
> 
> ## Workarounds I am using meanwhile
> 
> - Capture on the enslaved interface instead of on the VRF device.
> - `sysctl -w net.netfilter.nf_conntrack_checksum=0` in the affected netns.
> 
> Happy to run any test, apply any debug patch, or collect any additional data you
> need — just tell me exactly what to run.
> 
> Thanks for your time,
> Stefano

Re: [BUG] vrf: attaching a packet tap to a VRF device corrupts skb->csum for packets from L3 ingress devices (Udp6InCsumErrors, packets dropped)

[Ido Schimmel]

On Sat, Sep 05, 2026 at 02:59:13PM +0200, Stefano Sasso wrote:
> Naively, the fix looks like it should be to make the accounting symmetric:
> either drop the `skb_postpush_rcsum()` call, since the push/pull pair leaves the
> packet unchanged, or add the matching `skb_postpull_rcsum()` before
> `skb_pull_inline()`. I do not feel qualified to say which is correct, or whether
> there are other callers/paths that rely on the current behaviour, so I am not
> sending a patch.

I believe your analysis is correct. I will test [1]. Drivers that
support CHECKSUM_COMPLETE don't include the checksum of the Ethernet
header in skb->csum.

[1]
diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c
index a0557a3a7026..d4dc6d690a75 100644
--- a/drivers/net/vrf.c
+++ b/drivers/net/vrf.c
@@ -1175,8 +1175,6 @@ static int vrf_prepare_mac_header(struct sk_buff *skb,
 	skb->protocol = eth->h_proto;
 	skb->pkt_type = PACKET_HOST;
 
-	skb_postpush_rcsum(skb, skb->data, ETH_HLEN);
-
 	skb_pull_inline(skb, ETH_HLEN);
 
 	return 0;