[PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

This series makes IOAM insertion transparent to IPv6 ECMP, as required
by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
along the same path as the identical packet without them.

Patch 1 fixes the transit side: hash policy 0 hashed the raw IPv6
nexthdr, which becomes NEXTHDR_HOP once an IOAM Hop-by-Hop header is
inserted. The policy-0 key (and the ICMPv6-error flow selection) now
uses the nexthdr the packet would have had if the Hop-by-Hop header
carried only IOAM options and padding.

Patch 2 fixes the injecting node: the ioam6 output re-lookup on a
destination-cache miss hashed the transformed packet's NEXTHDR_HOP
instead of the original nexthdr (inline) or IPPROTO_IPV6 (encap), and
rebuilt the flow keys from the outer header only.

Patch 3 adds selftests/net/ioam6_ecmp.sh, covering both places: a
transit forwarder with a multipath route, and an encapsulating node
compared against a plain ip6ip6 baseline.

The issue was found while reviewing RFC9486. It can be reproduced and
verified by the newly added ioam6_ecmp.sh test.

Tested with:
ecmp_rehash.sh
fib_nexthops.sh
ioam6_ecmp.sh
tcp_ecmp_failover.sh
forwarding/gre_inner_v4_multipath.sh
forwarding/gre_inner_v6_multipath.sh
forwarding/gre_multipath_nh_res.sh
forwarding/gre_multipath_nh.sh
forwarding/gre_multipath.sh
forwarding/ip6gre_inner_v4_multipath.sh
forwarding/ip6gre_inner_v6_multipath.sh
forwarding/router_mpath_nh_res.sh
forwarding/router_mpath_nh.sh
forwarding/router_multipath.sh

Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
Hangbin Liu (3):
      ipv6: route: consider IOAM options in ECMP hash policy 0
      ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup
      selftests: net: add ioam6 ECMP test (RFC 9486 C2)

 include/uapi/linux/in6.h                  |   1 +
 net/ipv6/ioam6_iptunnel.c                 |   4 +-
 net/ipv6/route.c                          |  67 +++++-
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/ioam6_ecmp.sh | 349 ++++++++++++++++++++++++++++++
 5 files changed, 419 insertions(+), 3 deletions(-)
---
base-commit: 5d7f3375cd574fc83f0cd30839b21455c0e79ae2
change-id: 20261020-ipv6_ioam_ecmp-ae4ef0a4bdae

Best regards,
--  
Hangbin Liu <liuhangbin@kylinos.cn>

[PATCH 1/3] ipv6: route: consider IOAM options in ECMP hash policy 0

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
options is the same as the hash of the identical packet without them.

The default multipath hash policy (0) hashes basic.ip_proto, which
ip6_multipath_l3_keys() fills from the raw IPv6 nexthdr. An IOAM
Hop-by-Hop option is inserted ahead of the original next header, so a
transit router sees NEXTHDR_HOP (0) instead of e.g. IPPROTO_TCP and
may pick a different ECMP sibling than for the same packet without
IOAM.

Recover the nexthdr the packet would have had if its Hop-by-Hop
header carried only IOAM options (0x31, and 0x11 for DEX) plus
padding, and use it for the policy-0 key and for the ICMPv6-error
check, so errors that quote a flow keep hashing that flow. Any other
Hop-by-Hop option (Router Alert, Jumbo, ...) keeps hashing exactly as
before. The walk only runs for the rare nexthdr == NEXTHDR_HOP case
and is safe: ipv6_parse_hopopts() has already pulled and validated
the header before routing. ICMPv6-quoted inner headers are not
walked: they are payload and may be truncated.

Fixes: 3edede08ff37 ("ipv6: ioam: Support for IOAM injection with lwtunnels")
Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
The ECMP path with and without IOAM is verified by
tools/testing/selftests/net/ioam6_ecmp.sh in the next patch.
---
 include/uapi/linux/in6.h |  1 +
 net/ipv6/route.c         | 67 ++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/in6.h b/include/uapi/linux/in6.h
index 5a47339ef7d7..2e66dc37e1e5 100644
--- a/include/uapi/linux/in6.h
+++ b/include/uapi/linux/in6.h
@@ -145,6 +145,7 @@ struct in6_flowlabel_req {
 #define IPV6_TLV_PADN		1
 #define IPV6_TLV_ROUTERALERT	5
 #define IPV6_TLV_CALIPSO	7	/* RFC 5570 */
+#define IPV6_TLV_IOAM_DEX	17	/* RFC 9486 */
 #define IPV6_TLV_IOAM		49	/* RFC 9486 */
 #define IPV6_TLV_JUMBO		194
 #define IPV6_TLV_HAO		201	/* home address option */
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 94e5ef1c4f83..d0761f05d326 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2342,6 +2342,60 @@ struct dst_entry *ip6_route_input_lookup(struct net *net,
 }
 EXPORT_SYMBOL_GPL(ip6_route_input_lookup);
 
+/* RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
+ * options is the same as the hash of the identical packet without them.
+ * Recover the nexthdr if its Hop-by-Hop header carried only IOAM options
+ * and padding, every other packet is hashed exactly as before.
+ */
+static u8 ip6_nexthdr_without_ioam(const struct sk_buff *skb,
+				   const struct ipv6hdr *iph)
+{
+	const struct ipv6_opt_hdr *hhdr;
+	struct ipv6_opt_hdr hbuf;
+	bool has_ioam = false;
+	u8 obuf[2], *opt;
+	int off, len, i;
+
+	if (iph->nexthdr != NEXTHDR_HOP)
+		return iph->nexthdr;
+
+	/* The Hop-by-Hop header was already pulled and validated by
+	 * ipv6_parse_hopopts() before routing, so this walk is safe.
+	 */
+	off = sizeof(*iph);
+	hhdr = skb_header_pointer(skb, off, sizeof(hbuf), &hbuf);
+	if (!hhdr)
+		return iph->nexthdr;
+
+	len = (hhdr->hdrlen + 1) << 3;
+
+	for (i = 2; i < len; ) {
+		opt = skb_header_pointer(skb, off + i, sizeof(obuf), obuf);
+		if (!opt)
+			return iph->nexthdr;
+		if (opt[0] == IPV6_TLV_PAD1) {
+			i++;
+			continue;
+		}
+		if (opt[0] == IPV6_TLV_PADN) {
+			i += opt[1] + 2;
+			continue;
+		}
+		if (opt[0] != IPV6_TLV_IOAM && opt[0] != IPV6_TLV_IOAM_DEX)
+			return iph->nexthdr;
+
+		has_ioam = true;
+
+		/* Keep searching for the next TLV. As long as non-IOAM
+		 * options (Router Alert, Jumbo, etc.) exist in the HBH
+		 * header, we should preserve the original NEXTHDR_HOP.
+		 */
+		i += opt[1] + 2;
+	}
+
+	return has_ioam ? hhdr->nexthdr : iph->nexthdr;
+}
+
 static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 				  struct flow_keys *keys,
 				  struct flow_keys *flkeys)
@@ -2353,8 +2407,10 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 	const struct icmp6hdr *icmph;
 	struct ipv6hdr _inner_iph;
 	struct icmp6hdr _icmph;
+	u8 nexthdr;
 
-	if (likely(outer_iph->nexthdr != IPPROTO_ICMPV6))
+	nexthdr = ip6_nexthdr_without_ioam(skb, outer_iph);
+	if (likely(nexthdr != IPPROTO_ICMPV6))
 		goto out;
 
 	icmph = skb_header_pointer(skb, skb_transport_offset(skb),
@@ -2383,7 +2439,14 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 		keys->addrs.v6addrs.src = key_iph->saddr;
 		keys->addrs.v6addrs.dst = key_iph->daddr;
 		keys->tags.flow_label = ip6_flowlabel(key_iph);
-		keys->basic.ip_proto = key_iph->nexthdr;
+		/* Only for the outer header: an ICMPv6-quoted inner header
+		 * is payload, and its (possibly truncated) extension
+		 * headers must not be walked here.
+		 */
+		if (key_iph == outer_iph)
+			keys->basic.ip_proto = nexthdr;
+		else
+			keys->basic.ip_proto = key_iph->nexthdr;
 	}
 }
 

-- 
2.56.0

[PATCH 2/3] ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
options is the same as the hash of the identical packet without them.

When the ioam6 output path misses the destination cache, it re-looks
up the route for the transformed packet. Under the default hash
policy 0 that lookup hashed NEXTHDR_HOP rather than the nexthdr the
identical packet without IOAM would carry, so it could pick a
different ECMP sibling.

Use the Hop-by-Hop nexthdr already stored by do_inline()/do_encap()
(eh.nexthdr) for the re-lookup, and precompute the multipath hash
from the packet: ip6_route_output() lookups carry no skb, so without
this ICMPv6 errors would stop hashing their quoted flow and L4 hash
policy packets would lose their ports.

Fixes: 8cb3bf8bff3c ("ipv6: ioam: Add support for the ip6ip6 encapsulation")
Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
 net/ipv6/ioam6_iptunnel.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/net/ipv6/ioam6_iptunnel.c b/net/ipv6/ioam6_iptunnel.c
index cfb2c41634a0..713e625bfa3a 100644
--- a/net/ipv6/ioam6_iptunnel.c
+++ b/net/ipv6/ioam6_iptunnel.c
@@ -13,6 +13,7 @@
 #include <linux/ioam6.h>
 #include <linux/ioam6_iptunnel.h>
 #include <net/dst.h>
+#include <net/ip6_route.h>
 #include <net/sock.h>
 #include <net/lwtunnel.h>
 #include <net/ioam6.h>
@@ -412,7 +413,8 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
 		fl6.saddr = hdr->saddr;
 		fl6.flowlabel = ip6_flowinfo(hdr);
 		fl6.flowi6_mark = skb->mark;
-		fl6.flowi6_proto = hdr->nexthdr;
+		fl6.flowi6_proto = ilwt->tuninfo.eh.nexthdr;
+		fl6.mp_hash = rt6_multipath_hash(net, &fl6, skb, NULL);
 
 		dst = ip6_route_output(net, NULL, &fl6);
 		if (dst->error) {

-- 
2.56.0

[PATCH 3/3] selftests: net: add ioam6 ECMP test (RFC 9486 C2)

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

Add new test ioam6_ecmp.sh, verifying that the ECMP path taken with
IOAM options is identical to the path of the same packet without them
under the default hash policy 0, as required by RFC 9486 (C2).

  - transit input hashing: the sender inserts an IOAM Hop-by-Hop option
    (inline mode), the forwarder has a multipath route;
  - encapsulating node re-lookup: the forwarder itself encapsulates
    (ioam6 encap mode, tunnel dst equal to the original dst) over a
    multipath route, with a plain ip6ip6 tunnel as the
    identical-packet-without-IOAM baseline.

Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/ioam6_ecmp.sh | 349 ++++++++++++++++++++++++++++++
 2 files changed, 350 insertions(+)

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index df215e1a6262..397b19799cc9 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -52,6 +52,7 @@ TEST_PROGS := \
 	icmp_redirect.sh \
 	io_uring_zerocopy_tx.sh \
 	ioam6.sh \
+	ioam6_ecmp.sh \
 	ip6_gre_headroom.sh \
 	ip6_tunnel.sh \
 	ip_defrag.sh \
diff --git a/tools/testing/selftests/net/ioam6_ecmp.sh b/tools/testing/selftests/net/ioam6_ecmp.sh
new file mode 100755
index 000000000000..c5c82884c757
--- /dev/null
+++ b/tools/testing/selftests/net/ioam6_ecmp.sh
@@ -0,0 +1,349 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Verify RFC 9486 C2 for ECMP: with hash policy 0 (the default), the ECMP
+# nexthop chosen for a packet carrying IOAM options must be the same as for
+# the identical packet without them.
+#
+# Two scenarios, matching the two places where IOAM changed the hashed
+# IPv6 nexthdr:
+#
+# 1) transit node: alpha inserts an IOAM Hop-by-Hop option (inline mode);
+#    beta forwards with a multipath route. Policy 0 hashed the raw
+#    nexthdr (NEXTHDR_HOP vs IPPROTO_ICMPV6), so beta could pick a
+#    different ECMP sibling than for the same packet without IOAM.
+#
+# 2) encapsulating node: beta itself encapsulates (ioam6 encap mode,
+#    tunnel dst = original dst, tunnel src pinned) over a multipath
+#    route. The ioam6 re-lookup hashed NEXTHDR_HOP instead of the
+#    transformed packet's real nexthdr (IPPROTO_IPV6). The baseline is
+#    the same encapsulation without the IOAM option: a plain ip6ip6
+#    tunnel over the same multipath route, with the same outer
+#    addresses.
+#
+# Hash policy 0 hashes addresses, the flow label and the nexthdr (not L4
+# ports), so whether both variants land on the same ECMP sibling depends
+# on the per-netns hash seed. Each test therefore repeats with several
+# seeds and fails on the first divergence: pre-fix kernels fail with
+# high probability, fixed kernels always pass.
+#
+# Topology:
+#
+#   alpha                   beta                      gamma
+#   veth0 2001:db8:1::2 <-> veth0 2001:db8:1::1
+#                           veth1 2001:db8:2:1::1 <-> veth0 2001:db8:2:1::2
+#                           veth2 2001:db8:2:2::1 <-> veth1 2001:db8:2:2::2
+#
+# beta's ECMP route: 2001:db8:2:1::2/128
+#   nexthop via 2001:db8:2:1::2 dev veth1
+#   nexthop via 2001:db8:2:2::2 dev veth2
+#
+# Scenario 1 sends to 2001:db8:2:1::2 (gamma). Scenario 2 sends to
+# 2001:db8:3::2, which beta encapsulates towards 2001:db8:2:1::2. The
+# link gamma received the packets on is detected from tc flower
+# counters (read via jq) on the ingress of gamma's veth0/veth1. The
+# filters match the exact measurement packets: ICMPv6 echo requests
+# for scenario 1, and the outer addresses of the ip6ip6 packets for
+# scenario 2 (flower classifies outer headers only).
+
+# shellcheck disable=SC2154,SC2329
+source lib.sh
+
+readonly PKTS=3
+readonly SEEDS="1 2 3 4 5 6 7 8 9 10 11 12"
+
+ip6tnl_loaded=0
+
+cleanup()
+{
+	cleanup_all_ns
+
+	if [ "$ip6tnl_loaded" != 0 ]; then
+		modprobe -r ip6_tunnel &>/dev/null
+	fi
+}
+
+# Replace the tc measurement filters on gamma's ECMP links. $1 is the
+# flower match list, e.g. "ip_proto icmpv6 type 128 src_ip <sender>"
+# for plain IPv6 echo requests, or the outer addresses for ip6ip6
+# packets (flower classifies the outer header only, see
+# FLOW_DISSECTOR_F_STOP_BEFORE_ENCAP in cls_flower).
+set_measure_filters()
+{
+	local match=$1
+	local dev
+
+	for dev in veth0 veth1; do
+		tc -n "$ns_gamma" qdisc del dev "$dev" clsact &>/dev/null
+		tc -n "$ns_gamma" qdisc add dev "$dev" clsact || return 1
+		# shellcheck disable=SC2086
+		tc -n "$ns_gamma" filter add dev "$dev" ingress pref 1 \
+			protocol ipv6 flower skip_hw $match action pass || return 1
+	done
+}
+
+# Number of packets counted by the measurement filter on gamma's $1.
+icmp_packets()
+{
+	tc -n "$ns_gamma" -s -j filter show dev "$1" ingress |
+		jq '[.[] | .options.actions[]?.stats |
+		    (.packets // .basic.packets // 0)] | add // 0'
+}
+
+# Send PKTS pings from alpha to $1 and report which of gamma's links
+# received them: "A" (veth0), "B" (veth1), "TIE" or "LOST:<a>:<b>".
+measure_link()
+{
+	local dst=$1
+	local a0 b0 da db
+
+	a0=$(icmp_packets veth0)
+	b0=$(icmp_packets veth1)
+
+	ip netns exec "$ns_alpha" \
+		ping -6 -q -c "$PKTS" -i 0.2 -W 1 -w 2 "$dst" &>/dev/null || true
+
+	da=$(($(icmp_packets veth0) - a0))
+	db=$(($(icmp_packets veth1) - b0))
+
+	if [ $((da + db)) -lt "$PKTS" ]; then
+		echo "LOST:$da:$db"
+		return
+	fi
+
+	if [ "$da" -gt "$db" ]; then
+		echo "A"
+	elif [ "$db" -gt "$da" ]; then
+		echo "B"
+	else
+		echo "TIE"
+	fi
+}
+
+set_alpha_plain()
+{
+	ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+		via 2001:db8:1::1 dev veth0
+}
+
+set_alpha_ioam()
+{
+	ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+		encap ioam6 mode inline trace prealloc \
+		type 0x800000 ns 0 size 12 \
+		via 2001:db8:1::1 dev veth0
+}
+
+set_beta_tnl()
+{
+	# Recreate the tunnel: ip6_tnl caches the outer dst per CPU for a
+	# few seconds, and neither the route change below nor the
+	# fib_multipath_hash_seed write resets it. A stale cache would
+	# resolve the outer sibling under a previous seed (or on another
+	# CPU), making the baseline flip between attempts of one seed.
+	ip -netns "$ns_beta" link del ioamcmp &>/dev/null
+	ip -netns "$ns_beta" link add name ioamcmp type ip6tnl \
+		mode ip6ip6 local 2001:db8:2:1::1 remote 2001:db8:2:1::2
+	ip -netns "$ns_beta" link set ioamcmp up
+	ip -netns "$ns_beta" -6 route replace 2001:db8:3::2/128 dev ioamcmp
+}
+
+set_beta_ioam_encap()
+{
+	ip -netns "$ns_beta" -6 route replace 2001:db8:3::2/128 \
+		encap ioam6 mode encap tunsrc 2001:db8:2:1::1 tundst 2001:db8:2:1::2 \
+		trace prealloc type 0x800000 ns 0 size 12 dev veth1
+}
+
+# $1: description; $2: function installing the without-IOAM variant;
+# $3: function installing the with-IOAM variant; $4: destination.
+run_scenario()
+{
+	local desc=$1
+	local plain=$2
+	local ioam=$3
+	local dst=$4
+	local seed pl il
+
+	log_info "$desc"
+
+	for seed in $SEEDS; do
+		RET=0
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv4.fib_multipath_hash_seed="$seed"
+
+		if ! $plain; then
+			RET=$ksft_fail
+			log_test "$desc (plain route setup failed, seed $seed)"
+			continue
+		fi
+		pl=$(measure_link "$dst")
+		if ! $ioam; then
+			RET=$ksft_fail
+			log_test "$desc (ioam route setup failed, seed $seed)"
+			continue
+		fi
+		il=$(measure_link "$dst")
+
+		case "$pl$il" in
+		*LOST*)
+			RET=$ksft_fail
+			log_test "$desc (lost packets, seed $seed: without=$pl with=$il)"
+			continue
+			;;
+		*TIE*)
+			RET=$ksft_fail
+			log_test "$desc (no clear link, seed $seed)"
+			continue
+			;;
+		esac
+
+		if [ "$pl" = "$il" ]; then
+			log_test "$desc (seed $seed: same link $pl)"
+		else
+			RET=$ksft_fail
+			log_test "$desc (diverged on seed $seed: without IOAM via link $pl, with IOAM via link $il)"
+		fi
+	done
+}
+
+setup_topology()
+{
+	local dev
+
+	setup_ns ns_alpha ns_beta ns_gamma || return 1
+
+	ip link add veth0 netns "$ns_alpha" type veth \
+		peer name veth0 netns "$ns_beta" || return 1
+	ip link add veth1 netns "$ns_beta" type veth \
+		peer name veth0 netns "$ns_gamma" || return 1
+	ip link add veth2 netns "$ns_beta" type veth \
+		peer name veth1 netns "$ns_gamma" || return 1
+
+	ip -n "$ns_alpha" addr add 2001:db8:1::2/64 dev veth0 nodad
+	ip -n "$ns_beta" addr add 2001:db8:1::1/64 dev veth0 nodad
+	ip -n "$ns_beta" addr add 2001:db8:2:1::1/64 dev veth1 nodad
+	ip -n "$ns_beta" addr add 2001:db8:2:2::1/64 dev veth2 nodad
+	ip -n "$ns_gamma" addr add 2001:db8:2:1::2/64 dev veth0 nodad
+	ip -n "$ns_gamma" addr add 2001:db8:2:2::2/64 dev veth1 nodad
+
+	ip -n "$ns_alpha" link set veth0 up
+	ip -n "$ns_beta" link set veth0 up
+	ip -n "$ns_beta" link set veth1 up
+	ip -n "$ns_beta" link set veth2 up
+	ip -n "$ns_gamma" link set veth0 up
+	ip -n "$ns_gamma" link set veth1 up
+
+	ip netns exec "$ns_alpha" sysctl -wq net.ipv6.auto_flowlabels=0
+	ip netns exec "$ns_alpha" \
+		sysctl -wq net.ipv6.conf.veth0.accept_ra=0
+	ip netns exec "$ns_beta" sysctl -wq net.ipv6.conf.all.forwarding=1
+	ip netns exec "$ns_beta" \
+		sysctl -wq net.ipv6.fib_multipath_hash_policy=0
+	ip netns exec "$ns_gamma" sysctl -wq net.ipv6.conf.veth0.accept_ra=0
+	ip netns exec "$ns_gamma" sysctl -wq net.ipv6.conf.veth1.accept_ra=0
+
+	for dev in veth0 veth1 veth2
+	do
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv6.conf.$dev.accept_ra=0
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv6.conf.$dev.ioam6_enabled=1
+	done
+
+	# The ECMP route under test (scenario 1 hashes its selection at
+	# beta's input; scenario 2 at the encapsulating re-lookup).
+	ip -netns "$ns_beta" -6 route add 2001:db8:2:1::2/128 \
+		nexthop via 2001:db8:2:1::2 dev veth1 \
+		nexthop via 2001:db8:2:2::2 dev veth2
+
+	ip -netns "$ns_alpha" -6 route add 2001:db8:2:1::/64 \
+		via 2001:db8:1::1 dev veth0
+	ip -netns "$ns_alpha" -6 route add 2001:db8:3::/64 \
+		via 2001:db8:1::1 dev veth0
+	ip -netns "$ns_gamma" -6 route add 2001:db8:1::/64 \
+		via 2001:db8:2:1::1 dev veth0
+
+	# drop unreachable message when ping 2001:db8:3::2
+	tc -n "$ns_alpha" qdisc add dev veth0 clsact
+	tc -n "$ns_alpha" filter add dev veth0 ingress pref 1 \
+                protocol ipv6 flower skip_hw ip_proto icmpv6 type 1 action drop
+
+}
+
+setup_tunnel()
+{
+	if ! lsmod 2>/dev/null | grep -q "ip6_tunnel"; then
+		modprobe ip6_tunnel &>/dev/null
+		lsmod 2>/dev/null | grep -q "ip6_tunnel" || \
+			ip a 2>/dev/null | grep -q "ip6tnl0" || return 1
+		ip6tnl_loaded=1
+	fi
+
+	ip -netns "$ns_beta" link add name ioamcmp type ip6tnl \
+		mode ip6ip6 local 2001:db8:2:1::1 remote 2001:db8:2:1::2 \
+		2>/dev/null || return 1
+	ip -netns "$ns_beta" link set ioamcmp up
+}
+
+################################################################################
+#                                                                              #
+#                                     MAIN                                     #
+#                                                                              #
+################################################################################
+
+require_command ip
+require_command jq
+require_command ping
+require_command tc
+
+if [ ! -e /proc/sys/net/ipv4/fib_multipath_hash_seed ]; then
+	log_test_skip "Kernel without CONFIG_IP_ROUTE_MULTIPATH."
+	exit "${EXIT_STATUS}"
+fi
+
+setup_topology || { log_test_skip "Setup failed."; exit "${EXIT_STATUS}"; }
+trap cleanup EXIT
+
+# Compat check: can this kernel insert an IOAM trace into a route?
+if ! ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+	encap ioam6 mode inline trace prealloc type 0x800000 ns 0 size 12 \
+	via 2001:db8:1::1 dev veth0 &>/dev/null
+then
+	log_test_skip "Cannot attach an IOAM trace to a route"
+	exit "${EXIT_STATUS}"
+fi
+
+log_info "IOAM + ECMP (RFC 9486 C2, hash policy 0)"
+
+# Warm up both ECMP links (first-packet neighbor/RA activity) and check
+# basic connectivity before measuring.
+set_alpha_plain
+measure_link 2001:db8:2:1::2 >/dev/null
+
+# Scenario 1 counts the ICMPv6 echo requests sent by alpha.
+if ! set_measure_filters "ip_proto icmpv6 type 128 src_ip 2001:db8:1::2"; then
+	log_test_skip "Cannot install tc flower measurement filters"
+	exit "${EXIT_STATUS}"
+fi
+run_scenario \
+	"Transit input ECMP with/without IOAM" \
+	set_alpha_plain set_alpha_ioam 2001:db8:2:1::2
+
+if setup_tunnel; then
+	# Scenario 2 counts the outer ip6ip6 packets: the echo request is
+	# not visible to flower (outer headers only), so match the pinned
+	# tunnel addresses instead. No need to check return value again as
+	# we have done it at scenario 1
+	set_measure_filters "src_ip 2001:db8:2:1::1 dst_ip 2001:db8:2:1::2"
+	set_beta_tnl
+	measure_link 2001:db8:3::2 >/dev/null
+
+	run_scenario \
+		"Encapsulating node re-lookup ECMP with/without IOAM" \
+		set_beta_tnl set_beta_ioam_encap 2001:db8:3::2
+else
+	log_test_skip "Encapsulating node re-lookup ECMP with/without IOAM (no ip6_tunnel)"
+fi
+
+exit "${EXIT_STATUS}"

-- 
2.56.0

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> This series makes IOAM insertion transparent to IPv6 ECMP, as required
> by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> along the same path as the identical packet without them.

Sorry, forgot to add target repo "net". I will add it in next version.

Thanks
Hangbin

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Jakub Kicinski]

On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > along the same path as the identical packet without them.  
> 
> Sorry, forgot to add target repo "net". I will add it in next version.

CI bot says:

The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:

  TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
        link A, with IOAM via link B)   [FAIL]
  TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
  TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]

11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
(or the padding/option-length walk used to compute it) is correct for all IOAM option
layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
sibling than the equivalent packet without IOAM?

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

Hi Jakub,
On Wed, Oct 28, 2026 at 04:58:12PM -0700, Jakub Kicinski wrote:
> On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> > On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > > along the same path as the identical packet without them.  
> > 
> > Sorry, forgot to add target repo "net". I will add it in next version.
> 
> CI bot says:
> 
> The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
> is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:
> 
>   TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
>         link A, with IOAM via link B)   [FAIL]
>   TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
>   TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]
> 
> 11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
> scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
> nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
> infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
> (or the padding/option-length walk used to compute it) is correct for all IOAM option
> layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
> sibling than the equivalent packet without IOAM?

Thanks for the report. I run it locally multi times and didn't got this issue.
I will try reproduce it first and check the reason.

Thanks
Hangbin

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

Hi Jakub,
On Wed, Oct 28, 2026 at 04:58:12PM -0700, Jakub Kicinski wrote:
> On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> > On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > > along the same path as the identical packet without them.  
> > 
> > Sorry, forgot to add target repo "net". I will add it in next version.
> 
> CI bot says:
> 
> The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
> is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:
> 
>   TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
>         link A, with IOAM via link B)   [FAIL]
>   TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
>   TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]
> 
> 11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
> scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
> nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
> infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
> (or the padding/option-length walk used to compute it) is correct for all IOAM option
> layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
> sibling than the equivalent packet without IOAM?

OK, after running the test 50 times on a debug kernel and over 70 times on a
normal kernel, I've finally reproduced the issue and identified the root cause.

The reason is the ping `-w` parameter. I originally thought it set the maximum
wait time, but it actually keeps sending packets regardless of how many packets
have been sent or received. Combined with `-i 0.2`, it sometimes sends 10 packets,
which carry over into the seed 1 test and affect the result.

That's why the debug kernel case worked and the normal kernel failed in Jakub's
environment, while it always passed in my laptop test environment.

After removing the "-W 1 -w 2" parameters, I ran the test 200 times and all
passed. I will post the fix in the next version.

TL;DR
--------------------

Here are my testing steps. After reproducing the issue, I dumped the tc output
and found 10 extra ICMPv6 echo requests.

filter protocol ipv6 pref 1 flower chain 0
filter protocol ipv6 pref 1 flower chain 0 handle 0x1
  eth_type ipv6
  ip_proto icmpv6
  src_ip 2001:db8:1::2
  icmp_type 128
  skip_hw
  not_in_hw
        action order 1: gact action pass
         random type none pass val 0
         index 1 ref 1 bind 1 installed 0 sec
        Action statistics:
        Sent 1040 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

on gamma veth0 at the start of seed 1 testing, but *a0* is 0. This means some
echo requests are still in flight during this window. I then captured traffic
with tcpdump and confirmed those echo requests are present on gamma veth0:

15:47:09.161066 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 1, length 64
15:47:09.161083 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 2, length 64
15:47:09.161086 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 3, length 64
15:47:09.161088 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 4, length 64
15:47:09.161091 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 5, length 64
15:47:09.161093 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 6, length 64
15:47:09.161095 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 7, length 64
15:47:09.161097 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 8, length 64
15:47:09.161100 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 9, length 64
15:47:09.161102 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 10, length 64

The question is: where do these 10 packets come from, and why are they sent
within this narrow window?

We expect only 3 echo requests during the ECMP link warm-up, which does not
match the tcpdump capture. The timestamps, however, show the traffic starts
during the warm-up phase. The subsequent echo requests on veth1 are correctly
sent as 3 packets with a 0.2s interval.

15:47:09.221513 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 1, length 64
15:47:09.426247 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 2, length 64
15:47:09.633125 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 3, length 64

It looks like the skbs were held in the unresolved neighbour queue and flushed
after receiving the NA, causing the packets to be delivered to gamma ingress later.

Then I tried to send 1 packet on before the warm up.
ip netns exec "$ns_alpha" ping -6 -q -c 1 -i 0.2 -W 1 -w 2 2001:db8:2:1::2  &> /dev/null

Now the packet shows

16:05:20.744986 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 1, length 64
16:05:20.744987 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 2, length 64
16:05:20.744988 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 3, length 64
16:05:20.744988 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 4, length 64
16:05:20.744989 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 5, length 64
16:05:20.744990 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 6, length 64
16:05:20.744990 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 7, length 64
16:05:20.744991 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 8, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 9, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 10, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 1, length 64
16:05:20.936373 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 2, length 64
16:05:21.137153 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 3, length 64

The warm-up packets are correctly back to 3 with a 0.2s interval. However, the
first `-c 1` ping still generates 10 packets. With set -x, it shows

+ ip netns exec ns_alpha-nBg602 ping -6 -q -c 1 -i 0.2 -W 1 -w 2 2001:db8:2:1::2
PING 2001:db8:2:1::2 (2001:db8:2:1::2) 56 data bytes

--- 2001:db8:2:1::2 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 1865ms

After checking, I finally got the answer:

-w deadline: Specify a timeout, in seconds, before ping exits regardless of
how many packets have been sent or received.

When `-w` is used, the `-c` packet count no longer acts as a hard limit. Ping
keeps sending packets until either the expected number of replies are received
or the timeout expires. In our case, the timeout was hit. All 10 packets
(sent at a 0.2s interval within the 2s window) got deferred and surfaced
during the seed 1 test, corrupting the result.

Thanks
Hangbin

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> This series makes IOAM insertion transparent to IPv6 ECMP, as required
> by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> along the same path as the identical packet without them.
> 
> Patch 1 fixes the transit side: hash policy 0 hashed the raw IPv6
> nexthdr, which becomes NEXTHDR_HOP once an IOAM Hop-by-Hop header is
> inserted. The policy-0 key (and the ICMPv6-error flow selection) now
> uses the nexthdr the packet would have had if the Hop-by-Hop header
> carried only IOAM options and padding.
> 
> Patch 2 fixes the injecting node: the ioam6 output re-lookup on a
> destination-cache miss hashed the transformed packet's NEXTHDR_HOP
> instead of the original nexthdr (inline) or IPPROTO_IPV6 (encap), and
> rebuilt the flow keys from the outer header only.
> 
> Patch 3 adds selftests/net/ioam6_ecmp.sh, covering both places: a
> transit forwarder with a multipath route, and an encapsulating node
> compared against a plain ip6ip6 baseline.
> 
> The issue was found while reviewing RFC9486. It can be reproduced and
> verified by the newly added ioam6_ecmp.sh test.
> 
> Tested with:
> ecmp_rehash.sh
> fib_nexthops.sh
> ioam6_ecmp.sh
> tcp_ecmp_failover.sh
> forwarding/gre_inner_v4_multipath.sh
> forwarding/gre_inner_v6_multipath.sh
> forwarding/gre_multipath_nh_res.sh
> forwarding/gre_multipath_nh.sh
> forwarding/gre_multipath.sh
> forwarding/ip6gre_inner_v4_multipath.sh
> forwarding/ip6gre_inner_v6_multipath.sh
> forwarding/router_mpath_nh_res.sh
> forwarding/router_mpath_nh.sh
> forwarding/router_multipath.sh
> 
> Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
> ---
> Hangbin Liu (3):
>       ipv6: route: consider IOAM options in ECMP hash policy 0
>       ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup
>       selftests: net: add ioam6 ECMP test (RFC 9486 C2)
> 
>  include/uapi/linux/in6.h                  |   1 +
>  net/ipv6/ioam6_iptunnel.c                 |   4 +-
>  net/ipv6/route.c                          |  67 +++++-
>  tools/testing/selftests/net/Makefile      |   1 +
>  tools/testing/selftests/net/ioam6_ecmp.sh | 349 ++++++++++++++++++++++++++++++
>  5 files changed, 419 insertions(+), 3 deletions(-)
> ---
> base-commit: 5d7f3375cd574fc83f0cd30839b21455c0e79ae2
> change-id: 20261020-ipv6_ioam_ecmp-ae4ef0a4bdae
> 
> Best regards,
> --  
> Hangbin Liu <liuhangbin@kylinos.cn>
> 

Sashiko found some issues. I will fix them and post new version.

pw-bot: cr

Re: [PATCH 1/3] ipv6: route: consider IOAM options in ECMP hash policy 0

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 5 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 3 · Low: 2

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] ip6_nexthdr_without_ioam() (net/ipv6/route.c) reads the Hop-
  by-Hop header at `off = sizeof(*iph)`.
- [Medium] In ip6_multipath_l3_keys(), the outer header of an IOAM-
  carrying packet is now hashed with the recovered L4 protocol…
- [Medium] On IOAM-enabled ingress with a known namespace,
  ipv6_hop_ioam() calls ip6_route_input() while ip6_parse_tlv() is still…
- [Low] The new comment in ip6_nexthdr_without_ioam() says 'The Hop-by-
  Hop header was already pulled and validated by ipv6_parse_hopopts()…
- [Low] In the ip6_nexthdr_without_ioam() TLV loop,
  `skb_header_pointer(skb, off + i, sizeof(obuf), obuf)` fetches 2 bytes…

--

ipv6: route: consider IOAM options in ECMP hash policy 0

This patch adds ip6_nexthdr_without_ioam(). It walks a Hop-by-Hop header
and, if the header holds only IOAM options and padding, returns the
nexthdr that follows it. ip6_multipath_l3_keys() uses that value for the
policy 0 ip_proto key and for its ICMPv6 error check, so that packets
with and without IOAM hash the same way, as RFC 9486 (C2) requires.

> diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> index 94e5ef1c4f83e..d0761f05d3260 100644
> --- a/net/ipv6/route.c
> +++ b/net/ipv6/route.c
> @@ -2342,6 +2342,60 @@ struct dst_entry *ip6_route_input_lookup(struct net *net,

[ ... ]

> +static u8 ip6_nexthdr_without_ioam(const struct sk_buff *skb,
> +				   const struct ipv6hdr *iph)
> +{

[ ... ]

> +	if (iph->nexthdr != NEXTHDR_HOP)
> +		return iph->nexthdr;
> +
> +	/* The Hop-by-Hop header was already pulled and validated by
> +	 * ipv6_parse_hopopts() before routing, so this walk is safe.
> +	 */

[Severity: Low]
Does this comment hold for every caller of rt6_multipath_hash()?

icmp6_send() also hashes the offending skb:

net/ipv6/icmp.c:icmp6_send() {
    ...
	fl6.mp_hash = rt6_multipath_hash(net, &fl6, skb, NULL);
    ...
}

That skb can be generated locally and never pass through
ipv6_parse_hopopts(). One example is a packet built with IPV6_HOPOPTS
or a raw socket that hits a reject or unreachable route:

ip6_pkt_discard_out()->ip6_pkt_drop()->icmpv6_send()->icmp6_send()->
    rt6_multipath_hash()

On IOAM-enabled ingress, ipv6_hop_ioam() calls ip6_route_input() while
ip6_parse_tlv() is still running. So any TLVs after the IOAM option have
not been validated yet:

net/ipv6/exthdrs.c:ipv6_hop_ioam() {
    ...
		if (!skb_valid_dst(skb))
			ip6_route_input(skb);
    ...
}

Today the walk is memory safe only because every access goes through
skb_header_pointer(). With unvalidated input it can also return
hhdr->nexthdr for a malformed HBH header whose TLV lengths run past the
header.

Could the comment be updated to describe what actually makes the walk
safe?

> +	off = sizeof(*iph);
> +	hhdr = skb_header_pointer(skb, off, sizeof(hbuf), &hbuf);

[Severity: Medium]
Does this offset assume that skb->data sits at the network header?

skb_header_pointer() counts the offset from skb->data. iph is
ipv6_hdr(skb), i.e. skb_network_header(). The two only agree when
skb->data has not been pulled past the network header.

They do not agree when icmp6_send() is called from the L4 receive paths.
By then ip6_protocol_deliver_rcu() has already pulled to the transport
header:

net/ipv6/ip6_input.c:ip6_protocol_deliver_rcu() {
    ...
		if (!pskb_pull(skb, skb_transport_offset(skb)))
    ...
}

icmp6_send() then hashes the offending skb with policy 0:

__udp6_lib_rcv()->icmpv6_send()->icmp6_send()->rt6_multipath_hash()->
    ip6_multipath_l3_keys()->ip6_nexthdr_without_ioam()

icmp6_send() already handles this when it quotes the packet:

	msg.offset = skb_network_offset(skb);

The ICMPV6_UNK_NEXTHDR parameter problem in ip6_protocol_deliver_rcu()
and the tunnel6/xfrm6 errors take the same path. If the offending
packet has nexthdr == NEXTHDR_HOP (even PadN only), the helper treats
the bytes 40 past the L4 header as the HBH header and its TLVs. Those
bytes are payload controlled by the sender.

If the packet has a real IOAM-only HBH header, the locally generated
ICMP error is still hashed with proto 0. That is the mismatch this patch
sets out to fix.

A crafted payload can also make the helper return any nexthdr, including
IPPROTO_ICMPV6. ip6_multipath_l3_keys() then reads the UDP header at
transport offset 0 as an icmp6hdr. If the high byte of the source port
is 1-4, icmpv6_is_err() passes and the UDP payload is used as the inner
ipv6hdr. The payload then picks the addresses, flow label and proto
used as ECMP hash keys.

Before this patch, HBH packets never reached this parsing. All reads are
bounded by skb_header_pointer(), so the result is wrong hash input, not
an out-of-bounds access.

Would using skb_network_offset(skb) + sizeof(*iph) as the base offset
avoid this? The follow-up patch in the series only changes
ioam6_iptunnel.c and the selftests, so route.c is unchanged after it.

> +	if (!hhdr)
> +		return iph->nexthdr;
> +
> +	len = (hhdr->hdrlen + 1) << 3;
> +
> +	for (i = 2; i < len; ) {
> +		opt = skb_header_pointer(skb, off + i, sizeof(obuf), obuf);
> +		if (!opt)
> +			return iph->nexthdr;
> +		if (opt[0] == IPV6_TLV_PAD1) {
> +			i++;
> +			continue;
> +		}

[Severity: Low]
Can a trailing Pad1 make this 2-byte fetch fail?

ip6_parse_tlv() accepts Pad1 as the last byte of the HBH header, because
the Pad1 branch continues before the len < 2 check:

net/ipv6/exthdrs.c:ip6_parse_tlv() {
    ...
		if (nh[off] == IPV6_TLV_PAD1) {
			padlen++;
			if (padlen > 7)
				goto bad;
			off++;
			len--;
			continue;
		}
		if (len < 2)
			goto bad;
    ...
}

Suppose an IOAM-only HBH header ends in Pad1 and nothing follows it in
the skb, e.g. nexthdr is NEXTHDR_NONE with no payload. The fetch at
off + len - 1 needs one byte past skb->len, so it returns NULL. The
function then returns iph->nexthdr (0) even though has_ioam is true.
This packet then hashes differently from the same packet without IOAM.

Would it work to read one byte first and fetch the length byte only for
options other than Pad1?

[ ... ]

> @@ -2353,8 +2407,10 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
>  	const struct icmp6hdr *icmph;
>  	struct ipv6hdr _inner_iph;
>  	struct icmp6hdr _icmph;
> +	u8 nexthdr;
>  
> -	if (likely(outer_iph->nexthdr != IPPROTO_ICMPV6))
> +	nexthdr = ip6_nexthdr_without_ioam(skb, outer_iph);
> +	if (likely(nexthdr != IPPROTO_ICMPV6))
>  		goto out;
>  
>  	icmph = skb_header_pointer(skb, skb_transport_offset(skb),

[Severity: Medium]
On IOAM-enabled ingress, where does skb_transport_offset(skb) point
when this code runs?

For a known namespace, ipv6_hop_ioam() calls ip6_route_input() from
inside ip6_parse_tlv(). ip6_rcv_core() has set transport_header to the
HBH header:

	skb->transport_header = skb->network_header + sizeof(*hdr);

ipv6_parse_hopopts() moves it past the HBH header only after
ip6_parse_tlv() returns:

	skb->transport_header += extlen;

Take an ICMPv6 error with an IOAM-only HBH header.
ip6_nexthdr_without_ioam() now returns IPPROTO_ICMPV6, but icmph is read
from the HBH header itself. icmp6_type is then the HBH nexthdr (58), and
icmpv6_is_err(58) is false.

The error is therefore hashed on its outer addresses with proto 58. The
same error without IOAM would be hashed on the quoted inner header.
ip6_rcv_finish_core() only routes when !skb_valid_dst(skb), so this
early dst stays in place.

On IOAM transit nodes, does this match what the commit message claims?

  "and use it for the policy-0 key and for the ICMPv6-error
  check, so errors that quote a flow keep hashing that flow."

No later patch in the series addresses this.

[ ... ]

> @@ -2383,7 +2439,14 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
>  		keys->addrs.v6addrs.src = key_iph->saddr;
>  		keys->addrs.v6addrs.dst = key_iph->daddr;
>  		keys->tags.flow_label = ip6_flowlabel(key_iph);
> -		keys->basic.ip_proto = key_iph->nexthdr;
> +		/* Only for the outer header: an ICMPv6-quoted inner header
> +		 * is payload, and its (possibly truncated) extension
> +		 * headers must not be walked here.
> +		 */
> +		if (key_iph == outer_iph)
> +			keys->basic.ip_proto = nexthdr;
> +		else
> +			keys->basic.ip_proto = key_iph->nexthdr;

[Severity: Medium]
Does this give flow packets and the ICMPv6 errors that quote them
different ip_proto keys?

Consider a transit router using policy 0. A flow packet carrying IOAM
reaches this code through fib6_select_path() and is hashed with the
recovered proto, e.g. IPPROTO_TCP. An ICMPv6 error that quotes such a
packet takes the inner branch:

	key_iph = inner_iph;
	_flkeys = NULL;

It then hashes the raw nexthdr of the quoted header, which is
NEXTHDR_HOP (0).

Before this patch, when IOAM was present in both directions, both sides
hashed proto 0 and matched. Now they differ, so an error such as Packet
Too Big can go to a different ECMP sibling than the flow. Commit
23aebdacb05d ("ipv6: Compute multipath hash for ICMP errors from
offending packet") hashes ICMP errors on the inner header so that the
two would match.

This seems to contradict the commit message:

  "so errors that quote a flow keep hashing that flow."

The stated reason for not walking the quoted header is truncation.
However, every access in ip6_nexthdr_without_ioam() goes through
skb_header_pointer(), is bounded by skb->len, and falls back on failure.

Could the helper take a base offset and also be used for the quoted
header? No later patch in the series changes this.

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261028-ipv6_ioam_ecmp-v1-0-9b6715b6f4c0%40kylinos.cn

Re: [PATCH 2/3] ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 8 potential
issue(s) to consider.

Critical: 0 · High: 2 (1) · Medium: 3 · Low: 3

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [High] In ioam6_output() (net/ipv6/ioam6_iptunnel.c:416-417), the
  inline-mode re-lookup does not keep ICMPv6 errors hashed on their…
- [High] For locally generated packets that ioam6_output() handles in
  INLINE mode (or AUTO mode with !skb->dev), orig_dst came from…
- [Medium] ioam6_output() (ioam6_iptunnel.c:416) reads the per-route
  shared template field ilwt->tuninfo.eh.nexthdr into fl6.flowi6_proto…
- [Medium] The commit message credits the policy-0 fix to using
  eh.nexthdr as fl6.flowi6_proto in the re-lookup.
- [Medium] ipv6: route: wrong skb offset in ip6_nexthdr_without_ioam()
  via icmp6_send()
- [Low] Because mp_hash is precomputed from the skb, the main user-
  visible effect of changing fl6.flowi6_proto from hdr->nexthdr…
- [Low] fib6_select_path() (route.c:464-469) treats fl6->mp_hash == 0 as
  'not computed'.
- [Low] ipv6: ioam6: duplicate ip6_route.h include in ioam6_iptunnel.c

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the patch
  relies on the same racy template.

--

ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup

On a dst_cache miss, the ioam6 output path re-looks up the route. This
patch changes that lookup to use the Hop-by-Hop nexthdr stored in the
tunnel template (eh.nexthdr) as flowi6_proto instead of NEXTHDR_HOP. It
also precomputes fl6.mp_hash from the transformed skb with
rt6_multipath_hash(), so that the ECMP hash matches RFC 9486 (C2).

> diff --git a/net/ipv6/ioam6_iptunnel.c b/net/ipv6/ioam6_iptunnel.c
> index cfb2c41634a0b..713e625bfa3aa 100644
> --- a/net/ipv6/ioam6_iptunnel.c
> +++ b/net/ipv6/ioam6_iptunnel.c

[ ... ]

> @@ -412,7 +413,8 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
>  		fl6.saddr = hdr->saddr;
>  		fl6.flowlabel = ip6_flowinfo(hdr);
>  		fl6.flowi6_mark = skb->mark;
> -		fl6.flowi6_proto = hdr->nexthdr;
> +		fl6.flowi6_proto = ilwt->tuninfo.eh.nexthdr;

[Severity: Medium]
Is ilwt->tuninfo.eh.nexthdr guaranteed to still belong to this packet
when it is read here?

The template is shared per route. Every CPU sending through it writes
this field with plain stores:

ioam6_do_inline():
	tuninfo->eh.nexthdr = hdr->nexthdr;

ioam6_do_encap():
	tuninfo->eh.nexthdr = NEXTHDR_IPV6;

Nothing serializes the lwtunnel output path. Between the return of
ioam6_do_inline()/ioam6_do_encap() and this read, another CPU may have
stored a different packet's nexthdr. In inline mode that could be
TCP, UDP or ICMPv6. In auto mode the inline and encap values alternate.

The protocol read here is used by fib6_rule_match() for ipproto
selectors. fib6_select_path() also falls back to it when the
precomputed mp_hash is 0. The resulting dst is then cached in
ilwt->cache.

Could the value come from a local instead, for example
ipv6_hdr(skb)->nexthdr saved before ioam6_do_inline(), or NEXTHDR_IPV6
for encap? KCSAN would probably also flag this plain read/write pair.

[Severity: High]
This isn't a bug introduced by this patch, but the new read depends on
the same shared template, and the write side of it already looks racy.

ioam6_do_inline() stores to the template and then copies it into the
packet:

	tuninfo->eh.nexthdr = hdr->nexthdr;

	skb_set_transport_header(skb, sizeof(*hdr));
	skb_postpush_rcsum(skb, hdr, sizeof(*hdr) + hdrlen);

	memcpy(skb_transport_header(skb), (u8 *)tuninfo, hdrlen);

ioam6_do_encap() does the same with NEXTHDR_IPV6:

	tuninfo->eh.nexthdr = NEXTHDR_IPV6;
	memcpy(skb_transport_header(skb), (u8 *)tuninfo, hdrlen);

There is no locking here. Can another CPU's store land between this
CPU's store and its memcpy()? If so, the packet goes out with an HBH
nexthdr that does not describe its payload.

With mixed-protocol inline traffic, a TCP segment could be labelled
as UDP, and the receiver would drop it.

In auto mode, a local packet could be labelled nexthdr=41. The receiver
would then parse the UDP header and payload as an inner IPv6 header,
so UDP payload bytes 0-31 become the inner saddr/daddr. If the
destination runs an ip6ip6 decap tunnel, this might allow spoofed inner
packets to be injected there. That part depends on how the receiver is
configured.

[Severity: Low]
mp_hash is now precomputed from the skb. So the main visible effect of
this flowi6_proto change seems to be on policy routing, not on the
ECMP hash. fib6_rule_match() does:

	if (rule->ip_proto && (rule->ip_proto != fl6->flowi6_proto))
		return 0;

Before this patch, "ip -6 rule ... ipproto X" never matched the ioam6
re-lookup. Now it matches the inner L4 protocol in inline mode and
IPPROTO_IPV6 in encap mode. IOAM traffic can therefore be routed
through a different table than before.

This is arguably more consistent with the original lookup. Could the
commit message mention this routing behaviour change?

> +		fl6.mp_hash = rt6_multipath_hash(net, &fl6, skb, NULL);

[Severity: Medium]
This isn't introduced by this patch. It is in the helper added by the
earlier patch in this series, which this re-lookup now depends on, so
I am raising it here.

ip6_nexthdr_without_ioam() looks for the Hop-by-Hop header at
off = sizeof(*iph). It passes that offset to skb_header_pointer(),
which indexes from skb->data. That only works when skb->data points
at the IPv6 header. It does in ioam6_output(), but not for every
caller of rt6_multipath_hash().

icmp6_send() also calls rt6_multipath_hash() with the offending skb.
For locally delivered packets, ip6_protocol_deliver_rcu() has already
done:

	pskb_pull(skb, skb_transport_offset(skb))

After that, skb->data sits at the transport header and
skb_network_offset(skb) is negative. Two cases reach the helper with
such an skb when the packet carries a Hop-by-Hop header:

- udpv6_rcv() sends ICMPV6_PORT_UNREACH
- ip6_protocol_deliver_rcu() reports an unknown nexthdr

In both, ip6_multipath_l3_keys() calls ip6_nexthdr_without_ioam(),
which then parses L4 payload bytes as a Hop-by-Hop header and walks
them as options.

skb_header_pointer() keeps the reads bounds-checked. The problem is
the ip_proto fed into the policy 0 hash for the ICMPv6 error. It is
no longer the real HBH nexthdr. It is either derived from
remote-supplied payload bytes, or it falls back to NEXTHDR_HOP when
the payload is short. So the precomputed mp_hash in icmp6_send() no
longer reflects the offending flow, and the sender can influence which
ECMP sibling the error takes.

Should the helper start at skb_network_offset(skb) + sizeof(*iph), or
derive the offset from iph? That would match the rest of
ip6_multipath_l3_keys(), which works from the header offsets rather
than assuming skb->data is the network header.

[Severity: High]
Does this keep ICMPv6 errors hashed on their quoted flow in inline mode,
as the commit message says?

ioam6_do_inline() leaves the transport header pointing at the inserted
Hop-by-Hop header:

	skb_set_transport_header(skb, sizeof(*hdr));
	...
	memcpy(skb_transport_header(skb), (u8 *)tuninfo, hdrlen);

Under policy 0 and the policy 2 fallback, rt6_multipath_hash() calls
ip6_multipath_l3_keys(). That function gets IPPROTO_ICMPV6 back from
ip6_nexthdr_without_ioam(), but then reads the ICMPv6 header at the
transport offset:

	icmph = skb_header_pointer(skb, skb_transport_offset(skb),
				   sizeof(_icmph), &_icmph);
	...
	if (!icmpv6_is_err(icmph->icmp6_type))
		goto out;

Those bytes are the HBH header, so icmp6_type is eh.nexthdr (58) and
icmpv6_is_err() returns false. The outer addresses and flowlabel are
hashed instead of the quoted flow.

orig_dst was chosen on the quoted flow. For forwarded errors that was
done by ip6_route_input(). For local errors, icmp6_send() precomputes
fl6.mp_hash from the offending skb.

On a dst_cache miss, can this re-lookup pick a different ECMP sibling
and cache it for this lwtstate?

Also, this comment in ip6_nexthdr_without_ioam() does not hold for the
new output-path caller:

	/* The Hop-by-Hop header was already pulled and validated by
	 * ipv6_parse_hopopts() before routing, so this walk is safe.
	 */

Would it work to compute the hash before the HBH is inserted, or to
find the ICMPv6 header past the IOAM HBH?

[Severity: High]
Locally generated packets take inline mode, or auto mode with
!skb->dev. For them, orig_dst came from ip6_route_output() with no
skb, so fib6_select_path() used the fl6 branch of
rt6_multipath_hash(). This re-lookup uses the skb branch instead. For
the same flow, the two branches hash different keys:

Policy 0 with default sysctls: the socket lookup hashes
flowi6_get_flowlabel(fl6), which is 0 unless a label was set. The skb
branch hashes ip6_flowlabel(hdr), which is the auto flowlabel written
by ip6_make_flowlabel(). auto_flowlabels is on by default.

Policy 1: skb_set_hash_from_sk() sets skb->l4_hash from sk_txhash, so
the skb branch returns early instead of hashing the fl6 5-tuple:

	if (skb->l4_hash)
		return skb_get_hash_raw(skb) >> 1;

Policy 3: the skb branch uses rt6_multipath_custom_hash_skb(), while
the original lookup used rt6_multipath_custom_hash_fl6().

Each nexthop has its own lwtstate. A different sibling would fail the
orig_dst->lwtstate check, and its dst would be cached in ilwt->cache
for later packets.

In the default configuration, is the policy 0 divergence described in
the commit message still reachable for local traffic? Closing it seems
to need the re-lookup to reproduce the original lookup's hash. Changing
flowi6_proto alone would not do that.

[Severity: Medium]
The commit message credits the policy 0 fix to using eh.nexthdr as
flowi6_proto. With a non-NULL skb, though, no rt6_multipath_hash()
policy reads fl6->flowi6_proto:

- policy 0 and the policy 2 fallback use ip6_multipath_l3_keys()
- policy 1 uses the flow dissector
- policy 3 uses rt6_multipath_custom_hash_skb()

fib6_select_path() only recomputes when the precomputed value is zero:

	if (!fl6->mp_hash &&
	    (!match->nh || nexthop_is_multipath(match->nh)))
		fl6->mp_hash = rt6_multipath_hash(net, fl6, skb, NULL);

So the policy 0 behaviour comes entirely from
ip6_nexthdr_without_ioam(), which the earlier patch "ipv6: route:
consider IOAM options in ECMP hash policy 0" added. This commit still
carries its own tag:

    Fixes: 8cb3bf8bff3c ("ipv6: ioam: Add support for the ip6ip6 encapsulation")

If this commit is backported alone, ip6_multipath_l3_keys() would hash
outer_iph->nexthdr == NEXTHDR_HOP, and the stated bug would not be
fixed.

Could the commit message describe this dependency, and say which change
actually affects the hash?

[Severity: Low]
fib6_select_path() treats mp_hash == 0 as "not computed". Its comment
assumes that only ICMPv6 errors precompute, and that the value is
always non-zero:

	/* We might have already computed the hash for ICMPv6 errors. In such
	 * case it will always be non-zero. Otherwise now is the time to do it.
	 */

rt6_multipath_hash() returns mhash >> 1, and __flow_hash_from_keys()
maps 0 to 1, so the result can be 0. The policy 1 path,
skb_get_hash_raw(skb) >> 1, can also return 0. When that happens,
fib6_select_path() recomputes from fl6 with no skb, and the skb-based
hash is lost.

This is rare, and icmp6_send() already has the same sentinel collision.
Since ioam6_output() now precomputes too, should the comment be
updated, and is the zero case worth handling?

>  
>  		dst = ip6_route_output(net, NULL, &fl6);
>  		if (dst->error) {

[ ... ]

> @@ -13,6 +13,7 @@
>  #include <linux/ioam6.h>
>  #include <linux/ioam6_iptunnel.h>
>  #include <net/dst.h>
> +#include <net/ip6_route.h>
>  #include <net/sock.h>
>  #include <net/lwtunnel.h>
>  #include <net/ioam6.h>

[Severity: Low]
This has no runtime effect, but ioam6_iptunnel.c already includes
<net/ip6_route.h> further down, after <net/dst_cache.h>. The new
include at the top is a duplicate of it.

rt6_multipath_hash() was already visible through the existing
include. Could this hunk be dropped?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261028-ipv6_ioam_ecmp-v1-0-9b6715b6f4c0%40kylinos.cn

Re: [PATCH 3/3] selftests: net: add ioam6 ECMP test (RFC 9486 C2)

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 7 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 2 · Low: 5

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] The compatibility probe in MAIN (lines 308-315) only checks
  that an inline-mode IOAM trace can be attached (`encap ioam6 mode…
- [Medium] In MAIN, `setup_topology || { log_test_skip "Setup failed.";
  exit ...; }` (line 305) runs before `trap cleanup EXIT` (line 306).
- [Low] The commit message and the ioam6_ecmp.sh header comment describe
  scenario 2 differently from what the script does.
- [Low] ioam6_ecmp.sh skips the whole test when
  /proc/sys/net/ipv4/fib_multipath_hash_seed is missing (lines 300-303).
- [Low] The comment in set_beta_tnl() (lines 139-143) says "ip6_tnl
  caches the outer dst per CPU for a few seconds".
- [Low] setup_topology() does not check the return value of the multipath
  route install, which is the core of the test (`ip -netns $ns_beta -6…
- [Low] setup_tunnel() (lines 274-287) decides whether the test loaded
  ip6_tunnel only from `lsmod | grep -q ip6_tunnel`.

--

selftests: net: add ioam6 ECMP test (RFC 9486 C2)

This adds ioam6_ecmp.sh. The test checks that, with hash policy 0, the
ECMP nexthop chosen for a packet carrying IOAM options is the same as
for the packet without them. It covers a transit node forwarding
inline-inserted IOAM Hop-by-Hop packets, and an encapsulating node
doing the ioam6 encap re-lookup with a plain ip6ip6 tunnel as the
baseline.

> diff --git a/tools/testing/selftests/net/ioam6_ecmp.sh b/tools/testing/selftests/net/ioam6_ecmp.sh
> new file mode 100755
> index 0000000000000..c5c82884c7579
> --- /dev/null
> +++ b/tools/testing/selftests/net/ioam6_ecmp.sh
> @@ -0,0 +1,349 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0+

[ ... ]

> +# 2) encapsulating node: beta itself encapsulates (ioam6 encap mode,
> +#    tunnel dst = original dst, tunnel src pinned) over a multipath
> +#    route. The ioam6 re-lookup hashed NEXTHDR_HOP instead of the
> +#    transformed packet's real nexthdr (IPPROTO_IPV6). The baseline is
> +#    the same encapsulation without the IOAM option: a plain ip6ip6
> +#    tunnel over the same multipath route, with the same outer
> +#    addresses.

[Severity: Low]
Is "tunnel dst = original dst" accurate here? The commit message says
the same thing:

  - encapsulating node re-lookup: the forwarder itself encapsulates
    (ioam6 encap mode, tunnel dst equal to the original dst) over a
    multipath route, with a plain ip6ip6 tunnel as the
    identical-packet-without-IOAM baseline.

The script sends to 2001:db8:3::2, but set_beta_ioam_encap() installs
this route:

  2001:db8:3::2/128 encap ioam6 mode encap ... tundst 2001:db8:2:1::2

So the tunnel dst is not the original dst. The topology comment further
down describes this correctly.

If the tunnel dst really were the original dst, the re-lookup would
resolve back to the same ioam6 route:

net/ipv6/ioam6_iptunnel.c:ioam6_output() {
    ...
		if (orig_dst->lwtstate == dst->lwtstate)
			dst_cache_set_ip6(&ilwt->cache,
					  &ilwt->null_rt.dst, &fl6.saddr);
    ...
}

In that case orig_output would be used and the multipath route would
never be consulted.

The ip6ip6 baseline is also not an identical packet on the wire.
ip6_tnl_xmit() pushes an encap-limit Destination Options header when
encap_limit >= 0, and it sets the flow label with ip6_make_flowlabel()
after the route lookup. Only the hash inputs at beta's lookup match.

Could the commit message and this header comment be updated to match
what the test actually does?

[ ... ]

> +set_beta_tnl()
> +{
> +	# Recreate the tunnel: ip6_tnl caches the outer dst per CPU for a
> +	# few seconds, and neither the route change below nor the
> +	# fib_multipath_hash_seed write resets it. A stale cache would
> +	# resolve the outer sibling under a previous seed (or on another
> +	# CPU), making the baseline flip between attempts of one seed.

[Severity: Low]
This isn't a bug, but is "for a few seconds" accurate? ip6_tnl_xmit()
reads the cache with dst_cache_get(&t->dst_cache), and
dst_cache_per_cpu_get() only drops an entry in two cases:

net/core/dst_cache.c:dst_cache_per_cpu_get() {
    ...
	if (unlikely(!time_after(idst->refresh_ts,
				 READ_ONCE(dst_cache->reset_ts)) ||
		     (READ_ONCE(dst->obsolete) && !dst->ops->check(dst, idst->cookie)))) {
    ...
}

That means an explicit dst_cache_reset(), or an obsolete dst that fails
->check(). There is no time-based expiry, so a cached outer dst can
stay indefinitely.

Recreating the tunnel for every seed is still the right workaround. The
comment, though, could lead someone to replace the recreation with a
sleep.

[ ... ]

> +set_beta_ioam_encap()
> +{
> +	ip -netns "$ns_beta" -6 route replace 2001:db8:3::2/128 \
> +		encap ioam6 mode encap tunsrc 2001:db8:2:1::1 tundst 2001:db8:2:1::2 \
> +		trace prealloc type 0x800000 ns 0 size 12 dev veth1
> +}

[Severity: Medium]
What happens on a kernel or iproute2 that supports ioam6 inline and
encap modes but not the tunsrc attribute? The compat check in MAIN
only probes inline mode, so it passes, and setup_tunnel() succeeds.

This route install would then fail on every seed, and run_scenario()
takes this path:

		if ! $ioam; then
			RET=$ksft_fail
			log_test "$desc (ioam route setup failed, seed $seed)"

The result is 12 hard FAILs instead of a SKIP. ioam6.sh already treats
tunsrc as a feature under test.

Is the tunsrc pin needed at all? Without it, ioam6_do_encap() calls
ipv6_dev_get_saddr() on the original dst device (veth1), which should
pick 2001:db8:2:1::1 anyway.

[ ... ]

> +	# The ECMP route under test (scenario 1 hashes its selection at
> +	# beta's input; scenario 2 at the encapsulating re-lookup).
> +	ip -netns "$ns_beta" -6 route add 2001:db8:2:1::2/128 \
> +		nexthop via 2001:db8:2:1::2 dev veth1 \
> +		nexthop via 2001:db8:2:2::2 dev veth2

[Severity: Low]
Should this route add check its return value? If it fails, beta still
has the connected 2001:db8:2:1::/64 prefix on veth1, so all traffic to
2001:db8:2:1::2 leaves through veth1 and arrives on gamma's veth0.

measure_link() would then return "A" for both variants on every seed.
run_scenario() would report PASS even though no ECMP route exists.

Adding "|| return 1" here would catch this.

[ ... ]

> +setup_tunnel()
> +{
> +	if ! lsmod 2>/dev/null | grep -q "ip6_tunnel"; then
> +		modprobe ip6_tunnel &>/dev/null
> +		lsmod 2>/dev/null | grep -q "ip6_tunnel" || \
> +			ip a 2>/dev/null | grep -q "ip6tnl0" || return 1
> +		ip6tnl_loaded=1
> +	fi

[Severity: Low]
Can this unload an ip6_tunnel module the test did not load? Consider
lsmod being unavailable or failing while modprobe works, with
ip6_tunnel already loaded:

  - the first grep fails;
  - modprobe ip6_tunnel succeeds as a no-op;
  - grep ip6tnl0 matches the existing fallback device in the init
    netns;
  - ip6tnl_loaded=1 is set.

cleanup() then runs:

		modprobe -r ip6_tunnel &>/dev/null

This removes the module if nothing else is using it.

When ip6_tunnel is built in, lsmod does not list it either, so
ip6tnl_loaded=1 is set in that case too. The later modprobe -r then
fails harmlessly. ioam6.sh uses the same heuristic.

[ ... ]

> +if [ ! -e /proc/sys/net/ipv4/fib_multipath_hash_seed ]; then
> +	log_test_skip "Kernel without CONFIG_IP_ROUTE_MULTIPATH."
> +	exit "${EXIT_STATUS}"
> +fi

[Severity: Low]
Should tools/testing/selftests/net/config be updated for this test?
fib_multipath_hash_seed only exists with CONFIG_IP_ROUTE_MULTIPATH.
That option depends on CONFIG_IP_ADVANCED_ROUTER, neither has a
default, and neither is in the net selftests config fragment.

On a kernel built from a minimal base plus that fragment, this test
would always SKIP.

CONFIG_IPV6_TUNNEL, which scenario 2 needs, is not listed explicitly
either. It only gets enabled because CONFIG_IPV6_GRE=m selects it.

> +
> +setup_topology || { log_test_skip "Setup failed."; exit "${EXIT_STATUS}"; }
> +trap cleanup EXIT

[Severity: Medium]
Can this leak ns_alpha, ns_beta and ns_gamma? setup_ns has already
created them by the time setup_topology() returns failure. The trap is
installed only after this exit, so cleanup_all_ns never runs and the
namespaces and any veths stay on the host.

setup_topology() can fail after setup_ns through the veth "|| return 1"
paths. Its exit status is also that of its last command:

	tc -n "$ns_alpha" filter add dev veth0 ingress pref 1 \
                protocol ipv6 flower skip_hw ip_proto icmpv6 type 1 action drop

That command fails on a kernel without cls_flower or act_gact.
setup_ns only cleans up when ip netns add itself fails, and lib.sh
does not install an EXIT trap.

As a side effect, a missing flower is reported as "Setup failed.", and
the later "Cannot install tc flower measurement filters" skip is never
reached for that cause.

Would installing the trap before calling setup_topology() fix this?

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261028-ipv6_ioam_ecmp-v1-0-9b6715b6f4c0%40kylinos.cn