[RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider)

[RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider)

[Omar Ramadan]

This is an RFC and is NOT for merge. We are posting it to find a
provider: a Linux-attached device (DPU, SmartNIC, switch ASIC or P4
target) whose driver would implement the contract below. We will not
post a for-merge version until one in-tree driver with a named
maintainer implements it, either as a co-authored patch on this list
or as a public commitment from that driver's maintainer. Until then
the contract has no in-tree user other than netdevsim, and we do not
think it should be merged on netdevsim alone.

Problem

An AMT relay (RFC 7450) replicates every multicast data packet once
per gateway tunnel: one UDP-encapsulated unicast copy per gateway that
joined the group. In drivers/net/amt.c that is the loop in
amt_dev_xmit(), one skb copy and one route lookup per tunnel. At the
relay's egress the copy count, not the input rate, sets the
ceiling. We are bringing up an AMT relay port at the SFMIX exchange,
which is where that ceiling becomes concrete for us.

Hardware that can replicate one packet to N unicast destinations
exists, but there is no way for the amt driver to hand that work to
it. This series proposes one.

What the series does

  Patch 1 adds include/net/amt_offload.h: provider capabilities,
    opaque provider-owned endpoint and group handles, struct
    amt_offload_ops, the TX and ingress paths, the FALLBACK /
    CONSUMED / BUSY results and a per-binding u64 epoch. It is an
    in-kernel interface with no ABI promise.
  Patch 2 adds the provider registry, keyed by the lower netdev
    (amt_offload_register() / amt_offload_unregister()), and the
    lower-device lifecycle: a relay in auto or required mode binds
    when its stream device is a registered lower and the provider
    accepts it (otherwise it stays unbound, counted), and unbinds in
    a fixed order on NETDEV_UNREGISTER, provider unregister or relay
    deletion.
  Patch 3 adds one group object per (family, group) with its member
    endpoints, a generation and the desired state, plus a reconciler
    that programs the provider from process context, never under
    amt->lock, with a per-tick op budget, ENOSPC hysteresis with a
    software remainder, and reset-and-replay on op timeout.
  Patch 4, on a TX binding, hands each relay data packet of a group
    whose commit amt has published to the provider before the
    software loop, while the binding admits packets and its desired
    state is in sync; every other packet goes to software. Exactly
    one actor handles each packet handed over: CONSUMED means the
    provider handled every copy (sent it, or dropped and counted it)
    and owns the skb; FALLBACK and BUSY mean the provider sent no
    copy and amt runs the software loop once. It also exports
    amt_offload_encap_xmit(), so a simulated provider emits copies
    byte-identical to amt's own.
  Patch 5 adds IFLA_AMT_OFFLOAD_MODE (software, the default; auto;
    required). required with no usable provider refuses the link with
    -ENODEV and extack.
  Patch 6 adds 26 KUnit cases for the contract against an in-file
    fake provider.
  Patch 7 adds a TX reference provider to netdevsim. It keeps its own
    endpoint, group and generation tables, decides replication from
    them and emits each copy over the peer link. debugfs injects
    ENOSPC, reset and BUSY; devlink resources and an amt health
    reporter are registered on netdevsim's own devlink instance.
  Patch 8 adds tools/testing/selftests/net/amt_offload.sh. It counts
    the copies each gateway receives, and reads the provider's own
    hardware copy count, in the no-member, commit, generation-replace,
    BUSY, per-group forwarding, reset, partial-commit, disable and
    unlink-under-traffic cases. It also checks that required mode
    with no provider is refused, and that deleting a lower carrying
    two amt devices removes both without a warning.
  Patch 9 adds the new header and the netdevsim provider to the AMT
    MAINTAINERS entry.

Forwarding semantics are unchanged by design. The hardware key is
(family, group), because that is what the software path forwards on
today: a tunnel with only an INCLUDE(S1) membership still receives S2
for that group. A provider that keyed on (S, G) would change what
gateways receive, so the feature bit for it is reserved with no
semantics in this version.

With no provider registered, or with offload=software (the default),
amt_dev_xmit() runs the same loop it runs today, after one more RCU
pointer read and test (patch 4).

Questions for the list

1. This RFC registers providers in a registry keyed by the lower
   netdev and adds nothing to struct net_device or net_device_ops.
   Once a real driver exists, would you rather see a struct net_device
   field or an ndo instead?
2. Is netdevsim the right home for the reference provider, given it
   has no ingress classifier to simulate and so can only exercise the
   TX path?
3. Is an in-kernel provider contract for a single tunnel driver
   acceptable at all, or would you want it expressed through an
   existing offload framework? We did not find one that models
   one-to-N unicast replication with per-destination encapsulation.
4. The per-relay offload state and counters are exposed in a sysfs
   offload/ group on the amt device for this RFC. We expect to move
   them to netlink (link xstats) for a for-merge version, and would
   like guidance on that.

Performance

This series makes no performance claim. With no provider it adds one
more RCU pointer read and test per packet to the software path
(patch 4), which we have not measured, and netdevsim demonstrates the
contract's semantics, not speed.

For context only, we measured a separate software fan-out change,
which is not part of this series and has not been posted: it clones
each copy instead of copying it, clears the copy's hash and caches
each tunnel's outer route. Base and branch are both builds of our
downstream 6.18-based amt.c, out of tree against v6.18.48, not of
net-next's driver. Base is that driver just before the change. Branch
adds the change and an earlier 6.18 form of this series' patches 1, 2
and 5, which add no code to the data path; patch 4 was not in it. CPU
time per copy, median ns over 6 interleaved runs on two days,
virtme-ng guest, software path, no provider bound:

    N tunnels   base    branch   branch/base
            8   1898      1326         0.699
           64   1974      1392         0.705
          512   2480      1612         0.650

The table pools both days. Day 2 reproduced the direction but not the
spread: on day 2 alone the branch/base ratio was 0.706-0.730 (a 27-29%
saving), and the branch's day-2 medians at N=64 and N=512 fell outside
their day-1 ranges. The saving we would defend is therefore ~27-35%,
not a single figure.

No drops were counted on any step. We have not yet measured a
hardware AMT relay (a router-terminated relay, which would measure
relay capacity rather than this contract) against the Linux software
path, and the first-loss point is not yet measured on a physical NIC,
so we quote no hardware rate here.

We have also prepared a vendor capability appendix covering Linux amt,
Junos, Cisco, Arista and DPU/SmartNIC replication, with every row
either cited or marked unconfirmed. It is available on request to
anyone evaluating whether their device could be a provider.

Known limitation: tunnel lifetime

The offload desired state is only as sound as the relay's tunnel
lifetime, and the hooks in patches 3 and 4 inherit three windows that
exist in amt.c today:

  - an Update can add a group node, and with this series an offload
    member, after amt_clear_groups() on a tunnel that is about to be
    freed by kfree_rcu();
  - an Update can re-arm a tunnel's gc_wq while amt_tunnel_expire()
    unlinks and frees it, so the expiry, and with this series
    amt_ofl_tunnel_gone(), can run again on freed memory;
  - amt_dev_stop() unlinks a tunnel before
    cancel_delayed_work_sync(gc_wq).

None is introduced here. All three are closed by the in-review net
series "[PATCH net v6 0/6] amt: relay tunnel lifetime, validation and
request-path hardening"
(<20261107004935.2627329-1-omar@blockcast.net>), which a for-merge
version of this series will depend on.

Dependencies

This does not apply to bare net-next, and patchwork will report it as
not applying. It is written on v7.1 plus eight of our other amt
series, not all of them posted:

  - relay endpoint keying and visible pre-Query report drops, later
    versions of which are part of the net series cited above;
  - sending the relay's General Query directly, a later version of
    which is in mainline as commit afae89de73dd ("amt: send the
    relay's General Query directly from the receive path");
  - the IPv6 relay and gateway modes, a reworked version of which was
    posted, in a different split, as "[PATCH net-next 00/13] amt: add
    an IPv6 outer transport"
    (<20261009122426.551178-1-omar@blockcast.net>), on which changes
    were requested;
  - relay scaling (configurable hash buckets, group limit and queue
    count), the relay sending IGMPv3/MLDv2 host reports upstream, and
    a configurable per-source tunnel cap, none of which has been
    posted.

The net series instead bounds the tunnels admitted per source address
at a fixed 16. Our "[PATCH net-next v2] amt: make the per-tunnel group
and source limits configurable"
(<20261015231825.2852054-1-omar@blockcast.net>), on which changes were
requested, overlaps the scaling group limit, and its planned v3 also
sizes the hash buckets.

Patch 8's case for a lower carrying two amt devices also needs our fix
"amt: tear down every amt device on an unregistering lower"
(<20261105232302.494171-1-omar@blockcast.net>), posted to net
separately. Without it, amt_device_event() unregisters only the first
amt device on an unregistering lower, so the second stays linked and
trips the upper-device WARN in unregister_netdevice_many_notify().

To build the series and run its KUnit suite and the existing amt.sh
selftest, fetch this git branch:

  https://github.com/Blockcast/linux.git amt-offload-rfc-v1

It carries v7.1, the eight series, this RFC and a copy of that fix,
rebased onto this series, in that order.

IFLA_AMT_OFFLOAD_MODE is 16 on this base, after the seven attributes
those series add. None of those attributes is upstream, so it will be
renumbered to follow whatever is accepted. An iproute2 patch (ip link
... offload {software|auto|required}) is held until the uapi is
accepted. The patch 8 selftest skips when ip lacks the offload
keyword.

On that branch, under KASAN and lockdep, amt_offload.sh passes all 11
cases, amt.sh all 5 and the amt KUnit suite all 29, with no splat. The
ip came from our held iproute2 patches, the offload one renumbered for
this base (IFLA_AMT_OFFLOAD_MODE = 16); we can send them on request.

Testing: on this base, every patch builds with W=1 and the amt KUnit
suite (the 3 existing cases plus the 26 new ones) passes under UML
with CONFIG_IPV6=y and =n. net-next has no amt KUnit suite: the suite
and its 3 existing cases come from the unposted series that has the
relay send host reports upstream.

Omar Ramadan (9):
  amt: add the relay replication offload provider header
  amt: offload provider registry and lower-device lifecycle
  amt: offload group object, desired state and reconciler
  amt: offload TX handoff, ingress punt rule and counters
  amt: add IFLA_AMT_OFFLOAD_MODE
  amt: KUnit cases for the offload contract
  netdevsim: add a TX reference provider for AMT relay offload
  selftests: net: add amt_offload.sh for AMT relay offload
  MAINTAINERS: amt: cover the offload header and netdevsim provider

 MAINTAINERS                                |    2 +
 drivers/net/Kconfig                        |    1 +
 drivers/net/amt.c                          | 5036 +++++++++++++++++---
 drivers/net/netdevsim/Makefile             |    4 +
 drivers/net/netdevsim/amt.c                |  600 +++
 drivers/net/netdevsim/dev.c                |   68 +
 drivers/net/netdevsim/health.c             |   28 +-
 drivers/net/netdevsim/netdev.c             |   19 +-
 drivers/net/netdevsim/netdevsim.h          |   27 +
 include/net/amt.h                          |   72 +
 include/net/amt_offload.h                  |  353 ++
 include/uapi/linux/amt.h                   |   27 +
 tools/testing/selftests/net/Makefile       |    1 +
 tools/testing/selftests/net/amt_offload.sh |  608 +++
 tools/testing/selftests/net/config         |    2 +
 15 files changed, 6214 insertions(+), 634 deletions(-)
 create mode 100644 drivers/net/netdevsim/amt.c
 create mode 100644 include/net/amt_offload.h
 create mode 100755 tools/testing/selftests/net/amt_offload.sh

-- 
2.43.0

[RFC PATCH net-next 1/9] amt: add the relay replication offload provider header

[Omar Ramadan]

include/net/amt_offload.h: provider caps, opaque provider-owned endpoint
and group handles, struct amt_offload_ops, TX/INGRESS paths,
FALLBACK/CONSUMED/BUSY results, the per-binding u64 epoch, and the
amt_offload_register()/amt_offload_unregister() entry points. An
in-kernel interface with no ABI promise; nothing touches struct
net_device or net_device_ops.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 include/net/amt_offload.h | 263 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 263 insertions(+)
 create mode 100644 include/net/amt_offload.h

diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
new file mode 100644
index 000000000..3be5d129c
--- /dev/null
+++ b/include/net/amt_offload.h
@@ -0,0 +1,263 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * AMT relay replication offload: the contract between the amt driver and
+ * a provider that replicates relay multicast data to gateways in hardware
+ * (or, for testing, in a simulated device).
+ *
+ * This is an IN-KERNEL INTERFACE WITH NO ABI PROMISE. It may change in any
+ * release, like every other in-kernel API; out-of-tree providers get no
+ * stability guarantee.
+ *
+ * Registration is keyed by the lower netdev a relay transmits on (its
+ * IFLA_AMT_LINK stream device). Nothing here adds a field or an ndo to
+ * struct net_device.
+ */
+#ifndef _NET_AMT_OFFLOAD_H_
+#define _NET_AMT_OFFLOAD_H_
+
+#include <linux/bits.h>
+#include <linux/in6.h>
+#include <linux/types.h>
+
+struct net_device;
+struct sk_buff;
+
+/* Outer transport families, for &amt_offload_caps.families. */
+#define AMT_OFFLOAD_FAMILY_V4		BIT(0)
+#define AMT_OFFLOAD_FAMILY_V6		BIT(1)
+
+/*
+ * Reserved: (S,G)-keyed forwarding. v1 forwards per (family, group) exactly
+ * as the software path does, so this bit has NO v1 semantics. A provider
+ * must not set it; amt ignores it.
+ */
+#define AMT_OFFLOAD_F_SOURCE_FILTER	BIT(0)
+
+/**
+ * enum amt_offload_path - where a provider replicates
+ * @AMT_OFFLOAD_PATH_TX: amt hands the provider each relay data packet of
+ *	a group whose commit it has published (&amt_offload_ops.group_commit),
+ *	before its software fan-out (&amt_offload_ops.xmit), while the binding
+ *	admits packets and amt has not lost its desired state. amt withdraws a
+ *	group without a callback when a member leaves, until it publishes the
+ *	recommit. Every other packet goes straight to software.
+ * @AMT_OFFLOAD_PATH_INGRESS: the provider replicates on ingress and the
+ *	host only sees punted packets. Semantics only until a real provider
+ *	exists; no simulated provider implements it.
+ */
+enum amt_offload_path {
+	AMT_OFFLOAD_PATH_TX,
+	AMT_OFFLOAD_PATH_INGRESS,
+};
+
+/**
+ * struct amt_offload_caps - what a bound provider can do
+ * @families: AMT_OFFLOAD_FAMILY_* outer families it can encapsulate. A
+ *	relay whose outer family is missing stays unbound (counted), with the
+ *	outcome of a failed &amt_offload_ops.bind, except that amt first calls
+ *	&amt_offload_ops.unbind, since that bind succeeded.
+ * @paths: BIT(enum amt_offload_path) it implements.
+ * @op_timeout_ms: longest a programming op may take before amt treats the
+ *	provider as wedged, resets it and replays from desired state.
+ * @features: AMT_OFFLOAD_F_* bits. None has v1 semantics.
+ * @max_endpoints: endpoint table size; 0 means unbounded. Informational:
+ *	amt does not read it in this version. A provider out of room fails an
+ *	&amt_offload_ops.endpoint_add for a key it does not hold, by
+ *	convention with -ENOSPC.
+ * @max_groups: group table size; 0 means unbounded. Informational: amt
+ *	does not read it in this version. A provider out of room fails the
+ *	&amt_offload_ops.group_update that needs a new group, by convention
+ *	with -ENOSPC.
+ *
+ * Filled by &amt_offload_ops.bind.
+ */
+struct amt_offload_caps {
+	u32 families;
+	u32 paths;
+	u32 op_timeout_ms;
+	u32 features;
+	u32 max_endpoints;
+	u32 max_groups;
+};
+
+/**
+ * struct amt_offload_binding - one relay device bound to one provider
+ * @lower: the registered lower netdev (the relay's stream device).
+ * @dev: the amt relay netdev.
+ * @epoch: monotonic per relay device, bumped on every bind, reset and
+ *	teardown. A completion carrying an older epoch is stale and ignored.
+ * @priv: provider-owned; amt never touches it.
+ *
+ * Allocated and freed by amt. Valid from &amt_offload_ops.bind until
+ * &amt_offload_ops.unbind returns.
+ */
+struct amt_offload_binding {
+	struct net_device *lower;
+	struct net_device *dev;
+	u64 epoch;
+	void *priv;
+};
+
+/**
+ * struct amt_offload_endpoint_key - one gateway's outer endpoint
+ * @family: AF_INET or AF_INET6, the relay's outer family.
+ * @addr: gateway outer address.
+ * @addr.ip4: IPv4 address when @family is AF_INET.
+ * @addr.ip6: IPv6 address when @family is AF_INET6.
+ * @port: gateway outer UDP port (tunnel identity is address + port).
+ */
+struct amt_offload_endpoint_key {
+	u8 family;
+	union {
+		__be32 ip4;
+		struct in6_addr ip6;
+	} addr;
+	__be16 port;
+};
+
+/**
+ * struct amt_offload_group_key - one replicated group
+ * @family: AF_INET or AF_INET6, the INNER group family.
+ * @group: group address.
+ * @group.ip4: IPv4 group when @family is AF_INET.
+ * @group.ip6: IPv6 group when @family is AF_INET6.
+ *
+ * Forwarding is per (family, group), never per source: a gateway holding
+ * an INCLUDE(S1) membership still receives S2, exactly as the software
+ * fan-out does today.
+ */
+struct amt_offload_group_key {
+	u8 family;
+	union {
+		__be32 ip4;
+		struct in6_addr ip6;
+	} group;
+};
+
+/* Provider-owned table entries. amt only stores the pointers. */
+struct amt_offload_endpoint;
+struct amt_offload_group;
+
+/**
+ * enum amt_offload_result - outcome of &amt_offload_ops.xmit
+ * @AMT_OFFLOAD_FALLBACK: skb untouched; amt runs its software fan-out.
+ * @AMT_OFFLOAD_CONSUMED: the provider handled every copy of this packet
+ *	under the generation it was given and owns the skb; amt emits none and
+ *	retries none. A copy that failed is the provider's to count, except one
+ *	amt_offload_encap_xmit() could not copy or route, which amt has already
+ *	counted in the relay's tx_dropped.
+ * @AMT_OFFLOAD_BUSY: skb untouched; amt runs one synchronous software pass
+ *	and counts it. Never NETDEV_TX_BUSY: the relay is IFF_NO_QUEUE.
+ *
+ * Exactly one actor handles each packet, so no packet is duplicated.
+ */
+enum amt_offload_result {
+	AMT_OFFLOAD_FALLBACK,
+	AMT_OFFLOAD_CONSUMED,
+	AMT_OFFLOAD_BUSY,
+};
+
+/**
+ * struct amt_offload_stats - provider-side counters
+ * @tx_packets: copies the provider emitted.
+ * @tx_bytes: bytes of those copies on the wire.
+ * @tx_dropped: copies the provider failed to emit.
+ */
+struct amt_offload_stats {
+	u64 tx_packets;
+	u64 tx_bytes;
+	u64 tx_dropped;
+};
+
+/**
+ * struct amt_offload_ops - provider callbacks
+ * @bind: a relay transmitting on the registered lower device starts using
+ *	this provider. Fill @caps. Process context, RTNL held. Nonzero leaves
+ *	the relay unbound (counted): in software in auto mode; in required
+ *	mode refused if it is being created, else without carrier.
+ * @unbind: last call for a binding. Every endpoint and group has already
+ *	been deleted or disabled. Free @b->priv. RTNL held.
+ * @endpoint_add: program one gateway endpoint. Idempotent: an identical
+ *	key returns the existing handle. 0 means accepted and durable, not yet
+ *	forwarding. Process context, never under amt's spinlocks.
+ * @endpoint_del: remove an endpoint no group references any more.
+ * @group_update: stage the member set of one group under generation @gen,
+ *	allocating *@grp on first use. Not forwarding until @group_commit.
+ * @group_commit: start forwarding generation @gen of @grp. amt hands @gen
+ *	to @xmit only after this returns 0, and may never do so: an overrun
+ *	resets the provider, and on TX a member leaving meanwhile withholds
+ *	the commit until the next one. Until amt publishes @gen, any @xmit for
+ *	@grp carries an earlier generation this provider committed, and a call
+ *	already in flight may carry any of those or @gen itself; the provider
+ *	serves such a generation or declines it. A provider must never emit
+ *	one packet to one endpoint under two generations. On failure the group
+ *	stays in software and previously added endpoints stay valid.
+ * @group_disable: stop forwarding @grp and free it.
+ * @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
+ *	&enum amt_offload_result. Softirq context, under rcu_read_lock().
+ * @flush: drop every endpoint and group (reset after a timeout). amt then
+ *	replays from its own desired state under a new epoch, except when
+ *	the op that overran was a @group_disable or @endpoint_del of amt
+ *	giving everything back -- at teardown, where @unbind follows, or
+ *	after its desired state was lost: then nothing is replayed and the
+ *	epoch does not move.
+ * @stats: fill @st. Optional; amt does not call it in this version.
+ *
+ * Teardown order, on unregistration of the lower device and on
+ * amt_offload_unregister(): amt stops admitting packets to @xmit, waits for
+ * in-flight @xmit calls, cancels its programming work, calls @group_disable
+ * for every group, @endpoint_del for every endpoint, then @unbind, then
+ * synchronize_rcu(); the relay is back in software afterwards. If one of
+ * those @group_disable/@endpoint_del calls overruns op_timeout_ms, amt
+ * calls @flush once instead of the rest, then @unbind, with no replay.
+ */
+struct amt_offload_ops {
+	int (*bind)(struct amt_offload_binding *b,
+		    struct amt_offload_caps *caps);
+	void (*unbind)(struct amt_offload_binding *b);
+	int (*endpoint_add)(struct amt_offload_binding *b,
+			    const struct amt_offload_endpoint_key *key,
+			    struct amt_offload_endpoint **ep);
+	void (*endpoint_del)(struct amt_offload_binding *b,
+			     struct amt_offload_endpoint *ep);
+	int (*group_update)(struct amt_offload_binding *b,
+			    const struct amt_offload_group_key *key,
+			    struct amt_offload_endpoint *const *members,
+			    unsigned int nr_members, u64 gen,
+			    struct amt_offload_group **grp);
+	int (*group_commit)(struct amt_offload_binding *b,
+			    struct amt_offload_group *grp, u64 gen);
+	void (*group_disable)(struct amt_offload_binding *b,
+			      struct amt_offload_group *grp);
+	enum amt_offload_result (*xmit)(struct amt_offload_binding *b,
+					struct amt_offload_group *grp,
+					struct sk_buff *skb, u64 gen);
+	void (*flush)(struct amt_offload_binding *b);
+	void (*stats)(struct amt_offload_binding *b,
+		      struct amt_offload_stats *st);
+};
+
+/**
+ * amt_offload_register - offer a provider for relays on @lower
+ * @lower: the netdev the relays transmit on.
+ * @ops: provider callbacks; must outlive the registration.
+ *
+ * Every relay already using @lower in auto or required mode tries to bind
+ * now. Caller holds RTNL.
+ *
+ * Return: 0, -EEXIST if @lower already has a provider, or -ENOMEM.
+ */
+int amt_offload_register(struct net_device *lower,
+			 const struct amt_offload_ops *ops);
+
+/**
+ * amt_offload_unregister - withdraw the provider for @lower
+ * @lower: as passed to amt_offload_register().
+ *
+ * Tears every binding down in the documented order and returns with every
+ * relay on @lower in software. Caller holds RTNL. A no-op if @lower has no
+ * provider, which is the case after @lower itself was unregistered.
+ */
+void amt_offload_unregister(struct net_device *lower);
+
+#endif /* _NET_AMT_OFFLOAD_H_ */
-- 
2.43.0

[RFC PATCH net-next 2/9] amt: offload provider registry and lower-device lifecycle

[Omar Ramadan]

amt_offload_register()/amt_offload_unregister() (EXPORT_SYMBOL_GPL) keep
an RCU list of providers keyed by the lower netdev. Relays are listed at
newlink; a relay in auto or required mode whose stream_dev has a
provider binds there, and registering a provider on a stream_dev binds
the auto and required relays already using it at once (counted). bind()
fills caps; a relay whose bind() fails, or whose outer family the caps
lack, stays unbound (counted): in software in auto mode, refused at
creation or left without carrier in required mode.

On NETDEV_UNREGISTER of the lower device (before amt_dellink), on
provider unregistration and on relay deletion, teardown runs in one
order: stop admission and wait out in-flight xmit, cancel programming
work, group_disable all, endpoint_del all, unbind, unpublish,
synchronize_rcu, software. A required-mode relay that loses its provider
drops carrier. Counters are under the relay's offload/ sysfs group; the
global required-refusal count is the offload_required_refused module
parameter.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/amt.c        | 425 +++++++++++++++++++++++++++++++++++++++
 include/net/amt.h        |  34 ++++
 include/uapi/linux/amt.h |  20 ++
 3 files changed, 479 insertions(+)

diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 201421ad0..034221e1c 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -20,6 +20,7 @@
 #include <net/icmp.h>
 #include <net/mld.h>
 #include <net/amt.h>
+#include <net/amt_offload.h>
 #include <uapi/linux/amt.h>
 #include <linux/security.h>
 #include <net/gro_cells.h>
@@ -3464,9 +3465,11 @@ static int amt_dev_stop(struct net_device *dev)
  * lifetime matches vxlan/geneve and the rest of the netdev tree.
  */
 static const struct attribute_group amt_upstream_group;
+static const struct attribute_group amt_offload_group;
 
 static const struct attribute_group *amt_groups[] = {
 	&amt_upstream_group,
+	&amt_offload_group,
 	NULL,
 };
 
@@ -4028,6 +4031,414 @@ static void amt_link_setup(struct net_device *dev)
 	ether_setup(dev);
 }
 
+/* Relay replication offload: provider registry and lower-device lifecycle.
+ *
+ * A provider registers against the lower netdev relays transmit on (their
+ * stream_dev); there is no net_device field or ndo. Every relay is linked
+ * into amt_offload_devs at newlink so registration can find the relays
+ * already using that lower device. Both lists are written under RTNL;
+ * amt_offload_providers is an RCU list so a reader outside RTNL can test
+ * for a provider without it.
+ *
+ * Teardown has one order, whichever side goes first (the lower device is
+ * unregistered, the provider unregisters, or the relay is deleted): stop
+ * admitting packets to ops->xmit and wait out in-flight callers, cancel
+ * programming work, group_disable every group, endpoint_del every
+ * endpoint, unbind, unpublish and synchronize_rcu(). The relay is then
+ * in software, exactly the path it runs with no provider at all.
+ */
+struct amt_offload_provider {
+	struct list_head		node;
+	struct net_device		*lower;
+	const struct amt_offload_ops	*ops;
+	struct rcu_head			rcu;
+};
+
+/* One provider-owned handle amt must give back on teardown. */
+struct amt_offload_ref {
+	struct list_head		node;
+	void				*handle;
+};
+
+struct amt_offload_bound {
+	struct amt_offload_binding	pub;
+	const struct amt_offload_ops	*ops;
+	struct amt_offload_caps		caps;
+	/* Cleared first on teardown; ops->xmit is only called while set. */
+	bool				admitting;
+	struct list_head		groups;		/* amt_offload_ref */
+	struct list_head		endpoints;	/* amt_offload_ref */
+};
+
+static LIST_HEAD(amt_offload_providers);
+static LIST_HEAD(amt_offload_devs);
+
+/* Relays refused at newlink for want of a provider in required mode. The
+ * refused netdev is unregistered again before newlink returns, so the
+ * count cannot live on it.
+ */
+static atomic_t amt_offload_required_refused = ATOMIC_INIT(0);
+
+static int amt_offload_required_refused_get(char *buf,
+					    const struct kernel_param *kp)
+{
+	return sysfs_emit(buf, "%u\n",
+			  (u32)atomic_read(&amt_offload_required_refused));
+}
+
+/* Read-only, but a load-time value still reaches ->set: parse_one()
+ * calls it without a NULL check (only the sysfs store path checks), so
+ * a get-only ops oopses the loader. Ignore the value rather than fail
+ * the load, so a stray modprobe option cannot keep the relay down.
+ * NOARG: a bare "amt.offload_required_refused" would otherwise be
+ * -EINVAL in parse_one() before ->set, and fail the module load.
+ */
+static int amt_offload_required_refused_set(const char *val,
+					    const struct kernel_param *kp)
+{
+	pr_warn("offload_required_refused is read-only; ignoring \"%s\"\n",
+		val ? val : "");
+	return 0;
+}
+
+static const struct kernel_param_ops amt_offload_required_refused_ops = {
+	.flags = KERNEL_PARAM_OPS_FL_NOARG,
+	.set = amt_offload_required_refused_set,
+	.get = amt_offload_required_refused_get,
+};
+module_param_cb(offload_required_refused, &amt_offload_required_refused_ops,
+		NULL, 0444);
+MODULE_PARM_DESC(offload_required_refused,
+		 "Relays refused at creation: offload=required and no usable provider");
+
+static void amt_offload_count(struct amt_dev *amt,
+			      enum amt_offload_counter cnt)
+{
+	atomic_inc(&amt->offload_cnt[cnt]);
+}
+
+static struct amt_offload_provider *
+amt_offload_provider_find(const struct net_device *lower)
+{
+	struct amt_offload_provider *p;
+
+	ASSERT_RTNL();
+	list_for_each_entry(p, &amt_offload_providers, node)
+		if (p->lower == lower)
+			return p;
+	return NULL;
+}
+
+static u32 amt_offload_outer_family(const struct amt_dev *amt)
+{
+	return ipv6_addr_any(&amt->local_ipv6) ? AMT_OFFLOAD_FAMILY_V4 :
+						 AMT_OFFLOAD_FAMILY_V6;
+}
+
+/* The data path's way in: a binding is usable only while admitting. The
+ * caller holds rcu_read_lock(). The later TX handoff is its first caller.
+ */
+static __maybe_unused struct amt_offload_bound *
+amt_offload_tx_admit(struct amt_dev *amt)
+{
+	struct amt_offload_bound *b = rcu_dereference(amt->offload);
+
+	return b && READ_ONCE(b->admitting) ? b : NULL;
+}
+
+/* Record a handle the provider handed back; the later reconciler is the
+ * production caller.
+ */
+static int __maybe_unused amt_offload_track(struct list_head *list,
+					    void *handle)
+{
+	struct amt_offload_ref *ref = kzalloc(sizeof(*ref), GFP_KERNEL);
+
+	if (!ref)
+		return -ENOMEM;
+	ref->handle = handle;
+	list_add_tail(&ref->node, list);
+	return 0;
+}
+
+static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
+{
+	if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
+		return;
+	if (on)
+		netif_carrier_on(amt->dev);
+	else
+		netif_carrier_off(amt->dev);
+}
+
+/* Bind @amt to @p. 0 means bound; an error leaves @amt in software and
+ * counted. RTNL held.
+ */
+static int amt_offload_bind(struct amt_dev *amt,
+			    struct amt_offload_provider *p, bool on_register)
+{
+	struct amt_offload_bound *b;
+	int err;
+
+	ASSERT_RTNL();
+	if (amt->mode != AMT_MODE_RELAY ||
+	    amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE ||
+	    rtnl_dereference(amt->offload))
+		return -EINVAL;
+
+	b = kzalloc_obj(*b, GFP_KERNEL);
+	if (!b)
+		return -ENOMEM;
+	b->pub.lower = p->lower;
+	b->pub.dev = amt->dev;
+	b->pub.epoch = ++amt->offload_epoch;
+	b->ops = p->ops;
+	INIT_LIST_HEAD(&b->groups);
+	INIT_LIST_HEAD(&b->endpoints);
+
+	err = b->ops->bind(&b->pub, &b->caps);
+	if (err) {
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_FAILED);
+		kfree(b);
+		return err;
+	}
+	if (!(b->caps.families & amt_offload_outer_family(amt))) {
+		b->ops->unbind(&b->pub);
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
+		kfree(b);
+		return -EOPNOTSUPP;
+	}
+
+	b->admitting = true;
+	rcu_assign_pointer(amt->offload, b);
+	amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND);
+	if (on_register)
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
+	amt_offload_set_carrier(amt, true);
+	return 0;
+}
+
+/* Return @amt to software in the documented order. @cnt is the cause to
+ * count, or AMT_OFFLOAD_CNT_MAX for the relay's own deletion. RTNL held.
+ */
+static void amt_offload_teardown(struct amt_dev *amt,
+				 enum amt_offload_counter cnt)
+{
+	struct amt_offload_bound *b = rtnl_dereference(amt->offload);
+	struct amt_offload_ref *ref, *tmp;
+
+	if (!b)
+		return;
+
+	/* 1. Stop admission and wait for every ops->xmit already admitted. */
+	WRITE_ONCE(b->admitting, false);
+	synchronize_net();
+
+	/* 2. Programming work is cancelled here once the reconciler exists
+	 *    (next patch); nothing can program the provider past this point.
+	 */
+
+	/* 3. Disable every group, then 4. delete every endpoint. */
+	list_for_each_entry_safe(ref, tmp, &b->groups, node) {
+		b->ops->group_disable(&b->pub, ref->handle);
+		list_del(&ref->node);
+		kfree(ref);
+	}
+	list_for_each_entry_safe(ref, tmp, &b->endpoints, node) {
+		b->ops->endpoint_del(&b->pub, ref->handle);
+		list_del(&ref->node);
+		kfree(ref);
+	}
+
+	/* 5. Unbind, then 6. unpublish and let RCU readers drain. */
+	b->ops->unbind(&b->pub);
+	RCU_INIT_POINTER(amt->offload, NULL);
+	synchronize_rcu();
+	kfree(b);
+
+	/* 7. Software. */
+	amt->offload_epoch++;
+	if (cnt < AMT_OFFLOAD_CNT_MAX) {
+		amt_offload_count(amt, cnt);
+		if (amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
+			amt_offload_count(amt, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+			amt_offload_set_carrier(amt, false);
+		}
+	}
+}
+
+int amt_offload_register(struct net_device *lower,
+			 const struct amt_offload_ops *ops)
+{
+	struct amt_offload_provider *p;
+	struct amt_dev *amt;
+
+	ASSERT_RTNL();
+	if (!lower || !ops || !ops->bind || !ops->unbind ||
+	    !ops->group_disable || !ops->endpoint_del)
+		return -EINVAL;
+	if (amt_offload_provider_find(lower))
+		return -EEXIST;
+
+	p = kzalloc_obj(*p, GFP_KERNEL);
+	if (!p)
+		return -ENOMEM;
+	p->lower = lower;
+	p->ops = ops;
+	list_add_rcu(&p->node, &amt_offload_providers);
+
+	list_for_each_entry(amt, &amt_offload_devs, offload_node)
+		if (amt->stream_dev == lower)
+			amt_offload_bind(amt, p, true);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(amt_offload_register);
+
+static void amt_offload_provider_drop(struct net_device *lower,
+				      enum amt_offload_counter cnt)
+{
+	struct amt_offload_provider *p = amt_offload_provider_find(lower);
+	struct amt_dev *amt;
+
+	list_for_each_entry(amt, &amt_offload_devs, offload_node)
+		if (amt->stream_dev == lower)
+			amt_offload_teardown(amt, cnt);
+	if (p) {
+		list_del_rcu(&p->node);
+		kfree_rcu(p, rcu);
+	}
+}
+
+void amt_offload_unregister(struct net_device *lower)
+{
+	ASSERT_RTNL();
+	amt_offload_provider_drop(lower, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
+}
+EXPORT_SYMBOL_GPL(amt_offload_unregister);
+
+/* NETDEV_UNREGISTER of a stream_dev: tear its relays down before
+ * amt_dellink runs, and forget its provider so no entry outlives the
+ * netdev it names.
+ */
+static void amt_offload_lower_gone(struct net_device *lower)
+{
+	amt_offload_provider_drop(lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
+}
+
+/* newlink: list the relay and bind it if a provider is registered. A
+ * required-mode relay with no usable provider is refused (-ENODEV) and
+ * counted globally. RTNL held.
+ */
+static int amt_offload_attach(struct amt_dev *amt,
+			      struct netlink_ext_ack *extack)
+{
+	struct amt_offload_provider *p;
+	int err = -ENODEV;
+
+	INIT_LIST_HEAD(&amt->offload_node);
+	if (amt->mode != AMT_MODE_RELAY)
+		return 0;
+	list_add_tail(&amt->offload_node, &amt_offload_devs);
+	if (amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE)
+		return 0;
+
+	p = amt_offload_provider_find(amt->stream_dev);
+	if (p)
+		err = amt_offload_bind(amt, p, false);
+	if (!err || amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
+		return 0;
+
+	list_del_init(&amt->offload_node);
+	atomic_inc(&amt_offload_required_refused);
+	if (p)
+		NL_SET_ERR_MSG_MOD(extack,
+				   "Offload required but the stream device's provider cannot serve this relay");
+	else
+		NL_SET_ERR_MSG_MOD(extack,
+				   "Offload required but the stream device has no offload provider");
+	return -ENODEV;
+}
+
+/* The relay itself is going away. RTNL held. */
+static void amt_offload_detach(struct amt_dev *amt)
+{
+	amt_offload_teardown(amt, AMT_OFFLOAD_CNT_MAX);
+	if (!list_empty(&amt->offload_node))
+		list_del_init(&amt->offload_node);
+}
+
+/* offload/ sysfs group: mode, binding state and the counters. Counters
+ * are monotonic since netdev creation and wrap at u32.
+ */
+static ssize_t offload_mode_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	static const char * const names[] = {
+		[AMT_OFFLOAD_MODE_SOFTWARE] = "software",
+		[AMT_OFFLOAD_MODE_AUTO] = "auto",
+		[AMT_OFFLOAD_MODE_REQUIRED] = "required",
+	};
+	struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+	return sysfs_emit(buf, "%s\n", names[READ_ONCE(amt->offload_mode)]);
+}
+static struct device_attribute dev_attr_offload_mode =
+	__ATTR(mode, 0444, offload_mode_show, NULL);
+
+static ssize_t offload_bound_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+	return sysfs_emit(buf, "%d\n", !!rcu_access_pointer(amt->offload));
+}
+static struct device_attribute dev_attr_offload_bound =
+	__ATTR(bound, 0444, offload_bound_show, NULL);
+
+static ssize_t amt_offload_cnt_show(struct device *dev, char *buf,
+				    enum amt_offload_counter cnt)
+{
+	struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+	return sysfs_emit(buf, "%u\n", (u32)atomic_read(&amt->offload_cnt[cnt]));
+}
+
+#define AMT_OFFLOAD_CNT_ATTR(name, cnt)					\
+static ssize_t offload_##name##_show(struct device *dev,		\
+				     struct device_attribute *attr,	\
+				     char *buf)				\
+{									\
+	return amt_offload_cnt_show(dev, buf, cnt);			\
+}									\
+static struct device_attribute dev_attr_offload_##name =		\
+	__ATTR(name, 0444, offload_##name##_show, NULL)
+
+AMT_OFFLOAD_CNT_ATTR(binds, AMT_OFFLOAD_CNT_BIND);
+AMT_OFFLOAD_CNT_ATTR(binds_on_register, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
+AMT_OFFLOAD_CNT_ATTR(bind_failed, AMT_OFFLOAD_CNT_BIND_FAILED);
+AMT_OFFLOAD_CNT_ATTR(family_unsupported, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
+AMT_OFFLOAD_CNT_ATTR(teardown_lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
+AMT_OFFLOAD_CNT_ATTR(teardown_provider, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
+AMT_OFFLOAD_CNT_ATTR(required_unbound, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+
+static struct attribute *amt_offload_attrs[] = {
+	&dev_attr_offload_mode.attr,
+	&dev_attr_offload_bound.attr,
+	&dev_attr_offload_binds.attr,
+	&dev_attr_offload_binds_on_register.attr,
+	&dev_attr_offload_bind_failed.attr,
+	&dev_attr_offload_family_unsupported.attr,
+	&dev_attr_offload_teardown_lower.attr,
+	&dev_attr_offload_teardown_provider.attr,
+	&dev_attr_offload_required_unbound.attr,
+	NULL,
+};
+
+static const struct attribute_group amt_offload_group = {
+	.name = "offload",
+	.attrs = amt_offload_attrs,
+};
+
 static const struct nla_policy amt_policy[IFLA_AMT_MAX + 1] = {
 	[IFLA_AMT_MODE]		= { .type = NLA_U32 },
 	[IFLA_AMT_RELAY_PORT]	= { .type = NLA_U16 },
@@ -4284,6 +4695,13 @@ static int amt_newlink(struct net_device *dev,
 	INIT_DELAYED_WORK(&amt->secret_wq, amt_secret_work);
 	INIT_WORK(&amt->event_wq, amt_event_work);
 	INIT_LIST_HEAD(&amt->tunnel_list);
+
+	err = amt_offload_attach(amt, extack);
+	if (err) {
+		netdev_upper_dev_unlink(amt->stream_dev, dev);
+		unregister_netdevice(dev);
+		goto err;
+	}
 	return 0;
 err:
 	dev_put(amt->stream_dev);
@@ -4294,6 +4712,7 @@ static void amt_dellink(struct net_device *dev, struct list_head *head)
 {
 	struct amt_dev *amt = netdev_priv(dev);
 
+	amt_offload_detach(amt);
 	unregister_netdevice_queue(dev, head);
 	netdev_upper_dev_unlink(amt->stream_dev, dev);
 	dev_put(amt->stream_dev);
@@ -4438,6 +4857,12 @@ static int amt_device_event(struct notifier_block *unused,
 	LIST_HEAD(list);
 	int new_mtu;
 
+	/* Before amt_dellink below: the provider's tables are torn down
+	 * while the relay and its stream_dev are both still intact.
+	 */
+	if (event == NETDEV_UNREGISTER)
+		amt_offload_lower_gone(dev);
+
 	upper_dev = amt_lookup_upper_dev(dev);
 	if (!upper_dev)
 		return NOTIFY_DONE;
diff --git a/include/net/amt.h b/include/net/amt.h
index 5b3405f04..7d70f30e6 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -379,6 +379,29 @@ struct amt_events {
 	struct sk_buff *skb;
 };
 
+struct amt_offload_bound;
+
+/* Per-relay offload counters, exported under offload/ in sysfs. */
+enum amt_offload_counter {
+	/* bind() succeeded and the caps covered the outer family */
+	AMT_OFFLOAD_CNT_BIND,
+	/* ... of which: bound when a provider registered on the stream_dev
+	 * this relay already uses
+	 */
+	AMT_OFFLOAD_CNT_BIND_ON_REGISTER,
+	/* bind() returned an error; the relay stayed unbound */
+	AMT_OFFLOAD_CNT_BIND_FAILED,
+	/* caps lacked the relay's outer family; the relay stayed unbound */
+	AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED,
+	/* teardown because the stream_dev was unregistered */
+	AMT_OFFLOAD_CNT_TEARDOWN_LOWER,
+	/* teardown because the provider unregistered */
+	AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER,
+	/* a required-mode relay lost its provider and dropped carrier */
+	AMT_OFFLOAD_CNT_REQUIRED_UNBOUND,
+	AMT_OFFLOAD_CNT_MAX,
+};
+
 struct amt_dev {
 	struct net_device       *dev;
 	struct net_device       *stream_dev;
@@ -484,6 +507,17 @@ struct amt_dev {
 	struct socket		*stream_sock_v4;
 	struct socket		*stream_sock_v6;
 	atomic_t		upstream_setsockopt_slow_count;
+
+	/* Replication offload (relay mode). Everything here is written
+	 * under RTNL; offload is also read by the data path under RCU.
+	 * offload_node links the relay into amt_offload_devs so a provider
+	 * registering on its stream_dev can find it.
+	 */
+	u32			offload_mode;
+	struct amt_offload_bound __rcu *offload;
+	struct list_head	offload_node;
+	u64			offload_epoch;
+	atomic_t		offload_cnt[AMT_OFFLOAD_CNT_MAX];
 };
 
 #define AMT_TOS			0xc0
diff --git a/include/uapi/linux/amt.h b/include/uapi/linux/amt.h
index c04ed1fa4..de1ae1929 100644
--- a/include/uapi/linux/amt.h
+++ b/include/uapi/linux/amt.h
@@ -21,6 +21,26 @@ enum ifla_amt_mode {
 
 #define AMT_MODE_MAX (__AMT_MODE_MAX - 1)
 
+/* Relay replication offload (include/net/amt_offload.h). */
+enum ifla_amt_offload_mode {
+	/* Never use a provider. The default. */
+	AMT_OFFLOAD_MODE_SOFTWARE = 0,
+	/* Use the provider registered on the stream device when its bind
+	 * succeeds, it covers the relay's outer family and it implements the
+	 * TX path; software otherwise.
+	 */
+	AMT_OFFLOAD_MODE_AUTO,
+	/* Refuse to create the relay without a usable provider, and drop
+	 * carrier while none is bound. Unlike auto, also bind a provider
+	 * that replicates only on ingress. A bound relay still forwards in
+	 * software, with carrier up, what the provider does not serve.
+	 */
+	AMT_OFFLOAD_MODE_REQUIRED,
+	__AMT_OFFLOAD_MODE_MAX,
+};
+
+#define AMT_OFFLOAD_MODE_MAX (__AMT_OFFLOAD_MODE_MAX - 1)
+
 enum {
 	IFLA_AMT_UNSPEC,
 	/* This attribute specify mode etier Gateway or Relay. */
-- 
2.43.0

[RFC PATCH net-next 3/9] amt: offload group object, desired state and reconciler

[Omar Ramadan]

One group object per (family, group) carries its members (the gateway
endpoints joined to it), a generation and hw_active. Desired state
changes under a leaf bh spinlock: the first gnode of a group on a
tunnel adds the tunnel's endpoint as a member, the last removes it,
and a tunnel's unlink removes it from every group at once.
Forwarding stays per (family, group) and never per source, exactly as
the software fan-out: a gateway holding INCLUDE(S1) still receives S2.

A reconciler in process context, never under amt->lock or a tunnel lock,
converges the provider to desired state: endpoint_add for new endpoints,
group_update and group_commit at the group's generation, then hw_active;
group_disable for emptied groups and endpoint_del for unused endpoints
once no committed group can name them. Each endpoint counts the
committed groups naming it, so its deletion does not wait for every
dirty group to be reprogrammed. A pass starts no new group once it has
spent offload_op_budget ops (default 1000, an estimate), so it can
overrun the budget by one group's ops; if the budget left no room for
one, it still makes one endpoint_del, so a group re-dirtied on every
pass cannot starve endpoint collection. The pass that gives everything
back once desired state is lost is not budgeted. Changes between passes
coalesce into one update per group. A group the provider refuses is
served by software and retried with hysteresis (once the provider holds
an eighth fewer objects), after a reset or on a new binding; its
endpoints stay programmed. An op overrunning op_timeout_ms flushes the
provider, bumps the epoch and replays from desired state; a result from
an older epoch is ignored. A pass ends at a reset and at a teardown, and
checks for teardown before every endpoint_add, group_update,
group_commit and endpoint_del. A teardown under RTNL therefore waits out
at most one of those ops plus the cleanup it triggers or the pass has
already queued: a flush after an overrun, the group_disable that parks a
group it could not program, the group_disable calls for groups the pass
emptied, and, on the pass that gives everything back once desired state
is lost, the rest of that release. The group_disable and endpoint_del
calls that give everything back are timed too: the first to overrun
flushes the provider once and the remaining handles are forgotten, so a
wedged provider costs one op plus one flush, with no replay and no new
epoch (at teardown, unbind follows). The overrun is sampled as the
provider call returns, so amt's own bookkeeping is not charged to it.

The endpoint and group tables are sized once from max_tunnels and
max_tunnels * max_groups respectively, and hashed with the device's
hash_seed: gateways choose their groups and ports. Removing a member
withdraws its group from hardware at once (hw_active), so no packet
admitted after that is replicated to the departed endpoint; software
serves the group until the reconciler commits the new member set, and
hw_active is published only if no member was removed since the snapshot
the commit was built from. A join meanwhile does not withhold it: the
committed set is a subset that names no departed endpoint. A group
re-created for an emptied group's key waits, within the pass, for the
old provider object's group_disable, so a provider never holds two
groups for one key. Parking a group that holds a provider object always
waits out ops->xmit, since removing a member clears hw_active with no
grace period. The table size is clamped in 64 bits before
order_base_2(), which would truncate the product on 32-bit.

Binding replays desired state; teardown takes the reconciler mutex to
cancel programming, then gives every group and endpoint back in the
documented order. Desired state is freed at ndo_uninit, after the
tunnels. amt_tunnel_unlink() now carries the unlink that
amt_tunnel_expire() and amt_dev_stop() each open-coded, so the hook
runs on both. Software mode allocates nothing and runs no hook.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/amt.c         | 1166 +++++++++++++++++++++++++++++++++++--
 include/net/amt.h         |   35 +-
 include/net/amt_offload.h |   74 ++-
 3 files changed, 1225 insertions(+), 50 deletions(-)

diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 034221e1c..17337989a 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -7,6 +7,7 @@
 #include <linux/skbuff.h>
 #include <linux/udp.h>
 #include <linux/jhash.h>
+#include <linux/hashtable.h>
 #include <linux/if_tunnel.h>
 #include <linux/net.h>
 #include <linux/igmp.h>
@@ -33,6 +34,8 @@
 #include <net/inet_dscp.h>
 #include <net/ip6_checksum.h>
 
+#include <kunit/static_stub.h>
+
 static struct workqueue_struct *amt_wq;
 
 static HLIST_HEAD(source_gc_list);
@@ -304,6 +307,19 @@ static void amt_destroy_source(struct amt_source_node *snode)
 	spin_unlock_bh(&source_gc_lock);
 }
 
+/* Offload desired-state hooks, defined with the rest of the offload code.
+ * Each is a no-op on a relay in software mode.
+ */
+static void amt_ofl_member_add(struct amt_dev *amt,
+			       struct amt_tunnel_list *tunnel,
+			       const union amt_addr *group, bool v6);
+static void amt_ofl_member_del(struct amt_dev *amt,
+			       struct amt_tunnel_list *tunnel,
+			       const union amt_addr *group, bool v6);
+static void amt_ofl_tunnel_gone(struct amt_dev *amt,
+				struct amt_tunnel_list *tunnel);
+static void amt_ofl_state_free(struct amt_dev *amt);
+
 static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
 {
 	struct amt_source_node *snode;
@@ -315,6 +331,8 @@ static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
 	rhltable_remove(&amt->groups_rhl, &gnode->rhlnode, amt_gnode_params);
 	list_del(&gnode->tunnel_node);
 	gnode->tunnel_list->nr_groups--;
+	amt_ofl_member_del(amt, gnode->tunnel_list, &gnode->group_addr,
+			   gnode->v6);
 
 	if (!gnode->v6)
 		netdev_dbg(amt->dev, "Leave group %pI4\n",
@@ -569,6 +587,7 @@ static struct amt_group_node *amt_add_group(struct amt_dev *amt,
 	}
 	list_add(&gnode->tunnel_node, &tunnel->groups);
 	tunnel->nr_groups++;
+	amt_ofl_member_add(amt, tunnel, group, v6);
 
 	if (!gnode->v6)
 		netdev_dbg(amt->dev, "Join group %pI4\n",
@@ -1542,6 +1561,17 @@ static void amt_clear_groups(struct amt_tunnel_list *tunnel)
 	spin_unlock_bh(&tunnel->lock);
 }
 
+/* Unlink @tunnel from the relay. Caller holds amt->lock, or RTNL with the
+ * device stopping (amt_dev_stop). The groups stay until amt_clear_groups.
+ */
+static void amt_tunnel_unlink(struct amt_dev *amt,
+			      struct amt_tunnel_list *tunnel)
+{
+	list_del_rcu(&tunnel->list);
+	amt->nr_tunnels--;
+	amt_ofl_tunnel_gone(amt, tunnel);
+}
+
 static void amt_tunnel_expire(struct work_struct *work)
 {
 	struct amt_tunnel_list *tunnel = container_of(to_delayed_work(work),
@@ -1551,8 +1581,7 @@ static void amt_tunnel_expire(struct work_struct *work)
 
 	spin_lock_bh(&amt->lock);
 	rcu_read_lock();
-	list_del_rcu(&tunnel->list);
-	amt->nr_tunnels--;
+	amt_tunnel_unlink(amt, tunnel);
 	amt_clear_groups(tunnel);
 	rcu_read_unlock();
 	spin_unlock_bh(&amt->lock);
@@ -3445,8 +3474,7 @@ static int amt_dev_stop(struct net_device *dev)
 	amt->remote_ip = 0;
 
 	list_for_each_entry_safe(tunnel, tmp, &amt->tunnel_list, list) {
-		list_del_rcu(&tunnel->list);
-		amt->nr_tunnels--;
+		amt_tunnel_unlink(amt, tunnel);
 		cancel_delayed_work_sync(&tunnel->gc_wq);
 		amt_clear_groups(tunnel);
 		kfree_rcu(tunnel, rcu);
@@ -3996,6 +4024,7 @@ static void amt_dev_uninit(struct net_device *dev)
 	/* amt_dev_stop() has removed every tunnel and its groups. */
 	rhltable_destroy(&amt->groups_rhl);
 	gro_cells_destroy(&amt->gro_cells);
+	amt_ofl_state_free(amt);
 }
 
 static const struct net_device_ops amt_netdev_ops = {
@@ -4054,20 +4083,141 @@ struct amt_offload_provider {
 	struct rcu_head			rcu;
 };
 
-/* One provider-owned handle amt must give back on teardown. */
-struct amt_offload_ref {
-	struct list_head		node;
-	void				*handle;
-};
-
 struct amt_offload_bound {
 	struct amt_offload_binding	pub;
 	const struct amt_offload_ops	*ops;
 	struct amt_offload_caps		caps;
 	/* Cleared first on teardown; ops->xmit is only called while set. */
 	bool				admitting;
-	struct list_head		groups;		/* amt_offload_ref */
-	struct list_head		endpoints;	/* amt_offload_ref */
+};
+
+/* Desired state and its reconciler.
+ *
+ *   amt_add_group / amt_del_group      tunnel unlink
+ *   (first / last gnode of a group)    (every group of the tunnel)
+ *                 |                              |
+ *                 v                              v
+ *   +-- desired state, st->lock (bh, a leaf) -------------------------+
+ *   |  grp (family, group) --< member (grp, tunnel) >-- ep (addr,port) |
+ *   |  a member change bumps grp->gen and queues grp on st->dirty      |
+ *   +------------------------------------------------------------------+
+ *                 |  st->work, process context
+ *                 v
+ *   reconciler, st->mutex, never under amt->lock or a tunnel lock:
+ *     per dirty grp: endpoint_add each new ep, group_update(gen),
+ *     group_commit(gen), then publish hw_gen and hw_active; a grp
+ *     with no members is disabled; then unused eps get endpoint_del
+ *                 |
+ *                 v
+ *   provider tables; amt_dev_xmit reads grp->hw_active and hw_gen
+ *
+ * A tick starts no new group once st->budget ops are spent, so one
+ * group may overrun it by its own ops, and endpoint GC runs one
+ * endpoint_del past it when the budget left none, so a group re-dirtied
+ * on every tick cannot starve collection. Changes that land between ticks
+ * coalesce: a group is queued at most once and programmed at its latest
+ * generation. A group whose programming fails is parked and served by
+ * software (the remainder); parked groups are retried only when the
+ * provider holds an eighth fewer objects than when it refused one
+ * (hysteresis), after a reset, or on a new binding. A programming, gone
+ * or GC op overrunning op_timeout_ms resets the provider (flush) and
+ * replays every group from desired state under a new epoch; a result
+ * from an older epoch is ignored. A release op that overruns while amt
+ * gives everything back (teardown, or desired state lost) flushes once
+ * and forgets the remaining handles, with no new epoch and no replay.
+ * Only the reconciler frees grp and ep records, so it may use them
+ * without st->lock between ops.
+ *
+ * Forwarding is per (family, group) and never per source, exactly as the
+ * software fan-out in amt_dev_xmit: a gateway with an INCLUDE(S1)
+ * membership still receives S2. AMT_OFFLOAD_F_SOURCE_FILTER has no v1
+ * meaning.
+ */
+/* Both tables are sized once, from the caps, at state allocation: eps
+ * from max_tunnels, grps from max_tunnels * max_groups. Neither cap has a
+ * changelink, so a load factor of about one holds without a resize; the
+ * upper bound only matters past 64k entries.
+ */
+#define AMT_OFL_HASH_MIN_BITS	4
+#define AMT_OFL_HASH_MAX_BITS	16
+
+/* Default ops per reconciler tick. [estimated]: not measured on hardware. */
+static unsigned int amt_offload_op_budget = 1000;
+module_param_named(offload_op_budget, amt_offload_op_budget, uint, 0444);
+MODULE_PARM_DESC(offload_op_budget,
+		 "Offload programming ops per reconciler tick (default 1000)");
+
+enum amt_ofl_grp_state {
+	AMT_OFL_IDLE,
+	AMT_OFL_DIRTY,		/* on st->dirty */
+	AMT_OFL_PARKED,		/* on st->parked: served by software */
+};
+
+struct amt_ofl_ep {
+	struct hlist_node		hnode;		/* st->eps */
+	struct list_head		gc_node;	/* st->ep_gc */
+	struct amt_offload_endpoint_key	key;
+	u32				refs;		/* members using it */
+	u32				committed;	/* committed grps naming it */
+	bool				on_gc;
+	/* Reconciler-owned, under st->mutex. */
+	struct list_head		hw_node;	/* st->hw_eps */
+	struct amt_offload_endpoint	*hw;
+};
+
+struct amt_ofl_grp {
+	struct hlist_node		hnode;		/* st->grps, RCU */
+	struct list_head		q_node;	/* st->dirty or st->parked */
+	struct amt_offload_group_key	key;
+	struct list_head		members;
+	u32				nr_members;
+	u8				state;
+	u64				gen;		/* desired */
+	u32				left;		/* members removed */
+	/* Reconciler-owned, under st->mutex; the data path reads hw and
+	 * hw_gen once it has read hw_active true (acquire).
+	 */
+	struct list_head		hw_node;	/* st->hw_grps */
+	struct amt_offload_group	*hw;
+	/* The endpoints the last successful group_commit named; each counts
+	 * in its ep->committed, under st->lock.
+	 */
+	struct amt_ofl_ep		**ceps;
+	u32				nr_ceps;
+	u64				hw_gen;
+	bool				hw_active;
+	struct rcu_head			rcu;
+};
+
+struct amt_ofl_member {
+	struct list_head		grp_node;	/* grp->members */
+	struct hlist_node		tun_node;	/* tunnel->offload_members */
+	struct amt_ofl_grp		*grp;
+	struct amt_ofl_ep		*ep;
+	u32				gnodes;	/* the tunnel's gnodes for grp */
+};
+
+struct amt_offload_state {
+	struct amt_dev			*amt;
+	/* Desired state, written from the softirq decap path. */
+	spinlock_t			lock;
+	struct hlist_head		*grps;		/* RCU */
+	struct hlist_head		*eps;
+	u32				grp_bits;
+	u32				ep_bits;
+	struct list_head		dirty;
+	struct list_head		parked;
+	struct list_head		ep_gc;
+	bool				desync;
+	/* Reconciler. bind and teardown also take the mutex. */
+	struct mutex			mutex;
+	struct work_struct		work;
+	struct list_head		hw_grps;
+	struct list_head		hw_eps;
+	u32				hw_objs;
+	bool				enospc;
+	u32				enospc_at;
+	u32				budget;
 };
 
 static LIST_HEAD(amt_offload_providers);
@@ -4146,21 +4296,948 @@ amt_offload_tx_admit(struct amt_dev *amt)
 	return b && READ_ONCE(b->admitting) ? b : NULL;
 }
 
-/* Record a handle the provider handed back; the later reconciler is the
- * production caller.
+static void amt_ofl_grp_key(struct amt_offload_group_key *key,
+			    const union amt_addr *group, bool v6)
+{
+	memset(key, 0, sizeof(*key));
+#if IS_ENABLED(CONFIG_IPV6)
+	if (v6) {
+		key->family = AF_INET6;
+		key->group.ip6 = group->ip6;
+		return;
+	}
+#endif
+	key->family = AF_INET;
+	key->group.ip4 = group->ip4;
+}
+
+static void amt_ofl_ep_key(struct amt_offload_endpoint_key *key,
+			   const struct amt_dev *amt,
+			   const struct amt_tunnel_list *tunnel)
+{
+	memset(key, 0, sizeof(*key));
+	key->port = tunnel->source_port;
+#if IS_ENABLED(CONFIG_IPV6)
+	if (amt_v6(amt)) {
+		key->family = AF_INET6;
+		key->addr.ip6 = tunnel->addr.ip6;
+		return;
+	}
+#endif
+	key->family = AF_INET;
+	key->addr.ip4 = tunnel->addr.ip4;
+}
+
+/* Keys are built zeroed, so memcmp() over the padding is sound. Seeded
+ * per device: gateways choose their groups and ports, so an unseeded hash
+ * lets them pile entries into one chain under st->lock.
+ */
+static u32 amt_ofl_hash(const struct amt_offload_state *st,
+			const void *key, size_t len, u32 bits)
+{
+	return hash_32(jhash(key, len, st->amt->hash_seed), bits);
+}
+
+static struct hlist_head *
+amt_ofl_grp_bucket(const struct amt_offload_state *st,
+		   const struct amt_offload_group_key *key)
+{
+	return &st->grps[amt_ofl_hash(st, key, sizeof(*key), st->grp_bits)];
+}
+
+static struct hlist_head *
+amt_ofl_ep_bucket(const struct amt_offload_state *st,
+		  const struct amt_offload_endpoint_key *key)
+{
+	return &st->eps[amt_ofl_hash(st, key, sizeof(*key), st->ep_bits)];
+}
+
+/* Caller holds st->lock, or rcu_read_lock() for the data path. */
+static struct amt_ofl_grp *
+amt_ofl_grp_find(struct amt_offload_state *st,
+		 const struct amt_offload_group_key *key)
+{
+	struct amt_ofl_grp *g;
+
+	hlist_for_each_entry_rcu(g, amt_ofl_grp_bucket(st, key), hnode,
+				 lockdep_is_held(&st->lock))
+		if (!memcmp(&g->key, key, sizeof(*key)))
+			return g;
+	return NULL;
+}
+
+static struct amt_ofl_ep *
+amt_ofl_ep_find(struct amt_offload_state *st,
+		const struct amt_offload_endpoint_key *key)
+{
+	struct amt_ofl_ep *ep;
+
+	hlist_for_each_entry(ep, amt_ofl_ep_bucket(st, key), hnode)
+		if (!memcmp(&ep->key, key, sizeof(*key)))
+			return ep;
+	return NULL;
+}
+
+/* Queue @g for the reconciler. A parked group stays parked, served by
+ * software, until its retry is due -- unless it lost its last member,
+ * which the reconciler must see to free it. Caller holds st->lock.
+ */
+static void amt_ofl_grp_dirty(struct amt_offload_state *st,
+			      struct amt_ofl_grp *g)
+{
+	if (g->state == AMT_OFL_DIRTY)
+		return;
+	if (g->state == AMT_OFL_PARKED) {
+		if (g->nr_members)
+			return;
+		list_del(&g->q_node);
+	}
+	list_add_tail(&g->q_node, &st->dirty);
+	g->state = AMT_OFL_DIRTY;
+}
+
+static void amt_ofl_kick(struct amt_offload_state *st)
+{
+	queue_work(system_unbound_wq, &st->work);
+}
+
+/* Desired state could not record a change. Hardware would forward to a
+ * stale member set, so the relay leaves offload for good: the data path
+ * stops consulting the provider, and the next reconciler pass gives every
+ * provider object back (group_disable, endpoint_del; the binding stays,
+ * with no unbind). Desync does not schedule that pass: a pass already
+ * queued or a later removal of a recorded member (by a leave, a timer or
+ * a tunnel removal) does, and a teardown gives everything back itself
+ * and unbinds. Caller holds st->lock.
+ */
+static void amt_ofl_desync(struct amt_offload_state *st)
+{
+	if (st->desync)
+		return;
+	WRITE_ONCE(st->desync, true);
+	amt_offload_count(st->amt, AMT_OFFLOAD_CNT_DESYNC);
+}
+
+/* The first gnode of (@group, @v6) on @tunnel makes the tunnel's
+ * endpoint a member; later ones only count. Caller holds tunnel->lock.
+ */
+static void amt_ofl_member_add(struct amt_dev *amt,
+			       struct amt_tunnel_list *tunnel,
+			       const union amt_addr *group, bool v6)
+{
+	struct amt_offload_state *st = amt->offload_st;
+	struct amt_ofl_grp *g, *new_g = NULL;
+	struct amt_ofl_ep *ep, *new_ep = NULL;
+	struct amt_offload_endpoint_key ekey;
+	struct amt_offload_group_key gkey;
+	struct amt_ofl_member *m;
+
+	if (!st)
+		return;
+	amt_ofl_grp_key(&gkey, group, v6);
+	spin_lock_bh(&st->lock);
+	if (st->desync)
+		goto out;
+	hlist_for_each_entry(m, &tunnel->offload_members, tun_node) {
+		if (!memcmp(&m->grp->key, &gkey, sizeof(gkey))) {
+			m->gnodes++;
+			goto out;
+		}
+	}
+
+	g = amt_ofl_grp_find(st, &gkey);
+	if (!g) {
+		new_g = kzalloc_obj(*new_g, GFP_ATOMIC);
+		if (!new_g)
+			goto nomem;
+		g = new_g;
+		g->key = gkey;
+		INIT_LIST_HEAD(&g->members);
+		INIT_LIST_HEAD(&g->hw_node);
+	}
+	amt_ofl_ep_key(&ekey, amt, tunnel);
+	ep = amt_ofl_ep_find(st, &ekey);
+	if (!ep) {
+		new_ep = kzalloc_obj(*new_ep, GFP_ATOMIC);
+		if (!new_ep)
+			goto nomem;
+		ep = new_ep;
+		ep->key = ekey;
+		INIT_LIST_HEAD(&ep->hw_node);
+	}
+	m = kzalloc_obj(*m, GFP_ATOMIC);
+	if (!m)
+		goto nomem;
+
+	if (new_g)
+		hlist_add_head_rcu(&g->hnode, amt_ofl_grp_bucket(st, &gkey));
+	if (new_ep)
+		hlist_add_head(&ep->hnode, amt_ofl_ep_bucket(st, &ekey));
+	m->grp = g;
+	m->ep = ep;
+	m->gnodes = 1;
+	ep->refs++;
+	list_add_tail(&m->grp_node, &g->members);
+	hlist_add_head(&m->tun_node, &tunnel->offload_members);
+	g->nr_members++;
+	g->gen++;
+	amt_ofl_grp_dirty(st, g);
+	spin_unlock_bh(&st->lock);
+	amt_ofl_kick(st);
+	return;
+nomem:
+	/* One lost allocation takes the relay out of offload for good (see
+	 * amt_ofl_desync()). Rebuilding desired state from the tunnel walk
+	 * would avoid that, if GFP_ATOMIC failures ever show up here.
+	 */
+	kfree(new_g);
+	kfree(new_ep);
+	amt_ofl_desync(st);
+out:
+	spin_unlock_bh(&st->lock);
+}
+
+/* The reconciler's member arrays: up to max_tunnels entries, so not one
+ * contiguous block.
+ */
+static void *amt_ofl_alloc_array(u32 n, size_t size)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+	KUNIT_STATIC_STUB_REDIRECT(amt_ofl_alloc_array, n, size);
+#endif
+	return kvmalloc_array(n, size, GFP_KERNEL);
+}
+
+static void amt_ofl_free_array(void *p)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+	KUNIT_STATIC_STUB_REDIRECT(amt_ofl_free_array, p);
+#endif
+	kvfree(p);
+}
+
+/* @ep may be deleted once no member uses it and no committed group names
+ * it: queue it for endpoint GC. Caller holds st->lock.
+ */
+static void amt_ofl_ep_unused(struct amt_offload_state *st,
+			      struct amt_ofl_ep *ep)
+{
+	if (ep->refs || ep->committed || ep->on_gc)
+		return;
+	list_add_tail(&ep->gc_node, &st->ep_gc);
+	ep->on_gc = true;
+}
+
+/* @g's committed member set no longer reaches hardware: its provider
+ * object is disabled or flushed. Caller holds st->mutex.
+ */
+static void amt_ofl_grp_uncommit(struct amt_offload_state *st,
+				 struct amt_ofl_grp *g)
+{
+	struct amt_ofl_ep **ceps = g->ceps;
+	u32 i;
+
+	if (!ceps)
+		return;
+	spin_lock_bh(&st->lock);
+	for (i = 0; i < g->nr_ceps; i++) {
+		ceps[i]->committed--;
+		amt_ofl_ep_unused(st, ceps[i]);
+	}
+	g->ceps = NULL;
+	g->nr_ceps = 0;
+	spin_unlock_bh(&st->lock);
+	amt_ofl_free_array(ceps);
+}
+
+/* Caller holds st->lock. */
+static void amt_ofl_member_unlink(struct amt_offload_state *st,
+				  struct amt_ofl_member *m)
+{
+	struct amt_ofl_grp *g = m->grp;
+	struct amt_ofl_ep *ep = m->ep;
+
+	hlist_del(&m->tun_node);
+	list_del(&m->grp_node);
+	g->nr_members--;
+	g->gen++;
+	g->left++;
+	/* The committed member set still names this endpoint: stop handing
+	 * the group to hardware now, not when the reconciler recommits, so
+	 * no packet admitted after this is replicated to it. Software serves
+	 * the group meanwhile.
+	 */
+	WRITE_ONCE(g->hw_active, false);
+	amt_ofl_grp_dirty(st, g);
+	ep->refs--;
+	amt_ofl_ep_unused(st, ep);
+	kfree(m);
+}
+
+/* The last gnode of (@group, @v6) on @tunnel is gone: the endpoint
+ * leaves the group. Caller holds tunnel->lock.
+ */
+static void amt_ofl_member_del(struct amt_dev *amt,
+			       struct amt_tunnel_list *tunnel,
+			       const union amt_addr *group, bool v6)
+{
+	struct amt_offload_state *st = amt->offload_st;
+	struct amt_offload_group_key gkey;
+	struct amt_ofl_member *m;
+	bool kick = false;
+
+	if (!st)
+		return;
+	amt_ofl_grp_key(&gkey, group, v6);
+	spin_lock_bh(&st->lock);
+	hlist_for_each_entry(m, &tunnel->offload_members, tun_node) {
+		if (memcmp(&m->grp->key, &gkey, sizeof(gkey)))
+			continue;
+		if (!--m->gnodes) {
+			amt_ofl_member_unlink(st, m);
+			kick = true;
+		}
+		break;
+	}
+	spin_unlock_bh(&st->lock);
+	if (kick)
+		amt_ofl_kick(st);
+}
+
+/* @tunnel is torn down or unlinked: its endpoint leaves every group at
+ * once. The gnodes deleted afterwards find no member and do nothing.
+ * Caller holds amt->lock, or RTNL with the device stopping (amt_dev_stop).
+ */
+static void amt_ofl_tunnel_gone(struct amt_dev *amt,
+				struct amt_tunnel_list *tunnel)
+{
+	struct amt_offload_state *st = amt->offload_st;
+	struct amt_ofl_member *m;
+	struct hlist_node *tmp;
+	bool kick;
+
+	if (!st)
+		return;
+	spin_lock_bh(&st->lock);
+	kick = !hlist_empty(&tunnel->offload_members);
+	hlist_for_each_entry_safe(m, tmp, &tunnel->offload_members, tun_node)
+		amt_ofl_member_unlink(st, m);
+	spin_unlock_bh(&st->lock);
+	if (kick)
+		amt_ofl_kick(st);
+}
+
+/* Queue every group with members, parked ones included, for a full
+ * replay: on a new binding and after a reset.
+ */
+static void amt_ofl_requeue_all(struct amt_offload_state *st)
+{
+	struct amt_ofl_grp *g;
+	u32 bkt;
+
+	spin_lock_bh(&st->lock);
+	for (bkt = 0; bkt < 1U << st->grp_bits; bkt++) {
+		hlist_for_each_entry(g, &st->grps[bkt], hnode) {
+			if (g->state == AMT_OFL_PARKED) {
+				list_del(&g->q_node);
+				g->state = AMT_OFL_IDLE;
+			}
+			amt_ofl_grp_dirty(st, g);
+		}
+	}
+	st->enospc = false;
+	spin_unlock_bh(&st->lock);
+}
+
+/* Wait out every ops->xmit admitted before the caller cleared hw_active or
+ * admitting.
+ */
+static void amt_ofl_sync_net(void)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+	KUNIT_STATIC_STUB_REDIRECT(amt_ofl_sync_net);
+#endif
+	synchronize_net();
+}
+
+/* Forget every provider handle; the provider was flushed. Each group is
+ * unpublished too, so a later binding never sees hw_active over a NULL
+ * handle. Caller holds st->mutex.
+ */
+static void amt_ofl_hw_forget(struct amt_offload_state *st)
+{
+	struct amt_ofl_grp *g, *gt;
+	struct amt_ofl_ep *ep, *et;
+
+	list_for_each_entry_safe(g, gt, &st->hw_grps, hw_node) {
+		WRITE_ONCE(g->hw_active, false);
+		amt_ofl_grp_uncommit(st, g);
+		g->hw = NULL;
+		list_del_init(&g->hw_node);
+	}
+	list_for_each_entry_safe(ep, et, &st->hw_eps, hw_node) {
+		ep->hw = NULL;
+		list_del_init(&ep->hw_node);
+	}
+	st->hw_objs = 0;
+}
+
+/* An op issued at @t0 overran op_timeout_ms: the provider is wedged. */
+static bool amt_ofl_op_overran(struct amt_offload_bound *b, ktime_t t0)
+{
+	u32 tmo = b->caps.op_timeout_ms;
+
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+	KUNIT_STATIC_STUB_REDIRECT(amt_ofl_op_overran, b, t0);
+#endif
+	return tmo && ktime_ms_delta(ktime_get(), t0) > tmo;
+}
+
+/* Give every provider object back with group_disable and endpoint_del,
+ * groups first. A release op that overruns op_timeout_ms ends it: the
+ * provider is flushed once and the remaining handles are forgotten, so
+ * a wedged provider costs one op plus one flush, not one op per object,
+ * under RTNL at teardown. @quiesced: the caller already stopped
+ * admission and waited out ops->xmit. Caller holds st->mutex.
  */
-static int __maybe_unused amt_offload_track(struct list_head *list,
-					    void *handle)
+static void amt_ofl_hw_release(struct amt_offload_state *st,
+			       struct amt_offload_bound *b, bool quiesced)
 {
-	struct amt_offload_ref *ref = kzalloc(sizeof(*ref), GFP_KERNEL);
+	struct amt_ofl_grp *g, *gt;
+	struct amt_ofl_ep *ep, *et;
+	bool overran;
+	ktime_t t0;
+
+	lockdep_assert_held(&st->mutex);
+	if (!quiesced && !list_empty(&st->hw_grps)) {
+		list_for_each_entry(g, &st->hw_grps, hw_node)
+			WRITE_ONCE(g->hw_active, false);
+		amt_ofl_sync_net();
+	}
+	list_for_each_entry_safe(g, gt, &st->hw_grps, hw_node) {
+		WRITE_ONCE(g->hw_active, false);
+		t0 = ktime_get();
+		b->ops->group_disable(&b->pub, g->hw);
+		overran = amt_ofl_op_overran(b, t0);
+		amt_ofl_grp_uncommit(st, g);
+		g->hw = NULL;
+		list_del_init(&g->hw_node);
+		if (overran)
+			goto wedged;
+	}
+	list_for_each_entry_safe(ep, et, &st->hw_eps, hw_node) {
+		t0 = ktime_get();
+		b->ops->endpoint_del(&b->pub, ep->hw);
+		overran = amt_ofl_op_overran(b, t0);
+		ep->hw = NULL;
+		list_del_init(&ep->hw_node);
+		if (overran)
+			goto wedged;
+	}
+	st->hw_objs = 0;
+	return;
+wedged:
+	b->ops->flush(&b->pub);
+	amt_ofl_hw_forget(st);
+	amt_offload_count(st->amt, AMT_OFFLOAD_CNT_RESET);
+}
+
+/* An op overran op_timeout_ms: the provider is treated as wedged. Stop
+ * forwarding through it, flush its tables, forget every handle, bump the
+ * epoch so anything it still returns is stale, and replay. Caller holds
+ * st->mutex.
+ */
+static void amt_ofl_reset(struct amt_offload_state *st,
+			  struct amt_offload_bound *b)
+{
+	struct amt_ofl_grp *g;
+
+	lockdep_assert_held(&st->mutex);
+	list_for_each_entry(g, &st->hw_grps, hw_node)
+		WRITE_ONCE(g->hw_active, false);
+	amt_ofl_sync_net();
+	b->ops->flush(&b->pub);
+	amt_ofl_hw_forget(st);
+	b->pub.epoch = ++st->amt->offload_epoch;
+	amt_offload_count(st->amt, AMT_OFFLOAD_CNT_RESET);
+	amt_ofl_requeue_all(st);
+}
+
+/* Completion check for an op issued under @epoch; @overran was sampled
+ * right after the provider call returned, so amt's own bookkeeping is not
+ * charged to the provider. False means its result must be ignored -- the
+ * op overran op_timeout_ms (the provider is reset here) or the epoch moved
+ * under it.
+ */
+static bool amt_ofl_op_done(struct amt_offload_state *st,
+			    struct amt_offload_bound *b, u64 epoch, bool overran)
+{
+	if (overran)
+		amt_ofl_reset(st, b);
+	if (b->pub.epoch == epoch)
+		return true;
+	amt_offload_count(st->amt, AMT_OFFLOAD_CNT_STALE);
+	return false;
+}
+
+/* Stop forwarding @g in hardware and free its provider object. The caller
+ * cleared hw_active and waited out ops->xmit, or the group never had it.
+ * Returns whether group_disable overran op_timeout_ms.
+ */
+static bool amt_ofl_grp_hw_free(struct amt_offload_state *st,
+				struct amt_offload_bound *b,
+				struct amt_ofl_grp *g)
+{
+	ktime_t t0 = ktime_get();
+	bool overran;
+
+	b->ops->group_disable(&b->pub, g->hw);
+	overran = amt_ofl_op_overran(b, t0);
+	amt_ofl_grp_uncommit(st, g);
+	g->hw = NULL;
+	list_del_init(&g->hw_node);
+	st->hw_objs--;
+	return overran;
+}
 
-	if (!ref)
+/* Retry parked groups once the provider holds an eighth fewer objects
+ * than when it refused one. Refused with nothing programmed, only a
+ * reset or a new binding retries them. Caller holds st->mutex.
+ */
+static void amt_ofl_unpark(struct amt_offload_state *st)
+{
+	struct amt_ofl_grp *g, *t;
+
+	if (!st->enospc || !st->enospc_at ||
+	    st->hw_objs + max(st->enospc_at / 8, 1U) > st->enospc_at)
+		return;
+	spin_lock_bh(&st->lock);
+	list_for_each_entry_safe(g, t, &st->parked, q_node) {
+		list_del(&g->q_node);
+		g->state = AMT_OFL_IDLE;
+		amt_ofl_grp_dirty(st, g);
+	}
+	st->enospc = false;
+	spin_unlock_bh(&st->lock);
+}
+
+/* Programming @g failed: software serves it from now until a retry. Its
+ * endpoints stay programmed; they may serve other groups.
+ */
+static unsigned int amt_ofl_grp_park(struct amt_offload_state *st,
+				     struct amt_offload_bound *b,
+				     struct amt_ofl_grp *g)
+{
+	unsigned int ops = 0;
+
+	if (g->hw) {
+		/* Wait out ops->xmit even when hw_active already reads false:
+		 * amt_ofl_member_unlink() clears it from softirq with no grace
+		 * period, so a reader admitted before may still hold g->hw.
+		 */
+		WRITE_ONCE(g->hw_active, false);
+		amt_ofl_sync_net();
+		amt_ofl_grp_hw_free(st, b, g);
+		ops++;
+	}
+	if (!st->enospc) {
+		st->enospc = true;
+		st->enospc_at = st->hw_objs;
+	}
+	amt_offload_count(st->amt, AMT_OFFLOAD_CNT_SW_REMAINDER);
+	spin_lock_bh(&st->lock);
+	if (g->state == AMT_OFL_DIRTY && !g->nr_members) {
+		/* Emptied meanwhile: let the reconciler free it. */
+	} else {
+		if (g->state == AMT_OFL_DIRTY)
+			list_del(&g->q_node);
+		list_add_tail(&g->q_node, &st->parked);
+		g->state = AMT_OFL_PARKED;
+	}
+	spin_unlock_bh(&st->lock);
+	return ops;
+}
+
+/* Program @g at its current generation: add its endpoints that are not
+ * in hardware yet, stage the member set, commit, publish. Returns the
+ * ops spent. Caller holds st->mutex; @g is off every queue.
+ */
+static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
+					struct amt_offload_bound *b,
+					struct amt_ofl_grp *g)
+{
+	struct amt_offload_endpoint **hws = NULL;
+	struct amt_ofl_ep **eps = NULL;
+	struct amt_offload_group *hw;
+	struct amt_ofl_member *m;
+	unsigned int ops = 0;
+	u32 n, i;
+	ktime_t t0;
+	u64 epoch;
+	u32 left;
+	u64 gen;
+	int err;
+
+	for (;;) {
+		spin_lock_bh(&st->lock);
+		n = g->nr_members;
+		spin_unlock_bh(&st->lock);
+		if (!n)
+			goto requeue;
+		eps = amt_ofl_alloc_array(n, sizeof(*eps));
+		hws = amt_ofl_alloc_array(n, sizeof(*hws));
+		if (!eps || !hws)
+			goto park;
+		spin_lock_bh(&st->lock);
+		if (g->nr_members == n)
+			break;
+		spin_unlock_bh(&st->lock);
+		/* Membership moved under the allocation. The next pass may
+		 * find none and leave through out:, which frees both again.
+		 */
+		amt_ofl_free_array(eps);
+		amt_ofl_free_array(hws);
+		eps = NULL;
+		hws = NULL;
+	}
+	i = 0;
+	list_for_each_entry(m, &g->members, grp_node)
+		eps[i++] = m->ep;
+	gen = g->gen;
+	left = g->left;
+	spin_unlock_bh(&st->lock);
+
+	if (g->hw && READ_ONCE(g->hw_active) && g->hw_gen == gen)
+		goto out;	/* a replay of what is already committed */
+
+	for (i = 0; i < n; i++) {
+		struct amt_offload_endpoint *ehw = NULL;
+
+		if (!eps[i]->hw) {
+			if (!READ_ONCE(b->admitting))
+				goto out;	/* teardown gives it all back */
+			epoch = b->pub.epoch;
+			t0 = ktime_get();
+			err = b->ops->endpoint_add(&b->pub, &eps[i]->key, &ehw);
+			ops++;
+			if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+				goto out;	/* reset: replayed */
+			if (err)
+				goto park;
+			eps[i]->hw = ehw;
+			list_add_tail(&eps[i]->hw_node, &st->hw_eps);
+			st->hw_objs++;
+		}
+		hws[i] = eps[i]->hw;
+	}
+
+	if (!READ_ONCE(b->admitting))
+		goto out;
+	hw = g->hw;
+	epoch = b->pub.epoch;
+	t0 = ktime_get();
+	err = b->ops->group_update(&b->pub, &g->key, hws, n, gen, &hw);
+	ops++;
+	if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+		goto out;
+	if (!g->hw && hw) {
+		g->hw = hw;
+		list_add_tail(&g->hw_node, &st->hw_grps);
+		st->hw_objs++;
+	}
+	if (err)
+		goto park;
+
+	if (!READ_ONCE(b->admitting))
+		goto out;
+	epoch = b->pub.epoch;
+	t0 = ktime_get();
+	err = b->ops->group_commit(&b->pub, g->hw, gen);
+	ops++;
+	if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+		goto out;
+	if (err)
+		goto park;
+
+	/* Generation rule: packets submitted from here on carry @gen.
+	 * Release pairs with the acquire in amt_offload_tx(): a reader that
+	 * sees hw_active also sees hw and hw_gen. Publish only if no member
+	 * was removed since the snapshot: it is still in what was just
+	 * committed, and amt_ofl_member_unlink() already withdrew the group.
+	 * A join meanwhile does not hold the publication back: the committed
+	 * set is a subset that names no departed endpoint. Either change
+	 * re-queued @g; the next pass commits it.
+	 */
+	spin_lock_bh(&st->lock);
+	for (i = 0; i < n; i++)
+		eps[i]->committed++;
+	for (i = 0; i < g->nr_ceps; i++) {
+		g->ceps[i]->committed--;
+		amt_ofl_ep_unused(st, g->ceps[i]);
+	}
+	swap(g->ceps, eps);	/* out: frees the old set */
+	g->nr_ceps = n;
+	WRITE_ONCE(g->hw_gen, gen);
+	if (g->left == left)
+		smp_store_release(&g->hw_active, true);
+	spin_unlock_bh(&st->lock);
+	goto out;
+park:
+	ops += amt_ofl_grp_park(st, b, g);
+	goto out;
+requeue:
+	spin_lock_bh(&st->lock);
+	amt_ofl_grp_dirty(st, g);
+	spin_unlock_bh(&st->lock);
+out:
+	amt_ofl_free_array(eps);
+	amt_ofl_free_array(hws);
+	return ops;
+}
+
+/* Disable the unhashed groups on @gone: one grace period per flush, then
+ * disable. A group holds hw only while bound, so @b is set when any does.
+ */
+static void amt_ofl_gone_flush(struct amt_offload_state *st,
+			       struct amt_offload_bound *b,
+			       struct list_head *gone)
+{
+	struct amt_ofl_grp *g, *gt;
+	u64 e;
+
+	if (list_empty(gone))
+		return;
+	amt_ofl_sync_net();
+	list_for_each_entry_safe(g, gt, gone, q_node) {
+		list_del(&g->q_node);
+		/* A reset earlier in this flush or tick already flushed it. */
+		if (g->hw) {
+			e = b->pub.epoch;
+			amt_ofl_op_done(st, b, e, amt_ofl_grp_hw_free(st, b, g));
+		}
+		kfree_rcu(g, rcu);
+	}
+}
+
+/* Is a group with @key on @gone, its provider object still live? */
+static bool amt_ofl_gone_has(struct list_head *gone,
+			     const struct amt_offload_group_key *key)
+{
+	struct amt_ofl_grp *g;
+
+	list_for_each_entry(g, gone, q_node)
+		if (!memcmp(&g->key, key, sizeof(*key)))
+			return true;
+	return false;
+}
+
+/* One reconciler tick. Returns true while work remains. */
+static bool amt_ofl_reconcile(struct amt_dev *amt)
+{
+	struct amt_offload_state *st = amt->offload_st;
+	struct amt_ofl_grp *g;
+	struct amt_offload_bound *b;
+	LIST_HEAD(gone);
+	unsigned int spent = 0;
+	unsigned int budget;
+	struct amt_ofl_ep *ep;
+	bool gc_del = false;
+	u64 epoch = 0, e;
+	ktime_t t0;
+	bool more;
+
+	budget = max(READ_ONCE(st->budget), 1U);
+	mutex_lock(&st->mutex);
+	b = rcu_dereference_protected(amt->offload,
+				      lockdep_is_held(&st->mutex));
+	if (b && !READ_ONCE(b->admitting)) {
+		/* Teardown is under way and gives everything back. */
+		mutex_unlock(&st->mutex);
+		return false;
+	}
+	if (b && READ_ONCE(st->desync)) {
+		amt_ofl_hw_release(st, b, false);
+		b = NULL;
+	}
+	if (b)
+		epoch = b->pub.epoch;
+
+	while (spent < budget) {
+		/* A teardown waiting on st->mutex holds RTNL, and a reset
+		 * replays everything anyway: end the pass at either, and let
+		 * the work re-kick, rather than spend the whole budget
+		 * against a provider that may be wedged.
+		 */
+		if (b && (!READ_ONCE(b->admitting) || b->pub.epoch != epoch))
+			break;
+		spin_lock_bh(&st->lock);
+		g = list_first_entry_or_null(&st->dirty, struct amt_ofl_grp,
+					     q_node);
+		if (!g) {
+			spin_unlock_bh(&st->lock);
+			break;
+		}
+		list_del(&g->q_node);
+		g->state = AMT_OFL_IDLE;
+		if (!g->nr_members) {
+			hlist_del_init_rcu(&g->hnode);
+			spin_unlock_bh(&st->lock);
+			if (g->hw) {
+				WRITE_ONCE(g->hw_active, false);
+				list_add_tail(&g->q_node, &gone);
+				spent++;
+			} else {
+				kfree_rcu(g, rcu);
+			}
+			continue;
+		}
+		spin_unlock_bh(&st->lock);
+		if (!b)
+			continue;
+		/* A rejoin of a group emptied earlier in this tick hashed a
+		 * new object for the same key while the old one's provider
+		 * group is still live: disable that one first, or the
+		 * provider would hold two groups for one (family, group).
+		 */
+		if (amt_ofl_gone_has(&gone, &g->key)) {
+			amt_ofl_gone_flush(st, b, &gone);
+			/* A reset in that flush requeued @g: replay it. */
+			if (b->pub.epoch != epoch)
+				break;
+		}
+		spent += amt_ofl_grp_program(st, b, g);
+	}
+
+	amt_ofl_gone_flush(st, b, &gone);
+
+	/* Unused endpoints: no member uses one and no committed group names
+	 * it (ep->committed), so it goes whatever is still dirty. At least
+	 * one endpoint_del per tick, or a group re-dirtied on every pass
+	 * would spend the budget and starve it.
+	 */
+	for (;;) {
+		if (b && !READ_ONCE(b->admitting))
+			break;
+		spin_lock_bh(&st->lock);
+		ep = spent < budget || !gc_del ?
+		     list_first_entry_or_null(&st->ep_gc, struct amt_ofl_ep,
+					      gc_node) : NULL;
+		if (!ep) {
+			spin_unlock_bh(&st->lock);
+			break;
+		}
+		list_del(&ep->gc_node);
+		ep->on_gc = false;
+		if (ep->refs || ep->committed) {
+			/* Queued again when both reach zero. */
+			spin_unlock_bh(&st->lock);
+			continue;
+		}
+		hlist_del_init(&ep->hnode);
+		spin_unlock_bh(&st->lock);
+		if (ep->hw) {
+			list_del_init(&ep->hw_node);
+			st->hw_objs--;
+			e = b->pub.epoch;
+			t0 = ktime_get();
+			b->ops->endpoint_del(&b->pub, ep->hw);
+			spent++;
+			gc_del = true;
+			amt_ofl_op_done(st, b, e, amt_ofl_op_overran(b, t0));
+		}
+		kfree(ep);
+	}
+
+	if (b)
+		amt_ofl_unpark(st);
+	spin_lock_bh(&st->lock);
+	more = !list_empty(&st->dirty) || !list_empty(&st->ep_gc);
+	spin_unlock_bh(&st->lock);
+	mutex_unlock(&st->mutex);
+	return more;
+}
+
+static void amt_ofl_work(struct work_struct *work)
+{
+	struct amt_offload_state *st = container_of(work,
+						    struct amt_offload_state,
+						    work);
+
+	if (amt_ofl_reconcile(st->amt))
+		amt_ofl_kick(st);
+}
+
+static int amt_ofl_state_alloc(struct amt_dev *amt)
+{
+	struct amt_offload_state *st;
+
+	st = kzalloc_obj(*st, GFP_KERNEL);
+	if (!st)
 		return -ENOMEM;
-	ref->handle = handle;
-	list_add_tail(&ref->node, list);
+	st->ep_bits = clamp_t(u32, order_base_2(amt->max_tunnels),
+			      AMT_OFL_HASH_MIN_BITS, AMT_OFL_HASH_MAX_BITS);
+	/* Clamp in 64 bits: order_base_2() takes an unsigned long, which
+	 * would truncate the product on a 32-bit kernel.
+	 */
+	st->grp_bits = clamp_t(u32, order_base_2(min_t(u64,
+					(u64)amt->max_tunnels * amt->max_groups,
+					1U << AMT_OFL_HASH_MAX_BITS)),
+			       AMT_OFL_HASH_MIN_BITS, AMT_OFL_HASH_MAX_BITS);
+	st->grps = kvcalloc(1U << st->grp_bits, sizeof(*st->grps), GFP_KERNEL);
+	st->eps = kvcalloc(1U << st->ep_bits, sizeof(*st->eps), GFP_KERNEL);
+	if (!st->grps || !st->eps) {
+		kvfree(st->grps);
+		kvfree(st->eps);
+		kfree(st);
+		return -ENOMEM;
+	}
+	st->amt = amt;
+	spin_lock_init(&st->lock);
+	INIT_LIST_HEAD(&st->dirty);
+	INIT_LIST_HEAD(&st->parked);
+	INIT_LIST_HEAD(&st->ep_gc);
+	mutex_init(&st->mutex);
+	INIT_WORK(&st->work, amt_ofl_work);
+	INIT_LIST_HEAD(&st->hw_grps);
+	INIT_LIST_HEAD(&st->hw_eps);
+	st->budget = READ_ONCE(amt_offload_op_budget);
+	amt->offload_st = st;
 	return 0;
 }
 
+/* ndo_uninit: every tunnel is gone and the binding with it, so nothing
+ * here is in hardware. Members are freed without touching their tunnels.
+ */
+static void amt_ofl_state_free(struct amt_dev *amt)
+{
+	struct amt_offload_state *st = amt->offload_st;
+	struct amt_ofl_member *m, *mt;
+	struct hlist_node *tmp;
+	struct amt_ofl_grp *g;
+	struct amt_ofl_ep *ep;
+	u32 bkt;
+
+	if (!st)
+		return;
+	cancel_work_sync(&st->work);
+	for (bkt = 0; bkt < 1U << st->grp_bits; bkt++) {
+		hlist_for_each_entry_safe(g, tmp, &st->grps[bkt], hnode) {
+			list_for_each_entry_safe(m, mt, &g->members, grp_node)
+				kfree(m);
+			amt_ofl_free_array(g->ceps);
+			kfree(g);
+		}
+	}
+	for (bkt = 0; bkt < 1U << st->ep_bits; bkt++)
+		hlist_for_each_entry_safe(ep, tmp, &st->eps[bkt], hnode)
+			kfree(ep);
+	mutex_destroy(&st->mutex);
+	kvfree(st->grps);
+	kvfree(st->eps);
+	kfree(st);
+	amt->offload_st = NULL;
+}
+
 static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
 {
 	if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
@@ -4191,10 +5268,10 @@ static int amt_offload_bind(struct amt_dev *amt,
 		return -ENOMEM;
 	b->pub.lower = p->lower;
 	b->pub.dev = amt->dev;
-	b->pub.epoch = ++amt->offload_epoch;
 	b->ops = p->ops;
-	INIT_LIST_HEAD(&b->groups);
-	INIT_LIST_HEAD(&b->endpoints);
+	mutex_lock(&amt->offload_st->mutex);
+	b->pub.epoch = ++amt->offload_epoch;
+	mutex_unlock(&amt->offload_st->mutex);
 
 	err = b->ops->bind(&b->pub, &b->caps);
 	if (err) {
@@ -4210,7 +5287,12 @@ static int amt_offload_bind(struct amt_dev *amt,
 	}
 
 	b->admitting = true;
+	mutex_lock(&amt->offload_st->mutex);
 	rcu_assign_pointer(amt->offload, b);
+	/* Replay desired state: every group is programmed afresh. */
+	amt_ofl_requeue_all(amt->offload_st);
+	mutex_unlock(&amt->offload_st->mutex);
+	amt_ofl_kick(amt->offload_st);
 	amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND);
 	if (on_register)
 		amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
@@ -4225,7 +5307,7 @@ static void amt_offload_teardown(struct amt_dev *amt,
 				 enum amt_offload_counter cnt)
 {
 	struct amt_offload_bound *b = rtnl_dereference(amt->offload);
-	struct amt_offload_ref *ref, *tmp;
+	struct amt_offload_state *st = amt->offload_st;
 
 	if (!b)
 		return;
@@ -4234,21 +5316,14 @@ static void amt_offload_teardown(struct amt_dev *amt,
 	WRITE_ONCE(b->admitting, false);
 	synchronize_net();
 
-	/* 2. Programming work is cancelled here once the reconciler exists
-	 *    (next patch); nothing can program the provider past this point.
+	/* 2. Cancel programming: a reconciler tick holds st->mutex
+	 *    throughout, so taking it waits one out; a later tick finds
+	 *    admission stopped, then no binding, and programs nothing.
 	 */
+	mutex_lock(&st->mutex);
 
 	/* 3. Disable every group, then 4. delete every endpoint. */
-	list_for_each_entry_safe(ref, tmp, &b->groups, node) {
-		b->ops->group_disable(&b->pub, ref->handle);
-		list_del(&ref->node);
-		kfree(ref);
-	}
-	list_for_each_entry_safe(ref, tmp, &b->endpoints, node) {
-		b->ops->endpoint_del(&b->pub, ref->handle);
-		list_del(&ref->node);
-		kfree(ref);
-	}
+	amt_ofl_hw_release(st, b, true);
 
 	/* 5. Unbind, then 6. unpublish and let RCU readers drain. */
 	b->ops->unbind(&b->pub);
@@ -4256,8 +5331,10 @@ static void amt_offload_teardown(struct amt_dev *amt,
 	synchronize_rcu();
 	kfree(b);
 
-	/* 7. Software. */
+	/* 7. Software. Desired state stays for the next binding. */
 	amt->offload_epoch++;
+	mutex_unlock(&st->mutex);
+	amt_ofl_kick(st);
 	if (cnt < AMT_OFFLOAD_CNT_MAX) {
 		amt_offload_count(amt, cnt);
 		if (amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
@@ -4275,7 +5352,9 @@ int amt_offload_register(struct net_device *lower,
 
 	ASSERT_RTNL();
 	if (!lower || !ops || !ops->bind || !ops->unbind ||
-	    !ops->group_disable || !ops->endpoint_del)
+	    !ops->endpoint_add || !ops->endpoint_del ||
+	    !ops->group_update || !ops->group_commit ||
+	    !ops->group_disable || !ops->flush)
 		return -EINVAL;
 	if (amt_offload_provider_find(lower))
 		return -EEXIST;
@@ -4338,6 +5417,9 @@ static int amt_offload_attach(struct amt_dev *amt,
 	INIT_LIST_HEAD(&amt->offload_node);
 	if (amt->mode != AMT_MODE_RELAY)
 		return 0;
+	if (amt->offload_mode != AMT_OFFLOAD_MODE_SOFTWARE &&
+	    !amt->offload_st && amt_ofl_state_alloc(amt))
+		return -ENOMEM;
 	list_add_tail(&amt->offload_node, &amt_offload_devs);
 	if (amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE)
 		return 0;
@@ -4420,6 +5502,10 @@ AMT_OFFLOAD_CNT_ATTR(family_unsupported, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
 AMT_OFFLOAD_CNT_ATTR(teardown_lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
 AMT_OFFLOAD_CNT_ATTR(teardown_provider, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
 AMT_OFFLOAD_CNT_ATTR(required_unbound, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+AMT_OFFLOAD_CNT_ATTR(sw_remainder, AMT_OFFLOAD_CNT_SW_REMAINDER);
+AMT_OFFLOAD_CNT_ATTR(resets, AMT_OFFLOAD_CNT_RESET);
+AMT_OFFLOAD_CNT_ATTR(stale_completions, AMT_OFFLOAD_CNT_STALE);
+AMT_OFFLOAD_CNT_ATTR(desync, AMT_OFFLOAD_CNT_DESYNC);
 
 static struct attribute *amt_offload_attrs[] = {
 	&dev_attr_offload_mode.attr,
@@ -4431,6 +5517,10 @@ static struct attribute *amt_offload_attrs[] = {
 	&dev_attr_offload_teardown_lower.attr,
 	&dev_attr_offload_teardown_provider.attr,
 	&dev_attr_offload_required_unbound.attr,
+	&dev_attr_offload_sw_remainder.attr,
+	&dev_attr_offload_resets.attr,
+	&dev_attr_offload_stale_completions.attr,
+	&dev_attr_offload_desync.attr,
 	NULL,
 };
 
diff --git a/include/net/amt.h b/include/net/amt.h
index 7d70f30e6..0c746e4be 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -312,6 +312,11 @@ struct amt_tunnel_list {
 	struct rcu_head		rcu;
 	/* This tunnel's entries in amt->groups_rhl, under lock */
 	struct list_head	groups;
+	/* Offload desired state: one amt_ofl_member per group this tunnel
+	 * is joined to, under amt_offload_state.lock. Empty unless the
+	 * relay's offload mode is auto or required.
+	 */
+	struct hlist_head	offload_members;
 };
 
 /* RFC 3810
@@ -380,6 +385,7 @@ struct amt_events {
 };
 
 struct amt_offload_bound;
+struct amt_offload_state;
 
 /* Per-relay offload counters, exported under offload/ in sysfs. */
 enum amt_offload_counter {
@@ -399,6 +405,19 @@ enum amt_offload_counter {
 	AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER,
 	/* a required-mode relay lost its provider and dropped carrier */
 	AMT_OFFLOAD_CNT_REQUIRED_UNBOUND,
+	/* a group could not be programmed and is served by software */
+	AMT_OFFLOAD_CNT_SW_REMAINDER,
+	/* an op overran op_timeout_ms: flush and replay from desired state
+	 * (a release op while amt gives everything back: flush only)
+	 */
+	AMT_OFFLOAD_CNT_RESET,
+	/* an op result arrived under an older epoch and was ignored; a
+	 * programming, gone or GC op that overran counts here too, since its
+	 * reset moved the epoch (a release op's flush does not move it)
+	 */
+	AMT_OFFLOAD_CNT_STALE,
+	/* desired state lost an update (allocation failure); software only */
+	AMT_OFFLOAD_CNT_DESYNC,
 	AMT_OFFLOAD_CNT_MAX,
 };
 
@@ -508,16 +527,24 @@ struct amt_dev {
 	struct socket		*stream_sock_v6;
 	atomic_t		upstream_setsockopt_slow_count;
 
-	/* Replication offload (relay mode). Everything here is written
-	 * under RTNL; offload is also read by the data path under RCU.
-	 * offload_node links the relay into amt_offload_devs so a provider
-	 * registering on its stream_dev can find it.
+	/* Replication offload (relay mode). offload_mode is set at newlink.
+	 * offload is written under RTNL and offload_st's mutex, and read by
+	 * the data path under RCU. offload_epoch is written under
+	 * offload_st's mutex: at bind and teardown with RTNL held too, and
+	 * by the reconciler, without RTNL, when it resets the provider.
+	 * offload_node, under RTNL, links the relay into amt_offload_devs so
+	 * a provider registering on its stream_dev can find it. offload_cnt
+	 * is atomic and bumped from any context.
 	 */
 	u32			offload_mode;
 	struct amt_offload_bound __rcu *offload;
 	struct list_head	offload_node;
 	u64			offload_epoch;
 	atomic_t		offload_cnt[AMT_OFFLOAD_CNT_MAX];
+	/* Desired state and reconciler; allocated at newlink in auto and
+	 * required mode, freed at ndo_uninit once every tunnel is gone.
+	 */
+	struct amt_offload_state *offload_st;
 };
 
 #define AMT_TOS			0xc0
diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
index 3be5d129c..b5c119851 100644
--- a/include/net/amt_offload.h
+++ b/include/net/amt_offload.h
@@ -179,10 +179,17 @@ struct amt_offload_stats {
  *	been deleted or disabled. Free @b->priv. RTNL held.
  * @endpoint_add: program one gateway endpoint. Idempotent: an identical
  *	key returns the existing handle. 0 means accepted and durable, not yet
- *	forwarding. Process context, never under amt's spinlocks.
- * @endpoint_del: remove an endpoint no group references any more.
+ *	forwarding.
+ * @endpoint_del: remove an endpoint no committed group references any
+ *	more. amt only deletes endpoints once every group naming one has been
+ *	recommitted without it or disabled.
  * @group_update: stage the member set of one group under generation @gen,
- *	allocating *@grp on first use. Not forwarding until @group_commit.
+ *	allocating *@grp on first use (also on failure, if it allocated). On
+ *	any failure amt serves the group in software and disables *@grp if it
+ *	is set, a live group included. @members is valid only during the
+ *	call. Not forwarding until @group_commit. amt holds at most one group
+ *	per key per binding: a first use (*@grp NULL) never names a key whose
+ *	earlier group on this binding has not been disabled or flushed.
  * @group_commit: start forwarding generation @gen of @grp. amt hands @gen
  *	to @xmit only after this returns 0, and may never do so: an overrun
  *	resets the provider, and on TX a member leaving meanwhile withholds
@@ -190,9 +197,11 @@ struct amt_offload_stats {
  *	@grp carries an earlier generation this provider committed, and a call
  *	already in flight may carry any of those or @gen itself; the provider
  *	serves such a generation or declines it. A provider must never emit
- *	one packet to one endpoint under two generations. On failure the group
- *	stays in software and previously added endpoints stay valid.
- * @group_disable: stop forwarding @grp and free it.
+ *	one packet to one endpoint under two generations. On failure amt
+ *	disables the group and serves it in software; previously added
+ *	endpoints stay valid, since they may serve other groups.
+ * @group_disable: stop forwarding @grp and free it. amt has already
+ *	stopped submitting packets for @grp and waited out @xmit callers.
  * @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
  *	&enum amt_offload_result. Softirq context, under rcu_read_lock().
  * @flush: drop every endpoint and group (reset after a timeout). amt then
@@ -237,15 +246,64 @@ struct amt_offload_ops {
 		      struct amt_offload_stats *st);
 };
 
+/*
+ * Completion semantics. Every programming op (@endpoint_add,
+ * @endpoint_del, @group_update, @group_commit, @group_disable, @flush) is
+ * synchronous: amt calls it in process context, never under amt's
+ * spinlocks -- from one reconciler per relay, or with RTNL held when the
+ * binding is torn down -- so a provider may sleep in it but must not
+ * take RTNL. Return 0 means accepted and
+ * durable in the provider's tables. Nothing forwards before @group_commit
+ * returns 0. @xmit is only passed a generation for which @group_commit on
+ * that group returned 0, which can lag the one committed last (see
+ * @group_commit).
+ *
+ * An op that takes longer than &amt_offload_caps.op_timeout_ms (0: no
+ * limit) marks the provider wedged: amt stops submitting packets, calls
+ * @flush, forgets every handle, bumps &amt_offload_binding.epoch and
+ * replays every group from its own desired state. The exception is amt
+ * giving every object back (teardown, or desired state lost): there an
+ * overrunning @group_disable or @endpoint_del ends it with @flush, and the
+ * remaining handles are forgotten, with no new epoch and no replay. A
+ * result returned under an older epoch is ignored. After @flush the
+ * provider holds nothing for this binding; amt never passes it an older
+ * handle again.
+ *
+ * Cancellation is the teardown order on &struct amt_offload_ops. A
+ * reconciler pass running when a teardown starts finishes the op it is in
+ * and then makes no further @endpoint_add, @group_update, @group_commit or
+ * @endpoint_del, except that it may still call @flush (after an overrun),
+ * @group_disable (for a group it parks or found emptied) and, on a pass
+ * giving every object back once desired state is lost, the remaining
+ * @group_disable and @endpoint_del calls of that release.
+ *
+ * The reconciler starts no new group once a pass has spent the amt module
+ * parameter offload_op_budget (default 1000, an estimate) ops, and never
+ * splits a group's ops across passes, so a pass can overrun the budget by
+ * one group's ops. If the budget left no room for one, a pass still makes
+ * one @endpoint_del, so endpoint collection is never starved. The pass
+ * that gives every object back once desired state is lost is not
+ * budgeted. A group the provider refuses (-ENOSPC or any other error) is
+ * served by software and retried once the provider holds an eighth fewer
+ * objects than when it refused, after a reset, or on a new binding.
+ */
+
 /**
  * amt_offload_register - offer a provider for relays on @lower
  * @lower: the netdev the relays transmit on.
  * @ops: provider callbacks; must outlive the registration.
  *
  * Every relay already using @lower in auto or required mode tries to bind
- * now. Caller holds RTNL.
+ * now. Caller holds RTNL. Every op but @stats and @xmit is required.
+ * Whether a provider offers %AMT_OFFLOAD_PATH_TX shows only in the caps
+ * @bind fills, so @xmit is not checked here. Without it, once @bind
+ * succeeds and the caps cover the relay's outer family, amt treats TX as
+ * not offered: a relay in required mode whose caps offer
+ * %AMT_OFFLOAD_PATH_INGRESS binds on the ingress path, and any other
+ * relay stays unbound (counted as path_unsupported).
  *
- * Return: 0, -EEXIST if @lower already has a provider, or -ENOMEM.
+ * Return: 0, -EINVAL if a required op is missing, -EEXIST if @lower
+ * already has a provider, or -ENOMEM.
  */
 int amt_offload_register(struct net_device *lower,
 			 const struct amt_offload_ops *ops);
-- 
2.43.0

[RFC PATCH net-next 4/9] amt: offload TX handoff, ingress punt rule and counters

[Omar Ramadan]

On a TX binding, amt_dev_xmit hands each relay data packet of a group
the reconciler has published (hw_active, set with hw_gen after a
successful group_commit) to the provider before its software fan-out,
while the binding admits packets and desired state is in sync; every
other packet goes to software. Exactly one actor handles each packet
handed over: CONSUMED means the provider handled every copy (sent it, or
dropped it and counted it, apart from the copy and route failures
amt_offload_encap_xmit() counts itself) and owns the skb; FALLBACK
leaves the skb untouched for the software loop; BUSY runs one
synchronous software pass and counts it, never NETDEV_TX_BUSY since the
relay is IFF_NO_QUEUE. hw_gen is read once per packet. The reconciler
writes it after a successful group_commit, also when it then withholds
hw_active because a member left meanwhile, so a packet already past the
hw_active test can carry the new generation and one that read hw_gen
earlier any older committed one; the provider serves or declines it.

On the ingress path the provider replicates every packet of a group from
group_commit until group_disable or flush, and must never also punt one.
While the group's commit is published (hw_active, with the binding
admitting and desired state in sync), a punted copy that reaches the
relay is dropped and counted; at any other time (before the first
publication, after a park, during a reset or a teardown, after desync)
it goes to the software fan-out. The provider sees none of these
transitions as amt makes them, so the drop is only a safety net.
Removing a member on an ingress binding does not withdraw the group as
it does on TX: hardware replicates to the committed set until the
recommit regardless, and lifting the drop would send the remaining
members every punted packet a second time. For the same reason the
reconciler publishes a commit on an ingress binding even when a member
left while it was programmed. A relay binds on ingress only in required
mode: a packet the provider consumes on ingress never reaches the host,
so nothing on the host's forwarding path, mroute counters included, sees
it, and auto does not choose that. A provider offering no path the relay
may use leaves it unbound, counted. TX needs ops->xmit.

New offload/ counters: path_unsupported, busy_fallback, punt_dropped
(plus sw_remainder, resets, stale_completions and desync from the
reconciler). With no provider bound the data path does one more RCU
pointer read and test.

A simulated replication provider -- netdevsim is the reference one --
decides from its own tables which gateways get a copy of a relay data
packet, but has no business re-implementing AMT encapsulation: if its
copies differed on the wire from amt's own, a test could not tell a
provider bug from an encapsulation difference.

amt_offload_encap_xmit(binding, endpoint key, skb), exported, emits one
gateway's copy byte for byte as the software fan-out does: the copy, AMT
Multicast Data header and amt_udp_xmit() of amt_send_multicast_data(),
from the relay port to the gateway's port with AMT_TOS on an IPv4 outer
and the route's hop limit. Each failed copy or route lookup counts once
in tx_dropped; the software fan-out on this base counts neither. It
takes the endpoint key, not the provider's handle, which is opaque to
amt, and is for use from ops->xmit only.

With no provider registered nothing calls it.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/amt.c         | 206 +++++++++++++++++++++++++++++++++++---
 include/net/amt.h         |  13 ++-
 include/net/amt_offload.h |  38 ++++++-
 3 files changed, 237 insertions(+), 20 deletions(-)

diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 17337989a..377660e55 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -319,6 +319,8 @@ static void amt_ofl_member_del(struct amt_dev *amt,
 static void amt_ofl_tunnel_gone(struct amt_dev *amt,
 				struct amt_tunnel_list *tunnel);
 static void amt_ofl_state_free(struct amt_dev *amt);
+static bool amt_offload_tx(struct amt_dev *amt, struct sk_buff *skb,
+			   const union amt_addr *group, bool v6);
 
 static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
 {
@@ -1510,6 +1512,11 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
 	} else if (amt->mode == AMT_MODE_RELAY) {
 		if (!data)
 			goto free;
+		/* A bound offload provider replicates it, or declines; see
+		 * amt_offload_tx() for the cutover rules.
+		 */
+		if (amt_offload_tx(amt, skb, &group, v6))
+			return NETDEV_TX_OK;
 		key.group_addr = group;
 		key.v6 = v6;
 		list_for_each_entry_rcu(tunnel, &amt->tunnel_list, list) {
@@ -4087,6 +4094,7 @@ struct amt_offload_bound {
 	struct amt_offload_binding	pub;
 	const struct amt_offload_ops	*ops;
 	struct amt_offload_caps		caps;
+	enum amt_offload_path		path;
 	/* Cleared first on teardown; ops->xmit is only called while set. */
 	bool				admitting;
 };
@@ -4209,6 +4217,10 @@ struct amt_offload_state {
 	struct list_head		parked;
 	struct list_head		ep_gc;
 	bool				desync;
+	/* The binding is on the ingress path: a member change must not
+	 * clear hw_active, which there gates the punt-drop. Set at bind.
+	 */
+	bool				ingress;
 	/* Reconciler. bind and teardown also take the mutex. */
 	struct mutex			mutex;
 	struct work_struct		work;
@@ -4286,9 +4298,9 @@ static u32 amt_offload_outer_family(const struct amt_dev *amt)
 }
 
 /* The data path's way in: a binding is usable only while admitting. The
- * caller holds rcu_read_lock(). The later TX handoff is its first caller.
+ * caller holds rcu_read_lock().
  */
-static __maybe_unused struct amt_offload_bound *
+static struct amt_offload_bound *
 amt_offload_tx_admit(struct amt_dev *amt)
 {
 	struct amt_offload_bound *b = rcu_dereference(amt->offload);
@@ -4402,13 +4414,15 @@ static void amt_ofl_kick(struct amt_offload_state *st)
 }
 
 /* Desired state could not record a change. Hardware would forward to a
- * stale member set, so the relay leaves offload for good: the data path
- * stops consulting the provider, and the next reconciler pass gives every
- * provider object back (group_disable, endpoint_del; the binding stays,
- * with no unbind). Desync does not schedule that pass: a pass already
- * queued or a later removal of a recorded member (by a leave, a timer or
- * a tunnel removal) does, and a teardown gives everything back itself
- * and unbinds. Caller holds st->lock.
+ * stale member set, so the relay leaves offload for good. On TX that
+ * takes effect at once: the data path stops handing packets to the
+ * provider. The next reconciler pass gives every provider object back
+ * (group_disable, endpoint_del; the binding stays, with no unbind).
+ * Desync does not schedule that pass: a pass already queued or a later
+ * removal of a recorded member (by a leave, a timer or a tunnel removal)
+ * does, and a teardown gives everything back itself and unbinds. On an
+ * ingress binding the provider replicates to its committed sets until
+ * then. Caller holds st->lock.
  */
 static void amt_ofl_desync(struct amt_offload_state *st)
 {
@@ -4562,12 +4576,16 @@ static void amt_ofl_member_unlink(struct amt_offload_state *st,
 	g->nr_members--;
 	g->gen++;
 	g->left++;
-	/* The committed member set still names this endpoint: stop handing
-	 * the group to hardware now, not when the reconciler recommits, so
-	 * no packet admitted after this is replicated to it. Software serves
-	 * the group meanwhile.
+	/* TX: the committed member set still names this endpoint. Stop
+	 * handing the group to hardware now, not when the reconciler
+	 * recommits, so no packet admitted after this is replicated to it;
+	 * software serves the group meanwhile. Ingress: hardware replicates
+	 * on its own until the recommit whatever the host does, and
+	 * hw_active there only drops the punted copies, so clearing it
+	 * would send the remaining members every packet twice.
 	 */
-	WRITE_ONCE(g->hw_active, false);
+	if (!READ_ONCE(st->ingress))
+		WRITE_ONCE(g->hw_active, false);
 	amt_ofl_grp_dirty(st, g);
 	ep->refs--;
 	amt_ofl_ep_unused(st, ep);
@@ -4966,7 +4984,10 @@ static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
 	 * committed, and amt_ofl_member_unlink() already withdrew the group.
 	 * A join meanwhile does not hold the publication back: the committed
 	 * set is a subset that names no departed endpoint. Either change
-	 * re-queued @g; the next pass commits it.
+	 * re-queued @g; the next pass commits it. On ingress, hardware is
+	 * already replicating to the committed set whatever the host does,
+	 * and hw_active only drops the punted copies, so withholding it there
+	 * protects no one and sends the remaining members every packet twice.
 	 */
 	spin_lock_bh(&st->lock);
 	for (i = 0; i < n; i++)
@@ -4978,7 +4999,7 @@ static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
 	swap(g->ceps, eps);	/* out: frees the old set */
 	g->nr_ceps = n;
 	WRITE_ONCE(g->hw_gen, gen);
-	if (g->left == left)
+	if (READ_ONCE(st->ingress) || g->left == left)
 		smp_store_release(&g->hw_active, true);
 	spin_unlock_bh(&st->lock);
 	goto out;
@@ -5238,6 +5259,135 @@ static void amt_ofl_state_free(struct amt_dev *amt)
 	amt->offload_st = NULL;
 }
 
+/* Cutover, per packet; exactly one actor handles each one.
+ *
+ *   relay data packet (inner IP; Ethernet pulled)
+ *    |
+ *    +-- bound, admitting, desired state in sync, and (family, group)
+ *    |   hw_active?  no --> software fan-out
+ *    |
+ *    +-- TX path: ops->xmit(grp, hw_gen)
+ *    |     CONSUMED --> the provider handled every copy and owns the skb
+ *    |     FALLBACK --> skb untouched --> software fan-out
+ *    |     BUSY     --> skb untouched --> software fan-out, counted
+ *    |                  (never NETDEV_TX_BUSY: the relay is IFF_NO_QUEUE)
+ *    |
+ *    +-- ingress path: a punted copy of an hw_active group reaching here
+ *          is dropped and counted; one that took the "no" branch is
+ *          forwarded in software. A provider cannot tell which (amt
+ *          publishes after group_commit returns, and withdraws before
+ *          the callback on a reset, park or teardown and with no
+ *          callback on desync), so it must never punt a packet it
+ *          replicates.
+ *
+ * hw_gen is read once per packet. The reconciler writes it after
+ * group_commit returns 0, before it publishes hw_active, and also when a
+ * member left meanwhile and it withholds hw_active. So a packet already
+ * past the hw_active test may carry that new generation, and one that
+ * read hw_gen earlier any older committed one; the provider serves or
+ * declines it.
+ *
+ * Returns true when the skb is gone (consumed, or a punted copy dropped);
+ * false sends it to the software fan-out untouched.
+ */
+static bool amt_offload_tx(struct amt_dev *amt, struct sk_buff *skb,
+			   const union amt_addr *group, bool v6)
+{
+	struct amt_offload_group_key key;
+	enum amt_offload_result res;
+	struct amt_offload_bound *b;
+	struct amt_ofl_grp *g;
+	bool gone = false;
+
+	rcu_read_lock();
+	b = amt_offload_tx_admit(amt);
+	if (!b || READ_ONCE(amt->offload_st->desync))
+		goto out;
+	amt_ofl_grp_key(&key, group, v6);
+	g = amt_ofl_grp_find(amt->offload_st, &key);
+	/* Pairs with the release in amt_ofl_grp_program(). */
+	if (!g || !smp_load_acquire(&g->hw_active))
+		goto out;
+
+	if (b->path == AMT_OFFLOAD_PATH_INGRESS) {
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_PUNT_DROPPED);
+		consume_skb(skb);
+		gone = true;
+		goto out;
+	}
+
+	res = b->ops->xmit(&b->pub, g->hw, skb, READ_ONCE(g->hw_gen));
+	switch (res) {
+	case AMT_OFFLOAD_CONSUMED:
+		gone = true;
+		break;
+	case AMT_OFFLOAD_BUSY:
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_BUSY_FALLBACK);
+		break;
+	case AMT_OFFLOAD_FALLBACK:
+		break;
+	default:
+		WARN_ONCE(1, "amt: offload xmit returned %d\n", res);
+		break;
+	}
+out:
+	rcu_read_unlock();
+	return gone;
+}
+
+/* One gateway's copy for a simulated provider, built and sent exactly
+ * as amt_send_multicast_data() builds and sends a software copy: the
+ * same copy, AMT Multicast Data header, UDP ports, TOS and hop limit,
+ * through amt_udp_xmit(). Called from ops->xmit, in softirq context
+ * under rcu_read_lock().
+ */
+int amt_offload_encap_xmit(struct amt_offload_binding *b,
+			   const struct amt_offload_endpoint_key *key,
+			   struct sk_buff *oskb)
+{
+	struct amt_dev *amt = netdev_priv(b->dev);
+	struct amt_header_mcast_data *amtmd;
+	bool v6 = ip_hdr(oskb)->version == 6;
+	union amt_addr daddr = {0,};
+	struct socket *sock;
+	struct sk_buff *skb;
+
+	sock = rcu_dereference_bh(amt->sock);
+	if (!sock)
+		return -ENETDOWN;
+	if (key->family != (amt_v6(amt) ? AF_INET6 : AF_INET))
+		return -EAFNOSUPPORT;
+
+	skb = skb_copy_expand(oskb, sizeof(*amtmd) + amt_ip_hlen(amt) +
+			      sizeof(struct udphdr), 0, GFP_ATOMIC);
+	if (!skb)
+		goto drop;
+
+	skb_reset_inner_headers(skb);
+	amtmd = skb_push(skb, sizeof(*amtmd));
+	amtmd->version = 0;
+	amtmd->reserved = 0;
+	amtmd->type = AMT_MSG_MULTICAST_DATA;
+	skb_set_inner_protocol(skb, htons(v6 ? ETH_P_IPV6 : ETH_P_IP));
+
+#if IS_ENABLED(CONFIG_IPV6)
+	if (key->family == AF_INET6)
+		daddr.ip6 = key->addr.ip6;
+	else
+#endif
+		daddr.ip4 = key->addr.ip4;
+	if (amt_udp_xmit(amt, sock->sk, skb, &daddr, amt->relay_port,
+			 key->port, 0))
+		goto free;
+	return 0;
+free:
+	kfree_skb(skb);
+drop:
+	DEV_STATS_INC(amt->dev, tx_dropped);
+	return -ENETUNREACH;
+}
+EXPORT_SYMBOL_GPL(amt_offload_encap_xmit);
+
 static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
 {
 	if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
@@ -5285,9 +5435,27 @@ static int amt_offload_bind(struct amt_dev *amt,
 		kfree(b);
 		return -EOPNOTSUPP;
 	}
+	/* TX when offered. Ingress only for a required relay: a packet
+	 * consumed on ingress never reaches the host, so nothing on its
+	 * forwarding path sees it, and auto never chooses that.
+	 */
+	if ((b->caps.paths & BIT(AMT_OFFLOAD_PATH_TX)) && b->ops->xmit) {
+		b->path = AMT_OFFLOAD_PATH_TX;
+	} else if ((b->caps.paths & BIT(AMT_OFFLOAD_PATH_INGRESS)) &&
+		   amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
+		b->path = AMT_OFFLOAD_PATH_INGRESS;
+	} else {
+		b->ops->unbind(&b->pub);
+		amt_offload_count(amt, AMT_OFFLOAD_CNT_PATH_UNSUPPORTED);
+		kfree(b);
+		return -EOPNOTSUPP;
+	}
 
 	b->admitting = true;
 	mutex_lock(&amt->offload_st->mutex);
+	/* Before any group can hold hw under this binding. */
+	WRITE_ONCE(amt->offload_st->ingress,
+		   b->path == AMT_OFFLOAD_PATH_INGRESS);
 	rcu_assign_pointer(amt->offload, b);
 	/* Replay desired state: every group is programmed afresh. */
 	amt_ofl_requeue_all(amt->offload_st);
@@ -5506,6 +5674,9 @@ AMT_OFFLOAD_CNT_ATTR(sw_remainder, AMT_OFFLOAD_CNT_SW_REMAINDER);
 AMT_OFFLOAD_CNT_ATTR(resets, AMT_OFFLOAD_CNT_RESET);
 AMT_OFFLOAD_CNT_ATTR(stale_completions, AMT_OFFLOAD_CNT_STALE);
 AMT_OFFLOAD_CNT_ATTR(desync, AMT_OFFLOAD_CNT_DESYNC);
+AMT_OFFLOAD_CNT_ATTR(path_unsupported, AMT_OFFLOAD_CNT_PATH_UNSUPPORTED);
+AMT_OFFLOAD_CNT_ATTR(busy_fallback, AMT_OFFLOAD_CNT_BUSY_FALLBACK);
+AMT_OFFLOAD_CNT_ATTR(punt_dropped, AMT_OFFLOAD_CNT_PUNT_DROPPED);
 
 static struct attribute *amt_offload_attrs[] = {
 	&dev_attr_offload_mode.attr,
@@ -5521,6 +5692,9 @@ static struct attribute *amt_offload_attrs[] = {
 	&dev_attr_offload_resets.attr,
 	&dev_attr_offload_stale_completions.attr,
 	&dev_attr_offload_desync.attr,
+	&dev_attr_offload_path_unsupported.attr,
+	&dev_attr_offload_busy_fallback.attr,
+	&dev_attr_offload_punt_dropped.attr,
 	NULL,
 };
 
diff --git a/include/net/amt.h b/include/net/amt.h
index 0c746e4be..0b57c8604 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -389,7 +389,9 @@ struct amt_offload_state;
 
 /* Per-relay offload counters, exported under offload/ in sysfs. */
 enum amt_offload_counter {
-	/* bind() succeeded and the caps covered the outer family */
+	/* bind() succeeded, and the caps covered the outer family and offered
+	 * a data path this relay may use
+	 */
 	AMT_OFFLOAD_CNT_BIND,
 	/* ... of which: bound when a provider registered on the stream_dev
 	 * this relay already uses
@@ -418,6 +420,15 @@ enum amt_offload_counter {
 	AMT_OFFLOAD_CNT_STALE,
 	/* desired state lost an update (allocation failure); software only */
 	AMT_OFFLOAD_CNT_DESYNC,
+	/* no data path this relay may use: the caps offered none, or TX
+	 * without ops->xmit (ingress counts only in required mode); it stayed
+	 * unbound
+	 */
+	AMT_OFFLOAD_CNT_PATH_UNSUPPORTED,
+	/* xmit returned BUSY: one synchronous software pass ran instead */
+	AMT_OFFLOAD_CNT_BUSY_FALLBACK,
+	/* ingress: a punted copy of a group the provider replicates, dropped */
+	AMT_OFFLOAD_CNT_PUNT_DROPPED,
 	AMT_OFFLOAD_CNT_MAX,
 };
 
diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
index b5c119851..99f78699f 100644
--- a/include/net/amt_offload.h
+++ b/include/net/amt_offload.h
@@ -42,8 +42,16 @@ struct sk_buff;
  *	group without a callback when a member leaves, until it publishes the
  *	recommit. Every other packet goes straight to software.
  * @AMT_OFFLOAD_PATH_INGRESS: the provider replicates on ingress and the
- *	host only sees punted packets. Semantics only until a real provider
- *	exists; no simulated provider implements it.
+ *	host only sees punted packets. From &amt_offload_ops.group_commit
+ *	until &amt_offload_ops.group_disable or &amt_offload_ops.flush the
+ *	provider replicates every packet of the group, and it must never also
+ *	punt one. amt drops, and counts, a punted copy of a group only while
+ *	the group's commit is published: from some time after group_commit
+ *	returns until amt withdraws it, which it does before any group_disable
+ *	or flush and, with no callback, once its desired state is lost. A
+ *	punted copy outside that window is forwarded in software as well.
+ *	Semantics only until a real provider exists; no simulated provider
+ *	implements it, and only a relay in required mode binds on it.
  */
 enum amt_offload_path {
 	AMT_OFFLOAD_PATH_TX,
@@ -203,7 +211,12 @@ struct amt_offload_stats {
  * @group_disable: stop forwarding @grp and free it. amt has already
  *	stopped submitting packets for @grp and waited out @xmit callers.
  * @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
- *	&enum amt_offload_result. Softirq context, under rcu_read_lock().
+ *	&enum amt_offload_result. @skb is the relay data packet with its
+ *	data at the inner IP header. On %AMT_OFFLOAD_CONSUMED the provider
+ *	owns @skb and must have handled every copy under @gen alone: each one
+ *	handed to the stack, or dropped and counted (see
+ *	%AMT_OFFLOAD_CONSUMED). Otherwise it must not have sent any copy, nor
+ *	modified or freed @skb. Softirq context, under rcu_read_lock().
  * @flush: drop every endpoint and group (reset after a timeout). amt then
  *	replays from its own desired state under a new epoch, except when
  *	the op that overran was a @group_disable or @endpoint_del of amt
@@ -318,4 +331,23 @@ int amt_offload_register(struct net_device *lower,
  */
 void amt_offload_unregister(struct net_device *lower);
 
+/**
+ * amt_offload_encap_xmit - emit one gateway's copy of a relay data packet
+ * @b: the binding @skb was handed to the provider under.
+ * @key: the gateway endpoint, as amt passed it to endpoint_add.
+ * @skb: the packet given to &amt_offload_ops.xmit; not consumed.
+ *
+ * For simulated providers: encapsulates a copy of @skb as AMT Multicast
+ * Data to @key and transmits it on the relay's stream device, byte for
+ * byte as amt's own software fan-out would. Which endpoints get a copy
+ * is the provider's decision, made from its own tables. Call only from
+ * &amt_offload_ops.xmit.
+ *
+ * Return: 0 if the copy was handed to the stack, else a negative errno
+ * (counted in the relay's tx_dropped when the copy or route failed).
+ */
+int amt_offload_encap_xmit(struct amt_offload_binding *b,
+			   const struct amt_offload_endpoint_key *key,
+			   struct sk_buff *skb);
+
 #endif /* _NET_AMT_OFFLOAD_H_ */
-- 
2.43.0

[RFC PATCH net-next 5/9] amt: add IFLA_AMT_OFFLOAD_MODE

[Omar Ramadan]

software (default), auto or required, after
IFLA_AMT_MAX_TUNNELS_PER_SOURCE. An unknown value, or a non-software
mode on a gateway, is refused with extack. required with no usable
provider on the stream device refuses the link with -ENODEV and extack.
Emitted by fill_info.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/amt.c        | 25 ++++++++++++++++++++++++-
 include/uapi/linux/amt.h |  7 +++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 377660e55..58e9da443 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -5719,6 +5719,7 @@ static const struct nla_policy amt_policy[IFLA_AMT_MAX + 1] = {
 	[IFLA_AMT_NUM_QUEUES]	= NLA_POLICY_MAX(NLA_U32, AMT_MAX_QUEUES),
 	[IFLA_AMT_DISCOVERY_IP6] = NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
 	[IFLA_AMT_REMOTE_IP6]	= NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
+	[IFLA_AMT_OFFLOAD_MODE]	= { .type = NLA_U32 },
 };
 
 static int amt_validate(struct nlattr *tb[], struct nlattr *data[],
@@ -5745,6 +5746,22 @@ static int amt_validate(struct nlattr *tb[], struct nlattr *data[],
 		return -EINVAL;
 	}
 
+	if (data[IFLA_AMT_OFFLOAD_MODE]) {
+		u32 offload = nla_get_u32(data[IFLA_AMT_OFFLOAD_MODE]);
+
+		if (offload > AMT_OFFLOAD_MODE_MAX) {
+			NL_SET_ERR_MSG_ATTR(extack, data[IFLA_AMT_OFFLOAD_MODE],
+					    "Offload mode is not valid");
+			return -EINVAL;
+		}
+		if (offload != AMT_OFFLOAD_MODE_SOFTWARE &&
+		    nla_get_u32(data[IFLA_AMT_MODE]) != AMT_MODE_RELAY) {
+			NL_SET_ERR_MSG_ATTR(extack, data[IFLA_AMT_OFFLOAD_MODE],
+					    "Offload is only valid in relay mode");
+			return -EINVAL;
+		}
+	}
+
 	if (!data[IFLA_AMT_LOCAL_IP] && !data[IFLA_AMT_LOCAL_IP6]) {
 		NL_SET_ERR_MSG_MOD(extack,
 				   "Local IPv4 or IPv6 attribute is required");
@@ -5802,6 +5819,9 @@ static int amt_newlink(struct net_device *dev,
 
 	amt->net = link_net;
 	amt->mode = nla_get_u32(data[IFLA_AMT_MODE]);
+	amt->offload_mode = data[IFLA_AMT_OFFLOAD_MODE] ?
+			    nla_get_u32(data[IFLA_AMT_OFFLOAD_MODE]) :
+			    AMT_OFFLOAD_MODE_SOFTWARE;
 
 	if (data[IFLA_AMT_MAX_TUNNELS] &&
 	    nla_get_u32(data[IFLA_AMT_MAX_TUNNELS]))
@@ -5998,7 +6018,8 @@ static size_t amt_get_size(const struct net_device *dev)
 	       nla_total_size(sizeof(struct iphdr)) + /* IFLA_AMT_LOCAL_IP */
 	       nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_LOCAL_IP6 */
 	       nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_DISCOVERY_IP6 */
-	       nla_total_size(sizeof(struct in6_addr)); /* IFLA_AMT_REMOTE_IP6 */
+	       nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_REMOTE_IP6 */
+	       nla_total_size(sizeof(__u32)); /* IFLA_AMT_OFFLOAD_MODE */
 }
 
 static int amt_fill_info(struct sk_buff *skb, const struct net_device *dev)
@@ -6068,6 +6089,8 @@ static int amt_fill_info(struct sk_buff *skb, const struct net_device *dev)
 		goto nla_put_failure;
 	if (nla_put_u32(skb, IFLA_AMT_NUM_QUEUES, dev->real_num_tx_queues))
 		goto nla_put_failure;
+	if (nla_put_u32(skb, IFLA_AMT_OFFLOAD_MODE, amt->offload_mode))
+		goto nla_put_failure;
 
 	return 0;
 
diff --git a/include/uapi/linux/amt.h b/include/uapi/linux/amt.h
index de1ae1929..c67a1f763 100644
--- a/include/uapi/linux/amt.h
+++ b/include/uapi/linux/amt.h
@@ -122,6 +122,13 @@ enum {
 	 * AMT_MAX_TUNNELS_PER_SOURCE; raise it on a relay that fronts CGNAT.
 	 */
 	IFLA_AMT_MAX_TUNNELS_PER_SOURCE,
+	/* Relay mode: enum ifla_amt_offload_mode, default
+	 * AMT_OFFLOAD_MODE_SOFTWARE. Replication offload to a provider
+	 * registered on the IFLA_AMT_LINK device (include/net/amt_offload.h).
+	 * Required mode refuses creation with -ENODEV when no usable provider
+	 * is bound. Gateway mode accepts software only.
+	 */
+	IFLA_AMT_OFFLOAD_MODE,
 	__IFLA_AMT_MAX,
 };
 
-- 
2.43.0

[RFC PATCH net-next 6/9] amt: KUnit cases for the offload contract

[Omar Ramadan]

A relay fixture (amt_test_relay_alloc(), amt_test_tunnel_add()) builds a
v4 relay without registering or opening the device, and links tunnels on
it as amt_request_handler() admits them, listed and counted under
amt->lock with gc_wq armed for one GMI, but without the admission
bounds, the siphash key or the Request/Query exchange. An in-file fake
provider logs every teardown callback in order.

Registry, lifecycle and mode: register-when-bound,
caps-family-unsupported (and bind failure), unregister-mid-fanout
(stream_dev unregistered with groups programmed: order BGGEEU, no
fan-out admitted at any step, binding published until unbind),
provider-unregister ordering with required-mode carrier loss and
rebind, IFLA_AMT_OFFLOAD_MODE validation, and required refusal counted.

Reconciler and cutover, twenty cases: the fake also keeps its own
endpoint and group tables and can refuse with -ENOSPC, fail a commit,
overrun op_timeout_ms or answer xmit with any result. The reconciler
work is disabled in these cases and ticks are driven by hand, so a
static stub on amt_ofl_op_overran() reaches every timed op: the fake
records whether each op stalled, and only such an op overruns, never one
the scheduler happened to delay past a short op_timeout_ms.

  - members follow gnodes: a second inner host only counts, the last
    gnode removes the member, unlink removes it at once
  - churn coalesces into one update and one commit at the last
    generation; a tick stops starting groups once its budget is spent
  - a refused group is served by software, not retried on its own
    member changes, and retried past the low watermark
  - an op overrunning op_timeout_ms flushes, bumps the epoch, ignores
    the late result and replays; with groups still queued, the tick
    ends at the reset
  - a teardown whose release ops overrun op_timeout_ms makes one of
    them, then flushes, with groups programmed or with only endpoints
    (the group parked); every endpoint is still collected afterwards.
    The reconciler's gone and endpoint-GC loops reset on an
    overrunning release op
  - under churn in one group that spends every tick's budget, the
    endpoint that left another group is still deleted within three
    ticks, and the churning endpoint, which the last commit names, is
    never deleted
  - admission lost inside an endpoint_add, a group_update or an
    endpoint_del (a teardown racing the tick): no further provider op,
    and the tick ends with the remaining groups queued
  - endpoint_add is idempotent and a shared endpoint is added once
  - a failed commit disables the group, keeps its endpoints, and a new
    binding retries it; a leave followed by a failed recommit parks the
    group and disables its provider object, after a grace period (a
    static stub on the reconciler's synchronize_net() logs it)
  - the publish gate: a leave landing inside group_commit withholds
    that commit's publication and software serves the group, and the
    departed member's endpoint stays programmed; a join landing there
    does not withhold it
  - a group emptied and rejoined within one tick: the fake refuses a
    second live group per key, and the rejoined group still commits,
    after a grace period for the old group's disable. If that disable
    overruns op_timeout_ms, the provider is flushed once and the
    rejoined group waits for the next tick
  - CONSUMED/FALLBACK/BUSY ownership, unbound and uncommitted groups
    stay in software, the generation rule, and a member removal
    withdrawing the group from hardware at once
  - the last member leaving while the reconciler allocates (static
    stubs on its allocator and its free keep a ledger of live arrays):
    the retry frees nothing twice
  - both reconciler tables are indexed with the device's hash_seed and
    sized from max_tunnels at attach
  - ingress: a punted copy of a committed group is dropped and counted,
    and still is after a member leaves, including one landing inside
    the group's first commit, whose departed endpoint stays
    programmed; only a required relay binds on ingress

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/amt.c | 2062 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 2062 insertions(+)

diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 58e9da443..bafe21302 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -6201,6 +6201,7 @@ static struct notifier_block amt_notifier_block __read_mostly = {
 
 #if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
 #include <kunit/test.h>
+#include <linux/delay.h>
 
 /*
  * KUnit coverage for relay-mode upstream membership tracking. The cases
@@ -6513,10 +6514,2071 @@ static void amt_ex_transition_test(struct kunit *test)
 	free_netdev(test_dev);
 }
 
+#define AMT_TEST_GW	htonl(0xc6336401)	/* 198.51.100.1, one NAT */
+#define AMT_TEST_HOST	htonl(0x0a000002)	/* 10.0.0.2, inner host */
+#define AMT_TEST_GROUP	htonl(0xe801013c)	/* 232.1.1.60 */
+#define AMT_TEST_NONCE	htonl(0x5eed1234)
+#define AMT_TEST_MAC	0xa1b2c3d4e5f6ULL
+
+/* A v4 relay-mode amt_dev as amt_dev_init leaves it, minus the netdev
+ * registration. Nothing is sent: the device is never opened, and the
+ * upstream reconciler sees upstream_active == false.
+ */
+static struct amt_dev *amt_test_relay_alloc(struct kunit *test,
+					    const char *name)
+{
+	struct net_device *dev;
+	struct amt_dev *amt;
+
+	dev = alloc_netdev(sizeof(*amt), name, NET_NAME_UNKNOWN,
+			   amt_link_setup);
+	KUNIT_ASSERT_NOT_NULL(test, dev);
+	amt = netdev_priv(dev);
+	amt->dev = dev;
+	amt_upstream_init(amt);
+	spin_lock_init(&amt->lock);
+	INIT_LIST_HEAD(&amt->tunnel_list);
+	amt->mode = AMT_MODE_RELAY;
+	amt->hash_buckets = 8;
+	amt->max_groups = AMT_MAX_GROUP;
+	amt->max_tunnels = AMT_MAX_TUNNELS;
+	amt->max_tunnels_per_source = AMT_MAX_TUNNELS_PER_SOURCE;
+	amt->qrv = 2;
+	amt->qi = AMT_INIT_QUERY_INTERVAL;
+	amt->qri = 10;
+	KUNIT_ASSERT_EQ(test, rhltable_init(&amt->groups_rhl,
+					    &amt_gnode_params), 0);
+	return amt;
+}
+
+static void amt_test_relay_free(struct amt_dev *amt)
+{
+	cancel_delayed_work_sync(&amt->upstream_work);
+	__amt_source_gc_work();
+	rhltable_destroy(&amt->groups_rhl);
+	free_netdev(amt->dev);
+}
+
+/* A tunnel linked on @amt as amt_request_handler() admits one, holding
+ * the nonce and MAC an Update must echo. The admission bounds, the
+ * siphash key and the Request/Query exchange are skipped.
+ */
+static struct amt_tunnel_list *
+amt_test_tunnel_add(struct kunit *test, struct amt_dev *amt, __be32 addr,
+		    __be16 port, __be32 nonce, u64 mac)
+{
+	struct amt_tunnel_list *tunnel;
+
+	tunnel = kzalloc_obj(*tunnel, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, tunnel);
+	tunnel->amt = amt;
+	spin_lock_init(&tunnel->lock);
+	INIT_LIST_HEAD(&tunnel->groups);
+	tunnel->addr.ip4 = addr;
+	tunnel->source_port = port;
+	tunnel->nonce = nonce;
+	tunnel->mac = mac;
+	INIT_DELAYED_WORK(&tunnel->gc_wq, amt_tunnel_expire);
+
+	spin_lock_bh(&amt->lock);
+	list_add_tail_rcu(&tunnel->list, &amt->tunnel_list);
+	amt->nr_tunnels++;
+	mod_delayed_work(amt_wq, &tunnel->gc_wq,
+			 msecs_to_jiffies(amt_gmi(amt)));
+	spin_unlock_bh(&amt->lock);
+	return tunnel;
+}
+
+/* Offload contract: an in-file fake provider. It logs every
+ * teardown-side callback in order (B bind, G group_disable, E
+ * endpoint_del, U unbind), so registration, binding and the documented
+ * teardown order are asserted without a real provider, and keeps its
+ * own endpoint and group tables so the reconciler's programming can be
+ * checked against what the provider ended up holding.
+ */
+#define AMT_FAKE_LOG_MAX	32
+#define AMT_FAKE_SLOTS		16
+
+struct amt_fake_obj {
+	bool				used;
+	struct amt_offload_endpoint_key	key;	/* endpoints */
+	struct amt_offload_group_key	gkey;	/* groups */
+	u64				gen;	/* groups: staged */
+	u64				committed;
+	u32				nr;
+};
+
+static struct {
+	char			log[AMT_FAKE_LOG_MAX];
+	int			n;
+	u32			families;
+	u32			paths;
+	u32			op_timeout_ms;
+	int			bind_err;
+	/* Relay whose admission state the teardown callbacks sample. */
+	struct amt_dev		*amt;
+	bool			admitted_in_teardown;
+	bool			unpublished_in_teardown;
+	struct amt_fake_obj	eps[AMT_FAKE_SLOTS];
+	struct amt_fake_obj	grps[AMT_FAKE_SLOTS];
+	/* endpoint_add refuses with -ENOSPC once this many are held */
+	int			ep_cap;
+	int			commit_err;
+	/* run once from inside the next group_commit, then cleared */
+	void			(*commit_hook)(void);
+	/* the next endpoint_add overruns op_timeout_ms, then succeeds */
+	bool			add_stall;
+	/* every group_disable and endpoint_del overruns op_timeout_ms */
+	bool			release_stall;
+	/* whether the op that returned last overran, read by the overrun
+	 * stub; every op amt times sets it on entry, or it inherits the
+	 * previous op's verdict
+	 */
+	bool			stalled;
+	/* clear b->admitting from inside the next op named here ('A'
+	 * endpoint_add, 'U' group_update, 'E' endpoint_del), as a teardown
+	 * racing the reconciler would; then cleared
+	 */
+	char			drop_admit;
+	/* group_disable calls on a NULL handle; the contract allows none */
+	int			null_disables;
+	int			ep_adds;
+	/* first uses refused: the key's earlier group was still live */
+	int			dup_grps;
+	int			updates;
+	int			commits;
+	int			flushes;
+	enum amt_offload_result	xmit_res;
+	int			xmits;
+	u64			xmit_gen;
+} amt_fake;
+
+static void amt_fake_reset(u32 families)
+{
+	memset(&amt_fake, 0, sizeof(amt_fake));
+	amt_fake.families = families;
+	amt_fake.paths = BIT(AMT_OFFLOAD_PATH_TX);
+}
+
+static int amt_fake_live(const struct amt_fake_obj *objs)
+{
+	int i, n = 0;
+
+	for (i = 0; i < AMT_FAKE_SLOTS; i++)
+		n += objs[i].used;
+	return n;
+}
+
+static struct amt_fake_obj *amt_fake_slot(struct amt_fake_obj *objs)
+{
+	int i;
+
+	for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+		if (!objs[i].used) {
+			memset(&objs[i], 0, sizeof(objs[i]));
+			objs[i].used = true;
+			return &objs[i];
+		}
+	}
+	return NULL;
+}
+
+/* What a fan-out racing the teardown would see. */
+static void amt_fake_log(char c)
+{
+	if (amt_fake.n < AMT_FAKE_LOG_MAX - 1)
+		amt_fake.log[amt_fake.n++] = c;
+	if (!amt_fake.amt || c == 'B')
+		return;
+	rcu_read_lock();
+	if (amt_offload_tx_admit(amt_fake.amt))
+		amt_fake.admitted_in_teardown = true;
+	if (!rcu_access_pointer(amt_fake.amt->offload))
+		amt_fake.unpublished_in_teardown = true;
+	rcu_read_unlock();
+}
+
+static void amt_fake_drop_admit(struct amt_offload_binding *b, char op)
+{
+	if (amt_fake.drop_admit != op)
+		return;
+	amt_fake.drop_admit = 0;
+	WRITE_ONCE(container_of(b, struct amt_offload_bound, pub)->admitting,
+		   false);
+}
+
+static int amt_fake_bind(struct amt_offload_binding *b,
+			 struct amt_offload_caps *caps)
+{
+	amt_fake_log('B');
+	if (amt_fake.bind_err)
+		return amt_fake.bind_err;
+	caps->families = amt_fake.families;
+	caps->paths = amt_fake.paths;
+	caps->op_timeout_ms = amt_fake.op_timeout_ms;
+	return 0;
+}
+
+static void amt_fake_unbind(struct amt_offload_binding *b)
+{
+	amt_fake_log('U');
+}
+
+/* Idempotent on an identical key, as the contract requires. */
+static int amt_fake_endpoint_add(struct amt_offload_binding *b,
+				 const struct amt_offload_endpoint_key *key,
+				 struct amt_offload_endpoint **ep)
+{
+	struct amt_fake_obj *o;
+	int i;
+
+	amt_fake_drop_admit(b, 'A');
+	amt_fake.stalled = amt_fake.add_stall;
+	amt_fake.add_stall = false;
+	for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+		o = &amt_fake.eps[i];
+		if (o->used && !memcmp(&o->key, key, sizeof(*key))) {
+			*ep = (struct amt_offload_endpoint *)o;
+			return 0;
+		}
+	}
+	if (amt_fake.ep_cap && amt_fake_live(amt_fake.eps) >= amt_fake.ep_cap)
+		return -ENOSPC;
+	o = amt_fake_slot(amt_fake.eps);
+	if (!o)
+		return -ENOSPC;
+	o->key = *key;
+	amt_fake.ep_adds++;
+	*ep = (struct amt_offload_endpoint *)o;
+	return 0;
+}
+
+static void amt_fake_endpoint_del(struct amt_offload_binding *b,
+				  struct amt_offload_endpoint *ep)
+{
+	amt_fake.stalled = amt_fake.release_stall;
+	amt_fake_drop_admit(b, 'E');
+	((struct amt_fake_obj *)ep)->used = false;
+	amt_fake_log('E');
+}
+
+static int amt_fake_group_update(struct amt_offload_binding *b,
+				 const struct amt_offload_group_key *key,
+				 struct amt_offload_endpoint *const *members,
+				 unsigned int nr_members, u64 gen,
+				 struct amt_offload_group **grp)
+{
+	struct amt_fake_obj *o = (struct amt_fake_obj *)*grp;
+	int i;
+
+	amt_fake.stalled = false;
+	amt_fake_drop_admit(b, 'U');
+	if (!o) {
+		/* One group per key, as amt promises: refuse a second. */
+		for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+			o = &amt_fake.grps[i];
+			if (o->used && !memcmp(&o->gkey, key, sizeof(*key))) {
+				amt_fake.dup_grps++;
+				return -EEXIST;
+			}
+		}
+		o = amt_fake_slot(amt_fake.grps);
+		if (!o)
+			return -ENOSPC;
+		o->gkey = *key;
+		*grp = (struct amt_offload_group *)o;
+	}
+	o->gen = gen;
+	o->nr = nr_members;
+	amt_fake.updates++;
+	return 0;
+}
+
+static int amt_fake_group_commit(struct amt_offload_binding *b,
+				 struct amt_offload_group *grp, u64 gen)
+{
+	void (*hook)(void) = amt_fake.commit_hook;
+
+	amt_fake.stalled = false;
+	if (hook) {
+		amt_fake.commit_hook = NULL;
+		hook();
+	}
+	if (amt_fake.commit_err)
+		return amt_fake.commit_err;
+	((struct amt_fake_obj *)grp)->committed = gen;
+	amt_fake.commits++;
+	return 0;
+}
+
+static void amt_fake_group_disable(struct amt_offload_binding *b,
+				   struct amt_offload_group *grp)
+{
+	amt_fake.stalled = grp && amt_fake.release_stall;
+	if (!grp) {
+		amt_fake.null_disables++;
+		return;
+	}
+	((struct amt_fake_obj *)grp)->used = false;
+	amt_fake_log('G');
+}
+
+static void amt_fake_flush(struct amt_offload_binding *b)
+{
+	int i;
+
+	for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+		amt_fake.eps[i].used = false;
+		amt_fake.grps[i].used = false;
+	}
+	amt_fake.flushes++;
+}
+
+static enum amt_offload_result
+amt_fake_xmit(struct amt_offload_binding *b, struct amt_offload_group *grp,
+	      struct sk_buff *skb, u64 gen)
+{
+	amt_fake.xmits++;
+	amt_fake.xmit_gen = gen;
+	if (amt_fake.xmit_res == AMT_OFFLOAD_CONSUMED)
+		consume_skb(skb);
+	return amt_fake.xmit_res;
+}
+
+static const struct amt_offload_ops amt_fake_ops = {
+	.bind		= amt_fake_bind,
+	.unbind		= amt_fake_unbind,
+	.endpoint_add	= amt_fake_endpoint_add,
+	.endpoint_del	= amt_fake_endpoint_del,
+	.group_update	= amt_fake_group_update,
+	.group_commit	= amt_fake_group_commit,
+	.group_disable	= amt_fake_group_disable,
+	.flush		= amt_fake_flush,
+	.xmit		= amt_fake_xmit,
+};
+
+static struct net_device *amt_test_lower_alloc(struct kunit *test)
+{
+	struct net_device *lower;
+
+	lower = alloc_netdev(0, "amtlow%d", NET_NAME_UNKNOWN, ether_setup);
+	KUNIT_ASSERT_NOT_NULL(test, lower);
+	return lower;
+}
+
+/* A relay on @lower as amt_newlink leaves it, minus the netlink half.
+ * Caller holds RTNL.
+ */
+static struct amt_dev *amt_test_offload_relay(struct kunit *test,
+					      const char *name,
+					      struct net_device *lower,
+					      u32 mode)
+{
+	struct amt_dev *amt = amt_test_relay_alloc(test, name);
+
+	amt->stream_dev = lower;
+	amt->offload_mode = mode;
+	return amt;
+}
+
+static void amt_test_offload_free(struct amt_dev *amt)
+{
+	rtnl_lock();
+	amt_offload_detach(amt);
+	rtnl_unlock();
+	amt_ofl_state_free(amt);
+	amt_test_relay_free(amt);
+}
+
+/* A tunnel the offload hooks can name: endpoint identity only. */
+static struct amt_tunnel_list *amt_test_ofl_tunnel(struct kunit *test,
+						   __be32 addr, __be16 port)
+{
+	struct amt_tunnel_list *t = kunit_kzalloc(test, sizeof(*t),
+						  GFP_KERNEL);
+
+	KUNIT_ASSERT_NOT_NULL(test, t);
+	t->addr.ip4 = addr;
+	t->source_port = port;
+	return t;
+}
+
+static union amt_addr amt_test_ofl_group(int i)
+{
+	union amt_addr g = {};
+
+	g.ip4 = htonl(ntohl(AMT_TEST_GROUP) + i);
+	return g;
+}
+
+/* Run reconciler ticks until it reports no work left; returns the count. */
+static int amt_test_ofl_drain(struct amt_dev *amt)
+{
+	int ticks = 0;
+
+	while (ticks < 1000 && amt_ofl_reconcile(amt))
+		ticks++;
+	return ticks + 1;
+}
+
+static int amt_test_offload_cnt(struct amt_dev *amt,
+				enum amt_offload_counter cnt)
+{
+	return atomic_read(&amt->offload_cnt[cnt]);
+}
+
+/* Two groups and two endpoints the teardown must give back, programmed
+ * through desired state and the reconciler. RTNL held.
+ */
+static void amt_test_offload_seed(struct kunit *test, struct amt_dev *amt)
+{
+	union amt_addr grp;
+	int i;
+
+	KUNIT_ASSERT_NOT_NULL(test, rtnl_dereference(amt->offload));
+	for (i = 0; i < 2; i++) {
+		grp = amt_test_ofl_group(i);
+		amt_ofl_member_add(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+							   htons(4000 + i)),
+				   &grp, false);
+	}
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+}
+
+static bool amt_test_offload_admits(struct amt_dev *amt)
+{
+	bool ok;
+
+	rcu_read_lock();
+	ok = !!amt_offload_tx_admit(amt);
+	rcu_read_unlock();
+	return ok;
+}
+
+static void amt_offload_register_when_bound_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt, *sw;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4 | AMT_OFFLOAD_FAMILY_V6);
+	amt = amt_test_offload_relay(test, "amtob%d", lower,
+				     AMT_OFFLOAD_MODE_AUTO);
+	sw = amt_test_offload_relay(test, "amtos%d", lower,
+				    AMT_OFFLOAD_MODE_SOFTWARE);
+
+	rtnl_lock();
+	/* No provider yet: auto attaches in software. */
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(sw, NULL), 0);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+
+	/* Registering on a stream_dev in use binds the auto relay at once. */
+	KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	KUNIT_EXPECT_NOT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(sw->offload));
+	KUNIT_EXPECT_TRUE(test, amt_test_offload_admits(amt));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_BIND), 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 1);
+	KUNIT_EXPECT_EQ(test, amt->offload_epoch, 1);
+	KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops),
+			-EEXIST);
+
+	amt_offload_unregister(lower);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 1);
+	KUNIT_EXPECT_EQ(test, amt->offload_epoch, 2);
+	amt_offload_unregister(lower);	/* no provider: a no-op */
+	rtnl_unlock();
+
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BU");
+	amt_test_offload_free(sw);
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+static void amt_offload_family_unsupported_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt, *amt2;
+
+	/* A v6-only provider and a v4 relay. */
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V6);
+	amt = amt_test_offload_relay(test, "amtof%d", lower,
+				     AMT_OFFLOAD_MODE_AUTO);
+	amt2 = amt_test_offload_relay(test, "amtoe%d", lower,
+				      AMT_OFFLOAD_MODE_AUTO);
+
+	rtnl_lock();
+	KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED), 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_BIND), 0);
+
+	/* A bind() error is a different count and never reaches unbind(). */
+	amt_fake.bind_err = -EIO;
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt2, NULL), 0);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt2->offload));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt2,
+			AMT_OFFLOAD_CNT_BIND_FAILED), 1);
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BUB");
+	amt_test_offload_free(amt2);
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* The stream_dev is unregistered with groups programmed. A fan-out
+ * racing the teardown must not be admitted at any step, while the
+ * binding stays published until after unbind().
+ */
+static void amt_offload_unregister_mid_fanout_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt = amt_test_offload_relay(test, "amtom%d", lower,
+				     AMT_OFFLOAD_MODE_AUTO);
+
+	rtnl_lock();
+	KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	KUNIT_ASSERT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 0);
+	KUNIT_EXPECT_TRUE(test, amt_test_offload_admits(amt));
+	amt_test_offload_seed(test, amt);
+	amt_fake.amt = amt;
+
+	amt_offload_lower_gone(lower);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_NULL(test, amt_offload_provider_find(lower));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_TEARDOWN_LOWER), 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 0);
+	rtnl_unlock();
+
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGGEEU");
+	KUNIT_EXPECT_FALSE(test, amt_fake.admitted_in_teardown);
+	KUNIT_EXPECT_FALSE(test, amt_fake.unpublished_in_teardown);
+	KUNIT_EXPECT_FALSE(test, amt_test_offload_admits(amt));
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* Same order when the provider leaves first. A required relay drops
+ * carrier, counts it, and comes back when a provider registers again.
+ */
+static void amt_offload_provider_unregister_order_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt = amt_test_offload_relay(test, "amtop%d", lower,
+				     AMT_OFFLOAD_MODE_REQUIRED);
+
+	rtnl_lock();
+	KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	KUNIT_ASSERT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	amt_test_offload_seed(test, amt);
+	amt_fake.amt = amt;
+
+	amt_offload_unregister(lower);
+	KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_REQUIRED_UNBOUND), 1);
+	KUNIT_EXPECT_FALSE(test, netif_carrier_ok(amt->dev));
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGGEEU");
+	KUNIT_EXPECT_FALSE(test, amt_fake.admitted_in_teardown);
+
+	amt_fake.amt = NULL;
+	KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	KUNIT_EXPECT_NOT_NULL(test, rtnl_dereference(amt->offload));
+	KUNIT_EXPECT_TRUE(test, netif_carrier_ok(amt->dev));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 1);
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+struct amt_test_u32_attr {
+	struct nlattr	nla;
+	u32		val;
+};
+
+static struct nlattr *amt_test_u32(struct amt_test_u32_attr *a, int type,
+				   u32 val)
+{
+	a->nla.nla_len = nla_attr_size(sizeof(u32));
+	a->nla.nla_type = type;
+	a->val = val;
+	return &a->nla;
+}
+
+static int amt_test_validate_offload(u32 mode, u32 offload,
+				     struct netlink_ext_ack *extack)
+{
+	struct amt_test_u32_attr link, amode, local, off;
+	struct nlattr *data[IFLA_AMT_MAX + 1] = {};
+
+	data[IFLA_AMT_LINK] = amt_test_u32(&link, IFLA_AMT_LINK, 1);
+	data[IFLA_AMT_MODE] = amt_test_u32(&amode, IFLA_AMT_MODE, mode);
+	data[IFLA_AMT_LOCAL_IP] = amt_test_u32(&local, IFLA_AMT_LOCAL_IP,
+					       (__force u32)AMT_TEST_GW);
+	data[IFLA_AMT_OFFLOAD_MODE] = amt_test_u32(&off, IFLA_AMT_OFFLOAD_MODE,
+						   offload);
+	memset(extack, 0, sizeof(*extack));
+	return amt_validate(NULL, data, extack);
+}
+
+static void amt_offload_mode_validate_test(struct kunit *test)
+{
+	struct netlink_ext_ack extack;
+
+	KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+			AMT_OFFLOAD_MODE_SOFTWARE, &extack), 0);
+	KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+			AMT_OFFLOAD_MODE_AUTO, &extack), 0);
+	KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+			AMT_OFFLOAD_MODE_REQUIRED, &extack), 0);
+
+	KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+			AMT_OFFLOAD_MODE_MAX + 1, &extack), -EINVAL);
+	KUNIT_ASSERT_NOT_NULL(test, extack._msg);
+	KUNIT_EXPECT_STREQ(test, extack._msg, "Offload mode is not valid");
+
+	KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_GATEWAY,
+			AMT_OFFLOAD_MODE_AUTO, &extack), -EINVAL);
+	KUNIT_ASSERT_NOT_NULL(test, extack._msg);
+	KUNIT_EXPECT_STREQ(test, extack._msg,
+			   "Offload is only valid in relay mode");
+}
+
+static void amt_offload_required_refused_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct netlink_ext_ack extack = {};
+	struct amt_dev *amt;
+	int before;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V6);
+	amt = amt_test_offload_relay(test, "amtor%d", lower,
+				     AMT_OFFLOAD_MODE_REQUIRED);
+	before = atomic_read(&amt_offload_required_refused);
+
+	rtnl_lock();
+	/* No provider at all. */
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), -ENODEV);
+	KUNIT_EXPECT_NOT_NULL(test, extack._msg);
+	KUNIT_EXPECT_TRUE(test, list_empty(&amt->offload_node));
+	KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+			before + 1);
+
+	/* A provider that cannot carry this relay's outer family. */
+	KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	memset(&extack, 0, sizeof(extack));
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), -ENODEV);
+	KUNIT_EXPECT_NOT_NULL(test, extack._msg);
+	KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+			before + 2);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED), 1);
+
+	/* auto with the same provider is not a refusal. */
+	amt->offload_mode = AMT_OFFLOAD_MODE_AUTO;
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), 0);
+	KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+			before + 2);
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* Offload contract: desired state, reconciler and cutover, driven
+ * by hand -- the reconciler work is disabled and amt_ofl_reconcile()
+ * called directly, so every tick is the test's.
+ */
+static struct amt_dev *amt_test_ofl_attach(struct kunit *test,
+					   const char *name,
+					   struct net_device *lower, u32 mode)
+{
+	struct amt_dev *amt = amt_test_offload_relay(test, name, lower, mode);
+
+	rtnl_lock();
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	rtnl_unlock();
+	KUNIT_ASSERT_NOT_NULL(test, amt->offload_st);
+	disable_work_sync(&amt->offload_st->work);
+	return amt;
+}
+
+/* Whether a provider op overran op_timeout_ms is the fake's call, not the
+ * clock's: on a loaded machine a 10 ms timeout against ktime_get() also
+ * fires on ops that never stalled. The reconciler runs on the test thread
+ * here (its work is disabled), so the stub reaches every timed op.
+ */
+static bool amt_test_ofl_op_overran(struct amt_offload_bound *b, ktime_t t0)
+{
+	return b->caps.op_timeout_ms && amt_fake.stalled;
+}
+
+static void amt_test_ofl_register(struct kunit *test, struct net_device *lower)
+{
+	int err;
+
+	kunit_activate_static_stub(test, amt_ofl_op_overran,
+				   amt_test_ofl_op_overran);
+	rtnl_lock();
+	err = amt_offload_register(lower, &amt_fake_ops);
+	rtnl_unlock();
+	KUNIT_ASSERT_EQ(test, err, 0);
+}
+
+static void amt_test_ofl_done(struct amt_dev *amt, struct net_device *lower)
+{
+	rtnl_lock();
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+static void amt_test_ofl_join(struct amt_dev *amt, struct amt_tunnel_list *t,
+			      int i, bool join)
+{
+	union amt_addr g = amt_test_ofl_group(i);
+
+	if (join)
+		amt_ofl_member_add(amt, t, &g, false);
+	else
+		amt_ofl_member_del(amt, t, &g, false);
+}
+
+static struct amt_ofl_grp *amt_test_ofl_grp(struct amt_dev *amt, int i)
+{
+	union amt_addr a = amt_test_ofl_group(i);
+	struct amt_offload_group_key key;
+	struct amt_ofl_grp *g;
+
+	amt_ofl_grp_key(&key, &a, false);
+	spin_lock_bh(&amt->offload_st->lock);
+	g = amt_ofl_grp_find(amt->offload_st, &key);
+	spin_unlock_bh(&amt->offload_st->lock);
+	return g;
+}
+
+static bool amt_test_ofl_tx(struct amt_dev *amt, struct sk_buff *skb, int i)
+{
+	union amt_addr g = amt_test_ofl_group(i);
+
+	return amt_offload_tx(amt, skb, &g, false);
+}
+
+static bool amt_test_ofl_no_eps(struct amt_dev *amt)
+{
+	u32 i;
+
+	for (i = 0; i < 1U << amt->offload_st->ep_bits; i++)
+		if (!hlist_empty(&amt->offload_st->eps[i]))
+			return false;
+	return true;
+}
+
+/* Members follow gnodes: the first gnode of a group on a tunnel adds
+ * the member, a second (another inner host) only counts, the last one
+ * removes it; unlink removes it at once and the gnodes deleted after it
+ * find nothing.
+ */
+static void amt_offload_member_gnode_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	union amt_addr grp = amt_test_ofl_group(0), h1 = {}, h2 = {};
+	struct amt_group_node *g1, *g2;
+	struct amt_tunnel_list *tunnel;
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt = amt_test_ofl_attach(test, "amton%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	tunnel = amt_test_tunnel_add(test, amt, AMT_TEST_GW, htons(4000),
+				     AMT_TEST_NONCE, AMT_TEST_MAC);
+	h1.ip4 = AMT_TEST_HOST;
+	h2.ip4 = htonl(ntohl(AMT_TEST_HOST) + 1);
+
+	spin_lock_bh(&tunnel->lock);
+	g1 = amt_add_group(amt, tunnel, &grp, &h1, false);
+	g2 = amt_add_group(amt, tunnel, &grp, &h2, false);
+	spin_unlock_bh(&tunnel->lock);
+	KUNIT_ASSERT_FALSE(test, IS_ERR(g1));
+	KUNIT_ASSERT_FALSE(test, IS_ERR(g2));
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+	KUNIT_EXPECT_EQ(test, og->gen, 1);
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+
+	spin_lock_bh(&tunnel->lock);
+	amt_del_group(amt, g1);
+	spin_unlock_bh(&tunnel->lock);
+	KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+	KUNIT_EXPECT_EQ(test, og->gen, 1);
+	spin_lock_bh(&tunnel->lock);
+	amt_del_group(amt, g2);
+	spin_unlock_bh(&tunnel->lock);
+	KUNIT_EXPECT_EQ(test, og->nr_members, 0);
+	KUNIT_EXPECT_EQ(test, og->gen, 2);
+
+	spin_lock_bh(&tunnel->lock);
+	g1 = amt_add_group(amt, tunnel, &grp, &h1, false);
+	spin_unlock_bh(&tunnel->lock);
+	KUNIT_ASSERT_FALSE(test, IS_ERR(g1));
+	KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+	/* Unlink alone, as amt_dev_stop() does before it cancels the
+	 * tunnel's gc work: g1 is still in amt->groups_rhl and on
+	 * tunnel->groups, yet the member is gone.
+	 */
+	spin_lock_bh(&amt->lock);
+	amt_tunnel_unlink(amt, tunnel);
+	spin_unlock_bh(&amt->lock);
+	KUNIT_EXPECT_EQ(test, og->nr_members, 0);
+	KUNIT_EXPECT_EQ(test, og->gen, 4);
+	cancel_delayed_work_sync(&tunnel->gc_wq);
+	amt_clear_groups(tunnel);
+	kfree_rcu(tunnel, rcu);
+	KUNIT_EXPECT_EQ(test, og->gen, 4);	/* g1's delete found nothing */
+
+	/* Unbound: the reconciler only frees the emptied records. */
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* Churn between ticks coalesces into one update and one commit at the
+ * last generation; a tick starts no group once its budget is spent.
+ */
+static void amt_offload_budget_coalesce_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[4], *u[3];
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+	int i, round;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoc%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 4; i++)
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+	for (round = 0; round < 25; round++) {
+		for (i = 0; i < 4; i++) {
+			amt_test_ofl_join(amt, t[i], 0, true);
+			if (round < 24)
+				amt_test_ofl_join(amt, t[i], 0, false);
+		}
+	}
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_EQ(test, og->nr_members, 4);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.commits, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 4);
+	KUNIT_ASSERT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, og->hw_gen, og->gen);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->committed,
+			og->gen);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 4);
+
+	/* Three groups at three ops each (endpoint_add, update, commit):
+	 * a budget of 3 programs one group per tick.
+	 */
+	amt->offload_st->budget = 3;
+	for (i = 0; i < 3; i++) {
+		u[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(5000 + i));
+		amt_test_ofl_join(amt, u[i], i + 1, true);
+	}
+	KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 2);
+	KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 3);
+	KUNIT_EXPECT_FALSE(test, amt_ofl_reconcile(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 4);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 4);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* A group the provider refuses goes to software (the remainder) and is
+ * not retried on its own member changes; it is retried once the
+ * provider holds an eighth fewer objects than when it refused.
+ */
+static void amt_offload_enospc_hysteresis_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_ofl_grp *a, *b;
+	struct amt_tunnel_list *t[3];
+	struct amt_dev *amt;
+	int i, updates;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.ep_cap = 2;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtos%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 3; i++)
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+	amt_test_ofl_join(amt, t[0], 0, true);
+	amt_test_ofl_join(amt, t[1], 0, true);
+	amt_test_ofl_drain(amt);
+	a = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, a);
+	KUNIT_EXPECT_TRUE(test, a->hw_active);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 3);
+
+	amt_test_ofl_join(amt, t[2], 1, true);
+	amt_test_ofl_drain(amt);
+	b = amt_test_ofl_grp(amt, 1);
+	KUNIT_ASSERT_NOT_NULL(test, b);
+	KUNIT_EXPECT_FALSE(test, b->hw_active);
+	KUNIT_EXPECT_EQ(test, b->state, AMT_OFL_PARKED);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+	KUNIT_EXPECT_TRUE(test, a->hw_active);
+
+	/* Parked: a member change does not retry it. */
+	updates = amt_fake.updates;
+	amt_test_ofl_join(amt, t[1], 1, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, b->state, AMT_OFL_PARKED);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, updates);
+
+	/* t[0] leaves A: A is recommitted without it and its endpoint is
+	 * deleted, which crosses the low watermark and retries B.
+	 */
+	amt_test_ofl_join(amt, t[0], 0, false);
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_TRUE(test, a->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)a->hw)->nr, 1);
+	KUNIT_ASSERT_TRUE(test, b->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)b->hw)->nr, 2);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* An op that overruns op_timeout_ms resets the provider: flush, a new
+ * epoch, the late result ignored, and a replay from desired state.
+ */
+static void amt_offload_timeout_reset_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t;
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+	u64 epoch;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtot%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	epoch = amt->offload_epoch;
+	t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	amt_test_ofl_join(amt, t, 0, true);
+	amt_fake.add_stall = true;
+	amt_test_ofl_drain(amt);
+
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_STALE),
+			1);
+	KUNIT_EXPECT_EQ(test, amt->offload_epoch, epoch + 1);
+	/* The late handle died with the flush; the replay added it again. */
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 2);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* A reset late in a tick must not reach a group emptied earlier in it:
+ * the flush already took its handle, so amt neither disables it again nor
+ * counts it out of hw_objs a second time.
+ */
+static void amt_offload_reset_after_empty_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t1, *t2;
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoe%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	t1 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	t2 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+	amt_test_ofl_join(amt, t1, 0, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+
+	/* One tick: group 0 empties, then group 1's endpoint_add overruns. */
+	amt_test_ofl_join(amt, t1, 0, false);
+	amt_test_ofl_join(amt, t2, 1, true);
+	amt_fake.add_stall = true;
+	amt_test_ofl_drain(amt);
+
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	KUNIT_EXPECT_EQ(test, amt_fake.null_disables, 0);
+	KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 2U);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Teardown's release ops are timed: a provider that stalls
+ * in group_disable/endpoint_del costs one op under RTNL, then a flush,
+ * not one stall per object. Untimed, the log reads "BGGEEU".
+ */
+static void amt_offload_teardown_release_stall_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[2];
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtor%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++) {
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+		amt_test_ofl_join(amt, t[i], i, true);
+	}
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+
+	amt_fake.release_stall = true;
+	rtnl_lock();
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+	amt_fake.release_stall = false;
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGU");
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+	/* The group the overrun never reached is unpublished too. */
+	for (i = 0; i < 2; i++)
+		KUNIT_EXPECT_FALSE(test,
+				   READ_ONCE(amt_test_ofl_grp(amt, i)->hw_active));
+	/* Both groups were uncommitted, the disabled one and the forgotten
+	 * one, so once their members leave both endpoints are collected.
+	 */
+	for (i = 0; i < 2; i++)
+		amt_test_ofl_join(amt, t[i], i, false);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* Teardown with only endpoints programmed (the group parked): the first
+ * endpoint_del to overrun ends it. Untimed, the log reads "BGEEU".
+ */
+static void amt_offload_teardown_ep_stall_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_fake.commit_err = -EIO;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtot%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++)
+		amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+							   htons(4000 + i)),
+				  0, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_test_ofl_grp(amt, 0)->state, AMT_OFL_PARKED);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+
+	amt_fake.release_stall = true;
+	rtnl_lock();
+	amt_offload_unregister(lower);
+	rtnl_unlock();
+	amt_fake.release_stall = false;
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGEU");
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+	amt_test_offload_free(amt);
+	free_netdev(lower);
+}
+
+/* The gone and GC loops time their release ops too: one
+ * stall, then a flush, not one per object. Each part needs its own relay,
+ * because a gone-loop reset forgets every handle and the GC loop then has
+ * nothing to delete.
+ */
+static struct amt_dev *amt_test_ofl_stall_setup(struct kunit *test,
+						struct net_device *lower,
+						struct amt_tunnel_list **t)
+{
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtos%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++) {
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+		amt_test_ofl_join(amt, t[i], i, true);
+	}
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+	return amt;
+}
+
+static void amt_offload_reconcile_release_stall_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[3];
+	struct amt_dev *amt;
+	int i;
+
+	/* (a) Two groups empty in one tick: the first group_disable
+	 * overruns and resets. Untimed, the GC loop resets one op later
+	 * and the log reads "BGGE".
+	 */
+	amt = amt_test_ofl_stall_setup(test, lower, t);
+	for (i = 0; i < 2; i++)
+		amt_test_ofl_join(amt, t[i], i, false);
+	amt_fake.release_stall = true;
+	amt_ofl_reconcile(amt);
+	amt_fake.release_stall = false;
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BG");
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	KUNIT_EXPECT_EQ(test, amt_fake.null_disables, 0);
+	/* The reset uncommitted the group it forgot: both endpoints go. */
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+	amt_test_ofl_done(amt, lower);
+
+	/* (b) One endpoint leaves a group, nothing empties: its
+	 * endpoint_del overruns and resets. hw_objs is counted down before
+	 * the op, so the reset's zeroing is final.
+	 */
+	lower = amt_test_lower_alloc(test);
+	amt = amt_test_ofl_stall_setup(test, lower, t);
+	t[2] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4002));
+	amt_test_ofl_join(amt, t[2], 0, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+	amt_test_ofl_join(amt, t[2], 0, false);
+	amt_fake.release_stall = true;
+	amt_ofl_reconcile(amt);
+	amt_fake.release_stall = false;
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BE");
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs,
+			(unsigned int)(amt_fake_live(amt_fake.eps) +
+				       amt_fake_live(amt_fake.grps)));
+	amt_test_ofl_done(amt, lower);
+}
+
+/* One group re-dirtied from inside every group_commit: the churn the
+ * hook drives spends each tick's budget on that group.
+ */
+static struct {
+	struct amt_dev			*amt;
+	struct amt_tunnel_list		*t;
+	bool				in;
+	bool				on;
+} amt_test_churn;
+
+static void amt_test_churn_hook(void)
+{
+	if (!amt_test_churn.on)
+		return;
+	amt_test_churn.in = !amt_test_churn.in;
+	amt_test_ofl_join(amt_test_churn.amt, amt_test_churn.t, 0,
+			  amt_test_churn.in);
+	amt_fake.commit_hook = amt_test_churn_hook;
+}
+
+static bool amt_test_fake_ep_held(const struct amt_offload_endpoint_key *key)
+{
+	int i;
+
+	for (i = 0; i < AMT_FAKE_SLOTS; i++)
+		if (amt_fake.eps[i].used &&
+		    !memcmp(&amt_fake.eps[i].key, key, sizeof(*key)))
+			return true;
+	return false;
+}
+
+/* Endpoint GC does not wait for st->dirty to empty: under
+ * sustained churn in one group, a departed endpoint that no committed
+ * group names is deleted within a bounded number of ticks. Gated on an
+ * empty st->dirty it never is.
+ */
+static void amt_offload_gc_churn_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *stay, *churn, *gone;
+	struct amt_offload_endpoint_key key, ckey;
+	struct amt_dev *amt;
+	int tick;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtog%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	stay = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	churn = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+	gone = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4002));
+	amt_test_ofl_join(amt, stay, 0, true);
+	amt_test_ofl_join(amt, churn, 0, true);
+	amt_test_ofl_join(amt, gone, 1, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+	amt_ofl_ep_key(&key, amt, gone);
+	KUNIT_ASSERT_TRUE(test, amt_test_fake_ep_held(&key));
+	amt_ofl_ep_key(&ckey, amt, churn);
+
+	amt->offload_st->budget = 6;
+	memset(&amt_test_churn, 0, sizeof(amt_test_churn));
+	amt_test_churn.amt = amt;
+	amt_test_churn.t = churn;
+	amt_test_churn.on = true;
+	amt_test_ofl_join(amt, churn, 0, false);
+	amt_test_ofl_join(amt, gone, 1, false);
+	amt_fake.commit_hook = amt_test_churn_hook;
+	/* GC no longer waits for st->dirty, so nothing but the commit
+	 * accounting keeps it off an endpoint the last commit still names.
+	 */
+	for (tick = 0; tick < 3; tick++) {
+		KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+		KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&ckey));
+	}
+	KUNIT_EXPECT_FALSE(test, amt_test_fake_ep_held(&key));
+
+	amt_test_churn.on = false;
+	amt_fake.commit_hook = NULL;
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* endpoint_add is idempotent on an identical key, and amt adds an
+ * endpoint shared by several groups once.
+ */
+static void amt_offload_endpoint_idempotent_test(struct kunit *test)
+{
+	struct amt_offload_endpoint_key key = {};
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_offload_endpoint *h1 = NULL, *h2 = NULL;
+	struct amt_tunnel_list *t;
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	key.family = AF_INET;
+	key.addr.ip4 = AMT_TEST_GW;
+	key.port = htons(4000);
+	KUNIT_EXPECT_EQ(test, amt_fake_endpoint_add(NULL, &key, &h1), 0);
+	KUNIT_EXPECT_EQ(test, amt_fake_endpoint_add(NULL, &key, &h2), 0);
+	KUNIT_EXPECT_PTR_EQ(test, h1, h2);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoi%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	amt_test_ofl_join(amt, t, 0, true);
+	amt_test_ofl_join(amt, t, 1, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 2);
+	/* Leaving and rejoining one group keeps the endpoint programmed. */
+	amt_test_ofl_join(amt, t, 1, false);
+	amt_test_ofl_join(amt, t, 1, true);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* A failed group_commit disables the group and serves it in software;
+ * the endpoints it added stay programmed. A new binding retries it.
+ */
+static void amt_test_ofl_sync_net(void)
+{
+	amt_fake_log('S');
+	synchronize_net();
+}
+
+static void amt_offload_commit_fail_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[2];
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.commit_err = -EIO;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoe%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++) {
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+		amt_test_ofl_join(amt, t[i], 0, true);
+	}
+	amt_test_ofl_drain(amt);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_FALSE(test, og->hw_active);
+	KUNIT_EXPECT_NULL(test, og->hw);
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_PARKED);
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "BG");
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+
+	amt_fake.commit_err = 0;
+	rtnl_lock();
+	amt_offload_unregister(lower);
+	KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	rtnl_unlock();
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+
+	/* A leave withdraws the committed group, then its recommit fails:
+	 * park still disables the provider object it holds, and waits out
+	 * ops->xmit first although hw_active already reads false. 'S' is
+	 * that wait: park, endpoint GC of the departed endpoint, which
+	 * unparks, then the second failed commit parks again. Without the
+	 * wait the log reads "GEG".
+	 */
+	amt_test_ofl_join(amt, t[1], 0, false);
+	KUNIT_EXPECT_FALSE(test, og->hw_active);
+	amt_fake.commit_err = -EIO;
+	memset(amt_fake.log, 0, sizeof(amt_fake.log));
+	amt_fake.n = 0;
+	kunit_activate_static_stub(test, amt_ofl_sync_net,
+				   amt_test_ofl_sync_net);
+	amt_test_ofl_drain(amt);
+	kunit_deactivate_static_stub(test, amt_ofl_sync_net);
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "SGESG");
+	KUNIT_EXPECT_NULL(test, og->hw);
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_PARKED);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* The publish gate: a member removed while the
+ * reconciler programs the group is still in what it commits, so that
+ * commit is not published and software serves the group; a member added
+ * meanwhile does not hold the publication back. A budget of one ends the
+ * tick right after the program the hook lands in.
+ */
+static struct {
+	struct amt_dev			*amt;
+	struct amt_tunnel_list		*tunnel;
+	int				grp;
+	bool				join;
+} amt_test_ofl_hook;
+
+static void amt_test_ofl_commit_hook(void)
+{
+	amt_test_ofl_join(amt_test_ofl_hook.amt, amt_test_ofl_hook.tunnel,
+			  amt_test_ofl_hook.grp, amt_test_ofl_hook.join);
+}
+
+static void amt_offload_publish_gate_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_offload_endpoint_key key;
+	struct amt_tunnel_list *t[3];
+	struct amt_offload_state *st;
+	struct amt_ofl_grp *og;
+	struct sk_buff *skb;
+	struct amt_dev *amt;
+	u32 budget;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtop%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	st = amt->offload_st;
+	budget = st->budget;
+	for (i = 0; i < 3; i++)
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+	amt_test_ofl_hook.amt = amt;
+
+	/* A join inside the first commit: publish the committed subset. */
+	amt_test_ofl_join(amt, t[0], 0, true);
+	amt_test_ofl_hook.tunnel = t[1];
+	amt_test_ofl_hook.join = true;
+	amt_fake.commit_hook = amt_test_ofl_commit_hook;
+	st->budget = 1;
+	amt_ofl_reconcile(amt);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+	KUNIT_ASSERT_NOT_NULL(test, og->hw);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 1);
+	st->budget = budget;
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+
+	/* A leave inside a recommit: withhold it, software serves G. */
+	amt_test_ofl_join(amt, t[2], 0, true);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);	/* joins do not withdraw */
+	amt_test_ofl_hook.tunnel = t[0];
+	amt_test_ofl_hook.join = false;
+	amt_fake.commit_hook = amt_test_ofl_commit_hook;
+	st->budget = 1;
+	amt_ofl_reconcile(amt);
+	KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+	KUNIT_EXPECT_FALSE(test, og->hw_active);
+	/* The provider still replicates to t[0]: its endpoint stays. */
+	amt_ofl_ep_key(&key, amt, t[0]);
+	KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&key));
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+	i = amt_fake.xmits;
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, i);
+	st->budget = budget;
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+	kfree_skb(skb);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Same-key rejoin: K empties and goes on the tick's gone list with its
+ * provider group still live; a join inside the next group's commit hashes
+ * a new K, which the same tick programs. That first use must wait for the
+ * old group's disable: the fake refuses a second live group per key, and
+ * without the flush the new K parks in software.
+ */
+static void amt_offload_rejoin_gone_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[2];
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtor%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++)
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+	amt_test_ofl_join(amt, t[0], 0, true);
+	amt_test_ofl_drain(amt);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_ASSERT_TRUE(test, og->hw_active);
+
+	/* K empties first, then G dirties; G's commit rejoins K. */
+	amt_test_ofl_join(amt, t[0], 0, false);
+	amt_test_ofl_join(amt, t[1], 1, true);
+	amt_test_ofl_hook.amt = amt;
+	amt_test_ofl_hook.tunnel = t[0];
+	amt_test_ofl_hook.grp = 0;
+	amt_test_ofl_hook.join = true;
+	amt_fake.commit_hook = amt_test_ofl_commit_hook;
+	/* 'S' is the mid-pass flush's wait for ops->xmit on the old K before
+	 * its disable; without it the log reads "G".
+	 */
+	memset(amt_fake.log, 0, sizeof(amt_fake.log));
+	amt_fake.n = 0;
+	kunit_activate_static_stub(test, amt_ofl_sync_net,
+				   amt_test_ofl_sync_net);
+	amt_ofl_reconcile(amt);
+	kunit_deactivate_static_stub(test, amt_ofl_sync_net);
+	KUNIT_EXPECT_STREQ(test, amt_fake.log, "SG");
+	KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+	KUNIT_EXPECT_EQ(test, amt_fake.dup_grps, 0);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_IDLE);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_SW_REMAINDER), 0);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* The mid-pass flush above is timed like any release op: a stalled
+ * group_disable of the old K resets the provider once, and the pass ends
+ * there rather than program the new K against a provider it just
+ * flushed. The next ticks replay it.
+ */
+static void amt_offload_rejoin_gone_stall_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[2];
+	struct amt_ofl_grp *og;
+	struct amt_dev *amt;
+	int i, updates;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtos%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++)
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+	amt_test_ofl_join(amt, t[0], 0, true);
+	amt_test_ofl_drain(amt);
+
+	amt_test_ofl_join(amt, t[0], 0, false);
+	amt_test_ofl_join(amt, t[1], 1, true);
+	amt_test_ofl_hook.amt = amt;
+	amt_test_ofl_hook.tunnel = t[0];
+	amt_test_ofl_hook.grp = 0;
+	amt_test_ofl_hook.join = true;
+	amt_fake.commit_hook = amt_test_ofl_commit_hook;
+	amt_fake.release_stall = true;
+	updates = amt_fake.updates;
+	amt_ofl_reconcile(amt);
+	amt_fake.release_stall = false;
+	KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+			1);
+	/* G's update only: the new K was not programmed. */
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, updates + 1);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_EXPECT_NULL(test, og->hw);
+	KUNIT_EXPECT_FALSE(test, og->hw_active);
+
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, amt_fake.dup_grps, 0);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Exactly one actor per packet: CONSUMED gives the skb away, FALLBACK
+ * and BUSY return it untouched (BUSY counted); an uncommitted group and
+ * an unbound relay never reach the provider. A packet submitted after a
+ * commit carries the new generation.
+ */
+static void amt_offload_tx_ownership_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_tunnel_list *t[2];
+	struct amt_ofl_grp *og;
+	struct sk_buff *skb;
+	struct amt_dev *amt;
+	u64 gen;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt = amt_test_ofl_attach(test, "amtox%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	t[0] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	t[1] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+	amt_test_ofl_join(amt, t[0], 0, true);
+	amt_test_ofl_drain(amt);
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+
+	/* No provider: software, as with offload=software. */
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+
+	rtnl_lock();
+	KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+	rtnl_unlock();
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, 0);	/* not committed yet */
+	amt_test_ofl_drain(amt);
+	og = amt_test_ofl_grp(amt, 0);
+	KUNIT_ASSERT_NOT_NULL(test, og);
+	KUNIT_ASSERT_TRUE(test, og->hw_active);
+
+	amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	amt_fake.xmit_res = AMT_OFFLOAD_BUSY;
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_BUSY_FALLBACK), 1);
+	KUNIT_EXPECT_EQ(test, refcount_read(&skb->users), 1);
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 1));	/* no group */
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, 2);
+	KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, og->hw_gen);
+
+	/* Generation rule. */
+	gen = og->hw_gen;
+	amt_test_ofl_join(amt, t[1], 0, true);
+	amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, gen);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_GT(test, og->hw_gen, gen);
+	amt_fake.xmit_res = AMT_OFFLOAD_CONSUMED;
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0));	/* freed */
+	KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, og->hw_gen);
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, 4);
+
+	/* Removing a member withdraws the group from hardware at once, not
+	 * at the next commit: the committed set still names the departed
+	 * endpoint.
+	 */
+	amt_test_ofl_join(amt, t[1], 0, false);
+	KUNIT_EXPECT_FALSE(test, og->hw_active);
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, 4);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_TRUE(test, og->hw_active);
+	KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 1);
+	kfree_skb(skb);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Admission: a teardown waiting on st->mutex holds RTNL, so once admission
+ * stops a programming pass makes no further endpoint_add, group_update,
+ * group_commit or endpoint_del, and ends the tick. A pass releasing
+ * everything after desync does not check admission and finishes that
+ * release. The fake clears b->admitting from inside one op. The four
+ * blocks below fail with any one of these five admission checks reverted
+ * (before endpoint_add, group_update and group_commit, at the top of the
+ * pass loop, in the endpoint GC); the pass loop's epoch check is
+ * amt_offload_reset_ends_pass_test's.
+ */
+static void amt_offload_admit_guard_test(struct kunit *test)
+{
+	struct amt_tunnel_list *t[3];
+	struct net_device *lower;
+	struct amt_dev *amt;
+	int i;
+
+	/* A two-endpoint group, admission lost in the first endpoint_add:
+	 * no second endpoint_add.
+	 */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtga%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++)
+		amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+							   htons(4000 + i)),
+				  0, true);
+	amt_fake.drop_admit = 'A';
+	amt_ofl_reconcile(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+	amt_test_ofl_done(amt, lower);
+
+	/* Three one-endpoint groups, lost in the first endpoint_add: no
+	 * group_update, and the pass ends with the other two still queued.
+	 */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtgb%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 3; i++)
+		amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+							   htons(4000 + i)),
+				  i, true);
+	amt_fake.drop_admit = 'A';
+	KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+	KUNIT_EXPECT_EQ(test, amt_fake.commits, 0);
+	for (i = 1; i < 3; i++) {
+		struct amt_ofl_grp *g = amt_test_ofl_grp(amt, i);
+
+		KUNIT_ASSERT_NOT_NULL(test, g);
+		KUNIT_EXPECT_EQ(test, g->state, AMT_OFL_DIRTY);
+	}
+	amt_test_ofl_done(amt, lower);
+
+	/* Lost in group_update: no group_commit. */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtgc%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+						   htons(4000)), 0, true);
+	amt_fake.drop_admit = 'U';
+	amt_ofl_reconcile(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.commits, 0);
+	amt_test_ofl_done(amt, lower);
+
+	/* Two unused endpoints, lost in the first endpoint_del: the
+	 * endpoint GC deletes no second one.
+	 */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtgd%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 2; i++) {
+		t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+		amt_test_ofl_join(amt, t[i], i, true);
+	}
+	amt_test_ofl_drain(amt);
+	KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+	for (i = 0; i < 2; i++)
+		amt_test_ofl_join(amt, t[i], i, false);
+	amt_fake.drop_admit = 'E';
+	amt_ofl_reconcile(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* A reset (an op overran op_timeout_ms) replays every group, so
+ * the pass ends at it instead of spending the rest of the budget against
+ * a provider that may be wedged.
+ */
+static void amt_offload_reset_ends_pass_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt;
+	int i;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.op_timeout_ms = 10;
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtor%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	for (i = 0; i < 3; i++)
+		amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+							   htons(4000 + i)),
+				  i, true);
+	amt_fake.add_stall = true;
+	KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+
+	/* The next tick replays all three. */
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 3);
+	KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Hash seed: the offload tables are indexed with the device's hash_seed and
+ * sized from max_tunnels (x max_groups) at attach, so a gateway can
+ * neither precompute one chain nor fill a fixed-size table.
+ */
+static void amt_offload_hash_seed_test(struct kunit *test)
+{
+	struct amt_offload_endpoint_key ek;
+	struct amt_offload_group_key gk;
+	bool gmoved = false, emoved = false;
+	struct amt_offload_state *st;
+	struct hlist_head *g0, *e0;
+	struct net_device *lower;
+	struct amt_dev *amt;
+	u32 seed;
+
+	memset(&ek, 0, sizeof(ek));
+	ek.family = AF_INET;
+	ek.addr.ip4 = AMT_TEST_GW;
+	ek.port = htons(4000);
+	memset(&gk, 0, sizeof(gk));
+	gk.family = AF_INET;
+	gk.group.ip4 = AMT_TEST_GROUP;
+
+	/* Default caps: 128 tunnels x 32 groups. */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoh%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	st = amt->offload_st;
+	KUNIT_EXPECT_EQ(test, st->ep_bits, 7);
+	KUNIT_EXPECT_EQ(test, st->grp_bits, 12);
+	amt->hash_seed = 0;
+	g0 = amt_ofl_grp_bucket(st, &gk);
+	e0 = amt_ofl_ep_bucket(st, &ek);
+	for (seed = 1; seed < 64 && !(gmoved && emoved); seed++) {
+		amt->hash_seed = seed;
+		gmoved |= amt_ofl_grp_bucket(st, &gk) != g0;
+		emoved |= amt_ofl_ep_bucket(st, &ek) != e0;
+	}
+	KUNIT_EXPECT_TRUE_MSG(test, gmoved, "group bucket ignores hash_seed");
+	KUNIT_EXPECT_TRUE_MSG(test, emoved, "endpoint bucket ignores hash_seed");
+	amt_test_ofl_done(amt, lower);
+
+	/* max_tunnels set before attach sizes both tables. */
+	lower = amt_test_lower_alloc(test);
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_offload_relay(test, "amtoi%d", lower,
+				     AMT_OFFLOAD_MODE_AUTO);
+	amt->max_tunnels = 10000;
+	rtnl_lock();
+	KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+	rtnl_unlock();
+	KUNIT_ASSERT_NOT_NULL(test, amt->offload_st);
+	disable_work_sync(&amt->offload_st->work);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->ep_bits, 14);
+	KUNIT_EXPECT_EQ(test, amt->offload_st->grp_bits, 16);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* The last member leaves while the reconciler allocates its arrays: the
+ * retry finds no member and must not free the arrays a second time. The
+ * allocation and free stubs keep a ledger of live arrays, so a repeated
+ * free fails the case in every configuration, not only under KASAN.
+ */
+static struct {
+	struct kunit			*test;
+	struct amt_dev			*amt;
+	struct amt_tunnel_list		*tunnel;
+	int				calls;
+	void				*live[4];
+	int				nr_live;
+} amt_test_ofl_race;
+
+static void *amt_test_ofl_alloc_leave(u32 n, size_t size)
+{
+	void *p;
+
+	if (!amt_test_ofl_race.calls++)
+		amt_test_ofl_join(amt_test_ofl_race.amt,
+				  amt_test_ofl_race.tunnel, 0, false);
+	p = kvmalloc_array(n, size, GFP_KERNEL);
+	if (!p)
+		return NULL;
+	if (amt_test_ofl_race.nr_live == ARRAY_SIZE(amt_test_ofl_race.live)) {
+		/* A later free of p would read as a double free. */
+		KUNIT_FAIL(amt_test_ofl_race.test, "live-array ledger full");
+		return p;
+	}
+	amt_test_ofl_race.live[amt_test_ofl_race.nr_live++] = p;
+	return p;
+}
+
+static void amt_test_ofl_free_once(void *p)
+{
+	int i;
+
+	if (!p)
+		return;
+	for (i = 0; i < amt_test_ofl_race.nr_live; i++) {
+		if (amt_test_ofl_race.live[i] == p) {
+			amt_test_ofl_race.live[i] =
+				amt_test_ofl_race.live[--amt_test_ofl_race.nr_live];
+			kvfree(p);
+			return;
+		}
+	}
+	/* Not live: a second free. Leak it rather than corrupt the heap. */
+	KUNIT_FAIL(amt_test_ofl_race.test, "array %p freed twice", p);
+}
+
+static void amt_offload_alloc_race_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_dev *amt;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_test_ofl_register(test, lower);
+	amt = amt_test_ofl_attach(test, "amtoa%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	memset(&amt_test_ofl_race, 0, sizeof(amt_test_ofl_race));
+	amt_test_ofl_race.test = test;
+	amt_test_ofl_race.amt = amt;
+	amt_test_ofl_race.tunnel = amt_test_ofl_tunnel(test, AMT_TEST_GW,
+						       htons(4000));
+	amt_test_ofl_join(amt, amt_test_ofl_race.tunnel, 0, true);
+
+	kunit_activate_static_stub(test, amt_ofl_alloc_array,
+				   amt_test_ofl_alloc_leave);
+	kunit_activate_static_stub(test, amt_ofl_free_array,
+				   amt_test_ofl_free_once);
+	amt_test_ofl_drain(amt);
+	kunit_deactivate_static_stub(test, amt_ofl_free_array);
+	kunit_deactivate_static_stub(test, amt_ofl_alloc_array);
+
+	/* One allocation round, then the retry read no member. */
+	KUNIT_EXPECT_EQ(test, amt_test_ofl_race.calls, 2);
+	KUNIT_EXPECT_EQ(test, amt_test_ofl_race.nr_live, 0);
+	KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+	KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+	amt_test_ofl_done(amt, lower);
+}
+
+/* Ingress: the host drops a punted copy of a committed group and sends
+ * everything else to software. Only a required relay binds on ingress.
+ * A leave does not lift the drop: hardware keeps replicating to the
+ * committed set until the recommit, so software would send the remaining
+ * members every packet a second time.
+ */
+static void amt_offload_ingress_punt_test(struct kunit *test)
+{
+	struct net_device *lower = amt_test_lower_alloc(test);
+	struct amt_offload_endpoint_key key;
+	struct amt_tunnel_list *t, *t2;
+	struct amt_dev *amt, *aut;
+	struct sk_buff *skb;
+	u32 budget;
+
+	amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+	amt_fake.paths = BIT(AMT_OFFLOAD_PATH_INGRESS);
+	amt_test_ofl_register(test, lower);
+	aut = amt_test_ofl_attach(test, "amtoa%d", lower,
+				  AMT_OFFLOAD_MODE_AUTO);
+	KUNIT_EXPECT_NULL(test, rcu_access_pointer(aut->offload));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(aut,
+			AMT_OFFLOAD_CNT_PATH_UNSUPPORTED), 1);
+
+	amt = amt_test_ofl_attach(test, "amtog%d", lower,
+				  AMT_OFFLOAD_MODE_REQUIRED);
+	KUNIT_ASSERT_NOT_NULL(test, rcu_access_pointer(amt->offload));
+	t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+	t2 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+	amt_test_ofl_join(amt, t, 0, true);
+	amt_test_ofl_join(amt, t2, 0, true);
+	amt_test_ofl_join(amt, t, 2, true);
+	amt_test_ofl_drain(amt);
+	amt_test_ofl_join(amt, t, 3, true);	/* not committed */
+
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 1));
+	KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 3));
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_PUNT_DROPPED), 0);
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0));	/* dropped */
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_PUNT_DROPPED), 1);
+
+	amt_test_ofl_join(amt, t2, 0, false);
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0));	/* dropped */
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_PUNT_DROPPED), 2);
+
+	/* A leave inside a group's first commit: hardware replicates to the
+	 * committed set anyway, so the pass still publishes it and the
+	 * punted copy is dropped. Withholding it would send the remaining
+	 * member every packet twice until the next pass.
+	 */
+	amt_test_ofl_drain(amt);
+	amt_test_ofl_join(amt, t, 4, true);
+	amt_test_ofl_join(amt, t2, 4, true);
+	amt_test_ofl_hook.amt = amt;
+	amt_test_ofl_hook.tunnel = t2;
+	amt_test_ofl_hook.grp = 4;
+	amt_test_ofl_hook.join = false;
+	amt_fake.commit_hook = amt_test_ofl_commit_hook;
+	budget = amt->offload_st->budget;
+	amt->offload_st->budget = 1;
+	amt_ofl_reconcile(amt);
+	amt->offload_st->budget = budget;
+	amt_test_ofl_hook.grp = 0;
+	KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+	amt_ofl_ep_key(&key, amt, t2);
+	KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&key));
+	skb = alloc_skb(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, skb);
+	KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 4));	/* dropped */
+	KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+			AMT_OFFLOAD_CNT_PUNT_DROPPED), 3);
+	amt_test_ofl_drain(amt);
+	KUNIT_EXPECT_EQ(test, amt_fake.xmits, 0);
+	amt_test_offload_free(aut);
+	amt_test_ofl_done(amt, lower);
+}
+
 static struct kunit_case amt_test_cases[] = {
 	KUNIT_CASE(amt_lifecycle_test),
 	KUNIT_CASE(amt_upstream_refresh_idempotent_test),
 	KUNIT_CASE(amt_ex_transition_test),
+	KUNIT_CASE(amt_offload_register_when_bound_test),
+	KUNIT_CASE(amt_offload_family_unsupported_test),
+	KUNIT_CASE(amt_offload_unregister_mid_fanout_test),
+	KUNIT_CASE(amt_offload_provider_unregister_order_test),
+	KUNIT_CASE(amt_offload_mode_validate_test),
+	KUNIT_CASE(amt_offload_required_refused_test),
+	KUNIT_CASE(amt_offload_member_gnode_test),
+	KUNIT_CASE(amt_offload_budget_coalesce_test),
+	KUNIT_CASE(amt_offload_enospc_hysteresis_test),
+	KUNIT_CASE(amt_offload_timeout_reset_test),
+	KUNIT_CASE(amt_offload_reset_after_empty_test),
+	KUNIT_CASE(amt_offload_teardown_release_stall_test),
+	KUNIT_CASE(amt_offload_teardown_ep_stall_test),
+	KUNIT_CASE(amt_offload_reconcile_release_stall_test),
+	KUNIT_CASE(amt_offload_gc_churn_test),
+	KUNIT_CASE(amt_offload_endpoint_idempotent_test),
+	KUNIT_CASE(amt_offload_commit_fail_test),
+	KUNIT_CASE(amt_offload_publish_gate_test),
+	KUNIT_CASE(amt_offload_rejoin_gone_test),
+	KUNIT_CASE(amt_offload_rejoin_gone_stall_test),
+	KUNIT_CASE(amt_offload_tx_ownership_test),
+	KUNIT_CASE(amt_offload_alloc_race_test),
+	KUNIT_CASE(amt_offload_admit_guard_test),
+	KUNIT_CASE(amt_offload_reset_ends_pass_test),
+	KUNIT_CASE(amt_offload_hash_seed_test),
+	KUNIT_CASE(amt_offload_ingress_punt_test),
 	{}
 };
 
-- 
2.43.0

[RFC PATCH net-next 7/9] netdevsim: add a TX reference provider for AMT relay offload

[Omar Ramadan]

Each PF port offers itself as an AMT relay replication offload provider
(include/net/amt_offload.h) for relays whose stream device it is. It is
not a pass-through to amt's software fan-out: the port keeps its own
endpoint and group tables, with a staged and a committed member set per
group under the generation amt gave it, and decides from those which
endpoints get a copy of each packet amt hands to ->xmit. Each copy is
encapsulated by amt_offload_encap_xmit() and sent out of the port to its
peer, so any divergence between amt's desired state and what the
provider committed shows up as a wrong copy count at the peer. The set
committed before the last one is kept for packets already in flight,
and an endpoint stays allocated while a set names it.

debugfs (ports/<n>/amt/) shows the tables and counters and injects the
failures the contract has to survive: busy (xmit returns BUSY),
fail_commit (group_commit returns -ENOSPC), max_endpoints (endpoint_add
returns -ENOSPC), stall_ms (every programming op sleeps, so amt sees it
overrun op_timeout_ms and resets) and op_timeout_ms (advertised at
bind).

The provider owns its devlink objects, registered on netdevsim's own
devlink instance, not amt's: resources amt/endpoints, amt/groups,
amt/members and amt/cmdq, and an "amt" health reporter reported on
every flush (amt's reset). amt registers nothing on devlink.

Moving a netdev to another netns unregisters it there first, which
makes amt forget its provider; the port offers itself again once it is
registered in the new netns.

TX only: netdevsim has no ingress classifier to simulate the ingress
path with.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 drivers/net/Kconfig               |   1 +
 drivers/net/netdevsim/Makefile    |   4 +
 drivers/net/netdevsim/amt.c       | 600 ++++++++++++++++++++++++++++++
 drivers/net/netdevsim/dev.c       |  68 ++++
 drivers/net/netdevsim/health.c    |  28 +-
 drivers/net/netdevsim/netdev.c    |  19 +-
 drivers/net/netdevsim/netdevsim.h |  27 ++
 7 files changed, 744 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/netdevsim/amt.c

diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index fda19ae4b..6e45d2fa0 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -625,6 +625,7 @@ config NETDEVSIM
 	depends on INET
 	depends on PSAMPLE || PSAMPLE=n
 	depends on PTP_1588_CLOCK_MOCK || PTP_1588_CLOCK_MOCK=n
+	depends on AMT || AMT=n
 	select NET_DEVLINK
 	select PAGE_POOL
 	select NET_SHAPER
diff --git a/drivers/net/netdevsim/Makefile b/drivers/net/netdevsim/Makefile
index 87718204f..eec03812a 100644
--- a/drivers/net/netdevsim/Makefile
+++ b/drivers/net/netdevsim/Makefile
@@ -25,3 +25,7 @@ endif
 ifneq ($(CONFIG_MACSEC),)
 netdevsim-objs += macsec.o
 endif
+
+ifneq ($(CONFIG_AMT),)
+netdevsim-objs += amt.o
+endif
diff --git a/drivers/net/netdevsim/amt.c b/drivers/net/netdevsim/amt.c
new file mode 100644
index 000000000..ffb9aa4e7
--- /dev/null
+++ b/drivers/net/netdevsim/amt.c
@@ -0,0 +1,600 @@
+// SPDX-License-Identifier: GPL-2.0
+/* netdevsim: TX reference provider for AMT relay replication offload.
+ *
+ * Each PF port registers itself as a provider for relays that transmit
+ * on it (include/net/amt_offload.h). The port keeps its OWN endpoint and
+ * group tables, with a staged and a committed member set per group under
+ * the generation amt gave it, and decides replication from them: a packet
+ * amt hands to ->xmit is copied to the endpoints of the committed set of
+ * that generation, each copy encapsulated by amt_offload_encap_xmit() and
+ * sent out of this port to its peer. Any divergence between what amt
+ * wants and what the provider committed shows up at the peer as a wrong
+ * copy count. TX only: netdevsim has no ingress classifier to simulate.
+ *
+ * debugfs, per port, under ports/<n>/amt/:
+ *   tables         the provider's endpoints and groups, read-only
+ *   tx_copies, tx_busy, tx_stale_gen, tx_dropped, commits, flushes
+ *                  counters, read-only
+ *   busy           xmit returns BUSY
+ *   fail_commit    group_commit fails with -ENOSPC
+ *   max_endpoints  endpoint_add fails with -ENOSPC beyond this (0: none)
+ *   stall_ms       every programming op sleeps this long first
+ *   op_timeout_ms  advertised to amt at bind (0: no limit)
+ *
+ * devlink, per device: resources amt/{endpoints,groups,members,cmdq} and
+ * the "amt" health reporter, reported on every flush (amt's reset).
+ */
+
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/netdevice.h>
+#include <linux/rculist.h>
+#include <linux/refcount.h>
+#include <linux/seq_file.h>
+#include <linux/skbuff.h>
+#include <net/amt_offload.h>
+#include <net/devlink.h>
+
+#include "netdevsim.h"
+
+struct amt_offload_endpoint {
+	struct list_head		node;	/* nsim_amt_bind.eps */
+	struct amt_offload_endpoint_key	key;
+	/* One for amt's handle (all endpoint_add calls for the key), one per
+	 * member set naming it: an in-flight packet may still be copied to
+	 * an endpoint amt already deleted.
+	 */
+	refcount_t			refs;
+	u32				adds;
+	atomic64_t			copies;
+	struct rcu_head			rcu;
+};
+
+struct nsim_amt_set {
+	u64				gen;
+	u32				n;
+	struct rcu_head			rcu;
+	struct amt_offload_endpoint	*eps[] __counted_by(n);
+};
+
+struct amt_offload_group {
+	struct list_head		node;	/* nsim_amt_bind.grps */
+	struct amt_offload_group_key	key;
+	struct nsim_amt_set		*staged;
+	/* Committed last, and the one before it for packets in flight. */
+	struct nsim_amt_set __rcu	*cur;
+	struct nsim_amt_set __rcu	*prev;
+	struct rcu_head			rcu;
+};
+
+struct nsim_amt_bind {
+	struct list_head		node;	/* nsim_amt.binds */
+	struct amt_offload_binding	*b;
+	struct list_head		eps;
+	struct list_head		grps;
+	u32				nr_eps;
+};
+
+struct nsim_amt {
+	struct netdevsim		*ns;
+	struct mutex			lock;	/* binds and their tables */
+	struct list_head		binds;
+	struct dentry			*ddir;
+	bool				busy;
+	bool				fail_commit;
+	u32				max_endpoints;
+	u32				stall_ms;
+	u32				op_timeout_ms;
+	u64				commits;
+	u64				flushes;
+	atomic64_t			tx_copies;
+	atomic64_t			tx_busy;
+	atomic64_t			tx_stale_gen;
+	atomic64_t			tx_dropped;
+};
+
+#define nsim_amt_deref(nam, p) \
+	rcu_dereference_protected(p, lockdep_is_held(&(nam)->lock))
+
+static struct nsim_amt *nsim_amt_of(struct amt_offload_binding *b)
+{
+	struct netdevsim *ns = netdev_priv(b->lower);
+
+	return ns->amt;
+}
+
+/* Every programming op runs inside one of these: cmdq occupancy, and the
+ * injected stall that makes amt see an op overrun op_timeout_ms.
+ */
+static struct nsim_amt *nsim_amt_op_begin(struct amt_offload_binding *b)
+{
+	struct nsim_amt *nam = nsim_amt_of(b);
+	u32 stall = READ_ONCE(nam->stall_ms);
+
+	atomic_inc(&nam->ns->nsim_dev->amt.cmdq);
+	if (stall)
+		msleep(stall);
+	mutex_lock(&nam->lock);
+	return nam;
+}
+
+static void nsim_amt_op_end(struct nsim_amt *nam)
+{
+	mutex_unlock(&nam->lock);
+	atomic_dec(&nam->ns->nsim_dev->amt.cmdq);
+}
+
+static void nsim_amt_ep_put(struct amt_offload_endpoint *ep)
+{
+	if (refcount_dec_and_test(&ep->refs))
+		kfree_rcu(ep, rcu);
+}
+
+static void nsim_amt_set_free_rcu(struct rcu_head *head)
+{
+	struct nsim_amt_set *set = container_of(head, struct nsim_amt_set,
+						rcu);
+	u32 i;
+
+	for (i = 0; i < set->n; i++)
+		nsim_amt_ep_put(set->eps[i]);
+	kfree(set);
+}
+
+static void nsim_amt_set_free(struct nsim_amt_set *set)
+{
+	if (set)
+		call_rcu(&set->rcu, nsim_amt_set_free_rcu);
+}
+
+static int nsim_amt_bind(struct amt_offload_binding *b,
+			 struct amt_offload_caps *caps)
+{
+	struct nsim_amt *nam = nsim_amt_of(b);
+	struct nsim_amt_bind *nb;
+
+	nb = kzalloc_obj(*nb, GFP_KERNEL);
+	if (!nb)
+		return -ENOMEM;
+	nb->b = b;
+	INIT_LIST_HEAD(&nb->eps);
+	INIT_LIST_HEAD(&nb->grps);
+	b->priv = nb;
+
+	caps->families = AMT_OFFLOAD_FAMILY_V4 | AMT_OFFLOAD_FAMILY_V6;
+	caps->paths = BIT(AMT_OFFLOAD_PATH_TX);
+	caps->op_timeout_ms = READ_ONCE(nam->op_timeout_ms);
+	caps->max_endpoints = READ_ONCE(nam->max_endpoints);
+
+	mutex_lock(&nam->lock);
+	list_add_tail(&nb->node, &nam->binds);
+	mutex_unlock(&nam->lock);
+	return 0;
+}
+
+/* Drop every group and endpoint of @nb. Caller holds nam->lock; amt has
+ * stopped submitting packets for this binding.
+ */
+static void nsim_amt_bind_clear(struct nsim_amt *nam, struct nsim_amt_bind *nb)
+{
+	struct nsim_dev *nsim_dev = nam->ns->nsim_dev;
+	struct amt_offload_endpoint *ep, *et;
+	struct amt_offload_group *g, *gt;
+	struct nsim_amt_set *set;
+
+	list_for_each_entry_safe(g, gt, &nb->grps, node) {
+		set = nsim_amt_deref(nam, g->cur);
+		if (set)
+			atomic_sub(set->n, &nsim_dev->amt.members);
+		nsim_amt_set_free(set);
+		nsim_amt_set_free(nsim_amt_deref(nam, g->prev));
+		nsim_amt_set_free(g->staged);
+		list_del(&g->node);
+		atomic_dec(&nsim_dev->amt.groups);
+		kfree_rcu(g, rcu);
+	}
+	list_for_each_entry_safe(ep, et, &nb->eps, node) {
+		list_del(&ep->node);
+		nb->nr_eps--;
+		atomic_dec(&nsim_dev->amt.endpoints);
+		nsim_amt_ep_put(ep);
+	}
+}
+
+static void nsim_amt_unbind(struct amt_offload_binding *b)
+{
+	struct nsim_amt *nam = nsim_amt_of(b);
+	struct nsim_amt_bind *nb = b->priv;
+
+	mutex_lock(&nam->lock);
+	/* Empty unless amt broke the teardown order; then free, not leak. */
+	WARN_ON_ONCE(!list_empty(&nb->grps) || !list_empty(&nb->eps));
+	nsim_amt_bind_clear(nam, nb);
+	list_del(&nb->node);
+	mutex_unlock(&nam->lock);
+	kfree(nb);
+}
+
+static bool nsim_amt_ep_key_eq(const struct amt_offload_endpoint_key *a,
+			       const struct amt_offload_endpoint_key *b)
+{
+	if (a->family != b->family || a->port != b->port)
+		return false;
+	if (a->family == AF_INET6)
+		return ipv6_addr_equal(&a->addr.ip6, &b->addr.ip6);
+	return a->addr.ip4 == b->addr.ip4;
+}
+
+static int nsim_amt_endpoint_add(struct amt_offload_binding *b,
+				 const struct amt_offload_endpoint_key *key,
+				 struct amt_offload_endpoint **epp)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+	struct nsim_amt_bind *nb = b->priv;
+	struct amt_offload_endpoint *ep;
+	int err = 0;
+
+	list_for_each_entry(ep, &nb->eps, node) {
+		if (nsim_amt_ep_key_eq(&ep->key, key)) {
+			ep->adds++;
+			goto out;
+		}
+	}
+	if (nam->max_endpoints && nb->nr_eps >= nam->max_endpoints) {
+		err = -ENOSPC;
+		goto unlock;
+	}
+	ep = kzalloc_obj(*ep, GFP_KERNEL);
+	if (!ep) {
+		err = -ENOMEM;
+		goto unlock;
+	}
+	ep->key = *key;
+	refcount_set(&ep->refs, 1);
+	ep->adds = 1;
+	list_add_tail(&ep->node, &nb->eps);
+	nb->nr_eps++;
+	atomic_inc(&nam->ns->nsim_dev->amt.endpoints);
+out:
+	*epp = ep;
+unlock:
+	nsim_amt_op_end(nam);
+	return err;
+}
+
+static void nsim_amt_endpoint_del(struct amt_offload_binding *b,
+				  struct amt_offload_endpoint *ep)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+	struct nsim_amt_bind *nb = b->priv;
+
+	if (!--ep->adds) {
+		list_del(&ep->node);
+		nb->nr_eps--;
+		atomic_dec(&nam->ns->nsim_dev->amt.endpoints);
+		nsim_amt_ep_put(ep);
+	}
+	nsim_amt_op_end(nam);
+}
+
+static int nsim_amt_group_update(struct amt_offload_binding *b,
+				 const struct amt_offload_group_key *key,
+				 struct amt_offload_endpoint *const *members,
+				 unsigned int nr_members, u64 gen,
+				 struct amt_offload_group **grpp)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+	struct amt_offload_group *g = *grpp;
+	struct nsim_amt_bind *nb = b->priv;
+	struct nsim_amt_set *set;
+	int err = 0;
+	u32 i;
+
+	if (!g) {
+		g = kzalloc_obj(*g, GFP_KERNEL);
+		if (!g) {
+			err = -ENOMEM;
+			goto out;
+		}
+		g->key = *key;
+		list_add_tail(&g->node, &nb->grps);
+		atomic_inc(&nam->ns->nsim_dev->amt.groups);
+		*grpp = g;
+	}
+	set = kzalloc(struct_size(set, eps, nr_members), GFP_KERNEL);
+	if (!set) {
+		err = -ENOMEM;
+		goto out;
+	}
+	set->n = nr_members;
+	set->gen = gen;
+	for (i = 0; i < nr_members; i++) {
+		refcount_inc(&members[i]->refs);
+		set->eps[i] = members[i];
+	}
+	nsim_amt_set_free(g->staged);
+	g->staged = set;
+out:
+	nsim_amt_op_end(nam);
+	return err;
+}
+
+static int nsim_amt_group_commit(struct amt_offload_binding *b,
+				 struct amt_offload_group *g, u64 gen)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+	struct nsim_dev *nsim_dev = nam->ns->nsim_dev;
+	struct nsim_amt_set *cur, *prev;
+	int err = 0;
+
+	if (nam->fail_commit) {
+		err = -ENOSPC;
+		goto out;
+	}
+	if (!g->staged || g->staged->gen != gen) {
+		err = -EINVAL;
+		goto out;
+	}
+	cur = nsim_amt_deref(nam, g->cur);
+	prev = nsim_amt_deref(nam, g->prev);
+	atomic_add(g->staged->n, &nsim_dev->amt.members);
+	if (cur)
+		atomic_sub(cur->n, &nsim_dev->amt.members);
+	rcu_assign_pointer(g->prev, cur);
+	rcu_assign_pointer(g->cur, g->staged);
+	g->staged = NULL;
+	nsim_amt_set_free(prev);
+	nam->commits++;
+out:
+	nsim_amt_op_end(nam);
+	return err;
+}
+
+static void nsim_amt_group_disable(struct amt_offload_binding *b,
+				   struct amt_offload_group *g)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+	struct nsim_amt_set *set;
+
+	set = nsim_amt_deref(nam, g->cur);
+	if (set)
+		atomic_sub(set->n, &nam->ns->nsim_dev->amt.members);
+	nsim_amt_set_free(set);
+	nsim_amt_set_free(nsim_amt_deref(nam, g->prev));
+	nsim_amt_set_free(g->staged);
+	list_del(&g->node);
+	atomic_dec(&nam->ns->nsim_dev->amt.groups);
+	kfree_rcu(g, rcu);
+	nsim_amt_op_end(nam);
+}
+
+/* Softirq, under rcu_read_lock(). Copies go to the committed set of @gen
+ * only; a generation this provider never committed (or has already
+ * retired) is declined, never guessed at.
+ */
+static enum amt_offload_result
+nsim_amt_xmit(struct amt_offload_binding *b, struct amt_offload_group *g,
+	      struct sk_buff *skb, u64 gen)
+{
+	struct nsim_amt *nam = nsim_amt_of(b);
+	struct nsim_amt_set *set;
+	u32 i;
+
+	if (READ_ONCE(nam->busy)) {
+		atomic64_inc(&nam->tx_busy);
+		return AMT_OFFLOAD_BUSY;
+	}
+	set = rcu_dereference(g->cur);
+	if (!set || set->gen != gen) {
+		set = rcu_dereference(g->prev);
+		if (!set || set->gen != gen) {
+			atomic64_inc(&nam->tx_stale_gen);
+			return AMT_OFFLOAD_FALLBACK;
+		}
+	}
+	for (i = 0; i < set->n; i++) {
+		if (amt_offload_encap_xmit(b, &set->eps[i]->key, skb)) {
+			atomic64_inc(&nam->tx_dropped);
+			continue;
+		}
+		atomic64_inc(&set->eps[i]->copies);
+		atomic64_inc(&nam->tx_copies);
+	}
+	consume_skb(skb);
+	return AMT_OFFLOAD_CONSUMED;
+}
+
+/* Called under amt's offload mutex, like every programming op: from
+ * the reconciler with no other lock, or at teardown with RTNL held and,
+ * when the port is being deleted, the devlink instance lock too. The
+ * "amt" reporter has no .recover and no .dump (health.c), so reporting
+ * it takes no devlink lock in either case.
+ */
+static void nsim_amt_flush(struct amt_offload_binding *b)
+{
+	struct nsim_amt *nam = nsim_amt_op_begin(b);
+
+	nsim_amt_bind_clear(nam, b->priv);
+	nam->flushes++;
+	nsim_amt_op_end(nam);
+	devlink_health_report(nam->ns->nsim_dev->health.amt_reporter,
+			      "amt reset: provider tables flushed", NULL);
+}
+
+static const struct amt_offload_ops nsim_amt_ops = {
+	.bind		= nsim_amt_bind,
+	.unbind		= nsim_amt_unbind,
+	.endpoint_add	= nsim_amt_endpoint_add,
+	.endpoint_del	= nsim_amt_endpoint_del,
+	.group_update	= nsim_amt_group_update,
+	.group_commit	= nsim_amt_group_commit,
+	.group_disable	= nsim_amt_group_disable,
+	.xmit		= nsim_amt_xmit,
+	.flush		= nsim_amt_flush,
+};
+
+static void nsim_amt_show_ep(struct seq_file *m,
+			     const struct amt_offload_endpoint_key *key)
+{
+	if (key->family == AF_INET6)
+		seq_printf(m, "[%pI6c]:%u", &key->addr.ip6, ntohs(key->port));
+	else
+		seq_printf(m, "%pI4:%u", &key->addr.ip4, ntohs(key->port));
+}
+
+static void nsim_amt_show_set(struct seq_file *m, const char *name,
+			      const struct nsim_amt_set *set)
+{
+	u32 i;
+
+	if (!set)
+		return;
+	seq_printf(m, "  %s gen %llu members %u:", name, set->gen, set->n);
+	for (i = 0; i < set->n; i++) {
+		seq_putc(m, ' ');
+		nsim_amt_show_ep(m, &set->eps[i]->key);
+	}
+	seq_putc(m, '\n');
+}
+
+static int nsim_amt_tables_show(struct seq_file *m, void *unused)
+{
+	struct nsim_amt *nam = m->private;
+	struct amt_offload_endpoint *ep;
+	struct amt_offload_group *g;
+	struct nsim_amt_bind *nb;
+
+	mutex_lock(&nam->lock);
+	list_for_each_entry(nb, &nam->binds, node) {
+		seq_printf(m, "binding %s epoch %llu\n", netdev_name(nb->b->dev),
+			   nb->b->epoch);
+		list_for_each_entry(ep, &nb->eps, node) {
+			seq_puts(m, "endpoint ");
+			nsim_amt_show_ep(m, &ep->key);
+			seq_printf(m, " copies %llu\n",
+				   atomic64_read(&ep->copies));
+		}
+		list_for_each_entry(g, &nb->grps, node) {
+			if (g->key.family == AF_INET6)
+				seq_printf(m, "group %pI6c\n", &g->key.group.ip6);
+			else
+				seq_printf(m, "group %pI4\n", &g->key.group.ip4);
+			nsim_amt_show_set(m, "committed",
+					  nsim_amt_deref(nam, g->cur));
+			nsim_amt_show_set(m, "previous",
+					  nsim_amt_deref(nam, g->prev));
+			nsim_amt_show_set(m, "staged", g->staged);
+		}
+	}
+	mutex_unlock(&nam->lock);
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(nsim_amt_tables);
+
+static int nsim_amt_atomic64_get(void *data, u64 *val)
+{
+	*val = atomic64_read(data);
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(nsim_amt_atomic64_fops, nsim_amt_atomic64_get,
+			 NULL, "%llu\n");
+
+static void nsim_amt_debugfs_init(struct nsim_amt *nam)
+{
+	struct dentry *d;
+
+	d = debugfs_create_dir("amt", nam->ns->nsim_dev_port->ddir);
+	nam->ddir = d;
+	debugfs_create_file("tables", 0400, d, nam, &nsim_amt_tables_fops);
+	debugfs_create_file("tx_copies", 0400, d, &nam->tx_copies,
+			    &nsim_amt_atomic64_fops);
+	debugfs_create_file("tx_busy", 0400, d, &nam->tx_busy,
+			    &nsim_amt_atomic64_fops);
+	debugfs_create_file("tx_stale_gen", 0400, d, &nam->tx_stale_gen,
+			    &nsim_amt_atomic64_fops);
+	debugfs_create_file("tx_dropped", 0400, d, &nam->tx_dropped,
+			    &nsim_amt_atomic64_fops);
+	debugfs_create_u64("commits", 0400, d, &nam->commits);
+	debugfs_create_u64("flushes", 0400, d, &nam->flushes);
+	debugfs_create_bool("busy", 0600, d, &nam->busy);
+	debugfs_create_bool("fail_commit", 0600, d, &nam->fail_commit);
+	debugfs_create_u32("max_endpoints", 0600, d, &nam->max_endpoints);
+	debugfs_create_u32("stall_ms", 0600, d, &nam->stall_ms);
+	debugfs_create_u32("op_timeout_ms", 0600, d, &nam->op_timeout_ms);
+}
+
+int nsim_amt_init(struct netdevsim *ns)
+{
+	struct nsim_amt *nam;
+	int err;
+
+	nam = kzalloc_obj(*nam, GFP_KERNEL);
+	if (!nam)
+		return -ENOMEM;
+	nam->ns = ns;
+	mutex_init(&nam->lock);
+	INIT_LIST_HEAD(&nam->binds);
+	nam->op_timeout_ms = 1000;
+	nsim_amt_debugfs_init(nam);
+
+	rtnl_lock();
+	ns->amt = nam;
+	err = amt_offload_register(ns->netdev, &nsim_amt_ops);
+	if (err)
+		ns->amt = NULL;
+	rtnl_unlock();
+	if (err) {
+		debugfs_remove_recursive(nam->ddir);
+		mutex_destroy(&nam->lock);
+		kfree(nam);
+	}
+	return err;
+}
+
+void nsim_amt_uninit(struct netdevsim *ns)
+{
+	struct nsim_amt *nam = ns->amt;
+
+	if (!nam)
+		return;
+	rtnl_lock();
+	amt_offload_unregister(ns->netdev);
+	ns->amt = NULL;
+	rtnl_unlock();
+	debugfs_remove_recursive(nam->ddir);
+	mutex_destroy(&nam->lock);
+	kfree(nam);
+}
+
+/* amt forgets a provider when its lower device is unregistered, and
+ * moving a netdev to another netns unregisters it there first. Offer
+ * the port again once it is registered in its new netns.
+ */
+static int nsim_amt_netdev_event(struct notifier_block *nb,
+				 unsigned long event, void *ptr)
+{
+	struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+	struct netdevsim *ns;
+
+	if (event != NETDEV_REGISTER || !netdev_is_nsim(dev))
+		return NOTIFY_DONE;
+	ns = netdev_priv(dev);
+	if (ns->amt)
+		amt_offload_register(dev, &nsim_amt_ops);
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block nsim_amt_notifier = {
+	.notifier_call = nsim_amt_netdev_event,
+};
+
+int nsim_amt_module_init(void)
+{
+	return register_netdevice_notifier(&nsim_amt_notifier);
+}
+
+void nsim_amt_module_exit(void)
+{
+	unregister_netdevice_notifier(&nsim_amt_notifier);
+	rcu_barrier();	/* nsim_amt_set_free_rcu() is module text */
+}
diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
index f00fc2f9e..7fcdb0d67 100644
--- a/drivers/net/netdevsim/dev.c
+++ b/drivers/net/netdevsim/dev.c
@@ -444,6 +444,69 @@ static void nsim_dev_port_debugfs_exit(struct nsim_dev_port *nsim_dev_port)
 	debugfs_remove_recursive(nsim_dev_port->ddir);
 }
 
+static u64 nsim_dev_amt_endpoints_occ_get(void *priv)
+{
+	return atomic_read(&((struct nsim_dev *)priv)->amt.endpoints);
+}
+
+static u64 nsim_dev_amt_groups_occ_get(void *priv)
+{
+	return atomic_read(&((struct nsim_dev *)priv)->amt.groups);
+}
+
+static u64 nsim_dev_amt_members_occ_get(void *priv)
+{
+	return atomic_read(&((struct nsim_dev *)priv)->amt.members);
+}
+
+static u64 nsim_dev_amt_cmdq_occ_get(void *priv)
+{
+	return atomic_read(&((struct nsim_dev *)priv)->amt.cmdq);
+}
+
+static int
+nsim_dev_amt_resources_register(struct devlink *devlink,
+				const struct devlink_resource_size_params *p)
+{
+	static const struct {
+		const char *name;
+		u64 id;
+		devlink_resource_occ_get_t *occ_get;
+	} res[] = {
+		{ "endpoints", NSIM_RESOURCE_AMT_ENDPOINTS,
+		  nsim_dev_amt_endpoints_occ_get },
+		{ "groups", NSIM_RESOURCE_AMT_GROUPS,
+		  nsim_dev_amt_groups_occ_get },
+		{ "members", NSIM_RESOURCE_AMT_MEMBERS,
+		  nsim_dev_amt_members_occ_get },
+		{ "cmdq", NSIM_RESOURCE_AMT_CMDQ, nsim_dev_amt_cmdq_occ_get },
+	};
+	int err, i;
+
+	if (!IS_ENABLED(CONFIG_AMT))
+		return 0;
+	err = devl_resource_register(devlink, "amt", (u64)-1,
+				     NSIM_RESOURCE_AMT,
+				     DEVLINK_RESOURCE_ID_PARENT_TOP, p);
+	if (err) {
+		pr_err("Failed to register AMT top resource\n");
+		return err;
+	}
+	for (i = 0; i < ARRAY_SIZE(res); i++) {
+		err = devl_resource_register(devlink, res[i].name, (u64)-1,
+					     res[i].id, NSIM_RESOURCE_AMT, p);
+		if (err) {
+			pr_err("Failed to register AMT %s resource\n",
+			       res[i].name);
+			return err;
+		}
+		devl_resource_occ_get_register(devlink, res[i].id,
+					       res[i].occ_get,
+					       devlink_priv(devlink));
+	}
+	return 0;
+}
+
 static int nsim_dev_resources_register(struct devlink *devlink)
 {
 	struct devlink_resource_size_params params = {
@@ -514,6 +577,11 @@ static int nsim_dev_resources_register(struct devlink *devlink)
 		pr_err("Failed to register NEXTHOPS resource\n");
 		goto err_out;
 	}
+
+	/* AMT relay offload tables, owned by the ports' provider (amt.c) */
+	err = nsim_dev_amt_resources_register(devlink, &params);
+	if (err)
+		goto err_out;
 	return 0;
 
 err_out:
diff --git a/drivers/net/netdevsim/health.c b/drivers/net/netdevsim/health.c
index 3bd0e7a48..f6bc6cb51 100644
--- a/drivers/net/netdevsim/health.c
+++ b/drivers/net/netdevsim/health.c
@@ -175,6 +175,16 @@ static const struct file_operations nsim_dev_health_break_fops = {
 	.owner = THIS_MODULE,
 };
 
+/* Reported by the AMT relay offload provider (amt.c) each time amt resets
+ * it. Recovery is amt's own replay, so there is no .recover, and no .dump:
+ * with neither, devlink_health_report() never takes the devlink instance
+ * lock, which amt's ->flush() may not take (amt holds its offload mutex,
+ * below rtnl, and the instance lock sits above rtnl).
+ */
+static const struct devlink_health_reporter_ops nsim_dev_amt_reporter_ops = {
+	.name = "amt",
+};
+
 int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
 {
 	struct nsim_dev_health *health = &nsim_dev->health;
@@ -196,10 +206,21 @@ int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
 		goto err_empty_reporter_destroy;
 	}
 
+	if (IS_ENABLED(CONFIG_AMT)) {
+		health->amt_reporter =
+			devl_health_reporter_create(devlink,
+						    &nsim_dev_amt_reporter_ops,
+						    health);
+		if (IS_ERR(health->amt_reporter)) {
+			err = PTR_ERR(health->amt_reporter);
+			goto err_dummy_reporter_destroy;
+		}
+	}
+
 	health->ddir = debugfs_create_dir("health", nsim_dev->ddir);
 	if (IS_ERR(health->ddir)) {
 		err = PTR_ERR(health->ddir);
-		goto err_dummy_reporter_destroy;
+		goto err_amt_reporter_destroy;
 	}
 
 	health->recovered_break_msg = NULL;
@@ -213,6 +234,9 @@ int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
 			    &health->fail_recover);
 	return 0;
 
+err_amt_reporter_destroy:
+	if (health->amt_reporter)
+		devl_health_reporter_destroy(health->amt_reporter);
 err_dummy_reporter_destroy:
 	devl_health_reporter_destroy(health->dummy_reporter);
 err_empty_reporter_destroy:
@@ -226,6 +250,8 @@ void nsim_dev_health_exit(struct nsim_dev *nsim_dev)
 
 	debugfs_remove_recursive(health->ddir);
 	kfree(health->recovered_break_msg);
+	if (health->amt_reporter)
+		devl_health_reporter_destroy(health->amt_reporter);
 	devl_health_reporter_destroy(health->dummy_reporter);
 	devl_health_reporter_destroy(health->empty_reporter);
 }
diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c
index a75076891..4294c026d 100644
--- a/drivers/net/netdevsim/netdev.c
+++ b/drivers/net/netdevsim/netdev.c
@@ -1075,6 +1075,10 @@ static int nsim_init_netdevsim(struct netdevsim *ns)
 	if (err)
 		goto err_unregister_netdev;
 
+	err = nsim_amt_init(ns);
+	if (err)
+		goto err_psp_uninit;
+
 	if (IS_ENABLED(CONFIG_DEBUG_NET)) {
 		ns->nb.notifier_call = netdev_debug_event;
 		if (register_netdevice_notifier_dev_net(ns->netdev, &ns->nb,
@@ -1084,6 +1088,8 @@ static int nsim_init_netdevsim(struct netdevsim *ns)
 
 	return 0;
 
+err_psp_uninit:
+	nsim_psp_uninit(ns);
 err_unregister_netdev:
 	rtnl_lock();
 	peer = rtnl_dereference(ns->peer);
@@ -1182,8 +1188,10 @@ void nsim_destroy(struct netdevsim *ns)
 		unregister_netdevice_notifier_dev_net(ns->netdev, &ns->nb,
 						      &ns->nn);
 
-	if (nsim_dev_port_is_pf(ns->nsim_dev_port))
+	if (nsim_dev_port_is_pf(ns->nsim_dev_port)) {
+		nsim_amt_uninit(ns);
 		nsim_psp_uninit(ns);
+	}
 
 	rtnl_lock();
 	peer = rtnl_dereference(ns->peer);
@@ -1238,10 +1246,14 @@ static int __init nsim_module_init(void)
 {
 	int err;
 
-	err = nsim_dev_init();
+	err = nsim_amt_module_init();
 	if (err)
 		return err;
 
+	err = nsim_dev_init();
+	if (err)
+		goto err_amt_exit;
+
 	err = nsim_bus_init();
 	if (err)
 		goto err_dev_exit;
@@ -1256,6 +1268,8 @@ static int __init nsim_module_init(void)
 	nsim_bus_exit();
 err_dev_exit:
 	nsim_dev_exit();
+err_amt_exit:
+	nsim_amt_module_exit();
 	return err;
 }
 
@@ -1264,6 +1278,7 @@ static void __exit nsim_module_exit(void)
 	rtnl_link_unregister(&nsim_link_ops);
 	nsim_bus_exit();
 	nsim_dev_exit();
+	nsim_amt_module_exit();
 }
 
 module_init(nsim_module_init);
diff --git a/drivers/net/netdevsim/netdevsim.h b/drivers/net/netdevsim/netdevsim.h
index d909c4160..549afee0d 100644
--- a/drivers/net/netdevsim/netdevsim.h
+++ b/drivers/net/netdevsim/netdevsim.h
@@ -127,6 +127,8 @@ struct netdevsim {
 		u32 assoc_cnt;
 	} psp;
 
+	struct nsim_amt *amt;
+
 	struct nsim_bus_dev *nsim_bus_dev;
 
 	struct bpf_prog	*bpf_offloaded;
@@ -232,6 +234,11 @@ enum nsim_resource_id {
 	NSIM_RESOURCE_IPV6_FIB,
 	NSIM_RESOURCE_IPV6_FIB_RULES,
 	NSIM_RESOURCE_NEXTHOPS,
+	NSIM_RESOURCE_AMT,
+	NSIM_RESOURCE_AMT_ENDPOINTS,
+	NSIM_RESOURCE_AMT_GROUPS,
+	NSIM_RESOURCE_AMT_MEMBERS,
+	NSIM_RESOURCE_AMT_CMDQ,
 };
 
 enum nsim_port_resource_id {
@@ -241,6 +248,7 @@ enum nsim_port_resource_id {
 struct nsim_dev_health {
 	struct devlink_health_reporter *empty_reporter;
 	struct devlink_health_reporter *dummy_reporter;
+	struct devlink_health_reporter *amt_reporter;
 	struct dentry *ddir;
 	char *recovered_break_msg;
 	u32 binary_len;
@@ -351,6 +359,13 @@ struct nsim_dev {
 	bool fail_reload;
 	struct devlink_region *dummy_region;
 	struct nsim_dev_health health;
+	/* AMT relay offload tables of every port, for devlink resources */
+	struct {
+		atomic_t endpoints;
+		atomic_t groups;
+		atomic_t members;
+		atomic_t cmdq;
+	} amt;
 	struct nsim_dev_hwstats hwstats;
 	struct flow_action_cookie *fa_cookie;
 	spinlock_t fa_cookie_lock; /* protects fa_cookie */
@@ -472,6 +487,18 @@ nsim_psp_handle_ext(struct sk_buff *skb, struct skb_ext *psp_ext) {}
 int nsim_setup_tc(struct net_device *dev, enum tc_setup_type type,
 		  void *type_data);
 
+#if IS_ENABLED(CONFIG_AMT)
+int nsim_amt_init(struct netdevsim *ns);
+void nsim_amt_uninit(struct netdevsim *ns);
+int nsim_amt_module_init(void);
+void nsim_amt_module_exit(void);
+#else
+static inline int nsim_amt_init(struct netdevsim *ns) { return 0; }
+static inline void nsim_amt_uninit(struct netdevsim *ns) {}
+static inline int nsim_amt_module_init(void) { return 0; }
+static inline void nsim_amt_module_exit(void) {}
+#endif
+
 struct nsim_bus_dev {
 	struct device dev;
 	struct list_head list;
-- 
2.43.0

[RFC PATCH net-next 8/9] selftests: net: add amt_offload.sh for AMT relay offload

[Omar Ramadan]

Exercise the AMT relay replication offload contract against netdevsim's
TX reference provider. A relay in offload auto mode transmits on one
netdevsim port; four in-tree AMT gateways, each in its own netns (an amt
device binds relay_port on INADDR_ANY, so two gateways cannot share
one), reach its peer through a bridge. Copies are counted per gateway on
the decapsulating amtg device, and hardware copies on the provider's own
counter, so the eleven assertions are about what actually reached each
gateway:

 1. no membership: no copy anywhere
 2. one copy per committed endpoint per packet, all from the provider,
    and the provider's devlink resources agree
 3. members joining under traffic (generation replace): stable members
    get exactly one copy of every packet, and every copy comes from the
    provider
 4. xmit BUSY: exactly one software copy set, counted
 5. forwarding is per group: an INCLUDE(S1) member still receives S2
 6. a programming op overruns op_timeout_ms: amt resets the provider,
    ignores the stale result, replays, and copies stay at one; the
    provider's health reporter records it
 7. partial commit (group_update succeeds, group_commit fails): no
    hardware copy, the group is served in software
 8. every member leaves: once the relay times their sources out, the
    group is disabled, no copy anywhere
 9. offload required with no provider: refused with an extack, counted
10. the peer unlinked under traffic, relinked, then the relay's netdevsim
    deleted under traffic: ordered teardown, no splat
11. a lower carrying two amt devices deleted: both go with it, no splat
    (runs first, in its own netns, independent of the offload setup;
    one dmesg snapshot after its netns teardown ends its splat scan
    and starts case 10's)

No IGMPv3 leave takes effect at once on this base: a BLOCK or TO_IN
leaves the timer of each source it drops alone, so the relay keeps the
source until that timer expires, at least a GMI (260 s by default)
after the last report that refreshed it. So case 3 drives its
generation replaces with joins alone, and case 8 waits out the GMI.

The test SKIPs, loudly, when ip cannot set the offload mode or the
netdevsim has no provider, rather than silently testing software only.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 tools/testing/selftests/net/Makefile       |   1 +
 tools/testing/selftests/net/amt_offload.sh | 608 +++++++++++++++++++++
 tools/testing/selftests/net/config         |   2 +
 3 files changed, 611 insertions(+)
 create mode 100755 tools/testing/selftests/net/amt_offload.sh

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 5db68de0b..2d6c6de0d 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -11,6 +11,7 @@ TEST_PROGS := \
 	amt.sh \
 	amt_v6.sh \
 	amt_gw_v6.sh \
+	amt_offload.sh \
 	amt_scale.sh \
 	arp_ndisc_evict_nocarrier.sh \
 	arp_ndisc_untracked_subnets.sh \
diff --git a/tools/testing/selftests/net/amt_offload.sh b/tools/testing/selftests/net/amt_offload.sh
new file mode 100755
index 000000000..4e00fba0b
--- /dev/null
+++ b/tools/testing/selftests/net/amt_offload.sh
@@ -0,0 +1,608 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# AMT relay replication offload, against netdevsim's TX reference provider.
+#
+# netdevsim keeps its own endpoint and group tables and emits each copy
+# through amt_offload_encap_xmit() out of the relay's stream device to its
+# peer, so a divergence between what amt wants and what the provider
+# committed shows up as a wrong copy count at the gateways. Copies are
+# counted per gateway on the decapsulating amtg device (AF_PACKET), and
+# hardware copies on the provider's own counter in debugfs.
+#
+#   RELAY netns                                GATEWAY netns
+#   src0 (S1 172.17.0.2, S2 172.17.0.3)
+#     | sender, IP_MULTICAST_IF amtr
+#   amtr (relay, offload auto)
+#     | stream_dev
+#   nsim_r 10.0.0.2/24  <== netdevsim peer link ==>  nsim_g -- br0
+#                                                              |
+#                                     veth to GATEWAY netns i = 1..4:
+#                                     gw<i> 10.0.0.1<i>/24, amtg<i>
+#
+# One gateway per netns: an amt device binds its UDP socket to relay_port
+# on INADDR_ANY in its netns, gateways included, so a second gateway in
+# the same netns fails to come up with EADDRINUSE.
+#
+# The source sits in the relay netns and transmits on amtr directly: how
+# multicast reaches amtr (mroute) is not under test, amt_dev_xmit is.
+#
+# Eleven assertions, each printing one "TEST: ... [ OK ]" line; the run
+# passes only if all eleven printed OK.
+
+source "$(dirname "$0")"/lib.sh
+
+readonly RELAY=$(mktemp -u amtofl-r-XXXXXXXX)
+readonly GW=$(mktemp -u amtofl-g-XXXXXXXX)
+readonly TMP=$(mktemp -d)
+readonly G1=239.1.1.1 G2=239.1.1.2 G3=239.1.1.3 G4=232.1.1.4 G5=239.1.1.5
+readonly S1=172.17.0.2 S2=172.17.0.3
+readonly K=50
+NSIM_R_ID=$((256 + RANDOM % 256))
+NSIM_G_ID=$((NSIM_R_ID + 256))
+readonly NSIM_SYS=/sys/bus/netdevsim
+readonly DFS=/sys/kernel/debug/netdevsim/netdevsim$NSIM_R_ID/ports/0/amt
+readonly DL=netdevsim/netdevsim$NSIM_R_ID
+ERR=$ksft_skip
+OKS=0
+PIDS=()
+gwns() { echo "$GW-$1"; }
+declare -A JOIN
+
+cleanup()
+{
+	for p in "${PIDS[@]}" "${JOIN[@]}"; do
+		kill "$p" 2>/dev/null
+	done
+	wait 2>/dev/null
+	ip netns del "$RELAY" 2>/dev/null
+	ip netns del "$GW" 2>/dev/null
+	for i in 1 2 3 4 mu; do
+		ip netns del "$(gwns $i)" 2>/dev/null
+	done
+	echo "$NSIM_R_ID" > $NSIM_SYS/del_device 2>/dev/null
+	echo "$NSIM_G_ID" > $NSIM_SYS/del_device 2>/dev/null
+	rm -rf "$TMP"
+	exit $ERR
+}
+
+skip()
+{
+	echo "SKIP: $*"
+	# A failure already printed (assertion 11 runs before the provider
+	# checks) must not be reported as a skip.
+	[ "$ERR" = 1 ] || ERR=$ksft_skip
+	exit $ERR
+}
+
+ok()
+{
+	printf "TEST: %-62s  [ OK ]\n" "$1"
+	OKS=$((OKS + 1))
+}
+
+fail()
+{
+	printf "TEST: %-62s  [FAIL]\n" "$1"
+	shift
+	[ $# -gt 0 ] && echo "      $*"
+	ERR=1
+	return 0
+}
+
+# --- helpers --------------------------------------------------------------
+
+cat > "$TMP/join.py" <<'EOF'
+# join.py IFNAME GROUP [SOURCE]: hold an ASM or SSM membership on IFNAME
+# until killed; leaving is closing the socket.
+import signal, socket, struct, sys, time
+ifn, grp = sys.argv[1], sys.argv[2]
+src = sys.argv[3] if len(sys.argv) > 3 else None
+def sin(a):
+    return struct.pack('=HH4s120x', socket.AF_INET, 0, socket.inet_aton(a))
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+req = struct.pack('=I4x', socket.if_nametoindex(ifn)) + sin(grp)
+if src:
+    s.setsockopt(socket.IPPROTO_IP, 46, req + sin(src)) # MCAST_JOIN_SOURCE_GROUP
+else:
+    s.setsockopt(socket.IPPROTO_IP, 42, req)            # MCAST_JOIN_GROUP
+signal.signal(signal.SIGTERM, lambda *a: sys.exit(0))
+while True:
+    time.sleep(3600)
+EOF
+
+cat > "$TMP/send.py" <<'EOF'
+# send.py IFNAME SRC GROUP COUNT INTERVAL: COUNT UDP datagrams to GROUP:5000
+# from SRC out of IFNAME. Send errors (the device going away) are ignored.
+import socket, struct, sys, time
+ifn, src, grp, n, ivl = sys.argv[1:6]
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+s.bind((src, 0))
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF,
+             struct.pack('=4s4si', bytes(4), bytes(4),
+                         socket.if_nametoindex(ifn)))
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 8)
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_LOOP, 0)
+for i in range(int(n)):
+    try:
+        s.sendto(b'%08d' % i + bytes(56), (grp, 5000))
+    except OSError:
+        pass
+    time.sleep(float(ivl))
+EOF
+
+cat > "$TMP/count.py" <<'EOF'
+# count.py IFNAME...: count received UDP datagrams to port 5000 per
+# (interface, source, group) until SIGTERM, then read what is already
+# queued and print one line per key.
+import collections, select, signal, socket, sys
+socks = {}
+for ifn in sys.argv[1:]:
+    s = socket.socket(socket.AF_PACKET, socket.SOCK_DGRAM, socket.htons(0x0800))
+    s.bind((ifn, 0x0800))
+    socks[s] = ifn
+cnt = collections.Counter()
+done = False
+def stop(*a):
+    global done
+    done = True
+signal.signal(signal.SIGTERM, stop)
+sys.stdout.write('READY\n'); sys.stdout.flush()
+while True:
+    r = select.select(list(socks), [], [], 0 if done else 0.1)[0]
+    if not r and done:
+        break
+    for s in r:
+        d, a = s.recvfrom(4096)
+        if a[2] == socket.PACKET_OUTGOING or len(d) < 28 or d[9] != 17:
+            continue
+        hl = (d[0] & 15) * 4
+        if int.from_bytes(d[hl + 2:hl + 4], 'big') != 5000:
+            continue
+        cnt[(socks[s], socket.inet_ntoa(d[12:16]),
+             socket.inet_ntoa(d[16:20]))] += 1
+for (ifn, src, grp), n in sorted(cnt.items()):
+    print(ifn, src, grp, n)
+EOF
+
+dfs() { cat "$DFS/$1"; }
+dfs_set() { echo "$2" > "$DFS/$1"; }
+rsys() { ip netns exec "$RELAY" cat "/sys/class/net/amtr/offload/$1"; }
+
+# Committed member count of group $1 in the provider's own table, or 0.
+hw_members()
+{
+	awk -v g="group $1" '
+		$0 == g { in_g = 1; next }
+		/^group / || /^endpoint / || /^binding / { in_g = 0 }
+		in_g && $1 == "committed" { n = $5; sub(":", "", n); print n; f = 1; exit }
+		END { if (!f) print 0 }' "$DFS/tables"
+}
+
+hw_group_present()
+{
+	grep -qx "group $1" "$DFS/tables"
+}
+
+# wait_for SECONDS CMD...: poll CMD every 0.2s.
+wait_for()
+{
+	local t=$(( $1 * 5 ))
+
+	shift
+	while [ $t -gt 0 ]; do
+		"$@" && return 0
+		sleep 0.2
+		t=$((t - 1))
+	done
+	return 1
+}
+
+members_eq() { [ "$(hw_members "$1")" = "$2" ]; }
+group_gone() { ! hw_group_present "$1"; }
+sysfs_ge() { [ "$(rsys "$1")" -ge "$2" ]; }
+
+# Members that later leave join source-specific, so a relay that applies
+# INCLUDE leaves at once drops the group at once. This base does not: a
+# BLOCK or TO_IN leaves the timer of each source it drops alone, so the
+# relay keeps the source until that timer expires, at least a GMI after
+# the last report that refreshed it, and deletes the group with its last
+# source (an ASM leave waits out the group timer the same way). So case 8
+# waits out the GMI, and case 3 drives its generation replaces with joins
+# alone.
+join()	# join KEY IFNAME GROUP [SOURCE]
+{
+	local key=$1
+
+	shift
+	ip netns exec "$(gwns "${1#amtg}")" python3 "$TMP/join.py" "$@" &
+	JOIN[$key]=$!
+}
+
+leave()
+{
+	kill "${JOIN[$1]}" 2>/dev/null
+	wait "${JOIN[$1]}" 2>/dev/null
+	unset "JOIN[$1]"
+}
+
+# measure SRC GROUP COUNT INTERVAL [HOOK]: send, count per gateway, and
+# the provider's hardware copies. Fills CNT[amtgN] (copies from SRC to
+# GROUP) and HW (tx_copies delta). The counters run until a second after
+# the last packet is sent, however long sending takes. HOOK, a function
+# that must not block, runs once every counter is ready, just before the
+# first packet.
+declare -A CNT
+HW=0
+measure()
+{
+	local src=$1 grp=$2 n=$3 ivl=$4 hw0 line i cpids=()
+
+	hw0=$(dfs tx_copies)
+	for i in 1 2 3 4; do
+		ip netns exec "$(gwns $i)" python3 "$TMP/count.py" amtg$i \
+			> "$TMP/count$i.out" &
+		cpids+=($!)
+	done
+	for i in 1 2 3 4; do
+		wait_for 5 grep -q READY "$TMP/count$i.out"
+	done
+	[ -z "${5:-}" ] || "$5"
+	ip netns exec "$RELAY" python3 "$TMP/send.py" amtr "$src" "$grp" "$n" \
+		"$ivl"
+	sleep 1
+	kill -TERM "${cpids[@]}" 2>/dev/null
+	wait "${cpids[@]}"
+	HW=$(( $(dfs tx_copies) - hw0 ))
+	CNT=([amtg1]=0 [amtg2]=0 [amtg3]=0 [amtg4]=0)
+	while read -r ifn s g c; do
+		[ "$s" = "$src" ] && [ "$g" = "$grp" ] && CNT[$ifn]=$c
+	done < <(cat "$TMP"/count[1-4].out | grep -v READY)
+}
+
+counts() { echo "amtg1..4=${CNT[amtg1]},${CNT[amtg2]},${CNT[amtg3]},${CNT[amtg4]} hw=$HW"; }
+
+dl_occ()
+{
+	devlink -j resource show "$DL" | jq -r --arg r "$1" \
+		'.resources[][] | select(.name == "amt") | .resources[] |
+		 select(.name == $r) | .occ'
+}
+
+# --- preflight ------------------------------------------------------------
+
+[ "$(id -u)" -eq 0 ] || skip "needs root"
+for c in python3 jq devlink; do
+	command -v $c > /dev/null || skip "$c not found"
+done
+# A guest whose ip cannot set the mode would silently test software only.
+ip link help amt 2>&1 | grep -q 'offload OFFLOAD' ||
+	skip "ip lacks IFLA_AMT_OFFLOAD_MODE (iproute2 amt offload patch)"
+modprobe amt 2>/dev/null
+modprobe netdevsim 2>/dev/null
+[ -w $NSIM_SYS/new_device ] || skip "netdevsim not available"
+
+trap cleanup EXIT
+DMESG0=$(dmesg | wc -l)
+
+# --- 11. a lower carrying two amt devices is deleted ------------------------
+# Not offload, so it runs first, in its own netns, and is not lost when
+# the offload setup below fails: the amt notifier must take down every
+# amt device on an unregistering lower, not just the first.
+# One left linked trips the upper-dev WARN in
+# unregister_netdevice_many_notify() and a kernfs upper_* splat. The
+# devices stay down: two gateways cannot both bind relay_port in one
+# netns, and linking is all this needs.
+MU=$(gwns mu)
+ip netns add "$MU"
+ip -n "$MU" link add mu0 type dummy
+ip -n "$MU" addr add 10.9.0.1/24 dev mu0
+for d in mu_a mu_b; do
+	ip -n "$MU" link add $d type amt mode gateway local 10.9.0.1 \
+		discovery 10.9.0.2 dev mu0
+done
+made=$(ip -n "$MU" -o link show type amt | wc -l)
+ip -n "$MU" link del mu0
+left=$(ip -n "$MU" -o link show type amt | wc -l)
+ip netns del "$MU"
+# The netns teardown is asynchronous: give it a second. Then one dmesg
+# snapshot ends this case's scan and starts case 10's, so the two scans
+# tile and a splat this case provokes fails this case only.
+sleep 1
+dm=$(dmesg)
+DMESG1=$(printf '%s\n' "$dm" | wc -l)
+splat=$(printf '%s\n' "$dm" | tail -n +$((DMESG0 + 1)) |
+	grep -E 'WARNING:|BUG:|Oops|kernfs: can not remove|refcount_t|use-after-free')
+if [ "$made" -eq 2 ] && [ "$left" -eq 0 ] && [ -z "$splat" ]; then
+	ok "lower carrying two amt devices deleted: both gone"
+else
+	fail "lower carrying two amt devices deleted" \
+		"made=$made left=$left splat: $splat"
+fi
+
+echo "$NSIM_R_ID" > $NSIM_SYS/new_device
+echo "$NSIM_G_ID" > $NSIM_SYS/new_device
+udevadm settle 2>/dev/null
+[ -d "$DFS" ] || skip "netdevsim has no AMT offload provider ($DFS)"
+NR=$(ls $NSIM_SYS/devices/netdevsim$NSIM_R_ID/net)
+NG=$(ls $NSIM_SYS/devices/netdevsim$NSIM_G_ID/net)
+
+set -e
+ip netns add "$RELAY"
+ip netns add "$GW"
+ip link set "$NR" netns "$RELAY"
+ip link set "$NG" netns "$GW"
+ip -n "$RELAY" link set "$NR" name nsim_r
+ip -n "$GW" link set "$NG" name nsim_g
+for ns in "$RELAY" "$GW"; do
+	ip -n "$ns" link set lo up
+done
+ip -n "$RELAY" addr add 10.0.0.2/24 dev nsim_r
+ip -n "$RELAY" link set nsim_r up
+ip -n "$GW" link set nsim_g up
+
+exec {FD_R}<"/var/run/netns/$RELAY"
+exec {FD_G}<"/var/run/netns/$GW"
+IFX_R=$(ip netns exec "$RELAY" cat /sys/class/net/nsim_r/ifindex)
+IFX_G=$(ip netns exec "$GW" cat /sys/class/net/nsim_g/ifindex)
+echo "$FD_R:$IFX_R $FD_G:$IFX_G" > $NSIM_SYS/link_device
+
+ip -n "$RELAY" link add src0 type dummy
+ip -n "$RELAY" addr add $S1/24 dev src0
+ip -n "$RELAY" addr add $S2/24 dev src0
+ip -n "$RELAY" link set src0 up
+ip -n "$RELAY" link add amtr type amt mode relay local 10.0.0.2 dev nsim_r \
+	relay_port 2268 offload auto
+ip -n "$RELAY" link set amtr up
+ip -n "$GW" link add br0 type bridge
+ip -n "$GW" link set nsim_g master br0
+ip -n "$GW" link set br0 up
+for i in 1 2 3 4; do
+	ns=$(gwns $i)
+	ip netns add "$ns"
+	# No IPv6 on the gateways: amtg would join solicited-node groups over
+	# the tunnel and the relay would offload them alongside G1..G4.
+	ip netns exec "$ns" sysctl -qw net.ipv6.conf.default.disable_ipv6=1 \
+		2>/dev/null || true
+	ip -n "$ns" link set lo up
+	ip -n "$GW" link add br_g$i type veth peer name gw$i netns "$ns"
+	ip -n "$GW" link set br_g$i master br0 up
+	ip -n "$ns" addr add 10.0.0.1$i/24 dev gw$i
+	ip -n "$ns" link set gw$i up
+	ip -n "$ns" link add amtg$i type amt mode gateway local 10.0.0.1$i \
+		discovery 10.0.0.2 dev gw$i gateway_port 2268 relay_port 2268
+	ip -n "$ns" addr add 192.168.$i.1/32 dev amtg$i
+	ip -n "$ns" link set amtg$i up
+done
+set +e
+
+discovered()
+{
+	local i
+
+	for i in 1 2 3 4; do
+		[ "$(ip -n "$(gwns $i)" -d -j link show amtg$i |
+		     jq -r '.[0].linkinfo.info_data.remote')" = 10.0.0.2 ] ||
+			return 1
+	done
+}
+wait_for 30 discovered || { fail "setup" "gateways never discovered the relay"; exit; }
+
+mode=$(ip -n "$RELAY" -d -j link show amtr | jq -r '.[0].linkinfo.info_data.offload')
+[ "$mode" = auto ] && [ "$(rsys bound)" = 1 ] ||
+	{ fail "setup" "relay offload mode '$mode', bound $(rsys bound)"; exit; }
+ERR=0
+
+# --- 1. no membership, nothing committed: no copies anywhere ----------------
+measure $S1 $G1 $K 0.01
+if [ "$HW" -eq 0 ] && [ $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) -eq 0 ]; then
+	ok "0 copies before any group is committed"
+else
+	fail "0 copies before any group is committed" "$(counts)"
+fi
+
+# --- 2. one hardware copy per committed endpoint per packet -----------------
+for i in 1 2 3; do
+	join a$i amtg$i $G1 $S1
+done
+if ! wait_for 40 members_eq $G1 3; then
+	fail "1 copy per endpoint per packet" \
+		"G1 never committed with 3 members: $(cat "$DFS/tables")"
+else
+	measure $S1 $G1 $K 0.01
+	if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+	   [ "${CNT[amtg3]}" -eq $K ] && [ "${CNT[amtg4]}" -eq 0 ] &&
+	   [ "$HW" -eq $((3 * K)) ] && [ "$(dl_occ endpoints)" = 3 ] &&
+	   [ "$(dl_occ groups)" = 1 ] && [ "$(dl_occ members)" = 3 ]; then
+		ok "1 copy per endpoint per packet, all from the provider"
+	else
+		fail "1 copy per endpoint per packet" "$(counts)" \
+			"devlink endpoints/groups/members:" \
+			"$(dl_occ endpoints)/$(dl_occ groups)/$(dl_occ members)"
+	fi
+fi
+
+# --- 3. generation replace under traffic: no duplicate, no loss --------------
+# G5 has two stable members; two more join while it is sent, 3 s and 6 s
+# into a 12 s stream, so each join is a generation replace under traffic.
+late_joins()
+{
+	ip netns exec "$(gwns 3)" bash -c \
+		"sleep 3; exec python3 $TMP/join.py amtg3 $G5 $S1" &
+	JOIN[l3]=$!
+	ip netns exec "$(gwns 4)" bash -c \
+		"sleep 6; exec python3 $TMP/join.py amtg4 $G5 $S1" &
+	JOIN[l4]=$!
+}
+join l1 amtg1 $G5 $S1
+join l2 amtg2 $G5 $S1
+if ! wait_for 40 members_eq $G5 2; then
+	fail "generation replace under traffic" \
+		"G5 never committed with 2 members: $(cat "$DFS/tables")"
+else
+	c0=$(dfs commits)
+	st0=$(dfs tx_stale_gen)
+	measure $S1 $G5 600 0.02 late_joins
+	gens=$(( $(dfs commits) - c0 ))
+	stale=$(( $(dfs tx_stale_gen) - st0 ))
+	# The stable members got every packet once, the late members some
+	# but not all of them, and every copy came from the provider (HW is
+	# their sum), none from a software fallback on a stale generation.
+	if [ "${CNT[amtg1]}" -eq 600 ] && [ "${CNT[amtg2]}" -eq 600 ] &&
+	   [ "${CNT[amtg3]}" -gt 0 ] && [ "${CNT[amtg3]}" -lt 600 ] &&
+	   [ "${CNT[amtg4]}" -gt 0 ] && [ "${CNT[amtg4]}" -lt 600 ] &&
+	   [ "$HW" -eq $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) ] &&
+	   [ $gens -ge 2 ] && [ $stale -eq 0 ]; then
+		ok "generation replace: stable members get exactly 1 copy"
+	else
+		fail "generation replace under traffic" \
+			"$(counts) generations=$gens stale_gen=+$stale"
+	fi
+fi
+
+# --- 4. BUSY: exactly one software copy set -----------------------------------
+b0=$(rsys busy_fallback)
+t0=$(dfs tx_busy)
+dfs_set busy 1
+measure $S1 $G1 $K 0.01
+dfs_set busy 0
+if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+   [ "${CNT[amtg3]}" -eq $K ] && [ "$HW" -eq 0 ] &&
+   [ $(( $(rsys busy_fallback) - b0 )) -eq $K ] && [ $(( $(dfs tx_busy) - t0 )) -eq $K ]; then
+	ok "BUSY: exactly one software copy set per packet"
+else
+	fail "BUSY: exactly one software copy set" "$(counts)" \
+		"busy_fallback +$(( $(rsys busy_fallback) - b0 ))"
+fi
+
+# --- 5. forwarding is per group: INCLUDE(S1) still receives S2 ----------------
+join s1 amtg1 $G4 $S1
+if ! wait_for 40 members_eq $G4 1; then
+	fail "INCLUDE(S1) member receives S2" "G4 never committed: $(cat "$DFS/tables")"
+else
+	measure $S2 $G4 $K 0.01
+	if [ "${CNT[amtg1]}" -eq $K ] && [ "$HW" -eq $K ]; then
+		ok "INCLUDE(S1) member still receives S2, in hardware"
+	else
+		fail "INCLUDE(S1) member receives S2" "$(counts)"
+	fi
+fi
+
+# --- 6. op overruns op_timeout_ms: reset, stale results ignored, no dup -------
+r0=$(rsys resets)
+s0=$(rsys stale_completions)
+f0=$(dfs flushes)
+h0=$(devlink -j health show "$DL" reporter amt | jq '.health[][0].error')
+dfs_set stall_ms 1500
+join r2 amtg2 $G3
+wait_for 30 sysfs_ge resets $((r0 + 1))
+dfs_set stall_ms 0
+if ! wait_for 40 members_eq $G3 1; then
+	fail "reset: stale completion ignored" \
+		"G3 never committed after reset: $(cat "$DFS/tables")"
+else
+	measure $S1 $G3 $K 0.01
+	h1=$(devlink -j health show "$DL" reporter amt | jq '.health[][0].error')
+	if [ "${CNT[amtg2]}" -eq $K ] && [ "$HW" -eq $K ] &&
+	   [ "$(rsys resets)" -gt "$r0" ] && [ "$(rsys stale_completions)" -gt "$s0" ] &&
+	   [ "$(dfs flushes)" -gt "$f0" ] && [ "$h1" -gt "$h0" ]; then
+		ok "reset: stale completion ignored, replay gives 1 copy"
+	else
+		fail "reset: stale completion ignored" "$(counts)" \
+			"resets $r0->$(rsys resets)" \
+			"stale $s0->$(rsys stale_completions) health $h0->$h1"
+	fi
+fi
+
+# --- 7. partial commit: group staged, commit refused -> software -------------
+w0=$(rsys sw_remainder)
+dfs_set fail_commit 1
+for i in 1 2 3; do
+	join p$i amtg$i $G2 $S1
+done
+wait_for 40 sysfs_ge sw_remainder $((w0 + 1))
+sleep 2
+measure $S1 $G2 $K 0.01
+dfs_set fail_commit 0
+if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+   [ "${CNT[amtg3]}" -eq $K ] && [ "$HW" -eq 0 ] && [ "$(hw_members $G2)" = 0 ]; then
+	ok "partial commit: 0 hardware copies, software remainder"
+else
+	fail "partial commit: software remainder" "$(counts) sw_remainder $w0->$(rsys sw_remainder)"
+fi
+for i in 1 2 3; do
+	leave p$i
+done
+
+# --- 8. every member leaves: group disabled, no copies ------------------------
+# The relay drops S1, and G1 with it, at least a GMI after the last
+# report that refreshed it: qrv * 125 s query interval + 10 s response
+# interval. The timer wheel can fire it up to 8/63 of its delay late
+# (32.8 s of a 260 s GMI at HZ=1000), so wait a seventh more, and 10 s.
+gmi=$(( $(ip netns exec "$RELAY" sysctl -n net.ipv4.igmp_qrv) * 125 + 10 ))
+for i in 1 2 3; do
+	leave a$i
+done
+if wait_for $((gmi + gmi / 7 + 10)) group_gone $G1; then
+	measure $S1 $G1 $K 0.01
+	if [ "$HW" -eq 0 ] && [ $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) -eq 0 ]; then
+		ok "0 copies after the group is disabled"
+	else
+		fail "0 copies after the group is disabled" "$(counts)"
+	fi
+else
+	fail "0 copies after the group is disabled" "G1 still in the provider: $(cat "$DFS/tables")"
+fi
+
+# --- 9. required mode with no provider: refused with extack -----------------
+ip -n "$RELAY" link add dum0 type dummy
+ip -n "$RELAY" addr add 10.0.1.2/24 dev dum0
+q0=$(cat /sys/module/amt/parameters/offload_required_refused)
+msg=$(ip -n "$RELAY" link add amtx type amt mode relay local 10.0.1.2 \
+	dev dum0 offload required 2>&1)
+rc=$?
+q1=$(cat /sys/module/amt/parameters/offload_required_refused)
+if [ $rc -ne 0 ] && echo "$msg" | grep -q '^Error: ' && [ $((q1 - q0)) -eq 1 ] &&
+   ! ip -n "$RELAY" link show amtx > /dev/null 2>&1; then
+	ok "required with no provider: refused with extack, counted"
+else
+	fail "required refuses with extack" "rc=$rc refused $q0->$q1 msg: $msg"
+fi
+
+# --- 10. lower device torn away under traffic: ordered teardown ---------------
+# Unlink the peer mid-stream (copies stop, the binding survives), relink
+# (copies resume at 1 per packet), then delete the relay's netdevsim under
+# traffic: amt tears the binding down before the relay goes, with no splat.
+# The unlink-phase stream is from S1, so copies queued while unlinked and
+# released on relink are not counted as the relinked (S2, G4) copies.
+ip netns exec "$RELAY" python3 "$TMP/send.py" amtr $S1 $G4 100 0.02 &
+PIDS+=($!)
+sleep 0.5
+echo "$FD_R:$IFX_R" > $NSIM_SYS/unlink_device
+wait "${PIDS[-1]}"
+bound_after_unlink=$(rsys bound)
+echo "$FD_R:$IFX_R $FD_G:$IFX_G" > $NSIM_SYS/link_device
+wait_for 20 members_eq $G4 1
+measure $S2 $G4 $K 0.01
+relinked=${CNT[amtg1]}
+relinked_hw=$HW
+ip netns exec "$RELAY" python3 "$TMP/send.py" amtr $S2 $G4 200 0.01 &
+PIDS+=($!)
+sleep 0.5
+echo "$NSIM_R_ID" > $NSIM_SYS/del_device
+wait "${PIDS[-1]}"
+splat=$(dmesg | tail -n +$((DMESG1 + 1)) |
+	grep -E 'WARNING:|BUG:|Oops|refcount_t|use-after-free|list_(add|del) corruption')
+if [ "$bound_after_unlink" = 1 ] && [ "$relinked" -eq $K ] && [ "$relinked_hw" -eq $K ] &&
+   ! ip -n "$RELAY" link show amtr > /dev/null 2>&1 && [ ! -d "$DFS" ] &&
+   [ -z "$splat" ]; then
+	ok "lower device unlinked and deleted under traffic: ordered teardown"
+else
+	fail "ordered teardown under traffic" \
+		"bound=$bound_after_unlink relinked=$relinked hw=$relinked_hw splat: $splat"
+fi
+
+if [ $OKS -ne 11 ]; then
+	echo "only $OKS of 11 assertions passed"
+	ERR=1
+fi
diff --git a/tools/testing/selftests/net/config b/tools/testing/selftests/net/config
index 94d722770..88eb75060 100644
--- a/tools/testing/selftests/net/config
+++ b/tools/testing/selftests/net/config
@@ -11,6 +11,7 @@ CONFIG_CRYPTO_ARIA=y
 CONFIG_CRYPTO_CHACHA20POLY1305=m
 CONFIG_CRYPTO_SHA1=y
 CONFIG_CRYPTO_SM4_GENERIC=y
+CONFIG_DEBUG_FS=y
 CONFIG_DEBUG_INFO_BTF=y
 CONFIG_DEBUG_INFO_BTF_MODULES=n
 CONFIG_DUMMY=y
@@ -117,6 +118,7 @@ CONFIG_OPENVSWITCH=m
 CONFIG_OPENVSWITCH_GENEVE=m
 CONFIG_OPENVSWITCH_GRE=m
 CONFIG_OPENVSWITCH_VXLAN=m
+CONFIG_PACKET=y
 CONFIG_PROC_SYSCTL=y
 CONFIG_PSAMPLE=m
 CONFIG_RPS=y
-- 
2.43.0

[RFC PATCH net-next 9/9] MAINTAINERS: amt: cover the offload header and netdevsim provider

[Omar Ramadan]

The AMT entry does not list include/net/amt_offload.h or the netdevsim
reference provider drivers/net/netdevsim/amt.c, so get_maintainer.pl
does not find the AMT maintainer for changes to either. Add both.
tools/testing/selftests/net/amt_offload.sh is covered by the entry's
amt*.sh pattern, which the IPv6 outer transport series adds.

drivers/net/netdevsim/amt.c stays under the NETDEVSIM entry as well,
through its existing drivers/net/netdevsim/* pattern, so the netdevsim
maintainer is still found for it.

Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
 MAINTAINERS | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index b01fdeee6..f6d5b471f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1404,6 +1404,8 @@ S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
 F:	drivers/net/amt.c
+F:	drivers/net/netdevsim/amt.c
+F:	include/net/amt_offload.h
 F:	tools/testing/selftests/net/amt*.sh
 
 ANALOG DEVICES INC AD3530R DRIVER
-- 
2.43.0