* [RFC PATCH net-next 1/9] amt: add the relay replication offload provider header
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 2/9] amt: offload provider registry and lower-device lifecycle Omar Ramadan
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
include/net/amt_offload.h: provider caps, opaque provider-owned endpoint
and group handles, struct amt_offload_ops, TX/INGRESS paths,
FALLBACK/CONSUMED/BUSY results, the per-binding u64 epoch, and the
amt_offload_register()/amt_offload_unregister() entry points. An
in-kernel interface with no ABI promise; nothing touches struct
net_device or net_device_ops.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
include/net/amt_offload.h | 263 ++++++++++++++++++++++++++++++++++++++
1 file changed, 263 insertions(+)
create mode 100644 include/net/amt_offload.h
diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
new file mode 100644
index 000000000..3be5d129c
--- /dev/null
+++ b/include/net/amt_offload.h
@@ -0,0 +1,263 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * AMT relay replication offload: the contract between the amt driver and
+ * a provider that replicates relay multicast data to gateways in hardware
+ * (or, for testing, in a simulated device).
+ *
+ * This is an IN-KERNEL INTERFACE WITH NO ABI PROMISE. It may change in any
+ * release, like every other in-kernel API; out-of-tree providers get no
+ * stability guarantee.
+ *
+ * Registration is keyed by the lower netdev a relay transmits on (its
+ * IFLA_AMT_LINK stream device). Nothing here adds a field or an ndo to
+ * struct net_device.
+ */
+#ifndef _NET_AMT_OFFLOAD_H_
+#define _NET_AMT_OFFLOAD_H_
+
+#include <linux/bits.h>
+#include <linux/in6.h>
+#include <linux/types.h>
+
+struct net_device;
+struct sk_buff;
+
+/* Outer transport families, for &amt_offload_caps.families. */
+#define AMT_OFFLOAD_FAMILY_V4 BIT(0)
+#define AMT_OFFLOAD_FAMILY_V6 BIT(1)
+
+/*
+ * Reserved: (S,G)-keyed forwarding. v1 forwards per (family, group) exactly
+ * as the software path does, so this bit has NO v1 semantics. A provider
+ * must not set it; amt ignores it.
+ */
+#define AMT_OFFLOAD_F_SOURCE_FILTER BIT(0)
+
+/**
+ * enum amt_offload_path - where a provider replicates
+ * @AMT_OFFLOAD_PATH_TX: amt hands the provider each relay data packet of
+ * a group whose commit it has published (&amt_offload_ops.group_commit),
+ * before its software fan-out (&amt_offload_ops.xmit), while the binding
+ * admits packets and amt has not lost its desired state. amt withdraws a
+ * group without a callback when a member leaves, until it publishes the
+ * recommit. Every other packet goes straight to software.
+ * @AMT_OFFLOAD_PATH_INGRESS: the provider replicates on ingress and the
+ * host only sees punted packets. Semantics only until a real provider
+ * exists; no simulated provider implements it.
+ */
+enum amt_offload_path {
+ AMT_OFFLOAD_PATH_TX,
+ AMT_OFFLOAD_PATH_INGRESS,
+};
+
+/**
+ * struct amt_offload_caps - what a bound provider can do
+ * @families: AMT_OFFLOAD_FAMILY_* outer families it can encapsulate. A
+ * relay whose outer family is missing stays unbound (counted), with the
+ * outcome of a failed &amt_offload_ops.bind, except that amt first calls
+ * &amt_offload_ops.unbind, since that bind succeeded.
+ * @paths: BIT(enum amt_offload_path) it implements.
+ * @op_timeout_ms: longest a programming op may take before amt treats the
+ * provider as wedged, resets it and replays from desired state.
+ * @features: AMT_OFFLOAD_F_* bits. None has v1 semantics.
+ * @max_endpoints: endpoint table size; 0 means unbounded. Informational:
+ * amt does not read it in this version. A provider out of room fails an
+ * &amt_offload_ops.endpoint_add for a key it does not hold, by
+ * convention with -ENOSPC.
+ * @max_groups: group table size; 0 means unbounded. Informational: amt
+ * does not read it in this version. A provider out of room fails the
+ * &amt_offload_ops.group_update that needs a new group, by convention
+ * with -ENOSPC.
+ *
+ * Filled by &amt_offload_ops.bind.
+ */
+struct amt_offload_caps {
+ u32 families;
+ u32 paths;
+ u32 op_timeout_ms;
+ u32 features;
+ u32 max_endpoints;
+ u32 max_groups;
+};
+
+/**
+ * struct amt_offload_binding - one relay device bound to one provider
+ * @lower: the registered lower netdev (the relay's stream device).
+ * @dev: the amt relay netdev.
+ * @epoch: monotonic per relay device, bumped on every bind, reset and
+ * teardown. A completion carrying an older epoch is stale and ignored.
+ * @priv: provider-owned; amt never touches it.
+ *
+ * Allocated and freed by amt. Valid from &amt_offload_ops.bind until
+ * &amt_offload_ops.unbind returns.
+ */
+struct amt_offload_binding {
+ struct net_device *lower;
+ struct net_device *dev;
+ u64 epoch;
+ void *priv;
+};
+
+/**
+ * struct amt_offload_endpoint_key - one gateway's outer endpoint
+ * @family: AF_INET or AF_INET6, the relay's outer family.
+ * @addr: gateway outer address.
+ * @addr.ip4: IPv4 address when @family is AF_INET.
+ * @addr.ip6: IPv6 address when @family is AF_INET6.
+ * @port: gateway outer UDP port (tunnel identity is address + port).
+ */
+struct amt_offload_endpoint_key {
+ u8 family;
+ union {
+ __be32 ip4;
+ struct in6_addr ip6;
+ } addr;
+ __be16 port;
+};
+
+/**
+ * struct amt_offload_group_key - one replicated group
+ * @family: AF_INET or AF_INET6, the INNER group family.
+ * @group: group address.
+ * @group.ip4: IPv4 group when @family is AF_INET.
+ * @group.ip6: IPv6 group when @family is AF_INET6.
+ *
+ * Forwarding is per (family, group), never per source: a gateway holding
+ * an INCLUDE(S1) membership still receives S2, exactly as the software
+ * fan-out does today.
+ */
+struct amt_offload_group_key {
+ u8 family;
+ union {
+ __be32 ip4;
+ struct in6_addr ip6;
+ } group;
+};
+
+/* Provider-owned table entries. amt only stores the pointers. */
+struct amt_offload_endpoint;
+struct amt_offload_group;
+
+/**
+ * enum amt_offload_result - outcome of &amt_offload_ops.xmit
+ * @AMT_OFFLOAD_FALLBACK: skb untouched; amt runs its software fan-out.
+ * @AMT_OFFLOAD_CONSUMED: the provider handled every copy of this packet
+ * under the generation it was given and owns the skb; amt emits none and
+ * retries none. A copy that failed is the provider's to count, except one
+ * amt_offload_encap_xmit() could not copy or route, which amt has already
+ * counted in the relay's tx_dropped.
+ * @AMT_OFFLOAD_BUSY: skb untouched; amt runs one synchronous software pass
+ * and counts it. Never NETDEV_TX_BUSY: the relay is IFF_NO_QUEUE.
+ *
+ * Exactly one actor handles each packet, so no packet is duplicated.
+ */
+enum amt_offload_result {
+ AMT_OFFLOAD_FALLBACK,
+ AMT_OFFLOAD_CONSUMED,
+ AMT_OFFLOAD_BUSY,
+};
+
+/**
+ * struct amt_offload_stats - provider-side counters
+ * @tx_packets: copies the provider emitted.
+ * @tx_bytes: bytes of those copies on the wire.
+ * @tx_dropped: copies the provider failed to emit.
+ */
+struct amt_offload_stats {
+ u64 tx_packets;
+ u64 tx_bytes;
+ u64 tx_dropped;
+};
+
+/**
+ * struct amt_offload_ops - provider callbacks
+ * @bind: a relay transmitting on the registered lower device starts using
+ * this provider. Fill @caps. Process context, RTNL held. Nonzero leaves
+ * the relay unbound (counted): in software in auto mode; in required
+ * mode refused if it is being created, else without carrier.
+ * @unbind: last call for a binding. Every endpoint and group has already
+ * been deleted or disabled. Free @b->priv. RTNL held.
+ * @endpoint_add: program one gateway endpoint. Idempotent: an identical
+ * key returns the existing handle. 0 means accepted and durable, not yet
+ * forwarding. Process context, never under amt's spinlocks.
+ * @endpoint_del: remove an endpoint no group references any more.
+ * @group_update: stage the member set of one group under generation @gen,
+ * allocating *@grp on first use. Not forwarding until @group_commit.
+ * @group_commit: start forwarding generation @gen of @grp. amt hands @gen
+ * to @xmit only after this returns 0, and may never do so: an overrun
+ * resets the provider, and on TX a member leaving meanwhile withholds
+ * the commit until the next one. Until amt publishes @gen, any @xmit for
+ * @grp carries an earlier generation this provider committed, and a call
+ * already in flight may carry any of those or @gen itself; the provider
+ * serves such a generation or declines it. A provider must never emit
+ * one packet to one endpoint under two generations. On failure the group
+ * stays in software and previously added endpoints stay valid.
+ * @group_disable: stop forwarding @grp and free it.
+ * @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
+ * &enum amt_offload_result. Softirq context, under rcu_read_lock().
+ * @flush: drop every endpoint and group (reset after a timeout). amt then
+ * replays from its own desired state under a new epoch, except when
+ * the op that overran was a @group_disable or @endpoint_del of amt
+ * giving everything back -- at teardown, where @unbind follows, or
+ * after its desired state was lost: then nothing is replayed and the
+ * epoch does not move.
+ * @stats: fill @st. Optional; amt does not call it in this version.
+ *
+ * Teardown order, on unregistration of the lower device and on
+ * amt_offload_unregister(): amt stops admitting packets to @xmit, waits for
+ * in-flight @xmit calls, cancels its programming work, calls @group_disable
+ * for every group, @endpoint_del for every endpoint, then @unbind, then
+ * synchronize_rcu(); the relay is back in software afterwards. If one of
+ * those @group_disable/@endpoint_del calls overruns op_timeout_ms, amt
+ * calls @flush once instead of the rest, then @unbind, with no replay.
+ */
+struct amt_offload_ops {
+ int (*bind)(struct amt_offload_binding *b,
+ struct amt_offload_caps *caps);
+ void (*unbind)(struct amt_offload_binding *b);
+ int (*endpoint_add)(struct amt_offload_binding *b,
+ const struct amt_offload_endpoint_key *key,
+ struct amt_offload_endpoint **ep);
+ void (*endpoint_del)(struct amt_offload_binding *b,
+ struct amt_offload_endpoint *ep);
+ int (*group_update)(struct amt_offload_binding *b,
+ const struct amt_offload_group_key *key,
+ struct amt_offload_endpoint *const *members,
+ unsigned int nr_members, u64 gen,
+ struct amt_offload_group **grp);
+ int (*group_commit)(struct amt_offload_binding *b,
+ struct amt_offload_group *grp, u64 gen);
+ void (*group_disable)(struct amt_offload_binding *b,
+ struct amt_offload_group *grp);
+ enum amt_offload_result (*xmit)(struct amt_offload_binding *b,
+ struct amt_offload_group *grp,
+ struct sk_buff *skb, u64 gen);
+ void (*flush)(struct amt_offload_binding *b);
+ void (*stats)(struct amt_offload_binding *b,
+ struct amt_offload_stats *st);
+};
+
+/**
+ * amt_offload_register - offer a provider for relays on @lower
+ * @lower: the netdev the relays transmit on.
+ * @ops: provider callbacks; must outlive the registration.
+ *
+ * Every relay already using @lower in auto or required mode tries to bind
+ * now. Caller holds RTNL.
+ *
+ * Return: 0, -EEXIST if @lower already has a provider, or -ENOMEM.
+ */
+int amt_offload_register(struct net_device *lower,
+ const struct amt_offload_ops *ops);
+
+/**
+ * amt_offload_unregister - withdraw the provider for @lower
+ * @lower: as passed to amt_offload_register().
+ *
+ * Tears every binding down in the documented order and returns with every
+ * relay on @lower in software. Caller holds RTNL. A no-op if @lower has no
+ * provider, which is the case after @lower itself was unregistered.
+ */
+void amt_offload_unregister(struct net_device *lower);
+
+#endif /* _NET_AMT_OFFLOAD_H_ */
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 2/9] amt: offload provider registry and lower-device lifecycle
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 1/9] amt: add the relay replication offload provider header Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 3/9] amt: offload group object, desired state and reconciler Omar Ramadan
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
amt_offload_register()/amt_offload_unregister() (EXPORT_SYMBOL_GPL) keep
an RCU list of providers keyed by the lower netdev. Relays are listed at
newlink; a relay in auto or required mode whose stream_dev has a
provider binds there, and registering a provider on a stream_dev binds
the auto and required relays already using it at once (counted). bind()
fills caps; a relay whose bind() fails, or whose outer family the caps
lack, stays unbound (counted): in software in auto mode, refused at
creation or left without carrier in required mode.
On NETDEV_UNREGISTER of the lower device (before amt_dellink), on
provider unregistration and on relay deletion, teardown runs in one
order: stop admission and wait out in-flight xmit, cancel programming
work, group_disable all, endpoint_del all, unbind, unpublish,
synchronize_rcu, software. A required-mode relay that loses its provider
drops carrier. Counters are under the relay's offload/ sysfs group; the
global required-refusal count is the offload_required_refused module
parameter.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/amt.c | 425 +++++++++++++++++++++++++++++++++++++++
include/net/amt.h | 34 ++++
include/uapi/linux/amt.h | 20 ++
3 files changed, 479 insertions(+)
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 201421ad0..034221e1c 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -20,6 +20,7 @@
#include <net/icmp.h>
#include <net/mld.h>
#include <net/amt.h>
+#include <net/amt_offload.h>
#include <uapi/linux/amt.h>
#include <linux/security.h>
#include <net/gro_cells.h>
@@ -3464,9 +3465,11 @@ static int amt_dev_stop(struct net_device *dev)
* lifetime matches vxlan/geneve and the rest of the netdev tree.
*/
static const struct attribute_group amt_upstream_group;
+static const struct attribute_group amt_offload_group;
static const struct attribute_group *amt_groups[] = {
&amt_upstream_group,
+ &amt_offload_group,
NULL,
};
@@ -4028,6 +4031,414 @@ static void amt_link_setup(struct net_device *dev)
ether_setup(dev);
}
+/* Relay replication offload: provider registry and lower-device lifecycle.
+ *
+ * A provider registers against the lower netdev relays transmit on (their
+ * stream_dev); there is no net_device field or ndo. Every relay is linked
+ * into amt_offload_devs at newlink so registration can find the relays
+ * already using that lower device. Both lists are written under RTNL;
+ * amt_offload_providers is an RCU list so a reader outside RTNL can test
+ * for a provider without it.
+ *
+ * Teardown has one order, whichever side goes first (the lower device is
+ * unregistered, the provider unregisters, or the relay is deleted): stop
+ * admitting packets to ops->xmit and wait out in-flight callers, cancel
+ * programming work, group_disable every group, endpoint_del every
+ * endpoint, unbind, unpublish and synchronize_rcu(). The relay is then
+ * in software, exactly the path it runs with no provider at all.
+ */
+struct amt_offload_provider {
+ struct list_head node;
+ struct net_device *lower;
+ const struct amt_offload_ops *ops;
+ struct rcu_head rcu;
+};
+
+/* One provider-owned handle amt must give back on teardown. */
+struct amt_offload_ref {
+ struct list_head node;
+ void *handle;
+};
+
+struct amt_offload_bound {
+ struct amt_offload_binding pub;
+ const struct amt_offload_ops *ops;
+ struct amt_offload_caps caps;
+ /* Cleared first on teardown; ops->xmit is only called while set. */
+ bool admitting;
+ struct list_head groups; /* amt_offload_ref */
+ struct list_head endpoints; /* amt_offload_ref */
+};
+
+static LIST_HEAD(amt_offload_providers);
+static LIST_HEAD(amt_offload_devs);
+
+/* Relays refused at newlink for want of a provider in required mode. The
+ * refused netdev is unregistered again before newlink returns, so the
+ * count cannot live on it.
+ */
+static atomic_t amt_offload_required_refused = ATOMIC_INIT(0);
+
+static int amt_offload_required_refused_get(char *buf,
+ const struct kernel_param *kp)
+{
+ return sysfs_emit(buf, "%u\n",
+ (u32)atomic_read(&amt_offload_required_refused));
+}
+
+/* Read-only, but a load-time value still reaches ->set: parse_one()
+ * calls it without a NULL check (only the sysfs store path checks), so
+ * a get-only ops oopses the loader. Ignore the value rather than fail
+ * the load, so a stray modprobe option cannot keep the relay down.
+ * NOARG: a bare "amt.offload_required_refused" would otherwise be
+ * -EINVAL in parse_one() before ->set, and fail the module load.
+ */
+static int amt_offload_required_refused_set(const char *val,
+ const struct kernel_param *kp)
+{
+ pr_warn("offload_required_refused is read-only; ignoring \"%s\"\n",
+ val ? val : "");
+ return 0;
+}
+
+static const struct kernel_param_ops amt_offload_required_refused_ops = {
+ .flags = KERNEL_PARAM_OPS_FL_NOARG,
+ .set = amt_offload_required_refused_set,
+ .get = amt_offload_required_refused_get,
+};
+module_param_cb(offload_required_refused, &amt_offload_required_refused_ops,
+ NULL, 0444);
+MODULE_PARM_DESC(offload_required_refused,
+ "Relays refused at creation: offload=required and no usable provider");
+
+static void amt_offload_count(struct amt_dev *amt,
+ enum amt_offload_counter cnt)
+{
+ atomic_inc(&amt->offload_cnt[cnt]);
+}
+
+static struct amt_offload_provider *
+amt_offload_provider_find(const struct net_device *lower)
+{
+ struct amt_offload_provider *p;
+
+ ASSERT_RTNL();
+ list_for_each_entry(p, &amt_offload_providers, node)
+ if (p->lower == lower)
+ return p;
+ return NULL;
+}
+
+static u32 amt_offload_outer_family(const struct amt_dev *amt)
+{
+ return ipv6_addr_any(&amt->local_ipv6) ? AMT_OFFLOAD_FAMILY_V4 :
+ AMT_OFFLOAD_FAMILY_V6;
+}
+
+/* The data path's way in: a binding is usable only while admitting. The
+ * caller holds rcu_read_lock(). The later TX handoff is its first caller.
+ */
+static __maybe_unused struct amt_offload_bound *
+amt_offload_tx_admit(struct amt_dev *amt)
+{
+ struct amt_offload_bound *b = rcu_dereference(amt->offload);
+
+ return b && READ_ONCE(b->admitting) ? b : NULL;
+}
+
+/* Record a handle the provider handed back; the later reconciler is the
+ * production caller.
+ */
+static int __maybe_unused amt_offload_track(struct list_head *list,
+ void *handle)
+{
+ struct amt_offload_ref *ref = kzalloc(sizeof(*ref), GFP_KERNEL);
+
+ if (!ref)
+ return -ENOMEM;
+ ref->handle = handle;
+ list_add_tail(&ref->node, list);
+ return 0;
+}
+
+static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
+{
+ if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
+ return;
+ if (on)
+ netif_carrier_on(amt->dev);
+ else
+ netif_carrier_off(amt->dev);
+}
+
+/* Bind @amt to @p. 0 means bound; an error leaves @amt in software and
+ * counted. RTNL held.
+ */
+static int amt_offload_bind(struct amt_dev *amt,
+ struct amt_offload_provider *p, bool on_register)
+{
+ struct amt_offload_bound *b;
+ int err;
+
+ ASSERT_RTNL();
+ if (amt->mode != AMT_MODE_RELAY ||
+ amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE ||
+ rtnl_dereference(amt->offload))
+ return -EINVAL;
+
+ b = kzalloc_obj(*b, GFP_KERNEL);
+ if (!b)
+ return -ENOMEM;
+ b->pub.lower = p->lower;
+ b->pub.dev = amt->dev;
+ b->pub.epoch = ++amt->offload_epoch;
+ b->ops = p->ops;
+ INIT_LIST_HEAD(&b->groups);
+ INIT_LIST_HEAD(&b->endpoints);
+
+ err = b->ops->bind(&b->pub, &b->caps);
+ if (err) {
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_FAILED);
+ kfree(b);
+ return err;
+ }
+ if (!(b->caps.families & amt_offload_outer_family(amt))) {
+ b->ops->unbind(&b->pub);
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
+ kfree(b);
+ return -EOPNOTSUPP;
+ }
+
+ b->admitting = true;
+ rcu_assign_pointer(amt->offload, b);
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND);
+ if (on_register)
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
+ amt_offload_set_carrier(amt, true);
+ return 0;
+}
+
+/* Return @amt to software in the documented order. @cnt is the cause to
+ * count, or AMT_OFFLOAD_CNT_MAX for the relay's own deletion. RTNL held.
+ */
+static void amt_offload_teardown(struct amt_dev *amt,
+ enum amt_offload_counter cnt)
+{
+ struct amt_offload_bound *b = rtnl_dereference(amt->offload);
+ struct amt_offload_ref *ref, *tmp;
+
+ if (!b)
+ return;
+
+ /* 1. Stop admission and wait for every ops->xmit already admitted. */
+ WRITE_ONCE(b->admitting, false);
+ synchronize_net();
+
+ /* 2. Programming work is cancelled here once the reconciler exists
+ * (next patch); nothing can program the provider past this point.
+ */
+
+ /* 3. Disable every group, then 4. delete every endpoint. */
+ list_for_each_entry_safe(ref, tmp, &b->groups, node) {
+ b->ops->group_disable(&b->pub, ref->handle);
+ list_del(&ref->node);
+ kfree(ref);
+ }
+ list_for_each_entry_safe(ref, tmp, &b->endpoints, node) {
+ b->ops->endpoint_del(&b->pub, ref->handle);
+ list_del(&ref->node);
+ kfree(ref);
+ }
+
+ /* 5. Unbind, then 6. unpublish and let RCU readers drain. */
+ b->ops->unbind(&b->pub);
+ RCU_INIT_POINTER(amt->offload, NULL);
+ synchronize_rcu();
+ kfree(b);
+
+ /* 7. Software. */
+ amt->offload_epoch++;
+ if (cnt < AMT_OFFLOAD_CNT_MAX) {
+ amt_offload_count(amt, cnt);
+ if (amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+ amt_offload_set_carrier(amt, false);
+ }
+ }
+}
+
+int amt_offload_register(struct net_device *lower,
+ const struct amt_offload_ops *ops)
+{
+ struct amt_offload_provider *p;
+ struct amt_dev *amt;
+
+ ASSERT_RTNL();
+ if (!lower || !ops || !ops->bind || !ops->unbind ||
+ !ops->group_disable || !ops->endpoint_del)
+ return -EINVAL;
+ if (amt_offload_provider_find(lower))
+ return -EEXIST;
+
+ p = kzalloc_obj(*p, GFP_KERNEL);
+ if (!p)
+ return -ENOMEM;
+ p->lower = lower;
+ p->ops = ops;
+ list_add_rcu(&p->node, &amt_offload_providers);
+
+ list_for_each_entry(amt, &amt_offload_devs, offload_node)
+ if (amt->stream_dev == lower)
+ amt_offload_bind(amt, p, true);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(amt_offload_register);
+
+static void amt_offload_provider_drop(struct net_device *lower,
+ enum amt_offload_counter cnt)
+{
+ struct amt_offload_provider *p = amt_offload_provider_find(lower);
+ struct amt_dev *amt;
+
+ list_for_each_entry(amt, &amt_offload_devs, offload_node)
+ if (amt->stream_dev == lower)
+ amt_offload_teardown(amt, cnt);
+ if (p) {
+ list_del_rcu(&p->node);
+ kfree_rcu(p, rcu);
+ }
+}
+
+void amt_offload_unregister(struct net_device *lower)
+{
+ ASSERT_RTNL();
+ amt_offload_provider_drop(lower, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
+}
+EXPORT_SYMBOL_GPL(amt_offload_unregister);
+
+/* NETDEV_UNREGISTER of a stream_dev: tear its relays down before
+ * amt_dellink runs, and forget its provider so no entry outlives the
+ * netdev it names.
+ */
+static void amt_offload_lower_gone(struct net_device *lower)
+{
+ amt_offload_provider_drop(lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
+}
+
+/* newlink: list the relay and bind it if a provider is registered. A
+ * required-mode relay with no usable provider is refused (-ENODEV) and
+ * counted globally. RTNL held.
+ */
+static int amt_offload_attach(struct amt_dev *amt,
+ struct netlink_ext_ack *extack)
+{
+ struct amt_offload_provider *p;
+ int err = -ENODEV;
+
+ INIT_LIST_HEAD(&amt->offload_node);
+ if (amt->mode != AMT_MODE_RELAY)
+ return 0;
+ list_add_tail(&amt->offload_node, &amt_offload_devs);
+ if (amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE)
+ return 0;
+
+ p = amt_offload_provider_find(amt->stream_dev);
+ if (p)
+ err = amt_offload_bind(amt, p, false);
+ if (!err || amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
+ return 0;
+
+ list_del_init(&amt->offload_node);
+ atomic_inc(&amt_offload_required_refused);
+ if (p)
+ NL_SET_ERR_MSG_MOD(extack,
+ "Offload required but the stream device's provider cannot serve this relay");
+ else
+ NL_SET_ERR_MSG_MOD(extack,
+ "Offload required but the stream device has no offload provider");
+ return -ENODEV;
+}
+
+/* The relay itself is going away. RTNL held. */
+static void amt_offload_detach(struct amt_dev *amt)
+{
+ amt_offload_teardown(amt, AMT_OFFLOAD_CNT_MAX);
+ if (!list_empty(&amt->offload_node))
+ list_del_init(&amt->offload_node);
+}
+
+/* offload/ sysfs group: mode, binding state and the counters. Counters
+ * are monotonic since netdev creation and wrap at u32.
+ */
+static ssize_t offload_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ static const char * const names[] = {
+ [AMT_OFFLOAD_MODE_SOFTWARE] = "software",
+ [AMT_OFFLOAD_MODE_AUTO] = "auto",
+ [AMT_OFFLOAD_MODE_REQUIRED] = "required",
+ };
+ struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+ return sysfs_emit(buf, "%s\n", names[READ_ONCE(amt->offload_mode)]);
+}
+static struct device_attribute dev_attr_offload_mode =
+ __ATTR(mode, 0444, offload_mode_show, NULL);
+
+static ssize_t offload_bound_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+ return sysfs_emit(buf, "%d\n", !!rcu_access_pointer(amt->offload));
+}
+static struct device_attribute dev_attr_offload_bound =
+ __ATTR(bound, 0444, offload_bound_show, NULL);
+
+static ssize_t amt_offload_cnt_show(struct device *dev, char *buf,
+ enum amt_offload_counter cnt)
+{
+ struct amt_dev *amt = netdev_priv(to_net_dev(dev));
+
+ return sysfs_emit(buf, "%u\n", (u32)atomic_read(&amt->offload_cnt[cnt]));
+}
+
+#define AMT_OFFLOAD_CNT_ATTR(name, cnt) \
+static ssize_t offload_##name##_show(struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ return amt_offload_cnt_show(dev, buf, cnt); \
+} \
+static struct device_attribute dev_attr_offload_##name = \
+ __ATTR(name, 0444, offload_##name##_show, NULL)
+
+AMT_OFFLOAD_CNT_ATTR(binds, AMT_OFFLOAD_CNT_BIND);
+AMT_OFFLOAD_CNT_ATTR(binds_on_register, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
+AMT_OFFLOAD_CNT_ATTR(bind_failed, AMT_OFFLOAD_CNT_BIND_FAILED);
+AMT_OFFLOAD_CNT_ATTR(family_unsupported, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
+AMT_OFFLOAD_CNT_ATTR(teardown_lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
+AMT_OFFLOAD_CNT_ATTR(teardown_provider, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
+AMT_OFFLOAD_CNT_ATTR(required_unbound, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+
+static struct attribute *amt_offload_attrs[] = {
+ &dev_attr_offload_mode.attr,
+ &dev_attr_offload_bound.attr,
+ &dev_attr_offload_binds.attr,
+ &dev_attr_offload_binds_on_register.attr,
+ &dev_attr_offload_bind_failed.attr,
+ &dev_attr_offload_family_unsupported.attr,
+ &dev_attr_offload_teardown_lower.attr,
+ &dev_attr_offload_teardown_provider.attr,
+ &dev_attr_offload_required_unbound.attr,
+ NULL,
+};
+
+static const struct attribute_group amt_offload_group = {
+ .name = "offload",
+ .attrs = amt_offload_attrs,
+};
+
static const struct nla_policy amt_policy[IFLA_AMT_MAX + 1] = {
[IFLA_AMT_MODE] = { .type = NLA_U32 },
[IFLA_AMT_RELAY_PORT] = { .type = NLA_U16 },
@@ -4284,6 +4695,13 @@ static int amt_newlink(struct net_device *dev,
INIT_DELAYED_WORK(&amt->secret_wq, amt_secret_work);
INIT_WORK(&amt->event_wq, amt_event_work);
INIT_LIST_HEAD(&amt->tunnel_list);
+
+ err = amt_offload_attach(amt, extack);
+ if (err) {
+ netdev_upper_dev_unlink(amt->stream_dev, dev);
+ unregister_netdevice(dev);
+ goto err;
+ }
return 0;
err:
dev_put(amt->stream_dev);
@@ -4294,6 +4712,7 @@ static void amt_dellink(struct net_device *dev, struct list_head *head)
{
struct amt_dev *amt = netdev_priv(dev);
+ amt_offload_detach(amt);
unregister_netdevice_queue(dev, head);
netdev_upper_dev_unlink(amt->stream_dev, dev);
dev_put(amt->stream_dev);
@@ -4438,6 +4857,12 @@ static int amt_device_event(struct notifier_block *unused,
LIST_HEAD(list);
int new_mtu;
+ /* Before amt_dellink below: the provider's tables are torn down
+ * while the relay and its stream_dev are both still intact.
+ */
+ if (event == NETDEV_UNREGISTER)
+ amt_offload_lower_gone(dev);
+
upper_dev = amt_lookup_upper_dev(dev);
if (!upper_dev)
return NOTIFY_DONE;
diff --git a/include/net/amt.h b/include/net/amt.h
index 5b3405f04..7d70f30e6 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -379,6 +379,29 @@ struct amt_events {
struct sk_buff *skb;
};
+struct amt_offload_bound;
+
+/* Per-relay offload counters, exported under offload/ in sysfs. */
+enum amt_offload_counter {
+ /* bind() succeeded and the caps covered the outer family */
+ AMT_OFFLOAD_CNT_BIND,
+ /* ... of which: bound when a provider registered on the stream_dev
+ * this relay already uses
+ */
+ AMT_OFFLOAD_CNT_BIND_ON_REGISTER,
+ /* bind() returned an error; the relay stayed unbound */
+ AMT_OFFLOAD_CNT_BIND_FAILED,
+ /* caps lacked the relay's outer family; the relay stayed unbound */
+ AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED,
+ /* teardown because the stream_dev was unregistered */
+ AMT_OFFLOAD_CNT_TEARDOWN_LOWER,
+ /* teardown because the provider unregistered */
+ AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER,
+ /* a required-mode relay lost its provider and dropped carrier */
+ AMT_OFFLOAD_CNT_REQUIRED_UNBOUND,
+ AMT_OFFLOAD_CNT_MAX,
+};
+
struct amt_dev {
struct net_device *dev;
struct net_device *stream_dev;
@@ -484,6 +507,17 @@ struct amt_dev {
struct socket *stream_sock_v4;
struct socket *stream_sock_v6;
atomic_t upstream_setsockopt_slow_count;
+
+ /* Replication offload (relay mode). Everything here is written
+ * under RTNL; offload is also read by the data path under RCU.
+ * offload_node links the relay into amt_offload_devs so a provider
+ * registering on its stream_dev can find it.
+ */
+ u32 offload_mode;
+ struct amt_offload_bound __rcu *offload;
+ struct list_head offload_node;
+ u64 offload_epoch;
+ atomic_t offload_cnt[AMT_OFFLOAD_CNT_MAX];
};
#define AMT_TOS 0xc0
diff --git a/include/uapi/linux/amt.h b/include/uapi/linux/amt.h
index c04ed1fa4..de1ae1929 100644
--- a/include/uapi/linux/amt.h
+++ b/include/uapi/linux/amt.h
@@ -21,6 +21,26 @@ enum ifla_amt_mode {
#define AMT_MODE_MAX (__AMT_MODE_MAX - 1)
+/* Relay replication offload (include/net/amt_offload.h). */
+enum ifla_amt_offload_mode {
+ /* Never use a provider. The default. */
+ AMT_OFFLOAD_MODE_SOFTWARE = 0,
+ /* Use the provider registered on the stream device when its bind
+ * succeeds, it covers the relay's outer family and it implements the
+ * TX path; software otherwise.
+ */
+ AMT_OFFLOAD_MODE_AUTO,
+ /* Refuse to create the relay without a usable provider, and drop
+ * carrier while none is bound. Unlike auto, also bind a provider
+ * that replicates only on ingress. A bound relay still forwards in
+ * software, with carrier up, what the provider does not serve.
+ */
+ AMT_OFFLOAD_MODE_REQUIRED,
+ __AMT_OFFLOAD_MODE_MAX,
+};
+
+#define AMT_OFFLOAD_MODE_MAX (__AMT_OFFLOAD_MODE_MAX - 1)
+
enum {
IFLA_AMT_UNSPEC,
/* This attribute specify mode etier Gateway or Relay. */
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 3/9] amt: offload group object, desired state and reconciler
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 1/9] amt: add the relay replication offload provider header Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 2/9] amt: offload provider registry and lower-device lifecycle Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 4/9] amt: offload TX handoff, ingress punt rule and counters Omar Ramadan
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
One group object per (family, group) carries its members (the gateway
endpoints joined to it), a generation and hw_active. Desired state
changes under a leaf bh spinlock: the first gnode of a group on a
tunnel adds the tunnel's endpoint as a member, the last removes it,
and a tunnel's unlink removes it from every group at once.
Forwarding stays per (family, group) and never per source, exactly as
the software fan-out: a gateway holding INCLUDE(S1) still receives S2.
A reconciler in process context, never under amt->lock or a tunnel lock,
converges the provider to desired state: endpoint_add for new endpoints,
group_update and group_commit at the group's generation, then hw_active;
group_disable for emptied groups and endpoint_del for unused endpoints
once no committed group can name them. Each endpoint counts the
committed groups naming it, so its deletion does not wait for every
dirty group to be reprogrammed. A pass starts no new group once it has
spent offload_op_budget ops (default 1000, an estimate), so it can
overrun the budget by one group's ops; if the budget left no room for
one, it still makes one endpoint_del, so a group re-dirtied on every
pass cannot starve endpoint collection. The pass that gives everything
back once desired state is lost is not budgeted. Changes between passes
coalesce into one update per group. A group the provider refuses is
served by software and retried with hysteresis (once the provider holds
an eighth fewer objects), after a reset or on a new binding; its
endpoints stay programmed. An op overrunning op_timeout_ms flushes the
provider, bumps the epoch and replays from desired state; a result from
an older epoch is ignored. A pass ends at a reset and at a teardown, and
checks for teardown before every endpoint_add, group_update,
group_commit and endpoint_del. A teardown under RTNL therefore waits out
at most one of those ops plus the cleanup it triggers or the pass has
already queued: a flush after an overrun, the group_disable that parks a
group it could not program, the group_disable calls for groups the pass
emptied, and, on the pass that gives everything back once desired state
is lost, the rest of that release. The group_disable and endpoint_del
calls that give everything back are timed too: the first to overrun
flushes the provider once and the remaining handles are forgotten, so a
wedged provider costs one op plus one flush, with no replay and no new
epoch (at teardown, unbind follows). The overrun is sampled as the
provider call returns, so amt's own bookkeeping is not charged to it.
The endpoint and group tables are sized once from max_tunnels and
max_tunnels * max_groups respectively, and hashed with the device's
hash_seed: gateways choose their groups and ports. Removing a member
withdraws its group from hardware at once (hw_active), so no packet
admitted after that is replicated to the departed endpoint; software
serves the group until the reconciler commits the new member set, and
hw_active is published only if no member was removed since the snapshot
the commit was built from. A join meanwhile does not withhold it: the
committed set is a subset that names no departed endpoint. A group
re-created for an emptied group's key waits, within the pass, for the
old provider object's group_disable, so a provider never holds two
groups for one key. Parking a group that holds a provider object always
waits out ops->xmit, since removing a member clears hw_active with no
grace period. The table size is clamped in 64 bits before
order_base_2(), which would truncate the product on 32-bit.
Binding replays desired state; teardown takes the reconciler mutex to
cancel programming, then gives every group and endpoint back in the
documented order. Desired state is freed at ndo_uninit, after the
tunnels. amt_tunnel_unlink() now carries the unlink that
amt_tunnel_expire() and amt_dev_stop() each open-coded, so the hook
runs on both. Software mode allocates nothing and runs no hook.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/amt.c | 1166 +++++++++++++++++++++++++++++++++++--
include/net/amt.h | 35 +-
include/net/amt_offload.h | 74 ++-
3 files changed, 1225 insertions(+), 50 deletions(-)
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 034221e1c..17337989a 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -7,6 +7,7 @@
#include <linux/skbuff.h>
#include <linux/udp.h>
#include <linux/jhash.h>
+#include <linux/hashtable.h>
#include <linux/if_tunnel.h>
#include <linux/net.h>
#include <linux/igmp.h>
@@ -33,6 +34,8 @@
#include <net/inet_dscp.h>
#include <net/ip6_checksum.h>
+#include <kunit/static_stub.h>
+
static struct workqueue_struct *amt_wq;
static HLIST_HEAD(source_gc_list);
@@ -304,6 +307,19 @@ static void amt_destroy_source(struct amt_source_node *snode)
spin_unlock_bh(&source_gc_lock);
}
+/* Offload desired-state hooks, defined with the rest of the offload code.
+ * Each is a no-op on a relay in software mode.
+ */
+static void amt_ofl_member_add(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel,
+ const union amt_addr *group, bool v6);
+static void amt_ofl_member_del(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel,
+ const union amt_addr *group, bool v6);
+static void amt_ofl_tunnel_gone(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel);
+static void amt_ofl_state_free(struct amt_dev *amt);
+
static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
{
struct amt_source_node *snode;
@@ -315,6 +331,8 @@ static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
rhltable_remove(&amt->groups_rhl, &gnode->rhlnode, amt_gnode_params);
list_del(&gnode->tunnel_node);
gnode->tunnel_list->nr_groups--;
+ amt_ofl_member_del(amt, gnode->tunnel_list, &gnode->group_addr,
+ gnode->v6);
if (!gnode->v6)
netdev_dbg(amt->dev, "Leave group %pI4\n",
@@ -569,6 +587,7 @@ static struct amt_group_node *amt_add_group(struct amt_dev *amt,
}
list_add(&gnode->tunnel_node, &tunnel->groups);
tunnel->nr_groups++;
+ amt_ofl_member_add(amt, tunnel, group, v6);
if (!gnode->v6)
netdev_dbg(amt->dev, "Join group %pI4\n",
@@ -1542,6 +1561,17 @@ static void amt_clear_groups(struct amt_tunnel_list *tunnel)
spin_unlock_bh(&tunnel->lock);
}
+/* Unlink @tunnel from the relay. Caller holds amt->lock, or RTNL with the
+ * device stopping (amt_dev_stop). The groups stay until amt_clear_groups.
+ */
+static void amt_tunnel_unlink(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel)
+{
+ list_del_rcu(&tunnel->list);
+ amt->nr_tunnels--;
+ amt_ofl_tunnel_gone(amt, tunnel);
+}
+
static void amt_tunnel_expire(struct work_struct *work)
{
struct amt_tunnel_list *tunnel = container_of(to_delayed_work(work),
@@ -1551,8 +1581,7 @@ static void amt_tunnel_expire(struct work_struct *work)
spin_lock_bh(&amt->lock);
rcu_read_lock();
- list_del_rcu(&tunnel->list);
- amt->nr_tunnels--;
+ amt_tunnel_unlink(amt, tunnel);
amt_clear_groups(tunnel);
rcu_read_unlock();
spin_unlock_bh(&amt->lock);
@@ -3445,8 +3474,7 @@ static int amt_dev_stop(struct net_device *dev)
amt->remote_ip = 0;
list_for_each_entry_safe(tunnel, tmp, &amt->tunnel_list, list) {
- list_del_rcu(&tunnel->list);
- amt->nr_tunnels--;
+ amt_tunnel_unlink(amt, tunnel);
cancel_delayed_work_sync(&tunnel->gc_wq);
amt_clear_groups(tunnel);
kfree_rcu(tunnel, rcu);
@@ -3996,6 +4024,7 @@ static void amt_dev_uninit(struct net_device *dev)
/* amt_dev_stop() has removed every tunnel and its groups. */
rhltable_destroy(&amt->groups_rhl);
gro_cells_destroy(&amt->gro_cells);
+ amt_ofl_state_free(amt);
}
static const struct net_device_ops amt_netdev_ops = {
@@ -4054,20 +4083,141 @@ struct amt_offload_provider {
struct rcu_head rcu;
};
-/* One provider-owned handle amt must give back on teardown. */
-struct amt_offload_ref {
- struct list_head node;
- void *handle;
-};
-
struct amt_offload_bound {
struct amt_offload_binding pub;
const struct amt_offload_ops *ops;
struct amt_offload_caps caps;
/* Cleared first on teardown; ops->xmit is only called while set. */
bool admitting;
- struct list_head groups; /* amt_offload_ref */
- struct list_head endpoints; /* amt_offload_ref */
+};
+
+/* Desired state and its reconciler.
+ *
+ * amt_add_group / amt_del_group tunnel unlink
+ * (first / last gnode of a group) (every group of the tunnel)
+ * | |
+ * v v
+ * +-- desired state, st->lock (bh, a leaf) -------------------------+
+ * | grp (family, group) --< member (grp, tunnel) >-- ep (addr,port) |
+ * | a member change bumps grp->gen and queues grp on st->dirty |
+ * +------------------------------------------------------------------+
+ * | st->work, process context
+ * v
+ * reconciler, st->mutex, never under amt->lock or a tunnel lock:
+ * per dirty grp: endpoint_add each new ep, group_update(gen),
+ * group_commit(gen), then publish hw_gen and hw_active; a grp
+ * with no members is disabled; then unused eps get endpoint_del
+ * |
+ * v
+ * provider tables; amt_dev_xmit reads grp->hw_active and hw_gen
+ *
+ * A tick starts no new group once st->budget ops are spent, so one
+ * group may overrun it by its own ops, and endpoint GC runs one
+ * endpoint_del past it when the budget left none, so a group re-dirtied
+ * on every tick cannot starve collection. Changes that land between ticks
+ * coalesce: a group is queued at most once and programmed at its latest
+ * generation. A group whose programming fails is parked and served by
+ * software (the remainder); parked groups are retried only when the
+ * provider holds an eighth fewer objects than when it refused one
+ * (hysteresis), after a reset, or on a new binding. A programming, gone
+ * or GC op overrunning op_timeout_ms resets the provider (flush) and
+ * replays every group from desired state under a new epoch; a result
+ * from an older epoch is ignored. A release op that overruns while amt
+ * gives everything back (teardown, or desired state lost) flushes once
+ * and forgets the remaining handles, with no new epoch and no replay.
+ * Only the reconciler frees grp and ep records, so it may use them
+ * without st->lock between ops.
+ *
+ * Forwarding is per (family, group) and never per source, exactly as the
+ * software fan-out in amt_dev_xmit: a gateway with an INCLUDE(S1)
+ * membership still receives S2. AMT_OFFLOAD_F_SOURCE_FILTER has no v1
+ * meaning.
+ */
+/* Both tables are sized once, from the caps, at state allocation: eps
+ * from max_tunnels, grps from max_tunnels * max_groups. Neither cap has a
+ * changelink, so a load factor of about one holds without a resize; the
+ * upper bound only matters past 64k entries.
+ */
+#define AMT_OFL_HASH_MIN_BITS 4
+#define AMT_OFL_HASH_MAX_BITS 16
+
+/* Default ops per reconciler tick. [estimated]: not measured on hardware. */
+static unsigned int amt_offload_op_budget = 1000;
+module_param_named(offload_op_budget, amt_offload_op_budget, uint, 0444);
+MODULE_PARM_DESC(offload_op_budget,
+ "Offload programming ops per reconciler tick (default 1000)");
+
+enum amt_ofl_grp_state {
+ AMT_OFL_IDLE,
+ AMT_OFL_DIRTY, /* on st->dirty */
+ AMT_OFL_PARKED, /* on st->parked: served by software */
+};
+
+struct amt_ofl_ep {
+ struct hlist_node hnode; /* st->eps */
+ struct list_head gc_node; /* st->ep_gc */
+ struct amt_offload_endpoint_key key;
+ u32 refs; /* members using it */
+ u32 committed; /* committed grps naming it */
+ bool on_gc;
+ /* Reconciler-owned, under st->mutex. */
+ struct list_head hw_node; /* st->hw_eps */
+ struct amt_offload_endpoint *hw;
+};
+
+struct amt_ofl_grp {
+ struct hlist_node hnode; /* st->grps, RCU */
+ struct list_head q_node; /* st->dirty or st->parked */
+ struct amt_offload_group_key key;
+ struct list_head members;
+ u32 nr_members;
+ u8 state;
+ u64 gen; /* desired */
+ u32 left; /* members removed */
+ /* Reconciler-owned, under st->mutex; the data path reads hw and
+ * hw_gen once it has read hw_active true (acquire).
+ */
+ struct list_head hw_node; /* st->hw_grps */
+ struct amt_offload_group *hw;
+ /* The endpoints the last successful group_commit named; each counts
+ * in its ep->committed, under st->lock.
+ */
+ struct amt_ofl_ep **ceps;
+ u32 nr_ceps;
+ u64 hw_gen;
+ bool hw_active;
+ struct rcu_head rcu;
+};
+
+struct amt_ofl_member {
+ struct list_head grp_node; /* grp->members */
+ struct hlist_node tun_node; /* tunnel->offload_members */
+ struct amt_ofl_grp *grp;
+ struct amt_ofl_ep *ep;
+ u32 gnodes; /* the tunnel's gnodes for grp */
+};
+
+struct amt_offload_state {
+ struct amt_dev *amt;
+ /* Desired state, written from the softirq decap path. */
+ spinlock_t lock;
+ struct hlist_head *grps; /* RCU */
+ struct hlist_head *eps;
+ u32 grp_bits;
+ u32 ep_bits;
+ struct list_head dirty;
+ struct list_head parked;
+ struct list_head ep_gc;
+ bool desync;
+ /* Reconciler. bind and teardown also take the mutex. */
+ struct mutex mutex;
+ struct work_struct work;
+ struct list_head hw_grps;
+ struct list_head hw_eps;
+ u32 hw_objs;
+ bool enospc;
+ u32 enospc_at;
+ u32 budget;
};
static LIST_HEAD(amt_offload_providers);
@@ -4146,21 +4296,948 @@ amt_offload_tx_admit(struct amt_dev *amt)
return b && READ_ONCE(b->admitting) ? b : NULL;
}
-/* Record a handle the provider handed back; the later reconciler is the
- * production caller.
+static void amt_ofl_grp_key(struct amt_offload_group_key *key,
+ const union amt_addr *group, bool v6)
+{
+ memset(key, 0, sizeof(*key));
+#if IS_ENABLED(CONFIG_IPV6)
+ if (v6) {
+ key->family = AF_INET6;
+ key->group.ip6 = group->ip6;
+ return;
+ }
+#endif
+ key->family = AF_INET;
+ key->group.ip4 = group->ip4;
+}
+
+static void amt_ofl_ep_key(struct amt_offload_endpoint_key *key,
+ const struct amt_dev *amt,
+ const struct amt_tunnel_list *tunnel)
+{
+ memset(key, 0, sizeof(*key));
+ key->port = tunnel->source_port;
+#if IS_ENABLED(CONFIG_IPV6)
+ if (amt_v6(amt)) {
+ key->family = AF_INET6;
+ key->addr.ip6 = tunnel->addr.ip6;
+ return;
+ }
+#endif
+ key->family = AF_INET;
+ key->addr.ip4 = tunnel->addr.ip4;
+}
+
+/* Keys are built zeroed, so memcmp() over the padding is sound. Seeded
+ * per device: gateways choose their groups and ports, so an unseeded hash
+ * lets them pile entries into one chain under st->lock.
+ */
+static u32 amt_ofl_hash(const struct amt_offload_state *st,
+ const void *key, size_t len, u32 bits)
+{
+ return hash_32(jhash(key, len, st->amt->hash_seed), bits);
+}
+
+static struct hlist_head *
+amt_ofl_grp_bucket(const struct amt_offload_state *st,
+ const struct amt_offload_group_key *key)
+{
+ return &st->grps[amt_ofl_hash(st, key, sizeof(*key), st->grp_bits)];
+}
+
+static struct hlist_head *
+amt_ofl_ep_bucket(const struct amt_offload_state *st,
+ const struct amt_offload_endpoint_key *key)
+{
+ return &st->eps[amt_ofl_hash(st, key, sizeof(*key), st->ep_bits)];
+}
+
+/* Caller holds st->lock, or rcu_read_lock() for the data path. */
+static struct amt_ofl_grp *
+amt_ofl_grp_find(struct amt_offload_state *st,
+ const struct amt_offload_group_key *key)
+{
+ struct amt_ofl_grp *g;
+
+ hlist_for_each_entry_rcu(g, amt_ofl_grp_bucket(st, key), hnode,
+ lockdep_is_held(&st->lock))
+ if (!memcmp(&g->key, key, sizeof(*key)))
+ return g;
+ return NULL;
+}
+
+static struct amt_ofl_ep *
+amt_ofl_ep_find(struct amt_offload_state *st,
+ const struct amt_offload_endpoint_key *key)
+{
+ struct amt_ofl_ep *ep;
+
+ hlist_for_each_entry(ep, amt_ofl_ep_bucket(st, key), hnode)
+ if (!memcmp(&ep->key, key, sizeof(*key)))
+ return ep;
+ return NULL;
+}
+
+/* Queue @g for the reconciler. A parked group stays parked, served by
+ * software, until its retry is due -- unless it lost its last member,
+ * which the reconciler must see to free it. Caller holds st->lock.
+ */
+static void amt_ofl_grp_dirty(struct amt_offload_state *st,
+ struct amt_ofl_grp *g)
+{
+ if (g->state == AMT_OFL_DIRTY)
+ return;
+ if (g->state == AMT_OFL_PARKED) {
+ if (g->nr_members)
+ return;
+ list_del(&g->q_node);
+ }
+ list_add_tail(&g->q_node, &st->dirty);
+ g->state = AMT_OFL_DIRTY;
+}
+
+static void amt_ofl_kick(struct amt_offload_state *st)
+{
+ queue_work(system_unbound_wq, &st->work);
+}
+
+/* Desired state could not record a change. Hardware would forward to a
+ * stale member set, so the relay leaves offload for good: the data path
+ * stops consulting the provider, and the next reconciler pass gives every
+ * provider object back (group_disable, endpoint_del; the binding stays,
+ * with no unbind). Desync does not schedule that pass: a pass already
+ * queued or a later removal of a recorded member (by a leave, a timer or
+ * a tunnel removal) does, and a teardown gives everything back itself
+ * and unbinds. Caller holds st->lock.
+ */
+static void amt_ofl_desync(struct amt_offload_state *st)
+{
+ if (st->desync)
+ return;
+ WRITE_ONCE(st->desync, true);
+ amt_offload_count(st->amt, AMT_OFFLOAD_CNT_DESYNC);
+}
+
+/* The first gnode of (@group, @v6) on @tunnel makes the tunnel's
+ * endpoint a member; later ones only count. Caller holds tunnel->lock.
+ */
+static void amt_ofl_member_add(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel,
+ const union amt_addr *group, bool v6)
+{
+ struct amt_offload_state *st = amt->offload_st;
+ struct amt_ofl_grp *g, *new_g = NULL;
+ struct amt_ofl_ep *ep, *new_ep = NULL;
+ struct amt_offload_endpoint_key ekey;
+ struct amt_offload_group_key gkey;
+ struct amt_ofl_member *m;
+
+ if (!st)
+ return;
+ amt_ofl_grp_key(&gkey, group, v6);
+ spin_lock_bh(&st->lock);
+ if (st->desync)
+ goto out;
+ hlist_for_each_entry(m, &tunnel->offload_members, tun_node) {
+ if (!memcmp(&m->grp->key, &gkey, sizeof(gkey))) {
+ m->gnodes++;
+ goto out;
+ }
+ }
+
+ g = amt_ofl_grp_find(st, &gkey);
+ if (!g) {
+ new_g = kzalloc_obj(*new_g, GFP_ATOMIC);
+ if (!new_g)
+ goto nomem;
+ g = new_g;
+ g->key = gkey;
+ INIT_LIST_HEAD(&g->members);
+ INIT_LIST_HEAD(&g->hw_node);
+ }
+ amt_ofl_ep_key(&ekey, amt, tunnel);
+ ep = amt_ofl_ep_find(st, &ekey);
+ if (!ep) {
+ new_ep = kzalloc_obj(*new_ep, GFP_ATOMIC);
+ if (!new_ep)
+ goto nomem;
+ ep = new_ep;
+ ep->key = ekey;
+ INIT_LIST_HEAD(&ep->hw_node);
+ }
+ m = kzalloc_obj(*m, GFP_ATOMIC);
+ if (!m)
+ goto nomem;
+
+ if (new_g)
+ hlist_add_head_rcu(&g->hnode, amt_ofl_grp_bucket(st, &gkey));
+ if (new_ep)
+ hlist_add_head(&ep->hnode, amt_ofl_ep_bucket(st, &ekey));
+ m->grp = g;
+ m->ep = ep;
+ m->gnodes = 1;
+ ep->refs++;
+ list_add_tail(&m->grp_node, &g->members);
+ hlist_add_head(&m->tun_node, &tunnel->offload_members);
+ g->nr_members++;
+ g->gen++;
+ amt_ofl_grp_dirty(st, g);
+ spin_unlock_bh(&st->lock);
+ amt_ofl_kick(st);
+ return;
+nomem:
+ /* One lost allocation takes the relay out of offload for good (see
+ * amt_ofl_desync()). Rebuilding desired state from the tunnel walk
+ * would avoid that, if GFP_ATOMIC failures ever show up here.
+ */
+ kfree(new_g);
+ kfree(new_ep);
+ amt_ofl_desync(st);
+out:
+ spin_unlock_bh(&st->lock);
+}
+
+/* The reconciler's member arrays: up to max_tunnels entries, so not one
+ * contiguous block.
+ */
+static void *amt_ofl_alloc_array(u32 n, size_t size)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+ KUNIT_STATIC_STUB_REDIRECT(amt_ofl_alloc_array, n, size);
+#endif
+ return kvmalloc_array(n, size, GFP_KERNEL);
+}
+
+static void amt_ofl_free_array(void *p)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+ KUNIT_STATIC_STUB_REDIRECT(amt_ofl_free_array, p);
+#endif
+ kvfree(p);
+}
+
+/* @ep may be deleted once no member uses it and no committed group names
+ * it: queue it for endpoint GC. Caller holds st->lock.
+ */
+static void amt_ofl_ep_unused(struct amt_offload_state *st,
+ struct amt_ofl_ep *ep)
+{
+ if (ep->refs || ep->committed || ep->on_gc)
+ return;
+ list_add_tail(&ep->gc_node, &st->ep_gc);
+ ep->on_gc = true;
+}
+
+/* @g's committed member set no longer reaches hardware: its provider
+ * object is disabled or flushed. Caller holds st->mutex.
+ */
+static void amt_ofl_grp_uncommit(struct amt_offload_state *st,
+ struct amt_ofl_grp *g)
+{
+ struct amt_ofl_ep **ceps = g->ceps;
+ u32 i;
+
+ if (!ceps)
+ return;
+ spin_lock_bh(&st->lock);
+ for (i = 0; i < g->nr_ceps; i++) {
+ ceps[i]->committed--;
+ amt_ofl_ep_unused(st, ceps[i]);
+ }
+ g->ceps = NULL;
+ g->nr_ceps = 0;
+ spin_unlock_bh(&st->lock);
+ amt_ofl_free_array(ceps);
+}
+
+/* Caller holds st->lock. */
+static void amt_ofl_member_unlink(struct amt_offload_state *st,
+ struct amt_ofl_member *m)
+{
+ struct amt_ofl_grp *g = m->grp;
+ struct amt_ofl_ep *ep = m->ep;
+
+ hlist_del(&m->tun_node);
+ list_del(&m->grp_node);
+ g->nr_members--;
+ g->gen++;
+ g->left++;
+ /* The committed member set still names this endpoint: stop handing
+ * the group to hardware now, not when the reconciler recommits, so
+ * no packet admitted after this is replicated to it. Software serves
+ * the group meanwhile.
+ */
+ WRITE_ONCE(g->hw_active, false);
+ amt_ofl_grp_dirty(st, g);
+ ep->refs--;
+ amt_ofl_ep_unused(st, ep);
+ kfree(m);
+}
+
+/* The last gnode of (@group, @v6) on @tunnel is gone: the endpoint
+ * leaves the group. Caller holds tunnel->lock.
+ */
+static void amt_ofl_member_del(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel,
+ const union amt_addr *group, bool v6)
+{
+ struct amt_offload_state *st = amt->offload_st;
+ struct amt_offload_group_key gkey;
+ struct amt_ofl_member *m;
+ bool kick = false;
+
+ if (!st)
+ return;
+ amt_ofl_grp_key(&gkey, group, v6);
+ spin_lock_bh(&st->lock);
+ hlist_for_each_entry(m, &tunnel->offload_members, tun_node) {
+ if (memcmp(&m->grp->key, &gkey, sizeof(gkey)))
+ continue;
+ if (!--m->gnodes) {
+ amt_ofl_member_unlink(st, m);
+ kick = true;
+ }
+ break;
+ }
+ spin_unlock_bh(&st->lock);
+ if (kick)
+ amt_ofl_kick(st);
+}
+
+/* @tunnel is torn down or unlinked: its endpoint leaves every group at
+ * once. The gnodes deleted afterwards find no member and do nothing.
+ * Caller holds amt->lock, or RTNL with the device stopping (amt_dev_stop).
+ */
+static void amt_ofl_tunnel_gone(struct amt_dev *amt,
+ struct amt_tunnel_list *tunnel)
+{
+ struct amt_offload_state *st = amt->offload_st;
+ struct amt_ofl_member *m;
+ struct hlist_node *tmp;
+ bool kick;
+
+ if (!st)
+ return;
+ spin_lock_bh(&st->lock);
+ kick = !hlist_empty(&tunnel->offload_members);
+ hlist_for_each_entry_safe(m, tmp, &tunnel->offload_members, tun_node)
+ amt_ofl_member_unlink(st, m);
+ spin_unlock_bh(&st->lock);
+ if (kick)
+ amt_ofl_kick(st);
+}
+
+/* Queue every group with members, parked ones included, for a full
+ * replay: on a new binding and after a reset.
+ */
+static void amt_ofl_requeue_all(struct amt_offload_state *st)
+{
+ struct amt_ofl_grp *g;
+ u32 bkt;
+
+ spin_lock_bh(&st->lock);
+ for (bkt = 0; bkt < 1U << st->grp_bits; bkt++) {
+ hlist_for_each_entry(g, &st->grps[bkt], hnode) {
+ if (g->state == AMT_OFL_PARKED) {
+ list_del(&g->q_node);
+ g->state = AMT_OFL_IDLE;
+ }
+ amt_ofl_grp_dirty(st, g);
+ }
+ }
+ st->enospc = false;
+ spin_unlock_bh(&st->lock);
+}
+
+/* Wait out every ops->xmit admitted before the caller cleared hw_active or
+ * admitting.
+ */
+static void amt_ofl_sync_net(void)
+{
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+ KUNIT_STATIC_STUB_REDIRECT(amt_ofl_sync_net);
+#endif
+ synchronize_net();
+}
+
+/* Forget every provider handle; the provider was flushed. Each group is
+ * unpublished too, so a later binding never sees hw_active over a NULL
+ * handle. Caller holds st->mutex.
+ */
+static void amt_ofl_hw_forget(struct amt_offload_state *st)
+{
+ struct amt_ofl_grp *g, *gt;
+ struct amt_ofl_ep *ep, *et;
+
+ list_for_each_entry_safe(g, gt, &st->hw_grps, hw_node) {
+ WRITE_ONCE(g->hw_active, false);
+ amt_ofl_grp_uncommit(st, g);
+ g->hw = NULL;
+ list_del_init(&g->hw_node);
+ }
+ list_for_each_entry_safe(ep, et, &st->hw_eps, hw_node) {
+ ep->hw = NULL;
+ list_del_init(&ep->hw_node);
+ }
+ st->hw_objs = 0;
+}
+
+/* An op issued at @t0 overran op_timeout_ms: the provider is wedged. */
+static bool amt_ofl_op_overran(struct amt_offload_bound *b, ktime_t t0)
+{
+ u32 tmo = b->caps.op_timeout_ms;
+
+#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
+ KUNIT_STATIC_STUB_REDIRECT(amt_ofl_op_overran, b, t0);
+#endif
+ return tmo && ktime_ms_delta(ktime_get(), t0) > tmo;
+}
+
+/* Give every provider object back with group_disable and endpoint_del,
+ * groups first. A release op that overruns op_timeout_ms ends it: the
+ * provider is flushed once and the remaining handles are forgotten, so
+ * a wedged provider costs one op plus one flush, not one op per object,
+ * under RTNL at teardown. @quiesced: the caller already stopped
+ * admission and waited out ops->xmit. Caller holds st->mutex.
*/
-static int __maybe_unused amt_offload_track(struct list_head *list,
- void *handle)
+static void amt_ofl_hw_release(struct amt_offload_state *st,
+ struct amt_offload_bound *b, bool quiesced)
{
- struct amt_offload_ref *ref = kzalloc(sizeof(*ref), GFP_KERNEL);
+ struct amt_ofl_grp *g, *gt;
+ struct amt_ofl_ep *ep, *et;
+ bool overran;
+ ktime_t t0;
+
+ lockdep_assert_held(&st->mutex);
+ if (!quiesced && !list_empty(&st->hw_grps)) {
+ list_for_each_entry(g, &st->hw_grps, hw_node)
+ WRITE_ONCE(g->hw_active, false);
+ amt_ofl_sync_net();
+ }
+ list_for_each_entry_safe(g, gt, &st->hw_grps, hw_node) {
+ WRITE_ONCE(g->hw_active, false);
+ t0 = ktime_get();
+ b->ops->group_disable(&b->pub, g->hw);
+ overran = amt_ofl_op_overran(b, t0);
+ amt_ofl_grp_uncommit(st, g);
+ g->hw = NULL;
+ list_del_init(&g->hw_node);
+ if (overran)
+ goto wedged;
+ }
+ list_for_each_entry_safe(ep, et, &st->hw_eps, hw_node) {
+ t0 = ktime_get();
+ b->ops->endpoint_del(&b->pub, ep->hw);
+ overran = amt_ofl_op_overran(b, t0);
+ ep->hw = NULL;
+ list_del_init(&ep->hw_node);
+ if (overran)
+ goto wedged;
+ }
+ st->hw_objs = 0;
+ return;
+wedged:
+ b->ops->flush(&b->pub);
+ amt_ofl_hw_forget(st);
+ amt_offload_count(st->amt, AMT_OFFLOAD_CNT_RESET);
+}
+
+/* An op overran op_timeout_ms: the provider is treated as wedged. Stop
+ * forwarding through it, flush its tables, forget every handle, bump the
+ * epoch so anything it still returns is stale, and replay. Caller holds
+ * st->mutex.
+ */
+static void amt_ofl_reset(struct amt_offload_state *st,
+ struct amt_offload_bound *b)
+{
+ struct amt_ofl_grp *g;
+
+ lockdep_assert_held(&st->mutex);
+ list_for_each_entry(g, &st->hw_grps, hw_node)
+ WRITE_ONCE(g->hw_active, false);
+ amt_ofl_sync_net();
+ b->ops->flush(&b->pub);
+ amt_ofl_hw_forget(st);
+ b->pub.epoch = ++st->amt->offload_epoch;
+ amt_offload_count(st->amt, AMT_OFFLOAD_CNT_RESET);
+ amt_ofl_requeue_all(st);
+}
+
+/* Completion check for an op issued under @epoch; @overran was sampled
+ * right after the provider call returned, so amt's own bookkeeping is not
+ * charged to the provider. False means its result must be ignored -- the
+ * op overran op_timeout_ms (the provider is reset here) or the epoch moved
+ * under it.
+ */
+static bool amt_ofl_op_done(struct amt_offload_state *st,
+ struct amt_offload_bound *b, u64 epoch, bool overran)
+{
+ if (overran)
+ amt_ofl_reset(st, b);
+ if (b->pub.epoch == epoch)
+ return true;
+ amt_offload_count(st->amt, AMT_OFFLOAD_CNT_STALE);
+ return false;
+}
+
+/* Stop forwarding @g in hardware and free its provider object. The caller
+ * cleared hw_active and waited out ops->xmit, or the group never had it.
+ * Returns whether group_disable overran op_timeout_ms.
+ */
+static bool amt_ofl_grp_hw_free(struct amt_offload_state *st,
+ struct amt_offload_bound *b,
+ struct amt_ofl_grp *g)
+{
+ ktime_t t0 = ktime_get();
+ bool overran;
+
+ b->ops->group_disable(&b->pub, g->hw);
+ overran = amt_ofl_op_overran(b, t0);
+ amt_ofl_grp_uncommit(st, g);
+ g->hw = NULL;
+ list_del_init(&g->hw_node);
+ st->hw_objs--;
+ return overran;
+}
- if (!ref)
+/* Retry parked groups once the provider holds an eighth fewer objects
+ * than when it refused one. Refused with nothing programmed, only a
+ * reset or a new binding retries them. Caller holds st->mutex.
+ */
+static void amt_ofl_unpark(struct amt_offload_state *st)
+{
+ struct amt_ofl_grp *g, *t;
+
+ if (!st->enospc || !st->enospc_at ||
+ st->hw_objs + max(st->enospc_at / 8, 1U) > st->enospc_at)
+ return;
+ spin_lock_bh(&st->lock);
+ list_for_each_entry_safe(g, t, &st->parked, q_node) {
+ list_del(&g->q_node);
+ g->state = AMT_OFL_IDLE;
+ amt_ofl_grp_dirty(st, g);
+ }
+ st->enospc = false;
+ spin_unlock_bh(&st->lock);
+}
+
+/* Programming @g failed: software serves it from now until a retry. Its
+ * endpoints stay programmed; they may serve other groups.
+ */
+static unsigned int amt_ofl_grp_park(struct amt_offload_state *st,
+ struct amt_offload_bound *b,
+ struct amt_ofl_grp *g)
+{
+ unsigned int ops = 0;
+
+ if (g->hw) {
+ /* Wait out ops->xmit even when hw_active already reads false:
+ * amt_ofl_member_unlink() clears it from softirq with no grace
+ * period, so a reader admitted before may still hold g->hw.
+ */
+ WRITE_ONCE(g->hw_active, false);
+ amt_ofl_sync_net();
+ amt_ofl_grp_hw_free(st, b, g);
+ ops++;
+ }
+ if (!st->enospc) {
+ st->enospc = true;
+ st->enospc_at = st->hw_objs;
+ }
+ amt_offload_count(st->amt, AMT_OFFLOAD_CNT_SW_REMAINDER);
+ spin_lock_bh(&st->lock);
+ if (g->state == AMT_OFL_DIRTY && !g->nr_members) {
+ /* Emptied meanwhile: let the reconciler free it. */
+ } else {
+ if (g->state == AMT_OFL_DIRTY)
+ list_del(&g->q_node);
+ list_add_tail(&g->q_node, &st->parked);
+ g->state = AMT_OFL_PARKED;
+ }
+ spin_unlock_bh(&st->lock);
+ return ops;
+}
+
+/* Program @g at its current generation: add its endpoints that are not
+ * in hardware yet, stage the member set, commit, publish. Returns the
+ * ops spent. Caller holds st->mutex; @g is off every queue.
+ */
+static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
+ struct amt_offload_bound *b,
+ struct amt_ofl_grp *g)
+{
+ struct amt_offload_endpoint **hws = NULL;
+ struct amt_ofl_ep **eps = NULL;
+ struct amt_offload_group *hw;
+ struct amt_ofl_member *m;
+ unsigned int ops = 0;
+ u32 n, i;
+ ktime_t t0;
+ u64 epoch;
+ u32 left;
+ u64 gen;
+ int err;
+
+ for (;;) {
+ spin_lock_bh(&st->lock);
+ n = g->nr_members;
+ spin_unlock_bh(&st->lock);
+ if (!n)
+ goto requeue;
+ eps = amt_ofl_alloc_array(n, sizeof(*eps));
+ hws = amt_ofl_alloc_array(n, sizeof(*hws));
+ if (!eps || !hws)
+ goto park;
+ spin_lock_bh(&st->lock);
+ if (g->nr_members == n)
+ break;
+ spin_unlock_bh(&st->lock);
+ /* Membership moved under the allocation. The next pass may
+ * find none and leave through out:, which frees both again.
+ */
+ amt_ofl_free_array(eps);
+ amt_ofl_free_array(hws);
+ eps = NULL;
+ hws = NULL;
+ }
+ i = 0;
+ list_for_each_entry(m, &g->members, grp_node)
+ eps[i++] = m->ep;
+ gen = g->gen;
+ left = g->left;
+ spin_unlock_bh(&st->lock);
+
+ if (g->hw && READ_ONCE(g->hw_active) && g->hw_gen == gen)
+ goto out; /* a replay of what is already committed */
+
+ for (i = 0; i < n; i++) {
+ struct amt_offload_endpoint *ehw = NULL;
+
+ if (!eps[i]->hw) {
+ if (!READ_ONCE(b->admitting))
+ goto out; /* teardown gives it all back */
+ epoch = b->pub.epoch;
+ t0 = ktime_get();
+ err = b->ops->endpoint_add(&b->pub, &eps[i]->key, &ehw);
+ ops++;
+ if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+ goto out; /* reset: replayed */
+ if (err)
+ goto park;
+ eps[i]->hw = ehw;
+ list_add_tail(&eps[i]->hw_node, &st->hw_eps);
+ st->hw_objs++;
+ }
+ hws[i] = eps[i]->hw;
+ }
+
+ if (!READ_ONCE(b->admitting))
+ goto out;
+ hw = g->hw;
+ epoch = b->pub.epoch;
+ t0 = ktime_get();
+ err = b->ops->group_update(&b->pub, &g->key, hws, n, gen, &hw);
+ ops++;
+ if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+ goto out;
+ if (!g->hw && hw) {
+ g->hw = hw;
+ list_add_tail(&g->hw_node, &st->hw_grps);
+ st->hw_objs++;
+ }
+ if (err)
+ goto park;
+
+ if (!READ_ONCE(b->admitting))
+ goto out;
+ epoch = b->pub.epoch;
+ t0 = ktime_get();
+ err = b->ops->group_commit(&b->pub, g->hw, gen);
+ ops++;
+ if (!amt_ofl_op_done(st, b, epoch, amt_ofl_op_overran(b, t0)))
+ goto out;
+ if (err)
+ goto park;
+
+ /* Generation rule: packets submitted from here on carry @gen.
+ * Release pairs with the acquire in amt_offload_tx(): a reader that
+ * sees hw_active also sees hw and hw_gen. Publish only if no member
+ * was removed since the snapshot: it is still in what was just
+ * committed, and amt_ofl_member_unlink() already withdrew the group.
+ * A join meanwhile does not hold the publication back: the committed
+ * set is a subset that names no departed endpoint. Either change
+ * re-queued @g; the next pass commits it.
+ */
+ spin_lock_bh(&st->lock);
+ for (i = 0; i < n; i++)
+ eps[i]->committed++;
+ for (i = 0; i < g->nr_ceps; i++) {
+ g->ceps[i]->committed--;
+ amt_ofl_ep_unused(st, g->ceps[i]);
+ }
+ swap(g->ceps, eps); /* out: frees the old set */
+ g->nr_ceps = n;
+ WRITE_ONCE(g->hw_gen, gen);
+ if (g->left == left)
+ smp_store_release(&g->hw_active, true);
+ spin_unlock_bh(&st->lock);
+ goto out;
+park:
+ ops += amt_ofl_grp_park(st, b, g);
+ goto out;
+requeue:
+ spin_lock_bh(&st->lock);
+ amt_ofl_grp_dirty(st, g);
+ spin_unlock_bh(&st->lock);
+out:
+ amt_ofl_free_array(eps);
+ amt_ofl_free_array(hws);
+ return ops;
+}
+
+/* Disable the unhashed groups on @gone: one grace period per flush, then
+ * disable. A group holds hw only while bound, so @b is set when any does.
+ */
+static void amt_ofl_gone_flush(struct amt_offload_state *st,
+ struct amt_offload_bound *b,
+ struct list_head *gone)
+{
+ struct amt_ofl_grp *g, *gt;
+ u64 e;
+
+ if (list_empty(gone))
+ return;
+ amt_ofl_sync_net();
+ list_for_each_entry_safe(g, gt, gone, q_node) {
+ list_del(&g->q_node);
+ /* A reset earlier in this flush or tick already flushed it. */
+ if (g->hw) {
+ e = b->pub.epoch;
+ amt_ofl_op_done(st, b, e, amt_ofl_grp_hw_free(st, b, g));
+ }
+ kfree_rcu(g, rcu);
+ }
+}
+
+/* Is a group with @key on @gone, its provider object still live? */
+static bool amt_ofl_gone_has(struct list_head *gone,
+ const struct amt_offload_group_key *key)
+{
+ struct amt_ofl_grp *g;
+
+ list_for_each_entry(g, gone, q_node)
+ if (!memcmp(&g->key, key, sizeof(*key)))
+ return true;
+ return false;
+}
+
+/* One reconciler tick. Returns true while work remains. */
+static bool amt_ofl_reconcile(struct amt_dev *amt)
+{
+ struct amt_offload_state *st = amt->offload_st;
+ struct amt_ofl_grp *g;
+ struct amt_offload_bound *b;
+ LIST_HEAD(gone);
+ unsigned int spent = 0;
+ unsigned int budget;
+ struct amt_ofl_ep *ep;
+ bool gc_del = false;
+ u64 epoch = 0, e;
+ ktime_t t0;
+ bool more;
+
+ budget = max(READ_ONCE(st->budget), 1U);
+ mutex_lock(&st->mutex);
+ b = rcu_dereference_protected(amt->offload,
+ lockdep_is_held(&st->mutex));
+ if (b && !READ_ONCE(b->admitting)) {
+ /* Teardown is under way and gives everything back. */
+ mutex_unlock(&st->mutex);
+ return false;
+ }
+ if (b && READ_ONCE(st->desync)) {
+ amt_ofl_hw_release(st, b, false);
+ b = NULL;
+ }
+ if (b)
+ epoch = b->pub.epoch;
+
+ while (spent < budget) {
+ /* A teardown waiting on st->mutex holds RTNL, and a reset
+ * replays everything anyway: end the pass at either, and let
+ * the work re-kick, rather than spend the whole budget
+ * against a provider that may be wedged.
+ */
+ if (b && (!READ_ONCE(b->admitting) || b->pub.epoch != epoch))
+ break;
+ spin_lock_bh(&st->lock);
+ g = list_first_entry_or_null(&st->dirty, struct amt_ofl_grp,
+ q_node);
+ if (!g) {
+ spin_unlock_bh(&st->lock);
+ break;
+ }
+ list_del(&g->q_node);
+ g->state = AMT_OFL_IDLE;
+ if (!g->nr_members) {
+ hlist_del_init_rcu(&g->hnode);
+ spin_unlock_bh(&st->lock);
+ if (g->hw) {
+ WRITE_ONCE(g->hw_active, false);
+ list_add_tail(&g->q_node, &gone);
+ spent++;
+ } else {
+ kfree_rcu(g, rcu);
+ }
+ continue;
+ }
+ spin_unlock_bh(&st->lock);
+ if (!b)
+ continue;
+ /* A rejoin of a group emptied earlier in this tick hashed a
+ * new object for the same key while the old one's provider
+ * group is still live: disable that one first, or the
+ * provider would hold two groups for one (family, group).
+ */
+ if (amt_ofl_gone_has(&gone, &g->key)) {
+ amt_ofl_gone_flush(st, b, &gone);
+ /* A reset in that flush requeued @g: replay it. */
+ if (b->pub.epoch != epoch)
+ break;
+ }
+ spent += amt_ofl_grp_program(st, b, g);
+ }
+
+ amt_ofl_gone_flush(st, b, &gone);
+
+ /* Unused endpoints: no member uses one and no committed group names
+ * it (ep->committed), so it goes whatever is still dirty. At least
+ * one endpoint_del per tick, or a group re-dirtied on every pass
+ * would spend the budget and starve it.
+ */
+ for (;;) {
+ if (b && !READ_ONCE(b->admitting))
+ break;
+ spin_lock_bh(&st->lock);
+ ep = spent < budget || !gc_del ?
+ list_first_entry_or_null(&st->ep_gc, struct amt_ofl_ep,
+ gc_node) : NULL;
+ if (!ep) {
+ spin_unlock_bh(&st->lock);
+ break;
+ }
+ list_del(&ep->gc_node);
+ ep->on_gc = false;
+ if (ep->refs || ep->committed) {
+ /* Queued again when both reach zero. */
+ spin_unlock_bh(&st->lock);
+ continue;
+ }
+ hlist_del_init(&ep->hnode);
+ spin_unlock_bh(&st->lock);
+ if (ep->hw) {
+ list_del_init(&ep->hw_node);
+ st->hw_objs--;
+ e = b->pub.epoch;
+ t0 = ktime_get();
+ b->ops->endpoint_del(&b->pub, ep->hw);
+ spent++;
+ gc_del = true;
+ amt_ofl_op_done(st, b, e, amt_ofl_op_overran(b, t0));
+ }
+ kfree(ep);
+ }
+
+ if (b)
+ amt_ofl_unpark(st);
+ spin_lock_bh(&st->lock);
+ more = !list_empty(&st->dirty) || !list_empty(&st->ep_gc);
+ spin_unlock_bh(&st->lock);
+ mutex_unlock(&st->mutex);
+ return more;
+}
+
+static void amt_ofl_work(struct work_struct *work)
+{
+ struct amt_offload_state *st = container_of(work,
+ struct amt_offload_state,
+ work);
+
+ if (amt_ofl_reconcile(st->amt))
+ amt_ofl_kick(st);
+}
+
+static int amt_ofl_state_alloc(struct amt_dev *amt)
+{
+ struct amt_offload_state *st;
+
+ st = kzalloc_obj(*st, GFP_KERNEL);
+ if (!st)
return -ENOMEM;
- ref->handle = handle;
- list_add_tail(&ref->node, list);
+ st->ep_bits = clamp_t(u32, order_base_2(amt->max_tunnels),
+ AMT_OFL_HASH_MIN_BITS, AMT_OFL_HASH_MAX_BITS);
+ /* Clamp in 64 bits: order_base_2() takes an unsigned long, which
+ * would truncate the product on a 32-bit kernel.
+ */
+ st->grp_bits = clamp_t(u32, order_base_2(min_t(u64,
+ (u64)amt->max_tunnels * amt->max_groups,
+ 1U << AMT_OFL_HASH_MAX_BITS)),
+ AMT_OFL_HASH_MIN_BITS, AMT_OFL_HASH_MAX_BITS);
+ st->grps = kvcalloc(1U << st->grp_bits, sizeof(*st->grps), GFP_KERNEL);
+ st->eps = kvcalloc(1U << st->ep_bits, sizeof(*st->eps), GFP_KERNEL);
+ if (!st->grps || !st->eps) {
+ kvfree(st->grps);
+ kvfree(st->eps);
+ kfree(st);
+ return -ENOMEM;
+ }
+ st->amt = amt;
+ spin_lock_init(&st->lock);
+ INIT_LIST_HEAD(&st->dirty);
+ INIT_LIST_HEAD(&st->parked);
+ INIT_LIST_HEAD(&st->ep_gc);
+ mutex_init(&st->mutex);
+ INIT_WORK(&st->work, amt_ofl_work);
+ INIT_LIST_HEAD(&st->hw_grps);
+ INIT_LIST_HEAD(&st->hw_eps);
+ st->budget = READ_ONCE(amt_offload_op_budget);
+ amt->offload_st = st;
return 0;
}
+/* ndo_uninit: every tunnel is gone and the binding with it, so nothing
+ * here is in hardware. Members are freed without touching their tunnels.
+ */
+static void amt_ofl_state_free(struct amt_dev *amt)
+{
+ struct amt_offload_state *st = amt->offload_st;
+ struct amt_ofl_member *m, *mt;
+ struct hlist_node *tmp;
+ struct amt_ofl_grp *g;
+ struct amt_ofl_ep *ep;
+ u32 bkt;
+
+ if (!st)
+ return;
+ cancel_work_sync(&st->work);
+ for (bkt = 0; bkt < 1U << st->grp_bits; bkt++) {
+ hlist_for_each_entry_safe(g, tmp, &st->grps[bkt], hnode) {
+ list_for_each_entry_safe(m, mt, &g->members, grp_node)
+ kfree(m);
+ amt_ofl_free_array(g->ceps);
+ kfree(g);
+ }
+ }
+ for (bkt = 0; bkt < 1U << st->ep_bits; bkt++)
+ hlist_for_each_entry_safe(ep, tmp, &st->eps[bkt], hnode)
+ kfree(ep);
+ mutex_destroy(&st->mutex);
+ kvfree(st->grps);
+ kvfree(st->eps);
+ kfree(st);
+ amt->offload_st = NULL;
+}
+
static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
{
if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
@@ -4191,10 +5268,10 @@ static int amt_offload_bind(struct amt_dev *amt,
return -ENOMEM;
b->pub.lower = p->lower;
b->pub.dev = amt->dev;
- b->pub.epoch = ++amt->offload_epoch;
b->ops = p->ops;
- INIT_LIST_HEAD(&b->groups);
- INIT_LIST_HEAD(&b->endpoints);
+ mutex_lock(&amt->offload_st->mutex);
+ b->pub.epoch = ++amt->offload_epoch;
+ mutex_unlock(&amt->offload_st->mutex);
err = b->ops->bind(&b->pub, &b->caps);
if (err) {
@@ -4210,7 +5287,12 @@ static int amt_offload_bind(struct amt_dev *amt,
}
b->admitting = true;
+ mutex_lock(&amt->offload_st->mutex);
rcu_assign_pointer(amt->offload, b);
+ /* Replay desired state: every group is programmed afresh. */
+ amt_ofl_requeue_all(amt->offload_st);
+ mutex_unlock(&amt->offload_st->mutex);
+ amt_ofl_kick(amt->offload_st);
amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND);
if (on_register)
amt_offload_count(amt, AMT_OFFLOAD_CNT_BIND_ON_REGISTER);
@@ -4225,7 +5307,7 @@ static void amt_offload_teardown(struct amt_dev *amt,
enum amt_offload_counter cnt)
{
struct amt_offload_bound *b = rtnl_dereference(amt->offload);
- struct amt_offload_ref *ref, *tmp;
+ struct amt_offload_state *st = amt->offload_st;
if (!b)
return;
@@ -4234,21 +5316,14 @@ static void amt_offload_teardown(struct amt_dev *amt,
WRITE_ONCE(b->admitting, false);
synchronize_net();
- /* 2. Programming work is cancelled here once the reconciler exists
- * (next patch); nothing can program the provider past this point.
+ /* 2. Cancel programming: a reconciler tick holds st->mutex
+ * throughout, so taking it waits one out; a later tick finds
+ * admission stopped, then no binding, and programs nothing.
*/
+ mutex_lock(&st->mutex);
/* 3. Disable every group, then 4. delete every endpoint. */
- list_for_each_entry_safe(ref, tmp, &b->groups, node) {
- b->ops->group_disable(&b->pub, ref->handle);
- list_del(&ref->node);
- kfree(ref);
- }
- list_for_each_entry_safe(ref, tmp, &b->endpoints, node) {
- b->ops->endpoint_del(&b->pub, ref->handle);
- list_del(&ref->node);
- kfree(ref);
- }
+ amt_ofl_hw_release(st, b, true);
/* 5. Unbind, then 6. unpublish and let RCU readers drain. */
b->ops->unbind(&b->pub);
@@ -4256,8 +5331,10 @@ static void amt_offload_teardown(struct amt_dev *amt,
synchronize_rcu();
kfree(b);
- /* 7. Software. */
+ /* 7. Software. Desired state stays for the next binding. */
amt->offload_epoch++;
+ mutex_unlock(&st->mutex);
+ amt_ofl_kick(st);
if (cnt < AMT_OFFLOAD_CNT_MAX) {
amt_offload_count(amt, cnt);
if (amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
@@ -4275,7 +5352,9 @@ int amt_offload_register(struct net_device *lower,
ASSERT_RTNL();
if (!lower || !ops || !ops->bind || !ops->unbind ||
- !ops->group_disable || !ops->endpoint_del)
+ !ops->endpoint_add || !ops->endpoint_del ||
+ !ops->group_update || !ops->group_commit ||
+ !ops->group_disable || !ops->flush)
return -EINVAL;
if (amt_offload_provider_find(lower))
return -EEXIST;
@@ -4338,6 +5417,9 @@ static int amt_offload_attach(struct amt_dev *amt,
INIT_LIST_HEAD(&amt->offload_node);
if (amt->mode != AMT_MODE_RELAY)
return 0;
+ if (amt->offload_mode != AMT_OFFLOAD_MODE_SOFTWARE &&
+ !amt->offload_st && amt_ofl_state_alloc(amt))
+ return -ENOMEM;
list_add_tail(&amt->offload_node, &amt_offload_devs);
if (amt->offload_mode == AMT_OFFLOAD_MODE_SOFTWARE)
return 0;
@@ -4420,6 +5502,10 @@ AMT_OFFLOAD_CNT_ATTR(family_unsupported, AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED);
AMT_OFFLOAD_CNT_ATTR(teardown_lower, AMT_OFFLOAD_CNT_TEARDOWN_LOWER);
AMT_OFFLOAD_CNT_ATTR(teardown_provider, AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER);
AMT_OFFLOAD_CNT_ATTR(required_unbound, AMT_OFFLOAD_CNT_REQUIRED_UNBOUND);
+AMT_OFFLOAD_CNT_ATTR(sw_remainder, AMT_OFFLOAD_CNT_SW_REMAINDER);
+AMT_OFFLOAD_CNT_ATTR(resets, AMT_OFFLOAD_CNT_RESET);
+AMT_OFFLOAD_CNT_ATTR(stale_completions, AMT_OFFLOAD_CNT_STALE);
+AMT_OFFLOAD_CNT_ATTR(desync, AMT_OFFLOAD_CNT_DESYNC);
static struct attribute *amt_offload_attrs[] = {
&dev_attr_offload_mode.attr,
@@ -4431,6 +5517,10 @@ static struct attribute *amt_offload_attrs[] = {
&dev_attr_offload_teardown_lower.attr,
&dev_attr_offload_teardown_provider.attr,
&dev_attr_offload_required_unbound.attr,
+ &dev_attr_offload_sw_remainder.attr,
+ &dev_attr_offload_resets.attr,
+ &dev_attr_offload_stale_completions.attr,
+ &dev_attr_offload_desync.attr,
NULL,
};
diff --git a/include/net/amt.h b/include/net/amt.h
index 7d70f30e6..0c746e4be 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -312,6 +312,11 @@ struct amt_tunnel_list {
struct rcu_head rcu;
/* This tunnel's entries in amt->groups_rhl, under lock */
struct list_head groups;
+ /* Offload desired state: one amt_ofl_member per group this tunnel
+ * is joined to, under amt_offload_state.lock. Empty unless the
+ * relay's offload mode is auto or required.
+ */
+ struct hlist_head offload_members;
};
/* RFC 3810
@@ -380,6 +385,7 @@ struct amt_events {
};
struct amt_offload_bound;
+struct amt_offload_state;
/* Per-relay offload counters, exported under offload/ in sysfs. */
enum amt_offload_counter {
@@ -399,6 +405,19 @@ enum amt_offload_counter {
AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER,
/* a required-mode relay lost its provider and dropped carrier */
AMT_OFFLOAD_CNT_REQUIRED_UNBOUND,
+ /* a group could not be programmed and is served by software */
+ AMT_OFFLOAD_CNT_SW_REMAINDER,
+ /* an op overran op_timeout_ms: flush and replay from desired state
+ * (a release op while amt gives everything back: flush only)
+ */
+ AMT_OFFLOAD_CNT_RESET,
+ /* an op result arrived under an older epoch and was ignored; a
+ * programming, gone or GC op that overran counts here too, since its
+ * reset moved the epoch (a release op's flush does not move it)
+ */
+ AMT_OFFLOAD_CNT_STALE,
+ /* desired state lost an update (allocation failure); software only */
+ AMT_OFFLOAD_CNT_DESYNC,
AMT_OFFLOAD_CNT_MAX,
};
@@ -508,16 +527,24 @@ struct amt_dev {
struct socket *stream_sock_v6;
atomic_t upstream_setsockopt_slow_count;
- /* Replication offload (relay mode). Everything here is written
- * under RTNL; offload is also read by the data path under RCU.
- * offload_node links the relay into amt_offload_devs so a provider
- * registering on its stream_dev can find it.
+ /* Replication offload (relay mode). offload_mode is set at newlink.
+ * offload is written under RTNL and offload_st's mutex, and read by
+ * the data path under RCU. offload_epoch is written under
+ * offload_st's mutex: at bind and teardown with RTNL held too, and
+ * by the reconciler, without RTNL, when it resets the provider.
+ * offload_node, under RTNL, links the relay into amt_offload_devs so
+ * a provider registering on its stream_dev can find it. offload_cnt
+ * is atomic and bumped from any context.
*/
u32 offload_mode;
struct amt_offload_bound __rcu *offload;
struct list_head offload_node;
u64 offload_epoch;
atomic_t offload_cnt[AMT_OFFLOAD_CNT_MAX];
+ /* Desired state and reconciler; allocated at newlink in auto and
+ * required mode, freed at ndo_uninit once every tunnel is gone.
+ */
+ struct amt_offload_state *offload_st;
};
#define AMT_TOS 0xc0
diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
index 3be5d129c..b5c119851 100644
--- a/include/net/amt_offload.h
+++ b/include/net/amt_offload.h
@@ -179,10 +179,17 @@ struct amt_offload_stats {
* been deleted or disabled. Free @b->priv. RTNL held.
* @endpoint_add: program one gateway endpoint. Idempotent: an identical
* key returns the existing handle. 0 means accepted and durable, not yet
- * forwarding. Process context, never under amt's spinlocks.
- * @endpoint_del: remove an endpoint no group references any more.
+ * forwarding.
+ * @endpoint_del: remove an endpoint no committed group references any
+ * more. amt only deletes endpoints once every group naming one has been
+ * recommitted without it or disabled.
* @group_update: stage the member set of one group under generation @gen,
- * allocating *@grp on first use. Not forwarding until @group_commit.
+ * allocating *@grp on first use (also on failure, if it allocated). On
+ * any failure amt serves the group in software and disables *@grp if it
+ * is set, a live group included. @members is valid only during the
+ * call. Not forwarding until @group_commit. amt holds at most one group
+ * per key per binding: a first use (*@grp NULL) never names a key whose
+ * earlier group on this binding has not been disabled or flushed.
* @group_commit: start forwarding generation @gen of @grp. amt hands @gen
* to @xmit only after this returns 0, and may never do so: an overrun
* resets the provider, and on TX a member leaving meanwhile withholds
@@ -190,9 +197,11 @@ struct amt_offload_stats {
* @grp carries an earlier generation this provider committed, and a call
* already in flight may carry any of those or @gen itself; the provider
* serves such a generation or declines it. A provider must never emit
- * one packet to one endpoint under two generations. On failure the group
- * stays in software and previously added endpoints stay valid.
- * @group_disable: stop forwarding @grp and free it.
+ * one packet to one endpoint under two generations. On failure amt
+ * disables the group and serves it in software; previously added
+ * endpoints stay valid, since they may serve other groups.
+ * @group_disable: stop forwarding @grp and free it. amt has already
+ * stopped submitting packets for @grp and waited out @xmit callers.
* @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
* &enum amt_offload_result. Softirq context, under rcu_read_lock().
* @flush: drop every endpoint and group (reset after a timeout). amt then
@@ -237,15 +246,64 @@ struct amt_offload_ops {
struct amt_offload_stats *st);
};
+/*
+ * Completion semantics. Every programming op (@endpoint_add,
+ * @endpoint_del, @group_update, @group_commit, @group_disable, @flush) is
+ * synchronous: amt calls it in process context, never under amt's
+ * spinlocks -- from one reconciler per relay, or with RTNL held when the
+ * binding is torn down -- so a provider may sleep in it but must not
+ * take RTNL. Return 0 means accepted and
+ * durable in the provider's tables. Nothing forwards before @group_commit
+ * returns 0. @xmit is only passed a generation for which @group_commit on
+ * that group returned 0, which can lag the one committed last (see
+ * @group_commit).
+ *
+ * An op that takes longer than &amt_offload_caps.op_timeout_ms (0: no
+ * limit) marks the provider wedged: amt stops submitting packets, calls
+ * @flush, forgets every handle, bumps &amt_offload_binding.epoch and
+ * replays every group from its own desired state. The exception is amt
+ * giving every object back (teardown, or desired state lost): there an
+ * overrunning @group_disable or @endpoint_del ends it with @flush, and the
+ * remaining handles are forgotten, with no new epoch and no replay. A
+ * result returned under an older epoch is ignored. After @flush the
+ * provider holds nothing for this binding; amt never passes it an older
+ * handle again.
+ *
+ * Cancellation is the teardown order on &struct amt_offload_ops. A
+ * reconciler pass running when a teardown starts finishes the op it is in
+ * and then makes no further @endpoint_add, @group_update, @group_commit or
+ * @endpoint_del, except that it may still call @flush (after an overrun),
+ * @group_disable (for a group it parks or found emptied) and, on a pass
+ * giving every object back once desired state is lost, the remaining
+ * @group_disable and @endpoint_del calls of that release.
+ *
+ * The reconciler starts no new group once a pass has spent the amt module
+ * parameter offload_op_budget (default 1000, an estimate) ops, and never
+ * splits a group's ops across passes, so a pass can overrun the budget by
+ * one group's ops. If the budget left no room for one, a pass still makes
+ * one @endpoint_del, so endpoint collection is never starved. The pass
+ * that gives every object back once desired state is lost is not
+ * budgeted. A group the provider refuses (-ENOSPC or any other error) is
+ * served by software and retried once the provider holds an eighth fewer
+ * objects than when it refused, after a reset, or on a new binding.
+ */
+
/**
* amt_offload_register - offer a provider for relays on @lower
* @lower: the netdev the relays transmit on.
* @ops: provider callbacks; must outlive the registration.
*
* Every relay already using @lower in auto or required mode tries to bind
- * now. Caller holds RTNL.
+ * now. Caller holds RTNL. Every op but @stats and @xmit is required.
+ * Whether a provider offers %AMT_OFFLOAD_PATH_TX shows only in the caps
+ * @bind fills, so @xmit is not checked here. Without it, once @bind
+ * succeeds and the caps cover the relay's outer family, amt treats TX as
+ * not offered: a relay in required mode whose caps offer
+ * %AMT_OFFLOAD_PATH_INGRESS binds on the ingress path, and any other
+ * relay stays unbound (counted as path_unsupported).
*
- * Return: 0, -EEXIST if @lower already has a provider, or -ENOMEM.
+ * Return: 0, -EINVAL if a required op is missing, -EEXIST if @lower
+ * already has a provider, or -ENOMEM.
*/
int amt_offload_register(struct net_device *lower,
const struct amt_offload_ops *ops);
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 4/9] amt: offload TX handoff, ingress punt rule and counters
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (2 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 3/9] amt: offload group object, desired state and reconciler Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 5/9] amt: add IFLA_AMT_OFFLOAD_MODE Omar Ramadan
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
On a TX binding, amt_dev_xmit hands each relay data packet of a group
the reconciler has published (hw_active, set with hw_gen after a
successful group_commit) to the provider before its software fan-out,
while the binding admits packets and desired state is in sync; every
other packet goes to software. Exactly one actor handles each packet
handed over: CONSUMED means the provider handled every copy (sent it, or
dropped it and counted it, apart from the copy and route failures
amt_offload_encap_xmit() counts itself) and owns the skb; FALLBACK
leaves the skb untouched for the software loop; BUSY runs one
synchronous software pass and counts it, never NETDEV_TX_BUSY since the
relay is IFF_NO_QUEUE. hw_gen is read once per packet. The reconciler
writes it after a successful group_commit, also when it then withholds
hw_active because a member left meanwhile, so a packet already past the
hw_active test can carry the new generation and one that read hw_gen
earlier any older committed one; the provider serves or declines it.
On the ingress path the provider replicates every packet of a group from
group_commit until group_disable or flush, and must never also punt one.
While the group's commit is published (hw_active, with the binding
admitting and desired state in sync), a punted copy that reaches the
relay is dropped and counted; at any other time (before the first
publication, after a park, during a reset or a teardown, after desync)
it goes to the software fan-out. The provider sees none of these
transitions as amt makes them, so the drop is only a safety net.
Removing a member on an ingress binding does not withdraw the group as
it does on TX: hardware replicates to the committed set until the
recommit regardless, and lifting the drop would send the remaining
members every punted packet a second time. For the same reason the
reconciler publishes a commit on an ingress binding even when a member
left while it was programmed. A relay binds on ingress only in required
mode: a packet the provider consumes on ingress never reaches the host,
so nothing on the host's forwarding path, mroute counters included, sees
it, and auto does not choose that. A provider offering no path the relay
may use leaves it unbound, counted. TX needs ops->xmit.
New offload/ counters: path_unsupported, busy_fallback, punt_dropped
(plus sw_remainder, resets, stale_completions and desync from the
reconciler). With no provider bound the data path does one more RCU
pointer read and test.
A simulated replication provider -- netdevsim is the reference one --
decides from its own tables which gateways get a copy of a relay data
packet, but has no business re-implementing AMT encapsulation: if its
copies differed on the wire from amt's own, a test could not tell a
provider bug from an encapsulation difference.
amt_offload_encap_xmit(binding, endpoint key, skb), exported, emits one
gateway's copy byte for byte as the software fan-out does: the copy, AMT
Multicast Data header and amt_udp_xmit() of amt_send_multicast_data(),
from the relay port to the gateway's port with AMT_TOS on an IPv4 outer
and the route's hop limit. Each failed copy or route lookup counts once
in tx_dropped; the software fan-out on this base counts neither. It
takes the endpoint key, not the provider's handle, which is opaque to
amt, and is for use from ops->xmit only.
With no provider registered nothing calls it.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/amt.c | 206 +++++++++++++++++++++++++++++++++++---
include/net/amt.h | 13 ++-
include/net/amt_offload.h | 38 ++++++-
3 files changed, 237 insertions(+), 20 deletions(-)
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 17337989a..377660e55 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -319,6 +319,8 @@ static void amt_ofl_member_del(struct amt_dev *amt,
static void amt_ofl_tunnel_gone(struct amt_dev *amt,
struct amt_tunnel_list *tunnel);
static void amt_ofl_state_free(struct amt_dev *amt);
+static bool amt_offload_tx(struct amt_dev *amt, struct sk_buff *skb,
+ const union amt_addr *group, bool v6);
static void amt_del_group(struct amt_dev *amt, struct amt_group_node *gnode)
{
@@ -1510,6 +1512,11 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
} else if (amt->mode == AMT_MODE_RELAY) {
if (!data)
goto free;
+ /* A bound offload provider replicates it, or declines; see
+ * amt_offload_tx() for the cutover rules.
+ */
+ if (amt_offload_tx(amt, skb, &group, v6))
+ return NETDEV_TX_OK;
key.group_addr = group;
key.v6 = v6;
list_for_each_entry_rcu(tunnel, &amt->tunnel_list, list) {
@@ -4087,6 +4094,7 @@ struct amt_offload_bound {
struct amt_offload_binding pub;
const struct amt_offload_ops *ops;
struct amt_offload_caps caps;
+ enum amt_offload_path path;
/* Cleared first on teardown; ops->xmit is only called while set. */
bool admitting;
};
@@ -4209,6 +4217,10 @@ struct amt_offload_state {
struct list_head parked;
struct list_head ep_gc;
bool desync;
+ /* The binding is on the ingress path: a member change must not
+ * clear hw_active, which there gates the punt-drop. Set at bind.
+ */
+ bool ingress;
/* Reconciler. bind and teardown also take the mutex. */
struct mutex mutex;
struct work_struct work;
@@ -4286,9 +4298,9 @@ static u32 amt_offload_outer_family(const struct amt_dev *amt)
}
/* The data path's way in: a binding is usable only while admitting. The
- * caller holds rcu_read_lock(). The later TX handoff is its first caller.
+ * caller holds rcu_read_lock().
*/
-static __maybe_unused struct amt_offload_bound *
+static struct amt_offload_bound *
amt_offload_tx_admit(struct amt_dev *amt)
{
struct amt_offload_bound *b = rcu_dereference(amt->offload);
@@ -4402,13 +4414,15 @@ static void amt_ofl_kick(struct amt_offload_state *st)
}
/* Desired state could not record a change. Hardware would forward to a
- * stale member set, so the relay leaves offload for good: the data path
- * stops consulting the provider, and the next reconciler pass gives every
- * provider object back (group_disable, endpoint_del; the binding stays,
- * with no unbind). Desync does not schedule that pass: a pass already
- * queued or a later removal of a recorded member (by a leave, a timer or
- * a tunnel removal) does, and a teardown gives everything back itself
- * and unbinds. Caller holds st->lock.
+ * stale member set, so the relay leaves offload for good. On TX that
+ * takes effect at once: the data path stops handing packets to the
+ * provider. The next reconciler pass gives every provider object back
+ * (group_disable, endpoint_del; the binding stays, with no unbind).
+ * Desync does not schedule that pass: a pass already queued or a later
+ * removal of a recorded member (by a leave, a timer or a tunnel removal)
+ * does, and a teardown gives everything back itself and unbinds. On an
+ * ingress binding the provider replicates to its committed sets until
+ * then. Caller holds st->lock.
*/
static void amt_ofl_desync(struct amt_offload_state *st)
{
@@ -4562,12 +4576,16 @@ static void amt_ofl_member_unlink(struct amt_offload_state *st,
g->nr_members--;
g->gen++;
g->left++;
- /* The committed member set still names this endpoint: stop handing
- * the group to hardware now, not when the reconciler recommits, so
- * no packet admitted after this is replicated to it. Software serves
- * the group meanwhile.
+ /* TX: the committed member set still names this endpoint. Stop
+ * handing the group to hardware now, not when the reconciler
+ * recommits, so no packet admitted after this is replicated to it;
+ * software serves the group meanwhile. Ingress: hardware replicates
+ * on its own until the recommit whatever the host does, and
+ * hw_active there only drops the punted copies, so clearing it
+ * would send the remaining members every packet twice.
*/
- WRITE_ONCE(g->hw_active, false);
+ if (!READ_ONCE(st->ingress))
+ WRITE_ONCE(g->hw_active, false);
amt_ofl_grp_dirty(st, g);
ep->refs--;
amt_ofl_ep_unused(st, ep);
@@ -4966,7 +4984,10 @@ static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
* committed, and amt_ofl_member_unlink() already withdrew the group.
* A join meanwhile does not hold the publication back: the committed
* set is a subset that names no departed endpoint. Either change
- * re-queued @g; the next pass commits it.
+ * re-queued @g; the next pass commits it. On ingress, hardware is
+ * already replicating to the committed set whatever the host does,
+ * and hw_active only drops the punted copies, so withholding it there
+ * protects no one and sends the remaining members every packet twice.
*/
spin_lock_bh(&st->lock);
for (i = 0; i < n; i++)
@@ -4978,7 +4999,7 @@ static unsigned int amt_ofl_grp_program(struct amt_offload_state *st,
swap(g->ceps, eps); /* out: frees the old set */
g->nr_ceps = n;
WRITE_ONCE(g->hw_gen, gen);
- if (g->left == left)
+ if (READ_ONCE(st->ingress) || g->left == left)
smp_store_release(&g->hw_active, true);
spin_unlock_bh(&st->lock);
goto out;
@@ -5238,6 +5259,135 @@ static void amt_ofl_state_free(struct amt_dev *amt)
amt->offload_st = NULL;
}
+/* Cutover, per packet; exactly one actor handles each one.
+ *
+ * relay data packet (inner IP; Ethernet pulled)
+ * |
+ * +-- bound, admitting, desired state in sync, and (family, group)
+ * | hw_active? no --> software fan-out
+ * |
+ * +-- TX path: ops->xmit(grp, hw_gen)
+ * | CONSUMED --> the provider handled every copy and owns the skb
+ * | FALLBACK --> skb untouched --> software fan-out
+ * | BUSY --> skb untouched --> software fan-out, counted
+ * | (never NETDEV_TX_BUSY: the relay is IFF_NO_QUEUE)
+ * |
+ * +-- ingress path: a punted copy of an hw_active group reaching here
+ * is dropped and counted; one that took the "no" branch is
+ * forwarded in software. A provider cannot tell which (amt
+ * publishes after group_commit returns, and withdraws before
+ * the callback on a reset, park or teardown and with no
+ * callback on desync), so it must never punt a packet it
+ * replicates.
+ *
+ * hw_gen is read once per packet. The reconciler writes it after
+ * group_commit returns 0, before it publishes hw_active, and also when a
+ * member left meanwhile and it withholds hw_active. So a packet already
+ * past the hw_active test may carry that new generation, and one that
+ * read hw_gen earlier any older committed one; the provider serves or
+ * declines it.
+ *
+ * Returns true when the skb is gone (consumed, or a punted copy dropped);
+ * false sends it to the software fan-out untouched.
+ */
+static bool amt_offload_tx(struct amt_dev *amt, struct sk_buff *skb,
+ const union amt_addr *group, bool v6)
+{
+ struct amt_offload_group_key key;
+ enum amt_offload_result res;
+ struct amt_offload_bound *b;
+ struct amt_ofl_grp *g;
+ bool gone = false;
+
+ rcu_read_lock();
+ b = amt_offload_tx_admit(amt);
+ if (!b || READ_ONCE(amt->offload_st->desync))
+ goto out;
+ amt_ofl_grp_key(&key, group, v6);
+ g = amt_ofl_grp_find(amt->offload_st, &key);
+ /* Pairs with the release in amt_ofl_grp_program(). */
+ if (!g || !smp_load_acquire(&g->hw_active))
+ goto out;
+
+ if (b->path == AMT_OFFLOAD_PATH_INGRESS) {
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_PUNT_DROPPED);
+ consume_skb(skb);
+ gone = true;
+ goto out;
+ }
+
+ res = b->ops->xmit(&b->pub, g->hw, skb, READ_ONCE(g->hw_gen));
+ switch (res) {
+ case AMT_OFFLOAD_CONSUMED:
+ gone = true;
+ break;
+ case AMT_OFFLOAD_BUSY:
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_BUSY_FALLBACK);
+ break;
+ case AMT_OFFLOAD_FALLBACK:
+ break;
+ default:
+ WARN_ONCE(1, "amt: offload xmit returned %d\n", res);
+ break;
+ }
+out:
+ rcu_read_unlock();
+ return gone;
+}
+
+/* One gateway's copy for a simulated provider, built and sent exactly
+ * as amt_send_multicast_data() builds and sends a software copy: the
+ * same copy, AMT Multicast Data header, UDP ports, TOS and hop limit,
+ * through amt_udp_xmit(). Called from ops->xmit, in softirq context
+ * under rcu_read_lock().
+ */
+int amt_offload_encap_xmit(struct amt_offload_binding *b,
+ const struct amt_offload_endpoint_key *key,
+ struct sk_buff *oskb)
+{
+ struct amt_dev *amt = netdev_priv(b->dev);
+ struct amt_header_mcast_data *amtmd;
+ bool v6 = ip_hdr(oskb)->version == 6;
+ union amt_addr daddr = {0,};
+ struct socket *sock;
+ struct sk_buff *skb;
+
+ sock = rcu_dereference_bh(amt->sock);
+ if (!sock)
+ return -ENETDOWN;
+ if (key->family != (amt_v6(amt) ? AF_INET6 : AF_INET))
+ return -EAFNOSUPPORT;
+
+ skb = skb_copy_expand(oskb, sizeof(*amtmd) + amt_ip_hlen(amt) +
+ sizeof(struct udphdr), 0, GFP_ATOMIC);
+ if (!skb)
+ goto drop;
+
+ skb_reset_inner_headers(skb);
+ amtmd = skb_push(skb, sizeof(*amtmd));
+ amtmd->version = 0;
+ amtmd->reserved = 0;
+ amtmd->type = AMT_MSG_MULTICAST_DATA;
+ skb_set_inner_protocol(skb, htons(v6 ? ETH_P_IPV6 : ETH_P_IP));
+
+#if IS_ENABLED(CONFIG_IPV6)
+ if (key->family == AF_INET6)
+ daddr.ip6 = key->addr.ip6;
+ else
+#endif
+ daddr.ip4 = key->addr.ip4;
+ if (amt_udp_xmit(amt, sock->sk, skb, &daddr, amt->relay_port,
+ key->port, 0))
+ goto free;
+ return 0;
+free:
+ kfree_skb(skb);
+drop:
+ DEV_STATS_INC(amt->dev, tx_dropped);
+ return -ENETUNREACH;
+}
+EXPORT_SYMBOL_GPL(amt_offload_encap_xmit);
+
static void amt_offload_set_carrier(struct amt_dev *amt, bool on)
{
if (amt->offload_mode != AMT_OFFLOAD_MODE_REQUIRED)
@@ -5285,9 +5435,27 @@ static int amt_offload_bind(struct amt_dev *amt,
kfree(b);
return -EOPNOTSUPP;
}
+ /* TX when offered. Ingress only for a required relay: a packet
+ * consumed on ingress never reaches the host, so nothing on its
+ * forwarding path sees it, and auto never chooses that.
+ */
+ if ((b->caps.paths & BIT(AMT_OFFLOAD_PATH_TX)) && b->ops->xmit) {
+ b->path = AMT_OFFLOAD_PATH_TX;
+ } else if ((b->caps.paths & BIT(AMT_OFFLOAD_PATH_INGRESS)) &&
+ amt->offload_mode == AMT_OFFLOAD_MODE_REQUIRED) {
+ b->path = AMT_OFFLOAD_PATH_INGRESS;
+ } else {
+ b->ops->unbind(&b->pub);
+ amt_offload_count(amt, AMT_OFFLOAD_CNT_PATH_UNSUPPORTED);
+ kfree(b);
+ return -EOPNOTSUPP;
+ }
b->admitting = true;
mutex_lock(&amt->offload_st->mutex);
+ /* Before any group can hold hw under this binding. */
+ WRITE_ONCE(amt->offload_st->ingress,
+ b->path == AMT_OFFLOAD_PATH_INGRESS);
rcu_assign_pointer(amt->offload, b);
/* Replay desired state: every group is programmed afresh. */
amt_ofl_requeue_all(amt->offload_st);
@@ -5506,6 +5674,9 @@ AMT_OFFLOAD_CNT_ATTR(sw_remainder, AMT_OFFLOAD_CNT_SW_REMAINDER);
AMT_OFFLOAD_CNT_ATTR(resets, AMT_OFFLOAD_CNT_RESET);
AMT_OFFLOAD_CNT_ATTR(stale_completions, AMT_OFFLOAD_CNT_STALE);
AMT_OFFLOAD_CNT_ATTR(desync, AMT_OFFLOAD_CNT_DESYNC);
+AMT_OFFLOAD_CNT_ATTR(path_unsupported, AMT_OFFLOAD_CNT_PATH_UNSUPPORTED);
+AMT_OFFLOAD_CNT_ATTR(busy_fallback, AMT_OFFLOAD_CNT_BUSY_FALLBACK);
+AMT_OFFLOAD_CNT_ATTR(punt_dropped, AMT_OFFLOAD_CNT_PUNT_DROPPED);
static struct attribute *amt_offload_attrs[] = {
&dev_attr_offload_mode.attr,
@@ -5521,6 +5692,9 @@ static struct attribute *amt_offload_attrs[] = {
&dev_attr_offload_resets.attr,
&dev_attr_offload_stale_completions.attr,
&dev_attr_offload_desync.attr,
+ &dev_attr_offload_path_unsupported.attr,
+ &dev_attr_offload_busy_fallback.attr,
+ &dev_attr_offload_punt_dropped.attr,
NULL,
};
diff --git a/include/net/amt.h b/include/net/amt.h
index 0c746e4be..0b57c8604 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -389,7 +389,9 @@ struct amt_offload_state;
/* Per-relay offload counters, exported under offload/ in sysfs. */
enum amt_offload_counter {
- /* bind() succeeded and the caps covered the outer family */
+ /* bind() succeeded, and the caps covered the outer family and offered
+ * a data path this relay may use
+ */
AMT_OFFLOAD_CNT_BIND,
/* ... of which: bound when a provider registered on the stream_dev
* this relay already uses
@@ -418,6 +420,15 @@ enum amt_offload_counter {
AMT_OFFLOAD_CNT_STALE,
/* desired state lost an update (allocation failure); software only */
AMT_OFFLOAD_CNT_DESYNC,
+ /* no data path this relay may use: the caps offered none, or TX
+ * without ops->xmit (ingress counts only in required mode); it stayed
+ * unbound
+ */
+ AMT_OFFLOAD_CNT_PATH_UNSUPPORTED,
+ /* xmit returned BUSY: one synchronous software pass ran instead */
+ AMT_OFFLOAD_CNT_BUSY_FALLBACK,
+ /* ingress: a punted copy of a group the provider replicates, dropped */
+ AMT_OFFLOAD_CNT_PUNT_DROPPED,
AMT_OFFLOAD_CNT_MAX,
};
diff --git a/include/net/amt_offload.h b/include/net/amt_offload.h
index b5c119851..99f78699f 100644
--- a/include/net/amt_offload.h
+++ b/include/net/amt_offload.h
@@ -42,8 +42,16 @@ struct sk_buff;
* group without a callback when a member leaves, until it publishes the
* recommit. Every other packet goes straight to software.
* @AMT_OFFLOAD_PATH_INGRESS: the provider replicates on ingress and the
- * host only sees punted packets. Semantics only until a real provider
- * exists; no simulated provider implements it.
+ * host only sees punted packets. From &amt_offload_ops.group_commit
+ * until &amt_offload_ops.group_disable or &amt_offload_ops.flush the
+ * provider replicates every packet of the group, and it must never also
+ * punt one. amt drops, and counts, a punted copy of a group only while
+ * the group's commit is published: from some time after group_commit
+ * returns until amt withdraws it, which it does before any group_disable
+ * or flush and, with no callback, once its desired state is lost. A
+ * punted copy outside that window is forwarded in software as well.
+ * Semantics only until a real provider exists; no simulated provider
+ * implements it, and only a relay in required mode binds on it.
*/
enum amt_offload_path {
AMT_OFFLOAD_PATH_TX,
@@ -203,7 +211,12 @@ struct amt_offload_stats {
* @group_disable: stop forwarding @grp and free it. amt has already
* stopped submitting packets for @grp and waited out @xmit callers.
* @xmit: TX path. Replicate @skb for @grp under @gen or decline; see
- * &enum amt_offload_result. Softirq context, under rcu_read_lock().
+ * &enum amt_offload_result. @skb is the relay data packet with its
+ * data at the inner IP header. On %AMT_OFFLOAD_CONSUMED the provider
+ * owns @skb and must have handled every copy under @gen alone: each one
+ * handed to the stack, or dropped and counted (see
+ * %AMT_OFFLOAD_CONSUMED). Otherwise it must not have sent any copy, nor
+ * modified or freed @skb. Softirq context, under rcu_read_lock().
* @flush: drop every endpoint and group (reset after a timeout). amt then
* replays from its own desired state under a new epoch, except when
* the op that overran was a @group_disable or @endpoint_del of amt
@@ -318,4 +331,23 @@ int amt_offload_register(struct net_device *lower,
*/
void amt_offload_unregister(struct net_device *lower);
+/**
+ * amt_offload_encap_xmit - emit one gateway's copy of a relay data packet
+ * @b: the binding @skb was handed to the provider under.
+ * @key: the gateway endpoint, as amt passed it to endpoint_add.
+ * @skb: the packet given to &amt_offload_ops.xmit; not consumed.
+ *
+ * For simulated providers: encapsulates a copy of @skb as AMT Multicast
+ * Data to @key and transmits it on the relay's stream device, byte for
+ * byte as amt's own software fan-out would. Which endpoints get a copy
+ * is the provider's decision, made from its own tables. Call only from
+ * &amt_offload_ops.xmit.
+ *
+ * Return: 0 if the copy was handed to the stack, else a negative errno
+ * (counted in the relay's tx_dropped when the copy or route failed).
+ */
+int amt_offload_encap_xmit(struct amt_offload_binding *b,
+ const struct amt_offload_endpoint_key *key,
+ struct sk_buff *skb);
+
#endif /* _NET_AMT_OFFLOAD_H_ */
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 5/9] amt: add IFLA_AMT_OFFLOAD_MODE
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (3 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 4/9] amt: offload TX handoff, ingress punt rule and counters Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 6/9] amt: KUnit cases for the offload contract Omar Ramadan
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
software (default), auto or required, after
IFLA_AMT_MAX_TUNNELS_PER_SOURCE. An unknown value, or a non-software
mode on a gateway, is refused with extack. required with no usable
provider on the stream device refuses the link with -ENODEV and extack.
Emitted by fill_info.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/amt.c | 25 ++++++++++++++++++++++++-
include/uapi/linux/amt.h | 7 +++++++
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 377660e55..58e9da443 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -5719,6 +5719,7 @@ static const struct nla_policy amt_policy[IFLA_AMT_MAX + 1] = {
[IFLA_AMT_NUM_QUEUES] = NLA_POLICY_MAX(NLA_U32, AMT_MAX_QUEUES),
[IFLA_AMT_DISCOVERY_IP6] = NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
[IFLA_AMT_REMOTE_IP6] = NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
+ [IFLA_AMT_OFFLOAD_MODE] = { .type = NLA_U32 },
};
static int amt_validate(struct nlattr *tb[], struct nlattr *data[],
@@ -5745,6 +5746,22 @@ static int amt_validate(struct nlattr *tb[], struct nlattr *data[],
return -EINVAL;
}
+ if (data[IFLA_AMT_OFFLOAD_MODE]) {
+ u32 offload = nla_get_u32(data[IFLA_AMT_OFFLOAD_MODE]);
+
+ if (offload > AMT_OFFLOAD_MODE_MAX) {
+ NL_SET_ERR_MSG_ATTR(extack, data[IFLA_AMT_OFFLOAD_MODE],
+ "Offload mode is not valid");
+ return -EINVAL;
+ }
+ if (offload != AMT_OFFLOAD_MODE_SOFTWARE &&
+ nla_get_u32(data[IFLA_AMT_MODE]) != AMT_MODE_RELAY) {
+ NL_SET_ERR_MSG_ATTR(extack, data[IFLA_AMT_OFFLOAD_MODE],
+ "Offload is only valid in relay mode");
+ return -EINVAL;
+ }
+ }
+
if (!data[IFLA_AMT_LOCAL_IP] && !data[IFLA_AMT_LOCAL_IP6]) {
NL_SET_ERR_MSG_MOD(extack,
"Local IPv4 or IPv6 attribute is required");
@@ -5802,6 +5819,9 @@ static int amt_newlink(struct net_device *dev,
amt->net = link_net;
amt->mode = nla_get_u32(data[IFLA_AMT_MODE]);
+ amt->offload_mode = data[IFLA_AMT_OFFLOAD_MODE] ?
+ nla_get_u32(data[IFLA_AMT_OFFLOAD_MODE]) :
+ AMT_OFFLOAD_MODE_SOFTWARE;
if (data[IFLA_AMT_MAX_TUNNELS] &&
nla_get_u32(data[IFLA_AMT_MAX_TUNNELS]))
@@ -5998,7 +6018,8 @@ static size_t amt_get_size(const struct net_device *dev)
nla_total_size(sizeof(struct iphdr)) + /* IFLA_AMT_LOCAL_IP */
nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_LOCAL_IP6 */
nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_DISCOVERY_IP6 */
- nla_total_size(sizeof(struct in6_addr)); /* IFLA_AMT_REMOTE_IP6 */
+ nla_total_size(sizeof(struct in6_addr)) + /* IFLA_AMT_REMOTE_IP6 */
+ nla_total_size(sizeof(__u32)); /* IFLA_AMT_OFFLOAD_MODE */
}
static int amt_fill_info(struct sk_buff *skb, const struct net_device *dev)
@@ -6068,6 +6089,8 @@ static int amt_fill_info(struct sk_buff *skb, const struct net_device *dev)
goto nla_put_failure;
if (nla_put_u32(skb, IFLA_AMT_NUM_QUEUES, dev->real_num_tx_queues))
goto nla_put_failure;
+ if (nla_put_u32(skb, IFLA_AMT_OFFLOAD_MODE, amt->offload_mode))
+ goto nla_put_failure;
return 0;
diff --git a/include/uapi/linux/amt.h b/include/uapi/linux/amt.h
index de1ae1929..c67a1f763 100644
--- a/include/uapi/linux/amt.h
+++ b/include/uapi/linux/amt.h
@@ -122,6 +122,13 @@ enum {
* AMT_MAX_TUNNELS_PER_SOURCE; raise it on a relay that fronts CGNAT.
*/
IFLA_AMT_MAX_TUNNELS_PER_SOURCE,
+ /* Relay mode: enum ifla_amt_offload_mode, default
+ * AMT_OFFLOAD_MODE_SOFTWARE. Replication offload to a provider
+ * registered on the IFLA_AMT_LINK device (include/net/amt_offload.h).
+ * Required mode refuses creation with -ENODEV when no usable provider
+ * is bound. Gateway mode accepts software only.
+ */
+ IFLA_AMT_OFFLOAD_MODE,
__IFLA_AMT_MAX,
};
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 6/9] amt: KUnit cases for the offload contract
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (4 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 5/9] amt: add IFLA_AMT_OFFLOAD_MODE Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 7/9] netdevsim: add a TX reference provider for AMT relay offload Omar Ramadan
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
A relay fixture (amt_test_relay_alloc(), amt_test_tunnel_add()) builds a
v4 relay without registering or opening the device, and links tunnels on
it as amt_request_handler() admits them, listed and counted under
amt->lock with gc_wq armed for one GMI, but without the admission
bounds, the siphash key or the Request/Query exchange. An in-file fake
provider logs every teardown callback in order.
Registry, lifecycle and mode: register-when-bound,
caps-family-unsupported (and bind failure), unregister-mid-fanout
(stream_dev unregistered with groups programmed: order BGGEEU, no
fan-out admitted at any step, binding published until unbind),
provider-unregister ordering with required-mode carrier loss and
rebind, IFLA_AMT_OFFLOAD_MODE validation, and required refusal counted.
Reconciler and cutover, twenty cases: the fake also keeps its own
endpoint and group tables and can refuse with -ENOSPC, fail a commit,
overrun op_timeout_ms or answer xmit with any result. The reconciler
work is disabled in these cases and ticks are driven by hand, so a
static stub on amt_ofl_op_overran() reaches every timed op: the fake
records whether each op stalled, and only such an op overruns, never one
the scheduler happened to delay past a short op_timeout_ms.
- members follow gnodes: a second inner host only counts, the last
gnode removes the member, unlink removes it at once
- churn coalesces into one update and one commit at the last
generation; a tick stops starting groups once its budget is spent
- a refused group is served by software, not retried on its own
member changes, and retried past the low watermark
- an op overrunning op_timeout_ms flushes, bumps the epoch, ignores
the late result and replays; with groups still queued, the tick
ends at the reset
- a teardown whose release ops overrun op_timeout_ms makes one of
them, then flushes, with groups programmed or with only endpoints
(the group parked); every endpoint is still collected afterwards.
The reconciler's gone and endpoint-GC loops reset on an
overrunning release op
- under churn in one group that spends every tick's budget, the
endpoint that left another group is still deleted within three
ticks, and the churning endpoint, which the last commit names, is
never deleted
- admission lost inside an endpoint_add, a group_update or an
endpoint_del (a teardown racing the tick): no further provider op,
and the tick ends with the remaining groups queued
- endpoint_add is idempotent and a shared endpoint is added once
- a failed commit disables the group, keeps its endpoints, and a new
binding retries it; a leave followed by a failed recommit parks the
group and disables its provider object, after a grace period (a
static stub on the reconciler's synchronize_net() logs it)
- the publish gate: a leave landing inside group_commit withholds
that commit's publication and software serves the group, and the
departed member's endpoint stays programmed; a join landing there
does not withhold it
- a group emptied and rejoined within one tick: the fake refuses a
second live group per key, and the rejoined group still commits,
after a grace period for the old group's disable. If that disable
overruns op_timeout_ms, the provider is flushed once and the
rejoined group waits for the next tick
- CONSUMED/FALLBACK/BUSY ownership, unbound and uncommitted groups
stay in software, the generation rule, and a member removal
withdrawing the group from hardware at once
- the last member leaving while the reconciler allocates (static
stubs on its allocator and its free keep a ledger of live arrays):
the retry frees nothing twice
- both reconciler tables are indexed with the device's hash_seed and
sized from max_tunnels at attach
- ingress: a punted copy of a committed group is dropped and counted,
and still is after a member leaves, including one landing inside
the group's first commit, whose departed endpoint stays
programmed; only a required relay binds on ingress
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/amt.c | 2062 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 2062 insertions(+)
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 58e9da443..bafe21302 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -6201,6 +6201,7 @@ static struct notifier_block amt_notifier_block __read_mostly = {
#if IS_ENABLED(CONFIG_AMT_KUNIT_TEST)
#include <kunit/test.h>
+#include <linux/delay.h>
/*
* KUnit coverage for relay-mode upstream membership tracking. The cases
@@ -6513,10 +6514,2071 @@ static void amt_ex_transition_test(struct kunit *test)
free_netdev(test_dev);
}
+#define AMT_TEST_GW htonl(0xc6336401) /* 198.51.100.1, one NAT */
+#define AMT_TEST_HOST htonl(0x0a000002) /* 10.0.0.2, inner host */
+#define AMT_TEST_GROUP htonl(0xe801013c) /* 232.1.1.60 */
+#define AMT_TEST_NONCE htonl(0x5eed1234)
+#define AMT_TEST_MAC 0xa1b2c3d4e5f6ULL
+
+/* A v4 relay-mode amt_dev as amt_dev_init leaves it, minus the netdev
+ * registration. Nothing is sent: the device is never opened, and the
+ * upstream reconciler sees upstream_active == false.
+ */
+static struct amt_dev *amt_test_relay_alloc(struct kunit *test,
+ const char *name)
+{
+ struct net_device *dev;
+ struct amt_dev *amt;
+
+ dev = alloc_netdev(sizeof(*amt), name, NET_NAME_UNKNOWN,
+ amt_link_setup);
+ KUNIT_ASSERT_NOT_NULL(test, dev);
+ amt = netdev_priv(dev);
+ amt->dev = dev;
+ amt_upstream_init(amt);
+ spin_lock_init(&amt->lock);
+ INIT_LIST_HEAD(&amt->tunnel_list);
+ amt->mode = AMT_MODE_RELAY;
+ amt->hash_buckets = 8;
+ amt->max_groups = AMT_MAX_GROUP;
+ amt->max_tunnels = AMT_MAX_TUNNELS;
+ amt->max_tunnels_per_source = AMT_MAX_TUNNELS_PER_SOURCE;
+ amt->qrv = 2;
+ amt->qi = AMT_INIT_QUERY_INTERVAL;
+ amt->qri = 10;
+ KUNIT_ASSERT_EQ(test, rhltable_init(&amt->groups_rhl,
+ &amt_gnode_params), 0);
+ return amt;
+}
+
+static void amt_test_relay_free(struct amt_dev *amt)
+{
+ cancel_delayed_work_sync(&amt->upstream_work);
+ __amt_source_gc_work();
+ rhltable_destroy(&amt->groups_rhl);
+ free_netdev(amt->dev);
+}
+
+/* A tunnel linked on @amt as amt_request_handler() admits one, holding
+ * the nonce and MAC an Update must echo. The admission bounds, the
+ * siphash key and the Request/Query exchange are skipped.
+ */
+static struct amt_tunnel_list *
+amt_test_tunnel_add(struct kunit *test, struct amt_dev *amt, __be32 addr,
+ __be16 port, __be32 nonce, u64 mac)
+{
+ struct amt_tunnel_list *tunnel;
+
+ tunnel = kzalloc_obj(*tunnel, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, tunnel);
+ tunnel->amt = amt;
+ spin_lock_init(&tunnel->lock);
+ INIT_LIST_HEAD(&tunnel->groups);
+ tunnel->addr.ip4 = addr;
+ tunnel->source_port = port;
+ tunnel->nonce = nonce;
+ tunnel->mac = mac;
+ INIT_DELAYED_WORK(&tunnel->gc_wq, amt_tunnel_expire);
+
+ spin_lock_bh(&amt->lock);
+ list_add_tail_rcu(&tunnel->list, &amt->tunnel_list);
+ amt->nr_tunnels++;
+ mod_delayed_work(amt_wq, &tunnel->gc_wq,
+ msecs_to_jiffies(amt_gmi(amt)));
+ spin_unlock_bh(&amt->lock);
+ return tunnel;
+}
+
+/* Offload contract: an in-file fake provider. It logs every
+ * teardown-side callback in order (B bind, G group_disable, E
+ * endpoint_del, U unbind), so registration, binding and the documented
+ * teardown order are asserted without a real provider, and keeps its
+ * own endpoint and group tables so the reconciler's programming can be
+ * checked against what the provider ended up holding.
+ */
+#define AMT_FAKE_LOG_MAX 32
+#define AMT_FAKE_SLOTS 16
+
+struct amt_fake_obj {
+ bool used;
+ struct amt_offload_endpoint_key key; /* endpoints */
+ struct amt_offload_group_key gkey; /* groups */
+ u64 gen; /* groups: staged */
+ u64 committed;
+ u32 nr;
+};
+
+static struct {
+ char log[AMT_FAKE_LOG_MAX];
+ int n;
+ u32 families;
+ u32 paths;
+ u32 op_timeout_ms;
+ int bind_err;
+ /* Relay whose admission state the teardown callbacks sample. */
+ struct amt_dev *amt;
+ bool admitted_in_teardown;
+ bool unpublished_in_teardown;
+ struct amt_fake_obj eps[AMT_FAKE_SLOTS];
+ struct amt_fake_obj grps[AMT_FAKE_SLOTS];
+ /* endpoint_add refuses with -ENOSPC once this many are held */
+ int ep_cap;
+ int commit_err;
+ /* run once from inside the next group_commit, then cleared */
+ void (*commit_hook)(void);
+ /* the next endpoint_add overruns op_timeout_ms, then succeeds */
+ bool add_stall;
+ /* every group_disable and endpoint_del overruns op_timeout_ms */
+ bool release_stall;
+ /* whether the op that returned last overran, read by the overrun
+ * stub; every op amt times sets it on entry, or it inherits the
+ * previous op's verdict
+ */
+ bool stalled;
+ /* clear b->admitting from inside the next op named here ('A'
+ * endpoint_add, 'U' group_update, 'E' endpoint_del), as a teardown
+ * racing the reconciler would; then cleared
+ */
+ char drop_admit;
+ /* group_disable calls on a NULL handle; the contract allows none */
+ int null_disables;
+ int ep_adds;
+ /* first uses refused: the key's earlier group was still live */
+ int dup_grps;
+ int updates;
+ int commits;
+ int flushes;
+ enum amt_offload_result xmit_res;
+ int xmits;
+ u64 xmit_gen;
+} amt_fake;
+
+static void amt_fake_reset(u32 families)
+{
+ memset(&amt_fake, 0, sizeof(amt_fake));
+ amt_fake.families = families;
+ amt_fake.paths = BIT(AMT_OFFLOAD_PATH_TX);
+}
+
+static int amt_fake_live(const struct amt_fake_obj *objs)
+{
+ int i, n = 0;
+
+ for (i = 0; i < AMT_FAKE_SLOTS; i++)
+ n += objs[i].used;
+ return n;
+}
+
+static struct amt_fake_obj *amt_fake_slot(struct amt_fake_obj *objs)
+{
+ int i;
+
+ for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+ if (!objs[i].used) {
+ memset(&objs[i], 0, sizeof(objs[i]));
+ objs[i].used = true;
+ return &objs[i];
+ }
+ }
+ return NULL;
+}
+
+/* What a fan-out racing the teardown would see. */
+static void amt_fake_log(char c)
+{
+ if (amt_fake.n < AMT_FAKE_LOG_MAX - 1)
+ amt_fake.log[amt_fake.n++] = c;
+ if (!amt_fake.amt || c == 'B')
+ return;
+ rcu_read_lock();
+ if (amt_offload_tx_admit(amt_fake.amt))
+ amt_fake.admitted_in_teardown = true;
+ if (!rcu_access_pointer(amt_fake.amt->offload))
+ amt_fake.unpublished_in_teardown = true;
+ rcu_read_unlock();
+}
+
+static void amt_fake_drop_admit(struct amt_offload_binding *b, char op)
+{
+ if (amt_fake.drop_admit != op)
+ return;
+ amt_fake.drop_admit = 0;
+ WRITE_ONCE(container_of(b, struct amt_offload_bound, pub)->admitting,
+ false);
+}
+
+static int amt_fake_bind(struct amt_offload_binding *b,
+ struct amt_offload_caps *caps)
+{
+ amt_fake_log('B');
+ if (amt_fake.bind_err)
+ return amt_fake.bind_err;
+ caps->families = amt_fake.families;
+ caps->paths = amt_fake.paths;
+ caps->op_timeout_ms = amt_fake.op_timeout_ms;
+ return 0;
+}
+
+static void amt_fake_unbind(struct amt_offload_binding *b)
+{
+ amt_fake_log('U');
+}
+
+/* Idempotent on an identical key, as the contract requires. */
+static int amt_fake_endpoint_add(struct amt_offload_binding *b,
+ const struct amt_offload_endpoint_key *key,
+ struct amt_offload_endpoint **ep)
+{
+ struct amt_fake_obj *o;
+ int i;
+
+ amt_fake_drop_admit(b, 'A');
+ amt_fake.stalled = amt_fake.add_stall;
+ amt_fake.add_stall = false;
+ for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+ o = &amt_fake.eps[i];
+ if (o->used && !memcmp(&o->key, key, sizeof(*key))) {
+ *ep = (struct amt_offload_endpoint *)o;
+ return 0;
+ }
+ }
+ if (amt_fake.ep_cap && amt_fake_live(amt_fake.eps) >= amt_fake.ep_cap)
+ return -ENOSPC;
+ o = amt_fake_slot(amt_fake.eps);
+ if (!o)
+ return -ENOSPC;
+ o->key = *key;
+ amt_fake.ep_adds++;
+ *ep = (struct amt_offload_endpoint *)o;
+ return 0;
+}
+
+static void amt_fake_endpoint_del(struct amt_offload_binding *b,
+ struct amt_offload_endpoint *ep)
+{
+ amt_fake.stalled = amt_fake.release_stall;
+ amt_fake_drop_admit(b, 'E');
+ ((struct amt_fake_obj *)ep)->used = false;
+ amt_fake_log('E');
+}
+
+static int amt_fake_group_update(struct amt_offload_binding *b,
+ const struct amt_offload_group_key *key,
+ struct amt_offload_endpoint *const *members,
+ unsigned int nr_members, u64 gen,
+ struct amt_offload_group **grp)
+{
+ struct amt_fake_obj *o = (struct amt_fake_obj *)*grp;
+ int i;
+
+ amt_fake.stalled = false;
+ amt_fake_drop_admit(b, 'U');
+ if (!o) {
+ /* One group per key, as amt promises: refuse a second. */
+ for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+ o = &amt_fake.grps[i];
+ if (o->used && !memcmp(&o->gkey, key, sizeof(*key))) {
+ amt_fake.dup_grps++;
+ return -EEXIST;
+ }
+ }
+ o = amt_fake_slot(amt_fake.grps);
+ if (!o)
+ return -ENOSPC;
+ o->gkey = *key;
+ *grp = (struct amt_offload_group *)o;
+ }
+ o->gen = gen;
+ o->nr = nr_members;
+ amt_fake.updates++;
+ return 0;
+}
+
+static int amt_fake_group_commit(struct amt_offload_binding *b,
+ struct amt_offload_group *grp, u64 gen)
+{
+ void (*hook)(void) = amt_fake.commit_hook;
+
+ amt_fake.stalled = false;
+ if (hook) {
+ amt_fake.commit_hook = NULL;
+ hook();
+ }
+ if (amt_fake.commit_err)
+ return amt_fake.commit_err;
+ ((struct amt_fake_obj *)grp)->committed = gen;
+ amt_fake.commits++;
+ return 0;
+}
+
+static void amt_fake_group_disable(struct amt_offload_binding *b,
+ struct amt_offload_group *grp)
+{
+ amt_fake.stalled = grp && amt_fake.release_stall;
+ if (!grp) {
+ amt_fake.null_disables++;
+ return;
+ }
+ ((struct amt_fake_obj *)grp)->used = false;
+ amt_fake_log('G');
+}
+
+static void amt_fake_flush(struct amt_offload_binding *b)
+{
+ int i;
+
+ for (i = 0; i < AMT_FAKE_SLOTS; i++) {
+ amt_fake.eps[i].used = false;
+ amt_fake.grps[i].used = false;
+ }
+ amt_fake.flushes++;
+}
+
+static enum amt_offload_result
+amt_fake_xmit(struct amt_offload_binding *b, struct amt_offload_group *grp,
+ struct sk_buff *skb, u64 gen)
+{
+ amt_fake.xmits++;
+ amt_fake.xmit_gen = gen;
+ if (amt_fake.xmit_res == AMT_OFFLOAD_CONSUMED)
+ consume_skb(skb);
+ return amt_fake.xmit_res;
+}
+
+static const struct amt_offload_ops amt_fake_ops = {
+ .bind = amt_fake_bind,
+ .unbind = amt_fake_unbind,
+ .endpoint_add = amt_fake_endpoint_add,
+ .endpoint_del = amt_fake_endpoint_del,
+ .group_update = amt_fake_group_update,
+ .group_commit = amt_fake_group_commit,
+ .group_disable = amt_fake_group_disable,
+ .flush = amt_fake_flush,
+ .xmit = amt_fake_xmit,
+};
+
+static struct net_device *amt_test_lower_alloc(struct kunit *test)
+{
+ struct net_device *lower;
+
+ lower = alloc_netdev(0, "amtlow%d", NET_NAME_UNKNOWN, ether_setup);
+ KUNIT_ASSERT_NOT_NULL(test, lower);
+ return lower;
+}
+
+/* A relay on @lower as amt_newlink leaves it, minus the netlink half.
+ * Caller holds RTNL.
+ */
+static struct amt_dev *amt_test_offload_relay(struct kunit *test,
+ const char *name,
+ struct net_device *lower,
+ u32 mode)
+{
+ struct amt_dev *amt = amt_test_relay_alloc(test, name);
+
+ amt->stream_dev = lower;
+ amt->offload_mode = mode;
+ return amt;
+}
+
+static void amt_test_offload_free(struct amt_dev *amt)
+{
+ rtnl_lock();
+ amt_offload_detach(amt);
+ rtnl_unlock();
+ amt_ofl_state_free(amt);
+ amt_test_relay_free(amt);
+}
+
+/* A tunnel the offload hooks can name: endpoint identity only. */
+static struct amt_tunnel_list *amt_test_ofl_tunnel(struct kunit *test,
+ __be32 addr, __be16 port)
+{
+ struct amt_tunnel_list *t = kunit_kzalloc(test, sizeof(*t),
+ GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ t->addr.ip4 = addr;
+ t->source_port = port;
+ return t;
+}
+
+static union amt_addr amt_test_ofl_group(int i)
+{
+ union amt_addr g = {};
+
+ g.ip4 = htonl(ntohl(AMT_TEST_GROUP) + i);
+ return g;
+}
+
+/* Run reconciler ticks until it reports no work left; returns the count. */
+static int amt_test_ofl_drain(struct amt_dev *amt)
+{
+ int ticks = 0;
+
+ while (ticks < 1000 && amt_ofl_reconcile(amt))
+ ticks++;
+ return ticks + 1;
+}
+
+static int amt_test_offload_cnt(struct amt_dev *amt,
+ enum amt_offload_counter cnt)
+{
+ return atomic_read(&amt->offload_cnt[cnt]);
+}
+
+/* Two groups and two endpoints the teardown must give back, programmed
+ * through desired state and the reconciler. RTNL held.
+ */
+static void amt_test_offload_seed(struct kunit *test, struct amt_dev *amt)
+{
+ union amt_addr grp;
+ int i;
+
+ KUNIT_ASSERT_NOT_NULL(test, rtnl_dereference(amt->offload));
+ for (i = 0; i < 2; i++) {
+ grp = amt_test_ofl_group(i);
+ amt_ofl_member_add(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000 + i)),
+ &grp, false);
+ }
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+}
+
+static bool amt_test_offload_admits(struct amt_dev *amt)
+{
+ bool ok;
+
+ rcu_read_lock();
+ ok = !!amt_offload_tx_admit(amt);
+ rcu_read_unlock();
+ return ok;
+}
+
+static void amt_offload_register_when_bound_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt, *sw;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4 | AMT_OFFLOAD_FAMILY_V6);
+ amt = amt_test_offload_relay(test, "amtob%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ sw = amt_test_offload_relay(test, "amtos%d", lower,
+ AMT_OFFLOAD_MODE_SOFTWARE);
+
+ rtnl_lock();
+ /* No provider yet: auto attaches in software. */
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(sw, NULL), 0);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+
+ /* Registering on a stream_dev in use binds the auto relay at once. */
+ KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ KUNIT_EXPECT_NOT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(sw->offload));
+ KUNIT_EXPECT_TRUE(test, amt_test_offload_admits(amt));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_BIND), 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_epoch, 1);
+ KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops),
+ -EEXIST);
+
+ amt_offload_unregister(lower);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_epoch, 2);
+ amt_offload_unregister(lower); /* no provider: a no-op */
+ rtnl_unlock();
+
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BU");
+ amt_test_offload_free(sw);
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+static void amt_offload_family_unsupported_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt, *amt2;
+
+ /* A v6-only provider and a v4 relay. */
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V6);
+ amt = amt_test_offload_relay(test, "amtof%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ amt2 = amt_test_offload_relay(test, "amtoe%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+
+ rtnl_lock();
+ KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED), 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_BIND), 0);
+
+ /* A bind() error is a different count and never reaches unbind(). */
+ amt_fake.bind_err = -EIO;
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt2, NULL), 0);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt2->offload));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt2,
+ AMT_OFFLOAD_CNT_BIND_FAILED), 1);
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BUB");
+ amt_test_offload_free(amt2);
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* The stream_dev is unregistered with groups programmed. A fan-out
+ * racing the teardown must not be admitted at any step, while the
+ * binding stays published until after unbind().
+ */
+static void amt_offload_unregister_mid_fanout_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt = amt_test_offload_relay(test, "amtom%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+
+ rtnl_lock();
+ KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ KUNIT_ASSERT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 0);
+ KUNIT_EXPECT_TRUE(test, amt_test_offload_admits(amt));
+ amt_test_offload_seed(test, amt);
+ amt_fake.amt = amt;
+
+ amt_offload_lower_gone(lower);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_NULL(test, amt_offload_provider_find(lower));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_TEARDOWN_LOWER), 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 0);
+ rtnl_unlock();
+
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGGEEU");
+ KUNIT_EXPECT_FALSE(test, amt_fake.admitted_in_teardown);
+ KUNIT_EXPECT_FALSE(test, amt_fake.unpublished_in_teardown);
+ KUNIT_EXPECT_FALSE(test, amt_test_offload_admits(amt));
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* Same order when the provider leaves first. A required relay drops
+ * carrier, counts it, and comes back when a provider registers again.
+ */
+static void amt_offload_provider_unregister_order_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt = amt_test_offload_relay(test, "amtop%d", lower,
+ AMT_OFFLOAD_MODE_REQUIRED);
+
+ rtnl_lock();
+ KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ KUNIT_ASSERT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ amt_test_offload_seed(test, amt);
+ amt_fake.amt = amt;
+
+ amt_offload_unregister(lower);
+ KUNIT_EXPECT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_TEARDOWN_PROVIDER), 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_REQUIRED_UNBOUND), 1);
+ KUNIT_EXPECT_FALSE(test, netif_carrier_ok(amt->dev));
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGGEEU");
+ KUNIT_EXPECT_FALSE(test, amt_fake.admitted_in_teardown);
+
+ amt_fake.amt = NULL;
+ KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ KUNIT_EXPECT_NOT_NULL(test, rtnl_dereference(amt->offload));
+ KUNIT_EXPECT_TRUE(test, netif_carrier_ok(amt->dev));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_BIND_ON_REGISTER), 1);
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+struct amt_test_u32_attr {
+ struct nlattr nla;
+ u32 val;
+};
+
+static struct nlattr *amt_test_u32(struct amt_test_u32_attr *a, int type,
+ u32 val)
+{
+ a->nla.nla_len = nla_attr_size(sizeof(u32));
+ a->nla.nla_type = type;
+ a->val = val;
+ return &a->nla;
+}
+
+static int amt_test_validate_offload(u32 mode, u32 offload,
+ struct netlink_ext_ack *extack)
+{
+ struct amt_test_u32_attr link, amode, local, off;
+ struct nlattr *data[IFLA_AMT_MAX + 1] = {};
+
+ data[IFLA_AMT_LINK] = amt_test_u32(&link, IFLA_AMT_LINK, 1);
+ data[IFLA_AMT_MODE] = amt_test_u32(&amode, IFLA_AMT_MODE, mode);
+ data[IFLA_AMT_LOCAL_IP] = amt_test_u32(&local, IFLA_AMT_LOCAL_IP,
+ (__force u32)AMT_TEST_GW);
+ data[IFLA_AMT_OFFLOAD_MODE] = amt_test_u32(&off, IFLA_AMT_OFFLOAD_MODE,
+ offload);
+ memset(extack, 0, sizeof(*extack));
+ return amt_validate(NULL, data, extack);
+}
+
+static void amt_offload_mode_validate_test(struct kunit *test)
+{
+ struct netlink_ext_ack extack;
+
+ KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+ AMT_OFFLOAD_MODE_SOFTWARE, &extack), 0);
+ KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+ AMT_OFFLOAD_MODE_AUTO, &extack), 0);
+ KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+ AMT_OFFLOAD_MODE_REQUIRED, &extack), 0);
+
+ KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_RELAY,
+ AMT_OFFLOAD_MODE_MAX + 1, &extack), -EINVAL);
+ KUNIT_ASSERT_NOT_NULL(test, extack._msg);
+ KUNIT_EXPECT_STREQ(test, extack._msg, "Offload mode is not valid");
+
+ KUNIT_EXPECT_EQ(test, amt_test_validate_offload(AMT_MODE_GATEWAY,
+ AMT_OFFLOAD_MODE_AUTO, &extack), -EINVAL);
+ KUNIT_ASSERT_NOT_NULL(test, extack._msg);
+ KUNIT_EXPECT_STREQ(test, extack._msg,
+ "Offload is only valid in relay mode");
+}
+
+static void amt_offload_required_refused_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct netlink_ext_ack extack = {};
+ struct amt_dev *amt;
+ int before;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V6);
+ amt = amt_test_offload_relay(test, "amtor%d", lower,
+ AMT_OFFLOAD_MODE_REQUIRED);
+ before = atomic_read(&amt_offload_required_refused);
+
+ rtnl_lock();
+ /* No provider at all. */
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), -ENODEV);
+ KUNIT_EXPECT_NOT_NULL(test, extack._msg);
+ KUNIT_EXPECT_TRUE(test, list_empty(&amt->offload_node));
+ KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+ before + 1);
+
+ /* A provider that cannot carry this relay's outer family. */
+ KUNIT_ASSERT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ memset(&extack, 0, sizeof(extack));
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), -ENODEV);
+ KUNIT_EXPECT_NOT_NULL(test, extack._msg);
+ KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+ before + 2);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_FAMILY_UNSUPPORTED), 1);
+
+ /* auto with the same provider is not a refusal. */
+ amt->offload_mode = AMT_OFFLOAD_MODE_AUTO;
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, &extack), 0);
+ KUNIT_EXPECT_EQ(test, atomic_read(&amt_offload_required_refused),
+ before + 2);
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* Offload contract: desired state, reconciler and cutover, driven
+ * by hand -- the reconciler work is disabled and amt_ofl_reconcile()
+ * called directly, so every tick is the test's.
+ */
+static struct amt_dev *amt_test_ofl_attach(struct kunit *test,
+ const char *name,
+ struct net_device *lower, u32 mode)
+{
+ struct amt_dev *amt = amt_test_offload_relay(test, name, lower, mode);
+
+ rtnl_lock();
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ rtnl_unlock();
+ KUNIT_ASSERT_NOT_NULL(test, amt->offload_st);
+ disable_work_sync(&amt->offload_st->work);
+ return amt;
+}
+
+/* Whether a provider op overran op_timeout_ms is the fake's call, not the
+ * clock's: on a loaded machine a 10 ms timeout against ktime_get() also
+ * fires on ops that never stalled. The reconciler runs on the test thread
+ * here (its work is disabled), so the stub reaches every timed op.
+ */
+static bool amt_test_ofl_op_overran(struct amt_offload_bound *b, ktime_t t0)
+{
+ return b->caps.op_timeout_ms && amt_fake.stalled;
+}
+
+static void amt_test_ofl_register(struct kunit *test, struct net_device *lower)
+{
+ int err;
+
+ kunit_activate_static_stub(test, amt_ofl_op_overran,
+ amt_test_ofl_op_overran);
+ rtnl_lock();
+ err = amt_offload_register(lower, &amt_fake_ops);
+ rtnl_unlock();
+ KUNIT_ASSERT_EQ(test, err, 0);
+}
+
+static void amt_test_ofl_done(struct amt_dev *amt, struct net_device *lower)
+{
+ rtnl_lock();
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+static void amt_test_ofl_join(struct amt_dev *amt, struct amt_tunnel_list *t,
+ int i, bool join)
+{
+ union amt_addr g = amt_test_ofl_group(i);
+
+ if (join)
+ amt_ofl_member_add(amt, t, &g, false);
+ else
+ amt_ofl_member_del(amt, t, &g, false);
+}
+
+static struct amt_ofl_grp *amt_test_ofl_grp(struct amt_dev *amt, int i)
+{
+ union amt_addr a = amt_test_ofl_group(i);
+ struct amt_offload_group_key key;
+ struct amt_ofl_grp *g;
+
+ amt_ofl_grp_key(&key, &a, false);
+ spin_lock_bh(&amt->offload_st->lock);
+ g = amt_ofl_grp_find(amt->offload_st, &key);
+ spin_unlock_bh(&amt->offload_st->lock);
+ return g;
+}
+
+static bool amt_test_ofl_tx(struct amt_dev *amt, struct sk_buff *skb, int i)
+{
+ union amt_addr g = amt_test_ofl_group(i);
+
+ return amt_offload_tx(amt, skb, &g, false);
+}
+
+static bool amt_test_ofl_no_eps(struct amt_dev *amt)
+{
+ u32 i;
+
+ for (i = 0; i < 1U << amt->offload_st->ep_bits; i++)
+ if (!hlist_empty(&amt->offload_st->eps[i]))
+ return false;
+ return true;
+}
+
+/* Members follow gnodes: the first gnode of a group on a tunnel adds
+ * the member, a second (another inner host) only counts, the last one
+ * removes it; unlink removes it at once and the gnodes deleted after it
+ * find nothing.
+ */
+static void amt_offload_member_gnode_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ union amt_addr grp = amt_test_ofl_group(0), h1 = {}, h2 = {};
+ struct amt_group_node *g1, *g2;
+ struct amt_tunnel_list *tunnel;
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt = amt_test_ofl_attach(test, "amton%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ tunnel = amt_test_tunnel_add(test, amt, AMT_TEST_GW, htons(4000),
+ AMT_TEST_NONCE, AMT_TEST_MAC);
+ h1.ip4 = AMT_TEST_HOST;
+ h2.ip4 = htonl(ntohl(AMT_TEST_HOST) + 1);
+
+ spin_lock_bh(&tunnel->lock);
+ g1 = amt_add_group(amt, tunnel, &grp, &h1, false);
+ g2 = amt_add_group(amt, tunnel, &grp, &h2, false);
+ spin_unlock_bh(&tunnel->lock);
+ KUNIT_ASSERT_FALSE(test, IS_ERR(g1));
+ KUNIT_ASSERT_FALSE(test, IS_ERR(g2));
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+ KUNIT_EXPECT_EQ(test, og->gen, 1);
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+
+ spin_lock_bh(&tunnel->lock);
+ amt_del_group(amt, g1);
+ spin_unlock_bh(&tunnel->lock);
+ KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+ KUNIT_EXPECT_EQ(test, og->gen, 1);
+ spin_lock_bh(&tunnel->lock);
+ amt_del_group(amt, g2);
+ spin_unlock_bh(&tunnel->lock);
+ KUNIT_EXPECT_EQ(test, og->nr_members, 0);
+ KUNIT_EXPECT_EQ(test, og->gen, 2);
+
+ spin_lock_bh(&tunnel->lock);
+ g1 = amt_add_group(amt, tunnel, &grp, &h1, false);
+ spin_unlock_bh(&tunnel->lock);
+ KUNIT_ASSERT_FALSE(test, IS_ERR(g1));
+ KUNIT_EXPECT_EQ(test, og->nr_members, 1);
+ /* Unlink alone, as amt_dev_stop() does before it cancels the
+ * tunnel's gc work: g1 is still in amt->groups_rhl and on
+ * tunnel->groups, yet the member is gone.
+ */
+ spin_lock_bh(&amt->lock);
+ amt_tunnel_unlink(amt, tunnel);
+ spin_unlock_bh(&amt->lock);
+ KUNIT_EXPECT_EQ(test, og->nr_members, 0);
+ KUNIT_EXPECT_EQ(test, og->gen, 4);
+ cancel_delayed_work_sync(&tunnel->gc_wq);
+ amt_clear_groups(tunnel);
+ kfree_rcu(tunnel, rcu);
+ KUNIT_EXPECT_EQ(test, og->gen, 4); /* g1's delete found nothing */
+
+ /* Unbound: the reconciler only frees the emptied records. */
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* Churn between ticks coalesces into one update and one commit at the
+ * last generation; a tick starts no group once its budget is spent.
+ */
+static void amt_offload_budget_coalesce_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[4], *u[3];
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+ int i, round;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoc%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 4; i++)
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ for (round = 0; round < 25; round++) {
+ for (i = 0; i < 4; i++) {
+ amt_test_ofl_join(amt, t[i], 0, true);
+ if (round < 24)
+ amt_test_ofl_join(amt, t[i], 0, false);
+ }
+ }
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_EQ(test, og->nr_members, 4);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.commits, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 4);
+ KUNIT_ASSERT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, og->hw_gen, og->gen);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->committed,
+ og->gen);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 4);
+
+ /* Three groups at three ops each (endpoint_add, update, commit):
+ * a budget of 3 programs one group per tick.
+ */
+ amt->offload_st->budget = 3;
+ for (i = 0; i < 3; i++) {
+ u[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(5000 + i));
+ amt_test_ofl_join(amt, u[i], i + 1, true);
+ }
+ KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 2);
+ KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 3);
+ KUNIT_EXPECT_FALSE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 4);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 4);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* A group the provider refuses goes to software (the remainder) and is
+ * not retried on its own member changes; it is retried once the
+ * provider holds an eighth fewer objects than when it refused.
+ */
+static void amt_offload_enospc_hysteresis_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_ofl_grp *a, *b;
+ struct amt_tunnel_list *t[3];
+ struct amt_dev *amt;
+ int i, updates;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.ep_cap = 2;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtos%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 3; i++)
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[0], 0, true);
+ amt_test_ofl_join(amt, t[1], 0, true);
+ amt_test_ofl_drain(amt);
+ a = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, a);
+ KUNIT_EXPECT_TRUE(test, a->hw_active);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 3);
+
+ amt_test_ofl_join(amt, t[2], 1, true);
+ amt_test_ofl_drain(amt);
+ b = amt_test_ofl_grp(amt, 1);
+ KUNIT_ASSERT_NOT_NULL(test, b);
+ KUNIT_EXPECT_FALSE(test, b->hw_active);
+ KUNIT_EXPECT_EQ(test, b->state, AMT_OFL_PARKED);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+ KUNIT_EXPECT_TRUE(test, a->hw_active);
+
+ /* Parked: a member change does not retry it. */
+ updates = amt_fake.updates;
+ amt_test_ofl_join(amt, t[1], 1, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, b->state, AMT_OFL_PARKED);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, updates);
+
+ /* t[0] leaves A: A is recommitted without it and its endpoint is
+ * deleted, which crosses the low watermark and retries B.
+ */
+ amt_test_ofl_join(amt, t[0], 0, false);
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_TRUE(test, a->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)a->hw)->nr, 1);
+ KUNIT_ASSERT_TRUE(test, b->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)b->hw)->nr, 2);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* An op that overruns op_timeout_ms resets the provider: flush, a new
+ * epoch, the late result ignored, and a replay from desired state.
+ */
+static void amt_offload_timeout_reset_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t;
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+ u64 epoch;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtot%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ epoch = amt->offload_epoch;
+ t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ amt_test_ofl_join(amt, t, 0, true);
+ amt_fake.add_stall = true;
+ amt_test_ofl_drain(amt);
+
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_STALE),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_epoch, epoch + 1);
+ /* The late handle died with the flush; the replay added it again. */
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 2);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* A reset late in a tick must not reach a group emptied earlier in it:
+ * the flush already took its handle, so amt neither disables it again nor
+ * counts it out of hw_objs a second time.
+ */
+static void amt_offload_reset_after_empty_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t1, *t2;
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoe%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ t1 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ t2 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+ amt_test_ofl_join(amt, t1, 0, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+
+ /* One tick: group 0 empties, then group 1's endpoint_add overruns. */
+ amt_test_ofl_join(amt, t1, 0, false);
+ amt_test_ofl_join(amt, t2, 1, true);
+ amt_fake.add_stall = true;
+ amt_test_ofl_drain(amt);
+
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.null_disables, 0);
+ KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 2U);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Teardown's release ops are timed: a provider that stalls
+ * in group_disable/endpoint_del costs one op under RTNL, then a flush,
+ * not one stall per object. Untimed, the log reads "BGGEEU".
+ */
+static void amt_offload_teardown_release_stall_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[2];
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtor%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++) {
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[i], i, true);
+ }
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+
+ amt_fake.release_stall = true;
+ rtnl_lock();
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+ amt_fake.release_stall = false;
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGU");
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+ /* The group the overrun never reached is unpublished too. */
+ for (i = 0; i < 2; i++)
+ KUNIT_EXPECT_FALSE(test,
+ READ_ONCE(amt_test_ofl_grp(amt, i)->hw_active));
+ /* Both groups were uncommitted, the disabled one and the forgotten
+ * one, so once their members leave both endpoints are collected.
+ */
+ for (i = 0; i < 2; i++)
+ amt_test_ofl_join(amt, t[i], i, false);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* Teardown with only endpoints programmed (the group parked): the first
+ * endpoint_del to overrun ends it. Untimed, the log reads "BGEEU".
+ */
+static void amt_offload_teardown_ep_stall_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_fake.commit_err = -EIO;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtot%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++)
+ amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000 + i)),
+ 0, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_test_ofl_grp(amt, 0)->state, AMT_OFL_PARKED);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+
+ amt_fake.release_stall = true;
+ rtnl_lock();
+ amt_offload_unregister(lower);
+ rtnl_unlock();
+ amt_fake.release_stall = false;
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BGEU");
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+ amt_test_offload_free(amt);
+ free_netdev(lower);
+}
+
+/* The gone and GC loops time their release ops too: one
+ * stall, then a flush, not one per object. Each part needs its own relay,
+ * because a gone-loop reset forgets every handle and the GC loop then has
+ * nothing to delete.
+ */
+static struct amt_dev *amt_test_ofl_stall_setup(struct kunit *test,
+ struct net_device *lower,
+ struct amt_tunnel_list **t)
+{
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtos%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++) {
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[i], i, true);
+ }
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+ return amt;
+}
+
+static void amt_offload_reconcile_release_stall_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[3];
+ struct amt_dev *amt;
+ int i;
+
+ /* (a) Two groups empty in one tick: the first group_disable
+ * overruns and resets. Untimed, the GC loop resets one op later
+ * and the log reads "BGGE".
+ */
+ amt = amt_test_ofl_stall_setup(test, lower, t);
+ for (i = 0; i < 2; i++)
+ amt_test_ofl_join(amt, t[i], i, false);
+ amt_fake.release_stall = true;
+ amt_ofl_reconcile(amt);
+ amt_fake.release_stall = false;
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BG");
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.null_disables, 0);
+ /* The reset uncommitted the group it forgot: both endpoints go. */
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+ amt_test_ofl_done(amt, lower);
+
+ /* (b) One endpoint leaves a group, nothing empties: its
+ * endpoint_del overruns and resets. hw_objs is counted down before
+ * the op, so the reset's zeroing is final.
+ */
+ lower = amt_test_lower_alloc(test);
+ amt = amt_test_ofl_stall_setup(test, lower, t);
+ t[2] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4002));
+ amt_test_ofl_join(amt, t[2], 0, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+ amt_test_ofl_join(amt, t[2], 0, false);
+ amt_fake.release_stall = true;
+ amt_ofl_reconcile(amt);
+ amt_fake.release_stall = false;
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BE");
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs, 0U);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->hw_objs,
+ (unsigned int)(amt_fake_live(amt_fake.eps) +
+ amt_fake_live(amt_fake.grps)));
+ amt_test_ofl_done(amt, lower);
+}
+
+/* One group re-dirtied from inside every group_commit: the churn the
+ * hook drives spends each tick's budget on that group.
+ */
+static struct {
+ struct amt_dev *amt;
+ struct amt_tunnel_list *t;
+ bool in;
+ bool on;
+} amt_test_churn;
+
+static void amt_test_churn_hook(void)
+{
+ if (!amt_test_churn.on)
+ return;
+ amt_test_churn.in = !amt_test_churn.in;
+ amt_test_ofl_join(amt_test_churn.amt, amt_test_churn.t, 0,
+ amt_test_churn.in);
+ amt_fake.commit_hook = amt_test_churn_hook;
+}
+
+static bool amt_test_fake_ep_held(const struct amt_offload_endpoint_key *key)
+{
+ int i;
+
+ for (i = 0; i < AMT_FAKE_SLOTS; i++)
+ if (amt_fake.eps[i].used &&
+ !memcmp(&amt_fake.eps[i].key, key, sizeof(*key)))
+ return true;
+ return false;
+}
+
+/* Endpoint GC does not wait for st->dirty to empty: under
+ * sustained churn in one group, a departed endpoint that no committed
+ * group names is deleted within a bounded number of ticks. Gated on an
+ * empty st->dirty it never is.
+ */
+static void amt_offload_gc_churn_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *stay, *churn, *gone;
+ struct amt_offload_endpoint_key key, ckey;
+ struct amt_dev *amt;
+ int tick;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtog%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ stay = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ churn = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+ gone = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4002));
+ amt_test_ofl_join(amt, stay, 0, true);
+ amt_test_ofl_join(amt, churn, 0, true);
+ amt_test_ofl_join(amt, gone, 1, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+ amt_ofl_ep_key(&key, amt, gone);
+ KUNIT_ASSERT_TRUE(test, amt_test_fake_ep_held(&key));
+ amt_ofl_ep_key(&ckey, amt, churn);
+
+ amt->offload_st->budget = 6;
+ memset(&amt_test_churn, 0, sizeof(amt_test_churn));
+ amt_test_churn.amt = amt;
+ amt_test_churn.t = churn;
+ amt_test_churn.on = true;
+ amt_test_ofl_join(amt, churn, 0, false);
+ amt_test_ofl_join(amt, gone, 1, false);
+ amt_fake.commit_hook = amt_test_churn_hook;
+ /* GC no longer waits for st->dirty, so nothing but the commit
+ * accounting keeps it off an endpoint the last commit still names.
+ */
+ for (tick = 0; tick < 3; tick++) {
+ KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&ckey));
+ }
+ KUNIT_EXPECT_FALSE(test, amt_test_fake_ep_held(&key));
+
+ amt_test_churn.on = false;
+ amt_fake.commit_hook = NULL;
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 1);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* endpoint_add is idempotent on an identical key, and amt adds an
+ * endpoint shared by several groups once.
+ */
+static void amt_offload_endpoint_idempotent_test(struct kunit *test)
+{
+ struct amt_offload_endpoint_key key = {};
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_offload_endpoint *h1 = NULL, *h2 = NULL;
+ struct amt_tunnel_list *t;
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ key.family = AF_INET;
+ key.addr.ip4 = AMT_TEST_GW;
+ key.port = htons(4000);
+ KUNIT_EXPECT_EQ(test, amt_fake_endpoint_add(NULL, &key, &h1), 0);
+ KUNIT_EXPECT_EQ(test, amt_fake_endpoint_add(NULL, &key, &h2), 0);
+ KUNIT_EXPECT_PTR_EQ(test, h1, h2);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoi%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ amt_test_ofl_join(amt, t, 0, true);
+ amt_test_ofl_join(amt, t, 1, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 2);
+ /* Leaving and rejoining one group keeps the endpoint programmed. */
+ amt_test_ofl_join(amt, t, 1, false);
+ amt_test_ofl_join(amt, t, 1, true);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* A failed group_commit disables the group and serves it in software;
+ * the endpoints it added stay programmed. A new binding retries it.
+ */
+static void amt_test_ofl_sync_net(void)
+{
+ amt_fake_log('S');
+ synchronize_net();
+}
+
+static void amt_offload_commit_fail_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[2];
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.commit_err = -EIO;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoe%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++) {
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[i], 0, true);
+ }
+ amt_test_ofl_drain(amt);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_FALSE(test, og->hw_active);
+ KUNIT_EXPECT_NULL(test, og->hw);
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_PARKED);
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "BG");
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_SW_REMAINDER), 1);
+
+ amt_fake.commit_err = 0;
+ rtnl_lock();
+ amt_offload_unregister(lower);
+ KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ rtnl_unlock();
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+
+ /* A leave withdraws the committed group, then its recommit fails:
+ * park still disables the provider object it holds, and waits out
+ * ops->xmit first although hw_active already reads false. 'S' is
+ * that wait: park, endpoint GC of the departed endpoint, which
+ * unparks, then the second failed commit parks again. Without the
+ * wait the log reads "GEG".
+ */
+ amt_test_ofl_join(amt, t[1], 0, false);
+ KUNIT_EXPECT_FALSE(test, og->hw_active);
+ amt_fake.commit_err = -EIO;
+ memset(amt_fake.log, 0, sizeof(amt_fake.log));
+ amt_fake.n = 0;
+ kunit_activate_static_stub(test, amt_ofl_sync_net,
+ amt_test_ofl_sync_net);
+ amt_test_ofl_drain(amt);
+ kunit_deactivate_static_stub(test, amt_ofl_sync_net);
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "SGESG");
+ KUNIT_EXPECT_NULL(test, og->hw);
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_PARKED);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 0);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* The publish gate: a member removed while the
+ * reconciler programs the group is still in what it commits, so that
+ * commit is not published and software serves the group; a member added
+ * meanwhile does not hold the publication back. A budget of one ends the
+ * tick right after the program the hook lands in.
+ */
+static struct {
+ struct amt_dev *amt;
+ struct amt_tunnel_list *tunnel;
+ int grp;
+ bool join;
+} amt_test_ofl_hook;
+
+static void amt_test_ofl_commit_hook(void)
+{
+ amt_test_ofl_join(amt_test_ofl_hook.amt, amt_test_ofl_hook.tunnel,
+ amt_test_ofl_hook.grp, amt_test_ofl_hook.join);
+}
+
+static void amt_offload_publish_gate_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_offload_endpoint_key key;
+ struct amt_tunnel_list *t[3];
+ struct amt_offload_state *st;
+ struct amt_ofl_grp *og;
+ struct sk_buff *skb;
+ struct amt_dev *amt;
+ u32 budget;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtop%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ st = amt->offload_st;
+ budget = st->budget;
+ for (i = 0; i < 3; i++)
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_hook.amt = amt;
+
+ /* A join inside the first commit: publish the committed subset. */
+ amt_test_ofl_join(amt, t[0], 0, true);
+ amt_test_ofl_hook.tunnel = t[1];
+ amt_test_ofl_hook.join = true;
+ amt_fake.commit_hook = amt_test_ofl_commit_hook;
+ st->budget = 1;
+ amt_ofl_reconcile(amt);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+ KUNIT_ASSERT_NOT_NULL(test, og->hw);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 1);
+ st->budget = budget;
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+
+ /* A leave inside a recommit: withhold it, software serves G. */
+ amt_test_ofl_join(amt, t[2], 0, true);
+ KUNIT_EXPECT_TRUE(test, og->hw_active); /* joins do not withdraw */
+ amt_test_ofl_hook.tunnel = t[0];
+ amt_test_ofl_hook.join = false;
+ amt_fake.commit_hook = amt_test_ofl_commit_hook;
+ st->budget = 1;
+ amt_ofl_reconcile(amt);
+ KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+ KUNIT_EXPECT_FALSE(test, og->hw_active);
+ /* The provider still replicates to t[0]: its endpoint stays. */
+ amt_ofl_ep_key(&key, amt, t[0]);
+ KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&key));
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_DIRTY);
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+ i = amt_fake.xmits;
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, i);
+ st->budget = budget;
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 2);
+ kfree_skb(skb);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Same-key rejoin: K empties and goes on the tick's gone list with its
+ * provider group still live; a join inside the next group's commit hashes
+ * a new K, which the same tick programs. That first use must wait for the
+ * old group's disable: the fake refuses a second live group per key, and
+ * without the flush the new K parks in software.
+ */
+static void amt_offload_rejoin_gone_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[2];
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtor%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++)
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[0], 0, true);
+ amt_test_ofl_drain(amt);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_ASSERT_TRUE(test, og->hw_active);
+
+ /* K empties first, then G dirties; G's commit rejoins K. */
+ amt_test_ofl_join(amt, t[0], 0, false);
+ amt_test_ofl_join(amt, t[1], 1, true);
+ amt_test_ofl_hook.amt = amt;
+ amt_test_ofl_hook.tunnel = t[0];
+ amt_test_ofl_hook.grp = 0;
+ amt_test_ofl_hook.join = true;
+ amt_fake.commit_hook = amt_test_ofl_commit_hook;
+ /* 'S' is the mid-pass flush's wait for ops->xmit on the old K before
+ * its disable; without it the log reads "G".
+ */
+ memset(amt_fake.log, 0, sizeof(amt_fake.log));
+ amt_fake.n = 0;
+ kunit_activate_static_stub(test, amt_ofl_sync_net,
+ amt_test_ofl_sync_net);
+ amt_ofl_reconcile(amt);
+ kunit_deactivate_static_stub(test, amt_ofl_sync_net);
+ KUNIT_EXPECT_STREQ(test, amt_fake.log, "SG");
+ KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+ KUNIT_EXPECT_EQ(test, amt_fake.dup_grps, 0);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, og->state, AMT_OFL_IDLE);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 2);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_SW_REMAINDER), 0);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* The mid-pass flush above is timed like any release op: a stalled
+ * group_disable of the old K resets the provider once, and the pass ends
+ * there rather than program the new K against a provider it just
+ * flushed. The next ticks replay it.
+ */
+static void amt_offload_rejoin_gone_stall_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[2];
+ struct amt_ofl_grp *og;
+ struct amt_dev *amt;
+ int i, updates;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtos%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++)
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[0], 0, true);
+ amt_test_ofl_drain(amt);
+
+ amt_test_ofl_join(amt, t[0], 0, false);
+ amt_test_ofl_join(amt, t[1], 1, true);
+ amt_test_ofl_hook.amt = amt;
+ amt_test_ofl_hook.tunnel = t[0];
+ amt_test_ofl_hook.grp = 0;
+ amt_test_ofl_hook.join = true;
+ amt_fake.commit_hook = amt_test_ofl_commit_hook;
+ amt_fake.release_stall = true;
+ updates = amt_fake.updates;
+ amt_ofl_reconcile(amt);
+ amt_fake.release_stall = false;
+ KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt, AMT_OFFLOAD_CNT_RESET),
+ 1);
+ /* G's update only: the new K was not programmed. */
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, updates + 1);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_EXPECT_NULL(test, og->hw);
+ KUNIT_EXPECT_FALSE(test, og->hw_active);
+
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, amt_fake.dup_grps, 0);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Exactly one actor per packet: CONSUMED gives the skb away, FALLBACK
+ * and BUSY return it untouched (BUSY counted); an uncommitted group and
+ * an unbound relay never reach the provider. A packet submitted after a
+ * commit carries the new generation.
+ */
+static void amt_offload_tx_ownership_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_tunnel_list *t[2];
+ struct amt_ofl_grp *og;
+ struct sk_buff *skb;
+ struct amt_dev *amt;
+ u64 gen;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt = amt_test_ofl_attach(test, "amtox%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ t[0] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ t[1] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+ amt_test_ofl_join(amt, t[0], 0, true);
+ amt_test_ofl_drain(amt);
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+
+ /* No provider: software, as with offload=software. */
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+
+ rtnl_lock();
+ KUNIT_EXPECT_EQ(test, amt_offload_register(lower, &amt_fake_ops), 0);
+ rtnl_unlock();
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, 0); /* not committed yet */
+ amt_test_ofl_drain(amt);
+ og = amt_test_ofl_grp(amt, 0);
+ KUNIT_ASSERT_NOT_NULL(test, og);
+ KUNIT_ASSERT_TRUE(test, og->hw_active);
+
+ amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ amt_fake.xmit_res = AMT_OFFLOAD_BUSY;
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_BUSY_FALLBACK), 1);
+ KUNIT_EXPECT_EQ(test, refcount_read(&skb->users), 1);
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 1)); /* no group */
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, 2);
+ KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, og->hw_gen);
+
+ /* Generation rule. */
+ gen = og->hw_gen;
+ amt_test_ofl_join(amt, t[1], 0, true);
+ amt_fake.xmit_res = AMT_OFFLOAD_FALLBACK;
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, gen);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_GT(test, og->hw_gen, gen);
+ amt_fake.xmit_res = AMT_OFFLOAD_CONSUMED;
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0)); /* freed */
+ KUNIT_EXPECT_EQ(test, amt_fake.xmit_gen, og->hw_gen);
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, 4);
+
+ /* Removing a member withdraws the group from hardware at once, not
+ * at the next commit: the committed set still names the departed
+ * endpoint.
+ */
+ amt_test_ofl_join(amt, t[1], 0, false);
+ KUNIT_EXPECT_FALSE(test, og->hw_active);
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 0));
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, 4);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_TRUE(test, og->hw_active);
+ KUNIT_EXPECT_EQ(test, ((struct amt_fake_obj *)og->hw)->nr, 1);
+ kfree_skb(skb);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Admission: a teardown waiting on st->mutex holds RTNL, so once admission
+ * stops a programming pass makes no further endpoint_add, group_update,
+ * group_commit or endpoint_del, and ends the tick. A pass releasing
+ * everything after desync does not check admission and finishes that
+ * release. The fake clears b->admitting from inside one op. The four
+ * blocks below fail with any one of these five admission checks reverted
+ * (before endpoint_add, group_update and group_commit, at the top of the
+ * pass loop, in the endpoint GC); the pass loop's epoch check is
+ * amt_offload_reset_ends_pass_test's.
+ */
+static void amt_offload_admit_guard_test(struct kunit *test)
+{
+ struct amt_tunnel_list *t[3];
+ struct net_device *lower;
+ struct amt_dev *amt;
+ int i;
+
+ /* A two-endpoint group, admission lost in the first endpoint_add:
+ * no second endpoint_add.
+ */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtga%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++)
+ amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000 + i)),
+ 0, true);
+ amt_fake.drop_admit = 'A';
+ amt_ofl_reconcile(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+ amt_test_ofl_done(amt, lower);
+
+ /* Three one-endpoint groups, lost in the first endpoint_add: no
+ * group_update, and the pass ends with the other two still queued.
+ */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtgb%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 3; i++)
+ amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000 + i)),
+ i, true);
+ amt_fake.drop_admit = 'A';
+ KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+ KUNIT_EXPECT_EQ(test, amt_fake.commits, 0);
+ for (i = 1; i < 3; i++) {
+ struct amt_ofl_grp *g = amt_test_ofl_grp(amt, i);
+
+ KUNIT_ASSERT_NOT_NULL(test, g);
+ KUNIT_EXPECT_EQ(test, g->state, AMT_OFL_DIRTY);
+ }
+ amt_test_ofl_done(amt, lower);
+
+ /* Lost in group_update: no group_commit. */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtgc%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000)), 0, true);
+ amt_fake.drop_admit = 'U';
+ amt_ofl_reconcile(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.commits, 0);
+ amt_test_ofl_done(amt, lower);
+
+ /* Two unused endpoints, lost in the first endpoint_del: the
+ * endpoint GC deletes no second one.
+ */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtgd%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 2; i++) {
+ t[i] = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000 + i));
+ amt_test_ofl_join(amt, t[i], i, true);
+ }
+ amt_test_ofl_drain(amt);
+ KUNIT_ASSERT_EQ(test, amt_fake_live(amt_fake.eps), 2);
+ for (i = 0; i < 2; i++)
+ amt_test_ofl_join(amt, t[i], i, false);
+ amt_fake.drop_admit = 'E';
+ amt_ofl_reconcile(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 1);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* A reset (an op overran op_timeout_ms) replays every group, so
+ * the pass ends at it instead of spending the rest of the budget against
+ * a provider that may be wedged.
+ */
+static void amt_offload_reset_ends_pass_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt;
+ int i;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.op_timeout_ms = 10;
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtor%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ for (i = 0; i < 3; i++)
+ amt_test_ofl_join(amt, amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000 + i)),
+ i, true);
+ amt_fake.add_stall = true;
+ KUNIT_EXPECT_TRUE(test, amt_ofl_reconcile(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.flushes, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.ep_adds, 1);
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+
+ /* The next tick replays all three. */
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.grps), 3);
+ KUNIT_EXPECT_EQ(test, amt_fake_live(amt_fake.eps), 3);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Hash seed: the offload tables are indexed with the device's hash_seed and
+ * sized from max_tunnels (x max_groups) at attach, so a gateway can
+ * neither precompute one chain nor fill a fixed-size table.
+ */
+static void amt_offload_hash_seed_test(struct kunit *test)
+{
+ struct amt_offload_endpoint_key ek;
+ struct amt_offload_group_key gk;
+ bool gmoved = false, emoved = false;
+ struct amt_offload_state *st;
+ struct hlist_head *g0, *e0;
+ struct net_device *lower;
+ struct amt_dev *amt;
+ u32 seed;
+
+ memset(&ek, 0, sizeof(ek));
+ ek.family = AF_INET;
+ ek.addr.ip4 = AMT_TEST_GW;
+ ek.port = htons(4000);
+ memset(&gk, 0, sizeof(gk));
+ gk.family = AF_INET;
+ gk.group.ip4 = AMT_TEST_GROUP;
+
+ /* Default caps: 128 tunnels x 32 groups. */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoh%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ st = amt->offload_st;
+ KUNIT_EXPECT_EQ(test, st->ep_bits, 7);
+ KUNIT_EXPECT_EQ(test, st->grp_bits, 12);
+ amt->hash_seed = 0;
+ g0 = amt_ofl_grp_bucket(st, &gk);
+ e0 = amt_ofl_ep_bucket(st, &ek);
+ for (seed = 1; seed < 64 && !(gmoved && emoved); seed++) {
+ amt->hash_seed = seed;
+ gmoved |= amt_ofl_grp_bucket(st, &gk) != g0;
+ emoved |= amt_ofl_ep_bucket(st, &ek) != e0;
+ }
+ KUNIT_EXPECT_TRUE_MSG(test, gmoved, "group bucket ignores hash_seed");
+ KUNIT_EXPECT_TRUE_MSG(test, emoved, "endpoint bucket ignores hash_seed");
+ amt_test_ofl_done(amt, lower);
+
+ /* max_tunnels set before attach sizes both tables. */
+ lower = amt_test_lower_alloc(test);
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_offload_relay(test, "amtoi%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ amt->max_tunnels = 10000;
+ rtnl_lock();
+ KUNIT_EXPECT_EQ(test, amt_offload_attach(amt, NULL), 0);
+ rtnl_unlock();
+ KUNIT_ASSERT_NOT_NULL(test, amt->offload_st);
+ disable_work_sync(&amt->offload_st->work);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->ep_bits, 14);
+ KUNIT_EXPECT_EQ(test, amt->offload_st->grp_bits, 16);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* The last member leaves while the reconciler allocates its arrays: the
+ * retry finds no member and must not free the arrays a second time. The
+ * allocation and free stubs keep a ledger of live arrays, so a repeated
+ * free fails the case in every configuration, not only under KASAN.
+ */
+static struct {
+ struct kunit *test;
+ struct amt_dev *amt;
+ struct amt_tunnel_list *tunnel;
+ int calls;
+ void *live[4];
+ int nr_live;
+} amt_test_ofl_race;
+
+static void *amt_test_ofl_alloc_leave(u32 n, size_t size)
+{
+ void *p;
+
+ if (!amt_test_ofl_race.calls++)
+ amt_test_ofl_join(amt_test_ofl_race.amt,
+ amt_test_ofl_race.tunnel, 0, false);
+ p = kvmalloc_array(n, size, GFP_KERNEL);
+ if (!p)
+ return NULL;
+ if (amt_test_ofl_race.nr_live == ARRAY_SIZE(amt_test_ofl_race.live)) {
+ /* A later free of p would read as a double free. */
+ KUNIT_FAIL(amt_test_ofl_race.test, "live-array ledger full");
+ return p;
+ }
+ amt_test_ofl_race.live[amt_test_ofl_race.nr_live++] = p;
+ return p;
+}
+
+static void amt_test_ofl_free_once(void *p)
+{
+ int i;
+
+ if (!p)
+ return;
+ for (i = 0; i < amt_test_ofl_race.nr_live; i++) {
+ if (amt_test_ofl_race.live[i] == p) {
+ amt_test_ofl_race.live[i] =
+ amt_test_ofl_race.live[--amt_test_ofl_race.nr_live];
+ kvfree(p);
+ return;
+ }
+ }
+ /* Not live: a second free. Leak it rather than corrupt the heap. */
+ KUNIT_FAIL(amt_test_ofl_race.test, "array %p freed twice", p);
+}
+
+static void amt_offload_alloc_race_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_dev *amt;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_test_ofl_register(test, lower);
+ amt = amt_test_ofl_attach(test, "amtoa%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ memset(&amt_test_ofl_race, 0, sizeof(amt_test_ofl_race));
+ amt_test_ofl_race.test = test;
+ amt_test_ofl_race.amt = amt;
+ amt_test_ofl_race.tunnel = amt_test_ofl_tunnel(test, AMT_TEST_GW,
+ htons(4000));
+ amt_test_ofl_join(amt, amt_test_ofl_race.tunnel, 0, true);
+
+ kunit_activate_static_stub(test, amt_ofl_alloc_array,
+ amt_test_ofl_alloc_leave);
+ kunit_activate_static_stub(test, amt_ofl_free_array,
+ amt_test_ofl_free_once);
+ amt_test_ofl_drain(amt);
+ kunit_deactivate_static_stub(test, amt_ofl_free_array);
+ kunit_deactivate_static_stub(test, amt_ofl_alloc_array);
+
+ /* One allocation round, then the retry read no member. */
+ KUNIT_EXPECT_EQ(test, amt_test_ofl_race.calls, 2);
+ KUNIT_EXPECT_EQ(test, amt_test_ofl_race.nr_live, 0);
+ KUNIT_EXPECT_NULL(test, amt_test_ofl_grp(amt, 0));
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_no_eps(amt));
+ KUNIT_EXPECT_EQ(test, amt_fake.updates, 0);
+ amt_test_ofl_done(amt, lower);
+}
+
+/* Ingress: the host drops a punted copy of a committed group and sends
+ * everything else to software. Only a required relay binds on ingress.
+ * A leave does not lift the drop: hardware keeps replicating to the
+ * committed set until the recommit, so software would send the remaining
+ * members every packet a second time.
+ */
+static void amt_offload_ingress_punt_test(struct kunit *test)
+{
+ struct net_device *lower = amt_test_lower_alloc(test);
+ struct amt_offload_endpoint_key key;
+ struct amt_tunnel_list *t, *t2;
+ struct amt_dev *amt, *aut;
+ struct sk_buff *skb;
+ u32 budget;
+
+ amt_fake_reset(AMT_OFFLOAD_FAMILY_V4);
+ amt_fake.paths = BIT(AMT_OFFLOAD_PATH_INGRESS);
+ amt_test_ofl_register(test, lower);
+ aut = amt_test_ofl_attach(test, "amtoa%d", lower,
+ AMT_OFFLOAD_MODE_AUTO);
+ KUNIT_EXPECT_NULL(test, rcu_access_pointer(aut->offload));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(aut,
+ AMT_OFFLOAD_CNT_PATH_UNSUPPORTED), 1);
+
+ amt = amt_test_ofl_attach(test, "amtog%d", lower,
+ AMT_OFFLOAD_MODE_REQUIRED);
+ KUNIT_ASSERT_NOT_NULL(test, rcu_access_pointer(amt->offload));
+ t = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4000));
+ t2 = amt_test_ofl_tunnel(test, AMT_TEST_GW, htons(4001));
+ amt_test_ofl_join(amt, t, 0, true);
+ amt_test_ofl_join(amt, t2, 0, true);
+ amt_test_ofl_join(amt, t, 2, true);
+ amt_test_ofl_drain(amt);
+ amt_test_ofl_join(amt, t, 3, true); /* not committed */
+
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 1));
+ KUNIT_EXPECT_FALSE(test, amt_test_ofl_tx(amt, skb, 3));
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_PUNT_DROPPED), 0);
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0)); /* dropped */
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_PUNT_DROPPED), 1);
+
+ amt_test_ofl_join(amt, t2, 0, false);
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 0)); /* dropped */
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_PUNT_DROPPED), 2);
+
+ /* A leave inside a group's first commit: hardware replicates to the
+ * committed set anyway, so the pass still publishes it and the
+ * punted copy is dropped. Withholding it would send the remaining
+ * member every packet twice until the next pass.
+ */
+ amt_test_ofl_drain(amt);
+ amt_test_ofl_join(amt, t, 4, true);
+ amt_test_ofl_join(amt, t2, 4, true);
+ amt_test_ofl_hook.amt = amt;
+ amt_test_ofl_hook.tunnel = t2;
+ amt_test_ofl_hook.grp = 4;
+ amt_test_ofl_hook.join = false;
+ amt_fake.commit_hook = amt_test_ofl_commit_hook;
+ budget = amt->offload_st->budget;
+ amt->offload_st->budget = 1;
+ amt_ofl_reconcile(amt);
+ amt->offload_st->budget = budget;
+ amt_test_ofl_hook.grp = 0;
+ KUNIT_EXPECT_NULL(test, amt_fake.commit_hook);
+ amt_ofl_ep_key(&key, amt, t2);
+ KUNIT_EXPECT_TRUE(test, amt_test_fake_ep_held(&key));
+ skb = alloc_skb(64, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, skb);
+ KUNIT_EXPECT_TRUE(test, amt_test_ofl_tx(amt, skb, 4)); /* dropped */
+ KUNIT_EXPECT_EQ(test, amt_test_offload_cnt(amt,
+ AMT_OFFLOAD_CNT_PUNT_DROPPED), 3);
+ amt_test_ofl_drain(amt);
+ KUNIT_EXPECT_EQ(test, amt_fake.xmits, 0);
+ amt_test_offload_free(aut);
+ amt_test_ofl_done(amt, lower);
+}
+
static struct kunit_case amt_test_cases[] = {
KUNIT_CASE(amt_lifecycle_test),
KUNIT_CASE(amt_upstream_refresh_idempotent_test),
KUNIT_CASE(amt_ex_transition_test),
+ KUNIT_CASE(amt_offload_register_when_bound_test),
+ KUNIT_CASE(amt_offload_family_unsupported_test),
+ KUNIT_CASE(amt_offload_unregister_mid_fanout_test),
+ KUNIT_CASE(amt_offload_provider_unregister_order_test),
+ KUNIT_CASE(amt_offload_mode_validate_test),
+ KUNIT_CASE(amt_offload_required_refused_test),
+ KUNIT_CASE(amt_offload_member_gnode_test),
+ KUNIT_CASE(amt_offload_budget_coalesce_test),
+ KUNIT_CASE(amt_offload_enospc_hysteresis_test),
+ KUNIT_CASE(amt_offload_timeout_reset_test),
+ KUNIT_CASE(amt_offload_reset_after_empty_test),
+ KUNIT_CASE(amt_offload_teardown_release_stall_test),
+ KUNIT_CASE(amt_offload_teardown_ep_stall_test),
+ KUNIT_CASE(amt_offload_reconcile_release_stall_test),
+ KUNIT_CASE(amt_offload_gc_churn_test),
+ KUNIT_CASE(amt_offload_endpoint_idempotent_test),
+ KUNIT_CASE(amt_offload_commit_fail_test),
+ KUNIT_CASE(amt_offload_publish_gate_test),
+ KUNIT_CASE(amt_offload_rejoin_gone_test),
+ KUNIT_CASE(amt_offload_rejoin_gone_stall_test),
+ KUNIT_CASE(amt_offload_tx_ownership_test),
+ KUNIT_CASE(amt_offload_alloc_race_test),
+ KUNIT_CASE(amt_offload_admit_guard_test),
+ KUNIT_CASE(amt_offload_reset_ends_pass_test),
+ KUNIT_CASE(amt_offload_hash_seed_test),
+ KUNIT_CASE(amt_offload_ingress_punt_test),
{}
};
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 7/9] netdevsim: add a TX reference provider for AMT relay offload
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (5 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 6/9] amt: KUnit cases for the offload contract Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 8/9] selftests: net: add amt_offload.sh " Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 9/9] MAINTAINERS: amt: cover the offload header and netdevsim provider Omar Ramadan
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
Each PF port offers itself as an AMT relay replication offload provider
(include/net/amt_offload.h) for relays whose stream device it is. It is
not a pass-through to amt's software fan-out: the port keeps its own
endpoint and group tables, with a staged and a committed member set per
group under the generation amt gave it, and decides from those which
endpoints get a copy of each packet amt hands to ->xmit. Each copy is
encapsulated by amt_offload_encap_xmit() and sent out of the port to its
peer, so any divergence between amt's desired state and what the
provider committed shows up as a wrong copy count at the peer. The set
committed before the last one is kept for packets already in flight,
and an endpoint stays allocated while a set names it.
debugfs (ports/<n>/amt/) shows the tables and counters and injects the
failures the contract has to survive: busy (xmit returns BUSY),
fail_commit (group_commit returns -ENOSPC), max_endpoints (endpoint_add
returns -ENOSPC), stall_ms (every programming op sleeps, so amt sees it
overrun op_timeout_ms and resets) and op_timeout_ms (advertised at
bind).
The provider owns its devlink objects, registered on netdevsim's own
devlink instance, not amt's: resources amt/endpoints, amt/groups,
amt/members and amt/cmdq, and an "amt" health reporter reported on
every flush (amt's reset). amt registers nothing on devlink.
Moving a netdev to another netns unregisters it there first, which
makes amt forget its provider; the port offers itself again once it is
registered in the new netns.
TX only: netdevsim has no ingress classifier to simulate the ingress
path with.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
drivers/net/Kconfig | 1 +
drivers/net/netdevsim/Makefile | 4 +
drivers/net/netdevsim/amt.c | 600 ++++++++++++++++++++++++++++++
drivers/net/netdevsim/dev.c | 68 ++++
drivers/net/netdevsim/health.c | 28 +-
drivers/net/netdevsim/netdev.c | 19 +-
drivers/net/netdevsim/netdevsim.h | 27 ++
7 files changed, 744 insertions(+), 3 deletions(-)
create mode 100644 drivers/net/netdevsim/amt.c
diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index fda19ae4b..6e45d2fa0 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -625,6 +625,7 @@ config NETDEVSIM
depends on INET
depends on PSAMPLE || PSAMPLE=n
depends on PTP_1588_CLOCK_MOCK || PTP_1588_CLOCK_MOCK=n
+ depends on AMT || AMT=n
select NET_DEVLINK
select PAGE_POOL
select NET_SHAPER
diff --git a/drivers/net/netdevsim/Makefile b/drivers/net/netdevsim/Makefile
index 87718204f..eec03812a 100644
--- a/drivers/net/netdevsim/Makefile
+++ b/drivers/net/netdevsim/Makefile
@@ -25,3 +25,7 @@ endif
ifneq ($(CONFIG_MACSEC),)
netdevsim-objs += macsec.o
endif
+
+ifneq ($(CONFIG_AMT),)
+netdevsim-objs += amt.o
+endif
diff --git a/drivers/net/netdevsim/amt.c b/drivers/net/netdevsim/amt.c
new file mode 100644
index 000000000..ffb9aa4e7
--- /dev/null
+++ b/drivers/net/netdevsim/amt.c
@@ -0,0 +1,600 @@
+// SPDX-License-Identifier: GPL-2.0
+/* netdevsim: TX reference provider for AMT relay replication offload.
+ *
+ * Each PF port registers itself as a provider for relays that transmit
+ * on it (include/net/amt_offload.h). The port keeps its OWN endpoint and
+ * group tables, with a staged and a committed member set per group under
+ * the generation amt gave it, and decides replication from them: a packet
+ * amt hands to ->xmit is copied to the endpoints of the committed set of
+ * that generation, each copy encapsulated by amt_offload_encap_xmit() and
+ * sent out of this port to its peer. Any divergence between what amt
+ * wants and what the provider committed shows up at the peer as a wrong
+ * copy count. TX only: netdevsim has no ingress classifier to simulate.
+ *
+ * debugfs, per port, under ports/<n>/amt/:
+ * tables the provider's endpoints and groups, read-only
+ * tx_copies, tx_busy, tx_stale_gen, tx_dropped, commits, flushes
+ * counters, read-only
+ * busy xmit returns BUSY
+ * fail_commit group_commit fails with -ENOSPC
+ * max_endpoints endpoint_add fails with -ENOSPC beyond this (0: none)
+ * stall_ms every programming op sleeps this long first
+ * op_timeout_ms advertised to amt at bind (0: no limit)
+ *
+ * devlink, per device: resources amt/{endpoints,groups,members,cmdq} and
+ * the "amt" health reporter, reported on every flush (amt's reset).
+ */
+
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/netdevice.h>
+#include <linux/rculist.h>
+#include <linux/refcount.h>
+#include <linux/seq_file.h>
+#include <linux/skbuff.h>
+#include <net/amt_offload.h>
+#include <net/devlink.h>
+
+#include "netdevsim.h"
+
+struct amt_offload_endpoint {
+ struct list_head node; /* nsim_amt_bind.eps */
+ struct amt_offload_endpoint_key key;
+ /* One for amt's handle (all endpoint_add calls for the key), one per
+ * member set naming it: an in-flight packet may still be copied to
+ * an endpoint amt already deleted.
+ */
+ refcount_t refs;
+ u32 adds;
+ atomic64_t copies;
+ struct rcu_head rcu;
+};
+
+struct nsim_amt_set {
+ u64 gen;
+ u32 n;
+ struct rcu_head rcu;
+ struct amt_offload_endpoint *eps[] __counted_by(n);
+};
+
+struct amt_offload_group {
+ struct list_head node; /* nsim_amt_bind.grps */
+ struct amt_offload_group_key key;
+ struct nsim_amt_set *staged;
+ /* Committed last, and the one before it for packets in flight. */
+ struct nsim_amt_set __rcu *cur;
+ struct nsim_amt_set __rcu *prev;
+ struct rcu_head rcu;
+};
+
+struct nsim_amt_bind {
+ struct list_head node; /* nsim_amt.binds */
+ struct amt_offload_binding *b;
+ struct list_head eps;
+ struct list_head grps;
+ u32 nr_eps;
+};
+
+struct nsim_amt {
+ struct netdevsim *ns;
+ struct mutex lock; /* binds and their tables */
+ struct list_head binds;
+ struct dentry *ddir;
+ bool busy;
+ bool fail_commit;
+ u32 max_endpoints;
+ u32 stall_ms;
+ u32 op_timeout_ms;
+ u64 commits;
+ u64 flushes;
+ atomic64_t tx_copies;
+ atomic64_t tx_busy;
+ atomic64_t tx_stale_gen;
+ atomic64_t tx_dropped;
+};
+
+#define nsim_amt_deref(nam, p) \
+ rcu_dereference_protected(p, lockdep_is_held(&(nam)->lock))
+
+static struct nsim_amt *nsim_amt_of(struct amt_offload_binding *b)
+{
+ struct netdevsim *ns = netdev_priv(b->lower);
+
+ return ns->amt;
+}
+
+/* Every programming op runs inside one of these: cmdq occupancy, and the
+ * injected stall that makes amt see an op overrun op_timeout_ms.
+ */
+static struct nsim_amt *nsim_amt_op_begin(struct amt_offload_binding *b)
+{
+ struct nsim_amt *nam = nsim_amt_of(b);
+ u32 stall = READ_ONCE(nam->stall_ms);
+
+ atomic_inc(&nam->ns->nsim_dev->amt.cmdq);
+ if (stall)
+ msleep(stall);
+ mutex_lock(&nam->lock);
+ return nam;
+}
+
+static void nsim_amt_op_end(struct nsim_amt *nam)
+{
+ mutex_unlock(&nam->lock);
+ atomic_dec(&nam->ns->nsim_dev->amt.cmdq);
+}
+
+static void nsim_amt_ep_put(struct amt_offload_endpoint *ep)
+{
+ if (refcount_dec_and_test(&ep->refs))
+ kfree_rcu(ep, rcu);
+}
+
+static void nsim_amt_set_free_rcu(struct rcu_head *head)
+{
+ struct nsim_amt_set *set = container_of(head, struct nsim_amt_set,
+ rcu);
+ u32 i;
+
+ for (i = 0; i < set->n; i++)
+ nsim_amt_ep_put(set->eps[i]);
+ kfree(set);
+}
+
+static void nsim_amt_set_free(struct nsim_amt_set *set)
+{
+ if (set)
+ call_rcu(&set->rcu, nsim_amt_set_free_rcu);
+}
+
+static int nsim_amt_bind(struct amt_offload_binding *b,
+ struct amt_offload_caps *caps)
+{
+ struct nsim_amt *nam = nsim_amt_of(b);
+ struct nsim_amt_bind *nb;
+
+ nb = kzalloc_obj(*nb, GFP_KERNEL);
+ if (!nb)
+ return -ENOMEM;
+ nb->b = b;
+ INIT_LIST_HEAD(&nb->eps);
+ INIT_LIST_HEAD(&nb->grps);
+ b->priv = nb;
+
+ caps->families = AMT_OFFLOAD_FAMILY_V4 | AMT_OFFLOAD_FAMILY_V6;
+ caps->paths = BIT(AMT_OFFLOAD_PATH_TX);
+ caps->op_timeout_ms = READ_ONCE(nam->op_timeout_ms);
+ caps->max_endpoints = READ_ONCE(nam->max_endpoints);
+
+ mutex_lock(&nam->lock);
+ list_add_tail(&nb->node, &nam->binds);
+ mutex_unlock(&nam->lock);
+ return 0;
+}
+
+/* Drop every group and endpoint of @nb. Caller holds nam->lock; amt has
+ * stopped submitting packets for this binding.
+ */
+static void nsim_amt_bind_clear(struct nsim_amt *nam, struct nsim_amt_bind *nb)
+{
+ struct nsim_dev *nsim_dev = nam->ns->nsim_dev;
+ struct amt_offload_endpoint *ep, *et;
+ struct amt_offload_group *g, *gt;
+ struct nsim_amt_set *set;
+
+ list_for_each_entry_safe(g, gt, &nb->grps, node) {
+ set = nsim_amt_deref(nam, g->cur);
+ if (set)
+ atomic_sub(set->n, &nsim_dev->amt.members);
+ nsim_amt_set_free(set);
+ nsim_amt_set_free(nsim_amt_deref(nam, g->prev));
+ nsim_amt_set_free(g->staged);
+ list_del(&g->node);
+ atomic_dec(&nsim_dev->amt.groups);
+ kfree_rcu(g, rcu);
+ }
+ list_for_each_entry_safe(ep, et, &nb->eps, node) {
+ list_del(&ep->node);
+ nb->nr_eps--;
+ atomic_dec(&nsim_dev->amt.endpoints);
+ nsim_amt_ep_put(ep);
+ }
+}
+
+static void nsim_amt_unbind(struct amt_offload_binding *b)
+{
+ struct nsim_amt *nam = nsim_amt_of(b);
+ struct nsim_amt_bind *nb = b->priv;
+
+ mutex_lock(&nam->lock);
+ /* Empty unless amt broke the teardown order; then free, not leak. */
+ WARN_ON_ONCE(!list_empty(&nb->grps) || !list_empty(&nb->eps));
+ nsim_amt_bind_clear(nam, nb);
+ list_del(&nb->node);
+ mutex_unlock(&nam->lock);
+ kfree(nb);
+}
+
+static bool nsim_amt_ep_key_eq(const struct amt_offload_endpoint_key *a,
+ const struct amt_offload_endpoint_key *b)
+{
+ if (a->family != b->family || a->port != b->port)
+ return false;
+ if (a->family == AF_INET6)
+ return ipv6_addr_equal(&a->addr.ip6, &b->addr.ip6);
+ return a->addr.ip4 == b->addr.ip4;
+}
+
+static int nsim_amt_endpoint_add(struct amt_offload_binding *b,
+ const struct amt_offload_endpoint_key *key,
+ struct amt_offload_endpoint **epp)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+ struct nsim_amt_bind *nb = b->priv;
+ struct amt_offload_endpoint *ep;
+ int err = 0;
+
+ list_for_each_entry(ep, &nb->eps, node) {
+ if (nsim_amt_ep_key_eq(&ep->key, key)) {
+ ep->adds++;
+ goto out;
+ }
+ }
+ if (nam->max_endpoints && nb->nr_eps >= nam->max_endpoints) {
+ err = -ENOSPC;
+ goto unlock;
+ }
+ ep = kzalloc_obj(*ep, GFP_KERNEL);
+ if (!ep) {
+ err = -ENOMEM;
+ goto unlock;
+ }
+ ep->key = *key;
+ refcount_set(&ep->refs, 1);
+ ep->adds = 1;
+ list_add_tail(&ep->node, &nb->eps);
+ nb->nr_eps++;
+ atomic_inc(&nam->ns->nsim_dev->amt.endpoints);
+out:
+ *epp = ep;
+unlock:
+ nsim_amt_op_end(nam);
+ return err;
+}
+
+static void nsim_amt_endpoint_del(struct amt_offload_binding *b,
+ struct amt_offload_endpoint *ep)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+ struct nsim_amt_bind *nb = b->priv;
+
+ if (!--ep->adds) {
+ list_del(&ep->node);
+ nb->nr_eps--;
+ atomic_dec(&nam->ns->nsim_dev->amt.endpoints);
+ nsim_amt_ep_put(ep);
+ }
+ nsim_amt_op_end(nam);
+}
+
+static int nsim_amt_group_update(struct amt_offload_binding *b,
+ const struct amt_offload_group_key *key,
+ struct amt_offload_endpoint *const *members,
+ unsigned int nr_members, u64 gen,
+ struct amt_offload_group **grpp)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+ struct amt_offload_group *g = *grpp;
+ struct nsim_amt_bind *nb = b->priv;
+ struct nsim_amt_set *set;
+ int err = 0;
+ u32 i;
+
+ if (!g) {
+ g = kzalloc_obj(*g, GFP_KERNEL);
+ if (!g) {
+ err = -ENOMEM;
+ goto out;
+ }
+ g->key = *key;
+ list_add_tail(&g->node, &nb->grps);
+ atomic_inc(&nam->ns->nsim_dev->amt.groups);
+ *grpp = g;
+ }
+ set = kzalloc(struct_size(set, eps, nr_members), GFP_KERNEL);
+ if (!set) {
+ err = -ENOMEM;
+ goto out;
+ }
+ set->n = nr_members;
+ set->gen = gen;
+ for (i = 0; i < nr_members; i++) {
+ refcount_inc(&members[i]->refs);
+ set->eps[i] = members[i];
+ }
+ nsim_amt_set_free(g->staged);
+ g->staged = set;
+out:
+ nsim_amt_op_end(nam);
+ return err;
+}
+
+static int nsim_amt_group_commit(struct amt_offload_binding *b,
+ struct amt_offload_group *g, u64 gen)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+ struct nsim_dev *nsim_dev = nam->ns->nsim_dev;
+ struct nsim_amt_set *cur, *prev;
+ int err = 0;
+
+ if (nam->fail_commit) {
+ err = -ENOSPC;
+ goto out;
+ }
+ if (!g->staged || g->staged->gen != gen) {
+ err = -EINVAL;
+ goto out;
+ }
+ cur = nsim_amt_deref(nam, g->cur);
+ prev = nsim_amt_deref(nam, g->prev);
+ atomic_add(g->staged->n, &nsim_dev->amt.members);
+ if (cur)
+ atomic_sub(cur->n, &nsim_dev->amt.members);
+ rcu_assign_pointer(g->prev, cur);
+ rcu_assign_pointer(g->cur, g->staged);
+ g->staged = NULL;
+ nsim_amt_set_free(prev);
+ nam->commits++;
+out:
+ nsim_amt_op_end(nam);
+ return err;
+}
+
+static void nsim_amt_group_disable(struct amt_offload_binding *b,
+ struct amt_offload_group *g)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+ struct nsim_amt_set *set;
+
+ set = nsim_amt_deref(nam, g->cur);
+ if (set)
+ atomic_sub(set->n, &nam->ns->nsim_dev->amt.members);
+ nsim_amt_set_free(set);
+ nsim_amt_set_free(nsim_amt_deref(nam, g->prev));
+ nsim_amt_set_free(g->staged);
+ list_del(&g->node);
+ atomic_dec(&nam->ns->nsim_dev->amt.groups);
+ kfree_rcu(g, rcu);
+ nsim_amt_op_end(nam);
+}
+
+/* Softirq, under rcu_read_lock(). Copies go to the committed set of @gen
+ * only; a generation this provider never committed (or has already
+ * retired) is declined, never guessed at.
+ */
+static enum amt_offload_result
+nsim_amt_xmit(struct amt_offload_binding *b, struct amt_offload_group *g,
+ struct sk_buff *skb, u64 gen)
+{
+ struct nsim_amt *nam = nsim_amt_of(b);
+ struct nsim_amt_set *set;
+ u32 i;
+
+ if (READ_ONCE(nam->busy)) {
+ atomic64_inc(&nam->tx_busy);
+ return AMT_OFFLOAD_BUSY;
+ }
+ set = rcu_dereference(g->cur);
+ if (!set || set->gen != gen) {
+ set = rcu_dereference(g->prev);
+ if (!set || set->gen != gen) {
+ atomic64_inc(&nam->tx_stale_gen);
+ return AMT_OFFLOAD_FALLBACK;
+ }
+ }
+ for (i = 0; i < set->n; i++) {
+ if (amt_offload_encap_xmit(b, &set->eps[i]->key, skb)) {
+ atomic64_inc(&nam->tx_dropped);
+ continue;
+ }
+ atomic64_inc(&set->eps[i]->copies);
+ atomic64_inc(&nam->tx_copies);
+ }
+ consume_skb(skb);
+ return AMT_OFFLOAD_CONSUMED;
+}
+
+/* Called under amt's offload mutex, like every programming op: from
+ * the reconciler with no other lock, or at teardown with RTNL held and,
+ * when the port is being deleted, the devlink instance lock too. The
+ * "amt" reporter has no .recover and no .dump (health.c), so reporting
+ * it takes no devlink lock in either case.
+ */
+static void nsim_amt_flush(struct amt_offload_binding *b)
+{
+ struct nsim_amt *nam = nsim_amt_op_begin(b);
+
+ nsim_amt_bind_clear(nam, b->priv);
+ nam->flushes++;
+ nsim_amt_op_end(nam);
+ devlink_health_report(nam->ns->nsim_dev->health.amt_reporter,
+ "amt reset: provider tables flushed", NULL);
+}
+
+static const struct amt_offload_ops nsim_amt_ops = {
+ .bind = nsim_amt_bind,
+ .unbind = nsim_amt_unbind,
+ .endpoint_add = nsim_amt_endpoint_add,
+ .endpoint_del = nsim_amt_endpoint_del,
+ .group_update = nsim_amt_group_update,
+ .group_commit = nsim_amt_group_commit,
+ .group_disable = nsim_amt_group_disable,
+ .xmit = nsim_amt_xmit,
+ .flush = nsim_amt_flush,
+};
+
+static void nsim_amt_show_ep(struct seq_file *m,
+ const struct amt_offload_endpoint_key *key)
+{
+ if (key->family == AF_INET6)
+ seq_printf(m, "[%pI6c]:%u", &key->addr.ip6, ntohs(key->port));
+ else
+ seq_printf(m, "%pI4:%u", &key->addr.ip4, ntohs(key->port));
+}
+
+static void nsim_amt_show_set(struct seq_file *m, const char *name,
+ const struct nsim_amt_set *set)
+{
+ u32 i;
+
+ if (!set)
+ return;
+ seq_printf(m, " %s gen %llu members %u:", name, set->gen, set->n);
+ for (i = 0; i < set->n; i++) {
+ seq_putc(m, ' ');
+ nsim_amt_show_ep(m, &set->eps[i]->key);
+ }
+ seq_putc(m, '\n');
+}
+
+static int nsim_amt_tables_show(struct seq_file *m, void *unused)
+{
+ struct nsim_amt *nam = m->private;
+ struct amt_offload_endpoint *ep;
+ struct amt_offload_group *g;
+ struct nsim_amt_bind *nb;
+
+ mutex_lock(&nam->lock);
+ list_for_each_entry(nb, &nam->binds, node) {
+ seq_printf(m, "binding %s epoch %llu\n", netdev_name(nb->b->dev),
+ nb->b->epoch);
+ list_for_each_entry(ep, &nb->eps, node) {
+ seq_puts(m, "endpoint ");
+ nsim_amt_show_ep(m, &ep->key);
+ seq_printf(m, " copies %llu\n",
+ atomic64_read(&ep->copies));
+ }
+ list_for_each_entry(g, &nb->grps, node) {
+ if (g->key.family == AF_INET6)
+ seq_printf(m, "group %pI6c\n", &g->key.group.ip6);
+ else
+ seq_printf(m, "group %pI4\n", &g->key.group.ip4);
+ nsim_amt_show_set(m, "committed",
+ nsim_amt_deref(nam, g->cur));
+ nsim_amt_show_set(m, "previous",
+ nsim_amt_deref(nam, g->prev));
+ nsim_amt_show_set(m, "staged", g->staged);
+ }
+ }
+ mutex_unlock(&nam->lock);
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(nsim_amt_tables);
+
+static int nsim_amt_atomic64_get(void *data, u64 *val)
+{
+ *val = atomic64_read(data);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(nsim_amt_atomic64_fops, nsim_amt_atomic64_get,
+ NULL, "%llu\n");
+
+static void nsim_amt_debugfs_init(struct nsim_amt *nam)
+{
+ struct dentry *d;
+
+ d = debugfs_create_dir("amt", nam->ns->nsim_dev_port->ddir);
+ nam->ddir = d;
+ debugfs_create_file("tables", 0400, d, nam, &nsim_amt_tables_fops);
+ debugfs_create_file("tx_copies", 0400, d, &nam->tx_copies,
+ &nsim_amt_atomic64_fops);
+ debugfs_create_file("tx_busy", 0400, d, &nam->tx_busy,
+ &nsim_amt_atomic64_fops);
+ debugfs_create_file("tx_stale_gen", 0400, d, &nam->tx_stale_gen,
+ &nsim_amt_atomic64_fops);
+ debugfs_create_file("tx_dropped", 0400, d, &nam->tx_dropped,
+ &nsim_amt_atomic64_fops);
+ debugfs_create_u64("commits", 0400, d, &nam->commits);
+ debugfs_create_u64("flushes", 0400, d, &nam->flushes);
+ debugfs_create_bool("busy", 0600, d, &nam->busy);
+ debugfs_create_bool("fail_commit", 0600, d, &nam->fail_commit);
+ debugfs_create_u32("max_endpoints", 0600, d, &nam->max_endpoints);
+ debugfs_create_u32("stall_ms", 0600, d, &nam->stall_ms);
+ debugfs_create_u32("op_timeout_ms", 0600, d, &nam->op_timeout_ms);
+}
+
+int nsim_amt_init(struct netdevsim *ns)
+{
+ struct nsim_amt *nam;
+ int err;
+
+ nam = kzalloc_obj(*nam, GFP_KERNEL);
+ if (!nam)
+ return -ENOMEM;
+ nam->ns = ns;
+ mutex_init(&nam->lock);
+ INIT_LIST_HEAD(&nam->binds);
+ nam->op_timeout_ms = 1000;
+ nsim_amt_debugfs_init(nam);
+
+ rtnl_lock();
+ ns->amt = nam;
+ err = amt_offload_register(ns->netdev, &nsim_amt_ops);
+ if (err)
+ ns->amt = NULL;
+ rtnl_unlock();
+ if (err) {
+ debugfs_remove_recursive(nam->ddir);
+ mutex_destroy(&nam->lock);
+ kfree(nam);
+ }
+ return err;
+}
+
+void nsim_amt_uninit(struct netdevsim *ns)
+{
+ struct nsim_amt *nam = ns->amt;
+
+ if (!nam)
+ return;
+ rtnl_lock();
+ amt_offload_unregister(ns->netdev);
+ ns->amt = NULL;
+ rtnl_unlock();
+ debugfs_remove_recursive(nam->ddir);
+ mutex_destroy(&nam->lock);
+ kfree(nam);
+}
+
+/* amt forgets a provider when its lower device is unregistered, and
+ * moving a netdev to another netns unregisters it there first. Offer
+ * the port again once it is registered in its new netns.
+ */
+static int nsim_amt_netdev_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct netdevsim *ns;
+
+ if (event != NETDEV_REGISTER || !netdev_is_nsim(dev))
+ return NOTIFY_DONE;
+ ns = netdev_priv(dev);
+ if (ns->amt)
+ amt_offload_register(dev, &nsim_amt_ops);
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block nsim_amt_notifier = {
+ .notifier_call = nsim_amt_netdev_event,
+};
+
+int nsim_amt_module_init(void)
+{
+ return register_netdevice_notifier(&nsim_amt_notifier);
+}
+
+void nsim_amt_module_exit(void)
+{
+ unregister_netdevice_notifier(&nsim_amt_notifier);
+ rcu_barrier(); /* nsim_amt_set_free_rcu() is module text */
+}
diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
index f00fc2f9e..7fcdb0d67 100644
--- a/drivers/net/netdevsim/dev.c
+++ b/drivers/net/netdevsim/dev.c
@@ -444,6 +444,69 @@ static void nsim_dev_port_debugfs_exit(struct nsim_dev_port *nsim_dev_port)
debugfs_remove_recursive(nsim_dev_port->ddir);
}
+static u64 nsim_dev_amt_endpoints_occ_get(void *priv)
+{
+ return atomic_read(&((struct nsim_dev *)priv)->amt.endpoints);
+}
+
+static u64 nsim_dev_amt_groups_occ_get(void *priv)
+{
+ return atomic_read(&((struct nsim_dev *)priv)->amt.groups);
+}
+
+static u64 nsim_dev_amt_members_occ_get(void *priv)
+{
+ return atomic_read(&((struct nsim_dev *)priv)->amt.members);
+}
+
+static u64 nsim_dev_amt_cmdq_occ_get(void *priv)
+{
+ return atomic_read(&((struct nsim_dev *)priv)->amt.cmdq);
+}
+
+static int
+nsim_dev_amt_resources_register(struct devlink *devlink,
+ const struct devlink_resource_size_params *p)
+{
+ static const struct {
+ const char *name;
+ u64 id;
+ devlink_resource_occ_get_t *occ_get;
+ } res[] = {
+ { "endpoints", NSIM_RESOURCE_AMT_ENDPOINTS,
+ nsim_dev_amt_endpoints_occ_get },
+ { "groups", NSIM_RESOURCE_AMT_GROUPS,
+ nsim_dev_amt_groups_occ_get },
+ { "members", NSIM_RESOURCE_AMT_MEMBERS,
+ nsim_dev_amt_members_occ_get },
+ { "cmdq", NSIM_RESOURCE_AMT_CMDQ, nsim_dev_amt_cmdq_occ_get },
+ };
+ int err, i;
+
+ if (!IS_ENABLED(CONFIG_AMT))
+ return 0;
+ err = devl_resource_register(devlink, "amt", (u64)-1,
+ NSIM_RESOURCE_AMT,
+ DEVLINK_RESOURCE_ID_PARENT_TOP, p);
+ if (err) {
+ pr_err("Failed to register AMT top resource\n");
+ return err;
+ }
+ for (i = 0; i < ARRAY_SIZE(res); i++) {
+ err = devl_resource_register(devlink, res[i].name, (u64)-1,
+ res[i].id, NSIM_RESOURCE_AMT, p);
+ if (err) {
+ pr_err("Failed to register AMT %s resource\n",
+ res[i].name);
+ return err;
+ }
+ devl_resource_occ_get_register(devlink, res[i].id,
+ res[i].occ_get,
+ devlink_priv(devlink));
+ }
+ return 0;
+}
+
static int nsim_dev_resources_register(struct devlink *devlink)
{
struct devlink_resource_size_params params = {
@@ -514,6 +577,11 @@ static int nsim_dev_resources_register(struct devlink *devlink)
pr_err("Failed to register NEXTHOPS resource\n");
goto err_out;
}
+
+ /* AMT relay offload tables, owned by the ports' provider (amt.c) */
+ err = nsim_dev_amt_resources_register(devlink, ¶ms);
+ if (err)
+ goto err_out;
return 0;
err_out:
diff --git a/drivers/net/netdevsim/health.c b/drivers/net/netdevsim/health.c
index 3bd0e7a48..f6bc6cb51 100644
--- a/drivers/net/netdevsim/health.c
+++ b/drivers/net/netdevsim/health.c
@@ -175,6 +175,16 @@ static const struct file_operations nsim_dev_health_break_fops = {
.owner = THIS_MODULE,
};
+/* Reported by the AMT relay offload provider (amt.c) each time amt resets
+ * it. Recovery is amt's own replay, so there is no .recover, and no .dump:
+ * with neither, devlink_health_report() never takes the devlink instance
+ * lock, which amt's ->flush() may not take (amt holds its offload mutex,
+ * below rtnl, and the instance lock sits above rtnl).
+ */
+static const struct devlink_health_reporter_ops nsim_dev_amt_reporter_ops = {
+ .name = "amt",
+};
+
int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
{
struct nsim_dev_health *health = &nsim_dev->health;
@@ -196,10 +206,21 @@ int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
goto err_empty_reporter_destroy;
}
+ if (IS_ENABLED(CONFIG_AMT)) {
+ health->amt_reporter =
+ devl_health_reporter_create(devlink,
+ &nsim_dev_amt_reporter_ops,
+ health);
+ if (IS_ERR(health->amt_reporter)) {
+ err = PTR_ERR(health->amt_reporter);
+ goto err_dummy_reporter_destroy;
+ }
+ }
+
health->ddir = debugfs_create_dir("health", nsim_dev->ddir);
if (IS_ERR(health->ddir)) {
err = PTR_ERR(health->ddir);
- goto err_dummy_reporter_destroy;
+ goto err_amt_reporter_destroy;
}
health->recovered_break_msg = NULL;
@@ -213,6 +234,9 @@ int nsim_dev_health_init(struct nsim_dev *nsim_dev, struct devlink *devlink)
&health->fail_recover);
return 0;
+err_amt_reporter_destroy:
+ if (health->amt_reporter)
+ devl_health_reporter_destroy(health->amt_reporter);
err_dummy_reporter_destroy:
devl_health_reporter_destroy(health->dummy_reporter);
err_empty_reporter_destroy:
@@ -226,6 +250,8 @@ void nsim_dev_health_exit(struct nsim_dev *nsim_dev)
debugfs_remove_recursive(health->ddir);
kfree(health->recovered_break_msg);
+ if (health->amt_reporter)
+ devl_health_reporter_destroy(health->amt_reporter);
devl_health_reporter_destroy(health->dummy_reporter);
devl_health_reporter_destroy(health->empty_reporter);
}
diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c
index a75076891..4294c026d 100644
--- a/drivers/net/netdevsim/netdev.c
+++ b/drivers/net/netdevsim/netdev.c
@@ -1075,6 +1075,10 @@ static int nsim_init_netdevsim(struct netdevsim *ns)
if (err)
goto err_unregister_netdev;
+ err = nsim_amt_init(ns);
+ if (err)
+ goto err_psp_uninit;
+
if (IS_ENABLED(CONFIG_DEBUG_NET)) {
ns->nb.notifier_call = netdev_debug_event;
if (register_netdevice_notifier_dev_net(ns->netdev, &ns->nb,
@@ -1084,6 +1088,8 @@ static int nsim_init_netdevsim(struct netdevsim *ns)
return 0;
+err_psp_uninit:
+ nsim_psp_uninit(ns);
err_unregister_netdev:
rtnl_lock();
peer = rtnl_dereference(ns->peer);
@@ -1182,8 +1188,10 @@ void nsim_destroy(struct netdevsim *ns)
unregister_netdevice_notifier_dev_net(ns->netdev, &ns->nb,
&ns->nn);
- if (nsim_dev_port_is_pf(ns->nsim_dev_port))
+ if (nsim_dev_port_is_pf(ns->nsim_dev_port)) {
+ nsim_amt_uninit(ns);
nsim_psp_uninit(ns);
+ }
rtnl_lock();
peer = rtnl_dereference(ns->peer);
@@ -1238,10 +1246,14 @@ static int __init nsim_module_init(void)
{
int err;
- err = nsim_dev_init();
+ err = nsim_amt_module_init();
if (err)
return err;
+ err = nsim_dev_init();
+ if (err)
+ goto err_amt_exit;
+
err = nsim_bus_init();
if (err)
goto err_dev_exit;
@@ -1256,6 +1268,8 @@ static int __init nsim_module_init(void)
nsim_bus_exit();
err_dev_exit:
nsim_dev_exit();
+err_amt_exit:
+ nsim_amt_module_exit();
return err;
}
@@ -1264,6 +1278,7 @@ static void __exit nsim_module_exit(void)
rtnl_link_unregister(&nsim_link_ops);
nsim_bus_exit();
nsim_dev_exit();
+ nsim_amt_module_exit();
}
module_init(nsim_module_init);
diff --git a/drivers/net/netdevsim/netdevsim.h b/drivers/net/netdevsim/netdevsim.h
index d909c4160..549afee0d 100644
--- a/drivers/net/netdevsim/netdevsim.h
+++ b/drivers/net/netdevsim/netdevsim.h
@@ -127,6 +127,8 @@ struct netdevsim {
u32 assoc_cnt;
} psp;
+ struct nsim_amt *amt;
+
struct nsim_bus_dev *nsim_bus_dev;
struct bpf_prog *bpf_offloaded;
@@ -232,6 +234,11 @@ enum nsim_resource_id {
NSIM_RESOURCE_IPV6_FIB,
NSIM_RESOURCE_IPV6_FIB_RULES,
NSIM_RESOURCE_NEXTHOPS,
+ NSIM_RESOURCE_AMT,
+ NSIM_RESOURCE_AMT_ENDPOINTS,
+ NSIM_RESOURCE_AMT_GROUPS,
+ NSIM_RESOURCE_AMT_MEMBERS,
+ NSIM_RESOURCE_AMT_CMDQ,
};
enum nsim_port_resource_id {
@@ -241,6 +248,7 @@ enum nsim_port_resource_id {
struct nsim_dev_health {
struct devlink_health_reporter *empty_reporter;
struct devlink_health_reporter *dummy_reporter;
+ struct devlink_health_reporter *amt_reporter;
struct dentry *ddir;
char *recovered_break_msg;
u32 binary_len;
@@ -351,6 +359,13 @@ struct nsim_dev {
bool fail_reload;
struct devlink_region *dummy_region;
struct nsim_dev_health health;
+ /* AMT relay offload tables of every port, for devlink resources */
+ struct {
+ atomic_t endpoints;
+ atomic_t groups;
+ atomic_t members;
+ atomic_t cmdq;
+ } amt;
struct nsim_dev_hwstats hwstats;
struct flow_action_cookie *fa_cookie;
spinlock_t fa_cookie_lock; /* protects fa_cookie */
@@ -472,6 +487,18 @@ nsim_psp_handle_ext(struct sk_buff *skb, struct skb_ext *psp_ext) {}
int nsim_setup_tc(struct net_device *dev, enum tc_setup_type type,
void *type_data);
+#if IS_ENABLED(CONFIG_AMT)
+int nsim_amt_init(struct netdevsim *ns);
+void nsim_amt_uninit(struct netdevsim *ns);
+int nsim_amt_module_init(void);
+void nsim_amt_module_exit(void);
+#else
+static inline int nsim_amt_init(struct netdevsim *ns) { return 0; }
+static inline void nsim_amt_uninit(struct netdevsim *ns) {}
+static inline int nsim_amt_module_init(void) { return 0; }
+static inline void nsim_amt_module_exit(void) {}
+#endif
+
struct nsim_bus_dev {
struct device dev;
struct list_head list;
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 8/9] selftests: net: add amt_offload.sh for AMT relay offload
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (6 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 7/9] netdevsim: add a TX reference provider for AMT relay offload Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
2026-11-07 18:44 ` [RFC PATCH net-next 9/9] MAINTAINERS: amt: cover the offload header and netdevsim provider Omar Ramadan
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
Exercise the AMT relay replication offload contract against netdevsim's
TX reference provider. A relay in offload auto mode transmits on one
netdevsim port; four in-tree AMT gateways, each in its own netns (an amt
device binds relay_port on INADDR_ANY, so two gateways cannot share
one), reach its peer through a bridge. Copies are counted per gateway on
the decapsulating amtg device, and hardware copies on the provider's own
counter, so the eleven assertions are about what actually reached each
gateway:
1. no membership: no copy anywhere
2. one copy per committed endpoint per packet, all from the provider,
and the provider's devlink resources agree
3. members joining under traffic (generation replace): stable members
get exactly one copy of every packet, and every copy comes from the
provider
4. xmit BUSY: exactly one software copy set, counted
5. forwarding is per group: an INCLUDE(S1) member still receives S2
6. a programming op overruns op_timeout_ms: amt resets the provider,
ignores the stale result, replays, and copies stay at one; the
provider's health reporter records it
7. partial commit (group_update succeeds, group_commit fails): no
hardware copy, the group is served in software
8. every member leaves: once the relay times their sources out, the
group is disabled, no copy anywhere
9. offload required with no provider: refused with an extack, counted
10. the peer unlinked under traffic, relinked, then the relay's netdevsim
deleted under traffic: ordered teardown, no splat
11. a lower carrying two amt devices deleted: both go with it, no splat
(runs first, in its own netns, independent of the offload setup;
one dmesg snapshot after its netns teardown ends its splat scan
and starts case 10's)
No IGMPv3 leave takes effect at once on this base: a BLOCK or TO_IN
leaves the timer of each source it drops alone, so the relay keeps the
source until that timer expires, at least a GMI (260 s by default)
after the last report that refreshed it. So case 3 drives its
generation replaces with joins alone, and case 8 waits out the GMI.
The test SKIPs, loudly, when ip cannot set the offload mode or the
netdevsim has no provider, rather than silently testing software only.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
tools/testing/selftests/net/Makefile | 1 +
tools/testing/selftests/net/amt_offload.sh | 608 +++++++++++++++++++++
tools/testing/selftests/net/config | 2 +
3 files changed, 611 insertions(+)
create mode 100755 tools/testing/selftests/net/amt_offload.sh
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 5db68de0b..2d6c6de0d 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -11,6 +11,7 @@ TEST_PROGS := \
amt.sh \
amt_v6.sh \
amt_gw_v6.sh \
+ amt_offload.sh \
amt_scale.sh \
arp_ndisc_evict_nocarrier.sh \
arp_ndisc_untracked_subnets.sh \
diff --git a/tools/testing/selftests/net/amt_offload.sh b/tools/testing/selftests/net/amt_offload.sh
new file mode 100755
index 000000000..4e00fba0b
--- /dev/null
+++ b/tools/testing/selftests/net/amt_offload.sh
@@ -0,0 +1,608 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# AMT relay replication offload, against netdevsim's TX reference provider.
+#
+# netdevsim keeps its own endpoint and group tables and emits each copy
+# through amt_offload_encap_xmit() out of the relay's stream device to its
+# peer, so a divergence between what amt wants and what the provider
+# committed shows up as a wrong copy count at the gateways. Copies are
+# counted per gateway on the decapsulating amtg device (AF_PACKET), and
+# hardware copies on the provider's own counter in debugfs.
+#
+# RELAY netns GATEWAY netns
+# src0 (S1 172.17.0.2, S2 172.17.0.3)
+# | sender, IP_MULTICAST_IF amtr
+# amtr (relay, offload auto)
+# | stream_dev
+# nsim_r 10.0.0.2/24 <== netdevsim peer link ==> nsim_g -- br0
+# |
+# veth to GATEWAY netns i = 1..4:
+# gw<i> 10.0.0.1<i>/24, amtg<i>
+#
+# One gateway per netns: an amt device binds its UDP socket to relay_port
+# on INADDR_ANY in its netns, gateways included, so a second gateway in
+# the same netns fails to come up with EADDRINUSE.
+#
+# The source sits in the relay netns and transmits on amtr directly: how
+# multicast reaches amtr (mroute) is not under test, amt_dev_xmit is.
+#
+# Eleven assertions, each printing one "TEST: ... [ OK ]" line; the run
+# passes only if all eleven printed OK.
+
+source "$(dirname "$0")"/lib.sh
+
+readonly RELAY=$(mktemp -u amtofl-r-XXXXXXXX)
+readonly GW=$(mktemp -u amtofl-g-XXXXXXXX)
+readonly TMP=$(mktemp -d)
+readonly G1=239.1.1.1 G2=239.1.1.2 G3=239.1.1.3 G4=232.1.1.4 G5=239.1.1.5
+readonly S1=172.17.0.2 S2=172.17.0.3
+readonly K=50
+NSIM_R_ID=$((256 + RANDOM % 256))
+NSIM_G_ID=$((NSIM_R_ID + 256))
+readonly NSIM_SYS=/sys/bus/netdevsim
+readonly DFS=/sys/kernel/debug/netdevsim/netdevsim$NSIM_R_ID/ports/0/amt
+readonly DL=netdevsim/netdevsim$NSIM_R_ID
+ERR=$ksft_skip
+OKS=0
+PIDS=()
+gwns() { echo "$GW-$1"; }
+declare -A JOIN
+
+cleanup()
+{
+ for p in "${PIDS[@]}" "${JOIN[@]}"; do
+ kill "$p" 2>/dev/null
+ done
+ wait 2>/dev/null
+ ip netns del "$RELAY" 2>/dev/null
+ ip netns del "$GW" 2>/dev/null
+ for i in 1 2 3 4 mu; do
+ ip netns del "$(gwns $i)" 2>/dev/null
+ done
+ echo "$NSIM_R_ID" > $NSIM_SYS/del_device 2>/dev/null
+ echo "$NSIM_G_ID" > $NSIM_SYS/del_device 2>/dev/null
+ rm -rf "$TMP"
+ exit $ERR
+}
+
+skip()
+{
+ echo "SKIP: $*"
+ # A failure already printed (assertion 11 runs before the provider
+ # checks) must not be reported as a skip.
+ [ "$ERR" = 1 ] || ERR=$ksft_skip
+ exit $ERR
+}
+
+ok()
+{
+ printf "TEST: %-62s [ OK ]\n" "$1"
+ OKS=$((OKS + 1))
+}
+
+fail()
+{
+ printf "TEST: %-62s [FAIL]\n" "$1"
+ shift
+ [ $# -gt 0 ] && echo " $*"
+ ERR=1
+ return 0
+}
+
+# --- helpers --------------------------------------------------------------
+
+cat > "$TMP/join.py" <<'EOF'
+# join.py IFNAME GROUP [SOURCE]: hold an ASM or SSM membership on IFNAME
+# until killed; leaving is closing the socket.
+import signal, socket, struct, sys, time
+ifn, grp = sys.argv[1], sys.argv[2]
+src = sys.argv[3] if len(sys.argv) > 3 else None
+def sin(a):
+ return struct.pack('=HH4s120x', socket.AF_INET, 0, socket.inet_aton(a))
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+req = struct.pack('=I4x', socket.if_nametoindex(ifn)) + sin(grp)
+if src:
+ s.setsockopt(socket.IPPROTO_IP, 46, req + sin(src)) # MCAST_JOIN_SOURCE_GROUP
+else:
+ s.setsockopt(socket.IPPROTO_IP, 42, req) # MCAST_JOIN_GROUP
+signal.signal(signal.SIGTERM, lambda *a: sys.exit(0))
+while True:
+ time.sleep(3600)
+EOF
+
+cat > "$TMP/send.py" <<'EOF'
+# send.py IFNAME SRC GROUP COUNT INTERVAL: COUNT UDP datagrams to GROUP:5000
+# from SRC out of IFNAME. Send errors (the device going away) are ignored.
+import socket, struct, sys, time
+ifn, src, grp, n, ivl = sys.argv[1:6]
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+s.bind((src, 0))
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF,
+ struct.pack('=4s4si', bytes(4), bytes(4),
+ socket.if_nametoindex(ifn)))
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 8)
+s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_LOOP, 0)
+for i in range(int(n)):
+ try:
+ s.sendto(b'%08d' % i + bytes(56), (grp, 5000))
+ except OSError:
+ pass
+ time.sleep(float(ivl))
+EOF
+
+cat > "$TMP/count.py" <<'EOF'
+# count.py IFNAME...: count received UDP datagrams to port 5000 per
+# (interface, source, group) until SIGTERM, then read what is already
+# queued and print one line per key.
+import collections, select, signal, socket, sys
+socks = {}
+for ifn in sys.argv[1:]:
+ s = socket.socket(socket.AF_PACKET, socket.SOCK_DGRAM, socket.htons(0x0800))
+ s.bind((ifn, 0x0800))
+ socks[s] = ifn
+cnt = collections.Counter()
+done = False
+def stop(*a):
+ global done
+ done = True
+signal.signal(signal.SIGTERM, stop)
+sys.stdout.write('READY\n'); sys.stdout.flush()
+while True:
+ r = select.select(list(socks), [], [], 0 if done else 0.1)[0]
+ if not r and done:
+ break
+ for s in r:
+ d, a = s.recvfrom(4096)
+ if a[2] == socket.PACKET_OUTGOING or len(d) < 28 or d[9] != 17:
+ continue
+ hl = (d[0] & 15) * 4
+ if int.from_bytes(d[hl + 2:hl + 4], 'big') != 5000:
+ continue
+ cnt[(socks[s], socket.inet_ntoa(d[12:16]),
+ socket.inet_ntoa(d[16:20]))] += 1
+for (ifn, src, grp), n in sorted(cnt.items()):
+ print(ifn, src, grp, n)
+EOF
+
+dfs() { cat "$DFS/$1"; }
+dfs_set() { echo "$2" > "$DFS/$1"; }
+rsys() { ip netns exec "$RELAY" cat "/sys/class/net/amtr/offload/$1"; }
+
+# Committed member count of group $1 in the provider's own table, or 0.
+hw_members()
+{
+ awk -v g="group $1" '
+ $0 == g { in_g = 1; next }
+ /^group / || /^endpoint / || /^binding / { in_g = 0 }
+ in_g && $1 == "committed" { n = $5; sub(":", "", n); print n; f = 1; exit }
+ END { if (!f) print 0 }' "$DFS/tables"
+}
+
+hw_group_present()
+{
+ grep -qx "group $1" "$DFS/tables"
+}
+
+# wait_for SECONDS CMD...: poll CMD every 0.2s.
+wait_for()
+{
+ local t=$(( $1 * 5 ))
+
+ shift
+ while [ $t -gt 0 ]; do
+ "$@" && return 0
+ sleep 0.2
+ t=$((t - 1))
+ done
+ return 1
+}
+
+members_eq() { [ "$(hw_members "$1")" = "$2" ]; }
+group_gone() { ! hw_group_present "$1"; }
+sysfs_ge() { [ "$(rsys "$1")" -ge "$2" ]; }
+
+# Members that later leave join source-specific, so a relay that applies
+# INCLUDE leaves at once drops the group at once. This base does not: a
+# BLOCK or TO_IN leaves the timer of each source it drops alone, so the
+# relay keeps the source until that timer expires, at least a GMI after
+# the last report that refreshed it, and deletes the group with its last
+# source (an ASM leave waits out the group timer the same way). So case 8
+# waits out the GMI, and case 3 drives its generation replaces with joins
+# alone.
+join() # join KEY IFNAME GROUP [SOURCE]
+{
+ local key=$1
+
+ shift
+ ip netns exec "$(gwns "${1#amtg}")" python3 "$TMP/join.py" "$@" &
+ JOIN[$key]=$!
+}
+
+leave()
+{
+ kill "${JOIN[$1]}" 2>/dev/null
+ wait "${JOIN[$1]}" 2>/dev/null
+ unset "JOIN[$1]"
+}
+
+# measure SRC GROUP COUNT INTERVAL [HOOK]: send, count per gateway, and
+# the provider's hardware copies. Fills CNT[amtgN] (copies from SRC to
+# GROUP) and HW (tx_copies delta). The counters run until a second after
+# the last packet is sent, however long sending takes. HOOK, a function
+# that must not block, runs once every counter is ready, just before the
+# first packet.
+declare -A CNT
+HW=0
+measure()
+{
+ local src=$1 grp=$2 n=$3 ivl=$4 hw0 line i cpids=()
+
+ hw0=$(dfs tx_copies)
+ for i in 1 2 3 4; do
+ ip netns exec "$(gwns $i)" python3 "$TMP/count.py" amtg$i \
+ > "$TMP/count$i.out" &
+ cpids+=($!)
+ done
+ for i in 1 2 3 4; do
+ wait_for 5 grep -q READY "$TMP/count$i.out"
+ done
+ [ -z "${5:-}" ] || "$5"
+ ip netns exec "$RELAY" python3 "$TMP/send.py" amtr "$src" "$grp" "$n" \
+ "$ivl"
+ sleep 1
+ kill -TERM "${cpids[@]}" 2>/dev/null
+ wait "${cpids[@]}"
+ HW=$(( $(dfs tx_copies) - hw0 ))
+ CNT=([amtg1]=0 [amtg2]=0 [amtg3]=0 [amtg4]=0)
+ while read -r ifn s g c; do
+ [ "$s" = "$src" ] && [ "$g" = "$grp" ] && CNT[$ifn]=$c
+ done < <(cat "$TMP"/count[1-4].out | grep -v READY)
+}
+
+counts() { echo "amtg1..4=${CNT[amtg1]},${CNT[amtg2]},${CNT[amtg3]},${CNT[amtg4]} hw=$HW"; }
+
+dl_occ()
+{
+ devlink -j resource show "$DL" | jq -r --arg r "$1" \
+ '.resources[][] | select(.name == "amt") | .resources[] |
+ select(.name == $r) | .occ'
+}
+
+# --- preflight ------------------------------------------------------------
+
+[ "$(id -u)" -eq 0 ] || skip "needs root"
+for c in python3 jq devlink; do
+ command -v $c > /dev/null || skip "$c not found"
+done
+# A guest whose ip cannot set the mode would silently test software only.
+ip link help amt 2>&1 | grep -q 'offload OFFLOAD' ||
+ skip "ip lacks IFLA_AMT_OFFLOAD_MODE (iproute2 amt offload patch)"
+modprobe amt 2>/dev/null
+modprobe netdevsim 2>/dev/null
+[ -w $NSIM_SYS/new_device ] || skip "netdevsim not available"
+
+trap cleanup EXIT
+DMESG0=$(dmesg | wc -l)
+
+# --- 11. a lower carrying two amt devices is deleted ------------------------
+# Not offload, so it runs first, in its own netns, and is not lost when
+# the offload setup below fails: the amt notifier must take down every
+# amt device on an unregistering lower, not just the first.
+# One left linked trips the upper-dev WARN in
+# unregister_netdevice_many_notify() and a kernfs upper_* splat. The
+# devices stay down: two gateways cannot both bind relay_port in one
+# netns, and linking is all this needs.
+MU=$(gwns mu)
+ip netns add "$MU"
+ip -n "$MU" link add mu0 type dummy
+ip -n "$MU" addr add 10.9.0.1/24 dev mu0
+for d in mu_a mu_b; do
+ ip -n "$MU" link add $d type amt mode gateway local 10.9.0.1 \
+ discovery 10.9.0.2 dev mu0
+done
+made=$(ip -n "$MU" -o link show type amt | wc -l)
+ip -n "$MU" link del mu0
+left=$(ip -n "$MU" -o link show type amt | wc -l)
+ip netns del "$MU"
+# The netns teardown is asynchronous: give it a second. Then one dmesg
+# snapshot ends this case's scan and starts case 10's, so the two scans
+# tile and a splat this case provokes fails this case only.
+sleep 1
+dm=$(dmesg)
+DMESG1=$(printf '%s\n' "$dm" | wc -l)
+splat=$(printf '%s\n' "$dm" | tail -n +$((DMESG0 + 1)) |
+ grep -E 'WARNING:|BUG:|Oops|kernfs: can not remove|refcount_t|use-after-free')
+if [ "$made" -eq 2 ] && [ "$left" -eq 0 ] && [ -z "$splat" ]; then
+ ok "lower carrying two amt devices deleted: both gone"
+else
+ fail "lower carrying two amt devices deleted" \
+ "made=$made left=$left splat: $splat"
+fi
+
+echo "$NSIM_R_ID" > $NSIM_SYS/new_device
+echo "$NSIM_G_ID" > $NSIM_SYS/new_device
+udevadm settle 2>/dev/null
+[ -d "$DFS" ] || skip "netdevsim has no AMT offload provider ($DFS)"
+NR=$(ls $NSIM_SYS/devices/netdevsim$NSIM_R_ID/net)
+NG=$(ls $NSIM_SYS/devices/netdevsim$NSIM_G_ID/net)
+
+set -e
+ip netns add "$RELAY"
+ip netns add "$GW"
+ip link set "$NR" netns "$RELAY"
+ip link set "$NG" netns "$GW"
+ip -n "$RELAY" link set "$NR" name nsim_r
+ip -n "$GW" link set "$NG" name nsim_g
+for ns in "$RELAY" "$GW"; do
+ ip -n "$ns" link set lo up
+done
+ip -n "$RELAY" addr add 10.0.0.2/24 dev nsim_r
+ip -n "$RELAY" link set nsim_r up
+ip -n "$GW" link set nsim_g up
+
+exec {FD_R}<"/var/run/netns/$RELAY"
+exec {FD_G}<"/var/run/netns/$GW"
+IFX_R=$(ip netns exec "$RELAY" cat /sys/class/net/nsim_r/ifindex)
+IFX_G=$(ip netns exec "$GW" cat /sys/class/net/nsim_g/ifindex)
+echo "$FD_R:$IFX_R $FD_G:$IFX_G" > $NSIM_SYS/link_device
+
+ip -n "$RELAY" link add src0 type dummy
+ip -n "$RELAY" addr add $S1/24 dev src0
+ip -n "$RELAY" addr add $S2/24 dev src0
+ip -n "$RELAY" link set src0 up
+ip -n "$RELAY" link add amtr type amt mode relay local 10.0.0.2 dev nsim_r \
+ relay_port 2268 offload auto
+ip -n "$RELAY" link set amtr up
+ip -n "$GW" link add br0 type bridge
+ip -n "$GW" link set nsim_g master br0
+ip -n "$GW" link set br0 up
+for i in 1 2 3 4; do
+ ns=$(gwns $i)
+ ip netns add "$ns"
+ # No IPv6 on the gateways: amtg would join solicited-node groups over
+ # the tunnel and the relay would offload them alongside G1..G4.
+ ip netns exec "$ns" sysctl -qw net.ipv6.conf.default.disable_ipv6=1 \
+ 2>/dev/null || true
+ ip -n "$ns" link set lo up
+ ip -n "$GW" link add br_g$i type veth peer name gw$i netns "$ns"
+ ip -n "$GW" link set br_g$i master br0 up
+ ip -n "$ns" addr add 10.0.0.1$i/24 dev gw$i
+ ip -n "$ns" link set gw$i up
+ ip -n "$ns" link add amtg$i type amt mode gateway local 10.0.0.1$i \
+ discovery 10.0.0.2 dev gw$i gateway_port 2268 relay_port 2268
+ ip -n "$ns" addr add 192.168.$i.1/32 dev amtg$i
+ ip -n "$ns" link set amtg$i up
+done
+set +e
+
+discovered()
+{
+ local i
+
+ for i in 1 2 3 4; do
+ [ "$(ip -n "$(gwns $i)" -d -j link show amtg$i |
+ jq -r '.[0].linkinfo.info_data.remote')" = 10.0.0.2 ] ||
+ return 1
+ done
+}
+wait_for 30 discovered || { fail "setup" "gateways never discovered the relay"; exit; }
+
+mode=$(ip -n "$RELAY" -d -j link show amtr | jq -r '.[0].linkinfo.info_data.offload')
+[ "$mode" = auto ] && [ "$(rsys bound)" = 1 ] ||
+ { fail "setup" "relay offload mode '$mode', bound $(rsys bound)"; exit; }
+ERR=0
+
+# --- 1. no membership, nothing committed: no copies anywhere ----------------
+measure $S1 $G1 $K 0.01
+if [ "$HW" -eq 0 ] && [ $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) -eq 0 ]; then
+ ok "0 copies before any group is committed"
+else
+ fail "0 copies before any group is committed" "$(counts)"
+fi
+
+# --- 2. one hardware copy per committed endpoint per packet -----------------
+for i in 1 2 3; do
+ join a$i amtg$i $G1 $S1
+done
+if ! wait_for 40 members_eq $G1 3; then
+ fail "1 copy per endpoint per packet" \
+ "G1 never committed with 3 members: $(cat "$DFS/tables")"
+else
+ measure $S1 $G1 $K 0.01
+ if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+ [ "${CNT[amtg3]}" -eq $K ] && [ "${CNT[amtg4]}" -eq 0 ] &&
+ [ "$HW" -eq $((3 * K)) ] && [ "$(dl_occ endpoints)" = 3 ] &&
+ [ "$(dl_occ groups)" = 1 ] && [ "$(dl_occ members)" = 3 ]; then
+ ok "1 copy per endpoint per packet, all from the provider"
+ else
+ fail "1 copy per endpoint per packet" "$(counts)" \
+ "devlink endpoints/groups/members:" \
+ "$(dl_occ endpoints)/$(dl_occ groups)/$(dl_occ members)"
+ fi
+fi
+
+# --- 3. generation replace under traffic: no duplicate, no loss --------------
+# G5 has two stable members; two more join while it is sent, 3 s and 6 s
+# into a 12 s stream, so each join is a generation replace under traffic.
+late_joins()
+{
+ ip netns exec "$(gwns 3)" bash -c \
+ "sleep 3; exec python3 $TMP/join.py amtg3 $G5 $S1" &
+ JOIN[l3]=$!
+ ip netns exec "$(gwns 4)" bash -c \
+ "sleep 6; exec python3 $TMP/join.py amtg4 $G5 $S1" &
+ JOIN[l4]=$!
+}
+join l1 amtg1 $G5 $S1
+join l2 amtg2 $G5 $S1
+if ! wait_for 40 members_eq $G5 2; then
+ fail "generation replace under traffic" \
+ "G5 never committed with 2 members: $(cat "$DFS/tables")"
+else
+ c0=$(dfs commits)
+ st0=$(dfs tx_stale_gen)
+ measure $S1 $G5 600 0.02 late_joins
+ gens=$(( $(dfs commits) - c0 ))
+ stale=$(( $(dfs tx_stale_gen) - st0 ))
+ # The stable members got every packet once, the late members some
+ # but not all of them, and every copy came from the provider (HW is
+ # their sum), none from a software fallback on a stale generation.
+ if [ "${CNT[amtg1]}" -eq 600 ] && [ "${CNT[amtg2]}" -eq 600 ] &&
+ [ "${CNT[amtg3]}" -gt 0 ] && [ "${CNT[amtg3]}" -lt 600 ] &&
+ [ "${CNT[amtg4]}" -gt 0 ] && [ "${CNT[amtg4]}" -lt 600 ] &&
+ [ "$HW" -eq $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) ] &&
+ [ $gens -ge 2 ] && [ $stale -eq 0 ]; then
+ ok "generation replace: stable members get exactly 1 copy"
+ else
+ fail "generation replace under traffic" \
+ "$(counts) generations=$gens stale_gen=+$stale"
+ fi
+fi
+
+# --- 4. BUSY: exactly one software copy set -----------------------------------
+b0=$(rsys busy_fallback)
+t0=$(dfs tx_busy)
+dfs_set busy 1
+measure $S1 $G1 $K 0.01
+dfs_set busy 0
+if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+ [ "${CNT[amtg3]}" -eq $K ] && [ "$HW" -eq 0 ] &&
+ [ $(( $(rsys busy_fallback) - b0 )) -eq $K ] && [ $(( $(dfs tx_busy) - t0 )) -eq $K ]; then
+ ok "BUSY: exactly one software copy set per packet"
+else
+ fail "BUSY: exactly one software copy set" "$(counts)" \
+ "busy_fallback +$(( $(rsys busy_fallback) - b0 ))"
+fi
+
+# --- 5. forwarding is per group: INCLUDE(S1) still receives S2 ----------------
+join s1 amtg1 $G4 $S1
+if ! wait_for 40 members_eq $G4 1; then
+ fail "INCLUDE(S1) member receives S2" "G4 never committed: $(cat "$DFS/tables")"
+else
+ measure $S2 $G4 $K 0.01
+ if [ "${CNT[amtg1]}" -eq $K ] && [ "$HW" -eq $K ]; then
+ ok "INCLUDE(S1) member still receives S2, in hardware"
+ else
+ fail "INCLUDE(S1) member receives S2" "$(counts)"
+ fi
+fi
+
+# --- 6. op overruns op_timeout_ms: reset, stale results ignored, no dup -------
+r0=$(rsys resets)
+s0=$(rsys stale_completions)
+f0=$(dfs flushes)
+h0=$(devlink -j health show "$DL" reporter amt | jq '.health[][0].error')
+dfs_set stall_ms 1500
+join r2 amtg2 $G3
+wait_for 30 sysfs_ge resets $((r0 + 1))
+dfs_set stall_ms 0
+if ! wait_for 40 members_eq $G3 1; then
+ fail "reset: stale completion ignored" \
+ "G3 never committed after reset: $(cat "$DFS/tables")"
+else
+ measure $S1 $G3 $K 0.01
+ h1=$(devlink -j health show "$DL" reporter amt | jq '.health[][0].error')
+ if [ "${CNT[amtg2]}" -eq $K ] && [ "$HW" -eq $K ] &&
+ [ "$(rsys resets)" -gt "$r0" ] && [ "$(rsys stale_completions)" -gt "$s0" ] &&
+ [ "$(dfs flushes)" -gt "$f0" ] && [ "$h1" -gt "$h0" ]; then
+ ok "reset: stale completion ignored, replay gives 1 copy"
+ else
+ fail "reset: stale completion ignored" "$(counts)" \
+ "resets $r0->$(rsys resets)" \
+ "stale $s0->$(rsys stale_completions) health $h0->$h1"
+ fi
+fi
+
+# --- 7. partial commit: group staged, commit refused -> software -------------
+w0=$(rsys sw_remainder)
+dfs_set fail_commit 1
+for i in 1 2 3; do
+ join p$i amtg$i $G2 $S1
+done
+wait_for 40 sysfs_ge sw_remainder $((w0 + 1))
+sleep 2
+measure $S1 $G2 $K 0.01
+dfs_set fail_commit 0
+if [ "${CNT[amtg1]}" -eq $K ] && [ "${CNT[amtg2]}" -eq $K ] &&
+ [ "${CNT[amtg3]}" -eq $K ] && [ "$HW" -eq 0 ] && [ "$(hw_members $G2)" = 0 ]; then
+ ok "partial commit: 0 hardware copies, software remainder"
+else
+ fail "partial commit: software remainder" "$(counts) sw_remainder $w0->$(rsys sw_remainder)"
+fi
+for i in 1 2 3; do
+ leave p$i
+done
+
+# --- 8. every member leaves: group disabled, no copies ------------------------
+# The relay drops S1, and G1 with it, at least a GMI after the last
+# report that refreshed it: qrv * 125 s query interval + 10 s response
+# interval. The timer wheel can fire it up to 8/63 of its delay late
+# (32.8 s of a 260 s GMI at HZ=1000), so wait a seventh more, and 10 s.
+gmi=$(( $(ip netns exec "$RELAY" sysctl -n net.ipv4.igmp_qrv) * 125 + 10 ))
+for i in 1 2 3; do
+ leave a$i
+done
+if wait_for $((gmi + gmi / 7 + 10)) group_gone $G1; then
+ measure $S1 $G1 $K 0.01
+ if [ "$HW" -eq 0 ] && [ $((CNT[amtg1] + CNT[amtg2] + CNT[amtg3] + CNT[amtg4])) -eq 0 ]; then
+ ok "0 copies after the group is disabled"
+ else
+ fail "0 copies after the group is disabled" "$(counts)"
+ fi
+else
+ fail "0 copies after the group is disabled" "G1 still in the provider: $(cat "$DFS/tables")"
+fi
+
+# --- 9. required mode with no provider: refused with extack -----------------
+ip -n "$RELAY" link add dum0 type dummy
+ip -n "$RELAY" addr add 10.0.1.2/24 dev dum0
+q0=$(cat /sys/module/amt/parameters/offload_required_refused)
+msg=$(ip -n "$RELAY" link add amtx type amt mode relay local 10.0.1.2 \
+ dev dum0 offload required 2>&1)
+rc=$?
+q1=$(cat /sys/module/amt/parameters/offload_required_refused)
+if [ $rc -ne 0 ] && echo "$msg" | grep -q '^Error: ' && [ $((q1 - q0)) -eq 1 ] &&
+ ! ip -n "$RELAY" link show amtx > /dev/null 2>&1; then
+ ok "required with no provider: refused with extack, counted"
+else
+ fail "required refuses with extack" "rc=$rc refused $q0->$q1 msg: $msg"
+fi
+
+# --- 10. lower device torn away under traffic: ordered teardown ---------------
+# Unlink the peer mid-stream (copies stop, the binding survives), relink
+# (copies resume at 1 per packet), then delete the relay's netdevsim under
+# traffic: amt tears the binding down before the relay goes, with no splat.
+# The unlink-phase stream is from S1, so copies queued while unlinked and
+# released on relink are not counted as the relinked (S2, G4) copies.
+ip netns exec "$RELAY" python3 "$TMP/send.py" amtr $S1 $G4 100 0.02 &
+PIDS+=($!)
+sleep 0.5
+echo "$FD_R:$IFX_R" > $NSIM_SYS/unlink_device
+wait "${PIDS[-1]}"
+bound_after_unlink=$(rsys bound)
+echo "$FD_R:$IFX_R $FD_G:$IFX_G" > $NSIM_SYS/link_device
+wait_for 20 members_eq $G4 1
+measure $S2 $G4 $K 0.01
+relinked=${CNT[amtg1]}
+relinked_hw=$HW
+ip netns exec "$RELAY" python3 "$TMP/send.py" amtr $S2 $G4 200 0.01 &
+PIDS+=($!)
+sleep 0.5
+echo "$NSIM_R_ID" > $NSIM_SYS/del_device
+wait "${PIDS[-1]}"
+splat=$(dmesg | tail -n +$((DMESG1 + 1)) |
+ grep -E 'WARNING:|BUG:|Oops|refcount_t|use-after-free|list_(add|del) corruption')
+if [ "$bound_after_unlink" = 1 ] && [ "$relinked" -eq $K ] && [ "$relinked_hw" -eq $K ] &&
+ ! ip -n "$RELAY" link show amtr > /dev/null 2>&1 && [ ! -d "$DFS" ] &&
+ [ -z "$splat" ]; then
+ ok "lower device unlinked and deleted under traffic: ordered teardown"
+else
+ fail "ordered teardown under traffic" \
+ "bound=$bound_after_unlink relinked=$relinked hw=$relinked_hw splat: $splat"
+fi
+
+if [ $OKS -ne 11 ]; then
+ echo "only $OKS of 11 assertions passed"
+ ERR=1
+fi
diff --git a/tools/testing/selftests/net/config b/tools/testing/selftests/net/config
index 94d722770..88eb75060 100644
--- a/tools/testing/selftests/net/config
+++ b/tools/testing/selftests/net/config
@@ -11,6 +11,7 @@ CONFIG_CRYPTO_ARIA=y
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SM4_GENERIC=y
+CONFIG_DEBUG_FS=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=n
CONFIG_DUMMY=y
@@ -117,6 +118,7 @@ CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_GENEVE=m
CONFIG_OPENVSWITCH_GRE=m
CONFIG_OPENVSWITCH_VXLAN=m
+CONFIG_PACKET=y
CONFIG_PROC_SYSCTL=y
CONFIG_PSAMPLE=m
CONFIG_RPS=y
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH net-next 9/9] MAINTAINERS: amt: cover the offload header and netdevsim provider
2026-11-07 18:44 [RFC PATCH net-next 0/9] amt: relay replication offload provider contract (RFC, not for merge, seeking a provider) Omar Ramadan
` (7 preceding siblings ...)
2026-11-07 18:44 ` [RFC PATCH net-next 8/9] selftests: net: add amt_offload.sh " Omar Ramadan
@ 2026-11-07 18:44 ` Omar Ramadan
8 siblings, 0 replies; 10+ messages in thread
From: Omar Ramadan @ 2026-11-07 18:44 UTC (permalink / raw)
To: Taehee Yoo, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, michael.chan, pavan.chebbi, saeedm, leon,
tariqt, mbloch, brett.creeley, sgoutham, anthony.l.nguyen,
przemyslaw.kitszel, intel-wired-lan, Ido Schimmel, Satish Kharat,
Andrew Lunn, Simon Horman, Shuah Khan, Kees Cook,
Gustavo A. R. Silva, linux-kselftest, linux-hardening
The AMT entry does not list include/net/amt_offload.h or the netdevsim
reference provider drivers/net/netdevsim/amt.c, so get_maintainer.pl
does not find the AMT maintainer for changes to either. Add both.
tools/testing/selftests/net/amt_offload.sh is covered by the entry's
amt*.sh pattern, which the IPv6 outer transport series adds.
drivers/net/netdevsim/amt.c stays under the NETDEVSIM entry as well,
through its existing drivers/net/netdevsim/* pattern, so the netdevsim
maintainer is still found for it.
Assisted-by: LLM
Signed-off-by: Omar Ramadan <omar@blockcast.net>
---
MAINTAINERS | 2 ++
1 file changed, 2 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index b01fdeee6..f6d5b471f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1404,6 +1404,8 @@ S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
F: drivers/net/amt.c
+F: drivers/net/netdevsim/amt.c
+F: include/net/amt_offload.h
F: tools/testing/selftests/net/amt*.sh
ANALOG DEVICES INC AD3530R DRIVER
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread