[PATCH] vxlan: serialize nexthop fdb_list against cross-device concurrent updates

[PATCH] vxlan: serialize nexthop fdb_list against cross-device concurrent updates

[Seungwon Bae]

struct nexthop.fdb_list is a per-nexthop global list of the vxlan fdb
entries that reference an fdb-nexthop.  vxlan serializes it only with the
per-device vxlan->hash_lock.  When two vxlan devices reference the same
fdb-nexthop, their entries share one nh->fdb_list but each device takes
only its own hash_lock.  vxlan_cleanup() (the aging GC timer) runs in
softirq with only its device's hash_lock and calls vxlan_fdb_destroy() ->
list_del_rcu(&f->nh_list); a concurrent list_add_tail_rcu()/list_del_rcu()
from the other device corrupts the shared list, leaving a freed
kmalloc-128 struct vxlan_fdb linked on nh->fdb_list (use-after-free later
consumed by vxlan_fdb_nh_flush()).

On CONFIG_DEBUG_LIST/KASAN this reproduces as:

  list_del corruption. next->prev should be ..., but was dead000000000122.
  __list_del_entry_valid_or_report <- vxlan_fdb_destroy <- vxlan_cleanup
  <- run_timer_softirq
  BUG: KASAN: slab-use-after-free in vxlan_fdb_destroy

Serialize all nh->fdb_list mutations with the nexthop's own spinlock
(nh->lock), which - unlike rtnl/hash_lock - is usable from the softirq
aging path.  Lock order is hash_lock -> nh->lock at every mutation site;
vxlan_fdb_nh_flush() keeps traversing under rcu_read_lock().

Verified with a KASAN + CONFIG_DEBUG_LIST kernel and an unprivileged
(userns+netns) reproducer that runs two vxlan devices churning
add + aging-delete on a shared fdb-nexthop: BEFORE = 490 list_del
corruptions + slab-use-after-free; AFTER = 0 corruptions, 0 KASAN
reports, no deadlock.

Signed-off-by: Seungwon Bae <qotmddnjs@ajou.ac.kr>
---
 drivers/net/vxlan/vxlan_core.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index ac88d1c85..2a427e89e 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -843,11 +843,15 @@ static int vxlan_fdb_nh_update(struct vxlan_dev *vxlan, struct vxlan_fdb *fdb,
 	}
 
 	if (old_nh) {
+		spin_lock_bh(&old_nh->lock);
 		list_del_rcu(&fdb->nh_list);
+		spin_unlock_bh(&old_nh->lock);
 		nexthop_put(old_nh);
 	}
 	rcu_assign_pointer(fdb->nh, nh);
+	spin_lock_bh(&nh->lock);
 	list_add_tail_rcu(&fdb->nh_list, &nh->fdb_list);
+	spin_unlock_bh(&nh->lock);
 	return 1;
 
 err_inval:
@@ -938,6 +942,7 @@ static void vxlan_fdb_free(struct rcu_head *head)
 static void vxlan_fdb_destroy(struct vxlan_dev *vxlan, struct vxlan_fdb *f,
 			      bool do_notify, bool swdev_notify)
 {
+	struct nexthop *nh;
 	struct vxlan_rdst *rd;
 
 	netdev_dbg(vxlan->dev, "delete %pM\n", f->key.eth_addr);
@@ -956,7 +961,14 @@ static void vxlan_fdb_destroy(struct vxlan_dev *vxlan, struct vxlan_fdb *f,
 	hlist_del_init_rcu(&f->fdb_node);
 	rhashtable_remove_fast(&vxlan->fdb_hash_tbl, &f->rhnode,
 			       vxlan_fdb_rht_params);
-	list_del_rcu(&f->nh_list);
+	nh = rcu_dereference_protected(f->nh, lockdep_is_held(&vxlan->hash_lock));
+	if (nh) {
+		spin_lock_bh(&nh->lock);
+		list_del_rcu(&f->nh_list);
+		spin_unlock_bh(&nh->lock);
+	} else {
+		list_del_rcu(&f->nh_list);
+	}
 	call_rcu(&f->rcu, vxlan_fdb_free);
 }
 
-- 
2.43.0

Re: [PATCH] vxlan: serialize nexthop fdb_list against cross-device concurrent updates

[Ido Schimmel]

On Tue, Sep 01, 2026 at 02:02:53PM +0900, Seungwon Bae wrote:
> struct nexthop.fdb_list is a per-nexthop global list of the vxlan fdb
> entries that reference an fdb-nexthop.  vxlan serializes it only with the
> per-device vxlan->hash_lock.  When two vxlan devices reference the same
> fdb-nexthop, their entries share one nh->fdb_list but each device takes
> only its own hash_lock.  vxlan_cleanup() (the aging GC timer) runs in
> softirq with only its device's hash_lock and calls vxlan_fdb_destroy() ->
> list_del_rcu(&f->nh_list); a concurrent list_add_tail_rcu()/list_del_rcu()
> from the other device corrupts the shared list, leaving a freed
> kmalloc-128 struct vxlan_fdb linked on nh->fdb_list (use-after-free later
> consumed by vxlan_fdb_nh_flush()).
> 
> On CONFIG_DEBUG_LIST/KASAN this reproduces as:
> 
>   list_del corruption. next->prev should be ..., but was dead000000000122.
>   __list_del_entry_valid_or_report <- vxlan_fdb_destroy <- vxlan_cleanup
>   <- run_timer_softirq
>   BUG: KASAN: slab-use-after-free in vxlan_fdb_destroy
> 
> Serialize all nh->fdb_list mutations with the nexthop's own spinlock
> (nh->lock), which - unlike rtnl/hash_lock - is usable from the softirq
> aging path.  Lock order is hash_lock -> nh->lock at every mutation site;
> vxlan_fdb_nh_flush() keeps traversing under rcu_read_lock().
> 
> Verified with a KASAN + CONFIG_DEBUG_LIST kernel and an unprivileged
> (userns+netns) reproducer that runs two vxlan devices churning
> add + aging-delete on a shared fdb-nexthop: BEFORE = 490 list_del
> corruptions + slab-use-after-free; AFTER = 0 corruptions, 0 KASAN
> reports, no deadlock.

vxlan_cleanup() shouldn't age out FDB entries that point to a nexthop
ID. While their timestamp is updated by vxlan_snoop(), such entries
cannot roam, unlike normal entries.

I see two possible fixes:

1. Have vxlan_cleanup() skip such entries.

2. Reject FDB entries that point to a nexthop ID from being dynamic,
both when created and when an existing entry is updated.

The first is safer, but the second is the more correct fix and I doubt
anyone is relying on this behavior. Therefore, I suggest going with the
second option. Something like [1]. AFAICT, this means that
nexthop.fdb_list will only be manipulated under RTNL.

Also you need a Fixes tag:

Fixes: 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries")

Please read [2] and run [3] and [4] on v2 and future submissions.

Thanks

[1]
diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index 459f19f7071e..be95af64a1f5 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -996,6 +996,12 @@ static int vxlan_fdb_update_existing(struct vxlan_dev *vxlan,
 		return -EOPNOTSUPP;
 	}
 
+	if (rcu_access_pointer(f->nh) &&
+	    !(state & (NUD_PERMANENT | NUD_NOARP))) {
+		NL_SET_ERR_MSG(extack, "Cannot make a nexthop fdb dynamic");
+		return -EOPNOTSUPP;
+	}
+
 	/* Do not allow an externally learned entry to take over an entry added
 	 * by the user.
 	 */
@@ -1257,6 +1263,11 @@ static int vxlan_fdb_add(struct ndmsg *ndm, struct nlattr *tb[],
 	if (err)
 		return err;
 
+	if (nhid && !(ndm->ndm_state & (NUD_PERMANENT | NUD_NOARP))) {
+		NL_SET_ERR_MSG(extack, "A nexthop fdb cannot be dynamic");
+		return -EINVAL;
+	}
+
 	if (vxlan->default_dst.remote_ip.sa.sa_family != ip.sa.sa_family)
 		return -EAFNOSUPPORT;
 
diff --git a/tools/testing/selftests/net/fib_nexthops.sh b/tools/testing/selftests/net/fib_nexthops.sh
index 3d347126730a..431d7bed7622 100755
--- a/tools/testing/selftests/net/fib_nexthops.sh
+++ b/tools/testing/selftests/net/fib_nexthops.sh
@@ -533,6 +533,20 @@ ipv6_fdb_grp_fcnal()
 	run_cmd "$BRIDGE fdb add 02:02:00:00:00:14 dev vx10 nhid 61 self"
 	log_test $? 255 "Fdb mac add with nexthop"
 
+	# fdb entries with a nexthop group cannot be aged out
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:15 dev vx10 nhid 102 self static"
+	log_test $? 0 "Fdb mac add with nexthop group and static state"
+
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:16 dev vx10 nhid 102 self dynamic"
+	log_test $? 255 "Fdb mac add with nexthop group and dynamic state"
+
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:17 dev vx10 nhid 102 self"
+	run_cmd "$BRIDGE fdb replace 02:02:00:00:00:17 dev vx10 dst 2001:db8:91::11 self dynamic"
+	log_test $? 255 "Fdb mac replace with nexthop group and dynamic state"
+
+	run_cmd "$BRIDGE fdb append 02:02:00:00:00:17 dev vx10 dst 2001:db8:91::11 self dynamic"
+	log_test $? 255 "Fdb mac append with nexthop group and dynamic state"
+
 	run_cmd "$IP -6 ro add 2001:db8:101::1/128 nhid 66"
 	log_test $? 2 "Route add with fdb nexthop"
 
@@ -669,6 +683,20 @@ ipv4_fdb_grp_fcnal()
 	run_cmd "$BRIDGE fdb add 02:02:00:00:00:14 dev vx10 nhid 12 self"
 	log_test $? 255 "Fdb mac add with nexthop"
 
+	# fdb entries with a nexthop group cannot be aged out
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:15 dev vx10 nhid 102 self static"
+	log_test $? 0 "Fdb mac add with nexthop group and static state"
+
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:16 dev vx10 nhid 102 self dynamic"
+	log_test $? 255 "Fdb mac add with nexthop group and dynamic state"
+
+	run_cmd "$BRIDGE fdb add 02:02:00:00:00:17 dev vx10 nhid 102 self"
+	run_cmd "$BRIDGE fdb replace 02:02:00:00:00:17 dev vx10 dst 10.0.0.3 self dynamic"
+	log_test $? 255 "Fdb mac replace with nexthop group and dynamic state"
+
+	run_cmd "$BRIDGE fdb append 02:02:00:00:00:17 dev vx10 dst 10.0.0.3 self dynamic"
+	log_test $? 255 "Fdb mac append with nexthop group and dynamic state"
+
 	run_cmd "$IP ro add 172.16.0.0/22 nhid 16"
 	log_test $? 2 "Route add with fdb nexthop"

[2] https://docs.kernel.org/next/process/maintainer-netdev.html
[3] https://github.com/linux-netdev/nipa#running-locally
[4] https://github.com/sashiko-dev/sashiko#usage