[PATCH net-next v2] veth: set the real number of queues before registering the devices

[PATCH net-next v2] veth: set the real number of queues before registering the devices

[Qiliang Yuan]

Since commit 9d3684c24a52 ("veth: create by default nr_possible_cpus
queues"), each veth device is allocated with one rx and one tx queue
per possible CPU, and veth_newlink() reduces the real number of queues
to one unless the user asked for a number.

Both devices are already registered at that point, so
netif_set_real_num_tx_queues() and netif_set_real_num_rx_queues()
remove the sysfs objects of every extra queue that register_netdevice()
has just created, each one emitting an add and a remove uevent, and the
tx side also waits for an expedited RCU grace period. This happens with
RTNL held, twice per pair, and grows with the number of possible CPUs:
with 240 of them a veth pair emits about 1900 uevents, and udev takes
minutes to work through those of a few hundred pairs.

Set the number of queues before registering each device. For a device
that is not registered yet this only updates the counters, so the extra
queues never appear in sysfs, and the error paths that undid a
registration are no longer needed. Creating and then deleting 500 veth
pairs, one ip -batch each, in a 16 vCPU guest, median of 3 runs, per
pair with udev stopped, and how long udev stays busy after the deletion
with it running:

                                   before      after
  16 possible CPUs    create        902 us      295 us
                      delete       6180 us     6147 us
                      udev busy     > 300 s      4.5 s
                      uevents           126          6
  240 possible CPUs   create       9354 us      505 us
                      delete       7343 us     6916 us
                      udev busy     > 300 s      6.2 s
                      uevents          1918          6

Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
Tested-by: Wang Zhan <wang.zhan@smartx.com>
Reviewed-by: Wang Zhan <wang.zhan@smartx.com>
---
V1 -> V2:
- Rebase onto net-next, where the peer's queues are set from its own
  attributes (tbp), and keep setting them before registering the peer
- Retest in another guest with udev stopped, where deletion is not
  faster, and add how long udev stays busy afterwards, as deletion times
  with udev running mostly depend on its backlog
- Keep Wang Zhan's tags, as the code only changes by the rebase

v1: https://lore.kernel.org/r/20261016-bug-veth-init-queues-before-register-v1-1-f48eb2bb25a0@gmail.com
---
 drivers/net/veth.c | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index a61842b2585bf..0539b3661793d 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -1875,6 +1875,10 @@ static int veth_newlink(struct net_device *dev,
 
 	netif_inherit_tso_max(peer, dev);
 
+	err = veth_init_queues(peer, tbp);
+	if (err)
+		goto err_register_peer;
+
 	err = register_netdevice(peer);
 	if (err < 0)
 		goto err_register_peer;
@@ -1904,6 +1908,10 @@ static int veth_newlink(struct net_device *dev,
 	else
 		snprintf(dev->name, IFNAMSIZ, DRV_NAME "%%d");
 
+	err = veth_init_queues(dev, tb);
+	if (err)
+		goto err_register_dev;
+
 	err = register_netdevice(dev);
 	if (err < 0)
 		goto err_register_dev;
@@ -1917,16 +1925,10 @@ static int veth_newlink(struct net_device *dev,
 	priv = netdev_priv(dev);
 	rcu_assign_pointer(priv->peer, peer);
 	netdev_hold(peer, &priv->peer_tracker, GFP_KERNEL);
-	err = veth_init_queues(dev, tb);
-	if (err)
-		goto err_queues;
 
 	priv = netdev_priv(peer);
 	rcu_assign_pointer(priv->peer, dev);
 	netdev_hold(dev, &priv->peer_tracker, GFP_KERNEL);
-	err = veth_init_queues(peer, tbp);
-	if (err)
-		goto err_peer_queues;
 
 	veth_disable_gro(dev);
 	/* update XDP supported features */
@@ -1935,12 +1937,6 @@ static int veth_newlink(struct net_device *dev,
 
 	return 0;
 
-err_peer_queues:
-	netdev_put(dev, &priv->peer_tracker);
-	priv = netdev_priv(dev);
-err_queues:
-	netdev_put(peer, &priv->peer_tracker);
-	unregister_netdevice(dev);
 err_register_dev:
 	/* nothing to do */
 err_configure_peer:

---
base-commit: fa5196392691051963726787944db872420dfa75
change-id: 20261016-bug-veth-init-queues-before-register-eebe763a01a7

Best regards,
-- 
Qiliang Yuan <odys.yuan@gmail.com>