From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Alexei Starovoitov <ast@kernel.org>, <bpf@vger.kernel.org>,
"Carolina Jubran" <cjubran@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
"Daniel Borkmann" <daniel@iogearbox.net>,
Dragos Tatulea <dtatulea@nvidia.com>,
"Gal Pressman" <gal@nvidia.com>,
Jesper Dangaard Brouer <hawk@kernel.org>,
"John Fastabend" <john.fastabend@gmail.com>,
Kees Cook <kees+treewide@kernel.org>,
Leon Romanovsky <leon@kernel.org>,
open list <linux-kernel@vger.kernel.org>,
<linux-rdma@vger.kernel.org>, Li RongQing <lirongqing@baidu.com>,
Mark Bloch <mbloch@nvidia.com>,
Richard Cochran <richardcochran@gmail.com>,
"Saeed Mahameed" <saeedm@nvidia.com>,
Stanislav Fomichev <sdf@fomichev.me>,
"Tariq Toukan" <tariqt@nvidia.com>
Subject: [PATCH net-next V3 0/2] net/mlx5e: Batch RCU grace periods on channel open and close
Date: Thu, 19 Nov 2026 12:39:33 +0200 [thread overview]
Message-ID: <20261119103935.2190627-1-tariqt@nvidia.com> (raw)
Hi,
This series by Carolina reduces the time mlx5e spends waiting for RCU
grace periods when bringing an interface up or down with many channels.
Today the channel open and close flows call synchronize_net() per
object. On close, RQ, ICOSQ, TX SQ and NAPI teardown each require a
grace period, so the number of waits scales with the number of channels.
TX SQ teardown adds a wait per traffic class, and XDP and QoS can add
more. On activation, mlx5e_activate_channels() triggers the ICOSQ of
every channel, and each trigger waits for its own grace period. With a
large channel count this makes link down and link up slow.
Patch 1 batches the waits on channel close. Channel deactivate and close
are split into pre_sync and post_sync phases: the ENABLED state is first
cleared on all queues of all channels, a single grace period is awaited,
and only then does the teardown continue. Once NAPI is disabled on all
channels, one more grace period is awaited, as the poll may still arm
the CQs after napi_complete_done(). The queues are then closed and the
NAPI instances are unlinked with __netif_napi_del_locked(), followed by
one synchronize_net() before the channels are freed. The existing per
object synchronize_net() calls are kept for single queue paths. The
change touches the regular channels as well as the PTP, QoS, XDP and XSK
queues.
Patch 2 does the same for the ICOSQ trigger on activation: all channel
ICOSQs are first marked as requiring locking, one grace period is
awaited, and then all of them are triggered. mlx5e_trigger_napi_icosq()
keeps its existing semantics for standalone callers.
Results with 248 channels and rcu_normal=1:
- ip link set eth2 down: 11.837s before, 0.817s after.
- ip link set eth2 up: 4.087s before, 1.080s after.
Regards,
Tariq
V3:
- Wait for one more grace period after napi_disable() in deactivate, so
close can destroy the queues before its single synchronize_net().
This fixes the page pool and PTP SQ recovery issues from V2, and makes
the close path simpler and cleaner, with no separate page pool unlink
step.
- Drop the synchronize_net() in mlx5e_close_xdpsq() instead of splitting
it, as its callers already wait for a grace period or run before NAPI
is enabled.
- Test and clear the XSK state bit once on close, instead of using
xsk_pool to decide on XSK teardown.
- Add a sync_needed parameter to mlx5e_qos_close_queues() instead of a
separate variant without the wait.
- Keep the original mlx5e_xdp_tx_disable() name instead of renaming it
to _pre_sync, and only drop its synchronize_net().
- Build the single object paths from pre_sync, synchronize_net() and
post_sync, and keep mlx5e_close_locked() calling the existing
wrappers.
- Remove the priv level deactivate wrappers.
- Order the new functions as pre_sync, post_sync, then the wrapper.
V2:
https://lore.kernel.org/all/20261110185005.2932568-1-tariqt@nvidia.com/
V1:
https://lore.kernel.org/all/20261011133212.807979-1-tariqt@nvidia.com/
Carolina Jubran (2):
net/mlx5e: Batch NAPI RCU grace periods on channel close
net/mlx5e: Batch ICOSQ trigger RCU synchronization
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +
.../net/ethernet/mellanox/mlx5/core/en/ptp.c | 27 ++-
.../net/ethernet/mellanox/mlx5/core/en/ptp.h | 4 +-
.../net/ethernet/mellanox/mlx5/core/en/qos.c | 42 ++++-
.../net/ethernet/mellanox/mlx5/core/en/qos.h | 5 +-
.../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 -
.../mellanox/mlx5/core/en/xsk/setup.c | 22 ++-
.../mellanox/mlx5/core/en/xsk/setup.h | 2 +
.../net/ethernet/mellanox/mlx5/core/en_main.c | 154 ++++++++++++++----
9 files changed, 207 insertions(+), 55 deletions(-)
base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6
--
2.44.0
next reply other threads:[~2026-11-19 10:40 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 10:39 Tariq Toukan [this message]
2026-11-19 10:39 ` [PATCH net-next V3 1/2] net/mlx5e: Batch NAPI RCU grace periods on channel close Tariq Toukan
2026-11-19 10:39 ` [PATCH net-next V3 2/2] net/mlx5e: Batch ICOSQ trigger RCU synchronization Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119103935.2190627-1-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=cjubran@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dtatulea@nvidia.com \
--cc=edumazet@kernel.org \
--cc=gal@nvidia.com \
--cc=hawk@kernel.org \
--cc=john.fastabend@gmail.com \
--cc=kees+treewide@kernel.org \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=lirongqing@baidu.com \
--cc=mbloch@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=richardcochran@gmail.com \
--cc=saeedm@nvidia.com \
--cc=sdf@fomichev.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®