From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Andrea Parri <parri.andrea@gmail.com>,
Boris Pismenny <borisp@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
Dragos Tatulea <dtatulea@nvidia.com>,
"Gal Pressman" <gal@nvidia.com>,
Jeroen Massar <jmassar@nvidia.com>,
Jianbo Liu <jianbol@nvidia.com>,
Kuniyuki Iwashima <kuniyu@google.com>,
Leon Romanovsky <leon@kernel.org>,
open list <linux-kernel@vger.kernel.org>,
<linux-rdma@vger.kernel.org>, Mark Bloch <mbloch@nvidia.com>,
"Patrisious Haddad" <phaddad@nvidia.com>,
Prathamesh Deshpande <prathameshdeshpande7@gmail.com>,
Raed Salem <raeds@nvidia.com>, Roi Dayan <roid@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>, Shay Drory <shayd@nvidia.com>,
Simon Horman <horms@kernel.org>,
Steffen Klassert <steffen.klassert@secunet.com>,
Tariq Toukan <tariqt@nvidia.com>
Subject: [PATCH net V3 4/8] net/mlx5e: Fix E-Switch races in uplink unbind
Date: Thu, 5 Nov 2026 23:16:32 +0200 [thread overview]
Message-ID: <20261105211637.1083978-5-tariqt@nvidia.com> (raw)
In-Reply-To: <20261105211637.1083978-1-tariqt@nvidia.com>
From: Cosmin Ratiu <cratiu@nvidia.com>
Unbinding the uplink Ethernet auxiliary driver in switchdev mode with
offloaded IPsec policies deadlocks on the esw->mode_lock semaphore:
mlx5_eswitch_safe_aux_devs_remove()
mlx5_esw_hold()
down_read_trylock(&esw->mode_lock) (*)
mlx5_core_reps_aux_devs_remove()
...
mlx5e_vport_uplink_rep_unload()
unregister_netdev()
xfrm_dev_unregister()
xfrm_dev_policy_flush()
xfrm_policy_kill()
xfrm_dev_policy_delete()
mlx5e_xfrm_del_policy()
mlx5e_accel_ipsec_fs_del_pol()
rx_ft_put_policy()
rx_put()
mlx5_eswitch_unblock_mode()
down_write(&esw->mode_lock) (BOOM)
Another deadlock with IPsec states:
mlx5_eswitch_safe_aux_devs_remove()
mlx5_esw_hold()
down_read_trylock(&esw->mode_lock) (*)
mlx5_core_reps_aux_devs_remove()
...
mlx5e_vport_uplink_rep_unload()
unregister_netdev()
xfrm_dev_unregister()
xfrm_dev_state_flush()
xfrm_dev_state_free()
mlx5e_xfrm_free_state()
mlx5e_accel_ipsec_fs_del_rule()
rx_ft_put()
rx_put()
rx_destroy()
mlx5_eswitch_unblock_encap()
down_write(&esw->mode_lock) (BOOM)
To fix these, first refactor initiators of eswitch operations so they no
longer step on each other's toes.
The active/inactive toggle and setting the inline/encap mode now use the
new mlx5_esw_lock_wait(), which waits for an operation in progress to
finish. They all hold devl_lock, so the only operation they can find in
progress is an uplink unbind, which doesn't need devl_lock.
Setting the esw mode keeps using mlx5_esw_try_lock(), which also rejects
the change while eswitch users exist and bails out with -EBUSY.
mlx5_eswitch_safe_aux_devs_remove() now also waits for an operation in
progress with the same mlx5_esw_lock_wait() before initiating one,
dropping mode_lock and then removing representors.
The deadlocks are avoided.
Finishing an operation in progress now requires mlx5_esw_unlock() to
wake up any waiters instead of plain up_write(&mode_lock).
As a side effect, the timeout in mlx5_eswitch_safe_aux_devs_remove() is
gone, which fixes another scary corner case, of representor devs not
being removed on timeout but leaked instead. The timeout hid the SD
deadlock fixed in the previous patch. Now that that's gone, no reason
for the timeout to stay.
This was found by internal testing and reproduced on HW.
It was independently found by Sashiko during analysis of another patch
in this series.
Fixes: 2ae8c7edea87 ("net/mlx5: Fix Unbinding uplink-netdev in switchdev mode")
Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
Reviewed-by: Jianbo Liu <jianbol@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/eswitch.c | 24 +++++++++
.../net/ethernet/mellanox/mlx5/core/eswitch.h | 1 +
.../mellanox/mlx5/core/eswitch_offloads.c | 50 +++++++++----------
3 files changed, 48 insertions(+), 27 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index 2d289fb1353b..f50993cfcf7a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -3051,12 +3051,36 @@ int mlx5_esw_lock(struct mlx5_eswitch *esw)
return 0;
}
+static void esw_mode_lock(struct mlx5_eswitch *esw)
+{
+ down_write(&esw->mode_lock);
+}
+
+static void esw_mode_unlock(struct mlx5_eswitch *esw)
+{
+ up_write(&esw->mode_lock);
+}
+
+/* Lock the eswitch and wait for an operation in progress to finish. */
+void mlx5_esw_lock_wait(struct mlx5_eswitch *esw)
+{
+ down_write(&esw->mode_lock);
+ wait_var_event_any_lock(&esw->eswitch_operation_in_progress,
+ !esw->eswitch_operation_in_progress,
+ esw, esw_mode, TASK_UNINTERRUPTIBLE);
+}
+
/**
* mlx5_esw_unlock() - Release write lock on esw mode lock
* @esw: eswitch device.
+ *
+ * Also wakes up mlx5_esw_lock_wait() waiters, so it must be used to release
+ * mode_lock after clearing eswitch_operation_in_progress.
*/
void mlx5_esw_unlock(struct mlx5_eswitch *esw)
{
+ wake_up_var_locked(&esw->eswitch_operation_in_progress,
+ &esw->mode_lock);
up_write(&esw->mode_lock);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index 928b20e73d37..1fb67fc6d49a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -948,6 +948,7 @@ void mlx5_esw_get(struct mlx5_core_dev *dev);
void mlx5_esw_put(struct mlx5_core_dev *dev);
int mlx5_esw_try_lock(struct mlx5_eswitch *esw);
int mlx5_esw_lock(struct mlx5_eswitch *esw);
+void mlx5_esw_lock_wait(struct mlx5_eswitch *esw);
void mlx5_esw_unlock(struct mlx5_eswitch *esw);
void esw_vport_change_handle_locked(struct mlx5_vport *vport);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
index 4939e10336c2..4da89fcf215b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
@@ -4441,7 +4441,7 @@ static bool mlx5_devlink_switchdev_active_mode_change(struct mlx5_eswitch *esw,
!esw->offloads_inactive)
return false;
- down_write(&esw->mode_lock);
+ mlx5_esw_lock_wait(esw);
esw->offloads_inactive = !esw->offloads_inactive;
esw->eswitch_operation_in_progress = true;
up_write(&esw->mode_lock);
@@ -4453,34 +4453,30 @@ static bool mlx5_devlink_switchdev_active_mode_change(struct mlx5_eswitch *esw,
down_write(&esw->mode_lock);
esw->eswitch_operation_in_progress = false;
- up_write(&esw->mode_lock);
+ mlx5_esw_unlock(esw);
return true;
}
-#define MLX5_ESW_HOLD_TIMEOUT_MS 7000
-#define MLX5_ESW_HOLD_RETRY_DELAY_MS 500
-
void mlx5_eswitch_safe_aux_devs_remove(struct mlx5_core_dev *dev)
{
- unsigned long timeout;
- bool hold_esw = true;
+ struct mlx5_eswitch *esw = dev->priv.eswitch;
- /* Wait for any concurrent eswitch mode transition to complete. */
- if (!mlx5_esw_hold(dev)) {
- timeout = jiffies + msecs_to_jiffies(MLX5_ESW_HOLD_TIMEOUT_MS);
- while (!mlx5_esw_hold(dev)) {
- if (!time_before(jiffies, timeout)) {
- hold_esw = false;
- break;
- }
- msleep(MLX5_ESW_HOLD_RETRY_DELAY_MS);
- }
- }
- if (hold_esw) {
- if (mlx5_eswitch_mode(dev) == MLX5_ESWITCH_OFFLOADS)
- mlx5_core_reps_aux_devs_remove(dev);
- mlx5_esw_release(dev);
- }
+ if (!mlx5_esw_allowed(esw))
+ return;
+
+ /* Wait for any concurrent eswitch mode transition to complete.
+ * Deadlock risk if ever called from within an operation in progress.
+ */
+ mlx5_esw_lock_wait(esw);
+ esw->eswitch_operation_in_progress = true;
+ up_write(&esw->mode_lock);
+
+ if (mlx5_eswitch_mode(dev) == MLX5_ESWITCH_OFFLOADS)
+ mlx5_core_reps_aux_devs_remove(dev);
+
+ down_write(&esw->mode_lock);
+ esw->eswitch_operation_in_progress = false;
+ mlx5_esw_unlock(esw);
}
int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,
@@ -4656,7 +4652,7 @@ int mlx5_devlink_eswitch_inline_mode_set(struct devlink *devlink, u8 mode,
if (IS_ERR(esw))
return PTR_ERR(esw);
- down_write(&esw->mode_lock);
+ mlx5_esw_lock_wait(esw);
switch (MLX5_CAP_ETH(dev, wqe_inline_mode)) {
case MLX5_CAP_INLINE_MODE_NOT_REQUIRED:
@@ -4694,7 +4690,7 @@ int mlx5_devlink_eswitch_inline_mode_set(struct devlink *devlink, u8 mode,
down_write(&esw->mode_lock);
esw->eswitch_operation_in_progress = false;
- up_write(&esw->mode_lock);
+ mlx5_esw_unlock(esw);
return 0;
out:
@@ -4764,7 +4760,7 @@ int mlx5_devlink_eswitch_encap_mode_set(struct devlink *devlink,
if (IS_ERR(esw))
return PTR_ERR(esw);
- down_write(&esw->mode_lock);
+ mlx5_esw_lock_wait(esw);
if (encap != DEVLINK_ESWITCH_ENCAP_MODE_NONE &&
(!MLX5_CAP_ESW_FLOWTABLE_FDB(dev, reformat) ||
@@ -4820,7 +4816,7 @@ int mlx5_devlink_eswitch_encap_mode_set(struct devlink *devlink,
esw->eswitch_operation_in_progress = false;
unlock:
- up_write(&esw->mode_lock);
+ mlx5_esw_unlock(esw);
return err;
}
--
2.44.0
next prev parent reply other threads:[~2026-11-05 21:17 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 21:16 [PATCH net V3 0/8] net/mlx5e: Fix offload lifetime and exclusion bugs Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 1/8] net/mlx5e: Release IPsec root table reference on RX setup failure Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 2/8] net/mlx5e: Serialize ipsec encap reservations with eswitch operations Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 3/8] net/mlx5e: Take HCA devcom lock before esw->mode_lock Tariq Toukan
2026-11-05 21:16 ` Tariq Toukan [this message]
2026-11-05 21:16 ` [PATCH net V3 5/8] net/mlx5e: Fix ipsec core device access during profile changes Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 6/8] net/mlx5e: Block E-Switch mode changes during IPsec policy creation Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 7/8] net/mlx5e: Block E-Switch mode changes during IPsec state creation Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 8/8] net/mlx5e: Serialize TC and IPsec offload exclusion counters Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261105211637.1083978-5-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=borisp@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=dtatulea@nvidia.com \
--cc=edumazet@kernel.org \
--cc=gal@nvidia.com \
--cc=horms@kernel.org \
--cc=jianbol@nvidia.com \
--cc=jmassar@nvidia.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mbloch@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=parri.andrea@gmail.com \
--cc=phaddad@nvidia.com \
--cc=prathameshdeshpande7@gmail.com \
--cc=raeds@nvidia.com \
--cc=roid@nvidia.com \
--cc=saeedm@nvidia.com \
--cc=shayd@nvidia.com \
--cc=steffen.klassert@secunet.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®