From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Andrea Parri <parri.andrea@gmail.com>,
Boris Pismenny <borisp@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
Dragos Tatulea <dtatulea@nvidia.com>,
"Gal Pressman" <gal@nvidia.com>,
Jeroen Massar <jmassar@nvidia.com>,
Jianbo Liu <jianbol@nvidia.com>,
Kuniyuki Iwashima <kuniyu@google.com>,
Leon Romanovsky <leon@kernel.org>,
open list <linux-kernel@vger.kernel.org>,
<linux-rdma@vger.kernel.org>, Mark Bloch <mbloch@nvidia.com>,
"Patrisious Haddad" <phaddad@nvidia.com>,
Prathamesh Deshpande <prathameshdeshpande7@gmail.com>,
Raed Salem <raeds@nvidia.com>, Roi Dayan <roid@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>, Shay Drory <shayd@nvidia.com>,
Simon Horman <horms@kernel.org>,
Steffen Klassert <steffen.klassert@secunet.com>,
Tariq Toukan <tariqt@nvidia.com>
Subject: [PATCH net V3 2/8] net/mlx5e: Serialize ipsec encap reservations with eswitch operations
Date: Thu, 5 Nov 2026 23:16:30 +0200 [thread overview]
Message-ID: <20261105211637.1083978-3-tariqt@nvidia.com> (raw)
In-Reply-To: <20261105211637.1083978-1-tariqt@nvidia.com>
From: Cosmin Ratiu <cratiu@nvidia.com>
mlx5_eswitch_block_encap() takes mode_lock but ignores
eswitch_operation_in_progress. mlx5_devlink_eswitch_encap_mode_set()
claims the operation and drops mode_lock while rebuilding the FDB and
updating offloads.encap, so a reservation can read a stale or
rolled-back encap value. In diagram form:
T1: first IPsec SA (NIC table) T2: encap none -> basic
mlx5_devlink_eswitch_encap_mode_set()
down_write(&esw->mode_lock)
sees num_block_encap == 0
eswitch_operation_in_progress = true
up_write(&esw->mode_lock)
esw_destroy_offloads_fdb_tables()
mlx5e_xfrm_add_state()
mlx5e_ipsec_fs_tunnel_allowed()
mlx5_eswitch_block_encap()
down_write(&esw->mode_lock)
reads encap == NONE
num_block_encap++
up_write(&esw->mode_lock)
esw->offloads.encap = BASIC
esw_create_offloads_fdb_tables()
The race happens therefore because offloads.encap is not consistently
accessed under mode_lock. There are other variations of the above race
(e.g. in rollback, or in tx_create()).
Use mlx5_esw_lock() to reject a reservation while an operation is in
progress. This implies changing mlx5e_ipsec_fs_tunnel_allowed() to
propagate the error. Note that these are not new failure modes as it
seems on first glance, because if the eswitch_operation_in_progress flag
is set, rx_get()/tx_get()->mlx5_eswitch_block_mode() already fails with
-EBUSY. The new code simply protects against an operation initiated in
the short window after mlx5_eswitch_block_mode().
This race was found by Sashiko during inspection of another patch of
this series. No attempt has been made to trigger it, as it requires very
precise timing.
Fixes: baac8351f74c ("net/mlx5e: Reduce eswitch mode_lock protection context")
Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
Reviewed-by: Jianbo Liu <jianbol@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
.../mellanox/mlx5/core/en_accel/ipsec.c | 5 ++++-
.../mellanox/mlx5/core/en_accel/ipsec.h | 2 +-
.../mellanox/mlx5/core/en_accel/ipsec_fs.c | 21 ++++++++++++-------
.../net/ethernet/mellanox/mlx5/core/eswitch.h | 6 +++---
.../mellanox/mlx5/core/eswitch_offloads.c | 20 +++++++++++-------
5 files changed, 35 insertions(+), 19 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
index 7e991a09b3ff..7bc5d07b4321 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
@@ -812,7 +812,10 @@ static int mlx5e_xfrm_add_state(struct net_device *dev,
if (x->props.mode == XFRM_MODE_TUNNEL &&
x->xso.type == XFRM_DEV_OFFLOAD_PACKET) {
- allow_tunnel_mode = mlx5e_ipsec_fs_tunnel_allowed(sa_entry);
+ err = mlx5e_ipsec_fs_tunnel_allowed(sa_entry);
+ if (err < 0)
+ goto unblock_mode;
+ allow_tunnel_mode = err;
if (!allow_tunnel_mode) {
NL_SET_ERR_MSG_MOD(extack,
"Packet offload tunnel mode is disabled due to encap settings");
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h
index abcbd38db9db..f4b2d7b950c0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h
@@ -320,7 +320,7 @@ void mlx5e_accel_ipsec_fs_del_rule(struct mlx5e_ipsec_sa_entry *sa_entry);
int mlx5e_accel_ipsec_fs_add_pol(struct mlx5e_ipsec_pol_entry *pol_entry);
void mlx5e_accel_ipsec_fs_del_pol(struct mlx5e_ipsec_pol_entry *pol_entry);
void mlx5e_accel_ipsec_fs_modify(struct mlx5e_ipsec_sa_entry *sa_entry);
-bool mlx5e_ipsec_fs_tunnel_allowed(struct mlx5e_ipsec_sa_entry *sa_entry);
+int mlx5e_ipsec_fs_tunnel_allowed(struct mlx5e_ipsec_sa_entry *sa_entry);
int mlx5_ipsec_create_sa_ctx(struct mlx5e_ipsec_sa_entry *sa_entry);
void mlx5_ipsec_free_sa_ctx(struct mlx5e_ipsec_sa_entry *sa_entry);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c
index fd9fbf359333..1529d4730323 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c
@@ -1072,9 +1072,12 @@ static int rx_create(struct mlx5_core_dev *mdev, struct mlx5e_ipsec *ipsec,
goto err_fs_ft_sa_sel;
/* Create FT */
- if (mlx5_ipsec_device_caps(mdev) & MLX5_IPSEC_CAP_TUNNEL)
- rx->allow_tunnel_mode =
- mlx5_eswitch_block_encap(mdev, rx == ipsec->rx_esw);
+ if (mlx5_ipsec_device_caps(mdev) & MLX5_IPSEC_CAP_TUNNEL) {
+ err = mlx5_eswitch_block_encap(mdev, rx == ipsec->rx_esw);
+ if (err < 0)
+ goto err_fs_ft;
+ rx->allow_tunnel_mode = err;
+ }
if (rx->allow_tunnel_mode)
flags = MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
@@ -1315,9 +1318,12 @@ static int tx_create(struct mlx5e_ipsec *ipsec, struct mlx5e_ipsec_tx *tx,
if (err)
goto err_status_rule;
- if (mlx5_ipsec_device_caps(mdev) & MLX5_IPSEC_CAP_TUNNEL)
- tx->allow_tunnel_mode =
- mlx5_eswitch_block_encap(mdev, tx == ipsec->tx_esw);
+ if (mlx5_ipsec_device_caps(mdev) & MLX5_IPSEC_CAP_TUNNEL) {
+ err = mlx5_eswitch_block_encap(mdev, tx == ipsec->tx_esw);
+ if (err < 0)
+ goto err_encap;
+ tx->allow_tunnel_mode = err;
+ }
if (tx->allow_tunnel_mode)
flags = MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
@@ -1387,6 +1393,7 @@ static int tx_create(struct mlx5e_ipsec *ipsec, struct mlx5e_ipsec_tx *tx,
err_sa_ft:
if (tx->allow_tunnel_mode)
mlx5_eswitch_unblock_encap(mdev);
+err_encap:
mlx5_del_flow_rules(tx->status.rule);
err_status_rule:
mlx5_destroy_flow_table(tx->ft.status);
@@ -2854,7 +2861,7 @@ void mlx5e_accel_ipsec_fs_modify(struct mlx5e_ipsec_sa_entry *sa_entry)
memcpy(sa_entry, &sa_entry_shadow, sizeof(*sa_entry));
}
-bool mlx5e_ipsec_fs_tunnel_allowed(struct mlx5e_ipsec_sa_entry *sa_entry)
+int mlx5e_ipsec_fs_tunnel_allowed(struct mlx5e_ipsec_sa_entry *sa_entry)
{
struct mlx5e_ipsec *ipsec = sa_entry->ipsec;
struct xfrm_state *x = sa_entry->x;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index e9cbcd73b23f..928b20e73d37 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -967,7 +967,7 @@ void mlx5_eswitch_unload_reps(struct mlx5_eswitch *esw);
bool mlx5_eswitch_is_peer(struct mlx5_eswitch *esw,
struct mlx5_eswitch *peer_esw);
-bool mlx5_eswitch_block_encap(struct mlx5_core_dev *dev, bool from_fdb);
+int mlx5_eswitch_block_encap(struct mlx5_core_dev *dev, bool from_fdb);
void mlx5_eswitch_unblock_encap(struct mlx5_core_dev *dev);
int mlx5_eswitch_block_mode(struct mlx5_core_dev *dev);
@@ -1071,10 +1071,10 @@ mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)
static inline void
mlx5_eswitch_unload_reps(struct mlx5_eswitch *esw) {}
-static inline bool
+static inline int
mlx5_eswitch_block_encap(struct mlx5_core_dev *dev, bool from_fdb)
{
- return true;
+ return 1;
}
static inline void mlx5_eswitch_unblock_encap(struct mlx5_core_dev *dev)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
index c712848b202d..7a0131e26ff0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
@@ -4712,25 +4712,31 @@ int mlx5_devlink_eswitch_inline_mode_get(struct devlink *devlink, u8 *mode)
return esw_inline_mode_to_devlink(esw->offloads.inline_mode, mode);
}
-bool mlx5_eswitch_block_encap(struct mlx5_core_dev *dev, bool from_fdb)
+/* Return 1 if allowed, 0 if disallowed by the configuration, or -EBUSY
+ * if an eswitch operation is in progress.
+ */
+int mlx5_eswitch_block_encap(struct mlx5_core_dev *dev, bool from_fdb)
{
struct mlx5_eswitch *esw = dev->priv.eswitch;
enum devlink_eswitch_encap_mode encap;
- bool allow_tunnel = false;
+ int ret;
if (!mlx5_esw_allowed(esw))
- return true;
+ return 1;
+
+ ret = mlx5_esw_lock(esw);
+ if (ret)
+ return ret;
- down_write(&esw->mode_lock);
encap = esw->offloads.encap;
if (esw->mode == MLX5_ESWITCH_LEGACY ||
(encap == DEVLINK_ESWITCH_ENCAP_MODE_NONE && !from_fdb)) {
- allow_tunnel = true;
+ ret = 1;
esw->offloads.num_block_encap++;
}
- up_write(&esw->mode_lock);
+ mlx5_esw_unlock(esw);
- return allow_tunnel;
+ return ret;
}
void mlx5_eswitch_unblock_encap(struct mlx5_core_dev *dev)
--
2.44.0
next prev parent reply other threads:[~2026-11-05 21:17 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 21:16 [PATCH net V3 0/8] net/mlx5e: Fix offload lifetime and exclusion bugs Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 1/8] net/mlx5e: Release IPsec root table reference on RX setup failure Tariq Toukan
2026-11-05 21:16 ` Tariq Toukan [this message]
2026-11-05 21:16 ` [PATCH net V3 3/8] net/mlx5e: Take HCA devcom lock before esw->mode_lock Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 4/8] net/mlx5e: Fix E-Switch races in uplink unbind Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 5/8] net/mlx5e: Fix ipsec core device access during profile changes Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 6/8] net/mlx5e: Block E-Switch mode changes during IPsec policy creation Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 7/8] net/mlx5e: Block E-Switch mode changes during IPsec state creation Tariq Toukan
2026-11-05 21:16 ` [PATCH net V3 8/8] net/mlx5e: Serialize TC and IPsec offload exclusion counters Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261105211637.1083978-3-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=borisp@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=dtatulea@nvidia.com \
--cc=edumazet@kernel.org \
--cc=gal@nvidia.com \
--cc=horms@kernel.org \
--cc=jianbol@nvidia.com \
--cc=jmassar@nvidia.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mbloch@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=parri.andrea@gmail.com \
--cc=phaddad@nvidia.com \
--cc=prathameshdeshpande7@gmail.com \
--cc=raeds@nvidia.com \
--cc=roid@nvidia.com \
--cc=saeedm@nvidia.com \
--cc=shayd@nvidia.com \
--cc=steffen.klassert@secunet.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®