From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
Carolina Jubran <cjubran@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
Edward Srouji <edwards@nvidia.com>,
"Gal Pressman" <gal@nvidia.com>, Jason Gunthorpe <jgg@ziepe.ca>,
Kees Cook <kees@kernel.org>, Leon Romanovsky <leon@kernel.org>,
open list <linux-kernel@vger.kernel.org>,
<linux-pci@vger.kernel.org>, <linux-rdma@vger.kernel.org>,
Maher Sanalla <msanalla@nvidia.com>,
Mark Bloch <mbloch@nvidia.com>, Or Har-Toov <ohartoov@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>, Shay Drori <shayd@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>
Subject: [PATCH net V2 1/4] {RDMA,net}/mlx5: cache and use TX-enabled aggregate speed for vports
Date: Thu, 22 Oct 2026 22:30:50 +0300 [thread overview]
Message-ID: <20261022193053.3815866-2-tariqt@nvidia.com> (raw)
In-Reply-To: <20261022193053.3815866-1-tariqt@nvidia.com>
From: Or Har-Toov <ohartoov@nvidia.com>
Cache aggregate TX-enabled speed we calculated, and use it instead of
querying the bond speed so we can, in the following patch, limit it by
the PCI bandwidth.
The bond speed takes the calculation from PTYS, and PTYS is not bounded
by the PCI bandwidth of the NIC.
The max-speed fallback in mlx5_lag_set_vports_agg_speed() is local to
VF/SF vport rate-limit programming; agg_speed_mbps stores operational
speed only, and mlx5_lag_query_aggregated_speed() correctly returns zero
when all member links are down.
Until the aggregate has been measured, or after a failed measurement,
mlx5_lag_query_aggregated_speed() returns -ENODATA.
Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
drivers/infiniband/hw/mlx5/main.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/esw/qos.c | 1 +
.../net/ethernet/mellanox/mlx5/core/lag/lag.c | 86 +++++++++++++++----
.../net/ethernet/mellanox/mlx5/core/lag/lag.h | 16 +++-
include/linux/mlx5/driver.h | 2 +-
5 files changed, 84 insertions(+), 23 deletions(-)
v1:
https://lore.kernel.org/all/20260910102432.3845360-4-tariqt@nvidia.com/
Changes in v2:
- Count only TX-enabled, link-up ports from the LAG tracker for
non-MPESW modes; MPESW keeps the vport state check
- mlx5_lag_update_agg_speed() returns an error, and
mlx5_lag_set_vports_agg_speed() bails out on failure instead of
using the previous cached speed; the #else stub checks the return too
- Stop exporting mlx5_lag_query_bond_speed(), which has no users
outside mlx5_core
- Describe in the changelog what mlx5_lag_query_aggregated_speed()
returns when all member links are down
- Start agg_speed_mbps as SPEED_UNKNOWN and reset it to SPEED_UNKNOWN on
teardown; mlx5_lag_query_aggregated_speed() returns -ENODATA instead
of a zero speed until the first successful measurement
- A failed measurement invalidates the cached speed
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index a457647edacd..6594d38cf199 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1654,7 +1654,7 @@ static int mlx5_ib_query_port_speed_from_bond(struct mlx5_ib_dev *dev,
u32 bond_speed;
int err;
- err = mlx5_lag_query_bond_speed(mdev, &bond_speed);
+ err = mlx5_lag_query_aggregated_speed(mdev, &bond_speed);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
index 0d20f51b9702..0148306421ad 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
@@ -5,6 +5,7 @@
#include "lib/mlx5.h"
#include "esw/qos.h"
#include "en/port.h"
+#include "lag/lag.h"
#define CREATE_TRACE_POINTS
#include "diag/qos_tracepoint.h"
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 4e173b08cb37..57755ceab8a7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -360,6 +360,7 @@ static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev)
ldev->ports = MLX5_CAP_GEN(dev, num_lag_ports);
ldev->buckets = 1;
+ ldev->agg_speed_mbps = SPEED_UNKNOWN;
return ldev;
}
@@ -1431,10 +1432,10 @@ static bool mlx5_lag_should_disable_lag(struct mlx5_lag *ldev, bool do_bond)
ldev->mode != MLX5_LAG_MODE_MPESW;
}
-#ifdef CONFIG_MLX5_ESWITCH
static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
u32 *sum_speed)
{
+ bool mpesw = ldev->mode == MLX5_LAG_MODE_MPESW;
int pf_idx;
*sum_speed = 0;
@@ -1452,15 +1453,21 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
pf_mdev = pf->dev;
if (!pf_mdev)
continue;
- ret = mlx5_query_vport_max_tx_speed(pf_mdev, opmod, 0, 0,
- &speed, &state);
- if (ret) {
- mlx5_core_dbg(pf_mdev, "State query failed (err=%d)\n",
- ret);
- return ret;
- }
- if (state != VPORT_STATE_UP)
+ if (mpesw) {
+ ret = mlx5_query_vport_max_tx_speed(pf_mdev, opmod, 0,
+ 0, &speed, &state);
+ if (ret) {
+ mlx5_core_dbg(pf_mdev,
+ "State query failed (err=%d)\n",
+ ret);
+ return ret;
+ }
+ if (state != VPORT_STATE_UP)
+ continue;
+ } else if (!ldev->tracker.netdev_state[pf_idx].tx_enabled ||
+ !ldev->tracker.netdev_state[pf_idx].link_up) {
continue;
+ }
ret = mlx5_port_oper_linkspeed(pf_mdev, &speed);
if (ret) {
@@ -1476,6 +1483,31 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
return 0;
}
+int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
+{
+ u32 speed;
+ int err;
+
+ lockdep_assert_held(&ldev->lock);
+
+ err = mlx5_lag_get_devices_oper_speed(ldev, &speed);
+ if (err) {
+ WRITE_ONCE(ldev->agg_speed_mbps, SPEED_UNKNOWN);
+ return err;
+ }
+
+ WRITE_ONCE(ldev->agg_speed_mbps, speed);
+ return 0;
+}
+
+void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev)
+{
+ lockdep_assert_held(&ldev->lock);
+
+ WRITE_ONCE(ldev->agg_speed_mbps, SPEED_UNKNOWN);
+}
+
+#ifdef CONFIG_MLX5_ESWITCH
static int mlx5_lag_get_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed)
{
bool take_max;
@@ -1559,19 +1591,18 @@ void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev)
u32 speed;
int pf_idx;
- if (ldev->mode == MLX5_LAG_MODE_MPESW) {
- if (mlx5_lag_get_devices_oper_speed(ldev, &speed))
- return;
- } else {
- speed = ldev->tracker.bond_speed_mbps;
- if (speed == SPEED_UNKNOWN)
- return;
- }
+ if (mlx5_lag_update_agg_speed(ldev))
+ return;
+
+ speed = ldev->agg_speed_mbps;
- /* If speed is not set, fall back to the max achievable speed */
+ /* If speed is 0, fallback to the max speed */
if (!speed && mlx5_lag_get_devices_max_speed(ldev, &speed))
return;
+ if (!speed)
+ return;
+
speed = speed / MLX5_MAX_TX_SPEED_UNIT;
mlx5_ldev_for_each(pf_idx, 0, ldev) {
@@ -1594,6 +1625,7 @@ void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
int pf_idx;
int ret;
+ mlx5_lag_reset_agg_speed(ldev);
mlx5_ldev_for_each(pf_idx, 0, ldev) {
pf = mlx5_lag_pf(ldev, pf_idx);
if (!pf)
@@ -2089,6 +2121,23 @@ static int mlx5_handle_changeinfodata_event(struct mlx5_lag *ldev,
return 1;
}
+/* Returns speed in Mbps. */
+int mlx5_lag_query_aggregated_speed(struct mlx5_core_dev *mdev, u32 *speed)
+{
+ struct mlx5_lag *ldev;
+
+ ldev = mlx5_lag_dev(mdev);
+ if (!ldev)
+ return -ENODEV;
+
+ *speed = READ_ONCE(ldev->agg_speed_mbps);
+ if (*speed == SPEED_UNKNOWN)
+ return -ENODATA;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mlx5_lag_query_aggregated_speed);
+
static void mlx5_lag_update_tracker_speed(struct lag_tracker *tracker,
struct net_device *ndev)
{
@@ -2146,7 +2195,6 @@ int mlx5_lag_query_bond_speed(struct mlx5_core_dev *mdev, u32 *speed)
spin_unlock_irqrestore(&lag_lock, flags);
return ret;
}
-EXPORT_SYMBOL_GPL(mlx5_lag_query_bond_speed);
/* this handler is always registered to netdev events */
static int mlx5_lag_netdev_event(struct notifier_block *this,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
index e9f0ef83ce1d..45b0e8fd8f99 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
@@ -99,6 +99,7 @@ struct mlx5_lag {
/* Protect lag fields/state changes */
struct mutex lock;
struct lag_mpesw lag_mpesw;
+ u32 agg_speed_mbps;
};
static inline struct mlx5_lag *
@@ -258,12 +259,23 @@ void mlx5_lag_rescan_dev_locked(struct mlx5_lag *ldev,
void mlx5_lag_add_devices_filter(struct mlx5_lag *ldev, u32 filter);
struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev);
+int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev);
+void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev);
+int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
+
#ifdef CONFIG_MLX5_ESWITCH
void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev);
void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev);
#else
-static inline void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev) {}
-static inline void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev) {}
+static inline void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev)
+{
+ mlx5_lag_update_agg_speed(ldev);
+}
+
+static inline void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
+{
+ mlx5_lag_reset_agg_speed(ldev);
+}
#endif
static inline bool mlx5_lag_is_supported(struct mlx5_core_dev *dev)
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 83d0a83bbfbc..ff38086c6c56 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -1154,7 +1154,7 @@ int mlx5_cmd_destroy_vport_lag(struct mlx5_core_dev *dev);
bool mlx5_lag_is_roce(struct mlx5_core_dev *dev);
bool mlx5_lag_is_sriov(struct mlx5_core_dev *dev);
bool mlx5_lag_is_active(struct mlx5_core_dev *dev);
-int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
+int mlx5_lag_query_aggregated_speed(struct mlx5_core_dev *dev, u32 *speed);
bool mlx5_lag_mode_is_hash(struct mlx5_core_dev *dev);
bool mlx5_lag_is_master(struct mlx5_core_dev *dev);
bool mlx5_lag_is_shared_fdb(struct mlx5_core_dev *dev);
--
2.44.0
next prev parent reply other threads:[~2026-10-22 19:31 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-22 19:30 [PATCH net V2 0/4] net/mlx5: fixes for LAG max_tx_speed update flow Tariq Toukan
2026-10-22 19:30 ` Tariq Toukan [this message]
2026-10-22 19:30 ` [PATCH net V2 2/4] net/mlx5: Lag, limit aggregated TX speed by PCIe bandwidth Tariq Toukan
2026-10-22 19:30 ` [PATCH net V2 3/4] {RDMA,net}/mlx5: notify IB uplink on LAG aggregate speed change Tariq Toukan
2026-10-22 19:30 ` [PATCH net V2 4/4] net/mlx5: Lag, save and restore vport max_tx_speed across LAG Tariq Toukan
2026-10-22 19:37 ` [PATCH net V2 0/4] net/mlx5: fixes for LAG max_tx_speed update flow netdev-bot+sinfo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261022193053.3815866-2-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=bhelgaas@google.com \
--cc=cjubran@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=edwards@nvidia.com \
--cc=gal@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=kees@kernel.org \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mbloch@nvidia.com \
--cc=msanalla@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=ohartoov@nvidia.com \
--cc=pabeni@redhat.com \
--cc=saeedm@nvidia.com \
--cc=shayd@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®