[PATCH net V2 0/4] net/mlx5: fixes for LAG max_tx_speed update flow

[PATCH net V2 0/4] net/mlx5: fixes for LAG max_tx_speed update flow

[Tariq Toukan]

Hi,

Commit 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to
vports") added a mechanism that computes the aggregated LAG TX speed and
programs it into each vport's max_tx_speed via MODIFY_VPORT_STATE. This
series fixes bugs found in that flow.

The bugs:

1. The aggregate was not limited by the NIC's PCIe bandwidth. It was
   taken from the bond speed, which is calculated from PTYS, and PTYS
   is not bounded by the PCI bandwidth of the NIC. FW initializes
   max_tx_speed to the minimum of the port speed and the PCIe
   bandwidth, so modifying it with a value computed by different rules
   changes the meaning of the field rather than updating it.

2. RoCE LAG never told the RDMA layer about speed changes. The NIC is
   considered to be in a bare metal configuration, without a valid or
   enabled eswitch, so there are no VF, SF or PF rep vports to modify
   and no event of speed change was raised.

3. LAG teardown did not reset the vport speeds correctly. It wrote back
   the oper speed of the link, which reads 0 for a down link, and FW
   ignores max_tx_speed=0 as "not provided", so the aggregate stayed on
   the vports. With the link up, it replaced the max_tx_speed of a
   vport with the speed of the link instead of the value the vport had
   before the LAG, for example when FW limits it by the PCIe bandwidth.

Patch #1 caches the TX-enabled aggregate speed and programs that into
        the vports, instead of reading it back from the bond netdev.

Patch #2 limits the aggregated TX speed by the NIC's PCIe bandwidth.

Patch #3 notifies the RDMA layer of LAG aggregate speed changes via a
        new driver event.

Patch #4 saves the max_tx_speed of each vport when it is enabled and
        restores it at LAG teardown, with a link-derived fallback when
        no value was saved or the link is down. It fixes bug 3.

Regards,
Tariq

V2:
- Patch "cache and use TX-enabled aggregate speed for vports": only
  TX-enabled, link-up ports from the LAG tracker are counted in
  non-MPESW modes, which was part of the split patch in v1 and moved
  here, mlx5_lag_update_agg_speed() now returns int and
  propagates query errors, the cached speed starts as unknown and a
  failed measurement invalidates it, so that
  mlx5_lag_query_aggregated_speed() returns -ENODATA instead of a zero
  speed.
- Patch "limit aggregated TX speed by PCIe bandwidth": the aggregate is
  clamped as a whole rather than clamping each port's speed to its own
  PCIe bandwidth before summing, which could let the total reach a
  multiple of a shared link's real bandwidth. Only ports that
  contribute to the aggregate are counted. In Socket Direct the
  bandwidths of the mdevs of a port are summed, and a bandwidth that
  cannot be read fails the calculation instead of skipping the clamp.
- Patch "notify IB uplink on LAG aggregate speed change": the event is
  raised whenever the cached aggregate changes, in any LAG mode, and
  the commit message describes the gap it closes.
- New patch: "Lag, save and restore vport max_tx_speed across LAG".

V1:
https://lore.kernel.org/all/20260910102432.3845360-1-tariqt@nvidia.com/

Or Har-Toov (4):
  {RDMA,net}/mlx5: cache and use TX-enabled aggregate speed for vports
  net/mlx5: Lag, limit aggregated TX speed by PCIe bandwidth
  {RDMA,net}/mlx5: notify IB uplink on LAG aggregate speed change
  net/mlx5: Lag, save and restore vport max_tx_speed across LAG

 drivers/infiniband/hw/mlx5/main.c             |  12 +-
 .../net/ethernet/mellanox/mlx5/core/esw/qos.c |   1 +
 .../net/ethernet/mellanox/mlx5/core/eswitch.c |  22 ++
 .../net/ethernet/mellanox/mlx5/core/eswitch.h |   1 +
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 251 +++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h |  17 +-
 drivers/pci/pci.c                             |  16 ++
 include/linux/mlx5/device.h                   |   1 +
 include/linux/mlx5/driver.h                   |   2 +-
 include/linux/pci.h                           |   2 +
 10 files changed, 291 insertions(+), 34 deletions(-)


base-commit: f4377bf85887f57b59abb5479f7660a81ffe4449
-- 
2.44.0

[PATCH net V2 1/4] {RDMA,net}/mlx5: cache and use TX-enabled aggregate speed for vports

[Tariq Toukan]

From: Or Har-Toov <ohartoov@nvidia.com>

Cache aggregate TX-enabled speed we calculated, and use it instead of
querying the bond speed so we can, in the following patch, limit it by
the PCI bandwidth.

The bond speed takes the calculation from PTYS, and PTYS is not bounded
by the PCI bandwidth of the NIC.

The max-speed fallback in mlx5_lag_set_vports_agg_speed() is local to
VF/SF vport rate-limit programming; agg_speed_mbps stores operational
speed only, and mlx5_lag_query_aggregated_speed() correctly returns zero
when all member links are down.
Until the aggregate has been measured, or after a failed measurement,
mlx5_lag_query_aggregated_speed() returns -ENODATA.

Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c             |  2 +-
 .../net/ethernet/mellanox/mlx5/core/esw/qos.c |  1 +
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 86 +++++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 16 +++-
 include/linux/mlx5/driver.h                   |  2 +-
 5 files changed, 84 insertions(+), 23 deletions(-)

v1:
https://lore.kernel.org/all/20260910102432.3845360-4-tariqt@nvidia.com/

Changes in v2:
- Count only TX-enabled, link-up ports from the LAG tracker for
  non-MPESW modes; MPESW keeps the vport state check
- mlx5_lag_update_agg_speed() returns an error, and
  mlx5_lag_set_vports_agg_speed() bails out on failure instead of
  using the previous cached speed; the #else stub checks the return too
- Stop exporting mlx5_lag_query_bond_speed(), which has no users
  outside mlx5_core
- Describe in the changelog what mlx5_lag_query_aggregated_speed()
  returns when all member links are down
- Start agg_speed_mbps as SPEED_UNKNOWN and reset it to SPEED_UNKNOWN on
  teardown; mlx5_lag_query_aggregated_speed() returns -ENODATA instead
  of a zero speed until the first successful measurement
- A failed measurement invalidates the cached speed

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index a457647edacd..6594d38cf199 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1654,7 +1654,7 @@ static int mlx5_ib_query_port_speed_from_bond(struct mlx5_ib_dev *dev,
 	u32 bond_speed;
 	int err;
 
-	err = mlx5_lag_query_bond_speed(mdev, &bond_speed);
+	err = mlx5_lag_query_aggregated_speed(mdev, &bond_speed);
 	if (err)
 		return err;
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
index 0d20f51b9702..0148306421ad 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c
@@ -5,6 +5,7 @@
 #include "lib/mlx5.h"
 #include "esw/qos.h"
 #include "en/port.h"
+#include "lag/lag.h"
 #define CREATE_TRACE_POINTS
 #include "diag/qos_tracepoint.h"
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 4e173b08cb37..57755ceab8a7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -360,6 +360,7 @@ static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev)
 
 	ldev->ports = MLX5_CAP_GEN(dev, num_lag_ports);
 	ldev->buckets = 1;
+	ldev->agg_speed_mbps = SPEED_UNKNOWN;
 
 	return ldev;
 }
@@ -1431,10 +1432,10 @@ static bool mlx5_lag_should_disable_lag(struct mlx5_lag *ldev, bool do_bond)
 	       ldev->mode != MLX5_LAG_MODE_MPESW;
 }
 
-#ifdef CONFIG_MLX5_ESWITCH
 static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 					   u32 *sum_speed)
 {
+	bool mpesw = ldev->mode == MLX5_LAG_MODE_MPESW;
 	int pf_idx;
 
 	*sum_speed = 0;
@@ -1452,15 +1453,21 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 		pf_mdev = pf->dev;
 		if (!pf_mdev)
 			continue;
-		ret = mlx5_query_vport_max_tx_speed(pf_mdev, opmod, 0, 0,
-						    &speed, &state);
-		if (ret) {
-			mlx5_core_dbg(pf_mdev, "State query failed (err=%d)\n",
-				      ret);
-			return ret;
-		}
-		if (state != VPORT_STATE_UP)
+		if (mpesw) {
+			ret = mlx5_query_vport_max_tx_speed(pf_mdev, opmod, 0,
+							    0, &speed, &state);
+			if (ret) {
+				mlx5_core_dbg(pf_mdev,
+					      "State query failed (err=%d)\n",
+					      ret);
+				return ret;
+			}
+			if (state != VPORT_STATE_UP)
+				continue;
+		} else if (!ldev->tracker.netdev_state[pf_idx].tx_enabled ||
+			   !ldev->tracker.netdev_state[pf_idx].link_up) {
 			continue;
+		}
 
 		ret = mlx5_port_oper_linkspeed(pf_mdev, &speed);
 		if (ret) {
@@ -1476,6 +1483,31 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 	return 0;
 }
 
+int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
+{
+	u32 speed;
+	int err;
+
+	lockdep_assert_held(&ldev->lock);
+
+	err = mlx5_lag_get_devices_oper_speed(ldev, &speed);
+	if (err) {
+		WRITE_ONCE(ldev->agg_speed_mbps, SPEED_UNKNOWN);
+		return err;
+	}
+
+	WRITE_ONCE(ldev->agg_speed_mbps, speed);
+	return 0;
+}
+
+void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev)
+{
+	lockdep_assert_held(&ldev->lock);
+
+	WRITE_ONCE(ldev->agg_speed_mbps, SPEED_UNKNOWN);
+}
+
+#ifdef CONFIG_MLX5_ESWITCH
 static int mlx5_lag_get_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed)
 {
 	bool take_max;
@@ -1559,19 +1591,18 @@ void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev)
 	u32 speed;
 	int pf_idx;
 
-	if (ldev->mode == MLX5_LAG_MODE_MPESW) {
-		if (mlx5_lag_get_devices_oper_speed(ldev, &speed))
-			return;
-	} else {
-		speed = ldev->tracker.bond_speed_mbps;
-		if (speed == SPEED_UNKNOWN)
-			return;
-	}
+	if (mlx5_lag_update_agg_speed(ldev))
+		return;
+
+	speed = ldev->agg_speed_mbps;
 
-	/* If speed is not set, fall back to the max achievable speed */
+	/* If speed is 0, fallback to the max speed */
 	if (!speed && mlx5_lag_get_devices_max_speed(ldev, &speed))
 		return;
 
+	if (!speed)
+		return;
+
 	speed = speed / MLX5_MAX_TX_SPEED_UNIT;
 
 	mlx5_ldev_for_each(pf_idx, 0, ldev) {
@@ -1594,6 +1625,7 @@ void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
 	int pf_idx;
 	int ret;
 
+	mlx5_lag_reset_agg_speed(ldev);
 	mlx5_ldev_for_each(pf_idx, 0, ldev) {
 		pf = mlx5_lag_pf(ldev, pf_idx);
 		if (!pf)
@@ -2089,6 +2121,23 @@ static int mlx5_handle_changeinfodata_event(struct mlx5_lag *ldev,
 	return 1;
 }
 
+/* Returns speed in Mbps. */
+int mlx5_lag_query_aggregated_speed(struct mlx5_core_dev *mdev, u32 *speed)
+{
+	struct mlx5_lag *ldev;
+
+	ldev = mlx5_lag_dev(mdev);
+	if (!ldev)
+		return -ENODEV;
+
+	*speed = READ_ONCE(ldev->agg_speed_mbps);
+	if (*speed == SPEED_UNKNOWN)
+		return -ENODATA;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(mlx5_lag_query_aggregated_speed);
+
 static void mlx5_lag_update_tracker_speed(struct lag_tracker *tracker,
 					  struct net_device *ndev)
 {
@@ -2146,7 +2195,6 @@ int mlx5_lag_query_bond_speed(struct mlx5_core_dev *mdev, u32 *speed)
 	spin_unlock_irqrestore(&lag_lock, flags);
 	return ret;
 }
-EXPORT_SYMBOL_GPL(mlx5_lag_query_bond_speed);
 
 /* this handler is always registered to netdev events */
 static int mlx5_lag_netdev_event(struct notifier_block *this,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
index e9f0ef83ce1d..45b0e8fd8f99 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
@@ -99,6 +99,7 @@ struct mlx5_lag {
 	/* Protect lag fields/state changes */
 	struct mutex		  lock;
 	struct lag_mpesw	  lag_mpesw;
+	u32			  agg_speed_mbps;
 };
 
 static inline struct mlx5_lag *
@@ -258,12 +259,23 @@ void mlx5_lag_rescan_dev_locked(struct mlx5_lag *ldev,
 void mlx5_lag_add_devices_filter(struct mlx5_lag *ldev, u32 filter);
 struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev);
 
+int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev);
+void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev);
+int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
+
 #ifdef CONFIG_MLX5_ESWITCH
 void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev);
 void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev);
 #else
-static inline void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev) {}
-static inline void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev) {}
+static inline void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev)
+{
+	mlx5_lag_update_agg_speed(ldev);
+}
+
+static inline void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
+{
+	mlx5_lag_reset_agg_speed(ldev);
+}
 #endif
 
 static inline bool mlx5_lag_is_supported(struct mlx5_core_dev *dev)
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 83d0a83bbfbc..ff38086c6c56 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -1154,7 +1154,7 @@ int mlx5_cmd_destroy_vport_lag(struct mlx5_core_dev *dev);
 bool mlx5_lag_is_roce(struct mlx5_core_dev *dev);
 bool mlx5_lag_is_sriov(struct mlx5_core_dev *dev);
 bool mlx5_lag_is_active(struct mlx5_core_dev *dev);
-int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
+int mlx5_lag_query_aggregated_speed(struct mlx5_core_dev *dev, u32 *speed);
 bool mlx5_lag_mode_is_hash(struct mlx5_core_dev *dev);
 bool mlx5_lag_is_master(struct mlx5_core_dev *dev);
 bool mlx5_lag_is_shared_fdb(struct mlx5_core_dev *dev);
-- 
2.44.0

[PATCH net V2 2/4] net/mlx5: Lag, limit aggregated TX speed by PCIe bandwidth

[Tariq Toukan]

From: Or Har-Toov <ohartoov@nvidia.com>

FW initializes max_tx_speed to the minimum of the port speed and the
PCI bandwidth. Modifying it with a value computed by different rules
changes the meaning of the field rather than updating it.

Limit the aggregated LAG TX speed by the PCIe bandwidth. Clamping each
uplink separately and then summing could let the aggregate reach a
multiple of the link's real bandwidth, so clamp the sum once instead,
counting only the ports that contribute to it.

This assumes the bonded PFs share a single PCIe link, which holds when
they are functions of the same device. In Socket Direct, each port spans
multiple mdevs on separate PCIe links, so the PCIe budget for that port
is the sum of its mdevs' PCIe bandwidths. Where Socket Direct ports do
not have equal PCIe budgets, the aggregate is clamped to the narrowest
port, which under-reports rather than over-reports the available
bandwidth.

pcie_bandwidth_mbps() reads the endpoint link only, without traversing
upstream bridges, and uses the nominal per-lane transfer rate times the
link width, without encoding overhead. FW initializes max_tx_speed from
the same endpoint-only link status, so using the same calculation keeps
the clamp consistent with the value FW programs before LAG. Ethernet
speeds and max_tx_speed are likewise nominal rates, so the comparison is
like for like. A more accurate bandwidth estimate would make the
aggregate differ from the per-vport value FW sets up for the same link.

If the PCIe bandwidth of a contributing port cannot be read, the speed
calculation fails instead of returning an unclamped or partial result.
A device that has no Link Status register, an RC integrated endpoint,
has no PCIe budget to apply and is not clamped.

Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 79 ++++++++++++++++++-
 drivers/pci/pci.c                             | 16 ++++
 include/linux/pci.h                           |  2 +
 3 files changed, 95 insertions(+), 2 deletions(-)

v1: https://lore.kernel.org/all/20260910102432.3845360-5-tariqt@nvidia.com/

Changes in v2:
- Clamp the aggregate sum once by the narrowest PCIe link instead of
  clamping each uplink before summing, which could let the aggregate
  exceed the real bandwidth
- Document the shared PCIe link assumption in the changelog
- Add pcie_bandwidth_mbps() to drivers/pci/pci.c, which reads
  PCI_EXP_LNKSTA once and decodes both speed and width from that
  value, avoiding a race between separate reads of speed and width
- In Socket Direct, sum the PCIe bandwidths of all mdevs in the same
  port group instead of taking the minimum, since those links are
  independent
- Fail the aggregate and max speed calculations when a PCIe bandwidth
  cannot be read, instead of skipping the clamp; an RC integrated
  endpoint has no Link Status and is not clamped

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 57755ceab8a7..05e6d0aa31fc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -1432,10 +1432,55 @@ static bool mlx5_lag_should_disable_lag(struct mlx5_lag *ldev, bool do_bond)
 	       ldev->mode != MLX5_LAG_MODE_MPESW;
 }
 
+static int mlx5_lag_dev_pcie_bw(struct mlx5_core_dev *dev, u32 *bw)
+{
+	if (pci_pcie_type(dev->pdev) == PCI_EXP_TYPE_RC_END) {
+		*bw = U32_MAX;
+		return 0;
+	}
+
+	*bw = pcie_bandwidth_mbps(dev->pdev);
+	return *bw ? 0 : -EIO;
+}
+
+static int mlx5_lag_port_pcie_bw(struct mlx5_lag *ldev, struct lag_func *port,
+				 u32 *bw)
+{
+	struct lag_func *member;
+	bool unlimited = false;
+	u32 member_bw;
+	int idx;
+	int err;
+
+	if (!port->group_id)
+		return mlx5_lag_dev_pcie_bw(port->dev, bw);
+
+	*bw = 0;
+	mlx5_lag_for_each(idx, 0, ldev, port->group_id) {
+		member = mlx5_lag_pf(ldev, idx);
+		if (!member)
+			continue;
+
+		err = mlx5_lag_dev_pcie_bw(member->dev, &member_bw);
+		if (err)
+			return err;
+
+		if (member_bw == U32_MAX)
+			unlimited = true;
+		else
+			*bw += member_bw;
+	}
+
+	if (unlimited)
+		*bw = U32_MAX;
+	return 0;
+}
+
 static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 					   u32 *sum_speed)
 {
 	bool mpesw = ldev->mode == MLX5_LAG_MODE_MPESW;
+	u32 pci_bw = U32_MAX;
 	int pf_idx;
 
 	*sum_speed = 0;
@@ -1443,6 +1488,7 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 		u8 opmod = MLX5_VPORT_STATE_OP_MOD_VNIC_VPORT;
 		struct mlx5_core_dev *pf_mdev;
 		struct lag_func *pf;
+		u32 dev_pci_bw;
 		u32 speed;
 		u8 state;
 		int ret;
@@ -1477,9 +1523,20 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 			return ret;
 		}
 
+		ret = mlx5_lag_port_pcie_bw(ldev, pf, &dev_pci_bw);
+		if (ret) {
+			mlx5_core_dbg(pf_mdev,
+				      "Failed to get PCIe bandwidth (err=%d)\n",
+				      ret);
+			return ret;
+		}
+
+		pci_bw = min(pci_bw, dev_pci_bw);
 		*sum_speed += speed;
 	}
 
+	*sum_speed = min(*sum_speed, pci_bw);
+
 	return 0;
 }
 
@@ -1510,6 +1567,7 @@ void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev)
 #ifdef CONFIG_MLX5_ESWITCH
 static int mlx5_lag_get_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed)
 {
+	u32 pci_bw = U32_MAX;
 	bool take_max;
 	int pf_idx;
 
@@ -1521,6 +1579,7 @@ static int mlx5_lag_get_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed)
 	mlx5_ldev_for_each(pf_idx, 0, ldev) {
 		struct mlx5_core_dev *pf_mdev;
 		struct lag_func *pf;
+		u32 dev_pci_bw;
 		u32 speed;
 		int ret;
 
@@ -1539,10 +1598,26 @@ static int mlx5_lag_get_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed)
 			return ret;
 		}
 
-		*max_speed = take_max ?
-			max(*max_speed, speed) : *max_speed + speed;
+		ret = mlx5_lag_port_pcie_bw(ldev, pf, &dev_pci_bw);
+		if (ret) {
+			mlx5_core_dbg(pf_mdev,
+				      "Failed to get PCIe bandwidth (err=%d)\n",
+				      ret);
+			return ret;
+		}
+
+		if (take_max) {
+			speed = min(speed, dev_pci_bw);
+			*max_speed = max(*max_speed, speed);
+		} else {
+			pci_bw = min(pci_bw, dev_pci_bw);
+			*max_speed += speed;
+		}
 	}
 
+	if (!take_max)
+		*max_speed = min(*max_speed, pci_bw);
+
 	return 0;
 }
 
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8cc6a89130b5..064eef237d6f 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -6006,6 +6006,22 @@ int pcie_link_speed_mbps(struct pci_dev *pdev)
 }
 EXPORT_SYMBOL(pcie_link_speed_mbps);
 
+u32 pcie_bandwidth_mbps(struct pci_dev *pdev)
+{
+	u16 lnksta;
+	u32 width;
+	int speed;
+
+	if (pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta))
+		return 0;
+	speed = pcie_dev_speed_mbps(to_pcie_link_speed(lnksta));
+	if (speed <= 0)
+		return 0;
+	width = FIELD_GET(PCI_EXP_LNKSTA_NLW, lnksta);
+	return speed * width;
+}
+EXPORT_SYMBOL_GPL(pcie_bandwidth_mbps);
+
 /**
  * pcie_bandwidth_available - determine minimum link settings of a PCIe
  *			      device and its bandwidth limitation
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 5ffcf5ce8c96..202bc8680601 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1479,6 +1479,8 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 			     enum pci_bus_speed *speed,
 			     enum pcie_link_width *width);
 int pcie_link_speed_mbps(struct pci_dev *pdev);
+u32 pcie_bandwidth_mbps(struct pci_dev *pdev);
+
 void pcie_print_link_status(struct pci_dev *dev);
 int pcie_reset_flr(struct pci_dev *dev, bool probe);
 int pcie_flr(struct pci_dev *dev);
-- 
2.44.0

[PATCH net V2 3/4] {RDMA,net}/mlx5: notify IB uplink on LAG aggregate speed change

[Tariq Toukan]

From: Or Har-Toov <ohartoov@nvidia.com>

In switchdev LAG, a link-state change fires two
IB_EVENT_DEVICE_SPEED_CHANGE events: the physical port fires immediately
with the port state change, and once the LAG reconverges and pushes the
new aggregate speed to every vport via MODIFY_VPORT_STATE, FW fires a
PORT_CHANGE event per vport, which is how VF consumers learn the
aggregate actually changed.

The uplink vport is deliberately skipped when pushing the aggregate
speed, so the IB uplink device never gets a second event once the
aggregate has settled. In RoCE LAG there are no VF vports at all, so
nothing fires after reconvergence.

Raise IB_EVENT_DEVICE_SPEED_CHANGE from the LAG layer, in
mlx5_lag_update_agg_speed(), right after the cached aggregate is
updated. That cache is where the IB layer now takes the effective speed
from, so a consumer that queries after receiving the event is
guaranteed to read the new value.

blocking_notifier_call_chain() holds the blocking_notifier_head rwsem
for the full duration of lag_event();
blocking_notifier_chain_unregister() in mlx5_eth_lag_cleanup() takes the
write side and cannot return until any in-flight callback has completed,
so no event can be dispatched to a freed device after unregistration.

Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c             | 10 ++++++++++
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 20 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h |  1 +
 include/linux/mlx5/device.h                   |  1 +
 4 files changed, 32 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 6594d38cf199..cfa7c7ba2b73 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3690,6 +3690,16 @@ static int lag_event(struct notifier_block *nb, unsigned long event, void *data)
 			rdma_roce_rescan_port(ibdev, portnum + 1);
 		}
 		break;
+	case MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE: {
+		struct ib_event speed_event = {};
+
+		if (!dev->ib_active)
+			break;
+		speed_event.device = ibdev;
+		speed_event.event = IB_EVENT_DEVICE_SPEED_CHANGE;
+		ib_dispatch_event(&speed_event);
+		break;
+	}
 	default:
 		return NOTIFY_DONE;
 	}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 05e6d0aa31fc..afe0b9909f2d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -1540,8 +1540,24 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 	return 0;
 }
 
+void mlx5_lag_notify_speed_change(struct mlx5_lag *ldev)
+{
+	struct lag_func *pf;
+	int idx;
+
+	idx = mlx5_lag_get_dev_index_by_seq(ldev, MLX5_LAG_P1);
+	if (idx < 0)
+		return;
+	pf = mlx5_lag_pf(ldev, idx);
+	if (!pf)
+		return;
+	blocking_notifier_call_chain(&pf->dev->priv.lag_nh,
+				     MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE, NULL);
+}
+
 int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
 {
+	u32 old_speed;
 	u32 speed;
 	int err;
 
@@ -1553,7 +1569,11 @@ int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
 		return err;
 	}
 
+	old_speed = READ_ONCE(ldev->agg_speed_mbps);
 	WRITE_ONCE(ldev->agg_speed_mbps, speed);
+	if (speed != old_speed)
+		mlx5_lag_notify_speed_change(ldev);
+
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
index 45b0e8fd8f99..420655468d78 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
@@ -259,6 +259,7 @@ void mlx5_lag_rescan_dev_locked(struct mlx5_lag *ldev,
 void mlx5_lag_add_devices_filter(struct mlx5_lag *ldev, u32 filter);
 struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev);
 
+void mlx5_lag_notify_speed_change(struct mlx5_lag *ldev);
 int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev);
 void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev);
 int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 8cb321a9fb3d..1ec2331c24ee 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -374,6 +374,7 @@ enum mlx5_driver_event {
 	MLX5_DRIVER_EVENT_AFFILIATION_DONE,
 	MLX5_DRIVER_EVENT_AFFILIATION_REMOVED,
 	MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE,
+	MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE,
 };
 
 enum {
-- 
2.44.0

[PATCH net V2 4/4] net/mlx5: Lag, save and restore vport max_tx_speed across LAG

[Tariq Toukan]

From: Or Har-Toov <ohartoov@nvidia.com>

mlx5_lag_reset_vports_speed() queries the oper link speed at LAG
teardown and writes it back to each vport. When the link is down at
that moment, mlx5_port_oper_linkspeed() returns 0, which FW treats as
"not provided" and ignores, leaving the aggregate rate limit on the
vports after teardown.

Save each vport's max_tx_speed when the vport is enabled, before the
LAG aggregate or a deferred aggregate replay can overwrite it. While a
LAG is active, only a vport with a pending aggregate is saved, as FW
may already hold the aggregate for any other vport. Restore the saved
value per vport at teardown instead of querying the oper speed, and
update agg_max_tx_speed to the pre-LAG value so disabled vports replay
the correct rate when they are later enabled. A vport keeps its
saved value after teardown, as it is the baseline for the next LAG, and
for a disabled vport it is also the pending restore value.

If the max_tx_speed of a vport could not be saved, it stays zero and
mlx5_lag_get_restore_speed() falls back to a link-derived speed at
teardown. The aggregate is still applied to such a vport.

This also covers the case where the oper speed query fails and the PF
is skipped entirely, leaving the aggregate in FW and in
agg_max_tx_speed, as raised in the AI review of v1 of "net/mlx5: Lag,
reset vport speed on teardown". It is unrelated to that patch, so it is
fixed here separately.

Firmware never returns zero as a configured max_tx_speed; zero means
"not supported", so using it as a sentinel for "not yet saved" is safe.

A vport disabled for the entire LAG period was never assigned a
per-vport rate limit, so no original speed exists to restore; deferring
through agg_max_tx_speed on re-enable is correct.

Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Link: https://lore.kernel.org/all/20260915015118.875210-1-kuba@kernel.org/
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/eswitch.c | 22 ++++++
 .../net/ethernet/mellanox/mlx5/core/eswitch.h |  1 +
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 68 ++++++++++++++++---
 3 files changed, 81 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index fc197d1dc9df..7cf9c15ec65d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -935,6 +935,27 @@ static void esw_vport_cleanup(struct mlx5_eswitch *esw, struct mlx5_vport *vport
 	esw_vport_cleanup_acl(esw, vport);
 }
 
+static void mlx5_esw_vport_save_max_tx_speed(struct mlx5_eswitch *esw,
+					     struct mlx5_vport *vport)
+{
+	u16 op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT;
+	int ret;
+
+	if (!MLX5_CAP_ESW(esw->dev, esw_vport_state_max_tx_speed) ||
+	    vport->vport == MLX5_VPORT_UPLINK || vport->max_tx_speed)
+		return;
+
+	if (mlx5_lag_is_active(esw->dev) && !vport->agg_max_tx_speed)
+		return;
+
+	ret = mlx5_query_vport_max_tx_speed(esw->dev, op_mod, vport->vport,
+					    true, &vport->max_tx_speed, NULL);
+	if (ret)
+		mlx5_core_dbg(esw->dev,
+			      "Failed to save vport %d max tx speed, err=%d\n",
+			      vport->vport, ret);
+}
+
 static void mlx5_esw_vport_set_max_tx_speed(struct mlx5_eswitch *esw,
 					    struct mlx5_vport *vport)
 {
@@ -999,6 +1020,7 @@ int mlx5_esw_vport_enable(struct mlx5_eswitch *esw, struct mlx5_vport *vport,
 	esw->enabled_vports++;
 	esw_debug(esw->dev, "Enabled VPORT(%d)\n", vport_num);
 
+	mlx5_esw_vport_save_max_tx_speed(esw, vport);
 	if (vport->agg_max_tx_speed)
 		mlx5_esw_vport_set_max_tx_speed(esw, vport);
 done:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index 8b1f93b13ea9..3717fb6de0b6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -261,6 +261,7 @@ struct mlx5_vport {
 	int index;
 	struct mlx5_devlink_port *dl_port;
 	u32 agg_max_tx_speed;
+	u32 max_tx_speed;
 };
 
 struct mlx5_esw_indir_table;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index afe0b9909f2d..3e623c96b2a1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -1712,11 +1712,37 @@ void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev)
 	}
 }
 
+static int mlx5_lag_get_restore_speed(struct mlx5_core_dev *mdev,
+				      struct mlx5_vport *vport, u32 *speed)
+{
+	u32 pci_bw;
+
+	if (vport->max_tx_speed) {
+		*speed = vport->max_tx_speed;
+		return 0;
+	}
+
+	*speed = 0;
+	mlx5_port_oper_linkspeed(mdev, speed);
+	if (!*speed && mlx5_port_max_linkspeed(mdev, speed))
+		return -ENODATA;
+
+	pci_bw = pcie_bandwidth_mbps(mdev->pdev);
+	if (pci_bw)
+		*speed = min(*speed, pci_bw);
+	*speed /= MLX5_MAX_TX_SPEED_UNIT;
+	return 0;
+}
+
 void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
 {
+	u16 op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT;
 	struct mlx5_core_dev *mdev;
+	struct mlx5_eswitch *esw;
+	struct mlx5_vport *vport;
 	struct lag_func *pf;
-	u32 speed;
+	u32 restore_speed;
+	unsigned long i;
 	int pf_idx;
 	int ret;
 
@@ -1728,17 +1754,39 @@ void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev)
 		mdev = pf->dev;
 		if (!mdev)
 			continue;
-
-		ret = mlx5_port_oper_linkspeed(mdev, &speed);
-		if (ret) {
-			mlx5_core_dbg(mdev,
-				      "Failed to reset vports speed for device %s. Oper speed is not available (err=%d)\n",
-				      dev_name(mdev->device), ret);
+		esw = mdev->priv.eswitch;
+		if (!esw || !MLX5_CAP_ESW(mdev, esw_vport_state_max_tx_speed))
 			continue;
-		}
 
-		speed = speed / MLX5_MAX_TX_SPEED_UNIT;
-		mlx5_lag_modify_device_vports_speed(mdev, speed);
+		mutex_lock(&esw->state_lock);
+		mlx5_esw_for_each_vport(esw, i, vport) {
+			if (!vport || vport->vport == MLX5_VPORT_UPLINK)
+				continue;
+
+			if (!vport->enabled) {
+				vport->agg_max_tx_speed = vport->max_tx_speed;
+				continue;
+			}
+
+			if (mlx5_lag_get_restore_speed(mdev, vport,
+						       &restore_speed)) {
+				vport->agg_max_tx_speed = 0;
+				continue;
+			}
+
+			ret = mlx5_modify_vport_max_tx_speed(mdev, op_mod,
+							     vport->vport,
+							     true,
+							     restore_speed);
+			if (ret) {
+				mlx5_core_dbg(mdev,
+					      "Failed to restore vport %d max tx speed, err=%d\n",
+					      vport->vport, ret);
+				continue;
+			}
+			vport->agg_max_tx_speed = 0;
+		}
+		mutex_unlock(&esw->state_lock);
 	}
 }
 #endif
-- 
2.44.0

Re: [PATCH net V2 0/4] net/mlx5: fixes for LAG max_tx_speed update flow

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - How the issue was discovered, e.g. hit in production, hit during
   development, syzbot report, manual code inspection, LLM or static
   analysis tool scan.

 - Whether the issue was actually triggered, or is only theoretical
   (e.g. found by code inspection). If it was triggered please include
   the symptoms, like the stack trace or error messages.

 - What hardware the change was tested on. For driver fixes please
   mention the device (and if relevant firmware version) used for
   testing, or say that the change was not tested on real hardware.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.