[PATCH net V2 3/4] {RDMA,net}/mlx5: notify IB uplink on LAG aggregate speed change

[Tariq Toukan <tariqt@nvidia.com>]

From: Or Har-Toov <ohartoov@nvidia.com>

In switchdev LAG, a link-state change fires two
IB_EVENT_DEVICE_SPEED_CHANGE events: the physical port fires immediately
with the port state change, and once the LAG reconverges and pushes the
new aggregate speed to every vport via MODIFY_VPORT_STATE, FW fires a
PORT_CHANGE event per vport, which is how VF consumers learn the
aggregate actually changed.

The uplink vport is deliberately skipped when pushing the aggregate
speed, so the IB uplink device never gets a second event once the
aggregate has settled. In RoCE LAG there are no VF vports at all, so
nothing fires after reconvergence.

Raise IB_EVENT_DEVICE_SPEED_CHANGE from the LAG layer, in
mlx5_lag_update_agg_speed(), right after the cached aggregate is
updated. That cache is where the IB layer now takes the effective speed
from, so a consumer that queries after receiving the event is
guaranteed to read the new value.

blocking_notifier_call_chain() holds the blocking_notifier_head rwsem
for the full duration of lag_event();
blocking_notifier_chain_unregister() in mlx5_eth_lag_cleanup() takes the
write side and cannot return until any in-flight callback has completed,
so no event can be dispatched to a freed device after unregistration.

Fixes: 50f1d188c580 ("net/mlx5: Propagate LAG effective max_tx_speed to vports")
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c             | 10 ++++++++++
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 20 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h |  1 +
 include/linux/mlx5/device.h                   |  1 +
 4 files changed, 32 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 6594d38cf199..cfa7c7ba2b73 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3690,6 +3690,16 @@ static int lag_event(struct notifier_block *nb, unsigned long event, void *data)
 			rdma_roce_rescan_port(ibdev, portnum + 1);
 		}
 		break;
+	case MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE: {
+		struct ib_event speed_event = {};
+
+		if (!dev->ib_active)
+			break;
+		speed_event.device = ibdev;
+		speed_event.event = IB_EVENT_DEVICE_SPEED_CHANGE;
+		ib_dispatch_event(&speed_event);
+		break;
+	}
 	default:
 		return NOTIFY_DONE;
 	}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 05e6d0aa31fc..afe0b9909f2d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -1540,8 +1540,24 @@ static int mlx5_lag_get_devices_oper_speed(struct mlx5_lag *ldev,
 	return 0;
 }
 
+void mlx5_lag_notify_speed_change(struct mlx5_lag *ldev)
+{
+	struct lag_func *pf;
+	int idx;
+
+	idx = mlx5_lag_get_dev_index_by_seq(ldev, MLX5_LAG_P1);
+	if (idx < 0)
+		return;
+	pf = mlx5_lag_pf(ldev, idx);
+	if (!pf)
+		return;
+	blocking_notifier_call_chain(&pf->dev->priv.lag_nh,
+				     MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE, NULL);
+}
+
 int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
 {
+	u32 old_speed;
 	u32 speed;
 	int err;
 
@@ -1553,7 +1569,11 @@ int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev)
 		return err;
 	}
 
+	old_speed = READ_ONCE(ldev->agg_speed_mbps);
 	WRITE_ONCE(ldev->agg_speed_mbps, speed);
+	if (speed != old_speed)
+		mlx5_lag_notify_speed_change(ldev);
+
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
index 45b0e8fd8f99..420655468d78 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h
@@ -259,6 +259,7 @@ void mlx5_lag_rescan_dev_locked(struct mlx5_lag *ldev,
 void mlx5_lag_add_devices_filter(struct mlx5_lag *ldev, u32 filter);
 struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev);
 
+void mlx5_lag_notify_speed_change(struct mlx5_lag *ldev);
 int mlx5_lag_update_agg_speed(struct mlx5_lag *ldev);
 void mlx5_lag_reset_agg_speed(struct mlx5_lag *ldev);
 int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed);
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 8cb321a9fb3d..1ec2331c24ee 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -374,6 +374,7 @@ enum mlx5_driver_event {
 	MLX5_DRIVER_EVENT_AFFILIATION_DONE,
 	MLX5_DRIVER_EVENT_AFFILIATION_REMOVED,
 	MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE,
+	MLX5_DRIVER_EVENT_LAG_SPEED_CHANGE,
 };
 
 enum {
-- 
2.44.0