* [PATCH net-next] net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
@ 2026-11-16 21:09 Tariq Toukan
2026-11-18 9:12 ` netdev-bot+sashiko
0 siblings, 1 reply; 2+ messages in thread
From: Tariq Toukan @ 2026-11-16 21:09 UTC (permalink / raw)
To: Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
netdev, Paolo Abeni
Cc: Gal Pressman, Leon Romanovsky, open list, linux-rdma, Mark Bloch,
Moshe Shemesh, Saeed Mahameed, Shay Drori, Tariq Toukan, Uri Dar
From: Uri Dar <udar@nvidia.com>
The slow path FDB send-to-vport and peer-miss groups are sized from the
local eswitch total_vports, assuming every eswitch that may pair has the
same number of vports. Per-PF VF configuration (e.g. PF_NUM_OF_VF set
differently per port) breaks that assumption: the larger peer installs
rules sized by its own total_vports into the smaller peer's groups,
which cannot hold them, so pairing fails.
In SW and HW steering the flow table is unbounded (max_fte is INT_MAX)
and flow groups are backed by dynamically grown matchers, so size the
peer dependent groups to the architectural vport maximum. FW steering,
whose table is bounded by log_max_ft_size, keeps the current sizing.
Signed-off-by: Uri Dar <udar@nvidia.com>
Reviewed-by: Moshe Shemesh <moshe@nvidia.com>
Reviewed-by: Shay Drori <shayd@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
.../mellanox/mlx5/core/eswitch_offloads.c | 17 +++++++++++++++--
include/linux/mlx5/eswitch.h | 2 ++
2 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
index 474784131091..20eca8b89b8f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
@@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)
#endif
+/* SW and HW steering tables are unbounded, so they can be sized for any peer.
+ * FW steering is bounded by log_max_ft_size, so it uses local total_vports.
+ */
+static int esw_max_vports(const struct mlx5_eswitch *esw)
+{
+ if (esw->dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
+ return esw->total_vports;
+
+ return MLX5_ESW_MAX_VPORT_ID;
+}
+
static int
esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
struct mlx5_flow_table *fdb,
@@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
}
/* See comment at table_size calculation */
- count = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ);
+ count = MLX5_MAX_PORTS *
+ (esw_max_vports(esw) * MAX_SQ_NVPORTS + MAX_PF_SQ);
+
MLX5_SET(create_flow_group_in, flow_group_in, start_flow_index, 0);
MLX5_SET(create_flow_group_in, flow_group_in, end_flow_index, *ix + count - 1);
*ix += count;
@@ -1841,7 +1854,7 @@ esw_create_peer_esw_miss_group(struct mlx5_eswitch *esw,
u32 *flow_group_in,
int *ix)
{
- int max_peer_ports = (esw->total_vports - 1) * (MLX5_MAX_PORTS - 1);
+ int max_peer_ports = (esw_max_vports(esw) - 1) * (MLX5_MAX_PORTS - 1);
int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
struct mlx5_flow_group *g;
void *match_criteria;
diff --git a/include/linux/mlx5/eswitch.h b/include/linux/mlx5/eswitch.h
index 03d3620141c8..cfc52d0697e4 100644
--- a/include/linux/mlx5/eswitch.h
+++ b/include/linux/mlx5/eswitch.h
@@ -103,6 +103,8 @@ bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);
#define ESW_REG_C0_USER_DATA_METADATA_BITS (32 - ESW_SOURCE_PORT_METADATA_BITS)
#define ESW_REG_C0_USER_DATA_METADATA_MASK GENMASK(ESW_REG_C0_USER_DATA_METADATA_BITS - 1, 0)
+#define MLX5_ESW_MAX_VPORT_ID ((1 << ESW_VPORT_BITS) - 1)
+
static inline u32 mlx5_eswitch_get_vport_metadata_mask(void)
{
return GENMASK(31, 32 - ESW_SOURCE_PORT_METADATA_BITS);
base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6
--
2.44.0
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH net-next] net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
2026-11-16 21:09 [PATCH net-next] net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers Tariq Toukan
@ 2026-11-18 9:12 ` netdev-bot+sashiko
0 siblings, 0 replies; 2+ messages in thread
From: netdev-bot+sashiko @ 2026-11-18 9:12 UTC (permalink / raw)
To: tariqt
Cc: andrew+netdev, davem, edumazet, kuba, netdev, pabeni, gal, leon,
linux-kernel, linux-rdma, mbloch, moshe, saeedm, shayd, udar
Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] In SMFS/HMFS mode, esw_max_vports() returns the fixed
MLX5_ESW_MAX_VPORT_ID (4095) and ignores esw->total_vports.
- [Low] The patch changed how the send-to-vport and peer-miss groups are
sized, but left the comment and the table_size formula in…
--
net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
The patch changes how the slow path FDB send-to-vport and peer-miss groups
are sized in SW and HW steering modes. They now use a fixed architectural
vport maximum, MLX5_ESW_MAX_VPORT_ID, instead of the local total_vports.
This lets asymmetric peers pair. FW steering keeps the existing sizing.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
> index 4747841310914..20eca8b89b8f4 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
> @@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)
>
> #endif
>
> +/* SW and HW steering tables are unbounded, so they can be sized for any peer.
> + * FW steering is bounded by log_max_ft_size, so it uses local total_vports.
> + */
> +static int esw_max_vports(const struct mlx5_eswitch *esw)
> +{
> + if (esw->dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
> + return esw->total_vports;
> +
> + return MLX5_ESW_MAX_VPORT_ID;
> +}
[Severity: Medium]
In SMFS/HMFS mode, can esw_max_vports() return less than the old sizing?
MLX5_ESW_MAX_VPORT_ID is 4095, which is the width of the ESW_VPORT_BITS
field in the reg_c_0 source-port metadata. It limits the number of vports
only when vport-match metadata is enabled. In that case
esw_offloads_vport_metadata_setup() fails with -ENOSPC if there are more
than 4095 vports:
return vport->metadata ? 0 : -ENOSPC;
Metadata matching can be turned off, though. One way is the
esw_port_metadata devlink parameter. Another is a device without metadata
support. The FDB then matches on the 16-bit source_port/source_sqn space,
and nothing limits total_vports to 4095. total_vports comes from firmware
(VFs plus mlx5_sf_max_functions() and the host-PF/SPF SFs and EC VFs).
If total_vports goes above 4095 in that configuration, both groups end up
smaller than before:
send-to-vport: 8 * (4095 * 32 + 256) = 1,050,368 entries,
versus 8 * (32 * N + 256) before
peer-miss: 4094 * 7 = 28,658 entries,
versus (N - 1) * 7 before
Once the number of rules goes past the new capacity, would insert_fte()
fail here?
index = ida_alloc_max(&fg->fte_allocator, fg->max_ftes - 1, GFP_KERNEL);
Peer-miss or send-to-vport rule installation would then fail, where the old
sizing worked.
Would max(esw->total_vports, MLX5_ESW_MAX_VPORT_ID) keep the old sizing as
a lower bound?
A smaller point: the new public macro is named MLX5_ESW_MAX_VPORT_ID, but
real vport numbers are 16-bit (MLX5_VPORT_UPLINK is 0xffff and
MLX5_VPORT_ECPF is 0xfffe). Here it is used as a vport count, not an ID.
Would a name tied to the metadata field width be clearer?
> +
> static int
> esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
> struct mlx5_flow_table *fdb,
> @@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
> }
>
> /* See comment at table_size calculation */
> - count = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ);
> + count = MLX5_MAX_PORTS *
> + (esw_max_vports(esw) * MAX_SQ_NVPORTS + MAX_PF_SQ);
> +
[Severity: Low]
This isn't a bug, but should the comment and the table_size calculation in
esw_create_offloads_fdb_tables() be updated as well? The cross-reference
above now points at text that no longer matches this group:
/* To be strictly correct:
* MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ)
[ ... ]
* the peer miss group should also be created based on the number of
* total vports of the peer (currently is also uses esw->total_vports).
*/
table_size = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ) +
esw->total_vports * MLX5_MAX_PORTS + MLX5_ESW_MISS_FLOWS;
In SMFS/HMFS, both the send-to-vport group and the peer-miss group now use
esw_max_vports(). table_size, and so ft_attr.max_fte, is still computed
from total_vports. The group index layout therefore goes well past the
max_fte the table asks for.
This works today only because mlx5_cmd_dr_create_flow_table() and the HWS
create path overwrite the value:
ft->max_fte = INT_MAX;
Nothing at the table_size site records that dependency. Could the comment
mention it?
> MLX5_SET(create_flow_group_in, flow_group_in, start_flow_index, 0);
> MLX5_SET(create_flow_group_in, flow_group_in, end_flow_index, *ix + count - 1);
> *ix += count;
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116210917.1297111-1-tariqt%40nvidia.com
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-18 9:12 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 21:09 [PATCH net-next] net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers Tariq Toukan
2026-11-18 9:12 ` netdev-bot+sashiko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®