From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Carolina Jubran <cjubran@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>, Gal Pressman <gal@nvidia.com>,
Leon Romanovsky <leon@kernel.org>,
open list <linux-kernel@vger.kernel.org>,
<linux-rdma@vger.kernel.org>, Mark Bloch <mbloch@nvidia.com>,
Moshe Shemesh <moshe@nvidia.com>,
Or Har-Toov <ohartoov@nvidia.com>,
Patrisious Haddad <phaddad@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>, Shay Drory <shayd@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>, Uri Dar <udar@nvidia.com>
Subject: [PATCH net-next V2 00/10] net/mlx5: LAG, preparations for LAG over VFs/SFs
Date: Mon, 16 Nov 2026 12:30:00 +0200 [thread overview]
Message-ID: <20261116103010.1129666-1-tariqt@nvidia.com> (raw)
Hi,
This series by Shay is the first part of enabling a LAG over functions
that do not represent a physical port - a "virtual LAG" (virt LAG) of
VFs or SFs acting as nested e-switch managers. New FW lets a VF/SF front
the physical uplinks through a multi-uplink e-switch; such a function
reports num_lag_ports < 2 and drives port selection via the e-switch
(port_select_eswitch) rather than the HW port-selection flow table. A
virt LAG is therefore a software-managed, shared-FDB MPESW LAG: the
driver builds the LAG demux and vport-LAG in software.
This part prepares the LAG core. A follow-up series adds the e-switch
and representor support and the patch that admits a virt LAG; until then
no ldev is marked as a virt LAG and the virt LAG paths added here are
unused.
The patches fall into two parts.
LAG code cleanup (patches 5, 6, 7, 8, 9)
- Finish removing the fixed MLX5_MAX_PORTS sizing the first series
began: convert the remaining structures to dynamic, port-count-based
allocation and inline iteration. A virt LAG's member count is
unrelated to the physical port count.
- Rename the LAG "pf" naming to "fn", since a member is now any
function (PF, VF or SF), not necessarily a PF.
- Resolve the queue-affinity member indices only in that mode, so the
other port-select modes are not rejected for indices they never use.
Virtual LAG core (patches 1, 2, 3, 4, 10)
- Distinguish VFs/SFs from their PF in the SW system image GUID, so
they group into their own LAG instead of with the PF.
- Select the MPESW master from the live member snapshot: VF/SF members
are dynamic and not marked by the devcom PAIR event, so the master
is chosen from the current membership (tie-broken by PCI BDF for
VFs, sfnum for SFs).
- Skip the phys-LAG prerequisites (port_select_flow_table,
create_lag_when_not_master_up, ...) for a virt LAG, and require
port_select_eswitch instead.
- Treat a virt LAG as a software-managed LAG (demux + vport-LAG in
SW).
- Do not add a device to an already-active virt LAG.
Regards,
Tariq
V2:
- Split the patch set to ease the review: this is v1 patches 1-10; v1
patches 11-15 will be sent as a follow-up series.
- Patch 1: define mlx5_virt_lag_is_supported() in
include/linux/mlx5/driver.h, so mlx5_ib can use it in the follow-up
series.
- Patch 3: fix the commit message: a LAG of VFs/SFs still issues
CREATE_LAG, but a VF/SF issuing it does not create a LAG of physical
ports (Sashiko).
- Patch 10: retitled, and the commit message says the rejection applies
only to a virt LAG, and why the other join paths are safe (Sashiko).
V1:
https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/
Answers to sashiko comments on v1:
net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping
Q: A VF/SF that reports multi_uplink_eswitch, lag_master and
num_lag_ports < 2 now gets a key that differs from its PF's key, and
mlx5_same_hw_devs() requires the keys to match exactly. Does this break
PF to VF/SF same-HW detection, e.g. mlx5e_tc_is_vf_tunnel() and mirred
offload in NIC mode? Should mlx5_same_hw_devs() use a key without the
VF/SF markers?
[SD] No, this is the point of the patch. A VF/SF that reports
multi_uplink_eswitch and lag_master (and num_lag_ports < 2) is a nested
eswitch manager. That makes it a different switch domain from its
PF, not one more vport of the PF's FDB, so it must not be identified as
the same HW as the PF. Without the marker it would also be grouped into
the PF's LAG, which is what the patch prevents.
A VF/SF that does not report these caps, i.e. every VF/SF that exists
today, keeps its key unchanged and is still the same HW as its PF, as
the commit message says. Dropping the markers inside mlx5_same_hw_devs()
would make a nested eswitch manager the same HW as its PF again, i.e.
undo the patch.
net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG
> + if (mlx5_eswitch_mode(dev0) != MLX5_ESWITCH_OFFLOADS ||
> + !MLX5_CAP_PORT_SELECTION(dev0, port_select_eswitch) ||
Q: mlx5_virt_lag_is_supported() does not check port_select_eswitch. A
VF/SF that reports multi_uplink_eswitch and lag_master but not
port_select_eswitch would get the VF/SF marker in its SW GUID and join a
virt ldev, yet MPESW could never be enabled on it. Should
port_select_eswitch be checked in mlx5_virt_lag_is_supported() instead,
so the two checks agree?
[SD] The two checks are at different stages on purpose. A function with
the first two and without the third doesn't exist.
Even if it were, the result is benign: the function stays an inactive
member and esw_multiport fails cleanly with -EOPNOTSUPP, nothing else
changes.
net/mlx5: LAG, treat virt LAG as a software managed LAG
> -static int mlx5_lag_is_sw_lag(struct mlx5_core_dev *dev)
> +int mlx5_lag_is_sw_managed(struct mlx5_core_dev *dev)
> {
> - return mlx5_lag_is_sd(dev);
> + struct mlx5_lag *ldev = mlx5_lag_dev(dev);
Q: If dev is NULL, does mlx5_lag_dev(dev) NULL-deref dev->priv.lag? When
called from mlx5_lag_multipath_check_prereq() via the FIB notifier,
pf->dev can be NULL due to concurrent removal in mlx5_ldev_remove_mdev().
__mlx5_lag_is_sd() was safe (lockless pointer compare). Does this need a
NULL check?
[SD] mlx5_lag_is_sw_managed() assumes a valid dev pointer, like the rest
of the mlx5_lag_is_* APIs. It is true mlx5_lag_multipath_check_prereq()
can provide an invalid device pointer, but that is an existing bug there
which should be handled in its own patch.
net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
> - if (i < MLX5_MAX_PORTS) {
> + if (i < ldev->ports) {
> slave = bond_slave_get_rcu(ndev_tmp);
Q: Should this compare against ldev->max_funcs, like the other converted
call sites? A PORT-marked member can sit at an index in
[ldev->ports, max_funcs) if more PFs than num_lag_ports join the ldev
and a lower-index one is removed later. Physical ports also share the
xarray with VFs/SFs. Wouldn't that leave the bond silently not
offloaded?
[SD] No. mlx5 creates an ldev per sw_guid, and PFs have a different
sw_guid than VFs/SFs, so only PFs share this ldev. The only case with
more PFs than num_lag_ports in one ldev is SD, and for SD the netdev
notifier, and with it mlx5_handle_changeupper_event(), is not
registered. So a PORT-marked member here always sits at an index below
ldev->ports.
> - err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, MLX5_MAX_PORTS - 1),
> + err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, U8_MAX - 1),
Q: Physical members and Socket Direct secondaries share the xarray index
space. Can a physical member get an index >= MLX5_MAX_PORTS and make
mlx5_ldev_add_netdev() write beyond tracker.netdev_state[]?
[SD] Such HW doesn't exist; it needs LAG over SD with more than
MLX5_MAX_PORTS / 2 ports, and when it does MLX5_MAX_PORTS will be
increased. To protect a new HW running an old driver, a check that
num_lag_ports times the SD group size fits in MLX5_MAX_PORTS will be added
to mlx5_lag_is_supported() separately.
Q: Can removing the xa_alloc() limit let member indices reach 8 or more
while pf_metadata[], devs[], the cong counter array,
tracker.netdev_state[] and the port_sel rules[] are still sized
MLX5_MAX_PORTS? Could the limit stay at MLX5_MAX_PORTS - 1 until all of
these users are converted?
[SD] No. In this series a 9th member is impossible: an ldev holds only
PFs and SD secondaries, and no HW has more than MLX5_MAX_PORTS of them.
VFs/SFs join an ldev only in the follow-up series.
The limit can't be kept until all arrays are converted, since not all
of them will be. Some arrays are used only by a PF LAG, others by both a
PF LAG and a virt LAG. The shared ones, the fns xarray included, are
converted in this series (pf_metadata[] and devs[]; the cong counter
array is converted as well). The PF LAG only ones,
tracker.netdev_state[] and the port_sel rules[], keep their static
MLX5_MAX_PORTS size: they are walked only over PORT-marked members, in
the hash and queue affinity modes. A virt member is never PORT-marked
and sits in its own ldev, and a virt LAG supports only MPESW, so these
arrays never see more than MLX5_MAX_PORTS entries.
> + /* max_funcs is the iteration ceiling; grow it, never shrink. */
> + if (idx >= ldev->max_funcs)
> + ldev->max_funcs = idx + 1;
Q: Is this store safe against readers that only hold lag_lock?
mlx5_lag_get_next_peer_mdev(), mlx5_lag_get_slave_port() and
mlx5_lag_query_cong_counters() read max_funcs more than once per walk,
so a concurrent add could make the sentinel and the loop condition
disagree, and the body could see a pf whose dev is not set yet.
[SD] all the readers above running only while LAG is active, and
mlx5_lag_query_cong_counters() and mlx5_lag_get_slave_port() check
whether lag is active explicitly before walking over the lag
> - ldev->ports);
> + ldev->max_funcs);
Q: The commit message says "No functional change". Is that accurate?
max_slaves sizes the master's egress ACL, so the size of the hardware
flow table changes whenever max_funcs != ports.
[SD] Only the ACL size changes, and only for SD, where more PFs than
ports join the ldev and max_funcs > ports. The rules installed are the
same as before, so there is no functional change.
Q: end_flow_index is inclusive, so the master egress group spans
max_slaves + 1 entries in a table created with max_fte = max_slaves.
If max_slaves equals a flow table pool size, e.g. 128, can the group
creation fail?
[SD] This is theoretical only; there is no such use case. In this
series max_slaves is at most MLX5_MAX_PORTS, and the table is rounded
up to 128 entries, so the group fits.
Q: The patch also changes the xa_alloc() limit in mlx5_ldev_add_mdev(),
which now accepts a 9th member, and the found check in
mlx5_handle_changeupper_event(), which now uses i < ldev->ports. Could
the commit message describe these changes?
[SD] The xa_alloc() change is already in the commit message ("remove the
xa_alloc() limit of MLX5_MAX_PORTS"), and the answer above explains why
it can't let in a 9th member.
mlx5_handle_changeupper_event() runs only for a PF LAG without SD: the
netdev notifier is registered only when !mlx5_sd_is_supported(). There
each LAG port is one PF at an xarray index below ldev->ports
(num_lag_ports), so "i < ldev->ports" is true in exactly the same cases
as the old end-of-walk check.
net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation
> + ldev = mlx5_lag_dev(dev);
> + num_ports = ldev ? max_t(int, ldev->ports, 1) : 1;
> + mdev = kcalloc(num_ports, sizeof(mdev[0]), GFP_KERNEL);
Q: Could mlx5_lag_query_cong_counters() keep a freed ldev across this
sleeping allocation if teardown removes the association and drops the
last LAG reference before the query resumes?
[SD] No. The query runs on the IB device of dev, and dev leaves the LAG
only in its own teardown, after its IB device is unregistered.
Q: Can this loop write past the end of mdev[]? The array has
max(ldev->ports, 1) entries, mlx5_ldev_for_each() writes one entry per
PORT-marked member, and nothing caps the PORT-marked members at
ldev->ports. Should the loop break once idx >= num_ports?
[SD] It can't. The loop runs only for a non-virt LAG, and the
PORT-marked members of such an ldev are the PFs of one card, at most
num_lag_ports == ldev->ports of them (an SD secondary is not PORT-marked
and a virt member is excluded by the !ldev->virt_lag check right here).
Q: Can mlx5_mpesw_metadata_cleanup() index past the end of
pf_metadata[]? The buffer is sized by ldev->max_fns when MPESW is
enabled, but sd_lag_init() can add SD secondaries afterwards and grow
max_fns, and the cleanup loop stops at the current max_fns.
[SD] this is pre-existing bug, MPESW should failed if SD Secondary
is down. this will be handle in separate patch.
net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan
> + mlx5_lag_for_each(i, 0, ldev, filter) {
> + pf = mlx5_lag_pf(ldev, i);
> + ...
> + mlx5_rescan_drivers_locked(pf->dev);
Q: Is it safe to iterate the LAG PFs and deref pf->dev while ldev->lock is
dropped?
[SD] Valid. Membership can be removed by SD cleanup while ldev->lock
is dropped; this will be fixed in a separate patch.
Q: Does using the sparse XArray index directly as a dense-array index
cause OOB writes elsewhere? With Socket Direct the XArray index (i) for a
primary port can exceed ldev->ports, and it is used directly in
mlx5_infer_tx_affinity_mapping() (ports[idx]) and
mlx5_handle_changelowerstate_event() (tracker->netdev_state[idx]) - could
these overflow the heap/stack buffers?
[SD] All these buffers aren't used in LAG over SD; only MPESW is, and
MPESW doesn't use these APIs.
net/mlx5: LAG, do not add a device to an already-active virt LAG
Q: After this change only -EOPNOTSUPP is final. Every other error from
__mlx5_lag_dev_add_mdev() is retried with no limit. xa_alloc() returns
-EBUSY once all 255 slots are in use, and membership does not change
while the probe waits. Can mlx5_load() loop forever? Would it make sense
to retry only on -EAGAIN?
[SD] This is pre-existing: mlx5_lag_add_mdev() retried every error
before this series as well. for example future HW can return -EBUSY.
The fix will be sent as a separate patch.
Shay Drory (10):
net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG
grouping
net/mlx5: LAG, select the MPESW master from the live member snapshot
net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG
net/mlx5: LAG, treat virt LAG as a software managed LAG
net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
net/mlx5: LAG, rename PF naming to fn
net/mlx5: LAG, resolve affinity indices only in queue-affinity mode
net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic
allocation
net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan
net/mlx5: LAG, do not add a device to an already-active virt LAG
.../mellanox/mlx5/core/eswitch_offloads.c | 4 +
.../ethernet/mellanox/mlx5/core/lag/debugfs.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/lag/lag.c | 568 ++++++++++--------
.../net/ethernet/mellanox/mlx5/core/lag/lag.h | 62 +-
.../net/ethernet/mellanox/mlx5/core/lag/mp.c | 29 +-
.../ethernet/mellanox/mlx5/core/lag/mpesw.c | 125 ++--
.../ethernet/mellanox/mlx5/core/lag/mpesw.h | 2 +-
.../mellanox/mlx5/core/lag/port_sel.c | 25 +-
.../mellanox/mlx5/core/lag/shared_fdb.c | 44 +-
.../net/ethernet/mellanox/mlx5/core/lib/sd.c | 32 +-
.../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 12 +
.../net/ethernet/mellanox/mlx5/core/vport.c | 40 +-
include/linux/mlx5/driver.h | 11 +-
13 files changed, 573 insertions(+), 383 deletions(-)
base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6
--
2.44.0
next reply other threads:[~2026-11-16 10:31 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 10:30 Tariq Toukan [this message]
2026-11-16 10:30 ` [PATCH net-next V2 01/10] net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 02/10] net/mlx5: LAG, select the MPESW master from the live member snapshot Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 03/10] net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 04/10] net/mlx5: LAG, treat virt LAG as a software managed LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 06/10] net/mlx5: LAG, rename PF naming to fn Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 07/10] net/mlx5: LAG, resolve affinity indices only in queue-affinity mode Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 08/10] net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 09/10] net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 10/10] net/mlx5: LAG, do not add a device to an already-active virt LAG Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116103010.1129666-1-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=cjubran@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=gal@nvidia.com \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mbloch@nvidia.com \
--cc=moshe@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=ohartoov@nvidia.com \
--cc=pabeni@redhat.com \
--cc=phaddad@nvidia.com \
--cc=saeedm@nvidia.com \
--cc=shayd@nvidia.com \
--cc=udar@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®