mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@kernel.org>,
	Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
	Paolo Abeni <pabeni@redhat.com>
Cc: Carolina Jubran <cjubran@nvidia.com>,
	Cosmin Ratiu <cratiu@nvidia.com>, Gal Pressman <gal@nvidia.com>,
	Leon Romanovsky <leon@kernel.org>,
	open list <linux-kernel@vger.kernel.org>,
	<linux-rdma@vger.kernel.org>, Mark Bloch <mbloch@nvidia.com>,
	Moshe Shemesh <moshe@nvidia.com>,
	Or Har-Toov <ohartoov@nvidia.com>,
	Patrisious Haddad <phaddad@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>, Shay Drory <shayd@nvidia.com>,
	Tariq Toukan <tariqt@nvidia.com>, Uri Dar <udar@nvidia.com>
Subject: [PATCH net-next V2 00/10] net/mlx5: LAG, preparations for LAG over VFs/SFs
Date: Mon, 16 Nov 2026 12:30:00 +0200	[thread overview]
Message-ID: <20261116103010.1129666-1-tariqt@nvidia.com> (raw)

Hi,

This series by Shay is the first part of enabling a LAG over functions
that do not represent a physical port - a "virtual LAG" (virt LAG) of
VFs or SFs acting as nested e-switch managers. New FW lets a VF/SF front
the physical uplinks through a multi-uplink e-switch; such a function
reports num_lag_ports < 2 and drives port selection via the e-switch
(port_select_eswitch) rather than the HW port-selection flow table. A
virt LAG is therefore a software-managed, shared-FDB MPESW LAG: the
driver builds the LAG demux and vport-LAG in software.

This part prepares the LAG core. A follow-up series adds the e-switch
and representor support and the patch that admits a virt LAG; until then
no ldev is marked as a virt LAG and the virt LAG paths added here are
unused.

The patches fall into two parts.

LAG code cleanup (patches 5, 6, 7, 8, 9)
  - Finish removing the fixed MLX5_MAX_PORTS sizing the first series
    began: convert the remaining structures to dynamic, port-count-based
    allocation and inline iteration. A virt LAG's member count is
    unrelated to the physical port count.
  - Rename the LAG "pf" naming to "fn", since a member is now any
    function (PF, VF or SF), not necessarily a PF.
  - Resolve the queue-affinity member indices only in that mode, so the
    other port-select modes are not rejected for indices they never use.

Virtual LAG core (patches 1, 2, 3, 4, 10)
  - Distinguish VFs/SFs from their PF in the SW system image GUID, so
    they group into their own LAG instead of with the PF.
  - Select the MPESW master from the live member snapshot: VF/SF members
    are dynamic and not marked by the devcom PAIR event, so the master
    is chosen from the current membership (tie-broken by PCI BDF for
    VFs, sfnum for SFs).
  - Skip the phys-LAG prerequisites (port_select_flow_table,
    create_lag_when_not_master_up, ...) for a virt LAG, and require
    port_select_eswitch instead.
  - Treat a virt LAG as a software-managed LAG (demux + vport-LAG in
    SW).
  - Do not add a device to an already-active virt LAG.

Regards,
Tariq

V2:
- Split the patch set to ease the review: this is v1 patches 1-10; v1
  patches 11-15 will be sent as a follow-up series.
- Patch 1: define mlx5_virt_lag_is_supported() in
  include/linux/mlx5/driver.h, so mlx5_ib can use it in the follow-up
  series.
- Patch 3: fix the commit message: a LAG of VFs/SFs still issues
  CREATE_LAG, but a VF/SF issuing it does not create a LAG of physical
  ports (Sashiko).
- Patch 10: retitled, and the commit message says the rejection applies
  only to a virt LAG, and why the other join paths are safe (Sashiko).


V1:
https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/

Answers to sashiko comments on v1:

net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping

  Q: A VF/SF that reports multi_uplink_eswitch, lag_master and
  num_lag_ports < 2 now gets a key that differs from its PF's key, and
  mlx5_same_hw_devs() requires the keys to match exactly. Does this break
  PF to VF/SF same-HW detection, e.g. mlx5e_tc_is_vf_tunnel() and mirred
  offload in NIC mode? Should mlx5_same_hw_devs() use a key without the
  VF/SF markers?

  [SD] No, this is the point of the patch. A VF/SF that reports
  multi_uplink_eswitch and lag_master (and num_lag_ports < 2) is a nested
  eswitch manager. That makes it a different switch domain from its
  PF, not one more vport of the PF's FDB, so it must not be identified as
  the same HW as the PF. Without the marker it would also be grouped into
  the PF's LAG, which is what the patch prevents.
  A VF/SF that does not report these caps, i.e. every VF/SF that exists
  today, keeps its key unchanged and is still the same HW as its PF, as
  the commit message says. Dropping the markers inside mlx5_same_hw_devs()
  would make a nested eswitch manager the same HW as its PF again, i.e.
  undo the patch.

net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG

  > +	if (mlx5_eswitch_mode(dev0) != MLX5_ESWITCH_OFFLOADS ||
  > +	    !MLX5_CAP_PORT_SELECTION(dev0, port_select_eswitch) ||

  Q: mlx5_virt_lag_is_supported() does not check port_select_eswitch. A
  VF/SF that reports multi_uplink_eswitch and lag_master but not
  port_select_eswitch would get the VF/SF marker in its SW GUID and join a
  virt ldev, yet MPESW could never be enabled on it. Should
  port_select_eswitch be checked in mlx5_virt_lag_is_supported() instead,
  so the two checks agree?

  [SD] The two checks are at different stages on purpose. A function with
  the first two and without the third doesn't exist.
  Even if it were, the result is benign: the function stays an inactive
  member and esw_multiport fails cleanly with -EOPNOTSUPP, nothing else
  changes.

net/mlx5: LAG, treat virt LAG as a software managed LAG

  > -static int mlx5_lag_is_sw_lag(struct mlx5_core_dev *dev)
  > +int mlx5_lag_is_sw_managed(struct mlx5_core_dev *dev)
  >  {
  > -	return mlx5_lag_is_sd(dev);
  > +	struct mlx5_lag *ldev = mlx5_lag_dev(dev);

  Q: If dev is NULL, does mlx5_lag_dev(dev) NULL-deref dev->priv.lag? When
  called from mlx5_lag_multipath_check_prereq() via the FIB notifier,
  pf->dev can be NULL due to concurrent removal in mlx5_ldev_remove_mdev().
  __mlx5_lag_is_sd() was safe (lockless pointer compare). Does this need a
  NULL check?

  [SD] mlx5_lag_is_sw_managed() assumes a valid dev pointer, like the rest
  of the mlx5_lag_is_* APIs. It is true mlx5_lag_multipath_check_prereq()
  can provide an invalid device pointer, but that is an existing bug there
  which should be handled in its own patch.

net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS

  > -		if (i < MLX5_MAX_PORTS) {
  > +		if (i < ldev->ports) {
  >  			slave = bond_slave_get_rcu(ndev_tmp);

  Q: Should this compare against ldev->max_funcs, like the other converted
  call sites? A PORT-marked member can sit at an index in
  [ldev->ports, max_funcs) if more PFs than num_lag_ports join the ldev
  and a lower-index one is removed later. Physical ports also share the
  xarray with VFs/SFs. Wouldn't that leave the bond silently not
  offloaded?

  [SD] No. mlx5 creates an ldev per sw_guid, and PFs have a different
  sw_guid than VFs/SFs, so only PFs share this ldev. The only case with
  more PFs than num_lag_ports in one ldev is SD, and for SD the netdev
  notifier, and with it mlx5_handle_changeupper_event(), is not
  registered. So a PORT-marked member here always sits at an index below
  ldev->ports.

  > -	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, MLX5_MAX_PORTS - 1),
  > +	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, U8_MAX - 1),

  Q: Physical members and Socket Direct secondaries share the xarray index
  space. Can a physical member get an index >= MLX5_MAX_PORTS and make
  mlx5_ldev_add_netdev() write beyond tracker.netdev_state[]?

  [SD] Such HW doesn't exist; it needs LAG over SD with more than
  MLX5_MAX_PORTS / 2 ports, and when it does MLX5_MAX_PORTS will be
  increased. To protect a new HW running an old driver, a check that
  num_lag_ports times the SD group size fits in MLX5_MAX_PORTS will be added
  to mlx5_lag_is_supported() separately.

  Q: Can removing the xa_alloc() limit let member indices reach 8 or more
  while pf_metadata[], devs[], the cong counter array,
  tracker.netdev_state[] and the port_sel rules[] are still sized
  MLX5_MAX_PORTS? Could the limit stay at MLX5_MAX_PORTS - 1 until all of
  these users are converted?

  [SD] No. In this series a 9th member is impossible: an ldev holds only
  PFs and SD secondaries, and no HW has more than MLX5_MAX_PORTS of them.
  VFs/SFs join an ldev only in the follow-up series.
  The limit can't be kept until all arrays are converted, since not all
  of them will be. Some arrays are used only by a PF LAG, others by both a
  PF LAG and a virt LAG. The shared ones, the fns xarray included, are
  converted in this series (pf_metadata[] and devs[]; the cong counter
  array is converted as well). The PF LAG only ones,
  tracker.netdev_state[] and the port_sel rules[], keep their static
  MLX5_MAX_PORTS size: they are walked only over PORT-marked members, in
  the hash and queue affinity modes. A virt member is never PORT-marked
  and sits in its own ldev, and a virt LAG supports only MPESW, so these
  arrays never see more than MLX5_MAX_PORTS entries.

  > +	/* max_funcs is the iteration ceiling; grow it, never shrink. */
  > +	if (idx >= ldev->max_funcs)
  > +		ldev->max_funcs = idx + 1;

  Q: Is this store safe against readers that only hold lag_lock?
  mlx5_lag_get_next_peer_mdev(), mlx5_lag_get_slave_port() and
  mlx5_lag_query_cong_counters() read max_funcs more than once per walk,
  so a concurrent add could make the sentinel and the loop condition
  disagree, and the body could see a pf whose dev is not set yet.

  [SD] all the readers above running only while LAG is active, and
  mlx5_lag_query_cong_counters() and mlx5_lag_get_slave_port() check
  whether lag is active explicitly before walking over the lag

  > -								   ldev->ports);
  > +								   ldev->max_funcs);

  Q: The commit message says "No functional change". Is that accurate?
  max_slaves sizes the master's egress ACL, so the size of the hardware
  flow table changes whenever max_funcs != ports.

  [SD] Only the ACL size changes, and only for SD, where more PFs than
  ports join the ldev and max_funcs > ports. The rules installed are the
  same as before, so there is no functional change.

  Q: end_flow_index is inclusive, so the master egress group spans
  max_slaves + 1 entries in a table created with max_fte = max_slaves.
  If max_slaves equals a flow table pool size, e.g. 128, can the group
  creation fail?

  [SD] This is theoretical only; there is no such use case. In this
  series max_slaves is at most MLX5_MAX_PORTS, and the table is rounded
  up to 128 entries, so the group fits.

  Q: The patch also changes the xa_alloc() limit in mlx5_ldev_add_mdev(),
  which now accepts a 9th member, and the found check in
  mlx5_handle_changeupper_event(), which now uses i < ldev->ports. Could
  the commit message describe these changes?

  [SD] The xa_alloc() change is already in the commit message ("remove the
  xa_alloc() limit of MLX5_MAX_PORTS"), and the answer above explains why
  it can't let in a 9th member.
  mlx5_handle_changeupper_event() runs only for a PF LAG without SD: the
  netdev notifier is registered only when !mlx5_sd_is_supported(). There
  each LAG port is one PF at an xarray index below ldev->ports
  (num_lag_ports), so "i < ldev->ports" is true in exactly the same cases
  as the old end-of-walk check.

net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation

  > +	ldev = mlx5_lag_dev(dev);
  > +	num_ports = ldev ? max_t(int, ldev->ports, 1) : 1;
  > +	mdev = kcalloc(num_ports, sizeof(mdev[0]), GFP_KERNEL);

  Q: Could mlx5_lag_query_cong_counters() keep a freed ldev across this
  sleeping allocation if teardown removes the association and drops the
  last LAG reference before the query resumes?

  [SD] No. The query runs on the IB device of dev, and dev leaves the LAG
  only in its own teardown, after its IB device is unregistered.

  Q: Can this loop write past the end of mdev[]? The array has
  max(ldev->ports, 1) entries, mlx5_ldev_for_each() writes one entry per
  PORT-marked member, and nothing caps the PORT-marked members at
  ldev->ports. Should the loop break once idx >= num_ports?

  [SD] It can't. The loop runs only for a non-virt LAG, and the
  PORT-marked members of such an ldev are the PFs of one card, at most
  num_lag_ports == ldev->ports of them (an SD secondary is not PORT-marked
  and a virt member is excluded by the !ldev->virt_lag check right here).

  Q: Can mlx5_mpesw_metadata_cleanup() index past the end of
  pf_metadata[]? The buffer is sized by ldev->max_fns when MPESW is
  enabled, but sd_lag_init() can add SD secondaries afterwards and grow
  max_fns, and the cleanup loop stops at the current max_fns.

  [SD] this is pre-existing bug, MPESW should failed if SD Secondary
  is down. this will be handle in separate patch.

net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan

  > +	mlx5_lag_for_each(i, 0, ldev, filter) {
  > +		pf = mlx5_lag_pf(ldev, i);
  > +		...
  > +		mlx5_rescan_drivers_locked(pf->dev);

  Q: Is it safe to iterate the LAG PFs and deref pf->dev while ldev->lock is
  dropped?

  [SD] Valid. Membership can be removed by SD cleanup while ldev->lock
  is dropped; this will be fixed in a separate patch.

  Q: Does using the sparse XArray index directly as a dense-array index
  cause OOB writes elsewhere? With Socket Direct the XArray index (i) for a
  primary port can exceed ldev->ports, and it is used directly in
  mlx5_infer_tx_affinity_mapping() (ports[idx]) and
  mlx5_handle_changelowerstate_event() (tracker->netdev_state[idx]) - could
  these overflow the heap/stack buffers?

  [SD] All these buffers aren't used in LAG over SD; only MPESW is, and
  MPESW doesn't use these APIs.

net/mlx5: LAG, do not add a device to an already-active virt LAG

  Q: After this change only -EOPNOTSUPP is final. Every other error from
  __mlx5_lag_dev_add_mdev() is retried with no limit. xa_alloc() returns
  -EBUSY once all 255 slots are in use, and membership does not change
  while the probe waits. Can mlx5_load() loop forever? Would it make sense
  to retry only on -EAGAIN?

  [SD] This is pre-existing: mlx5_lag_add_mdev() retried every error
  before this series as well. for example future HW can return -EBUSY.
  The fix will be sent as a separate patch.

Shay Drory (10):
  net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG
    grouping
  net/mlx5: LAG, select the MPESW master from the live member snapshot
  net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG
  net/mlx5: LAG, treat virt LAG as a software managed LAG
  net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
  net/mlx5: LAG, rename PF naming to fn
  net/mlx5: LAG, resolve affinity indices only in queue-affinity mode
  net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic
    allocation
  net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan
  net/mlx5: LAG, do not add a device to an already-active virt LAG

 .../mellanox/mlx5/core/eswitch_offloads.c     |   4 +
 .../ethernet/mellanox/mlx5/core/lag/debugfs.c |   2 +-
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 568 ++++++++++--------
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h |  62 +-
 .../net/ethernet/mellanox/mlx5/core/lag/mp.c  |  29 +-
 .../ethernet/mellanox/mlx5/core/lag/mpesw.c   | 125 ++--
 .../ethernet/mellanox/mlx5/core/lag/mpesw.h   |   2 +-
 .../mellanox/mlx5/core/lag/port_sel.c         |  25 +-
 .../mellanox/mlx5/core/lag/shared_fdb.c       |  44 +-
 .../net/ethernet/mellanox/mlx5/core/lib/sd.c  |  32 +-
 .../ethernet/mellanox/mlx5/core/sf/dev/dev.h  |  12 +
 .../net/ethernet/mellanox/mlx5/core/vport.c   |  40 +-
 include/linux/mlx5/driver.h                   |  11 +-
 13 files changed, 573 insertions(+), 383 deletions(-)


base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6
-- 
2.44.0


             reply	other threads:[~2026-11-16 10:31 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 10:30 Tariq Toukan [this message]
2026-11-16 10:30 ` [PATCH net-next V2 01/10] net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 02/10] net/mlx5: LAG, select the MPESW master from the live member snapshot Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 03/10] net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 04/10] net/mlx5: LAG, treat virt LAG as a software managed LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 06/10] net/mlx5: LAG, rename PF naming to fn Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 07/10] net/mlx5: LAG, resolve affinity indices only in queue-affinity mode Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 08/10] net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 09/10] net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 10/10] net/mlx5: LAG, do not add a device to an already-active virt LAG Tariq Toukan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116103010.1129666-1-tariqt@nvidia.com \
    --to=tariqt@nvidia.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=cjubran@nvidia.com \
    --cc=cratiu@nvidia.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=gal@nvidia.com \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mbloch@nvidia.com \
    --cc=moshe@nvidia.com \
    --cc=netdev@vger.kernel.org \
    --cc=ohartoov@nvidia.com \
    --cc=pabeni@redhat.com \
    --cc=phaddad@nvidia.com \
    --cc=saeedm@nvidia.com \
    --cc=shayd@nvidia.com \
    --cc=udar@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®