mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Jacob Keller <jacob.e.keller@intel.com>
To: Petr Oros <poros@redhat.com>, <netdev@vger.kernel.org>
Cc: Tony Nguyen <anthony.l.nguyen@intel.com>,
	Przemek Kitszel <przemyslaw.kitszel@intel.com>,
	Andrew Lunn <andrew+netdev@lunn.ch>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@kernel.org>,
	"Jakub Kicinski" <kuba@kernel.org>,
	Paolo Abeni <pabeni@redhat.com>,
	"Sebastian Andrzej Siewior" <bigeasy@linutronix.de>,
	Clark Williams <clrkwllms@kernel.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Dave Ertman <david.m.ertman@intel.com>,
	Mateusz Palczewski <mateusz.palczewski@intel.com>,
	Leon Romanovsky <leon@kernel.org>,
	"Rafal Rogalski" <rafalx.rogalski@intel.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Aleksandr Loktionov <aleksandr.loktionov@intel.com>,
	<intel-wired-lan@lists.osuosl.org>,
	<linux-kernel@vger.kernel.org>, <linux-rt-devel@lists.linux.dev>
Subject: Re: [PATCH iwl-net v2 0/5] ice: fix RTNL vs auxiliary device lock inversion
Date: Wed, 4 Nov 2026 15:10:55 -0800	[thread overview]
Message-ID: <4b07e441-de15-44f8-a7ff-f67c9c442966@intel.com> (raw)
In-Reply-To: <20261023070258.730168-1-poros@redhat.com>

On 10/23/2026 12:02 AM, Petr Oros wrote:
> ice takes the auxiliary device lock of its RDMA auxiliary device from
> three places that run with the RTNL lock held. The RDMA auxiliary probe
> takes the two in the opposite order, because __driver_attach() holds the
> auxiliary device lock across ib_register_device(), which needs the RTNL
> lock. The result is a hard deadlock of everything that needs the RTNL
> lock, so the boot never completes:
> 

Sashiko has a couple of "high" complaints that look plausible to me.
Could you please confirm if you are working on a v3 or respond to the
reports and explain if any of these following issues are faulty reports
from the AI?

> - [High] ice: orphaned channel VSIs and memory leak when ice_setup_tc() fails teardown
> --
> - [High] ice: premature AFTER_TC_CHANGE and unlock before ice_dcb_noncontig_cfg()
> - [High] ice: inconsistent QoS and DCB state on error path in ice_pf_dcb_cfg()
> - [Medium] ice: unrolled state mutations when ice_pf_dcb_cfg() fails with -EBUSY
> --

I've left the series as "new" in the IWL patchwork until I heard one way
or another.

Thanks,
Jake

>   systemd-udevd   state:D
>     really_probe                  <- holds device_lock(&adev->dev)
>     icrdma_probe
>     irdma_ib_register_device
>     ib_register_device
>     setup_device
>     irdma_roce_port_immutable
>     ib_query_port
>     irdma_query_port
>     ib_get_eth_speed              <- blocks on rtnl_lock()
> 
>   lldpad          state:D
>     rtnetlink_rcv_msg             <- holds rtnl_lock()
>     dcb_doit
>     dcbnl_ieee_set
>     ice_dcbnl_setets
>     ice_pf_dcb_cfg
>     ice_send_event_to_aux         <- blocks on device_lock(&adev->dev)
> 
> Lockdep cannot flag it, because device_initialize() marks dev->mutex
> with lockdep_set_novalidate_class(). A PREEMPT_RT kernel with
> CONFIG_DEBUG_RT_MUTEXES reports "rtmutex deadlock detected" instead.
> 
> The probe side cannot be changed, so the three ice callers use
> device_trylock() and refuse the operation with -EBUSY when the driver
> core holds the lock. Patches 1 and 2 are new in v2 and fix two separate
> problems on the same sites, the auxiliary device pointer was tested
> before pf->adev_mutex was taken, and ndo_setup_tc() took pf->adev_mutex
> before pf->tc_mutex while the dcbnl handlers take them the other way
> round. Patch 3 keeps the locks across the whole DCB reconfiguration so
> that BEFORE_TC_CHANGE and AFTER_TC_CHANGE are either both delivered or
> the request is refused before anything changes, and re-arms the service
> task events. Patches 4 and 5 turn the remaining two locks into trylocks.
> 
> The proper fix is provider-owned registration as in i40e and mlx5, which
> makes the inversion impossible by construction. That is a net-next change
> across ice, irdma and iidc_rdma.h, this series is the backportable part.
> 
> Tested on an E810-XXV with two PFs, irdma bound to both auxiliary
> devices, on net tip commit 191aa41548b2d2 ("MAINTAINERS: add Bobby
> Eshleman as vsock reviewer") with and without this series. A reproducer
> binds the irdma auxiliary driver while driving one of the three RTNL
> held callers into the probe window:
> 
>                         without the series   with the series
>   ethtool set_channels  deadlock 10/10       0/10, window entered 300x
>   ndo_setup_tc mqprio   deadlock             0/10, window entered 50x
>   dcbnl ets set         deadlock             0/10, window entered 30x
> 
> A window entry is a contended device_trylock(), so each one used to
> close the cycle. 25 reboots on the patched kernel were clean.
> 
> Changes since v1:
> - new patch 1 for the auxiliary device pointer race
> - new patch 2 for the pf->tc_mutex order in ndo_setup_tc()
> - patch 3 holds the auxiliary device locks across the whole DCB
>   reconfiguration and sends AFTER_TC_CHANGE under them instead of
>   dropping it when the lock is contended, and takes the locks before it
>   consumes pf->oicr_err_reg in the service task
> - patch 5 warns instead of netdev_dbg(), the qdisc teardown ignores the
>   return value
> 
> Link to v1:
> https://lore.kernel.org/all/20261019065301.3983537-1-poros@redhat.com/
> 
> Petr Oros (5):
>   ice: check the auxiliary device pointer under adev_mutex
>   ice: take tc_mutex before the auxiliary device locks in ndo_setup_tc
>   ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver
>   ice: do not block on the aux device lock in ethtool set_channels
>   ice: do not block on the aux device lock in ndo_setup_tc
> 
>  drivers/net/ethernet/intel/ice/ice_dcb_lib.c | 46 ++++++++++++++++---
>  drivers/net/ethernet/intel/ice/ice_ethtool.c | 26 +++++++----
>  drivers/net/ethernet/intel/ice/ice_idc.c     | 48 ++++++++++++++++----
>  drivers/net/ethernet/intel/ice/ice_idc_int.h |  2 +
>  drivers/net/ethernet/intel/ice/ice_main.c    | 47 +++++++++++++------
>  5 files changed, 130 insertions(+), 39 deletions(-)
> 
> 
> base-commit: 07bd391eb9572e74f073c49acef327611234f9bb


      parent reply	other threads:[~2026-11-04 23:11 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-23  7:02 Petr Oros
2026-10-23  7:02 ` [PATCH iwl-net v2 1/5] ice: check the auxiliary device pointer under adev_mutex Petr Oros
2026-10-23 14:55   ` Loktionov, Aleksandr
2026-10-23  7:02 ` [PATCH iwl-net v2 2/5] ice: take tc_mutex before the auxiliary device locks in ndo_setup_tc Petr Oros
2026-10-23 14:56   ` Loktionov, Aleksandr
2026-10-23  7:02 ` [PATCH iwl-net v2 3/5] ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver Petr Oros
2026-10-24  7:03   ` sashiko-bot
2026-10-23  7:02 ` [PATCH iwl-net v2 4/5] ice: do not block on the aux device lock in ethtool set_channels Petr Oros
2026-10-23 14:56   ` Loktionov, Aleksandr
2026-10-23  7:02 ` [PATCH iwl-net v2 5/5] ice: do not block on the aux device lock in ndo_setup_tc Petr Oros
2026-10-24  7:03   ` sashiko-bot
2026-11-04 23:10 ` Jacob Keller [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4b07e441-de15-44f8-a7ff-f67c9c442966@intel.com \
    --to=jacob.e.keller@intel.com \
    --cc=aleksandr.loktionov@intel.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=anthony.l.nguyen@intel.com \
    --cc=bigeasy@linutronix.de \
    --cc=clrkwllms@kernel.org \
    --cc=davem@davemloft.net \
    --cc=david.m.ertman@intel.com \
    --cc=edumazet@kernel.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=mateusz.palczewski@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=poros@redhat.com \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=rafalx.rogalski@intel.com \
    --cc=rostedt@goodmis.org \
    --cc=shiraz.saleem@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®