From: Jacob Keller <jacob.e.keller@intel.com>
To: Petr Oros <poros@redhat.com>, <netdev@vger.kernel.org>
Cc: Tony Nguyen <anthony.l.nguyen@intel.com>,
Przemek Kitszel <przemyslaw.kitszel@intel.com>,
Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
"Jakub Kicinski" <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
"Sebastian Andrzej Siewior" <bigeasy@linutronix.de>,
Clark Williams <clrkwllms@kernel.org>,
Steven Rostedt <rostedt@goodmis.org>,
Dave Ertman <david.m.ertman@intel.com>,
Mateusz Palczewski <mateusz.palczewski@intel.com>,
Leon Romanovsky <leon@kernel.org>,
"Rafal Rogalski" <rafalx.rogalski@intel.com>,
Shiraz Saleem <shiraz.saleem@intel.com>,
Aleksandr Loktionov <aleksandr.loktionov@intel.com>,
<intel-wired-lan@lists.osuosl.org>,
<linux-kernel@vger.kernel.org>, <linux-rt-devel@lists.linux.dev>
Subject: Re: [PATCH iwl-net v2 0/5] ice: fix RTNL vs auxiliary device lock inversion
Date: Wed, 4 Nov 2026 15:10:55 -0800 [thread overview]
Message-ID: <4b07e441-de15-44f8-a7ff-f67c9c442966@intel.com> (raw)
In-Reply-To: <20261023070258.730168-1-poros@redhat.com>
On 10/23/2026 12:02 AM, Petr Oros wrote:
> ice takes the auxiliary device lock of its RDMA auxiliary device from
> three places that run with the RTNL lock held. The RDMA auxiliary probe
> takes the two in the opposite order, because __driver_attach() holds the
> auxiliary device lock across ib_register_device(), which needs the RTNL
> lock. The result is a hard deadlock of everything that needs the RTNL
> lock, so the boot never completes:
>
Sashiko has a couple of "high" complaints that look plausible to me.
Could you please confirm if you are working on a v3 or respond to the
reports and explain if any of these following issues are faulty reports
from the AI?
> - [High] ice: orphaned channel VSIs and memory leak when ice_setup_tc() fails teardown
> --
> - [High] ice: premature AFTER_TC_CHANGE and unlock before ice_dcb_noncontig_cfg()
> - [High] ice: inconsistent QoS and DCB state on error path in ice_pf_dcb_cfg()
> - [Medium] ice: unrolled state mutations when ice_pf_dcb_cfg() fails with -EBUSY
> --
I've left the series as "new" in the IWL patchwork until I heard one way
or another.
Thanks,
Jake
> systemd-udevd state:D
> really_probe <- holds device_lock(&adev->dev)
> icrdma_probe
> irdma_ib_register_device
> ib_register_device
> setup_device
> irdma_roce_port_immutable
> ib_query_port
> irdma_query_port
> ib_get_eth_speed <- blocks on rtnl_lock()
>
> lldpad state:D
> rtnetlink_rcv_msg <- holds rtnl_lock()
> dcb_doit
> dcbnl_ieee_set
> ice_dcbnl_setets
> ice_pf_dcb_cfg
> ice_send_event_to_aux <- blocks on device_lock(&adev->dev)
>
> Lockdep cannot flag it, because device_initialize() marks dev->mutex
> with lockdep_set_novalidate_class(). A PREEMPT_RT kernel with
> CONFIG_DEBUG_RT_MUTEXES reports "rtmutex deadlock detected" instead.
>
> The probe side cannot be changed, so the three ice callers use
> device_trylock() and refuse the operation with -EBUSY when the driver
> core holds the lock. Patches 1 and 2 are new in v2 and fix two separate
> problems on the same sites, the auxiliary device pointer was tested
> before pf->adev_mutex was taken, and ndo_setup_tc() took pf->adev_mutex
> before pf->tc_mutex while the dcbnl handlers take them the other way
> round. Patch 3 keeps the locks across the whole DCB reconfiguration so
> that BEFORE_TC_CHANGE and AFTER_TC_CHANGE are either both delivered or
> the request is refused before anything changes, and re-arms the service
> task events. Patches 4 and 5 turn the remaining two locks into trylocks.
>
> The proper fix is provider-owned registration as in i40e and mlx5, which
> makes the inversion impossible by construction. That is a net-next change
> across ice, irdma and iidc_rdma.h, this series is the backportable part.
>
> Tested on an E810-XXV with two PFs, irdma bound to both auxiliary
> devices, on net tip commit 191aa41548b2d2 ("MAINTAINERS: add Bobby
> Eshleman as vsock reviewer") with and without this series. A reproducer
> binds the irdma auxiliary driver while driving one of the three RTNL
> held callers into the probe window:
>
> without the series with the series
> ethtool set_channels deadlock 10/10 0/10, window entered 300x
> ndo_setup_tc mqprio deadlock 0/10, window entered 50x
> dcbnl ets set deadlock 0/10, window entered 30x
>
> A window entry is a contended device_trylock(), so each one used to
> close the cycle. 25 reboots on the patched kernel were clean.
>
> Changes since v1:
> - new patch 1 for the auxiliary device pointer race
> - new patch 2 for the pf->tc_mutex order in ndo_setup_tc()
> - patch 3 holds the auxiliary device locks across the whole DCB
> reconfiguration and sends AFTER_TC_CHANGE under them instead of
> dropping it when the lock is contended, and takes the locks before it
> consumes pf->oicr_err_reg in the service task
> - patch 5 warns instead of netdev_dbg(), the qdisc teardown ignores the
> return value
>
> Link to v1:
> https://lore.kernel.org/all/20261019065301.3983537-1-poros@redhat.com/
>
> Petr Oros (5):
> ice: check the auxiliary device pointer under adev_mutex
> ice: take tc_mutex before the auxiliary device locks in ndo_setup_tc
> ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver
> ice: do not block on the aux device lock in ethtool set_channels
> ice: do not block on the aux device lock in ndo_setup_tc
>
> drivers/net/ethernet/intel/ice/ice_dcb_lib.c | 46 ++++++++++++++++---
> drivers/net/ethernet/intel/ice/ice_ethtool.c | 26 +++++++----
> drivers/net/ethernet/intel/ice/ice_idc.c | 48 ++++++++++++++++----
> drivers/net/ethernet/intel/ice/ice_idc_int.h | 2 +
> drivers/net/ethernet/intel/ice/ice_main.c | 47 +++++++++++++------
> 5 files changed, 130 insertions(+), 39 deletions(-)
>
>
> base-commit: 07bd391eb9572e74f073c49acef327611234f9bb
prev parent reply other threads:[~2026-11-04 23:11 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-23 7:02 Petr Oros
2026-10-23 7:02 ` [PATCH iwl-net v2 1/5] ice: check the auxiliary device pointer under adev_mutex Petr Oros
2026-10-23 14:55 ` Loktionov, Aleksandr
2026-10-23 7:02 ` [PATCH iwl-net v2 2/5] ice: take tc_mutex before the auxiliary device locks in ndo_setup_tc Petr Oros
2026-10-23 14:56 ` Loktionov, Aleksandr
2026-10-23 7:02 ` [PATCH iwl-net v2 3/5] ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver Petr Oros
2026-10-24 7:03 ` sashiko-bot
2026-10-23 7:02 ` [PATCH iwl-net v2 4/5] ice: do not block on the aux device lock in ethtool set_channels Petr Oros
2026-10-23 14:56 ` Loktionov, Aleksandr
2026-10-23 7:02 ` [PATCH iwl-net v2 5/5] ice: do not block on the aux device lock in ndo_setup_tc Petr Oros
2026-10-24 7:03 ` sashiko-bot
2026-11-04 23:10 ` Jacob Keller [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4b07e441-de15-44f8-a7ff-f67c9c442966@intel.com \
--to=jacob.e.keller@intel.com \
--cc=aleksandr.loktionov@intel.com \
--cc=andrew+netdev@lunn.ch \
--cc=anthony.l.nguyen@intel.com \
--cc=bigeasy@linutronix.de \
--cc=clrkwllms@kernel.org \
--cc=davem@davemloft.net \
--cc=david.m.ertman@intel.com \
--cc=edumazet@kernel.org \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=mateusz.palczewski@intel.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=poros@redhat.com \
--cc=przemyslaw.kitszel@intel.com \
--cc=rafalx.rogalski@intel.com \
--cc=rostedt@goodmis.org \
--cc=shiraz.saleem@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®