From: Niklas Schnelle <schnelle@linux.ibm.com>
To: Benjamin Block <bblock@linux.ibm.com>,
Benjamin Block <bebl@ategam.org>,
Bjorn Helgaas <bhelgaas@google.com>
Cc: Farhan Ali <alifm@linux.ibm.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Heiko Carstens <hca@linux.ibm.com>,
Dragos Tatulea <dtatulea@nvidia.com>,
Omar Elghoul <oelghoul@linux.ibm.com>,
Lukas Wunner <lukas@wunner.de>,
Matthew Brost <matthew.brost@intel.com>,
Halil Pasic <pasic@linux.ibm.com>,
Ionut Nechita <ionut.nechita@windriver.com>,
Sven Schnelle <svens@linux.ibm.com>,
Alexander Gordeev <agordeev@linux.ibm.com>,
Ionut Nechita <ionut_n2001@yahoo.com>,
Tobias Schumacher <ts@linux.ibm.com>,
Julian Ruess <julianr@linux.ibm.com>,
Ionut Nechita <sunlightlinux@gmail.com>,
Michal Wajdeczko <michal.wajdeczko@intel.com>,
Keith Busch <kbusch@kernel.org>,
Matthew Rosato <mjrosato@linux.ibm.com>,
Christian Borntraeger <borntraeger@linux.ibm.com>,
Gerd Bayer <gbayer@linux.ibm.com>,
linux-intel-xe <intel-xe@lists.freedesktop.org>,
Andreas Krebbel <krebbel@linux.ibm.com>,
Ramesh Errabolu <ramesh@linux.ibm.com>,
linux-s390 <linux-s390@vger.kernel.org>,
Vasily Gorbik <gor@linux.ibm.com>,
Manivannan Sadhasivam <mani@kernel.org>,
piotr.piorkowski@intel.com, Guenter Roeck <linux@roeck-us.net>,
linux-pci <linux-pci@vger.kernel.org>,
stable@vger.kernel.org
Subject: Re: [PATCH v16 5/5] s390/pci: Fix circular/recursive deadlocks in PCI-bus and -device release
Date: Wed, 18 Nov 2026 11:09:01 +0100 [thread overview]
Message-ID: <ccb3af54ee02d7cb56871d55e977502c9845b6d5.camel@linux.ibm.com> (raw)
In-Reply-To: <4c7d6e0ca4772a80059e9feb8e4b113154f420c3.1790945242.git.bblock@linux.ibm.com>
On Fri, 2026-10-02 at 14:55 +0200, Benjamin Block wrote:
> When removing PCI device or PCI bus objects there are a couple of
> call-chains where it is possible that the kernel runs into a circular
> deadlock involving the central `pci_rescan_remove_lock`.
>
> Two examples:
>
> (A) Thread α receives a PCI event notifying the kernel that a PCI
> virtual function has been moved into Reserved state, and so the PCI
> subsystem will try to remove that PCI function. The call-chain for that
> looks like this:
> __zpci_event_availability()
> -> zpci_zdev_put() # will lock(zpci_add_remove_lock),
> # and lock(zpci_list_lock)
> -> zpci_release_device() # will unlock(zpci_list_lock)
> -> zpci_cleanup_bus_resources() # will lock(pci_rescan_remove_lock)
>
> Thread β is triggered by userspace writing 0 into the SysFS attribute
> `sriov_numvfs` of the parent PCI physical function of the same function
> we just try to remove. This will also try to release the PCI virtual
> function; but this time the call-chain looks like this:
> sriov_numvfs_store()
> -> ... (device driver dependent)
> -> sriov_disable()
> -> sriov_del_vfs() # will lock(pci_rescan_remove_lock)
> -> ... (deep chain)
> -> pci_release_dev()
> -> pcibios_release_device()
> -> zpci_zdev_put() # will lock(zpci_add_remove_lock)
>
> If thread α and β coincide, this will result in a cyclic deadlock.
>
> (B) Thread γ receives a PCI event notifying the kernel that one or more
> PCI functions got hot plugged, and need to be configured. A possible
> call-chain that might happen while the PCI subsystem is trying to add
> those new function looks like this:
> __zpci_event_availability()
> -> ... (multiple ways in which a device gets added)
> -> zpci_add_device() # will lock(zpci_add_remove_lock)
> -> zpci_bus_device_register()
> -> zpci_bus_put() # will lock(zbus_list_lock)
> -> zpci_bus_release() # will unlock(zbus_list_lock)
> # will lock(pci_rescan_remove_lock)
>
> Now the same thread β as above in (A) might coincide, and again could
> result in a cyclic deadlock.
>
> `pci_rescan_remove_lock` has to be and is taken at a "high level" in
> most call-chains since it is intended to protect/mutual exclude all
> rescan and/or removal actions taken in the PCI subsystem. So to prevent
> the outlined deadlock scenarios above remove it instead from the "low
> level" release functions for both the PCI device and PCI bus objects.
>
> Instead, lock `pci_rescan_remove_lock` in all call-chains leading to
> those release functions:
> * initialization of the PCI subsystem;
> * processing of availability events (CRWs) for PCI functions;
> * processing of error events (CRWs) for PCI functions;
> * architecture specific release PCI device implementation.
>
> Additionally, remove `pci_rescan_remove_lock` from zpci_bus_scan_bus()
> since its only caller zpci_scan_devices() is now always called with
> `pci_rescan_remove_lock` already held.
>
> Lastly, document the new locking expectations after these changes. Add
> sparse and lockdep annotations to functions that previously locked
> `pci_rescan_remove_lock` explicitly, making sure the lock is now
> already held when called. Additionally also add the annotations to
> zpci_zdev_put() and zpci_bus_put() to make sure that every function that
> potentially drops the last reference already holds the lock to prevent
> surprises.
>
> Fixes: 05bc1be6db4b2 ("s390/pci: create zPCI bus")
> Fixes: ab909509850b2 ("PCI: s390: Fix use-after-free of PCI resources with per-function hotplug")
> Cc: stable@vger.kernel.org
> Signed-off-by: Benjamin Block <bblock@linux.ibm.com>
Thank you for the very detailed analysis. I agree with it and the fix
approach.
> ---
> arch/s390/pci/pci.c | 10 ++++++++--
> arch/s390/pci/pci_bus.c | 15 ++++++++-------
> arch/s390/pci/pci_event.c | 29 +++++++++++++++++++----------
> arch/s390/pci/pci_iov.c | 3 +--
> arch/s390/pci/pci_sysfs.c | 25 ++++++++++++-------------
> 5 files changed, 48 insertions(+), 34 deletions(-)
--- snip ---
>
> diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
> index 36a4807285fa..c1b48b572e86 100644
> --- a/arch/s390/pci/pci_bus.c
> +++ b/arch/s390/pci/pci_bus.c
> @@ -82,9 +82,8 @@ int zpci_bus_scan_device(struct zpci_dev *zdev)
> if (!pdev)
> return -ENODEV;
>
> - pci_lock_rescan_remove();
> + guard(pci_rescan_remove)();
> pci_bus_add_device(pdev);
> - pci_unlock_rescan_remove();
This is a pre-existing issue not introduced by your patches, but I
think pci_scan_single_device() further up in zpci_bus_scan_device()
should also be done under the pci_rescan_remove lock.
>
> return 0;
> }
> @@ -132,10 +131,13 @@ void zpci_bus_remove_device(struct zpci_dev *zdev, bool set_error)
> * Return: 0 on success, an error value otherwise
> */
> int zpci_bus_scan_bus(struct zpci_bus *zbus)
> + __must_hold(&pci_rescan_remove_lock)
> {
> struct zpci_dev *zdev;
> int devfn, rc, ret = 0;
>
> + lockdep_assert_held(&pci_rescan_remove_lock);
> +
> for (devfn = 0; devfn < ZPCI_FUNCTIONS_PER_BUS; devfn++) {
> zdev = zbus->function[devfn];
> if (zdev && zdev->state == ZPCI_FN_STATE_CONFIGURED) {
> @@ -145,10 +147,8 @@ int zpci_bus_scan_bus(struct zpci_bus *zbus)
> }
> }
>
> - pci_lock_rescan_remove();
> pci_scan_child_bus(zbus->bus);
> pci_bus_add_devices(zbus->bus);
> - pci_unlock_rescan_remove();
>
> return ret;
> }
--- snip ---
Apart from the pre-existing issue this all looks good to me. The
remaining issue is close enough in nature that I think it should be
part of this patch and it does become simpler due to the guard(). The
addition is simple enough that you can go ahead and add my:
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Thanks,
Niklas
next prev parent reply other threads:[~2026-11-18 10:10 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-24 16:29 [PATCH v15 0/5] PCI/IOV: Fix SR-IOV locking races and AB-BA deadlocks Benjamin Block
2026-09-24 16:29 ` [PATCH v15 1/5] PCI/IOV: Make pci_lock_rescan_remove() reentrant and protect sriov_add_vfs/sriov_del_vfs Benjamin Block
2026-11-11 13:28 ` Niklas Schnelle
2026-09-24 16:29 ` [PATCH v15 2/5] PCI: Fix AB-BA deadlock between device_lock and pci_rescan_remove_lock in remove_store Benjamin Block
2026-09-24 16:29 ` [PATCH v15 3/5] PCI: Move declaration of pci_rescan_remove_lock into public pci.h Benjamin Block
2026-09-24 16:29 ` [PATCH v15 4/5] PCI: Provide lock guard for pci_rescan_remove_lock Benjamin Block
2026-09-24 16:29 ` [PATCH v15 5/5] s390/pci: Fix circular/recursive deadlocks in PCI-bus and -device release Benjamin Block
2026-10-02 12:55 ` [PATCH v16 " Benjamin Block
2026-11-18 10:09 ` Niklas Schnelle [this message]
2026-11-04 11:16 ` [PATCH v15 0/5] PCI/IOV: Fix SR-IOV locking races and AB-BA deadlocks Benjamin Block
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ccb3af54ee02d7cb56871d55e977502c9845b6d5.camel@linux.ibm.com \
--to=schnelle@linux.ibm.com \
--cc=agordeev@linux.ibm.com \
--cc=alifm@linux.ibm.com \
--cc=bblock@linux.ibm.com \
--cc=bebl@ategam.org \
--cc=bhelgaas@google.com \
--cc=borntraeger@linux.ibm.com \
--cc=dtatulea@nvidia.com \
--cc=gbayer@linux.ibm.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=ionut.nechita@windriver.com \
--cc=ionut_n2001@yahoo.com \
--cc=julianr@linux.ibm.com \
--cc=kbusch@kernel.org \
--cc=krebbel@linux.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux@roeck-us.net \
--cc=lukas@wunner.de \
--cc=mani@kernel.org \
--cc=matthew.brost@intel.com \
--cc=michal.wajdeczko@intel.com \
--cc=mjrosato@linux.ibm.com \
--cc=oelghoul@linux.ibm.com \
--cc=pasic@linux.ibm.com \
--cc=piotr.piorkowski@intel.com \
--cc=ramesh@linux.ibm.com \
--cc=stable@vger.kernel.org \
--cc=sunlightlinux@gmail.com \
--cc=svens@linux.ibm.com \
--cc=ts@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®