mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Niklas Schnelle <schnelle@linux.ibm.com>
To: Benjamin Block <bblock@linux.ibm.com>,
	Benjamin Block <bebl@ategam.org>,
	Bjorn Helgaas <bhelgaas@google.com>
Cc: Farhan Ali <alifm@linux.ibm.com>,
	linux-kernel	 <linux-kernel@vger.kernel.org>,
	Heiko Carstens <hca@linux.ibm.com>,
	Dragos Tatulea <dtatulea@nvidia.com>,
	Omar Elghoul <oelghoul@linux.ibm.com>,
	Lukas Wunner <lukas@wunner.de>,
	Matthew Brost <matthew.brost@intel.com>,
	Halil Pasic <pasic@linux.ibm.com>,
	Ionut Nechita	 <ionut.nechita@windriver.com>,
	Sven Schnelle <svens@linux.ibm.com>,
	Alexander Gordeev <agordeev@linux.ibm.com>,
	Ionut Nechita <ionut_n2001@yahoo.com>,
	Tobias Schumacher <ts@linux.ibm.com>,
	Julian Ruess <julianr@linux.ibm.com>,
	Ionut Nechita <sunlightlinux@gmail.com>,
	Michal Wajdeczko	 <michal.wajdeczko@intel.com>,
	Keith Busch <kbusch@kernel.org>,
	Matthew Rosato	 <mjrosato@linux.ibm.com>,
	Christian Borntraeger <borntraeger@linux.ibm.com>,
	Gerd Bayer <gbayer@linux.ibm.com>,
	linux-intel-xe <intel-xe@lists.freedesktop.org>,
	Andreas Krebbel	 <krebbel@linux.ibm.com>,
	Ramesh Errabolu <ramesh@linux.ibm.com>,
	linux-s390 <linux-s390@vger.kernel.org>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Manivannan Sadhasivam <mani@kernel.org>,
	piotr.piorkowski@intel.com, Guenter Roeck <linux@roeck-us.net>,
	linux-pci	 <linux-pci@vger.kernel.org>,
	stable@vger.kernel.org
Subject: Re: [PATCH v16 5/5] s390/pci: Fix circular/recursive deadlocks in PCI-bus and -device release
Date: Wed, 18 Nov 2026 11:09:01 +0100	[thread overview]
Message-ID: <ccb3af54ee02d7cb56871d55e977502c9845b6d5.camel@linux.ibm.com> (raw)
In-Reply-To: <4c7d6e0ca4772a80059e9feb8e4b113154f420c3.1790945242.git.bblock@linux.ibm.com>

On Fri, 2026-10-02 at 14:55 +0200, Benjamin Block wrote:
> When removing PCI device or PCI bus objects there are a couple of
> call-chains where it is possible that the kernel runs into a circular
> deadlock involving the central `pci_rescan_remove_lock`.
> 
> Two examples:
> 
> (A) Thread α receives a PCI event notifying the kernel that a PCI
> virtual function has been moved into Reserved state, and so the PCI
> subsystem will try to remove that PCI function. The call-chain for that
> looks like this:
>   __zpci_event_availability()
>     -> zpci_zdev_put()              # will   lock(zpci_add_remove_lock),
>                                     # and    lock(zpci_list_lock)
>     -> zpci_release_device()        # will unlock(zpci_list_lock)
>     -> zpci_cleanup_bus_resources() # will   lock(pci_rescan_remove_lock)
> 
> Thread β is triggered by userspace writing 0 into the SysFS attribute
> `sriov_numvfs` of the parent PCI physical function of the same function
> we just try to remove. This will also try to release the PCI virtual
> function; but this time the call-chain looks like this:
>   sriov_numvfs_store()
>     -> ... (device driver dependent)
>     -> sriov_disable()
>     -> sriov_del_vfs()              # will   lock(pci_rescan_remove_lock)
>     -> ... (deep chain)
>     -> pci_release_dev()
>     -> pcibios_release_device()
>     -> zpci_zdev_put()              # will   lock(zpci_add_remove_lock)
> 
> If thread α and β coincide, this will result in a cyclic deadlock.
> 
> (B) Thread γ receives a PCI event notifying the kernel that one or more
> PCI functions got hot plugged, and need to be configured. A possible
> call-chain that might happen while the PCI subsystem is trying to add
> those new function looks like this:
>   __zpci_event_availability()
>     -> ... (multiple ways in which a device gets added)
>     -> zpci_add_device()            # will   lock(zpci_add_remove_lock)
>     -> zpci_bus_device_register()
>     -> zpci_bus_put()               # will   lock(zbus_list_lock)
>     -> zpci_bus_release()           # will unlock(zbus_list_lock)
>                                     # will   lock(pci_rescan_remove_lock)
> 
> Now the same thread β as above in (A) might coincide, and again could
> result in a cyclic deadlock.
> 
> `pci_rescan_remove_lock` has to be and is taken at a "high level" in
> most call-chains since it is intended to protect/mutual exclude all
> rescan and/or removal actions taken in the PCI subsystem. So to prevent
> the outlined deadlock scenarios above remove it instead from the "low
> level" release functions for both the PCI device and PCI bus objects.
> 
> Instead, lock `pci_rescan_remove_lock` in all call-chains leading to
> those release functions:
>   * initialization of the PCI subsystem;
>   * processing of availability events (CRWs) for PCI functions;
>   * processing of error events (CRWs) for PCI functions;
>   * architecture specific release PCI device implementation.
> 
> Additionally, remove `pci_rescan_remove_lock` from zpci_bus_scan_bus()
> since its only caller zpci_scan_devices() is now always called with
> `pci_rescan_remove_lock` already held.
> 
> Lastly, document the new locking expectations after these changes. Add
> sparse and lockdep annotations to functions that previously locked
> `pci_rescan_remove_lock` explicitly, making sure the lock is now
> already held when called. Additionally also add the annotations to
> zpci_zdev_put() and zpci_bus_put() to make sure that every function that
> potentially drops the last reference already holds the lock to prevent
> surprises.
> 
> Fixes: 05bc1be6db4b2 ("s390/pci: create zPCI bus")
> Fixes: ab909509850b2 ("PCI: s390: Fix use-after-free of PCI resources with per-function hotplug")
> Cc: stable@vger.kernel.org
> Signed-off-by: Benjamin Block <bblock@linux.ibm.com>

Thank you for the very detailed analysis. I agree with it and the fix
approach.

> ---
>  arch/s390/pci/pci.c       | 10 ++++++++--
>  arch/s390/pci/pci_bus.c   | 15 ++++++++-------
>  arch/s390/pci/pci_event.c | 29 +++++++++++++++++++----------
>  arch/s390/pci/pci_iov.c   |  3 +--
>  arch/s390/pci/pci_sysfs.c | 25 ++++++++++++-------------
>  5 files changed, 48 insertions(+), 34 deletions(-)
--- snip ---
>  
> diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
> index 36a4807285fa..c1b48b572e86 100644
> --- a/arch/s390/pci/pci_bus.c
> +++ b/arch/s390/pci/pci_bus.c
> @@ -82,9 +82,8 @@ int zpci_bus_scan_device(struct zpci_dev *zdev)
>  	if (!pdev)
>  		return -ENODEV;
>  
> -	pci_lock_rescan_remove();
> +	guard(pci_rescan_remove)();
>  	pci_bus_add_device(pdev);
> -	pci_unlock_rescan_remove();

This is a pre-existing issue not introduced by your patches, but I
think pci_scan_single_device() further up in zpci_bus_scan_device()
should also be done under the pci_rescan_remove lock.

>  
>  	return 0;
>  }
> @@ -132,10 +131,13 @@ void zpci_bus_remove_device(struct zpci_dev *zdev, bool set_error)
>   * Return: 0 on success, an error value otherwise
>   */
>  int zpci_bus_scan_bus(struct zpci_bus *zbus)
> +	__must_hold(&pci_rescan_remove_lock)
>  {
>  	struct zpci_dev *zdev;
>  	int devfn, rc, ret = 0;
>  
> +	lockdep_assert_held(&pci_rescan_remove_lock);
> +
>  	for (devfn = 0; devfn < ZPCI_FUNCTIONS_PER_BUS; devfn++) {
>  		zdev = zbus->function[devfn];
>  		if (zdev && zdev->state == ZPCI_FN_STATE_CONFIGURED) {
> @@ -145,10 +147,8 @@ int zpci_bus_scan_bus(struct zpci_bus *zbus)
>  		}
>  	}
>  
> -	pci_lock_rescan_remove();
>  	pci_scan_child_bus(zbus->bus);
>  	pci_bus_add_devices(zbus->bus);
> -	pci_unlock_rescan_remove();
>  
>  	return ret;
>  }
--- snip ---

Apart from the pre-existing issue this all looks good to me. The
remaining issue is close enough in nature that I think it should be
part of this patch and it does become simpler due to the guard(). The
addition is simple enough that you can go ahead and add my:

Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>

Thanks,
Niklas

  reply	other threads:[~2026-11-18 10:10 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-24 16:29 [PATCH v15 0/5] PCI/IOV: Fix SR-IOV locking races and AB-BA deadlocks Benjamin Block
2026-09-24 16:29 ` [PATCH v15 1/5] PCI/IOV: Make pci_lock_rescan_remove() reentrant and protect sriov_add_vfs/sriov_del_vfs Benjamin Block
2026-11-11 13:28   ` Niklas Schnelle
2026-09-24 16:29 ` [PATCH v15 2/5] PCI: Fix AB-BA deadlock between device_lock and pci_rescan_remove_lock in remove_store Benjamin Block
2026-09-24 16:29 ` [PATCH v15 3/5] PCI: Move declaration of pci_rescan_remove_lock into public pci.h Benjamin Block
2026-09-24 16:29 ` [PATCH v15 4/5] PCI: Provide lock guard for pci_rescan_remove_lock Benjamin Block
2026-09-24 16:29 ` [PATCH v15 5/5] s390/pci: Fix circular/recursive deadlocks in PCI-bus and -device release Benjamin Block
2026-10-02 12:55   ` [PATCH v16 " Benjamin Block
2026-11-18 10:09     ` Niklas Schnelle [this message]
2026-11-04 11:16 ` [PATCH v15 0/5] PCI/IOV: Fix SR-IOV locking races and AB-BA deadlocks Benjamin Block

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ccb3af54ee02d7cb56871d55e977502c9845b6d5.camel@linux.ibm.com \
    --to=schnelle@linux.ibm.com \
    --cc=agordeev@linux.ibm.com \
    --cc=alifm@linux.ibm.com \
    --cc=bblock@linux.ibm.com \
    --cc=bebl@ategam.org \
    --cc=bhelgaas@google.com \
    --cc=borntraeger@linux.ibm.com \
    --cc=dtatulea@nvidia.com \
    --cc=gbayer@linux.ibm.com \
    --cc=gor@linux.ibm.com \
    --cc=hca@linux.ibm.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=ionut.nechita@windriver.com \
    --cc=ionut_n2001@yahoo.com \
    --cc=julianr@linux.ibm.com \
    --cc=kbusch@kernel.org \
    --cc=krebbel@linux.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=lukas@wunner.de \
    --cc=mani@kernel.org \
    --cc=matthew.brost@intel.com \
    --cc=michal.wajdeczko@intel.com \
    --cc=mjrosato@linux.ibm.com \
    --cc=oelghoul@linux.ibm.com \
    --cc=pasic@linux.ibm.com \
    --cc=piotr.piorkowski@intel.com \
    --cc=ramesh@linux.ibm.com \
    --cc=stable@vger.kernel.org \
    --cc=sunlightlinux@gmail.com \
    --cc=svens@linux.ibm.com \
    --cc=ts@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®