mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH] vfio/pci: Only lock the upstream bridge if the reset may need it
@ 2026-11-12 23:57 Samiullah Khawaja
  2026-11-19  0:37 ` Alex Williamson
  0 siblings, 1 reply; 3+ messages in thread
From: Samiullah Khawaja @ 2026-11-12 23:57 UTC (permalink / raw)
  To: Alex Williamson, Bjorn Helgaas
  Cc: Kevin Tian, Jason Gunthorpe, Anthony Pighin, Dave Jiang,
	Dan Williams, kvm, linux-pci, linux-kernel, Samiullah Khawaja,
	stable

Commit 962ae6892d8b ("vfio/pci: Lock upstream bridge for
vfio_pci_core_disable()") makes vfio_pci_core_disable() trylock the
upstream bridge before resetting the device, and skips the reset if the
trylock fails.

All VFs of a PF have the same upstream bridge. Take a device with two
VFs, both used by one process. When the process is killed, both VFs can
be released at the same time. The first one takes the bridge lock and
does its FLR, the second one fails the trylock and is never reset.

The bridge lock is only needed for a bus reset, which is done through
the bridge. A VF is reset with FLR, and pci_parent_bus_reset() never
resets the bus for it because the PF and the other VFs are on the same
bus.

Add pci_reset_needs_bridge_lock(), which returns true if any reset
method enabled for the device is a bus reset, and only lock the bridge
in vfio_pci_core_disable() when it does.

Fixes: 962ae6892d8b ("vfio/pci: Lock upstream bridge for vfio_pci_core_disable()")
Cc: stable@vger.kernel.org
Signed-off-by: Samiullah Khawaja <skhawaja@google.com>
---
Tested in QEMU with an igb PF behind a PCIe root port, bound to vfio-pci
with two VFs. Both VFs are opened through VFIO and then closed from two
threads at the same time, counting FLRs with a kprobe on pcie_flr().
Without this patch one VF missed its FLR in 50 of 50 rounds, with it
both VFs were reset in every round. I will send a VFIO selftest for this
separately.
---
 drivers/pci/pci.c                | 28 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c |  3 ++-
 include/linux/pci.h              |  1 +
 3 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 7c6f2aba8144..c44e4d4d8774 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -5256,6 +5256,34 @@ int __pci_reset_function_locked(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(__pci_reset_function_locked);
 
+/**
+ * pci_reset_needs_bridge_lock - check whether a reset of @dev may need the
+ * upstream bridge to be locked
+ * @dev: PCI device to check
+ *
+ * A bus reset of @dev is done through its upstream bridge, so the caller must
+ * hold the bridge lock. The other reset methods only access @dev itself.
+ *
+ * Return: true if any of the reset methods enabled for @dev is a bus reset.
+ */
+bool pci_reset_needs_bridge_lock(struct pci_dev *dev)
+{
+	int i, m;
+
+	for (i = 0; i < PCI_NUM_RESET_METHODS; i++) {
+		m = dev->reset_methods[i];
+		if (!m)
+			break;
+
+		if (pci_reset_fn_methods[m].reset_fn == pci_reset_bus_function ||
+		    pci_reset_fn_methods[m].reset_fn == cxl_reset_bus_function)
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(pci_reset_needs_bridge_lock);
+
 /**
  * pci_init_reset_methods - check whether device can be safely reset
  * and store supported reset mechanisms.
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..ba3dfdadae39 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -777,7 +777,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	 * overwrite the previously restored configuration information.
 	 */
 	if (vdev->reset_works) {
-		bridge = pci_upstream_bridge(pdev);
+		bridge = pci_reset_needs_bridge_lock(pdev) ?
+			 pci_upstream_bridge(pdev) : NULL;
 		if (bridge && !pci_dev_trylock(bridge))
 			goto out_restore_state;
 		if (pci_dev_trylock(pdev)) {
diff --git a/include/linux/pci.h b/include/linux/pci.h
index aa3765a20a96..cb2856f444ca 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1486,6 +1486,7 @@ void pcie_print_link_status(struct pci_dev *dev);
 int pcie_reset_flr(struct pci_dev *dev, bool probe);
 int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
+bool pci_reset_needs_bridge_lock(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
 int pci_reset_function_locked(struct pci_dev *dev);
 int pci_try_reset_function(struct pci_dev *dev);

base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
-- 
2.56.0.611.g1030f0b940-goog


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] vfio/pci: Only lock the upstream bridge if the reset may need it
  2026-11-12 23:57 [PATCH] vfio/pci: Only lock the upstream bridge if the reset may need it Samiullah Khawaja
@ 2026-11-19  0:37 ` Alex Williamson
  2026-11-19 22:32   ` Alex Williamson
  0 siblings, 1 reply; 3+ messages in thread
From: Alex Williamson @ 2026-11-19  0:37 UTC (permalink / raw)
  To: Samiullah Khawaja
  Cc: Bjorn Helgaas, Kevin Tian, Jason Gunthorpe, Anthony Pighin,
	Dave Jiang, Dan Williams, kvm, linux-pci, linux-kernel, stable,
	alex

On Thu, 12 Nov 2026 23:57:00 +0000
Samiullah Khawaja <skhawaja@google.com> wrote:

> Commit 962ae6892d8b ("vfio/pci: Lock upstream bridge for
> vfio_pci_core_disable()") makes vfio_pci_core_disable() trylock the
> upstream bridge before resetting the device, and skips the reset if the
> trylock fails.
> 
> All VFs of a PF have the same upstream bridge. Take a device with two
> VFs, both used by one process. When the process is killed, both VFs can
> be released at the same time. The first one takes the bridge lock and
> does its FLR, the second one fails the trylock and is never reset.
> 
> The bridge lock is only needed for a bus reset, which is done through
> the bridge. A VF is reset with FLR, and pci_parent_bus_reset() never
> resets the bus for it because the PF and the other VFs are on the same
> bus.
> 
> Add pci_reset_needs_bridge_lock(), which returns true if any reset
> method enabled for the device is a bus reset, and only lock the bridge
> in vfio_pci_core_disable() when it does.
> 
> Fixes: 962ae6892d8b ("vfio/pci: Lock upstream bridge for vfio_pci_core_disable()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Samiullah Khawaja <skhawaja@google.com>
> ---
> Tested in QEMU with an igb PF behind a PCIe root port, bound to vfio-pci
> with two VFs. Both VFs are opened through VFIO and then closed from two
> threads at the same time, counting FLRs with a kprobe on pcie_flr().
> Without this patch one VF missed its FLR in 50 of 50 rounds, with it
> both VFs were reset in every round. I will send a VFIO selftest for this
> separately.
> ---
>  drivers/pci/pci.c                | 28 ++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_core.c |  3 ++-
>  include/linux/pci.h              |  1 +
>  3 files changed, 31 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 7c6f2aba8144..c44e4d4d8774 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -5256,6 +5256,34 @@ int __pci_reset_function_locked(struct pci_dev *dev)
>  }
>  EXPORT_SYMBOL_GPL(__pci_reset_function_locked);
>  
> +/**
> + * pci_reset_needs_bridge_lock - check whether a reset of @dev may need the
> + * upstream bridge to be locked
> + * @dev: PCI device to check
> + *
> + * A bus reset of @dev is done through its upstream bridge, so the caller must
> + * hold the bridge lock. The other reset methods only access @dev itself.
> + *
> + * Return: true if any of the reset methods enabled for @dev is a bus reset.
> + */
> +bool pci_reset_needs_bridge_lock(struct pci_dev *dev)
> +{
> +	int i, m;
> +
> +	for (i = 0; i < PCI_NUM_RESET_METHODS; i++) {
> +		m = dev->reset_methods[i];
> +		if (!m)
> +			break;
> +
> +		if (pci_reset_fn_methods[m].reset_fn == pci_reset_bus_function ||
> +		    pci_reset_fn_methods[m].reset_fn == cxl_reset_bus_function)

This looks like a maintenance nightmare.  What if instead we did
something like:

bool pci_parent_bus_reset_possible(struct pci_dev *dev)
{
	struct pci_dev *pdev;
   
	if (pci_is_root_bus(dev->bus) || dev->subordinate ||
	    !dev->bus->self || dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET)
		return false;

	list_for_each_entry(pdev, &dev->bus->devices, bus_list)
		if (pdev != dev)
			return false;

	return true;
}
EXPORT_SYMBOL_GPL(pci_parent_bus_reset_possible);
  
static int pci_parent_bus_reset(struct pci_dev *dev, bool probe)
{
	if (!pci_parent_bus_reset_possible(dev))
		return -ENOTTY;

	if (probe)
		return 0;

	return pci_bridge_secondary_bus_reset(dev->bus->self);
}

Where we'd then condition acquiring the bridge lock on whether it's
even possible for a function scoped reset to perform a bus reset.

It's a bit broader of a test than whether we actually have the necessary
methods in our reset_fn list, but since we're not locking the reset_fn
list between test and function-reset, it's also safer.  Thanks,

Alex

> +			return true;
> +	}
> +
> +	return false;
> +}
> +EXPORT_SYMBOL_GPL(pci_reset_needs_bridge_lock);
> +
>  /**
>   * pci_init_reset_methods - check whether device can be safely reset
>   * and store supported reset mechanisms.
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index fcbb2da860e2..ba3dfdadae39 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -777,7 +777,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
>  	 * overwrite the previously restored configuration information.
>  	 */
>  	if (vdev->reset_works) {
> -		bridge = pci_upstream_bridge(pdev);
> +		bridge = pci_reset_needs_bridge_lock(pdev) ?
> +			 pci_upstream_bridge(pdev) : NULL;
>  		if (bridge && !pci_dev_trylock(bridge))
>  			goto out_restore_state;
>  		if (pci_dev_trylock(pdev)) {
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index aa3765a20a96..cb2856f444ca 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1486,6 +1486,7 @@ void pcie_print_link_status(struct pci_dev *dev);
>  int pcie_reset_flr(struct pci_dev *dev, bool probe);
>  int pcie_flr(struct pci_dev *dev);
>  int __pci_reset_function_locked(struct pci_dev *dev);
> +bool pci_reset_needs_bridge_lock(struct pci_dev *dev);
>  int pci_reset_function(struct pci_dev *dev);
>  int pci_reset_function_locked(struct pci_dev *dev);
>  int pci_try_reset_function(struct pci_dev *dev);
> 
> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] vfio/pci: Only lock the upstream bridge if the reset may need it
  2026-11-19  0:37 ` Alex Williamson
@ 2026-11-19 22:32   ` Alex Williamson
  0 siblings, 0 replies; 3+ messages in thread
From: Alex Williamson @ 2026-11-19 22:32 UTC (permalink / raw)
  To: Samiullah Khawaja
  Cc: Bjorn Helgaas, Kevin Tian, Jason Gunthorpe, Anthony Pighin,
	Dave Jiang, Dan Williams, kvm, linux-pci, linux-kernel, stable,
	alex

On Wed, 18 Nov 2026 16:37:06 -0800
Alex Williamson <alex@shazbot.org> wrote:

> On Thu, 12 Nov 2026 23:57:00 +0000
> Samiullah Khawaja <skhawaja@google.com> wrote:
> 
> > Commit 962ae6892d8b ("vfio/pci: Lock upstream bridge for
> > vfio_pci_core_disable()") makes vfio_pci_core_disable() trylock the
> > upstream bridge before resetting the device, and skips the reset if the
> > trylock fails.
> > 
> > All VFs of a PF have the same upstream bridge. Take a device with two
> > VFs, both used by one process. When the process is killed, both VFs can
> > be released at the same time. The first one takes the bridge lock and
> > does its FLR, the second one fails the trylock and is never reset.
> > 
> > The bridge lock is only needed for a bus reset, which is done through
> > the bridge. A VF is reset with FLR, and pci_parent_bus_reset() never
> > resets the bus for it because the PF and the other VFs are on the same
> > bus.
> > 
> > Add pci_reset_needs_bridge_lock(), which returns true if any reset
> > method enabled for the device is a bus reset, and only lock the bridge
> > in vfio_pci_core_disable() when it does.
> > 
> > Fixes: 962ae6892d8b ("vfio/pci: Lock upstream bridge for vfio_pci_core_disable()")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Samiullah Khawaja <skhawaja@google.com>
> > ---
> > Tested in QEMU with an igb PF behind a PCIe root port, bound to vfio-pci
> > with two VFs. Both VFs are opened through VFIO and then closed from two
> > threads at the same time, counting FLRs with a kprobe on pcie_flr().
> > Without this patch one VF missed its FLR in 50 of 50 rounds, with it
> > both VFs were reset in every round. I will send a VFIO selftest for this
> > separately.
> > ---
> >  drivers/pci/pci.c                | 28 ++++++++++++++++++++++++++++
> >  drivers/vfio/pci/vfio_pci_core.c |  3 ++-
> >  include/linux/pci.h              |  1 +
> >  3 files changed, 31 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index 7c6f2aba8144..c44e4d4d8774 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -5256,6 +5256,34 @@ int __pci_reset_function_locked(struct pci_dev *dev)
> >  }
> >  EXPORT_SYMBOL_GPL(__pci_reset_function_locked);
> >  
> > +/**
> > + * pci_reset_needs_bridge_lock - check whether a reset of @dev may need the
> > + * upstream bridge to be locked
> > + * @dev: PCI device to check
> > + *
> > + * A bus reset of @dev is done through its upstream bridge, so the caller must
> > + * hold the bridge lock. The other reset methods only access @dev itself.
> > + *
> > + * Return: true if any of the reset methods enabled for @dev is a bus reset.
> > + */
> > +bool pci_reset_needs_bridge_lock(struct pci_dev *dev)
> > +{
> > +	int i, m;
> > +
> > +	for (i = 0; i < PCI_NUM_RESET_METHODS; i++) {
> > +		m = dev->reset_methods[i];
> > +		if (!m)
> > +			break;
> > +
> > +		if (pci_reset_fn_methods[m].reset_fn == pci_reset_bus_function ||
> > +		    pci_reset_fn_methods[m].reset_fn == cxl_reset_bus_function)  
> 
> This looks like a maintenance nightmare.  What if instead we did
> something like:
> 
> bool pci_parent_bus_reset_possible(struct pci_dev *dev)
> {
> 	struct pci_dev *pdev;
>    
> 	if (pci_is_root_bus(dev->bus) || dev->subordinate ||
> 	    !dev->bus->self || dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET)
> 		return false;
> 
> 	list_for_each_entry(pdev, &dev->bus->devices, bus_list)
> 		if (pdev != dev)
> 			return false;
> 
> 	return true;
> }
> EXPORT_SYMBOL_GPL(pci_parent_bus_reset_possible);
>   
> static int pci_parent_bus_reset(struct pci_dev *dev, bool probe)
> {
> 	if (!pci_parent_bus_reset_possible(dev))
> 		return -ENOTTY;
> 
> 	if (probe)
> 		return 0;
> 
> 	return pci_bridge_secondary_bus_reset(dev->bus->self);
> }
> 
> Where we'd then condition acquiring the bridge lock on whether it's
> even possible for a function scoped reset to perform a bus reset.
> 
> It's a bit broader of a test than whether we actually have the necessary
> methods in our reset_fn list, but since we're not locking the reset_fn
> list between test and function-reset, it's also safer.  Thanks,

After exploring Ionut's patch[1], what I propose above is probably
wrong.  We don't have any guarantees that the sibling topology is
static and this bus-reset-possible function presents it as immutable.
The existing bus-reset function has the same race, but the gap is
smaller, let's not make it bigger.

There are however immutable properties we can test that cover a fair
number of cases.  For example:

bool pci_reset_function_needs_bridge_lock(struct pci_dev *dev)
{
	return pci_upstream_bridge(dev) &&
		!(dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET) && !dev->is_virtfn;
}
EXPORT_SYMBOL_GPL(pci_reset_function_needs_bridge_lock);

We should be able to use this as a condition on whether we acquire the
bridge device lock, in pci_reset_function() and in vfio-pci code.
Thanks,

Alex

[1]https://lore.kernel.org/all/20261119123504.25906-2-ionut.nechita@windriver.com/

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-19 22:32 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-12 23:57 [PATCH] vfio/pci: Only lock the upstream bridge if the reset may need it Samiullah Khawaja
2026-11-19  0:37 ` Alex Williamson
2026-11-19 22:32   ` Alex Williamson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®