* [PATCH V1 1/4] PCI/CXL: Don't offer 'cxl_bus' reset where a bus reset can't work
2026-11-19 10:06 [PATCH V1 0/4] PCI: Don't list reset methods that can't reset the device Vidya Sagar
@ 2026-11-19 10:06 ` Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 2/4] PCI: Don't run IOMMU reset hooks when only probing reset methods Vidya Sagar
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Vidya Sagar @ 2026-11-19 10:06 UTC (permalink / raw)
To: Bjorn Helgaas, Dan Williams, Dave Jiang, Jonathan Cameron,
Joerg Roedel, Jason Gunthorpe, Nicolin Chen, Kevin Tian
Cc: vsethi, kthota, mmaddireddy, kumarahul, sagar.tv, linux-pci,
linux-kernel, Vidya Sagar
cxl_reset_bus_function() reports the "cxl_bus" reset method as supported
whenever the device's upstream bridge has a CXL Port DVSEC. The reset
itself goes through pci_reset_bus_function(), which can only reset the
device through a hotplug slot or a Secondary Bus Reset of the parent
bridge. It returns -ENOTTY when neither is possible, e.g., when other
functions share the device's bus, when the device is a bridge, or when it
has PCI_DEV_FLAGS_NO_BUS_RESET.
"cxl_bus" is then listed in "reset_method" but always fails. For example,
on function 1 of a 4-function ConnectX-8 below a CXL-capable Root Port
without a hotplug slot, "bus" is correctly not offered, but "cxl_bus" is:
# cat /sys/bus/pci/devices/0001:01:00.1/reset_method
flr cxl_bus
# echo cxl_bus > /sys/bus/pci/devices/0001:01:00.1/reset_method
# echo 1 > /sys/bus/pci/devices/0001:01:00.1/reset
bash: echo: write error: Inappropriate ioctl for device
Each attempt still temporarily sets "Unmask SBR" in the Root Port and runs
the IOMMU reset prepare/done hooks, although no reset happens.
Move the slot and parent bus reset from pci_reset_bus_function() into a
new __pci_reset_bus_function() and use it for the "cxl_bus" probe. The
probe now makes the same slot and parent-bus checks as the "bus" probe.
"cxl_bus" is then offered under the same conditions as "bus", except for
the CXL SBR mask that "cxl_bus" exists to clear.
Fixes: 53c49b6e6dd2 ("PCI/CXL: Add 'cxl_bus' reset method for devices below CXL Ports")
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
drivers/pci/pci.c | 27 +++++++++++++++++++++------
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 7c6f2aba8144..921d076c4c1d 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -5030,6 +5030,21 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
return true;
}
+/*
+ * Reset "dev" through its hotplug slot or, failing that, a Secondary Bus
+ * Reset of its parent bridge. Callers deal with CXL SBR masking.
+ */
+static int __pci_reset_bus_function(struct pci_dev *dev, bool probe)
+{
+ int rc;
+
+ rc = pci_dev_reset_slot_function(dev, probe);
+ if (rc != -ENOTTY)
+ return rc;
+
+ return pci_parent_bus_reset(dev, probe);
+}
+
static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
{
struct pci_dev *bridge = pci_upstream_bridge(dev);
@@ -5048,12 +5063,8 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
return rc;
}
- rc = pci_dev_reset_slot_function(dev, probe);
- if (rc != -ENOTTY)
- goto done;
+ rc = __pci_reset_bus_function(dev, probe);
- rc = pci_parent_bus_reset(dev, probe);
-done:
pci_dev_reset_iommu_done(dev);
return rc;
}
@@ -5072,8 +5083,12 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
if (!dvsec)
return -ENOTTY;
+ /*
+ * Unmasking SBR only helps if a slot or bus reset can reach "dev" at
+ * all, e.g., not when other functions share its bus.
+ */
if (probe)
- return 0;
+ return __pci_reset_bus_function(dev, probe);
rc = pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL, ®);
if (rc)
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH V1 2/4] PCI: Don't run IOMMU reset hooks when only probing reset methods
2026-11-19 10:06 [PATCH V1 0/4] PCI: Don't list reset methods that can't reset the device Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 1/4] PCI/CXL: Don't offer 'cxl_bus' reset where a bus reset can't work Vidya Sagar
@ 2026-11-19 10:06 ` Vidya Sagar
2026-11-19 16:39 ` Nicolin Chen
2026-11-19 10:06 ` [PATCH V1 3/4] PCI/sysfs: Take the device lock when setting " Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 4/4] PCI: Re-check function 0 reset methods after scanning its slot Vidya Sagar
3 siblings, 1 reply; 6+ messages in thread
From: Vidya Sagar @ 2026-11-19 10:06 UTC (permalink / raw)
To: Bjorn Helgaas, Dan Williams, Dave Jiang, Jonathan Cameron,
Joerg Roedel, Jason Gunthorpe, Nicolin Chen, Kevin Tian
Cc: vsethi, kthota, mmaddireddy, kumarahul, sagar.tv, linux-pci,
linux-kernel, Vidya Sagar
pci_reset_bus_function() and __pci_dev_specific_reset() call
pci_dev_reset_iommu_prepare() and pci_dev_reset_iommu_done() even when
"probe" is set and no reset follows. The other reset methods that use
these hooks, pcie_flr(), pci_af_flr(), pci_pm_reset() and
pci_dev_acpi_reset(), only call them when actually resetting.
Reset methods are probed at enumeration, before the IOMMU is set up, but
also whenever "reset_method" is written in sysfs. For an ATS-capable
device behind an IOMMU, each such probe of "bus" or of a device-specific
reset attaches the blocking domain and then re-attaches the device's
domains. That blocks any DMA in the window and, on arm-smmu-v3, sends
ATC invalidations, even while a driver is bound and using the device.
Return before the IOMMU hooks when probing, as the other reset methods
do.
Fixes: f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device")
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
drivers/pci/pci.c | 3 +++
drivers/pci/quirks.c | 3 +++
2 files changed, 6 insertions(+)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 921d076c4c1d..918a4d95f914 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -5057,6 +5057,9 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
if (bridge && pcie_is_cxl(bridge) && cxl_sbr_masked(bridge))
return -ENOTTY;
+ if (probe)
+ return __pci_reset_bus_function(dev, probe);
+
rc = pci_dev_reset_iommu_prepare(dev);
if (rc) {
pci_err(dev, "failed to stop IOMMU for a PCI reset: %d\n", rc);
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 7f1f47ea2ee4..79c326b0e80f 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -4406,6 +4406,9 @@ static int __pci_dev_specific_reset(struct pci_dev *dev, bool probe,
{
int ret;
+ if (probe)
+ return i->reset(dev, probe);
+
ret = pci_dev_reset_iommu_prepare(dev);
if (ret) {
pci_err(dev, "failed to stop IOMMU for a PCI reset: %d\n", ret);
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH V1 2/4] PCI: Don't run IOMMU reset hooks when only probing reset methods
2026-11-19 10:06 ` [PATCH V1 2/4] PCI: Don't run IOMMU reset hooks when only probing reset methods Vidya Sagar
@ 2026-11-19 16:39 ` Nicolin Chen
0 siblings, 0 replies; 6+ messages in thread
From: Nicolin Chen @ 2026-11-19 16:39 UTC (permalink / raw)
To: Vidya Sagar
Cc: Bjorn Helgaas, Dan Williams, Dave Jiang, Jonathan Cameron,
Joerg Roedel, Jason Gunthorpe, Kevin Tian, vsethi, kthota,
mmaddireddy, kumarahul, sagar.tv, linux-pci, linux-kernel
On Thu, Nov 19, 2026 at 03:36:26PM +0530, Vidya Sagar wrote:
> pci_reset_bus_function() and __pci_dev_specific_reset() call
> pci_dev_reset_iommu_prepare() and pci_dev_reset_iommu_done() even when
> "probe" is set and no reset follows. The other reset methods that use
> these hooks, pcie_flr(), pci_af_flr(), pci_pm_reset() and
> pci_dev_acpi_reset(), only call them when actually resetting.
>
> Reset methods are probed at enumeration, before the IOMMU is set up, but
> also whenever "reset_method" is written in sysfs. For an ATS-capable
> device behind an IOMMU, each such probe of "bus" or of a device-specific
> reset attaches the blocking domain and then re-attaches the device's
> domains. That blocks any DMA in the window and, on arm-smmu-v3, sends
> ATC invalidations, even while a driver is bound and using the device.
>
> Return before the IOMMU hooks when probing, as the other reset methods
> do.
>
> Fixes: f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device")
We already have a similar fix here that Bjorn acked:
https://lore.kernel.org/linux-iommu/c66d5221a8e4356316d4f614c8584a80dc9d71b6.1793729707.git.nicolinc@nvidia.com/
I am waiting for Joerg to take the series.
Nicolin
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH V1 3/4] PCI/sysfs: Take the device lock when setting reset methods
2026-11-19 10:06 [PATCH V1 0/4] PCI: Don't list reset methods that can't reset the device Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 1/4] PCI/CXL: Don't offer 'cxl_bus' reset where a bus reset can't work Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 2/4] PCI: Don't run IOMMU reset hooks when only probing reset methods Vidya Sagar
@ 2026-11-19 10:06 ` Vidya Sagar
2026-11-19 10:06 ` [PATCH V1 4/4] PCI: Re-check function 0 reset methods after scanning its slot Vidya Sagar
3 siblings, 0 replies; 6+ messages in thread
From: Vidya Sagar @ 2026-11-19 10:06 UTC (permalink / raw)
To: Bjorn Helgaas, Dan Williams, Dave Jiang, Jonathan Cameron,
Joerg Roedel, Jason Gunthorpe, Nicolin Chen, Kevin Tian
Cc: vsethi, kthota, mmaddireddy, kumarahul, sagar.tv, linux-pci,
linux-kernel, Vidya Sagar
reset_method_store() updates dev->reset_methods without holding any
lock. A reset can run at the same time: __pci_reset_function_locked()
walks the list one entry at a time while the write copies the new list
over it, so the reset may use a mix of the old and the new list.
Take the device lock in reset_method_store(), as
__pci_reset_function_locked() already requires. This also serializes
concurrent writes. The reset methods already run under this lock during
a reset, and probing a method does less than running it, so probing them
with the lock held is safe. A write now waits while a reset of the
device is in progress.
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
drivers/pci/pci-sysfs.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index 6669e9ad5bd3..dff3c2d9408c 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -1643,6 +1643,9 @@ static ssize_t reset_method_store(struct device *dev,
int m, n;
u8 reset_methods[PCI_NUM_RESET_METHODS] = {};
+ /* Serialize against resets and other updates of the list */
+ guard(device)(dev);
+
if (sysfs_streq(buf, "")) {
pdev->reset_methods[0] = 0;
pci_warn(pdev, "All device reset methods disabled by user");
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH V1 4/4] PCI: Re-check function 0 reset methods after scanning its slot
2026-11-19 10:06 [PATCH V1 0/4] PCI: Don't list reset methods that can't reset the device Vidya Sagar
` (2 preceding siblings ...)
2026-11-19 10:06 ` [PATCH V1 3/4] PCI/sysfs: Take the device lock when setting " Vidya Sagar
@ 2026-11-19 10:06 ` Vidya Sagar
3 siblings, 0 replies; 6+ messages in thread
From: Vidya Sagar @ 2026-11-19 10:06 UTC (permalink / raw)
To: Bjorn Helgaas, Dan Williams, Dave Jiang, Jonathan Cameron,
Joerg Roedel, Jason Gunthorpe, Nicolin Chen, Kevin Tian
Cc: vsethi, kthota, mmaddireddy, kumarahul, sagar.tv, linux-pci,
linux-kernel, Vidya Sagar
pci_device_add() probes a device's reset methods before it adds the
device to bus->devices. pci_scan_slot() adds function 0 first, so when
function 0 is probed it is still alone on its bus and the "bus" probe,
which refuses a bus reset if other devices share the bus, succeeds. Once
the other functions are added, the same check refuses the reset, so
function 0 of a multi-function device lists "bus" but can't use it. For
example, on function 0 of a 4-function ConnectX-8:
# cat /sys/bus/pci/devices/0002:01:00.0/reset_method
flr bus cxl_bus
# echo bus > /sys/bus/pci/devices/0002:01:00.0/reset_method
bash: echo: write error: Invalid argument
# echo default > /sys/bus/pci/devices/0002:01:00.0/reset_method
# cat /sys/bus/pci/devices/0002:01:00.0/reset_method
flr
The other functions are probed after function 0 is on the bus, so they
are not affected.
At the end of pci_scan_slot(), if the scan added function 0 and other
functions, probe each of function 0's reset methods again and drop the
ones that now fail. By then device_add() has already created function
0's "reset_method" file and sent the "add" uevent, so userspace may have
changed the list already. Add pci_prune_reset_methods() for this: unlike
pci_init_reset_methods(), it only removes methods and keeps the order of
the rest. Hold the device lock while doing so. reset_method_store()
takes the same lock, so a write can't land between reading the list and
storing the result. pci_bus_add_device() takes this lock later in the
same scan, for device_initial_probe().
No driver can bind to function 0, and its runtime PM is not enabled,
until pci_bus_add_device(), which runs after the slot scan.
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
drivers/pci/pci.c | 35 +++++++++++++++++++++++++++++++++++
drivers/pci/pci.h | 1 +
drivers/pci/probe.c | 16 +++++++++++++++-
3 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 918a4d95f914..d9b26208a76c 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -5306,6 +5306,41 @@ void pci_init_reset_methods(struct pci_dev *dev)
dev->reset_methods[i] = 0;
}
+/**
+ * pci_prune_reset_methods - drop reset methods that no longer work
+ * @dev: PCI device to check
+ *
+ * Probe each of @dev's current reset methods again and remove the ones
+ * that fail, keeping the order of the rest. Unlike pci_init_reset_methods(),
+ * this never adds a method, so a list set through sysfs is kept, except for
+ * methods that can no longer work.
+ *
+ * Context: The caller must hold the device lock of @dev, which serializes
+ * this against resets and writes to the "reset_method" sysfs file.
+ */
+void pci_prune_reset_methods(struct pci_dev *dev)
+{
+ u8 methods[PCI_NUM_RESET_METHODS] = {};
+ int m, i, n = 0, rc;
+
+ might_sleep();
+ device_lock_assert(&dev->dev);
+
+ for (i = 0; i < PCI_NUM_RESET_METHODS; i++) {
+ m = dev->reset_methods[i];
+ if (!m)
+ break;
+
+ rc = pci_reset_fn_methods[m].reset_fn(dev, PCI_RESET_PROBE);
+ if (!rc)
+ methods[n++] = m;
+ else if (rc != -ENOTTY)
+ break;
+ }
+
+ memcpy(dev->reset_methods, methods, sizeof(dev->reset_methods));
+}
+
/**
* pci_reset_function - quiesce and reset a PCI device function
* @dev: PCI device to reset
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 77efa4c5c960..6e82c82acdb6 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -242,6 +242,7 @@ int pci_mmap_fits(struct pci_dev *pdev, int resno, struct vm_area_struct *vmai,
bool pci_reset_supported(struct pci_dev *dev);
void pci_init_reset_methods(struct pci_dev *dev);
+void pci_prune_reset_methods(struct pci_dev *dev);
int pci_bridge_secondary_bus_reset(struct pci_dev *dev);
int pci_bus_error_reset(struct pci_dev *dev);
int pci_try_reset_bridge(struct pci_dev *bridge);
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 9a178e0e5f7f..a36bf679c393 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2874,7 +2874,7 @@ static int only_one_child(struct pci_bus *bus)
*/
int pci_scan_slot(struct pci_bus *bus, int devfn)
{
- struct pci_dev *dev;
+ struct pci_dev *dev, *dev0 = NULL;
int fn = 0, nr = 0;
if (only_one_child(bus) && (devfn > 0))
@@ -2887,6 +2887,8 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
nr++;
if (fn > 0)
dev->multifunction = 1;
+ else
+ dev0 = dev;
} else if (fn == 0) {
/*
* Function 0 is required unless we are running on
@@ -2899,6 +2901,18 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
fn = next_fn(bus, dev, fn);
} while (fn >= 0);
+ /*
+ * Function 0's reset methods were probed before the other functions
+ * were on bus->devices, so a bus reset looked possible then. Its
+ * "reset_method" file already exists and userspace may have changed
+ * the list, so only drop the methods that can't work now. The device
+ * lock keeps a write to that file from landing in the middle.
+ */
+ if (dev0 && !pci_dev_is_added(dev0) && nr > 1) {
+ guard(device)(&dev0->dev);
+ pci_prune_reset_methods(dev0);
+ }
+
/* Only one slot has PCIe device */
if (bus->self && nr)
pcie_aspm_init_link_state(bus->self);
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread