[PATCH] nvme-pci: skip FLR after a failed controller reset

[PATCH] nvme-pci: skip FLR after a failed controller reset

[Haowen Bai]

nvme_disable_ctrl() already waits up to CAP.TO for CSTS.RDY to clear.
If that times out, nvme_pci_configure_admin_queue() currently issues a
PCIe Function Level Reset and retries.

FLR is performed with PCI config cycles. Those cycles take
pci_config_lock, a raw spinlock, and wait for the endpoint to complete
the transaction. A wedged NVMe function can stall that completion.
Other CPUs then spin in pci_conf1_read() -- including ACPI PCI config
from an unrelated device -- and the NMI watchdog reports a hard lockup.

This was observed on an x86_64 UGREEN DXP4800 (kernel 6.18.15) with two
ZHITAI Ti600 NVMe devices used as bcache. Each disk independently:

  nvme: I/O timeout, reset controller
  nvme: Device not ready; aborting reset, CSTS=0x1
  nvme: Device not ready; aborting reset, CSTS=0x1
  watchdog: Watchdog detected hard LOCKUP
  RIP: native_queued_spin_lock_slowpath
  pci_conf1_read -> acpi_pci_set_power_state -> mmc runtime resume

The two "aborting reset" messages are nvme_wait_ready() timeouts, 128s
apart, matching (CAP.TO+1)/2. The lockup is ~11s after the second
timeout, i.e. on the post-FLR cleanup path, not in the wait loop.
Linux 6.12 has no FLR fallback here; 6.18 and 7.3 still do.

Skip FLR when the controller is already in NVME_CTRL_RESETTING (I/O
timeout recovery). Keep the FLR hammer for initial probe, where the
device may simply have been left enabled by firmware. Reset work then
marks namespaces dead instead of hard-locking the host.

Cc: linux-nvme@lists.infradead.org
Cc: linux-pci@vger.kernel.org
Cc: Keith Busch <kbusch@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Haowen Bai <calvin.bai@ugreen.com>
---
 drivers/nvme/host/pci.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 5440cf18b55b..1df475fab950 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2373,12 +2373,18 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
 		struct pci_dev *pdev = to_pci_dev(dev->dev);
 
 		/*
-		 * The NVMe Controller Reset method did not get an expected
-		 * CSTS.RDY transition, so something with the device appears to
-		 * be stuck. Use the lower level and bigger hammer PCIe
-		 * Function Level Reset to attempt restoring the device to its
-		 * initial state, and try again.
+		 * Controller Reset did not clear CSTS.RDY. FLR can recover
+		 * some devices, but it issues PCI config cycles with
+		 * pci_config_lock held. A wedged function can stall those
+		 * cycles and hard-lock unrelated PCI users.
+		 *
+		 * Only try FLR during initial probe. On I/O-timeout reset
+		 * the controller is already known stuck; fail the reset
+		 * instead of risking a host lockup.
 		 */
+		if (nvme_ctrl_state(&dev->ctrl) == NVME_CTRL_RESETTING)
+			return result;
+
 		result = pcie_reset_flr(pdev, false);
 		if (result < 0)
 			return result;
-- 
2.47.3

Re: [PATCH] nvme-pci: skip FLR after a failed controller reset

[Keith Busch]

On Mon, Sep 21, 2026 at 10:07:32PM +0800, Haowen Bai wrote:
> nvme_disable_ctrl() already waits up to CAP.TO for CSTS.RDY to clear.
> If that times out, nvme_pci_configure_admin_queue() currently issues a
> PCIe Function Level Reset and retries.
> 
> FLR is performed with PCI config cycles. Those cycles take
> pci_config_lock, a raw spinlock, and wait for the endpoint to complete
> the transaction. A wedged NVMe function can stall that completion.

Shouldn't PCIe CTO have kicked in to fail the transaction? Do you know
which transaction is failing? Is the stall specific to FLR or could any
config access stall in your setup?

> Other CPUs then spin in pci_conf1_read() -- including ACPI PCI config
> from an unrelated device -- and the NMI watchdog reports a hard lockup.
> 
> This was observed on an x86_64 UGREEN DXP4800 (kernel 6.18.15) with two
> ZHITAI Ti600 NVMe devices used as bcache. Each disk independently:

Are you able to fix the device instead? Maybe add your device to the
"quirk_no_flr" list if you can't fix it. This sounds like a pretty nasty
bug on that side that forces driver to remove one of its recovery
options.
 
>   nvme: I/O timeout, reset controller
>   nvme: Device not ready; aborting reset, CSTS=0x1
>   nvme: Device not ready; aborting reset, CSTS=0x1
>   watchdog: Watchdog detected hard LOCKUP
>   RIP: native_queued_spin_lock_slowpath
>   pci_conf1_read -> acpi_pci_set_power_state -> mmc runtime resume
> 
> The two "aborting reset" messages are nvme_wait_ready() timeouts, 128s
> apart, matching (CAP.TO+1)/2. The lockup is ~11s after the second
> timeout, i.e. on the post-FLR cleanup path, not in the wait loop.
> Linux 6.12 has no FLR fallback here; 6.18 and 7.3 still do.
> 
> Skip FLR when the controller is already in NVME_CTRL_RESETTING (I/O
> timeout recovery). Keep the FLR hammer for initial probe, where the
> device may simply have been left enabled by firmware. Reset work then
> marks namespaces dead instead of hard-locking the host.

While I don't see this happen very often, I've seen FLR recover devices
both on first probe and IO timeout, so skipping for RESETTING will miss
recovering when it was possible for some conditions.

Re: [PATCH] nvme-pci: skip FLR after a failed controller reset

[Haowen Bai]

Keith,

Thanks for the review.

The goal on our side is that a dead NVMe function must not hard-lock
the host. Losing the device (and bcache) is acceptable; an NMI
lockup of unrelated PCI users is not. I don't want to take FLR away
from a path that still recovers some devices — I want teardown not to
pin pci_config_lock across a hung config cycle.

> Shouldn't PCIe CTO have kicked in to fail the transaction? Do you
> know which transaction is failing? Is the stall specific to FLR or
> could any config access stall in your setup?

I don't know which config cycle is stuck. There is no vmcore / lock
owner. The NMI captures the waiter (an unrelated eMMC runtime-resume
spinning in pci_conf1_read -> acpi_pci_set_power_state), not the
holder.

What the pstore timestamps do show:

  [t+0]     nvme_wait_ready timeout, CSTS=0x1   (MMIO, first disable)
  [t+128s]  nvme_wait_ready timeout, CSTS=0x1   (second disable after FLR)
  [t+139s]  hard lockup on pci_config_lock

So the 128s gap is CAP.TO on the second nvme_disable_ctrl(), which is
MMIO and does not take pci_config_lock. The lockup is ~11s after that
returns, i.e. on the post-FLR teardown path (pci_free_irq_vectors /
pci_disable_device or a config access still in flight), not inside
nvme_wait_ready().

I cannot prove the stall is unique to the FLR write vs any later
config access to that function. Both events went through
disable-timeout -> FLR -> disable-timeout -> teardown. Linux 6.12 has
no FLR fallback here; the same class of NVMe drop usually only took
the cache offline. That is correlation, not a single-cycle trace.

There is no AER / UR / completion-timeout message in the log. I do
not know whether CTO was disabled, longer than the NMI watchdog
(~10s), or not applicable because the root port never completed. I
won't claim CTO is broken. Even if CTO should have aborted the
cycle, it did not save the machine here.

> Are you able to fix the device instead? Maybe add your device to the
> "quirk_no_flr" list if you can't fix it.

The two ZHITAI Ti600 functions (1e49:0081) each reproduced the same
sequence independently, so I agree this is a nasty device bug. We
are taking them out of the bcache path on the affected machines.

quirk_no_flr would stop nvme from requesting FLR, but it would not
stop pci_disable_device() from touching config on the way out, which
is where the lockup lines up. Pinning host protection to one VID:DID
also misses the next broken device. A bad endpoint should be allowed
to die; it should not be able to stall pci_config_lock and take the
rest of the platform with it. I can add a quirk as a device note if
you want it on record; I don't think it is the host fix.

> I've seen FLR recover devices both on first probe and IO timeout, so
> skipping for RESETTING will miss recovering when it was possible

Agreed — that makes v1 too broad. I'll drop the RESETTING special
case rather than take FLR away from a path that still recovers some
devices.

If a v2 is useful, I think it needs to stop issuing config cycles to
a function that already failed CC.EN and FLR (so teardown cannot
hold pci_config_lock across a hung inl), without skipping FLR on the
reset path. I have not written that patch yet.

Thanks,
Haowen