[bug report] nvme-pci: NVMe lost after S3 resume, CSTS reads all-ones for a few 100 ms

[Moritz Hiller <m.hiller@pc-heini.de>]

Hi,

on my desktop the root NVMe drive sporadically does not come back after
S3 resume (roughly 5-10% of resumes in automated runs):

  nvme nvme0: Disabling device after reset failure: -19

The single readl(CSTS) in nvme_pci_enable() returns all-ones about
200 ms after resume. The controller is not dead, it just needs a few
hundred ms more. A small wait loop (diff below) fixes it here, tested
on CachyOS kernels and on vanilla v7.4-rc2. I am not asking for the
diff to be merged as is, I would like to know where this should be
fixed properly.

System:
- ASUS ROG STRIX X570-F GAMING (BIOS 5044), Ryzen 7 5800X, S3 (deep)
- Samsung 9100 PRO 2TB [144d:a810], FW 0B2QNXH7, at 0000:01:00.0,
  directly behind root port 0000:00:01.1 [1022:1483]

Observations:
- pci_pm_resume_noirq takes ~0.3 ms for the SSD, config space answers.
- At the failing CSTS read config space is intact, the link is up
  (16 GT/s x4), the device is in D0, no AER errors:

  vendor=144d cmd=0006 (saved 0002) bar0=fcf00004 (saved fcf00004)
  pmcsr=0008 lnksta=1044 rp_lnksta=3044

- sysfs remove + rescan brings the drive back after ~3 s.
- ASPM is off on this link, pcie_aspm=off and
  nvme_core.default_ps_max_latency_us=0 make no difference.
- Unpatched, 2 of 15 and 2 of 24 cycles failed in automated runs
  (90 s sleep, RTC wake). With the wait, no cycle failed (30/30 on
  CachyOS, 15/15 on v7.4-rc2); the controller answered after
  200-400 ms whenever the wait triggered.

My reading: pci_power_up() finds PMCSR already in D0 and does not
wait, and pci_bridge_wait_for_secondary_bus() only runs after D3cold
and only checks config space. So nothing waits for MMIO to come back,
and nvme_pci_enable() gives up on the first all-ones read.

The diff polls CSTS every 100 ms for up to 10 s, only if
NVME_CC_SHN_NORMAL is still set (the was_suspend test in
nvme_reset_work()) and while pci_device_is_present(). Probe and
runtime resets are unchanged. Known downsides:
- The 10 s bound is arbitrary.
- It also triggers on thaw/restore after hibernation (same bit set).
- The wait sleeps with shutdown_lock held, as nvme_wait_ready() already
  does in the same call chain.
- A controller that answers config space but is really dead after
  resume is declared dead 10 s later than today.
- A wait that times out is not logged.

Should this go into nvme-pci, the PCI core, or a quirk for this board
or drive? I can provide lspci -vvv, full dmesg or test patches.

The problem has existed on this machine for about a year, possibly
also with the previous SSD in this slot. It was hard to pin down: when
it happens the root filesystem is gone, so nothing gets written to
disk. I only got the first log of a failure in August 2026 via
netconsole, and every failure captured since then looks the same.

Disclosure: I am not a kernel developer. Log analysis, code reading
and the diff were done with an LLM assistant (Claude); I ran all tests
on my hardware. I can send patch with Signed-off-by if useful.

Thanks,
Moritz Hiller

---8<--- proof of concept, against v7.3.4 ---8<---

--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -7,6 +7,7 @@
 #include <linux/acpi.h>
 #include <linux/async.h>
 #include <linux/blkdev.h>
+#include <linux/delay.h>
 #include <linux/blk-mq-dma.h>
 #include <linux/blk-integrity.h>
 #include <linux/dmi.h>
@@ -3206,6 +3207,38 @@
 	return true;
 }
 
+/*
+ * A controller that was shut down for suspend may have lost power while the
+ * PCI core still treats it as D3hot, so nothing waited for it to come back.
+ * Config space can already answer while MMIO still reads all-ones. Give such
+ * a controller some time instead of declaring it dead on the first read.
+ */
+#define NVME_RESUME_MMIO_TIMEOUT_MS	10000
+#define NVME_RESUME_MMIO_POLL_MS	100
+
+static bool nvme_pci_wait_mmio_after_suspend(struct nvme_dev *dev)
+{
+	struct pci_dev *pdev = to_pci_dev(dev->dev);
+	unsigned int waited = 0;
+
+	/* Same test nvme_reset_work() uses to detect a resume. */
+	if (!(dev->ctrl.ctrl_config & NVME_CC_SHN_NORMAL))
+		return false;
+
+	while (waited < NVME_RESUME_MMIO_TIMEOUT_MS &&
+	       pci_device_is_present(pdev)) {
+		msleep(NVME_RESUME_MMIO_POLL_MS);
+		waited += NVME_RESUME_MMIO_POLL_MS;
+		if (readl(dev->bar + NVME_REG_CSTS) != -1) {
+			dev_warn(dev->ctrl.device,
+				 "CSTS read all-ones after resume, controller ready after %u ms\n",
+				 waited);
+			return true;
+		}
+	}
+	return false;
+}
+
 static int nvme_pci_enable(struct nvme_dev *dev)
 {
 	int result = -ENOMEM;
@@ -3217,7 +3250,8 @@
 
 	pci_set_master(pdev);
 
-	if (readl(dev->bar + NVME_REG_CSTS) == -1) {
+	if (readl(dev->bar + NVME_REG_CSTS) == -1 &&
+	    !nvme_pci_wait_mmio_after_suspend(dev)) {
 		dev_dbg(dev->ctrl.device, "reading CSTS register failed\n");
 		result = -ENODEV;
 		goto disable;