mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
       [not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
@ 2026-11-06  8:56 ` Ziyou Wang (Lenovo)
  2026-11-06 15:21   ` Bjorn Helgaas
  0 siblings, 1 reply; 4+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-06  8:56 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Lukas Wunner, Kuppuswamy Sathyanarayanan, linux-pci,
	linux-kernel, stable, Ziyou Wang (Lenovo)

Some NVMe devices need more than 5 seconds to recover after DPC
containment, but pci_dpc_recovered() waits only 4 seconds.  If the device
hasn't recovered by then, pciehp reports "device recovery failed" and
the device is re-enumerated.

Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
used in dpc.c.  Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
so pciehp waits as long as pci_dev_wait() does, which is sufficient for
all known devices.

Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
injection:

Upstream kernel (7.3.0-master+, 4s timeout):
  [123.018691] nvme: not ready 1023ms after DPC; waiting
  [124.106690] nvme: not ready 2047ms after DPC; waiting
  [125.642703] pciehp: Slot(64): Link Down
  [126.218696] AER: subordinate device reset failed
  [126.218700] AER: device recovery failed
  Device re-enumerated as nvme1

With this patch (60s timeout):
  [74.894811] nvme: not ready 1023ms after DPC; waiting
  [75.982813] nvme: not ready 2047ms after DPC; waiting
  [78.094810] nvme: ready 4095ms after DPC
  [78.094816] nvme: restart after slot reset
  [78.111620] AER: device recovery successful
  Device works normally

Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
Cc: stable@vger.kernel.org
Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
---
 drivers/pci/pci.c      |  1 -
 drivers/pci/pci.h      | 10 ++++++++++
 drivers/pci/pcie/dpc.c | 11 ++++++-----
 3 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8cc6a89130b5..08f08f25e597 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -80,7 +80,6 @@ struct pci_pme_device {
  * limit, but 60 sec ought to be enough for any device to become
  * responsive.
  */
-#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
 
 static void pci_dev_d3_sleep(struct pci_dev *dev)
 {
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index ba3c3fddddc2..1e8ddcc013ff 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -61,6 +61,16 @@ struct pcie_tlp_log;
  */
 #define PCIE_RESET_CONFIG_WAIT_MS	100
 
+/*
+ * Time to wait for a device to become ready after a reset.
+ *
+ * Devices may extend the 1 sec period through Request Retry Status
+ * completions (PCIe r6.0 sec 2.3.1).  The spec does not provide an upper
+ * limit, but 60 sec ought to be enough for any device to become
+ * responsive.
+ */
+#define PCIE_RESET_READY_POLL_MS	60000	/* msec */
+
 /* Parameters for the waiting for link up routine */
 #define PCIE_LINK_WAIT_MAX_RETRIES	10
 #define PCIE_LINK_WAIT_SLEEP_MS		90
diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
index 2b779bd1d861..df1c469fa24c 100644
--- a/drivers/pci/pcie/dpc.c
+++ b/drivers/pci/pcie/dpc.c
@@ -116,12 +116,13 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
 		return false;
 
 	/*
-	 * Need a timeout in case DPC never completes due to failure of
-	 * dpc_wait_rp_inactive().  The spec doesn't mandate a time limit,
-	 * but reports indicate that DPC completes within 4 seconds.
+	 * Wait for DPC recovery completion.  Use PCIE_RESET_READY_POLL_MS
+	 * (60s) to match pci_dev_wait() timeout.  Some devices need >5 seconds
+	 * to recover after DPC containment.  The 4 second timeout is too short.
 	 */
-	wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
-			   msecs_to_jiffies(4000));
+	wait_event_timeout(dpc_completed_waitqueue,
+			   dpc_completed(pdev),
+			   msecs_to_jiffies(PCIE_RESET_READY_POLL_MS));
 
 	return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
 }
-- 
2.53.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
  2026-11-06  8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
@ 2026-11-06 15:21   ` Bjorn Helgaas
  2026-11-19  3:30     ` Ziyou Wang (Lenovo)
  0 siblings, 1 reply; 4+ messages in thread
From: Bjorn Helgaas @ 2026-11-06 15:21 UTC (permalink / raw)
  To: Ziyou Wang (Lenovo)
  Cc: Bjorn Helgaas, Lukas Wunner, Kuppuswamy Sathyanarayanan,
	linux-pci, linux-kernel, stable

On Fri, Nov 06, 2026 at 04:56:55PM +0800, Ziyou Wang (Lenovo) wrote:
> Some NVMe devices need more than 5 seconds to recover after DPC
> containment, but pci_dpc_recovered() waits only 4 seconds.  If the device
> hasn't recovered by then, pciehp reports "device recovery failed" and
> the device is re-enumerated.
> 
> Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
> used in dpc.c.  Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
> so pciehp waits as long as pci_dev_wait() does, which is sufficient for
> all known devices.
> 
> Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
> injection:
> 
> Upstream kernel (7.3.0-master+, 4s timeout):
>   [123.018691] nvme: not ready 1023ms after DPC; waiting
>   [124.106690] nvme: not ready 2047ms after DPC; waiting

Was there a "not ready ... after DPC; giving up" message here?
pci_dev_wait() should log *something* about how it resolved this
situation.

>   [125.642703] pciehp: Slot(64): Link Down
>   [126.218696] AER: subordinate device reset failed
>   [126.218700] AER: device recovery failed
>   Device re-enumerated as nvme1
> 
> With this patch (60s timeout):
>   [74.894811] nvme: not ready 1023ms after DPC; waiting
>   [75.982813] nvme: not ready 2047ms after DPC; waiting
>   [78.094810] nvme: ready 4095ms after DPC
>   [78.094816] nvme: restart after slot reset
>   [78.111620] AER: device recovery successful
>   Device works normally
> 
> Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
> ---

30 minutes after v1 is too soon to post a v2.  A couple days is a
reasonable interval.

v2 should include a note here (after the "---") about what changed
since v1 and ideally a lore link to the v1 posting.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
  2026-11-06 15:21   ` Bjorn Helgaas
@ 2026-11-19  3:30     ` Ziyou Wang (Lenovo)
  2026-11-19  9:01       ` Lukas Wunner
  0 siblings, 1 reply; 4+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-19  3:30 UTC (permalink / raw)
  To: helgaas
  Cc: bhelgaas, lukas, sathyanarayanan.kuppuswamy, linux-pci,
	linux-kernel, stable, ziyou.dev

Hi Bjorn,

Thanks for your review!

On Fri, Nov 06, 2026 at 09:21:43AM -0600, Bjorn Helgaas wrote:
> Was there a "not ready ... after DPC; giving up" message here?
> pci_dev_wait() should log *something* about how it resolved this
> situation.

No.  Here is the complete log from the DPC event to the re-enumeration:

  [121.693046] pcieport 0000:00:03.1: EDR: EDR event received
  [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
  [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
  [121.695829] pcieport 0000:00:03.1:   device [1022:1108] error status/mask=00200000/00000000
  [121.695831] pcieport 0000:00:03.1:   [21] ACSViol           | Receiver    | Transaction Layer (First)
  [121.695834] pcieport 0000:00:03.1: AER:   TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
  [121.695841] nvme nvme0: frozen state error detected, reset controller
  [121.721168] nvme0n1: I/O Cmd(0x2) @ LBA 67600640, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
  [121.721182] I/O error, dev nvme0n1, sector 67600640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
  [121.721193] nvme0n1: I/O Cmd(0x2) @ LBA 67600384, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
  [121.721196] I/O error, dev nvme0n1, sector 67600384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
  [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting
  [124.106690] nvme 0000:01:00.0: not ready 2047ms after DPC; waiting
  [125.642703] pcieport 0000:00:03.1: pciehp: Slot(64): Link Down
  [125.647840] Buffer I/O error on dev nvme0n1, logical block 8450048, async page read
  [125.647846] Buffer I/O error on dev nvme0n1, logical block 8450049, async page read
  [125.647847] Buffer I/O error on dev nvme0n1, logical block 8450050, async page read
  [125.647848] Buffer I/O error on dev nvme0n1, logical block 8450051, async page read
  [125.647850] Buffer I/O error on dev nvme0n1, logical block 8450052, async page read
  [125.647850] Buffer I/O error on dev nvme0n1, logical block 8450053, async page read
  [125.647851] Buffer I/O error on dev nvme0n1, logical block 8450054, async page read
  [125.647852] Buffer I/O error on dev nvme0n1, logical block 8450055, async page read
  [125.647853] Buffer I/O error on dev nvme0n1, logical block 8450056, async page read
  [125.647854] Buffer I/O error on dev nvme0n1, logical block 8450057, async page read
  [125.849608] pcieport 0000:00:03.1: pciehp: Slot(64): Card present
  [125.849614] pcieport 0000:00:03.1: pciehp: Slot(64): Link Up
  [126.218696] pcieport 0000:00:03.1: AER: subordinate device reset failed
  [126.218700] pcieport 0000:00:03.1: AER: device recovery failed
  [126.705745] pci 0000:01:00.0: [1dbe:5668] type 00 class 0x010802 PCIe Endpoint
  [126.705786] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit]
  [126.705793] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
  [126.705801] pci 0000:01:00.0: Max Payload Size set to 512 (was 128, max 512)
  [126.705987] pci 0000:01:00.0: _HPX attempts Link Control setting (AND 0xffff OR 0x0040)
  [126.706073] pci 0000:01:00.0: supports D1 D2
  [126.706074] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
  [126.706432] pci 0000:01:00.0: Adding to iommu group 16
  [126.706481] pci 0000:01:00.0: ROM [mem 0xde300000-0xde30ffff pref]: assigned
  [126.706483] pci 0000:01:00.0: BAR 0 [mem 0xde310000-0xde313fff 64bit]: assigned
  [126.706491] pcieport 0000:00:03.1: PCI bridge to [bus 01]
  [126.706494] pcieport 0000:00:03.1:   bridge window [io  0x1000-0x1fff]
  [126.706498] pcieport 0000:00:03.1:   bridge window [mem 0xde300000-0xde3fffff]
  [126.706500] pcieport 0000:00:03.1:   bridge window [mem 0x10021600000-0x100217fffff 64bit pref]
  [126.709685] nvme nvme1: pci function 0000:01:00.0
  [126.709692] nvme 0000:01:00.0: enabling device (0140 -> 0142)
  [126.712068] nvme nvme1: D3 entry latency set to 10 seconds
  [126.720936] nvme nvme1: 129/0/0 default/read/poll queues
  [126.727879]  nvme0n2: p1 p2 p3

pci_dev_wait() never reached its timeout.  The 4s pci_dpc_recovered()
timeout expired first, so pciehp removed the device and marked it
disconnected.  When pci_dev_wait() woke up, it took the
pci_dev_is_disconnected() path, which only logs at debug level and
returns -ENOTTY.  That is why the -ENOTTY surfaced as "AER:
subordinate device reset failed" instead.

The device needs a little over 4s to become ready ("ready 4095ms
after DPC" with this patch), so the 4s timeout is losing that race.

One thing worth noting: before e8df1d194c39b ("PCI: Don't treat ~0 as
ready in pci_dev_wait() with RRS SV"), this failure was masked on this
system.  The RRS SV path only checked for the 0x0001 RRS Vendor ID, so
a ~0 read from the not-yet-ready device was treated as "ready",
pci_bridge_wait_for_secondary_bus() returned early, and the DPC
"recovery" appeared to succeed within the 4s window, even though the
device was not actually usable (nvme then failed with -19 after the
"successful" recovery).  With e8df1d194c39b, pci_dev_wait() correctly
waits, which re-exposed this 4s race.

Best regards,
Ziyou Wang


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
  2026-11-19  3:30     ` Ziyou Wang (Lenovo)
@ 2026-11-19  9:01       ` Lukas Wunner
  0 siblings, 0 replies; 4+ messages in thread
From: Lukas Wunner @ 2026-11-19  9:01 UTC (permalink / raw)
  To: Ziyou Wang (Lenovo)
  Cc: helgaas, bhelgaas, sathyanarayanan.kuppuswamy, linux-pci,
	linux-kernel, stable

On Thu, Nov 19, 2026 at 11:30:36AM +0800, Ziyou Wang (Lenovo) wrote:
>   [121.693046] pcieport 0000:00:03.1: EDR: EDR event received
>   [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
>   [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
>   [121.695829] pcieport 0000:00:03.1:   device [1022:1108] error status/mask=00200000/00000000
>   [121.695831] pcieport 0000:00:03.1:   [21] ACSViol           | Receiver    | Transaction Layer (First)
>   [121.695834] pcieport 0000:00:03.1: AER:   TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
[...]
>   [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting

I'm wondering why the ACS Violation Error is raised in the first place.
The TLP causing the DPC event is a Memory Write Request from the NVMe
drive to memory address 0x000000177e325c00.  The Requester ID 0x100
seems to match the bus address 0000:01:00.0.

What are the ACS settings at the Root Port and why is it signaling an
ACS violation?

TLP decoded with tlp-tool (https://github.com/mmpg-x86/tlp-tool):

+----------+-------------+----------------------+
| TLP Type | MemWriteReq | 4DW with Data Header |
+----------+-------------+----------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
|            | (bits) | (bits) |       |
+------------+--------+--------+-------+
| Fmt        | 0      | 3      | 3     |
| Type       | 3      | 5      | 0     |
| T9         | 8      | 1      | 0     |
| TC         | 9      | 3      | 0     |
| T8         | 12     | 1      | 0     |
| Attr_b2    | 13     | 1      | 0     |
| LN         | 14     | 1      | 0     |
| TH         | 15     | 1      | 0     |
| Td         | 16     | 1      | 0     |
| Ep         | 17     | 1      | 0     |
| Attr       | 18     | 2      | 2     |
| AT         | 20     | 2      | 0     |
| Length     | 22     | 10     | 128   |
+------------+--------+--------+-------+
+-----------------+----------------------+
| TLP:            | 4DW with Data Header |
+-----------------+----------------------+
| Req ID          | 0x100                |
| Tag             | 0x42                 |
| Last DW BE      | 0xF                  |
| First DW BE     | 0xF                  |
| Addr High (DW2) | 0x00000017           |
| Addr Low  (DW3) | 0x7E325C00           |
+-----------------+----------------------+

Thanks,

Lukas

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-19  9:01 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
2026-11-06  8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 15:21   ` Bjorn Helgaas
2026-11-19  3:30     ` Ziyou Wang (Lenovo)
2026-11-19  9:01       ` Lukas Wunner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®