[PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Jennifer Chan]

On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
downstream port.  "nvidia-smi -r" resets all GPUs: NVML unbinds the
switches from the nvidia-nvswitch driver, resets each one through sysfs
(reset_method "bus", a Secondary Bus Reset of its port) and binds them
again.  The PCI core waits 100 ms after the link comes back, the switch
answers config reads and the reset completes.  When nvidia-nvswitch then
probes the switches, one fails in pci_enable_device() with

  Unable to change power state from D3cold to D0, device inaccessible

four others fail to initialize with -EIO, and the NVSwitch fabric does
not come back.

The same systems recover every time on a kernel that sleeps 1 s after
each Secondary Bus Reset, as pci_reset_secondary_bus() did before commit
ac91e6980563 ("PCI: Unify delay handling for reset and resume").  Adding
a 1 s sleep before the link wait on the PEX 8725 ports of the failing
kernel fixes it too.  Config reads already succeed at that point, so
pci_dev_wait() does not cover it, and the switch's state there is not
known.

Give LR10 a 1 s d3cold_delay, so that pci_bridge_wait_for_secondary_bus()
waits 1 s after the downstream link comes up before the switch is used.
This adds 1 s to each LR10 reset (about 6 s to an "nvidia-smi -r" on
these systems) and to D3cold resume of its port.  Resets issued further
up the hierarchy keep the default delay.

Fixes: ac91e6980563 ("PCI: Unify delay handling for reset and resume")
Cc: stable@vger.kernel.org
Signed-off-by: Jennifer Chan <jenniferchan@meta.com>
---
 drivers/pci/quirks.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 7f1f47ea2ee4..4a84095e736f 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -6569,12 +6569,14 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_TOSHIBA, 0x0220, of_pci_make_dev_node);
  * after reset recovery or resume from D3cold:
  *
  * VideoPropulsion (aka Genroco) Torrent QN16e MPEG QAM Modulator
+ * NVIDIA LR10 NVSwitch (HGX A100)
  */
 static void pci_fixup_d3cold_delay_1sec(struct pci_dev *pdev)
 {
 	pdev->d3cold_delay = 1000;
 }
 DECLARE_PCI_FIXUP_FINAL(0x5555, 0x0004, pci_fixup_d3cold_delay_1sec);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x1af1, pci_fixup_d3cold_delay_1sec);
 
 #ifdef CONFIG_PCIEAER
 static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
-- 
2.53.0-Meta

Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Bjorn Helgaas]

[+cc Lukas, author of ac91e6980563, Jason in case NVIDIA knows
anything]

On Thu, Nov 05, 2026 at 12:10:10PM -0800, Jennifer Chan wrote:
> On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
> NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
> downstream port.  "nvidia-smi -r" resets all GPUs: NVML unbinds the
> switches from the nvidia-nvswitch driver, resets each one through sysfs
> (reset_method "bus", a Secondary Bus Reset of its port) and binds them
> again.  The PCI core waits 100 ms after the link comes back, the switch
> answers config reads and the reset completes.  When nvidia-nvswitch then
> probes the switches, one fails in pci_enable_device() with
> 
>   Unable to change power state from D3cold to D0, device inaccessible
> 
> four others fail to initialize with -EIO, and the NVSwitch fabric does
> not come back.

Do you know where the -EIO is from?

> The same systems recover every time on a kernel that sleeps 1 s after
> each Secondary Bus Reset, as pci_reset_secondary_bus() did before commit
> ac91e6980563 ("PCI: Unify delay handling for reset and resume").  Adding
> a 1 s sleep before the link wait on the PEX 8725 ports of the failing
> kernel fixes it too.  Config reads already succeed at that point, so
> pci_dev_wait() does not cover it, and the switch's state there is not
> known.

Ugh, a second is a long time to wait.  Does the kernel you're testing
include e8df1d194c39 ("PCI: Don't treat ~0 as ready in pci_dev_wait()
with RRS SV"), by chance?  It only appeared in v7.3, so it's pretty
new.

> Give LR10 a 1 s d3cold_delay, so that pci_bridge_wait_for_secondary_bus()
> waits 1 s after the downstream link comes up before the switch is used.
> This adds 1 s to each LR10 reset (about 6 s to an "nvidia-smi -r" on
> these systems) and to D3cold resume of its port.  Resets issued further
> up the hierarchy keep the default delay.
> 
> Fixes: ac91e6980563 ("PCI: Unify delay handling for reset and resume")
> Cc: stable@vger.kernel.org
> Signed-off-by: Jennifer Chan <jenniferchan@meta.com>
> ---
>  drivers/pci/quirks.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 7f1f47ea2ee4..4a84095e736f 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -6569,12 +6569,14 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_TOSHIBA, 0x0220, of_pci_make_dev_node);
>   * after reset recovery or resume from D3cold:
>   *
>   * VideoPropulsion (aka Genroco) Torrent QN16e MPEG QAM Modulator
> + * NVIDIA LR10 NVSwitch (HGX A100)
>   */
>  static void pci_fixup_d3cold_delay_1sec(struct pci_dev *pdev)
>  {
>  	pdev->d3cold_delay = 1000;
>  }
>  DECLARE_PCI_FIXUP_FINAL(0x5555, 0x0004, pci_fixup_d3cold_delay_1sec);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x1af1, pci_fixup_d3cold_delay_1sec);
>  
>  #ifdef CONFIG_PCIEAER
>  static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
> -- 
> 2.53.0-Meta
> 

Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Jennifer Chan]

On Thu, Nov 05, 2026, Bjorn Helgaas wrote:
> Do you know where the -EIO is from?

My mistake in the changelog, sorry: it isn't -EIO.  The four switches
fail in nvidia-nvswitch chip init, which logs "Failed to initialize
device : -5".  That -5 is the driver's own status code
(NVL_INITIALIZATION_TOTAL_FAILURE), not an errno, and probe then returns
-ENODEV.  Before that the driver warns while reading its engine
discovery table from BAR0, which looks like MMIO not returning real data
yet.  My logs cut off the driver's error text, so I can't say which
access failed.

> Ugh, a second is a long time to wait.  Does the kernel you're testing
> include e8df1d194c39 ("PCI: Don't treat ~0 as ready in pci_dev_wait()
> with RRS SV"), by chance?  It only appeared in v7.3, so it's pretty
> new.

No, both kernels are v7.1-based without it.  The fifth switch fails
because PM_CTRL reads ~0 in pci_power_up(), so the device may well be
returning ~0 after pci_dev_wait() declares it ready.  I haven't yet
checked whether the Root Port has RRS SV enabled.  I'll check that and
test v7.1 with e8df1d194c39 backported and without this quirk, logging
what pci_dev_wait() reads.  If that fixes it, I'll drop this patch.

Thanks,
Jennifer

Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Lukas Wunner]

On Thu, Nov 05, 2026 at 03:51:48PM -0600, Bjorn Helgaas wrote:
> On Thu, Nov 05, 2026 at 12:10:10PM -0800, Jennifer Chan wrote:
> > On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
> > NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
> > downstream port.  "nvidia-smi -r" resets all GPUs: NVML unbinds the
> > switches from the nvidia-nvswitch driver, resets each one through sysfs
> > (reset_method "bus", a Secondary Bus Reset of its port) and binds them
> > again.  The PCI core waits 100 ms after the link comes back, the switch
> > answers config reads and the reset completes.

Broadcom PEX 89xxx switches support a "virtual switch" feature whereby
the switch spoofs responses from downstream devices when they're
inaccessible.  I just checked the datasheet of the PEX 8725 and it
seems to support this feature as well.

This feature makes the kernel believe that the downstream device is
ready even though it actually isn't.

I was made aware of this internally because a Supermicro server used
for testing contained two PEX 89xxx switches which had that feature
enabled.  I see you're using a Supermicro server as well, so perhaps
it has the same problem.

I developed a patch (included below, goes on top of v7.4-rc1) to work
around this misbehavior.  You may want to give it a spin and see if
it helps.  I never upstreamed the patch because after a lengthy
back and forth with Broadcom it turned out that the Supermicro server
shipped with an outdated firmware on the PEX 89xxx switches.  Updating
to a newer firmware made the issue go away.  The BIOS of the Supermicro
server was up to date, but it does not include the switch firmware.
I had to obtain a separate firmware update specifically for the switches
and install it through an EFI shell.

Obviously, a blanket wait of 1 second will work around Broadcom's virtual
switch feature as well, but it's not a proper fix.

The amdgpu driver contains a workaround for the virtual switch feature,
see 1dd2fa0e00f1.  Obviously, this only helps amdgpu and other drivers
don't benefit.  So we'd need to solve this for everyone in the PCI core,
as my patch below does.  I kind of hate putting a workaround for this
vendor-specific misbehavior in core code, but problem reports caused
by it keep landing on my desk, so maybe I should amend the patch with
a commit message and submit it.  But please test it first because perhaps
the issue you're seeing has some entirely different cause.

Thanks,

Lukas

-- >8 --

From 6ae81de682389f8522e96cea309e5e5304829b55 Mon Sep 17 00:00:00 2001
From: Lukas Wunner <lukas@wunner.de>
Date: Thu, 16 Oct 2025 10:56:15 +0200
Subject: [PATCH] PCI: Check for Broadcom placeholder device in pci_dev_wait()

Signed-off-by: Lukas Wunner <lukas@wunner.de>
---
 drivers/pci/pci.c | 70 +++++++++++++++++++++++++++++------------------
 1 file changed, 44 insertions(+), 26 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 9f82b0bc72ff..437e82cb47ac 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -1225,18 +1225,19 @@ void pci_resume_bus(struct pci_bus *bus)
 		pci_walk_bus(bus, pci_resume_one, NULL);
 }
 
-static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
+static bool pci_dev_ready(struct pci_dev *dev, struct pci_dev *root,
+			  struct pci_dev *bridge)
 {
-	int delay = 1;
-	bool retrain = false;
-	struct pci_dev *root, *bridge;
-
-	root = pcie_find_root_port(dev);
+	u32 id;
 
-	if (pci_is_pcie(dev)) {
-		bridge = pci_upstream_bridge(dev);
-		if (bridge)
-			retrain = true;
+	/*
+	 * Broadcom (LSI Logic) PEX 89000 PCIe switches spoof responses from a
+	 * placeholder device with ID 0x02b2 if the real device is not ready.
+	 */
+	if (bridge && bridge->vendor == PCI_VENDOR_ID_LSI_LOGIC) {
+		pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+		if (id == (PCI_VENDOR_ID_LSI_LOGIC | 0x02b2 << 16))
+			return false;
 	}
 
 	/*
@@ -1260,28 +1261,45 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 	 * ID for VFs and non-existent devices also returns ~0, so read the
 	 * Command register until it returns something other than ~0.
 	 */
-	for (;;) {
-		u32 id;
+	if (root && root->config_rrs_sv) {
+		pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+		if (!pci_bus_rrs_vendor_id(id)) {
+			if (PCI_POSSIBLE_ERROR(id) && dev->is_virtfn)
+				pci_read_config_dword(dev, PCI_COMMAND, &id);
+			if (!PCI_POSSIBLE_ERROR(id))
+				return true;
+		}
+	} else {
+		pci_read_config_dword(dev, PCI_COMMAND, &id);
+		if (!PCI_POSSIBLE_ERROR(id))
+			return true;
+	}
+
+	return false;
+}
+
+static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
+{
+	int delay = 1;
+	bool retrain = false;
+	struct pci_dev *root, *bridge = NULL;
 
+	root = pcie_find_root_port(dev);
+
+	if (pci_is_pcie(dev)) {
+		bridge = pci_upstream_bridge(dev);
+		if (bridge)
+			retrain = true;
+	}
+
+	for (;;) {
 		if (pci_dev_is_disconnected(dev)) {
 			pci_dbg(dev, "disconnected; not waiting\n");
 			return -ENOTTY;
 		}
 
-		if (root && root->config_rrs_sv) {
-			pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
-			if (!pci_bus_rrs_vendor_id(id)) {
-				if (PCI_POSSIBLE_ERROR(id) && dev->is_virtfn)
-					pci_read_config_dword(dev, PCI_COMMAND,
-							      &id);
-				if (!PCI_POSSIBLE_ERROR(id))
-					break;
-			}
-		} else {
-			pci_read_config_dword(dev, PCI_COMMAND, &id);
-			if (!PCI_POSSIBLE_ERROR(id))
-				break;
-		}
+		if (pci_dev_ready(dev, root, bridge))
+			break;
 
 		if (delay > timeout) {
 			pci_err(dev, "not ready %dms after %s; giving up\n",
-- 
2.53.0

Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Jason Gunthorpe]

On Thu, Nov 05, 2026 at 03:51:48PM -0600, Bjorn Helgaas wrote:
> [+cc Lukas, author of ac91e6980563, Jason in case NVIDIA knows
> anything]

Yeah, please don't take quirks for our server silicon unless we send
them. I've seen this a few times and rarely do users understand enough
to get this right.

Lukas is correct that many bugs trace back go Broadcom's exciting
interpretation of PCIe :\

The user should open a case with their NVIDIA support and if a kernel
quirk is in fact the correct fix the someone from NVIDIA will send
it..

Jason