Bjorn Helgaas writes: > Can you please preserve the cc list, including the mailing lists, when > responding? This whole conversation is important for future > maintenance. My mistake - that one went out to you alone. Same content below, unchanged, now with the list on cc. > You said that even a later retrain isn't sufficient, so I don't know > what you envision. Can you post a working patch? I wrote the (b) variant you described and tested it. It fixes the PCIe link, but it does not restore the console, and I think that changes the picture for this bug. Details below. The patch, on top of v7.3.4: ret = pcie_set_target_speed(dev, speed_cap, false); if (ret) goto err; /* * The bus below this port has not been enumerated yet, so * pcie_set_target_speed() cannot tell whether the link really * retrained: it skips its "speed did not change" check on an * empty bus, and the DLLLA wait in pcie_retrain_link() may * return at once because DLLLA still reports the state from * before the retrain. Give the LTSSM time to retrain and * verify that the link survived; if it did not, fall back to * the original target speed as in the path above. */ msleep(100); pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); if (!(lnksta & PCI_EXP_LNKSTA_DLLLA)) { pci_info(dev, "link went down after removing the restriction\n"); goto err; } The 100 ms is measured, not guessed: on the live machine the link drops 25 ms after the Retrain Link write. With the patch: [ 1.146706] pci 0000:00:1c.3: removing 2.5GT/s downstream link speed restriction [ 1.246826] pci 0000:00:1c.3: link went down after removing the restriction [ 1.246895] pci 0000:00:1c.3: retraining failed [ 1.265256] pci 0000:07:00.0: [18ca:0027] type 00 class 0x030000 PCIe Endpoint [ 1.281487] pci 0000:07:00.0: BAR 0 [mem 0x90000000-0x93ffffff pref]: assigned So it does what it says: detects the failure, falls back, the endpoint is enumerated again, 79 PCI devices, LnkCtl2 back to 2.5GT/s, DLActive+. Your reading of the code was right - pcie_retrain_link() returns success because DLLLA is read before the retrain actually starts. But the console is still dead - not black, the monitor reports no signal at all. This is the part I did not expect. Four kernels, same hardware model (three different units of this box, all XH320 V2 with the same 18ca:0027 endpoint), same .config except the hunk under test: kernel 18ca:0027 00:1c.3 link local console -------------------------------------------------------------- 6.8.0-31 (Ubuntu) present 2.5GT/s works 7.3.4 (mainline) absent 5GT/s, down black 7.3.4 + revert present 2.5GT/s works 7.3.4 + this patch present 2.5GT/s black The device is back, the link is up and stable, and the console is still dead. Probing the VGA registers directly over /dev/port explains why: 6.8.0-31 7.3.4 + patch ------------------------------------------------------------- 0x3DA (30 samples) 00, 01, 09 31, constant 0x3C2 (Input Status 0) 10 00 CRTC 0x17 (mode control) A3 23 PCI_COMMAND of 07:00.0 0007 0000 0x3DA toggling means the CRTC is scanning out. Constant 0x31 means the display engine is stopped - not black, no signal at all. 0x3C2 bit 4 (display enable) is clear. So the endpoint is present and reaches its BARs, but its display state was wiped when the link went down, and nothing in Linux puts it back: unbinding and rebinding vgacon, which re-runs vgacon_startup(), does not restart the scan. Only a VBE mode set can, and the kernel has no path to that for this device. The framebuffer is a second, separate casualty. v7.3.4 added sysfb_pci_dev_is_enabled() in drivers/firmware/sysfb.c, which reads PCI_COMMAND and returns -ENODEV if PCI_COMMAND_MEMORY is clear: pci_read_config_word(pdev, PCI_COMMAND, &command); if (!(command & PCI_COMMAND_MEMORY)) return false; sysfb_init() then returns silently, without a single printk. Since no driver has enabled this device (no fb0, enable refcount is 0), the memory decoding is clear precisely because the reset cleared it, so the framebuffer is never created. I set PCI_COMMAND back to 0x0007 by hand and the BARs became readable, but the scan still does not start. What this means for the fix Your (b) argument was that it is better because it responds to something that went wrong. The test above says the damage is already done by the time we can respond: falling back to 2.5GT/s restores the link and the PCI topology, but not the device state that the failed training wiped. (b) turns "device disappears" into "device is back but its display is dead", which is a smaller failure but still a failure, and the local console stays unusable either way. I am not saying (b) is useless: for a NIC or an HBA behind such a port the same reset happens, the device is re-enumerated and its driver probes it again, so (b) does repair those. The problem is specific to devices whose only initialisation happened in firmware and that have no driver to redo it - a VGA adapter used through vgacon or simpledrm is exactly that case, and it is the machine that loses its console. The only fix measured to work on this machine is the revert: gating the lift so the port is not pushed to 5GT/s when the endpoint behind it is Gen1-only. I understand your objection that this is more aggressive than the spec requires, and that a working Gen2 link with a Gen1 endpoint is the normal case - it is, on chipsets that can train it. This one cannot, and the cost is not a slow link but a dead one. If (a) in its original form is not acceptable, could the gate key off the link state instead of the endpoint's capabilities? At the point pcie_failed_link_retrain() runs, LnkSta already reports whether the link is active (DLLLA) and at which speed it was trained, so a gate of "DLActive set, negotiated speed still 2.5GT/s" would skip exactly the ports where raising the target speed re-trains an already-working link, and still allow the lift everywhere else. I am willing to build and test any variant you prefer on this machine - I still have it, and it reproduces on every boot. Attached: fix-lift-verify.diff the (b) patch above, applies to v7.3.4 dmesg-7.3.4-fix.txt full dmesg, patched v7.3.4 lspci-00-1c.3-7.3.4-fix.txt lspci -vvv, root port 00:1c.3 lspci-07-00.0-7.3.4-fix.txt lspci -vvv, endpoint 07:00.0 boot-fix.txt uname/pci count/console state summary vga-port-probe.txt the /dev/port dump in the table above Thanks, chh