* [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link
@ 2026-08-01 20:11 Andreas Wild
2026-08-03 5:39 ` Thorsten Leemhuis
` (2 more replies)
0 siblings, 3 replies; 19+ messages in thread
From: Andreas Wild @ 2026-08-01 20:11 UTC (permalink / raw)
To: linux-pci; +Cc: bhelgaas, macro, linux-kernel, Andreas Wild
From: "Maciej W. Rozycki" <macro@orcam.me.uk>
Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
failed link retraining") the Target Speed quirk lifts a firmware-imposed
2.5GT/s restriction on any downstream port, without checking whether the
link is up. Where nothing is plugged in, the retraining that follows can
never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The
quirk makes two of them -- the initial one and the restore on the error
path -- adding a fixed 2 s to every boot.
On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
slot, running v7.2-rc5:
0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction
1.541 pci 0000:00:1c.0: retraining failed
2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400
Where the Link Speed has already been clamped at 2.5GT/s and no link has
been established there is nothing worth doing, which is what the kerneldoc
for the quirk already describes: the restriction is to be lifted where
firmware arranged it "and the port reports its link already being up".
Bail out early in that case, before either the ASM2824 workaround or the
removal of the restriction is considered.
Ports whose link is up are unaffected, and so is the ASM2824 workaround,
which is reached with the Target Link Speed not clamped.
With this applied the quirk returns without touching the port: both
messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2
in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from
3.011 s to 1.036 s.
Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining")
Cc: stable@vger.kernel.org
Reported-by: Andreas Wild <andiwild@gmail.com>
Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
Tested-by: Andreas Wild <andiwild@gmail.com>
Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
Signed-off-by: Andreas Wild <andiwild@gmail.com>
---
v3:
- Replaced with Maciej's much simpler approach: bail out of the quirk
entirely when the Target Link Speed is already clamped at 2.5GT/s and no
link has been established, rather than programming the speed and skipping
only the retraining. One function, no new API, no bwctrl changes.
- Note this leaves the Target Link Speed clamped on such a port, where v2
left it at the Port's maximum. A device hot-plugged there later trains at
2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when
pcie_wait_for_link_status() fails, so a link that comes up cleanly at
2.5GT/s never re-runs it. Flagging in case that matters; the clamp is
firmware's, so honouring it on an unoccupied Port seems defensible.
- v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/
- v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
drivers/pci/quirks.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index b09f27f..9e407c4 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2);
- if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) {
+ if (lnksta & PCI_EXP_LNKSTA_DLLLA) {
+ ;
+ } else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) {
+ return ret;
+ } else if (pcie_lbms_seen(dev, lnksta)) {
pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n");
ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false);
if (ret)
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-01 20:11 [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link Andreas Wild @ 2026-08-03 5:39 ` Thorsten Leemhuis 2026-08-03 22:07 ` Maciej W. Rozycki 2026-08-07 19:59 ` Aoxtj 2026-09-17 23:33 ` Bjorn Helgaas 2 siblings, 1 reply; 19+ messages in thread From: Thorsten Leemhuis @ 2026-08-03 5:39 UTC (permalink / raw) To: Andreas Wild, linux-pci Cc: bhelgaas, macro, linux-kernel, Linux kernel regressions list, edwardmalik95 On 8/1/26 22:11, Andreas Wild wrote: > From: "Maciej W. Rozycki" <macro@orcam.me.uk> > > Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe > failed link retraining") the Target Speed quirk lifts a firmware-imposed > 2.5GT/s restriction on any downstream port, without checking whether the > link is up. Where nothing is plugged in, the retraining that follows can > never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The > quirk makes two of them -- the initial one and the restore on the error > path -- adding a fixed 2 s to every boot. > > On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1 > slot, running v7.2-rc5: > > 0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction > 1.541 pci 0000:00:1c.0: retraining failed > 2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400 TWIMC, this patch fixes a regression Edward reported here: https://bugzilla.kernel.org/show_bug.cgi?id=221801 [side note: the report could be more specific, yes -- I wanted to ask Edward to clarify a few things (like the actual slowdown) before forwarding it, but it didn't came to that when I noticed this patch and asked Edward to just check if it helped -- which is did.] Ciao, Thorsten > Where the Link Speed has already been clamped at 2.5GT/s and no link has > been established there is nothing worth doing, which is what the kerneldoc > for the quirk already describes: the restriction is to be lifted where > firmware arranged it "and the port reports its link already being up". > Bail out early in that case, before either the ASM2824 workaround or the > removal of the restriction is considered. > > Ports whose link is up are unaffected, and so is the ASM2824 workaround, > which is reached with the Target Link Speed not clamped. > > With this applied the quirk returns without touching the port: both > messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2 > in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from > 3.011 s to 1.036 s. > > Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining") > Cc: stable@vger.kernel.org > Reported-by: Andreas Wild <andiwild@gmail.com> > Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/ > Tested-by: Andreas Wild <andiwild@gmail.com> > Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk> > Signed-off-by: Andreas Wild <andiwild@gmail.com> > --- > v3: > - Replaced with Maciej's much simpler approach: bail out of the quirk > entirely when the Target Link Speed is already clamped at 2.5GT/s and no > link has been established, rather than programming the speed and skipping > only the retraining. One function, no new API, no bwctrl changes. > - Note this leaves the Target Link Speed clamped on such a port, where v2 > left it at the Port's maximum. A device hot-plugged there later trains at > 2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when > pcie_wait_for_link_status() fails, so a link that comes up cleanly at > 2.5GT/s never re-runs it. Flagging in case that matters; the clamp is > firmware's, so honouring it on an unoccupied Port seems defensible. > - v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/ > - v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/ > drivers/pci/quirks.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index b09f27f..9e407c4 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev) > > pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); > pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2); > - if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) { > + if (lnksta & PCI_EXP_LNKSTA_DLLLA) { > + ; > + } else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) { > + return ret; > + } else if (pcie_lbms_seen(dev, lnksta)) { > pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n"); > ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false); > if (ret) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-03 5:39 ` Thorsten Leemhuis @ 2026-08-03 22:07 ` Maciej W. Rozycki 2026-08-04 5:32 ` Thorsten Leemhuis 0 siblings, 1 reply; 19+ messages in thread From: Maciej W. Rozycki @ 2026-08-03 22:07 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, Linux kernel regressions list, edwardmalik95 On Mon, 3 Aug 2026, Thorsten Leemhuis wrote: > > Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe > > failed link retraining") the Target Speed quirk lifts a firmware-imposed > > 2.5GT/s restriction on any downstream port, without checking whether the > > link is up. Where nothing is plugged in, the retraining that follows can > > never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The > > quirk makes two of them -- the initial one and the restore on the error > > path -- adding a fixed 2 s to every boot. > > TWIMC, this patch fixes a regression Edward reported here: > https://bugzilla.kernel.org/show_bug.cgi?id=221801 Thank you for the pointer. As it happens attempts were made to cc me on that bug, which however went nowhere as messages were sent to my long-defunct <macro@linux-mips.org> e-mail address, a clear oversight of mine as I went, back in the day, through numerous sites to get this updated. I have now asked bugzilla admins to help me recover my account. > [side note: the report could be more specific, yes -- I wanted to ask > Edward to clarify a few things (like the actual slowdown) before > forwarding it, but it didn't came to that when I noticed this patch and > asked Edward to just check if it helped -- which is did.] Tested-by tags welcome! And thank you for your input overall, really useful and appreciated! Maciej ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-03 22:07 ` Maciej W. Rozycki @ 2026-08-04 5:32 ` Thorsten Leemhuis 0 siblings, 0 replies; 19+ messages in thread From: Thorsten Leemhuis @ 2026-08-04 5:32 UTC (permalink / raw) To: Maciej W. Rozycki Cc: Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, Linux kernel regressions list, edwardmalik95 On 8/4/26 00:07, Maciej W. Rozycki wrote: > On Mon, 3 Aug 2026, Thorsten Leemhuis wrote: >>> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe >>> failed link retraining") the Target Speed quirk lifts a firmware-imposed >>> 2.5GT/s restriction on any downstream port, without checking whether the >>> link is up. Where nothing is plugged in, the retraining that follows can >>> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The >>> quirk makes two of them -- the initial one and the restore on the error >>> path -- adding a fixed 2 s to every boot. >> >> TWIMC, this patch fixes a regression Edward reported here: >> https://bugzilla.kernel.org/show_bug.cgi?id=221801 > > Thank you for the pointer. yw > As it happens attempts were made to cc me on that bug, which however went > nowhere as messages were sent to my long-defunct <macro@linux-mips.org> > e-mail address, a clear oversight of mine as I went, back in the day, > through numerous sites to get this updated. > > I have now asked bugzilla admins to help me recover my account. FWIW, Artem does that and I'm not a fan of it at all -- I think it is only appropriate when one knows the person occasionally interacts directly (e.g. not via mail) with bugzilla (and thus I only do it in those cases). >> [side note: the report could be more specific, yes -- I wanted to ask >> Edward to clarify a few things (like the actual slowdown) before >> forwarding it, but it didn't came to that when I noticed this patch and >> asked Edward to just check if it helped -- which is did.] > > Tested-by tags welcome! FWIW, as I forgot to mention it easlier: Edward is CCed. @Edward, if you'd like a record of your test result end up in the history, reply with Tested-by: Full Name <email address> to this thread, just omit the space in the beginning and change it accordingly. Ciao, Thorsten ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-01 20:11 [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link Andreas Wild 2026-08-03 5:39 ` Thorsten Leemhuis @ 2026-08-07 19:59 ` Aoxtj 2026-08-08 6:31 ` Andreas Wild 2026-09-17 23:33 ` Bjorn Helgaas 2 siblings, 1 reply; 19+ messages in thread From: Aoxtj @ 2026-08-07 19:59 UTC (permalink / raw) To: Andreas Wild, linux-pci; +Cc: bhelgaas, macro, linux-kernel Hi, I'm the one who reported a PCIe/NVMe boot regression on the Proxmox forum: https://forum.proxmox.com/threads/7-0-14-7-kernel-breaks-pci-link-training.185549/ A Proxmox dev pointed me here and asked me to try this patch. I'm not a kernel developer and did most of the debugging with AI help, so please bear with me. TL;DR: - Proxmox Kernel 7.0.14-6 to 7.0.14-8: SK hynix system drive disappears during boot. - Narrowed down to commit 72780f796468 and verified by reverting it. - 7.0.14-8 + revert 72780f796468: boots reliably. - 7.0.14-8 + this v3 patch: only boots sometimes; failed boots lose the drive the same way. Machine: ASRock B450M Pro4-F, SK hynix PE4010 NVMe system drive on a passive PCIe bifurcation card. AMD Renoir PCIe GPP Bridge 0000:00:02.4 [1022:1633] -> bus 07 -> SK hynix PE4010 0000:07:00.0 [1c5c:2527] Initramfs dmesg from a failed v3 boot: [ 0.555805] pci 0000:00:02.4: [1022:1633] type 01 class 0x060400 PCIe Root Port [ 0.555828] pci 0000:00:02.4: PCI bridge to [bus 07] [ 0.555871] pci 0000:00:02.4: removing 2.5GT/s downstream link speed restriction [ 1.636965] pci 0000:00:02.4: retraining failed Nothing shows up on bus 07 afterwards, so my system drive is gone. With the revert instead, the working link reports: Root Port 0000:00:02.4: LnkCap: Speed 8GT/s, Width x4 LnkSta: Speed 2.5GT/s, Width x2, DLActive+ LnkCtl2: Target Link Speed: 2.5GT/s, SpeedDis+ LnkSta2: EqualizationComplete-, EqualizationPhase1/2/3- Endpoint 0000:07:00.0: LnkCap: Speed 8GT/s, Width x4 LnkSta: Speed 2.5GT/s (downgraded), Width x2 (downgraded) LnkCtl2: Target Link Speed: 8GT/s LnkSta2: EqualizationComplete-, EqualizationPhase1/2/3- My guess: this motherboard supports different PCIe generations depending on which CPU is installed, and with this CPU the link just can't run at the advertised 8GT/s — which would explain why the 8GT/s retrain fails. Tested on a locally rebuilt Proxmox/Ubuntu 7.0.14-8 backport, not mainline. Also replied in kernel Bugzilla #221801: https://bugzilla.kernel.org/show_bug.cgi?id=221801 Happy to test a diagnostic patch. Best regards, Aoxtj ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-07 19:59 ` Aoxtj @ 2026-08-08 6:31 ` Andreas Wild 2026-08-11 3:43 ` Aoxtj 0 siblings, 1 reply; 19+ messages in thread From: Andreas Wild @ 2026-08-08 6:31 UTC (permalink / raw) To: Aoxtj; +Cc: linux-pci, Bjorn Helgaas, Maciej W. Rozycki, linux-kernel On Sat, 1 Aug 2026, Aoxtj wrote: > - 7.0.14-8 + revert 72780f796468: boots reliably. > - 7.0.14-8 + this v3 patch: only boots sometimes; failed boots lose the > drive the same way. Thanks for testing it. v3 only covers ports with no link at all, so it does not address your case. 72780f796468 removed two guards at once: the Data Link Layer Link Active check, and the device ID match against the ASMedia ASM2824. My regression comes from losing the first; yours looks like it comes from losing the second. v3 only bails out where DLLLA is clear. Your link is up: > LnkSta: Speed 2.5GT/s, Width x2, DLActive+ so the quirk runs past the v3 early return, lifts the restriction and retrains at 8GT/s, exactly as it did before. Reverting works for you because your AMD Renoir Root Port didn't match the ASM2824 ID list that used to gate this. A possible workaround: If your BIOS lets you pin that slot to Gen1, you could try that. With the Root Port advertising only 2.5GT/s the quirk never reaches the retrain at all: on current mainline it returns at speed_cap = pcie_get_speed_cap(dev); if (speed_cap <= PCIE_SPEED_2_5GT) return ret; and on the older code your 7.0.14 backport is based on, the (lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB condition is false, so the restriction is never lifted. Since the link already runs at 2.5GT/s x2, that should cost you nothing - it just makes the Root Port advertise what the link can actually do. Something that is different in your case: Your Root Port reports > LnkCtl2: Target Link Speed: 2.5GT/s, SpeedDis+ where mine reports the same clamp with SpeedDis-. SpeedDis is PCI_EXP_LNKCTL2_HASD, and as far as I can tell the quirk never looks at it; the only user in the tree is pcie-designware.c. Firmware setting both the clamp and Hardware Autonomous Speed Disable looks like a deliberate pin rather than an incidental one. I am aware that HASD is specified as disabling *hardware autonomous* speed changes and so does not literally forbid a software-initiated retrain, so this is just a heuristic about firmware intent. About the drive disappearing: The error path is err: pci_info(dev, "retraining failed\n"); pcie_set_target_speed(dev, PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2), true); which reaches pcie_retrain_link(pdev, use_lt=true), whose last wait is rc = pcie_wait_for_link_status(pdev, use_lt, !use_lt); With use_lt set that waits for the Link Training bit to clear (for training to stop) and not for DLLLA to come back. So the restore can return having never confirmed the link recovered. Since the quirk runs from pci_device_add(), before pci_scan_bridge_extend() creates the subordinate bus, a link that comes up shortly afterwards is never scanned and the device behind it is simply not there. That would fit only booting sometimes better than anything about speed policy does. I could easily be wrong about this - I am just reading the code, and I came to the PCI core a few days ago via this one bug. Since you offered to test a diagnostic, here is one. It only touches the err: block, which is identical in mainline and in the 7.0.14 code your backport is based on, so it should apply to your tree (it applied to v7.1.5 here with a -2 line offset): diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index b09f27f..e4fcdfb 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -126,7 +126,32 @@ int pcie_failed_link_retrain(struct pci_dev *dev) return ret; err: pci_info(dev, "retraining failed\n"); + { + /* DIAGNOSTIC ONLY -- not for merging */ + u16 sta = 0, ctl2 = 0; + + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &sta); + pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &ctl2); + pci_info(dev, "diag: pre ret=%d DLLLA=%d sta=%#06x ctl2=%#06x\n", + ret, !!(sta & PCI_EXP_LNKSTA_DLLLA), sta, ctl2); + } pcie_set_target_speed(dev, PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2), true); + { + /* DIAGNOSTIC ONLY -- not for merging */ + u16 sta = 0, ctl2 = 0; + int i; + + for (i = 0; i <= 100; i++) { + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &sta); + if (sta & PCI_EXP_LNKSTA_DLLLA) + break; + msleep(10); + } + pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &ctl2); + pci_info(dev, "diag: post DLLLA=%d after %dms sta=%#06x\n", + !!(sta & PCI_EXP_LNKSTA_DLLLA), i * 10, sta); + pci_info(dev, "diag: post ctl2=%#06x\n", ctl2); + } return ret; } Expected output: - "diag: pre" gives the error the retrain returned and whether the link was already down at that point. - "diag: post DLLLA=1 after <n>ms" would mean the link does come back, just not before pci_device_add() returns and the bus below is scanned. That would point at the error path needing to wait for DLLLA rather than only for the Link Training bit to clear. - "diag: post DLLLA=0 after 1000ms" would mean the link is genuinely down and staying down, which is a different problem and probably a worse one. One caveat: the poll loop waits up to a second in the error path, so it changes timing. If the diagnostic build happens to boot reliably where the plain v3 build did not, that is also worth reporting. Best regards, Andreas Wild ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-08 6:31 ` Andreas Wild @ 2026-08-11 3:43 ` Aoxtj 2026-08-11 6:42 ` Andreas Wild 0 siblings, 1 reply; 19+ messages in thread From: Aoxtj @ 2026-08-11 3:43 UTC (permalink / raw) To: Andreas Wild, Aoxtj Cc: linux-pci, Bjorn Helgaas, Maciej W. Rozycki, linux-kernel On 2026/8/8 14:31, Andreas Wild wrote: > Expected output: > > - "diag: pre" gives the error the retrain returned and whether the > link was already down at that point. > - "diag: post DLLLA=1 after <n>ms" would mean the link does come back, > just not before pci_device_add() returns and the bus below is > scanned. That would point at the error path needing to wait for > DLLLA rather than only for the Link Training bit to clear. > - "diag: post DLLLA=0 after 1000ms" would mean the link is genuinely > down and staying down, which is a different problem and > probably a worse one. > > One caveat: the poll loop waits up to a second in the error path, so it > changes timing. If the diagnostic build happens to boot reliably where > the plain v3 build did not, that is also worth reporting. Diagnostic result: link is genuinely down, not recovering. Tested 7.0.14-8-pve++ (v3 patch + diagnostic patch) on the same machine. Boot fails: SK hynix never enumerates on bus 07. [0.542354] pci 0000:00:02.4: removing 2.5GT/s downstream link speed restriction [1.623696] pci 0000:00:02.4: retraining failed [1.623701] diag: pre ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023 [3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823 [3.734702] diag: post ctl2=0x0021 Sorry for taking so long to reply; I've been a little busy this week. Happy to test anything else. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-11 3:43 ` Aoxtj @ 2026-08-11 6:42 ` Andreas Wild 2026-08-24 14:26 ` Thorsten Leemhuis 0 siblings, 1 reply; 19+ messages in thread From: Andreas Wild @ 2026-08-11 6:42 UTC (permalink / raw) To: Aoxtj; +Cc: linux-pci, Bjorn Helgaas, Maciej W. Rozycki, linux-kernel On Mon, 11 Aug 2026, Aoxtj wrote: > Diagnostic result: link is genuinely down, not recovering. > > [1.623701] diag: pre ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023 > [3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823 > [3.734702] diag: post ctl2=0x0021 So the link really isn't coming back at all rather than too late for the bus scan. after the failed 8GT/s retrain (ret=-110, -ETIMEDOUT): LNKSTA 0x9023 CLS 8.0GT/s width x2 LT 0 DLLLA 0 LNKCTL2 0x0023 TLS 8.0GT/s HASD set after the restore, plus 1010 ms of polling: LNKSTA 0x9823 CLS 8.0GT/s width x2 LT 1 DLLLA 0 LNKCTL2 0x0021 TLS 2.5GT/s HASD set The register restore works, LNKCTL2 goes back to 2.5GT/s, but LT is still asserted a second later, with DLLLA never returning, so the port is stuck in link training rather than merely slow to recover. On the HASD bit: Your LNKCTL2 has bit 5 (PCI_EXP_LNKCTL2_HASD, "SpeedDis+") set in both samples. So firmware clamped the Target Link Speed to 2.5GT/s and also disabled hardware autonomous speed changes, on a link that turns out not to work at 8GT/s. If you would like something to try, the change below skips lifting the restriction if HASD is set. It applies to your tree (I checked it against v7.1.6 and on top of v3). Mainline would need a small adaptation, since that block no longer has the LNKCAP test. diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -115,6 +115,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev) pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap); if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT && (lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB) { + if (lnkctl2 & PCI_EXP_LNKCTL2_HASD) { + pci_info(dev, "2.5GT/s restriction left, firmware set HASD\n"); + return ret; + } + pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n"); ret = pcie_set_target_speed(dev, PCIE_LNKCAP_SLS2SPEED(lnkcap), false); if (ret) Some things to consider: I cannot verify the change myself - it compiles without warnings and that is all I can say. HASD is specified as disabling *hardware autonomous* speed changes, so it does not strictly forbid a software-initiated retrain. Interpreting it as "firmware meant this" is just my guess. As I said before, I have no background in this code. I came to it through one boot-time regression on my own machine, and that is the extent of it. So please take it as "here is one thing that could help this particular case", not as a view on how the quirk ought to work. Maciej and Bjorn are far better placed to judge whether the answer is this, or restoring the device ID match, or something else entirely. One further thought for people who know the code better than I do: Since retraining apparently cannot recover the link once it is wedged, would a secondary bus reset be the appropriate recovery in the error path? Best regards, Andreas ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-11 6:42 ` Andreas Wild @ 2026-08-24 14:26 ` Thorsten Leemhuis 2026-08-25 7:05 ` Andreas Wild 0 siblings, 1 reply; 19+ messages in thread From: Thorsten Leemhuis @ 2026-08-24 14:26 UTC (permalink / raw) To: Andreas Wild, Aoxtj Cc: linux-pci, Bjorn Helgaas, Maciej W. Rozycki, linux-kernel, Linux kernel regressions list On 8/11/26 08:42, Andreas Wild wrote: > On Mon, 11 Aug 2026, Aoxtj wrote: > >> Diagnostic result: link is genuinely down, not recovering. >> >> [1.623701] diag: pre ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023 >> [3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823 >> [3.734702] diag: post ctl2=0x0021 > > So the link really isn't coming back at all rather than too late for the > bus scan. > [...] > If you would like something to try, the change below skips lifting the > restriction if HASD is set. It applies to your tree (I checked it against > v7.1.6 and on top of v3). Mainline would need a small adaptation, since > that block no longer has the LNKCAP test. Side note: what's the status of all this? From the outside things looks stalled, but maybe there was progress and I just missed it. Are we waiting for a reply from Aoxtj before moving on with the fix at the start of the thread? And whatever the answer is: would it make sense to move on with that independently to get the other regression (boot time delay) resolved? Ciao, Thorsten > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -115,6 +115,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev) > pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap); > if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT && > (lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB) { > + if (lnkctl2 & PCI_EXP_LNKCTL2_HASD) { > + pci_info(dev, "2.5GT/s restriction left, firmware set HASD\n"); > + return ret; > + } > + > pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n"); > ret = pcie_set_target_speed(dev, PCIE_LNKCAP_SLS2SPEED(lnkcap), false); > if (ret) > > Some things to consider: > > I cannot verify the change myself - it compiles without warnings and that > is all I can say. > > HASD is specified as disabling *hardware autonomous* speed changes, so it > does not strictly forbid a software-initiated retrain. Interpreting it as > "firmware meant this" is just my guess. > > As I said before, I have no background in this code. I came to it through > one boot-time regression on my own machine, and that is the extent of it. > > So please take it as "here is one thing that could help this particular > case", not as a view on how the quirk ought to work. Maciej and Bjorn are > far better placed to judge whether the answer is this, or restoring the > device ID match, or something else entirely. > > One further thought for people who know the code better than I do: > > Since retraining apparently cannot recover the link once it is wedged, > would a secondary bus reset be the appropriate recovery in the error path? > > Best regards, > Andreas > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-24 14:26 ` Thorsten Leemhuis @ 2026-08-25 7:05 ` Andreas Wild 2026-08-25 10:18 ` Maciej W. Rozycki 0 siblings, 1 reply; 19+ messages in thread From: Andreas Wild @ 2026-08-25 7:05 UTC (permalink / raw) To: Thorsten Leemhuis, Aoxtj Cc: linux-pci, Bjorn Helgaas, Maciej W. Rozycki, linux-kernel, Linux kernel regressions list On Mon, 24 Aug 2026, Thorsten Leemhuis wrote: > Side note: what's the status of all this? From the outside things looks > stalled, but maybe there was progress and I just missed it. Nothing has moved, so you have not missed anything. > Are we waiting for a reply from Aoxtj before moving on with the fix at > the start of the thread? And whatever the answer is: would it make sense > to move on with that independently to get the other regression (boot > time delay) resolved? Not on my side. Aoxtj tested v3 early on and it does not help them, which is expected: their link is up at 2.5GT/s and v3 only skips ports with no link at all. The two are separate failure modes needing separate fixes, so I see no reason to hold v3 back. Best regards, Andreas ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-25 7:05 ` Andreas Wild @ 2026-08-25 10:18 ` Maciej W. Rozycki 2026-09-02 15:48 ` Thorsten Leemhuis 0 siblings, 1 reply; 19+ messages in thread From: Maciej W. Rozycki @ 2026-08-25 10:18 UTC (permalink / raw) To: Andreas Wild Cc: Thorsten Leemhuis, Aoxtj, linux-pci, Bjorn Helgaas, linux-kernel, Linux kernel regressions list On Tue, 25 Aug 2026, Andreas Wild wrote: > > Are we waiting for a reply from Aoxtj before moving on with the fix at > > the start of the thread? And whatever the answer is: would it make sense > > to move on with that independently to get the other regression (boot > > time delay) resolved? > > Not on my side. Aoxtj tested v3 early on and it does not help them, which > is expected: their link is up at 2.5GT/s and v3 only skips ports with no > link at all. The two are separate failure modes needing separate fixes, > so I see no reason to hold v3 back. I'd be happy to hear any results of your posted HASD check though. Maciej ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-25 10:18 ` Maciej W. Rozycki @ 2026-09-02 15:48 ` Thorsten Leemhuis 2026-09-07 12:33 ` Maciej W. Rozycki 0 siblings, 1 reply; 19+ messages in thread From: Thorsten Leemhuis @ 2026-09-02 15:48 UTC (permalink / raw) To: Maciej W. Rozycki, Bjorn Helgaas Cc: Aoxtj, linux-pci, Andreas Wild, linux-kernel, Linux kernel regressions list, restafvalendergelijke On 8/25/26 12:18, Maciej W. Rozycki wrote: > On Tue, 25 Aug 2026, Andreas Wild wrote: > >>> Are we waiting for a reply from Aoxtj before moving on with the fix at >>> the start of the thread? And whatever the answer is: would it make sense >>> to move on with that independently to get the other regression (boot >>> time delay) resolved? >> >> Not on my side. Aoxtj tested v3 early on and it does not help them, which >> is expected: their link is up at 2.5GT/s and v3 only skips ports with no >> link at all. The two are separate failure modes needing separate fixes, >> so I see no reason to hold v3 back. Thx for the clarification. Would afaics be really good to get that fix in to solve that problem. Bjorn, is this somewhere on your todo list? > I'd be happy to hear any results of your posted HASD check though. There is another report from Mich (now CCed). Mich, can you maybe test that if you haven't yet (your report is really verbose, which makes things somewhat hard to follow)? FWIW, Mich's report can be found here: https://bugzilla.kernel.org/show_bug.cgi?id=221919 Quoting from there: """ > Mich 2026-08-25 09:56:05 UTC > > Created attachment 310724 [details] > Various collection of log files. > > Overview > ======== > After the Vendor:Device ID matching was removed from the > pcie_failed_link_retrain() quirk, the kernel now lifts 2.5GT/s link speed > clamps that platform firmware has deliberately applied. On a Dell PowerEdge > R610 this causes the affected links to fail to come back up, the device > behind them to disappear from the PCI bus, and a fatal PCIe error resulting > in a kernel panic. > > Kernel version > ============== > Failing : 7.0.14-12-pve (Proxmox, based on Ubuntu-7.0.0-31.31) > Working : 7.0.14-5-pve (Proxmox, based on an earlier Ubuntu base) > > The Proxmox changelog shows 7.0.14-7 as the release that pulled in the large > upstream update (Ubuntu-7.0.0-28.28i3); all later releases in that series are > CVE cherry-picks only. Forum reports independently confirm -6 works and -7 > onwards do not. > > Hardware > ======== > Dell PowerEdge R610 (11G) > Chipset : Intel 5520/5500 I/O Hub > BIOS : 6.4.0 (2013-07-23) on one machine, 6.6.0 (latest) on the other > NIC : Intel 82571EB quad-port PCIe (the device that disappears) > RAID : LSI MegaRAID SAS 2108 (PERC H700) > > Reproduced on two separate R610 machines with different BIOS versions, so > this is not a firmware revision issue. > > What happens > ============ > [ 1.666183] pci 0000:00:07.0: removing 2.5GT/s downstream link speed restriction > <- exactly 1 second gap (link training timeout) > [ 2.666510] pci 0000:00:09.0: removing 2.5GT/s downstream link speed restriction > [ 2.673748] Uhhuh. NMI received for unknown reason 21 on CPU 0. > > Resulting link state (captured while booted with ghes.disable=y): > 00:07.0 LnkCap: Speed 5GT/s, Width x8 / LnkSta: Speed 2.5GT/s, Width x0 > 00:09.0 LnkCap: Speed 5GT/s, Width x8 / LnkSta: Speed 2.5GT/s, Width x0 > > Width x0 means the link is down. The Intel 82571EB installed behind one of > these ports is not enumerated at all - it is completely absent from lspci. > Only the onboard BCM5709 ports remain visible. > > Without ghes.disable=y the machine panics before reaching userspace: > Kernel panic - not syncing: GHES: Fatal hardware error > The call trace shows this happens during acpi_ghes_init: > acpi_ghes_init+0xd3/0x160 > acpi_init+0x401/0x490 > do_one_initcall+0x5f/0x350 > kernel_init_freeable+0x243/0x2f0 > > The front panel LCD and iDRAC SEL both report "E171F PCIe fatal". > > The ACPI BERT table is empty (48 bytes, header only), confirming this is a > live error raised during PCI enumeration rather than a stale firmware record. > > Suspected cause > =============== > Commit a89c82249c37 ("PCI: Work around PCIe link training failures") > introduced pcie_failed_link_retrain(), originally limited to the ASMedia > ASM2824 by Vendor:Device ID. > > A later change - "PCI: Always lift 2.5GT/s restriction in PCIe failed link > retraining" (Maciej W. Rozycki, Dec 2025) - removed that ID matching, so the > quirk now lifts 2.5GT/s clamps on any downstream port, explicitly including > clamps "the firmware may have already arranged". > > On this platform the firmware sets that clamp intentionally and the hardware > cannot negotiate a higher speed, so lifting it takes the link down entirely. > > During review of that patch, concerns were raised on linux-pci that lifting > firmware-set clamps on arbitrary hardware would cause regressions on real > systems. This appears to be exactly such a case. > > Workarounds tried > ================= > ghes.disable=y : machine boots, but the NIC is still missing, so the node is > unusable. Only suppresses the reporting. > pci=noaer : no effect (the link is already down before AER is involved). > Pinning kernel 7.0.14-5-pve : works, and is what we are currently using. > > Question > ======== > Would it be possible to restrict this quirk back to the original ASMedia > device IDs, or to skip lifting clamps that were set by firmware rather than > by the quirk itself? On platforms where firmware deliberately limits a link, > overriding that appears to be unsafe. > > Attachments > =========== > Full dmesg, lspci -nnvvv, dmidecode, ACPI tables (BERT/HEST) attached. > > Comment 1 The Linux kernel's regression tracker (Thorsten Leemhuis) 2026-08-26 04:07:17 UTC > > Please use latest upstream kernels when reporting problem upstream, those downstream kernels are patched in various ways and thus might have issues that never even happened upstream or were fixed long ago. Nevertheless: > > There is some discussion about an issue that might be the same here here: https://lore.kernel.org/all/20260801201244.4421-1-andiwild@gmail.com/ > > Would be best if you could try if the fix or one of the test-patches in that thread helps and report back there; alternatively, report back here please, ideally while allowing me to CC you on a mail about this (which would expose your email to the world) > > Comment 2 Mich 2026-08-26 04:59:18 UTC > > (In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #1) >> Please use latest upstream kernels when reporting problem upstream, those >> downstream kernels are patched in various ways and thus might have issues >> that never even happened upstream or were fixed long ago. Nevertheless: >> >> There is some discussion about an issue that might be the same here here: >> https://lore.kernel.org/all/20260801201244.4421-1-andiwild@gmail.com/ >> >> Would be best if you could try if the fix or one of the test-patches in that >> thread helps and report back there; alternatively, report back here please, >> ideally while allowing me to CC you on a mail about this (which would expose >> your email to the world) > > Thank you for the response, I will take a look at it! > I tried adding my "spam" collection mailbox, but I am not allowed unfortunately. If you want to add this to CC " > restafvalendergelijke@outlook.com" it's no problem. Thank you once again. > > Comment 3 Mich 2026-09-01 09:46:23 UTC > > Created attachment 310765 [details] > Full diagnostics attached (dmesg, lspci -nnvvv, dmidecode, ACPI tables). > > == Second platform, and a case the proposed v3 fix would NOT cover == > > Reproduced on a second machine from a different vendor: > > HP ProLiant DL360 G7, BIOS P68 (2015-08-16) > Intel 5520 I/O Hub (same chipset generation as the Dell R610) > Intel 82571EB quad-port NIC [8086:10bc] - same card as the R610s > Kernel 7.0.14-14-pve (Proxmox, Ubuntu-based). 7.0.14-5-pve works. > > This rules out a Dell-specific firmware quirk: two vendors, different BIOS > versions, same Intel 5520 chipset, identical failure mode. > > == The important difference from my earlier R610 report == > > On the R610 the affected ports were EMPTY: LnkSta Width x0, DLActive- , no > device behind them. That case is covered by the v3 approach (bail out when > Target Link Speed is already 2.5GT/s and no link has been established). > > On this HP machine one of the affected ports had an ACTIVE link with a > device behind it before the quirk ran: > > 00:08.0 before: LnkCap 5GT/s x4, LnkSta 2.5GT/s Width x4, DLActive+ > (Intel 82571EB, bus 04, working normally on 7.0.14-5) > after : LnkCap 5GT/s x4, LnkSta 2.5GT/s Width x0 > (bus 04 now empty, NIC absent from lspci) > > Since DLLLA was set, pcie_failed_link_retrain() would take the first branch > of the v3 patch and fall through to the clamp-removal block regardless. The > clamp is then lifted on a port with a working device attached, the link > fails to retrain, and the device is lost. > > If that reading is correct, v3 fixes the empty-port case but not this one. > I'm happy to be corrected - I'm reporting observed behaviour, not reading > the patch as an expert. > > == Full quirk trace on the HP == > > The quirk walks every root port, each with a one-second retrain timeout: > > [ 1.883010] pci 0000:00:02.0: removing 2.5GT/s downstream link speed restriction > [ 2.882097] pci 0000:00:02.0: retraining failed > [ 2.882422] pci 0000:00:03.0: removing 2.5GT/s downstream link speed restriction > [ 3.882097] pci 0000:00:03.0: retraining failed > [ 3.882441] pci 0000:00:04.0: removing 2.5GT/s downstream link speed restriction > [ 4.882097] pci 0000:00:04.0: retraining failed > [ 4.882417] pci 0000:00:05.0: removing 2.5GT/s downstream link speed restriction > [ 5.882097] pci 0000:00:05.0: retraining failed > [ 5.882409] pci 0000:00:06.0: removing 2.5GT/s downstream link speed restriction > [ 6.882097] pci 0000:00:06.0: retraining failed > [ 6.882425] pci 0000:00:07.0: removing 2.5GT/s downstream link speed restriction > [ 6.882746] pci 0000:00:08.0: removing 2.5GT/s downstream link speed restriction > [ 7.882097] pci 0000:00:08.0: retraining failed > [ 7.882415] pci 0000:00:09.0: removing 2.5GT/s downstream link speed restriction > [ 7.882742] pci 0000:00:0a.0: removing 2.5GT/s downstream link speed restriction > [ 7.906501] NMI: IOCK error (debug interrupt?) for reason 61 on CPU 0. > [ 8.882097] pci 0000:00:0a.0: retraining failed > > Note 00:07.0 gets the same treatment but is NOT reported as failed - the > onboard BCM5709 behind it survives at Width x4. So on the same machine, in > the same boot, one device survives the retrain and another does not. > > Also of note: roughly 7 seconds of boot time are spent on these one-second > timeouts, on a machine where nothing needed retraining in the first place. > > == Firmware clamps every port on both platforms == > > Dell R610 (working kernel, quirk not triggered): > 00:01.0 LnkCap 5GT/s x4 / LnkSta 2.5GT/s, Width x4 BCM5709, link up > 00:03.0 LnkCap 5GT/s x4 / LnkSta 2.5GT/s, Width x4 BCM5709, link up > 00:07.0 LnkCap 5GT/s x8 / LnkSta 2.5GT/s, Width x0 empty > 00:09.0 LnkCap 5GT/s x8 / LnkSta 2.5GT/s, Width x0 empty (Intel NIC on the failing node) > > HP DL360 G7: same pattern across nine root ports. > > Both platforms clamp every 5GT/s-capable port to 2.5GT/s, occupied or not. > Moving the card to another slot is therefore not a workaround. > > Full diagnostics attached (dmesg, lspci -nnvvv, dmidecode, ACPI tables). > """ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-02 15:48 ` Thorsten Leemhuis @ 2026-09-07 12:33 ` Maciej W. Rozycki 0 siblings, 0 replies; 19+ messages in thread From: Maciej W. Rozycki @ 2026-09-07 12:33 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Bjorn Helgaas, Aoxtj, linux-pci, Andreas Wild, linux-kernel, Linux kernel regressions list, restafvalendergelijke On Wed, 2 Sep 2026, Thorsten Leemhuis wrote: > Thx for the clarification. Would afaics be really good to get that fix > in to solve that problem. Bjorn, is this somewhere on your todo list? > > I'd be happy to hear any results of your posted HASD check though. > There is another report from Mich (now CCed). Mich, can you maybe test > that if you haven't yet (your report is really verbose, which makes > things somewhat hard to follow)? Thank you for the update. I'm back from holiday today and will review the situation in detail in the coming few days. Maciej ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-08-01 20:11 [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link Andreas Wild 2026-08-03 5:39 ` Thorsten Leemhuis 2026-08-07 19:59 ` Aoxtj @ 2026-09-17 23:33 ` Bjorn Helgaas 2026-09-18 5:39 ` Thorsten Leemhuis 2 siblings, 1 reply; 19+ messages in thread From: Bjorn Helgaas @ 2026-09-17 23:33 UTC (permalink / raw) To: Andreas Wild Cc: linux-pci, bhelgaas, macro, linux-kernel, M M, Thorsten Leemhuis, regressions [+cc M M, Thorsten] On Sat, Aug 01, 2026 at 10:11:48PM +0200, Andreas Wild wrote: > From: "Maciej W. Rozycki" <macro@orcam.me.uk> > > Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe > failed link retraining") the Target Speed quirk lifts a firmware-imposed > 2.5GT/s restriction on any downstream port, without checking whether the > link is up. Where nothing is plugged in, the retraining that follows can > never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The > quirk makes two of them -- the initial one and the restore on the error > path -- adding a fixed 2 s to every boot. > > On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1 > slot, running v7.2-rc5: > > 0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction > 1.541 pci 0000:00:1c.0: retraining failed > 2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400 > > Where the Link Speed has already been clamped at 2.5GT/s and no link has > been established there is nothing worth doing, which is what the kerneldoc > for the quirk already describes: the restriction is to be lifted where > firmware arranged it "and the port reports its link already being up". > Bail out early in that case, before either the ASM2824 workaround or the > removal of the restriction is considered. > > Ports whose link is up are unaffected, and so is the ASM2824 workaround, > which is reached with the Target Link Speed not clamped. > > With this applied the quirk returns without touching the port: both > messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2 > in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from > 3.011 s to 1.036 s. > > Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining") > Cc: stable@vger.kernel.org > Reported-by: Andreas Wild <andiwild@gmail.com> > Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/ > Tested-by: Andreas Wild <andiwild@gmail.com> > Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk> > Signed-off-by: Andreas Wild <andiwild@gmail.com> I provisionally applied this on pci/misc. It seems to address at least part of a v7.2 regression, so it probably should go on pci/for-linus for v7.3. But I'd like to have it in -next for a bit before asking Linus to pull it. I added: Closes: https://bugzilla.kernel.org/show_bug.cgi?id=221801 based on Edward's testing at https://bugzilla.kernel.org/show_bug.cgi?id=221801#c13. Edward, I bcc'd you because I haven't seen a public email response, but if you want your Tested-by to appear in the git commit, just respond to this email thread. But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining") even with this patch (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc) And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3) So evidently there are still issues but maybe this patch fixes part of them? > --- > v3: > - Replaced with Maciej's much simpler approach: bail out of the quirk > entirely when the Target Link Speed is already clamped at 2.5GT/s and no > link has been established, rather than programming the speed and skipping > only the retraining. One function, no new API, no bwctrl changes. > - Note this leaves the Target Link Speed clamped on such a port, where v2 > left it at the Port's maximum. A device hot-plugged there later trains at > 2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when > pcie_wait_for_link_status() fails, so a link that comes up cleanly at > 2.5GT/s never re-runs it. Flagging in case that matters; the clamp is > firmware's, so honouring it on an unoccupied Port seems defensible. > - v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/ > - v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/ > drivers/pci/quirks.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index b09f27f..9e407c4 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev) > > pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); > pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2); > - if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) { > + if (lnksta & PCI_EXP_LNKSTA_DLLLA) { > + ; > + } else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) { > + return ret; > + } else if (pcie_lbms_seen(dev, lnksta)) { > pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n"); > ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false); > if (ret) > -- > 2.55.0 > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-17 23:33 ` Bjorn Helgaas @ 2026-09-18 5:39 ` Thorsten Leemhuis 2026-09-18 10:39 ` Maciej W. Rozycki 0 siblings, 1 reply; 19+ messages in thread From: Thorsten Leemhuis @ 2026-09-18 5:39 UTC (permalink / raw) To: Bjorn Helgaas, Andreas Wild Cc: linux-pci, bhelgaas, macro, linux-kernel, M M, regressions, restafvalendergelijke On 9/18/26 01:33, Bjorn Helgaas wrote: > On Sat, Aug 01, 2026 at 10:11:48PM +0200, Andreas Wild wrote: >> From: "Maciej W. Rozycki" <macro@orcam.me.uk> >> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe >> failed link retraining") the Target Speed quirk lifts a firmware-imposed >> 2.5GT/s restriction on any downstream port, without checking whether the >> link is up. Where nothing is plugged in, the retraining that follows can >> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS. The >> quirk makes two of them -- the initial one and the restore on the error >> path -- adding a fixed 2 s to every boot. >> >> On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1 >> slot, running v7.2-rc5: >> [...] >> Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining") >> Cc: stable@vger.kernel.org >> Reported-by: Andreas Wild <andiwild@gmail.com> >> Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/ >> Tested-by: Andreas Wild <andiwild@gmail.com> >> Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk> >> Signed-off-by: Andreas Wild <andiwild@gmail.com> > > I provisionally applied this on pci/misc. It seems to address at > least part of a v7.2 regression, so it probably should go on > pci/for-linus for v7.3. > > But I'd like to have it in -next for a bit before asking Linus to pull > it. [...] Great, thx -- and yes, -next sounds totally fine for me, as this fixes a delay, not a hard breakage. > But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift > 2.5GT/s restriction in PCIe failed link retraining") even with this > patch > (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc) > > And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by > this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3) > > So evidently there are still issues but maybe this patch fixes part of > them? Yeah, Maciej ten days expressed that we to look into this. Anyway, I suggest someone that is affected by that problem starts a new thread (please CC all those that are affected by it and the regression list) with summarizing the current state + this patch, as this thread got a bit confusing... (please drop a link to that thread here afterwards) Ciao, Thorsten ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-18 5:39 ` Thorsten Leemhuis @ 2026-09-18 10:39 ` Maciej W. Rozycki 2026-09-18 11:00 ` Thorsten Leemhuis 0 siblings, 1 reply; 19+ messages in thread From: Maciej W. Rozycki @ 2026-09-18 10:39 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Bjorn Helgaas, Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, M M, regressions, M M On Fri, 18 Sep 2026, Thorsten Leemhuis wrote: > > But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift > > 2.5GT/s restriction in PCIe failed link retraining") even with this > > patch > > (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc) > > > > And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by > > this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3) > > > > So evidently there are still issues but maybe this patch fixes part of > > them? > Yeah, Maciej ten days expressed that we to look into this. Anyway, I > suggest someone that is affected by that problem starts a new thread > (please CC all those that are affected by it and the regression list) > with summarizing the current state + this patch, as this thread got a > bit confusing... (please drop a link to that thread here afterwards) It remains on my radar, no worries. Hectic time here, a lot is going on. I'll do my best to look into it as soon as possible. Also I'll be at LPC in Prague in a couple of weeks' time, so this may be an opportunity to discuss the issues in person. Maciej ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-18 10:39 ` Maciej W. Rozycki @ 2026-09-18 11:00 ` Thorsten Leemhuis 2026-09-18 12:18 ` Maciej W. Rozycki 0 siblings, 1 reply; 19+ messages in thread From: Thorsten Leemhuis @ 2026-09-18 11:00 UTC (permalink / raw) To: Maciej W. Rozycki Cc: Bjorn Helgaas, Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, M M, regressions On 9/18/26 12:39, Maciej W. Rozycki wrote: > On Fri, 18 Sep 2026, Thorsten Leemhuis wrote: > >>> But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift >>> 2.5GT/s restriction in PCIe failed link retraining") even with this >>> patch >>> (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc) >>> >>> And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by >>> this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3) >>> >>> So evidently there are still issues but maybe this patch fixes part of >>> them? >> Yeah, Maciej ten days expressed that we to look into this. Anyway, I >> suggest someone that is affected by that problem starts a new thread >> (please CC all those that are affected by it and the regression list) >> with summarizing the current state + this patch, as this thread got a >> bit confusing... (please drop a link to that thread here afterwards) > > It remains on my radar, no worries. Hectic time here, a lot is going on. > I'll do my best to look into it as soon as possible. Also I'll be at LPC > in Prague in a couple of weeks' time, so this may be an opportunity to > discuss the issues in person. Sorry, I didn't want to prod you; I mainly wrote what I wrote because the reports about the two problems got mixed up, so things became confusing -- which is why I think a new thread starting with a summary might be best for all of us. That being said and wrt to the "a lot going on": seems this causes some devices to not work for iirc at least two reporters. Would it maybe be better to revert the culprit and reapply it later once that is sorted out? Or is this not an option or a bad one for some reason? Reminder, normally Linus in a situation like this wants regressions ideally 'fixed "within a week", preferably before the next rc': https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/ Ciao, Thorsten ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-18 11:00 ` Thorsten Leemhuis @ 2026-09-18 12:18 ` Maciej W. Rozycki 2026-09-18 12:50 ` Thorsten Leemhuis 0 siblings, 1 reply; 19+ messages in thread From: Maciej W. Rozycki @ 2026-09-18 12:18 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Bjorn Helgaas, Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, M M, regressions On Fri, 18 Sep 2026, Thorsten Leemhuis wrote: > That being said and wrt to the "a lot going on": seems this causes some > devices to not work for iirc at least two reporters. Would it maybe be > better to revert the culprit and reapply it later once that is sorted > out? Or is this not an option or a bad one for some reason? Reminder, > normally Linus in a situation like this wants regressions ideally 'fixed > "within a week", preferably before the next rc': > https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/ Reverting will cause other configurations to regress, so this would be replacing one regression with another. We seem to be in a situation where there are all kinds of breakage out there worked around by the system boot firmware by capping the link speed, and this quirk interferes with that. One way to handle this could be by having a list of downstream devices never to remove link capping for. But that seems like a recipe for an ongoing maintaince burden. Ideally we'd handle all the cases gracefully, even possibly out-of-spec ones, but it seems more and more evident that this may not be possible. It may be that the quirk will have to be dropped entirely after all. Thoughts? Maciej ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link 2026-09-18 12:18 ` Maciej W. Rozycki @ 2026-09-18 12:50 ` Thorsten Leemhuis 0 siblings, 0 replies; 19+ messages in thread From: Thorsten Leemhuis @ 2026-09-18 12:50 UTC (permalink / raw) To: Maciej W. Rozycki Cc: Bjorn Helgaas, Andreas Wild, linux-pci, Bjorn Helgaas, linux-kernel, M M, regressions On 9/18/26 14:18, Maciej W. Rozycki wrote: > On Fri, 18 Sep 2026, Thorsten Leemhuis wrote: >> That being said and wrt to the "a lot going on": seems this causes some >> devices to not work for iirc at least two reporters. Would it maybe be >> better to revert the culprit and reapply it later once that is sorted >> out? Or is this not an option or a bad one for some reason? Reminder, >> normally Linus in a situation like this wants regressions ideally 'fixed >> "within a week", preferably before the next rc': >> https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/ Disclaimer/reminder: I'm an outsider trying to ensure regressions are handled well, I'm neither affected by the problems nor involved in PCI subsystem development. > Reverting will cause other configurations to regress, so this would be > replacing one regression with another. Well, guess so, as regressions and fixes for them iirc have stacked here, so there might be no good established base to go back to. But *normally* going back to one is what Linus afaics wants to actually prevent exactly a back-and-forth or an exchange of one bug for another. He wants to ensure that people can always feel safe to update. See these quotes from Linus for details: https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-back-and-forth So yes, not reverting might be the best at this point, as 7.2 is out for a while already and so far there are only two reports iirc. You folks are the best to judge here. That's why I'll also leave the other aspects you raised to others. Ciao, Thorsten ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2026-09-18 12:51 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-08-01 20:11 [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link Andreas Wild 2026-08-03 5:39 ` Thorsten Leemhuis 2026-08-03 22:07 ` Maciej W. Rozycki 2026-08-04 5:32 ` Thorsten Leemhuis 2026-08-07 19:59 ` Aoxtj 2026-08-08 6:31 ` Andreas Wild 2026-08-11 3:43 ` Aoxtj 2026-08-11 6:42 ` Andreas Wild 2026-08-24 14:26 ` Thorsten Leemhuis 2026-08-25 7:05 ` Andreas Wild 2026-08-25 10:18 ` Maciej W. Rozycki 2026-09-02 15:48 ` Thorsten Leemhuis 2026-09-07 12:33 ` Maciej W. Rozycki 2026-09-17 23:33 ` Bjorn Helgaas 2026-09-18 5:39 ` Thorsten Leemhuis 2026-09-18 10:39 ` Maciej W. Rozycki 2026-09-18 11:00 ` Thorsten Leemhuis 2026-09-18 12:18 ` Maciej W. Rozycki 2026-09-18 12:50 ` Thorsten Leemhuis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®