From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F104C41535 for ; Fri, 10 Nov 2023 18:11:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345130AbjKJSKr (ORCPT ); Fri, 10 Nov 2023 13:10:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344597AbjKJSI0 (ORCPT ); Fri, 10 Nov 2023 13:08:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 615F339741 for ; Fri, 10 Nov 2023 06:35:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699626905; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E0jbSLocU8Qu+t0aHk24/qSe4dawu00kk74v0Zid600=; b=hWrlm3a9XDPdDm2Vpq5VRRM4IfYZuRjPEOrhTCg+IEC993f5XOyb/EpS5MRsrw0OCkOZPQ o9NOSS6HB990vu0npdjCtxZd4obCV2r+C/c2v7yrLnlI6iB/af04S4iTfYtVxUOCSNmS98 bbPj2MFn2v1wWLU7LfewSdtsAmIo6fY= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-58-vw51FS5xPzCGJNkOHf0xIg-1; Fri, 10 Nov 2023 09:35:03 -0500 X-MC-Unique: vw51FS5xPzCGJNkOHf0xIg-1 Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-9e5e190458dso111362266b.1 for ; Fri, 10 Nov 2023 06:35:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699626902; x=1700231702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E0jbSLocU8Qu+t0aHk24/qSe4dawu00kk74v0Zid600=; b=CJ8TpECY37XJ/SzJoPrk4rcIrquSClxjVMwqDOKweGgHrwHnHhiuMqYOewp7ce/49d S6CxihXKfUvTP24lyDbl6k3sDnu78/U4yTFurR1berBqPsk0jInevEO1G9TN4lFbfyI/ e5A5WbArxlDhniTRx85wwUduucbOMMKhYvPXbfHAAo4yGr7Ph+E8TpdRLq7mJc6Hq5cI 5v1smoTvpUfnW4H+WNAk084tfPUzzHFLewW4GSlJAwv1dg8QU2sGlvB453cgUlPm0af1 /87KF1OqEvbJHbZMqHqF1EOLDEKInd+hDpQbMD0xEnhn/g72wIQ2dfEgCRPENMu8TFg2 I/Dg== X-Gm-Message-State: AOJu0YwBnj8My7UDOKS2x3PRjLNH4cGQKEEKvx1E6IBE+CRGrrvX/bJe Siw/2ICe1uLZIMdsZ0P/FxvJ+IrnzPCtSAJLMHkwfKvF6YX89tt0X3BbmVj+wvOgzThfRZ//E0J 7SXLl3fBxW5bUtIyKYF1uQYjC X-Received: by 2002:a17:906:1c8e:b0:9d5:96e7:5ae1 with SMTP id g14-20020a1709061c8e00b009d596e75ae1mr2213150ejh.12.1699626902631; Fri, 10 Nov 2023 06:35:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IH9uHdtYMXLxTcY/0b1/LwOT+o0u++Jc/LSsPYHzKMr5iLqhZpN/FN2epi+fQaVQ3HJwY3CwQ== X-Received: by 2002:a17:906:1c8e:b0:9d5:96e7:5ae1 with SMTP id g14-20020a1709061c8e00b009d596e75ae1mr2213136ejh.12.1699626902351; Fri, 10 Nov 2023 06:35:02 -0800 (PST) Received: from imammedo.users.ipa.redhat.com ([185.140.112.229]) by smtp.gmail.com with ESMTPSA id ti11-20020a170907c20b00b009e62bb0439bsm909806ejc.212.2023.11.10.06.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 06:35:01 -0800 (PST) Date: Fri, 10 Nov 2023 15:35:00 +0100 From: Igor Mammedov To: "Maciej W. Rozycki" Cc: Bjorn Helgaas , Stefan Roese , Jim Wilson , David Abdurachmanov , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] pci: Work around ASMedia ASM2824 PCIe link training failures Message-ID: <20231110153500.316862fd@imammedo.users.ipa.redhat.com> In-Reply-To: References: X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, 2 Jan 2022 23:25:25 +0000 (GMT) "Maciej W. Rozycki" wrote: > Attempt to handle cases with a downstream port of the ASMedia ASM2824 > PCIe switch where link training never completes and the link continues > switching between speeds indefinitely with the data link layer never > reaching the active state. > > It has been observed with a downstream port of the ASMedia ASM2824 Gen 3 > switch wired to the upstream port of the Pericom PI7C9X2G304 Gen 2 > switch, using a Delock Riser Card PCI Express x1 > 2 x PCIe x1 device, > P/N 41433, wired to a SiFive HiFive Unmatched board. In this setup the > switches are supposed to negotiate the link speed of preferably 5.0GT/s, > falling back to 2.5GT/s. It looks like I might have similar issue on EPYC system with this switch. In my case ASM2824 switch board connected to EPYC's root port (via SlimSAS 8i cable and on upstream site it negotiates expected 8GT/s x8) were supposed to provide 10x slots with PCIe x1 Gen3 links. However lspci shows that link is capped at 2.5GT/s and NVME plugged into one of the slots is downgraded to 2.5GT/s. I wonder if it could be fixed to get expected 8GT/s per lane. Downstream port in question: 07:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L1, Exit Latency L1 <64us ClockPM+ Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1 TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 0W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- ARIFwd+ AtomicOpsCap: Routing- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd- AtomicOpsCtl: EgressBlck- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+, Selectable De-emphasis: -3.5dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [c0] Subsystem: Device 0000:0000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [140 v1] Power Budgeting Capabilities: [160 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [180 v1] Multicast McastCap: MaxGroups 64, ECRCRegen- McastCtl: NumGroups 1, Enable- McastBAR: IndexPos 0, BaseAddr 0000000000000000 McastReceiveVec: 0000000000000000 McastBlockAllVec: 0000000000000000 McastBlockUntransVec: 0000000000000000 McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000 Capabilities: [1c0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [200 v1] Vendor Specific Information: ID=001a Rev=0 Len=000 Kernel driver in use: pcieport NVME: 08:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT (DRAM-less) NVMe SSD Controllers (rev 03) (prog-if 02 [NVM Express]) Subsystem: Biwin Storage Technology Co., Ltd. Device 2263 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-