Re: [PATCH v2] PCI: qcom: Block accesses to downstream devices on link down

[Bjorn Helgaas <helgaas@kernel.org>]

On Wed, Sep 16, 2026 at 09:03:26AM +0100, Marc Zyngier wrote:
> On Wed, 16 Sep 2026 00:14:21 +0100,
> Bjorn Helgaas <helgaas@kernel.org> wrote:
> > 
> > [+cc James, Andrew, Marc for guidance on generic arm64 PCIe error
> > recovery; beginning of thread:
> > https://lore.kernel.org/all/20260819-ecam_blocker-v2-1-e7a8fdc1c5cb@oss.qualcomm.com]
> > 
> > On Tue, Sep 15, 2026 at 07:16:11PM +0200, Manivannan Sadhasivam wrote:
> > > On Fri, Sep 11, 2026 at 12:18:07PM -0500, Bjorn Helgaas wrote:
> > > > On Fri, Sep 11, 2026 at 08:17:29AM +0200, Manivannan Sadhasivam wrote:
> > > > > On Tue, Sep 08, 2026 at 06:00:30PM -0500, Bjorn Helgaas wrote:
> > > > > > On Wed, Aug 19, 2026 at 11:36:54PM -0700, Qiang Yu wrote:
> > > > > > > After a PCIe link goes down, software may still access the
> > > > > > > BAR (MMIO) space or configuration space of devices behind
> > > > > > > that link before recovery has run. As the link is down,
> > > > > > > these accesses never complete, resulting in a storm of
> > > > > > > Completion Timeout AERs.
> > > > > > 
> > > > > > What is special about qcom here?  It seems like the Completion
> > > > > > Timeouts and AER interrupts should happen with every PCIe
> > > > > > controller.
> > > > > 
> > > > > The special behavior which is common across many (not all) ARM
> > > > > SoCs is that they don't synthesize all-one response for
> > > > > completion timeouts, unlike RCs in x86 machines. Rather, they
> > > > > return AXI error response, resulting in CPU treating them as
> > > > > SError, in-addition to AER storm.
> > > > > 
> > > > > Commit message missed mentioning SError though.
> > > > 
> > > > I don't know how SError works, but this sounds like a pretty big
> > > > open issue with respect to RAS.  I don't think we want a kernel
> > > > panic because a device failed to respond to a config or MMIO
> > > > access, e.g., if a card or Thunderbolt cable got unplugged.
> > 
> > I still don't know anything about arm64 or SError, but I see these KVM
> > commits about using ESB to manage SError in some cases:
> > 
> >   0e5b9c085dce ("KVM: arm64: Consume pending SError as early as possible")
> >   472fc011ccd3 ("KVM: arm64: nVHE: Don't consume host SErrors with ESB")
> > 
> > I assume it's not possible or practical to use ESB in the native host
> > case to deal with issues like this in a generic way instead of the
> > vendor-specific feature Qcom is using here?
> 
> In general, an SError is fatal. No ifs, no buts. It is asynchronous,
> and has no syndrome information. Once you find out about it, you're
> already dead.
> 
> ESB is a way to paper over this sorry state of affair by making sure
> that the error is observed at a known point. You can think of it as a
> barrier synchronising potential errors for transactions in flight. KVM
> uses that to attribute SErrors triggered by a guest, on an exception
> boundary.
> 
> That's not applicable to the kernel itself, unless you want to place
> an ESB after each and every load/store in the kernel (/s).
> 
> Modern versions of the architecture allow these errors to be reported
> as *synchronous* exceptions, with all the syndrome information you
> want. I guess this HW doesn't implement it, which is a pretty big
> mistake for systems that allow surprise removal of devices...

Thanks a lot for this background, Marc.

I guess we should take advantage of this qcom feature to at least
avoid some issues.  I think we just need to update the commit log to
clarify:

  - The reason this is important: apparently lots of AER interrupts
    but sounds like unrecoverable SError in some cases; I think it
    would be useful to understand the differences and whether CT and
    UR are handled differently.

  - What the result of the ECAM blocker is.  IIUC it blocks both
    config and MMIO.  I assume it prevents transactions from reaching
    PCIe so they don't cause UR or CT; maybe writes are dropped and
    reads return ~0 for both config and MMIO?

  - This is best-effort but not 100% because accesses between
    link-down and ECAM blocker being enabled can still cause AER
    interrupts and SError.

I'm more interested in these PCIe-level details than in the nitty
gritty of programming the controller; we need that too, but maybe some
of that can be addressed by code comments, e.g., the mutex could have
a one-line comment about avoiding races with RP resets via AER
recovery.