Re: PCI IO ECS access is no longer possible for AMD family 17h

[Yazen Ghannam <yazen.ghannam@amd.com>]

On Mon, Mar 03, 2025 at 01:20:21PM +0100, Rostyslav Khudolii wrote:
> * Ingo Molnar <mingo@kernel.org> wrote:
> >
> >
> > * Rostyslav Khudolii <ros@qtec.com> wrote:
> >
> > > Hi,
> > >
> > > > Rostyslav, I would like to ask you, do you have patches / updates for
> > > > enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> > > > adjust my lspci changes for new machines.
> > >
> > > Pali, sorry for the late reply. Do I understand correctly, that even
> > > though you have access to the ECS via
> > > the MMCFG you still want the legacy (direct IO) to work for the
> > > debugging purposes? I can prepare a
> > > simple patch that will allow you to do so if that's the case.
> > >
> > > >
> > > > So what is the practical impact here? Do things start breaking
> > > > unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled?
> > > > Then I'd suggest fixing that in the Kconfig space, either by adding a
> > > > dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those
> > > > must-have pieces of infrastructure.
> > > >
> > >
> > > Ingo, thank you for the reply.
> > >
> > > The way I understand the access to the PCI ECS (via raw_pci_ext_ops)
> > > works, is the following:
> > > 1. If CONFIG_ACPI_MCFG or CONFIG_PCI_MMCONFIG are enabled - set the
> > > raw_pci_ext_ops to use
> > >     MMCFG to access ECS. See pci_mmcfg_early_init() / pci_mmcfg_late_init();
> > > 2. If CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled - set the
> > > raw_pci_ext_ops to use
> > >     the 'direct' access to ECS. See pci_direct_init(). The direct
> > > access is conditional on the PCI_HAS_IO_ECS
> > >     flag being set.
> > >
> > > On AMD, the kernel enables the ECS IO access via the
> > > amd_bus_cpu_online() and pci_enable_pci_io_ecs().
> > > Except those functions have no desired effect on the AMD 17h+ family
> > > because the register (EnableCf8ExtCfg),
> > > they access, has been moved. What is important though, is that the
> > > PCI_HAS_IO_ECS flag is set unconditionally.
> > > See pci_io_ecs_init() in amd_bus.c
> > >
> > > Therefore I was wondering whether we should add support for the 17h+
> > > family in those functions to have
> > > the direct access work for those families as well.
> >
> > Yeah, I think so, but I'm really just guessing:
> >
> > > Regarding your suggestion to address this in the Kconfig space - I'm
> > > not quite sure I follow, since right now the kernel will use
> > > raw_pci_ext_ops whenever access beyond the first 256 bytes is
> > > requested. Say we want to make that conditional on CONFIG_ACPI_MCFG
> > > and CONFIG_PCI_MMCONFIG, does it also mean then we want to drop
> > > support for the 'direct' PCI IO ECS access altogether?
> >
> > I thought that enabling CONFIG_ACPI_MCFG would solve the problem, and
> > other architectures are selecting it pretty broadly:
> >
> >  arch/arm64/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
> >  arch/loongarch/Kconfig: select ACPI_MCFG if ACPI
> >  arch/riscv/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
> >
> > While x86 allows it to be user-configured, which may result in
> > misconfiguration, given that PCI_HAS_IO_ECS is being followed
> > unconditionally if a platform provides it?
> >
> > Thanks,
> >
> >         Ingo
> 
> So it seems that the config option that needs to be enabled is
> CONFIG_PCI_MMCONFIG.
> The MCFG kernel support depends on it (aka not compiled when disabled).
> With that said - the IO ECS question remains. Do we want to support it
> for AMD 17h+ or
> it's not required anymore?
> 

Hi all,

I'd say it is not required (nor encouraged) anymore based on my reading
of the AMD docs and PCI Firmware spec.

The possibility for direct access is still in hardware for backwards
compatibility. But this doesn't mean that it *must* be used. And it
seems that it shouldn't be used, if possible.

I'm for the option to address this issue through Kconfig.

Ros,
Is/was there a reason why you didn't have the MCFG/MMCONFIG options
enabled in your kernel?

Was this a side effect of trying to build a minimal kernel or similar?

Thanks,
Yazen

P.S. Sorry for the late reply. My mailbox is missing Ros's reply to me.