Re: [PATCH v23 2/5] PCI: Allow per function PCI slots to fix slot reset on s390

[Niklas Schnelle <schnelle@linux.ibm.com>]

On Fri, 2026-08-14 at 08:26 -0500, Bjorn Helgaas wrote:
> On Fri, Aug 14, 2026 at 10:46:28AM +0200, Niklas Schnelle wrote:
> > On Thu, 2026-08-13 at 18:25 -0500, Bjorn Helgaas wrote:
> > > On Wed, Aug 05, 2026 at 09:55:15AM -0700, Farhan Ali wrote:
> > > > On s390 systems, which use a machine level hypervisor, PCI devices are
> > > > always accessed through a form of PCI pass-through which fundamentally
> > > > operates on a per PCI function granularity. This is also reflected in the
> > > > s390 PCI hotplug driver which creates hotplug slots for individual PCI
> > > > functions. Its reset_slot() function, which is a wrapper for
> > > > zpci_hot_reset_device(), thus also resets individual functions.
> > > 
> > > Sorry to come back to this yet again.  I understand the issue with
> > > the wrong pci_slot being assigned for these s390 functions.
> > > 
> > > What I don't understand is why we would use slot_reset() in the first
> > > place.  I would expect FLR instead.
> > > 
> > > The hotplug slot_reset() path is used by pci_reset_bus_function().
> > > But given the order in pci_reset_fn_methods[], we would typically try
> > > pcie_reset_flr() first, and we would only get to
> > > pci_reset_bus_function() if FLR and the other resets are not
> > > available.
> > > 
> > > Since these are actually multi-function devices, I'm surprised that
> > > they wouldn't advertise FLR support.
> > 
> > Good question. The problem isn't that FLR isn't advertised or
> > unsupported. Rather we end up needing to use the slot reset when the
> > platform has put the PCI function in the architected error state which
> > blocks both MMIO and DMA similar to DPC and which we can only get out
> > of with the platform specific CLP Set PCI Function Disable/Enable
> > hypercalls. FLR still works if you have a function that wasn't put in
> > the error state but for most real world errors as well as some service
> > scenarios we do end up in the error state where a FLR won't work.
> 
> I assume these are standard PCIe devices, but this architected error
> state doesn't sound like something from the PCIe spec.
> 
> DPC works by disabling the link, but of course that blocks traffic to
> all the functions of an MFD, so maybe this is some s390-specific thing
> outside the endpoint, e.g., something in a Downstream Port that can
> selectively block traffic to/from a specific function?

Yes the error state is an s390 concept. It's implemented by firmware
which controls the PCIe root controllers which are hidden from Linux.
This firmware also controls the s390 HW IOMMU, though Linux controls
the translation tables. And it even controls the PCIe switches between
the root port and the endpoint. So firmware can disable MMIO, block DMA
and interrupts on a per function basis without dropping the link. If
the link is dropped it will of course affect the entire multi-function
device or even an entire I/O cage but then firmware will do the PCIe
level resets, link recovery etc. Then it will generate error events for
all affected functions and require a CLP based reset for each
individually. So even in that case we still work through it on a per
PCI function basis.

> 
> To get to pci_reset_bus_function() where we can use the slot reset, I
> think all the previous methods, including pcie_reset_flr(), must have
> failed with -ENOTTY.  But I don't see a place that would do that.
> Maybe you remove the other methods from dev->reset_methods[]?

In our normal error recovery flow we use zpci_hot_reset_device()
directly. But with Farhan's series QEMU now needs to drive the reset
through vfio-pci. This happens on behalf of a guest which itself does
the CLP Set PCI Function Disable.

> 
> In addition to whatever the CLP Set PCI Function Disable/Enable
> hypercall does to unblock traffic to/from the endpoint, I suppose it
> does an FLR internally? 

Yes that's at least one of the options, it can also do PERST#, swap
links to alternate paths retrain links etc.

>  It must use some standard PCIe mechanism
> because the endpoint doesn't know anything about s390 or the
> hypervisor.

Yes, though our machines only support a limited set of devices and
firmware also knows what kind of device is plugged where, so while the
devices don't know about s390 the firmware may do device specific
recovery steps.

> 
> I wonder if we should make some kind of direct platform-specific reset
> method, or maybe a pcibios_*()-style hook in the pcie_reset_flr() path
> instead of this somewhat convoluted pci_slot stuff.  But
> s390_pci_hpc.c is pretty simple and maybe it's used for things other
> than reset.
> 
> Bjorn

s390_pci_hpc.c is also used for "sharing" PCI functions between
different Linux instances. Basically a hotplug slot with the power
attribute reading 0 represents a PCI function which is in a pool of
standby PCI functions which are seen by multiple Linux instances at
once. Once one instance write 1 to power the PCI function is configured
(aka attached) to that Linux instance and becomes exclusively owned and
the hotplug slot becomes invisible/is hot unplugged for all other Linux
instances that could previously see it. On the other hand when one
Linux instance write 0 to a hotplug slot the PCI function is returned
to the pool and other Linux instances get a hotplug slot hot plugged.

Note that even without this patch resetting through s390_pci_hpc.c
worked on a per-function basis since the linking from the hotplug slot
to struct pci_slot was ok which is also why this stayed hidden for so
long.

Thanks,
Niklas