Re: [RFC PATCH 1/5] PCI: Refuse function reset of an SR-IOV PF with enabled VFs

[Samiullah Khawaja <skhawaja@google.com>]

On Thu, Aug 27, 2026 at 05:35:17AM +0000, Tian, Kevin wrote:
>> From: Alex Williamson <alex.williamson@nvidia.com>
>> Sent: Wednesday, August 19, 2026 4:39 AM
>>
>> On Tue, 18 Aug 2026 11:03:04 -0300
>> Jason Gunthorpe <jgg@ziepe.ca> wrote:
>>
>> > On Tue, Aug 18, 2026 at 07:07:20AM +0000, Tian, Kevin wrote:
>> > > 3) resetting a PF due to ad-hoc FW error
>> > >
>> > > e.g. pdsc_check_pci_health() calls pci_reset_function() when FW
>> > > reports broken PCI. Sounds a similar condition to AER.
>> > >
>> > > but as it calls pci_reset_function() this series causes a regression
>> > > in case of active VFs. Is it desired?
>> >
>> > mlx5 does something similar. RAS flows require detecting the PF has
>> > blown up and resetting it to bring it back.
>> >
>> > This has to work while VFs exist and bound to kernel drivers or vfio.
>> >
>> > In the VFIO KVM case I would expect the VM to experiance a RAS error
>> > on its vPCI device and trigger a VF flr
>>
>> This would need to be a device level RAS flow as we're sorely lacking
>> on forwarding reset notification to VF drivers or providing any
>> mechanism for recovery from the vfio error eventfd.
>>
>> > But we have to sort all this out through the hypervisor to not crash
>> > the machine or otherwise :)
>>
>> In general guarding the high level pci_reset_function() family on
>> (pci_num_vfs() == 0) seems to have more pros than cons and the
>> restriction matches the scoped intent of the interface.
>>
>> The low level __pci_reset_function_locked() interface remains
>> untouched, and as the prefix implies, relies on the caller to do
>> necessary verification and preparation.  A "reset the PF regardless of
>> VFs" seems like an underscore prefix sort of operation.  We'll need to
>> export the pci_dev_save_and_disable() and pci_dev_restore() functions
>> to provide handlers the full suite of tools they need to replace the
>> existing function and port those we find (maybe also questioning the
>> author's intent on VF reset and recovery).
>>
>> Of course making the guard parameterized with wrappers to expand this
>> family of calls is also an option,
>> pci_reset_function_locked_with_vfs()...
>
>+1 for wrappers. Maybe call it pci_force_reset_function_locked() which
>is slightly forward-looking if in future other conditions may reject the
>reset request using the base function.
>
>>
>> The mlx5 driver doesn't actually seem to be affected by these changes,
>> doing its own link toggle or a PCI bus reset, not a
>> pci_reset_function() call.
>>
>> On the other improvements, triggering a pci_reset_function_locked() on
>> unbind after SR-IOV is disabled, when needs_reset is indicated, looks
>> pretty trivial.
>>
>> The blocking domain operation looks like it might be simplest to
>> implement in the IOMMU core.  We can set a flag for a default blocking
>> domain on the IOMMU group when we take_dma_ownership of the group.
>> Then
>> release_dma_ownership picks the blocking rather than default domain.
>
>IMHO we should change release_dma_ownership() to always picks the
>blocking. No new flag.
>
>this take/release interfaces are invented for drivers which manage dma
>on their own, so it sounds reasonable to not assume the operated device
>is in a good shape after the ownership is released.
>
>Then only a clean reset can unwind it.
>
>>
>> This is then unwound in use_default_domain, called via dma_configure,
>> attaching the device to the default domain in probe of the next driver.
>> Therefore until probe by another driver, a device used by vfio would
>> remain in a blocking domain even while unused and unbound.
>>
>
>so unwinding it blindly at dma_configure time is also problematic, as the
>device may still have a bad state to hurt the kernel.
>
>Instead use_default_domain should fail in such case (in blocked with
>owner_cnt being '0') then a reset is expected to unwind it (Nicolin's
>work will carry the resetting status to pci_dev_reset_iommu_done())

Thinking about this more, it might be tricky in multiple ways,

- This will only work if drivers do reset after dma ownership is
   released. Looking at the code, vfio currently resets the device first
   and then dma ownership is released.
- For some devices with no reset mechanism, vfio-pci uses bus or slot
   resets. This will need extra care to propagate the
   reset_iommu_prepare/reset_iommu_done to subordinate devices to bring
   the device out of blocking domain.
- AFAIU looking at vfio-pci code, some devices do not have proper reset
   mechanism, not even slot/bus, so those devices would be stuck in the
   blocking domain state. Probably need a mechanism to let IOMMU know
   that there is no reset mechanism?

Thanks,
Sami