[BUG/RFC] PCI/EDR/DPC: vfio-pci endpoint left disabled after successful recovery

[BUG/RFC] PCI/EDR/DPC: vfio-pci endpoint left disabled after successful recovery

[Engel, Amit]

Hello,

I am investigating a firmware-first PCIe DPC/EDR recovery issue involving an NVMe endpoint bound to vfio-pci.
Linux reports that DPC recovery completed successfully and sends EDR _OST status 0x80 to firmware.
However, the downstream endpoint's PCI configuration is not restored: PCI_COMMAND remains 0x0000, Memory Space Enable is disabled, and the device cannot accept MMIO accesses.
A manual Secondary Bus Reset later causes hot-remove/re-enumeration and restores the endpoint.
Environment:
Platform: Dell PowerEdge, firmware-first DPC/EDR
Kernel: 6.4.0-150600.23.92-default, SUSE Linux Enterprise 15
Endpoint: 0000:07:00.0, KIOXIA NVMe
Endpoint driver: vfio-pci
DPC downstream port: 0000:04:04.0
EDR notification Root Port: 0000:00:01.1

Reproduction:
1. Start IO
2. Confirm the endpoint's initial PCI Command register:
setpci -s 0000:07:00.0 COMMAND
0406
3. Clear the Memory Space Enable bit:
setpci -s 0000:07:00.0 COMMAND=0000:0002
With IO active, a component attempts to access the disabled MMIO BAR. This results in an ERR_NONFATAL event and DPC containment.
Observed EDR/DPC recovery
The kernel reports:
pcieport 0000:00:01.1: EDR: EDR event received
pcieport 0000:00:01.1: EDR: Reported EDR dev: 0000:04:04.0
pcieport 0000:04:04.0: DPC: containment event, status:0x2003, ERR_NONFATAL received from 0000:07:00.0
pcieport 0000:04:04.0: pciehp: Slot(167): Link Down/Up ignored (recovered by DPC)
pcieport 0000:04:04.0: AER: device recovery successful
pcieport 0000:04:04.0: EDR: DPC port successfully recovered
pcieport 0000:00:01.1: EDR: Status for 0000:04:04.0: 0x80

DPC Trigger Status was cleared,
Therefore, the link-level DPC recovery appears to have succeeded.
The endpoint remains present in PCI enumeration and sysfs and is still bound to vfio-pci. However:
# setpci -s 0000:07:00.0 COMMAND
0000
lspci reports:
Control: I/O- Mem- BusMaster-
Region 0: Memory at ef000000 [disabled]
Kernel driver in use: vfio-pci
The endpoint cannot accept MMIO operations in this state.

Later, I manually issued an SBR on 0000:04:04.0. This caused pciehp removal and re-enumeration:
pcieport 0000:04:04.0: pciehp: Slot(167): Link Down
vfio-pci 0000:07:00.0: Relaying device request to user
vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
...
pcieport 0000:04:04.0: pciehp: Slot(167): Link Up
pci 0000:07:00.0: BAR 0: assigned [mem 0xef000000-0xef00ffff 64bit]
vfio-pci 0000:07:00.0: enabling device

After re-enumeration, PCI_COMMAND and MMIO access were restored.

This seems related to:
[PATCH v18 3/4] vfio/pci: Add a reset_done callback for vfio-pci driver
https://lore.kernel.org/all/20260603182415.2324-4-alifm@linux.ibm.com/
That patch restored VFIO's initial saved PCI state using:
pci_load_saved_state(pdev, vdev->pci_saved_state);
pci_restore_state(pdev);

This is conceptually similar to what appears to be missing in this reproduction.
However, reset_done() appears to be associated with PCI function-reset operations. The EDR/DPC path uses dpc_reset_link() through pcie_do_recovery(), so I do not believe the proposed reset_done callback would be invoked for this case.
I also did not see the generic VFIO reset_done patch in the later v19/v20 VFIO series. Was it intentionally dropped?

Questions
1. Is it expected that DPC link recovery succeeded but no endpoint driver callback confirmed that the PCI function was restored?
2. Should vfio-pci implement slot_reset(), or another post-DPC callback, that reloads vdev->pci_saved_state after the link becomes accessible?
3. Alternatively, should the PCI recovery core restore generic PCI configuration state for downstream devices before reporting recovery success?
4. If the endpoint's PCI state cannot be restored, should EDR return a failure status instead of sending _OST 0x80?

Before reporting successful recovery, I would expect the kernel/VFIO layer either to:
* restore a safe PCI handoff state, including BAR configuration and
Memory Space Enable; Bus Master Enable may remain disabled until userspace reinitializes DMA, or
* report that recovery failed.

Leaving the endpoint enumerated and bound to vfio-pci with PCI_COMMAND=0x0000, while reporting successful recovery, appears to be a false-success condition.

Please let me know whether this behavior is already known or whether the expected fix belongs in vfio-pci?

Regards,
Amit

Re: [BUG/RFC] PCI/EDR/DPC: vfio-pci endpoint left disabled after successful recovery

[Alex Williamson]

On Sun, 6 Sep 2026 18:34:02 +0000
"Engel, Amit" <Amit.Engel@Dell.com> wrote:

> Hello,
> 
> I am investigating a firmware-first PCIe DPC/EDR recovery issue involving an NVMe endpoint bound to vfio-pci.
> Linux reports that DPC recovery completed successfully and sends EDR _OST status 0x80 to firmware.
> However, the downstream endpoint's PCI configuration is not restored: PCI_COMMAND remains 0x0000, Memory Space Enable is disabled, and the device cannot accept MMIO accesses.
> A manual Secondary Bus Reset later causes hot-remove/re-enumeration and restores the endpoint.
> Environment:
> Platform: Dell PowerEdge, firmware-first DPC/EDR
> Kernel: 6.4.0-150600.23.92-default, SUSE Linux Enterprise 15
> Endpoint: 0000:07:00.0, KIOXIA NVMe
> Endpoint driver: vfio-pci
> DPC downstream port: 0000:04:04.0
> EDR notification Root Port: 0000:00:01.1
> 
> Reproduction:
> 1. Start IO
> 2. Confirm the endpoint's initial PCI Command register:
> setpci -s 0000:07:00.0 COMMAND
> 0406
> 3. Clear the Memory Space Enable bit:
> setpci -s 0000:07:00.0 COMMAND=0000:0002
> With IO active, a component attempts to access the disabled MMIO BAR. This results in an ERR_NONFATAL event and DPC containment.
> Observed EDR/DPC recovery
> The kernel reports:
> pcieport 0000:00:01.1: EDR: EDR event received
> pcieport 0000:00:01.1: EDR: Reported EDR dev: 0000:04:04.0
> pcieport 0000:04:04.0: DPC: containment event, status:0x2003, ERR_NONFATAL received from 0000:07:00.0
> pcieport 0000:04:04.0: pciehp: Slot(167): Link Down/Up ignored (recovered by DPC)
> pcieport 0000:04:04.0: AER: device recovery successful
> pcieport 0000:04:04.0: EDR: DPC port successfully recovered
> pcieport 0000:00:01.1: EDR: Status for 0000:04:04.0: 0x80
> 
> DPC Trigger Status was cleared,
> Therefore, the link-level DPC recovery appears to have succeeded.
> The endpoint remains present in PCI enumeration and sysfs and is still bound to vfio-pci. However:
> # setpci -s 0000:07:00.0 COMMAND
> 0000
> lspci reports:
> Control: I/O- Mem- BusMaster-
> Region 0: Memory at ef000000 [disabled]
> Kernel driver in use: vfio-pci
> The endpoint cannot accept MMIO operations in this state.
> 
> Later, I manually issued an SBR on 0000:04:04.0. This caused pciehp removal and re-enumeration:
> pcieport 0000:04:04.0: pciehp: Slot(167): Link Down
> vfio-pci 0000:07:00.0: Relaying device request to user
> vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
> ...
> pcieport 0000:04:04.0: pciehp: Slot(167): Link Up
> pci 0000:07:00.0: BAR 0: assigned [mem 0xef000000-0xef00ffff 64bit]
> vfio-pci 0000:07:00.0: enabling device
> 
> After re-enumeration, PCI_COMMAND and MMIO access were restored.
> 
> This seems related to:
> [PATCH v18 3/4] vfio/pci: Add a reset_done callback for vfio-pci driver
> https://lore.kernel.org/all/20260603182415.2324-4-alifm@linux.ibm.com/
> That patch restored VFIO's initial saved PCI state using:
> pci_load_saved_state(pdev, vdev->pci_saved_state);
> pci_restore_state(pdev);
> 
> This is conceptually similar to what appears to be missing in this reproduction.
> However, reset_done() appears to be associated with PCI function-reset operations. The EDR/DPC path uses dpc_reset_link() through pcie_do_recovery(), so I do not believe the proposed reset_done callback would be invoked for this case.
> I also did not see the generic VFIO reset_done patch in the later v19/v20 VFIO series. Was it intentionally dropped?
> 
> Questions
> 1. Is it expected that DPC link recovery succeeded but no endpoint driver callback confirmed that the PCI function was restored?
> 2. Should vfio-pci implement slot_reset(), or another post-DPC callback, that reloads vdev->pci_saved_state after the link becomes accessible?
> 3. Alternatively, should the PCI recovery core restore generic PCI configuration state for downstream devices before reporting recovery success?
> 4. If the endpoint's PCI state cannot be restored, should EDR return a failure status instead of sending _OST 0x80?
> 
> Before reporting successful recovery, I would expect the kernel/VFIO layer either to:
> * restore a safe PCI handoff state, including BAR configuration and
> Memory Space Enable; Bus Master Enable may remain disabled until userspace reinitializes DMA, or
> * report that recovery failed.
> 
> Leaving the endpoint enumerated and bound to vfio-pci with PCI_COMMAND=0x0000, while reporting successful recovery, appears to be a false-success condition.
> 
> Please let me know whether this behavior is already known or whether the expected fix belongs in vfio-pci?

Known behavior, vfio-pci has never supported recovery from an
uncorrected error.  Such an event triggers the error eventfd to signal
the error to userspace, but does not provide any means for userspace to
participate or observe the error recovery.  The typical VM behavior is
a VM_STOP on such error.

There are efforts[1] in motion to improve this, which you're welcome to
contribute to, but the userspace driver plays a part in any uncorrected
error event, this isn't simply a matter of restoring the command
register and continuing.  Thanks,

Alex

[1]https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/