Re: [PATCH v2 0/4] vfio: Fix cdev second-open and harden selftests

[Alex Williamson <alex@shazbot.org>]

On Fri, 11 Sep 2026 11:04:23 -0600
Alex Williamson <alex.williamson@nvidia.com> wrote:

> v2:
>  - Re-try on -EBUSY reworked to harness issue and posted separately[1],
>    dropped here.
>  - vfio selftests subjects updated for consistency, Reviewed-bys
>    incorporated, Assisted-bys updated to current standards.
>  - Added the selftest suggested by David, but split it into two tests,
>    one that validates vf_token is not clobbered regardless of the 2nd
>    bind errno, and another that enforces the -EBUSY expected errno.
>    This allows us to observe that an old kernel fails both and avoids
>    conflating the uAPI expectation vs the underlying data clobber.
> 
> [1]https://lore.kernel.org/all/20260910230254.1198094-1-alex.williamson@nvidia.com/
> 
> v1:
> 
> In porting some testing infrastructure to a different system I found
> the igb selftest failing in mix_and_match and generating a cascade
> failure through the remaining tests.  The difference in the new system
> is the firmware error handling.  When the igb device gets wedged due
> to bad DMA in mix_and_match, we need to FLR the device.  The igb holds
> transaction pending asserted for the full duration of
> pci_wait_for_pending().  Meanwhile, firmware based error handling is
> triggering SMIs and stealing time, such that the 700ms total delay
> in pci_wait_for_pending() turns into several seconds.  With 10 cases
> in mix_and_match generating bad DMAs, the transaction pending delays
> alone push us close to the 30s per-test timeout.  The worst case I
> observed for the total test was ~45s.  Increasing the mix_and_match
> timeout to 90s provides plenty of headroom to get a passing test and
> avoid the cascade failure.
> 
> When the process is killed via timeout, the release occurs through a
> scheduled fput(), which is also delayed by the SMI storm.  Each
> subsequent test then sees a non-zero open_count (for the group open
> in legacy mode or on the iommufd bind in the cdev mode), resulting in
> the cascade failure.  Group mode already uses -EBUSY when the group
> fd open count is elevated, which allows selftests to interpret the
> failure as potentially transient and implement a bounded retry.  The
> cdev path instead returns -EINVAL for this case.  -EBUSY seems
> justified here and allows userspace to have compatible retry flows
> for group open and cdev bind operations.
> 
> A local sashiko review then found two existing issues.  First, in
> analyzing the exit flow from the iommufd bind, we can see that the
> vf_token and kvm pointers are clobbered by the second process before
> the open count test.  The open_count test in vfio_df_open() is only
> for the cdev path (!df->group) and is called under the dev_set lock,
> so we really only need to relocate the test to
> vfio_df_ioctl_bind_iommufd() prior to vf_token/kvm manipulation.
> 
> The second existing issue is that when executed via the kselftest
> runner, all tests have a 45s timeout, which is the cumulative time
> across each sub-test of the execution.  The pci_driver test already
> fails this with ioatdma, nv_falcon, and obviously with physical igb.
> Running in parallel across both ports of an igb on the system prone
> to SMI overhead, the worst case I saw was 450s.  Therefore, we not
> only need to extend the mix_and_match timeout to 90s to handle the
> extra transaction pending delay, we need to extend the default
> timeout to allow the full pci-driver test to complete when executed
> via the runner.  600s is picked here as a "sufficient" margin.
> 
> Please review and comment.  Thanks,
> 
> Alex
> 
> Alex Williamson (4):
>   vfio: Reject a second cdev open before mutating shared device state
>   vfio: selftests: Verify a failed second open preserves the vf_token
>   vfio: selftests: Extend mix_and_match timeout to 90s
>   vfio: selftests: Extend timeout for runner executions
> 
>  drivers/vfio/device_cdev.c                    | 12 ++++
>  drivers/vfio/vfio_main.c                      |  7 ---
>  tools/testing/selftests/vfio/.gitignore       |  1 +
>  tools/testing/selftests/vfio/settings         |  5 ++
>  .../selftests/vfio/vfio_pci_driver_test.c     |  2 +-
>  .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 57 +++++++++++++++++++
>  6 files changed, 76 insertions(+), 8 deletions(-)
>  create mode 100644 tools/testing/selftests/vfio/settings
> 
> 
> base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

Applied 1,3-4 to vfio next branch for v7.4.  A respin of the selftest
is forthcoming.  Thanks,

Alex