mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement
@ 2026-11-19  0:01 David Matlack
  2026-11-19  0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

This series can be found on GitHub:

  https://github.com/dmatlack/linux/tree/vfio/set_power_state/v2

This series fixes a race between closing a vfio-pci device and enabling
SR-IOV on it, both of which can transition the device from D3hot to D0
without being serialized. On devices with NoSoftRst-, this can restore
the PF's BARs after VFs have been enabled (triggering a WARN in
pci_iov_update_resource()), or double-free vdev->pm_save.

This series also adds a selftest that exercises VFIO's handling of
power state transitions requested by userspace through the PM
capability, including a test case that reproduces the race.

This series applies on top of vfio/next (vfio-v7.4-rc1).

Series Overview
---------------

This series implements the following:

  1. Hold memory_lock for all calls to vfio_pci_set_power_state() and
     assert that it is held, which fixes the race (patch 1).

  2. Annotate the intentional lockless accesses to vdev->sriov_active
     with READ_ONCE() and WRITE_ONCE() (patch 2).

  3. Skip vfio_pci_sriov_uapi_test instead of failing when SR-IOV
     cannot be configured on the PF, e.g. because vfio-pci's
     enable_sriov module parameter is not set (patches 3-4).

  4. Add library helpers to find a PCI capability, read
     /sys/kernel/warn_count, and print kernel warnings from /dev/kmsg
     (patches 5-7).

  5. Add vfio_pci_power_state_test, which covers closing a device in
     D3hot and reopening it, enabling SR-IOV while the PF is in D3hot,
     and racing a close in D3hot against enabling SR-IOV. The test fails
     if /sys/kernel/warn_count changes during a test case, and prints
     the warnings logged to /dev/kmsg to help debug the failure
     (patch 8).

Testing
-------

All patches were build-tested individually. The series was tested on
x86_64 in a QEMU VM with an emulated Intel 82576 NIC (igb), which has
NoSoftRst- and supports SR-IOV, bound to vfio-pci:

  - Without patch 1 (KASAN kernel), close_in_d3hot_races_sriov_enable
    fails, hitting the pci_iov_update_resource() WARN on 9 or 10 of the
    test's 10 iterations.

  - With the full series (KASAN and lockdep kernels), all VFIO
    selftests pass with no warnings, KASAN reports, or lockdep splats.

To run the new test (the SR-IOV test cases are skipped unless
vfio_pci.enable_sriov=1):

  # tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0
  # tools/testing/selftests/vfio/scripts/run.sh \
      tools/testing/selftests/vfio/vfio_pci_power_state_test

Changelog
---------

v2:
 - Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that
   it does not verify the D0 transition during close, since that is not
   reliably observable from userspace (Alex)
 - Print the kernel warnings logged to /dev/kmsg when the warn_count
   check fails, so the failure can be debugged from the test log (Alex)
 - Drop the claim that memory_lock protects the sriov_active check from
   the comment in vfio_pci_set_power_state(), since it is cleared
   without memory_lock
 - Skip the SR-IOV tests, including the existing
   vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF
   (e.g. vfio_pci.enable_sriov=N) instead of failing
 - Use kernel-doc for the new library helpers
 - Clean up commit messages, comments, and the cover letter

v1: https://lore.kernel.org/kvm/20261029203324.1637772-1-dmatlack@google.com/

David Matlack (8):
  vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()
  vfio/pci: Annotate lockless accesses to vdev->sriov_active
  vfio: selftests: Add __sysfs_sriov_numvfs_set()
  vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be
    enabled
  vfio: selftests: Add vfio_pci_find_capability()
  vfio: selftests: Add sysfs_kernel_warn_count_get()
  vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
  vfio: selftests: Add test for device power state transitions

 drivers/vfio/pci/vfio_pci_core.c              |  16 +-
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../selftests/vfio/lib/include/libvfio/kmsg.h |   8 +
 .../vfio/lib/include/libvfio/sysfs.h          |   2 +
 .../lib/include/libvfio/vfio_pci_device.h     |   2 +
 tools/testing/selftests/vfio/lib/kmsg.c       | 134 +++++++++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 tools/testing/selftests/vfio/lib/sysfs.c      |  66 ++++-
 .../selftests/vfio/lib/vfio_pci_device.c      |  36 +++
 .../vfio/vfio_pci_power_state_test.c          | 256 ++++++++++++++++++
 .../selftests/vfio/vfio_pci_sriov_uapi_test.c |   8 +-
 12 files changed, 512 insertions(+), 19 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
 create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c


base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active David Matlack
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Hold vdev->memory_lock for write when calling vfio_pci_set_power_state()
from vfio_pci_core_disable() and vfio_pci_core_register_device(), and
assert that the lock is held for all power state transitions, not just
transitions out of D0.

vfio_pci_core_disable() calls vfio_pci_set_power_state(vdev, PCI_D0)
without holding memory_lock, which allows it to race with
vfio_pci_core_sriov_configure() when userspace writes to sriov_numvfs
while the device is being closed. If the user left the device in D3hot
via a config space write, both threads can transition the device from
D3hot to D0 concurrently. Since a config space write does not change
the runtime PM status, the device stays RPM_ACTIVE, pm_runtime_resume()
in the close path is a no-op, and vfio_pci_set_power_state() performs
the D3hot->D0 transition.

For devices with vdev->needs_pm_restore set (NoSoftRst-), this can
result in:

 - vfio_pci_core_disable() restoring the PF's BARs after
   vfio_pci_core_sriov_configure() has enabled VFs, since the D3hot->D0
   transition sleeps before restoring device state. This reproduces
   reliably with vfio_pci_power_state_test and triggers the following
   warning:

     vfio-pci 0000:01:00.0: can't update enabled VF BAR0 [mem 0xe0200000000-0xe020001ffff 64bit pref]
     WARNING: drivers/pci/iov.c:1062 at pci_iov_update_resource+0x170/0x3b0
     Call Trace:
      pci_update_resource+0x1a3/0x5b0
      __pci_set_power_state+0x597/0xb90
      vfio_pci_core_disable+0x13b/0x1180
      vfio_pci_core_close_device+0x144/0x260
      vfio_df_close+0xf2/0x210
      vfio_df_unbind_iommufd+0x89/0x140
      vfio_device_fops_release+0x85/0xf0
      __fput+0x2d6/0x720
      fput_close_sync+0xe8/0x180
      __x64_sys_close+0x7f/0xc0
      do_syscall_64+0xbc/0x320
      entry_SYSCALL_64_after_hwframe+0x76/0x7e

 - Both threads observing pdev->current_state >= PCI_D3hot and calling
   pci_load_and_free_saved_state(pdev, &vdev->pm_save) concurrently,
   resulting in a double-free of vdev->pm_save.

vfio_pci_core_register_device() cannot race with other callers since
the device is not yet registered and device_lock is held across probe,
but take the lock there as well so that the lockdep assertion can be
made unconditional and catch any future unlocked callers.

Taking memory_lock in these paths does not introduce any new lock
dependencies. vfio_pci_core_disable() already acquires memory_lock
under dev_set->lock via vfio_pci_runtime_pm_exit(), and
vfio_pci_core_register_device(), like vfio_pci_core_sriov_configure(),
runs under device_lock, which is already taken before memory_lock.

Fixes: f4162eb1e2fc ("vfio/pci: Change the PF power state to D0 before enabling VFs")
Cc: stable@vger.kernel.org
Signed-off-by: David Matlack <dmatlack@google.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..4289ec9bc32f 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -325,12 +325,12 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	bool needs_restore = false, needs_save = false;
 	int ret;
 
+	/* Serialize power state changes and accesses to vdev->pm_save. */
+	lockdep_assert_held_write(&vdev->memory_lock);
+
 	/* Prevent changing power state for PFs with VFs enabled */
-	if (state > PCI_D0) {
-		lockdep_assert_held_write(&vdev->memory_lock);
-		if (vdev->sriov_active)
-			return -EBUSY;
-	}
+	if (state > PCI_D0 && vdev->sriov_active)
+		return -EBUSY;
 
 	if (vdev->needs_pm_restore) {
 		if (pdev->current_state < PCI_D3hot && state >= PCI_D3hot) {
@@ -699,7 +699,9 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	 * reset without restoring the original state (saved locally in
 	 * 'vdev->pm_save').
 	 */
+	down_write(&vdev->memory_lock);
 	vfio_pci_set_power_state(vdev, PCI_D0);
+	up_write(&vdev->memory_lock);
 
 	/* Stop the device from further DMA */
 	pci_clear_master(pdev);
@@ -2273,7 +2275,9 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
 	 * be able to get to D3.  Therefore first do a D0 transition
 	 * before enabling runtime PM.
 	 */
+	down_write(&vdev->memory_lock);
 	vfio_pci_set_power_state(vdev, PCI_D0);
+	up_write(&vdev->memory_lock);
 
 	pm_runtime_allow(dev);
 	if (!vdev->disable_idle_d3)
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
  2026-11-19  0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set() David Matlack
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Use READ_ONCE() and WRITE_ONCE() to annotate the read of
vdev->sriov_active in vfio_pci_set_power_state() and the lockless write
that clears it in vfio_pci_core_sriov_configure().

vfio_pci_core_sriov_configure() intentionally clears sriov_active
without holding memory_lock, so the write can race with the read in
vfio_pci_set_power_state() (which runs under memory_lock). The race is
benign: at worst it causes a spurious -EBUSY while the device stays in
D0. Mark the accesses anyway to document the intentional data race and
avoid KCSAN reports.

The write that sets sriov_active is done under memory_lock and is
serialized against the lockless write by device_lock, so it does not
need to be annotated.

Fixes: 40ef3edf151e ("vfio/pci: Use a private flag to prevent power state change with VFs")
Signed-off-by: David Matlack <dmatlack@google.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 4289ec9bc32f..511dcb49d2f6 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -329,7 +329,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	lockdep_assert_held_write(&vdev->memory_lock);
 
 	/* Prevent changing power state for PFs with VFs enabled */
-	if (state > PCI_D0 && vdev->sriov_active)
+	if (state > PCI_D0 && READ_ONCE(vdev->sriov_active))
 		return -EBUSY;
 
 	if (vdev->needs_pm_restore) {
@@ -2393,7 +2393,7 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 	 * state transition would at most result in an -EBUSY, leaving the
 	 * device in PCI_D0.
 	 */
-	vdev->sriov_active = false;
+	WRITE_ONCE(vdev->sriov_active, false);
 
 	mutex_lock(&vfio_pci_sriov_pfs_mutex);
 	list_del_init(&vdev->sriov_pfs_item);
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set()
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
  2026-11-19  0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
  2026-11-19  0:01 ` [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled David Matlack
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Add __sysfs_sriov_numvfs_set(), which returns the error from writing to
a PF's sriov_numvfs file instead of asserting that the write succeeds,
and implement sysfs_sriov_numvfs_set() on top of it.

This will be used in subsequent commits to skip SR-IOV tests when the
kernel returns -ENOENT, which means SR-IOV cannot be configured on the
PF, e.g. because vfio-pci's enable_sriov module parameter is not set.

Signed-off-by: David Matlack <dmatlack@google.com>
---
 .../vfio/lib/include/libvfio/sysfs.h          |  1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 38 +++++++++++++++----
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
index c9ab1ea8f5a9..ede114181cf8 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
@@ -4,6 +4,7 @@
 
 int sysfs_sriov_totalvfs_get(const char *bdf);
 int sysfs_sriov_numvfs_get(const char *bdf);
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
 int sysfs_iommu_group_get(const char *bdf);
diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c
index d88e348a1e0d..308da0526543 100644
--- a/tools/testing/selftests/vfio/lib/sysfs.c
+++ b/tools/testing/selftests/vfio/lib/sysfs.c
@@ -56,17 +56,24 @@ static int sysfs_val_get_int(const char *component, const char *name,
 	return ret;
 }
 
-static void sysfs_val_set(const char *component, const char *name,
-			  const char *file, const char *val)
+static int sysfs_val_set(const char *component, const char *name,
+			 const char *file, const char *val)
 {
 	char path[PATH_MAX];
+	ssize_t ret;
 	int fd;
 
 	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
 	VFIO_ASSERT_GT(fd = open(path, O_WRONLY), 0);
 
-	VFIO_ASSERT_EQ(write(fd, val, strlen(val)), strlen(val));
+	ret = write(fd, val, strlen(val));
+	if (ret < 0)
+		ret = -errno;
+	else
+		VFIO_ASSERT_EQ(ret, strlen(val));
+
 	VFIO_ASSERT_EQ(close(fd), 0);
+	return ret < 0 ? ret : 0;
 }
 
 static int sysfs_device_val_get(const char *bdf, const char *file)
@@ -74,17 +81,17 @@ static int sysfs_device_val_get(const char *bdf, const char *file)
 	return sysfs_val_get_int("devices", bdf, file);
 }
 
-static void sysfs_device_val_set(const char *bdf, const char *file, const char *val)
+static int sysfs_device_val_set(const char *bdf, const char *file, const char *val)
 {
-	sysfs_val_set("devices", bdf, file, val);
+	return sysfs_val_set("devices", bdf, file, val);
 }
 
-static void sysfs_device_val_set_int(const char *bdf, const char *file, int val)
+static int sysfs_device_val_set_int(const char *bdf, const char *file, int val)
 {
 	char val_str[32];
 
 	snprintf_assert(val_str, sizeof(val_str), "%d", val);
-	sysfs_device_val_set(bdf, file, val_str);
+	return sysfs_device_val_set(bdf, file, val_str);
 }
 
 int sysfs_sriov_totalvfs_get(const char *bdf)
@@ -97,9 +104,24 @@ int sysfs_sriov_numvfs_get(const char *bdf)
 	return sysfs_device_val_get(bdf, "sriov_numvfs");
 }
 
+/**
+ * __sysfs_sriov_numvfs_set() - Set the number of VFs enabled on a PF
+ * @bdf: BDF of the PF.
+ * @numvfs: Number of VFs to enable, or 0 to disable SR-IOV.
+ *
+ * Return: 0 on success, or a negative errno on failure. The kernel returns
+ * -ENOENT if SR-IOV cannot be configured on the PF, e.g. because its driver
+ * does not support configuring SR-IOV or has it disabled (vfio-pci unless its
+ * enable_sriov module parameter is set).
+ */
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
+{
+	return sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+}
+
 void sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
 {
-	sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+	VFIO_ASSERT_EQ(__sysfs_sriov_numvfs_set(bdf, numvfs), 0);
 }
 
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i)
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
                   ` (2 preceding siblings ...)
  2026-11-19  0:01 ` [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set() David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability() David Matlack
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Skip vfio_pci_sriov_uapi_test if writing to the PF's sriov_numvfs file
fails with -ENOENT, which means SR-IOV cannot be configured on the PF.

vfio-pci returns -ENOENT unless its enable_sriov module parameter is
set, which it is not by default, so the test currently fails during
setup instead of skipping.

Fixes: a6e4e726600d ("vfio: selftests: Add tests to validate SR-IOV UAPI")
Signed-off-by: David Matlack <dmatlack@google.com>
---
 tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
index f3a018b8699d..f27418fa5960 100644
--- a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
+++ b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
@@ -210,6 +210,7 @@ static void vf_setup(void)
 {
 	char *vf_driver;
 	int nr_vfs;
+	int ret;
 
 	nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
 	if (nr_vfs <= 0)
@@ -220,7 +221,12 @@ static void vf_setup(void)
 		ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
 
 	/* Create only one VF for testing */
-	sysfs_sriov_numvfs_set(pf_bdf, 1);
+	ret = __sysfs_sriov_numvfs_set(pf_bdf, 1);
+	if (ret == -ENOENT)
+		ksft_exit_skip("Cannot enable SR-IOV on %s (vfio-pci requires enable_sriov=1)\n",
+			       pf_bdf);
+
+	VFIO_ASSERT_EQ(ret, 0);
 
 	/*
 	 * Setup an exit handler to destroy the VF in case of failures
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability()
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
                   ` (3 preceding siblings ...)
  2026-11-19  0:01 ` [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get() David Matlack
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Add vfio_pci_find_capability(), which walks a device's standard
capability list like the kernel's pci_find_capability().

This will be used in a subsequent commit to locate the PM capability.

Signed-off-by: David Matlack <dmatlack@google.com>
---
 .../lib/include/libvfio/vfio_pci_device.h     |  2 ++
 .../selftests/vfio/lib/vfio_pci_device.c      | 36 +++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
index 007adc322c1e..0fc03bb79448 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
@@ -79,6 +79,8 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
 #define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
 #define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
 
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id);
+
 static inline void vfio_pci_cmd_set(struct vfio_pci_device *device, u16 bits)
 {
 	u16 cmd = vfio_pci_config_readw(device, PCI_COMMAND);
diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
index 4063a0e2b3df..ee3f76e2b874 100644
--- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c
+++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
@@ -260,6 +260,42 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
 		       write ? "write to" : "read from", config);
 }
 
+/* Bound the capability list walk in case the list contains a loop. */
+#define PCI_FIND_CAP_TTL 48
+
+/**
+ * vfio_pci_find_capability() - Find a capability in a device's config space
+ * @device: Device to search.
+ * @cap_id: Capability ID to find (PCI_CAP_ID_*).
+ *
+ * Walk the device's standard capability list, like the kernel's
+ * pci_find_capability().
+ *
+ * Return: The config space offset of the first capability with ID @cap_id, or
+ * 0 if the device does not have the capability.
+ */
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id)
+{
+	int ttl = PCI_FIND_CAP_TTL;
+	u8 pos;
+
+	if (!(vfio_pci_config_readw(device, PCI_STATUS) & PCI_STATUS_CAP_LIST))
+		return 0;
+
+	pos = vfio_pci_config_readb(device, PCI_CAPABILITY_LIST);
+
+	while (pos >= PCI_STD_HEADER_SIZEOF && ttl--) {
+		pos &= ~3;
+
+		if (vfio_pci_config_readb(device, pos + PCI_CAP_LIST_ID) == cap_id)
+			return pos;
+
+		pos = vfio_pci_config_readb(device, pos + PCI_CAP_LIST_NEXT);
+	}
+
+	return 0;
+}
+
 int __vfio_pci_device_reset(struct vfio_pci_device *device)
 {
 	if (ioctl(device->fd, VFIO_DEVICE_RESET, NULL))
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get()
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
                   ` (4 preceding siblings ...)
  2026-11-19  0:01 ` [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability() David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg David Matlack
  2026-11-19  0:01 ` [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions David Matlack
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Add a helper to read /sys/kernel/warn_count, which counts the number of
warnings the kernel has encountered since boot. Return a negative value
if the file does not exist, e.g. on kernels that predate it.

Factor the code to read an integer from a sysfs file out of
sysfs_val_get_int() so that it can be shared by both helpers.

This will be used in a subsequent commit to detect kernel warnings
triggered by tests.

Signed-off-by: David Matlack <dmatlack@google.com>
---
 .../vfio/lib/include/libvfio/sysfs.h          |  1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 28 ++++++++++++++++---
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
index ede114181cf8..5ef515eda686 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
@@ -9,5 +9,6 @@ void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
 int sysfs_iommu_group_get(const char *bdf);
 char *sysfs_driver_get(const char *bdf);
+int sysfs_kernel_warn_count_get(void);
 
 #endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H */
diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c
index 308da0526543..3845d81825e3 100644
--- a/tools/testing/selftests/vfio/lib/sysfs.c
+++ b/tools/testing/selftests/vfio/lib/sysfs.c
@@ -33,15 +33,12 @@ static void readlink_base(const char *path, const char *data_fmt, void *out_data
 	VFIO_ASSERT_EQ(ret, 1);
 }
 
-static int sysfs_val_get_int(const char *component, const char *name,
-			     const char *file)
+static int sysfs_path_get_int(const char *path)
 {
-	char path[PATH_MAX];
 	char buf[32];
 	int ret;
 	int fd;
 
-	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
 	fd = open(path, O_RDONLY);
 	if (fd < 0)
 		return fd;
@@ -56,6 +53,15 @@ static int sysfs_val_get_int(const char *component, const char *name,
 	return ret;
 }
 
+static int sysfs_val_get_int(const char *component, const char *name,
+			     const char *file)
+{
+	char path[PATH_MAX];
+
+	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
+	return sysfs_path_get_int(path);
+}
+
 static int sysfs_val_set(const char *component, const char *name,
 			 const char *file, const char *val)
 {
@@ -170,3 +176,17 @@ char *sysfs_driver_get(const char *bdf)
 
 	return out_driver;
 }
+
+/**
+ * sysfs_kernel_warn_count_get() - Get the number of kernel warnings since boot
+ *
+ * Read /sys/kernel/warn_count, which counts the warnings (e.g. WARN(), KASAN,
+ * and UBSAN reports) the kernel has encountered since boot.
+ *
+ * Return: The number of warnings, or a negative value if the kernel does not
+ * support /sys/kernel/warn_count.
+ */
+int sysfs_kernel_warn_count_get(void)
+{
+	return sysfs_path_get_int("/sys/kernel/warn_count");
+}
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
                   ` (5 preceding siblings ...)
  2026-11-19  0:01 ` [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get() David Matlack
@ 2026-11-19  0:01 ` David Matlack
  2026-11-19  0:01 ` [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions David Matlack
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Add kmsg_open() and kmsg_print_warnings() to the library. kmsg_open()
opens /dev/kmsg positioned after the last message in the kernel log, and
kmsg_print_warnings() prints any kernel messages at KERN_WARNING or
higher priority that have been logged since.

Bound how much kmsg_print_warnings() reads and prints, since other
processes (e.g. tests running in parallel) may log a large number of
messages, and note in the output when messages were lost because the
kernel log wrapped. If /dev/kmsg cannot be opened, kmsg_open() prints a
message saying that kernel warnings will not be printed, so that callers
do not each need to.

This will be used in a subsequent commit to print kernel warnings to
the test log when a test triggers one, so the failure can be debugged
from the test log.

Signed-off-by: David Matlack <dmatlack@google.com>
---
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../selftests/vfio/lib/include/libvfio/kmsg.h |   8 ++
 tools/testing/selftests/vfio/lib/kmsg.c       | 134 ++++++++++++++++++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 4 files changed, 144 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
 create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c

diff --git a/tools/testing/selftests/vfio/lib/include/libvfio.h b/tools/testing/selftests/vfio/lib/include/libvfio.h
index 07862b470777..dca9a9af961e 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio.h
@@ -5,6 +5,7 @@
 #include <libvfio/assert.h>
 #include <libvfio/iommu.h>
 #include <libvfio/iova_allocator.h>
+#include <libvfio/kmsg.h>
 #include <libvfio/sysfs.h>
 #include <libvfio/vfio_pci_device.h>
 #include <libvfio/vfio_pci_driver.h>
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h b/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
new file mode 100644
index 000000000000..9b9e89129b8f
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+
+int kmsg_open(void);
+void kmsg_print_warnings(int fd);
+
+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H */
diff --git a/tools/testing/selftests/vfio/lib/kmsg.c b/tools/testing/selftests/vfio/lib/kmsg.c
new file mode 100644
index 000000000000..a25770d2247a
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/kmsg.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <syslog.h>
+#include <unistd.h>
+
+#include <libvfio.h>
+
+/* Large enough to hold any single /dev/kmsg record. */
+#define KMSG_RECORD_MAX 8192
+
+/* Limit how much is printed if the kernel logs a flood of warnings. */
+#define KMSG_PRINT_MAX 512
+
+/*
+ * Limit how many records are read, so that this terminates even if other
+ * processes are logging messages faster than they can be read.
+ */
+#define KMSG_READ_MAX 65536
+
+/**
+ * kmsg_open() - Open /dev/kmsg to read kernel messages logged from now on
+ *
+ * Open /dev/kmsg positioned after the last message currently in the kernel
+ * log, so that kmsg_print_warnings() only prints messages logged after this
+ * call. If /dev/kmsg cannot be opened, e.g. because the caller is not allowed
+ * to read the kernel log, print a message saying that kernel warnings will not
+ * be printed.
+ *
+ * Return: A file descriptor on success, or a negative errno on failure.
+ */
+int kmsg_open(void)
+{
+	int fd, ret;
+
+	fd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC);
+	if (fd < 0) {
+		ret = -errno;
+		goto out_err;
+	}
+
+	if (lseek(fd, 0, SEEK_END) < 0) {
+		ret = -errno;
+		close(fd);
+		goto out_err;
+	}
+
+	return fd;
+
+out_err:
+	fprintf(stderr,
+		"# kmsg: Failed to open /dev/kmsg (%s), kernel warnings will not be printed\n",
+		strerror(-ret));
+	return ret;
+}
+
+/**
+ * kmsg_print_warnings() - Print kernel warnings logged since kmsg_open()
+ * @fd: File descriptor returned by kmsg_open().
+ *
+ * Print all kernel messages at KERN_WARNING or higher priority that have been
+ * logged since kmsg_open() (or since the last call to this function), e.g. to
+ * provide context for a test failure caused by a kernel warning.
+ *
+ * Messages may be missing if the kernel log wrapped since they were logged,
+ * and output is limited if the kernel logged a large number of messages. A
+ * note is printed in either case.
+ */
+void kmsg_print_warnings(int fd)
+{
+	char buf[KMSG_RECORD_MAX];
+	bool truncated = true;
+	bool lost = false;
+	int printed = 0;
+	int omitted = 0;
+	int i;
+
+	for (i = 0; i < KMSG_READ_MAX; i++) {
+		char *text, *end;
+		ssize_t len;
+		int prio;
+
+		len = read(fd, buf, sizeof(buf) - 1);
+		if (len < 0 && errno == EPIPE) {
+			/* Messages were overwritten before being read. */
+			lost = true;
+			continue;
+		}
+
+		/* Stop at EAGAIN (no more messages) or on any other error. */
+		if (len <= 0) {
+			truncated = false;
+			break;
+		}
+
+		buf[len] = '\0';
+
+		/*
+		 * Each record is "<prio>,<seq>,<ts>,<flags>;<message>\n",
+		 * optionally followed by " KEY=VALUE\n" lines. See
+		 * Documentation/ABI/testing/dev-kmsg.
+		 */
+		text = strchr(buf, ';');
+		if (!text || sscanf(buf, "%d,", &prio) != 1)
+			continue;
+
+		text++;
+		end = strchr(text, '\n');
+		if (end)
+			*end = '\0';
+
+		if (LOG_FAC(prio) != LOG_FAC(LOG_KERN) || LOG_PRI(prio) > LOG_WARNING)
+			continue;
+
+		if (printed < KMSG_PRINT_MAX) {
+			fprintf(stderr, "# kmsg: %s\n", text);
+			printed++;
+		} else {
+			omitted++;
+		}
+	}
+
+	if (omitted)
+		fprintf(stderr, "# kmsg: (%d more messages omitted)\n", omitted);
+
+	if (lost)
+		fprintf(stderr, "# kmsg: (some messages were lost, see the kernel log)\n");
+
+	if (truncated)
+		fprintf(stderr, "# kmsg: (stopped reading, see the kernel log)\n");
+}
diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk
index eff95828719e..f8af0c14c3d0 100644
--- a/tools/testing/selftests/vfio/lib/libvfio.mk
+++ b/tools/testing/selftests/vfio/lib/libvfio.mk
@@ -5,6 +5,7 @@ LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
 
 LIBVFIO_C := iommu.c
 LIBVFIO_C += iova_allocator.c
+LIBVFIO_C += kmsg.c
 LIBVFIO_C += libvfio.c
 LIBVFIO_C += sysfs.c
 LIBVFIO_C += vfio_pci_device.c
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions
  2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
                   ` (6 preceding siblings ...)
  2026-11-19  0:01 ` [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg David Matlack
@ 2026-11-19  0:01 ` David Matlack
  7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19  0:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-kselftest, stable
  Cc: Abhishek Sahu, Alex Williamson, David Matlack,
	Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma

Add vfio_pci_power_state_test to exercise VFIO's handling of device
power state changes initiated by userspace through the PM capability in
config space.

The test covers:

 - Closing a device in D3hot and then reopening it, which must leave
   the device in D0.
 - Enabling SR-IOV while the PF is in D3hot, which must return the PF to
   D0 and prevent it from leaving D0 while VFs are enabled.
 - Racing a close of the device while it is in D3hot against enabling
   SR-IOV, both of which transition the device back to D0.

The SR-IOV tests are skipped if the device does not support SR-IOV,
already has VFs enabled, or SR-IOV cannot be configured on it (e.g.
vfio-pci with vfio_pci.enable_sriov=N). If an SR-IOV test case fails,
any VFs it left enabled are disabled so they don't affect later test
cases.

Fail the test if /sys/kernel/warn_count changes between fixture setup
and teardown, since bugs such as the close vs. SR-IOV enable race show
up only as kernel warnings. When that happens, print the warnings
logged to /dev/kmsg during the test so that the failure can be debugged
from the test log.

Signed-off-by: David Matlack <dmatlack@google.com>
---
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../vfio/vfio_pci_power_state_test.c          | 256 ++++++++++++++++++
 2 files changed, 257 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c

diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile
index 2c32c48db509..e2b152528560 100644
--- a/tools/testing/selftests/vfio/Makefile
+++ b/tools/testing/selftests/vfio/Makefile
@@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test
 TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
+TEST_GEN_PROGS += vfio_pci_power_state_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
 
 TEST_FILES += scripts/cleanup.sh
diff --git a/tools/testing/selftests/vfio/vfio_pci_power_state_test.c b/tools/testing/selftests/vfio/vfio_pci_power_state_test.c
new file mode 100644
index 000000000000..5b185c174036
--- /dev/null
+++ b/tools/testing/selftests/vfio/vfio_pci_power_state_test.c
@@ -0,0 +1,256 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <pthread.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <linux/pci_regs.h>
+
+#include <libvfio.h>
+
+#include "kselftest_harness.h"
+
+/* PCI_PM_CTRL_STATE_MASK values */
+#define D0	0
+#define D3HOT	3
+
+static const char *device_bdf;
+
+static u16 power_state_get(struct vfio_pci_device *device, u8 pm_cap)
+{
+	return vfio_pci_config_readw(device, pm_cap + PCI_PM_CTRL) &
+	       PCI_PM_CTRL_STATE_MASK;
+}
+
+static void power_state_set(struct vfio_pci_device *device, u8 pm_cap,
+			    u16 state)
+{
+	u16 pmcsr = vfio_pci_config_readw(device, pm_cap + PCI_PM_CTRL);
+
+	pmcsr &= ~PCI_PM_CTRL_STATE_MASK;
+	pmcsr |= state;
+
+	vfio_pci_config_writew(device, pm_cap + PCI_PM_CTRL, pmcsr);
+}
+
+/*
+ * Set by test cases that enable VFs, so that disable_vfs() can disable any VFs
+ * left enabled if the test case fails. Only set in the child process that runs
+ * the test case.
+ */
+static bool test_enabled_vfs;
+
+/*
+ * Disable VFs left enabled by a failed test case, so they don't affect later
+ * test cases. Called from fixture teardown, which runs when an ASSERT fails,
+ * and from an atexit() handler, which runs when a library helper exits the
+ * test process early (e.g. on a VFIO_ASSERT() failure).
+ */
+static void disable_vfs(void)
+{
+	if (test_enabled_vfs && sysfs_sriov_numvfs_get(device_bdf) > 0)
+		__sysfs_sriov_numvfs_set(device_bdf, 0);
+}
+
+/*
+ * Check whether the test can enable VFs on the device, briefly enabling one VF
+ * to find out. Returns the reason to skip the test, or NULL if it can run.
+ */
+static const char *sriov_skip_reason(void)
+{
+	int ret;
+
+	if (sysfs_sriov_totalvfs_get(device_bdf) <= 0)
+		return "Device does not support SR-IOV";
+
+	if (sysfs_sriov_numvfs_get(device_bdf) != 0)
+		return "SR-IOV is already enabled on the device";
+
+	/* From here on, any VFs on the device were enabled by this test. */
+	test_enabled_vfs = true;
+
+	/* -ENOENT means SR-IOV cannot be configured on the device. */
+	ret = __sysfs_sriov_numvfs_set(device_bdf, 1);
+	if (ret == -ENOENT)
+		return "Cannot enable SR-IOV on the device (vfio-pci requires enable_sriov=1)";
+
+	VFIO_ASSERT_EQ(ret, 0);
+	sysfs_sriov_numvfs_set(device_bdf, 0);
+	return NULL;
+}
+
+FIXTURE(vfio_pci_power_state_test) {
+	struct iommu *iommu;
+	struct vfio_pci_device *device;
+	u8 pm_cap;
+	int warn_count;
+	int kmsg_fd;
+};
+
+FIXTURE_SETUP(vfio_pci_power_state_test)
+{
+	/*
+	 * Snapshot the kernel's warning count so that teardown can check that
+	 * the test did not trigger any kernel warnings, which are otherwise
+	 * invisible to the test. Open /dev/kmsg first so that any warning
+	 * counted after the snapshot can be printed if the check fails.
+	 */
+	self->kmsg_fd = kmsg_open();
+	self->warn_count = sysfs_kernel_warn_count_get();
+
+	self->iommu = iommu_init(default_iommu_mode);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
+
+	self->pm_cap = vfio_pci_find_capability(self->device, PCI_CAP_ID_PM);
+	if (!self->pm_cap)
+		SKIP(return, "Device does not have a PM capability\n");
+
+	ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+FIXTURE_TEARDOWN(vfio_pci_power_state_test)
+{
+	disable_vfs();
+
+	if (self->device)
+		vfio_pci_device_cleanup(self->device);
+
+	iommu_cleanup(self->iommu);
+
+	if (self->warn_count >= 0) {
+		int warn_count = sysfs_kernel_warn_count_get();
+
+		if (warn_count != self->warn_count && self->kmsg_fd >= 0)
+			kmsg_print_warnings(self->kmsg_fd);
+
+		EXPECT_EQ(warn_count, self->warn_count);
+	}
+
+	if (self->kmsg_fd >= 0)
+		close(self->kmsg_fd);
+}
+
+/*
+ * Put the device into D3hot via config space, close it, and then reopen it.
+ * Closing the device while it is in D3hot must not trigger any kernel warnings
+ * (checked during fixture teardown), and the device must be in D0 when it is
+ * reopened.
+ *
+ * Note that this does not verify that VFIO returns the device to D0 during
+ * close, since that is not reliably observable from userspace: the device may
+ * runtime suspend after it is closed, and opening it also transitions it to
+ * D0.
+ */
+TEST_F(vfio_pci_power_state_test, reopen_after_close_in_d3hot)
+{
+	power_state_set(self->device, self->pm_cap, D3HOT);
+	ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+	vfio_pci_device_cleanup(self->device);
+
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
+	ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+/*
+ * Put the device into D3hot via config space and then enable SR-IOV. VFIO
+ * must return the PF to D0 before enabling VFs, and must not allow the PF to
+ * leave D0 while VFs are enabled.
+ */
+TEST_F(vfio_pci_power_state_test, sriov_enable_in_d3hot)
+{
+	const char *skip_reason = sriov_skip_reason();
+
+	if (skip_reason)
+		SKIP(return, "%s", skip_reason);
+
+	power_state_set(self->device, self->pm_cap, D3HOT);
+	ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+	sysfs_sriov_numvfs_set(device_bdf, 1);
+	EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+
+	power_state_set(self->device, self->pm_cap, D3HOT);
+	EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+
+	sysfs_sriov_numvfs_set(device_bdf, 0);
+
+	power_state_set(self->device, self->pm_cap, D3HOT);
+	EXPECT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+	power_state_set(self->device, self->pm_cap, D0);
+	EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+struct sriov_enable_args {
+	pthread_barrier_t barrier;
+	int delay_us;
+};
+
+static void *sriov_enable_thread(void *arg)
+{
+	struct sriov_enable_args *args = arg;
+
+	pthread_barrier_wait(&args->barrier);
+	usleep(args->delay_us);
+	sysfs_sriov_numvfs_set(device_bdf, 1);
+	return NULL;
+}
+
+/*
+ * Race closing the device while it is in D3hot against enabling SR-IOV. Both
+ * paths transition the device back to D0, which must be serialized by VFIO,
+ * e.g. to avoid restoring the PF's BARs after VFs have been enabled or
+ * double-freeing the saved PM state.
+ *
+ * The D3hot->D0 transition during close includes a delay (10ms by default)
+ * before the device's state is restored. Use a barrier to start closing the
+ * device and enabling SR-IOV at the same time, and then delay enabling SR-IOV
+ * by a few milliseconds so that it lands inside that window. Vary the delay
+ * across iterations to account for differences in timing across systems and
+ * kernel configs.
+ */
+TEST_F(vfio_pci_power_state_test, close_in_d3hot_races_sriov_enable)
+{
+	struct sriov_enable_args args;
+	pthread_t thread;
+	const char *skip_reason;
+	int i;
+
+	skip_reason = sriov_skip_reason();
+	if (skip_reason)
+		SKIP(return, "%s", skip_reason);
+
+	ASSERT_EQ(pthread_barrier_init(&args.barrier, NULL, 2), 0);
+
+	/*
+	 * Keep the number of iterations low since each one enables and
+	 * disables SR-IOV, which can take over a second.
+	 */
+	for (i = 0; i < 10; i++) {
+		args.delay_us = 2000 + i * 500;
+
+		power_state_set(self->device, self->pm_cap, D3HOT);
+		ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+		ASSERT_EQ(pthread_create(&thread, NULL, sriov_enable_thread, &args), 0);
+		pthread_barrier_wait(&args.barrier);
+		vfio_pci_device_cleanup(self->device);
+		self->device = NULL;
+		ASSERT_EQ(pthread_join(thread, NULL), 0);
+
+		sysfs_sriov_numvfs_set(device_bdf, 0);
+
+		self->device = vfio_pci_device_init(device_bdf, self->iommu);
+		ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+	}
+
+	ASSERT_EQ(pthread_barrier_destroy(&args.barrier), 0);
+}
+
+int main(int argc, char *argv[])
+{
+	device_bdf = vfio_selftests_get_bdf(&argc, argv);
+	VFIO_ASSERT_EQ(atexit(disable_vfs), 0);
+
+	return test_harness_run(argc, argv);
+}
-- 
2.56.1.708.g77570a6a1d-goog


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-19  0:02 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19  0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
2026-11-19  0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
2026-11-19  0:01 ` [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active David Matlack
2026-11-19  0:01 ` [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set() David Matlack
2026-11-19  0:01 ` [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled David Matlack
2026-11-19  0:01 ` [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability() David Matlack
2026-11-19  0:01 ` [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get() David Matlack
2026-11-19  0:01 ` [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg David Matlack
2026-11-19  0:01 ` [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions David Matlack

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®