* [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active David Matlack
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Hold vdev->memory_lock for write when calling vfio_pci_set_power_state()
from vfio_pci_core_disable() and vfio_pci_core_register_device(), and
assert that the lock is held for all power state transitions, not just
transitions out of D0.
vfio_pci_core_disable() calls vfio_pci_set_power_state(vdev, PCI_D0)
without holding memory_lock, which allows it to race with
vfio_pci_core_sriov_configure() when userspace writes to sriov_numvfs
while the device is being closed. If the user left the device in D3hot
via a config space write, both threads can transition the device from
D3hot to D0 concurrently. Since a config space write does not change
the runtime PM status, the device stays RPM_ACTIVE, pm_runtime_resume()
in the close path is a no-op, and vfio_pci_set_power_state() performs
the D3hot->D0 transition.
For devices with vdev->needs_pm_restore set (NoSoftRst-), this can
result in:
- vfio_pci_core_disable() restoring the PF's BARs after
vfio_pci_core_sriov_configure() has enabled VFs, since the D3hot->D0
transition sleeps before restoring device state. This reproduces
reliably with vfio_pci_power_state_test and triggers the following
warning:
vfio-pci 0000:01:00.0: can't update enabled VF BAR0 [mem 0xe0200000000-0xe020001ffff 64bit pref]
WARNING: drivers/pci/iov.c:1062 at pci_iov_update_resource+0x170/0x3b0
Call Trace:
pci_update_resource+0x1a3/0x5b0
__pci_set_power_state+0x597/0xb90
vfio_pci_core_disable+0x13b/0x1180
vfio_pci_core_close_device+0x144/0x260
vfio_df_close+0xf2/0x210
vfio_df_unbind_iommufd+0x89/0x140
vfio_device_fops_release+0x85/0xf0
__fput+0x2d6/0x720
fput_close_sync+0xe8/0x180
__x64_sys_close+0x7f/0xc0
do_syscall_64+0xbc/0x320
entry_SYSCALL_64_after_hwframe+0x76/0x7e
- Both threads observing pdev->current_state >= PCI_D3hot and calling
pci_load_and_free_saved_state(pdev, &vdev->pm_save) concurrently,
resulting in a double-free of vdev->pm_save.
vfio_pci_core_register_device() cannot race with other callers since
the device is not yet registered and device_lock is held across probe,
but take the lock there as well so that the lockdep assertion can be
made unconditional and catch any future unlocked callers.
Taking memory_lock in these paths does not introduce any new lock
dependencies. vfio_pci_core_disable() already acquires memory_lock
under dev_set->lock via vfio_pci_runtime_pm_exit(), and
vfio_pci_core_register_device(), like vfio_pci_core_sriov_configure(),
runs under device_lock, which is already taken before memory_lock.
Fixes: f4162eb1e2fc ("vfio/pci: Change the PF power state to D0 before enabling VFs")
Cc: stable@vger.kernel.org
Signed-off-by: David Matlack <dmatlack@google.com>
---
drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..4289ec9bc32f 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -325,12 +325,12 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
bool needs_restore = false, needs_save = false;
int ret;
+ /* Serialize power state changes and accesses to vdev->pm_save. */
+ lockdep_assert_held_write(&vdev->memory_lock);
+
/* Prevent changing power state for PFs with VFs enabled */
- if (state > PCI_D0) {
- lockdep_assert_held_write(&vdev->memory_lock);
- if (vdev->sriov_active)
- return -EBUSY;
- }
+ if (state > PCI_D0 && vdev->sriov_active)
+ return -EBUSY;
if (vdev->needs_pm_restore) {
if (pdev->current_state < PCI_D3hot && state >= PCI_D3hot) {
@@ -699,7 +699,9 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
* reset without restoring the original state (saved locally in
* 'vdev->pm_save').
*/
+ down_write(&vdev->memory_lock);
vfio_pci_set_power_state(vdev, PCI_D0);
+ up_write(&vdev->memory_lock);
/* Stop the device from further DMA */
pci_clear_master(pdev);
@@ -2273,7 +2275,9 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
* be able to get to D3. Therefore first do a D0 transition
* before enabling runtime PM.
*/
+ down_write(&vdev->memory_lock);
vfio_pci_set_power_state(vdev, PCI_D0);
+ up_write(&vdev->memory_lock);
pm_runtime_allow(dev);
if (!vdev->disable_idle_d3)
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
2026-11-19 0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set() David Matlack
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Use READ_ONCE() and WRITE_ONCE() to annotate the read of
vdev->sriov_active in vfio_pci_set_power_state() and the lockless write
that clears it in vfio_pci_core_sriov_configure().
vfio_pci_core_sriov_configure() intentionally clears sriov_active
without holding memory_lock, so the write can race with the read in
vfio_pci_set_power_state() (which runs under memory_lock). The race is
benign: at worst it causes a spurious -EBUSY while the device stays in
D0. Mark the accesses anyway to document the intentional data race and
avoid KCSAN reports.
The write that sets sriov_active is done under memory_lock and is
serialized against the lockless write by device_lock, so it does not
need to be annotated.
Fixes: 40ef3edf151e ("vfio/pci: Use a private flag to prevent power state change with VFs")
Signed-off-by: David Matlack <dmatlack@google.com>
---
drivers/vfio/pci/vfio_pci_core.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 4289ec9bc32f..511dcb49d2f6 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -329,7 +329,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
lockdep_assert_held_write(&vdev->memory_lock);
/* Prevent changing power state for PFs with VFs enabled */
- if (state > PCI_D0 && vdev->sriov_active)
+ if (state > PCI_D0 && READ_ONCE(vdev->sriov_active))
return -EBUSY;
if (vdev->needs_pm_restore) {
@@ -2393,7 +2393,7 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
* state transition would at most result in an -EBUSY, leaving the
* device in PCI_D0.
*/
- vdev->sriov_active = false;
+ WRITE_ONCE(vdev->sriov_active, false);
mutex_lock(&vfio_pci_sriov_pfs_mutex);
list_del_init(&vdev->sriov_pfs_item);
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set()
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
2026-11-19 0:01 ` [PATCH v2 1/8] vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() David Matlack
2026-11-19 0:01 ` [PATCH v2 2/8] vfio/pci: Annotate lockless accesses to vdev->sriov_active David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled David Matlack
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Add __sysfs_sriov_numvfs_set(), which returns the error from writing to
a PF's sriov_numvfs file instead of asserting that the write succeeds,
and implement sysfs_sriov_numvfs_set() on top of it.
This will be used in subsequent commits to skip SR-IOV tests when the
kernel returns -ENOENT, which means SR-IOV cannot be configured on the
PF, e.g. because vfio-pci's enable_sriov module parameter is not set.
Signed-off-by: David Matlack <dmatlack@google.com>
---
.../vfio/lib/include/libvfio/sysfs.h | 1 +
tools/testing/selftests/vfio/lib/sysfs.c | 38 +++++++++++++++----
2 files changed, 31 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
index c9ab1ea8f5a9..ede114181cf8 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
@@ -4,6 +4,7 @@
int sysfs_sriov_totalvfs_get(const char *bdf);
int sysfs_sriov_numvfs_get(const char *bdf);
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
int sysfs_iommu_group_get(const char *bdf);
diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c
index d88e348a1e0d..308da0526543 100644
--- a/tools/testing/selftests/vfio/lib/sysfs.c
+++ b/tools/testing/selftests/vfio/lib/sysfs.c
@@ -56,17 +56,24 @@ static int sysfs_val_get_int(const char *component, const char *name,
return ret;
}
-static void sysfs_val_set(const char *component, const char *name,
- const char *file, const char *val)
+static int sysfs_val_set(const char *component, const char *name,
+ const char *file, const char *val)
{
char path[PATH_MAX];
+ ssize_t ret;
int fd;
snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
VFIO_ASSERT_GT(fd = open(path, O_WRONLY), 0);
- VFIO_ASSERT_EQ(write(fd, val, strlen(val)), strlen(val));
+ ret = write(fd, val, strlen(val));
+ if (ret < 0)
+ ret = -errno;
+ else
+ VFIO_ASSERT_EQ(ret, strlen(val));
+
VFIO_ASSERT_EQ(close(fd), 0);
+ return ret < 0 ? ret : 0;
}
static int sysfs_device_val_get(const char *bdf, const char *file)
@@ -74,17 +81,17 @@ static int sysfs_device_val_get(const char *bdf, const char *file)
return sysfs_val_get_int("devices", bdf, file);
}
-static void sysfs_device_val_set(const char *bdf, const char *file, const char *val)
+static int sysfs_device_val_set(const char *bdf, const char *file, const char *val)
{
- sysfs_val_set("devices", bdf, file, val);
+ return sysfs_val_set("devices", bdf, file, val);
}
-static void sysfs_device_val_set_int(const char *bdf, const char *file, int val)
+static int sysfs_device_val_set_int(const char *bdf, const char *file, int val)
{
char val_str[32];
snprintf_assert(val_str, sizeof(val_str), "%d", val);
- sysfs_device_val_set(bdf, file, val_str);
+ return sysfs_device_val_set(bdf, file, val_str);
}
int sysfs_sriov_totalvfs_get(const char *bdf)
@@ -97,9 +104,24 @@ int sysfs_sriov_numvfs_get(const char *bdf)
return sysfs_device_val_get(bdf, "sriov_numvfs");
}
+/**
+ * __sysfs_sriov_numvfs_set() - Set the number of VFs enabled on a PF
+ * @bdf: BDF of the PF.
+ * @numvfs: Number of VFs to enable, or 0 to disable SR-IOV.
+ *
+ * Return: 0 on success, or a negative errno on failure. The kernel returns
+ * -ENOENT if SR-IOV cannot be configured on the PF, e.g. because its driver
+ * does not support configuring SR-IOV or has it disabled (vfio-pci unless its
+ * enable_sriov module parameter is set).
+ */
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
+{
+ return sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+}
+
void sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
{
- sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+ VFIO_ASSERT_EQ(__sysfs_sriov_numvfs_set(bdf, numvfs), 0);
}
char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i)
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
` (2 preceding siblings ...)
2026-11-19 0:01 ` [PATCH v2 3/8] vfio: selftests: Add __sysfs_sriov_numvfs_set() David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability() David Matlack
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Skip vfio_pci_sriov_uapi_test if writing to the PF's sriov_numvfs file
fails with -ENOENT, which means SR-IOV cannot be configured on the PF.
vfio-pci returns -ENOENT unless its enable_sriov module parameter is
set, which it is not by default, so the test currently fails during
setup instead of skipping.
Fixes: a6e4e726600d ("vfio: selftests: Add tests to validate SR-IOV UAPI")
Signed-off-by: David Matlack <dmatlack@google.com>
---
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
index f3a018b8699d..f27418fa5960 100644
--- a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
+++ b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
@@ -210,6 +210,7 @@ static void vf_setup(void)
{
char *vf_driver;
int nr_vfs;
+ int ret;
nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
if (nr_vfs <= 0)
@@ -220,7 +221,12 @@ static void vf_setup(void)
ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
/* Create only one VF for testing */
- sysfs_sriov_numvfs_set(pf_bdf, 1);
+ ret = __sysfs_sriov_numvfs_set(pf_bdf, 1);
+ if (ret == -ENOENT)
+ ksft_exit_skip("Cannot enable SR-IOV on %s (vfio-pci requires enable_sriov=1)\n",
+ pf_bdf);
+
+ VFIO_ASSERT_EQ(ret, 0);
/*
* Setup an exit handler to destroy the VF in case of failures
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability()
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
` (3 preceding siblings ...)
2026-11-19 0:01 ` [PATCH v2 4/8] vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get() David Matlack
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Add vfio_pci_find_capability(), which walks a device's standard
capability list like the kernel's pci_find_capability().
This will be used in a subsequent commit to locate the PM capability.
Signed-off-by: David Matlack <dmatlack@google.com>
---
.../lib/include/libvfio/vfio_pci_device.h | 2 ++
.../selftests/vfio/lib/vfio_pci_device.c | 36 +++++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
index 007adc322c1e..0fc03bb79448 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
@@ -79,6 +79,8 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
#define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
#define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id);
+
static inline void vfio_pci_cmd_set(struct vfio_pci_device *device, u16 bits)
{
u16 cmd = vfio_pci_config_readw(device, PCI_COMMAND);
diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
index 4063a0e2b3df..ee3f76e2b874 100644
--- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c
+++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
@@ -260,6 +260,42 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
write ? "write to" : "read from", config);
}
+/* Bound the capability list walk in case the list contains a loop. */
+#define PCI_FIND_CAP_TTL 48
+
+/**
+ * vfio_pci_find_capability() - Find a capability in a device's config space
+ * @device: Device to search.
+ * @cap_id: Capability ID to find (PCI_CAP_ID_*).
+ *
+ * Walk the device's standard capability list, like the kernel's
+ * pci_find_capability().
+ *
+ * Return: The config space offset of the first capability with ID @cap_id, or
+ * 0 if the device does not have the capability.
+ */
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id)
+{
+ int ttl = PCI_FIND_CAP_TTL;
+ u8 pos;
+
+ if (!(vfio_pci_config_readw(device, PCI_STATUS) & PCI_STATUS_CAP_LIST))
+ return 0;
+
+ pos = vfio_pci_config_readb(device, PCI_CAPABILITY_LIST);
+
+ while (pos >= PCI_STD_HEADER_SIZEOF && ttl--) {
+ pos &= ~3;
+
+ if (vfio_pci_config_readb(device, pos + PCI_CAP_LIST_ID) == cap_id)
+ return pos;
+
+ pos = vfio_pci_config_readb(device, pos + PCI_CAP_LIST_NEXT);
+ }
+
+ return 0;
+}
+
int __vfio_pci_device_reset(struct vfio_pci_device *device)
{
if (ioctl(device->fd, VFIO_DEVICE_RESET, NULL))
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get()
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
` (4 preceding siblings ...)
2026-11-19 0:01 ` [PATCH v2 5/8] vfio: selftests: Add vfio_pci_find_capability() David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg David Matlack
2026-11-19 0:01 ` [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions David Matlack
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Add a helper to read /sys/kernel/warn_count, which counts the number of
warnings the kernel has encountered since boot. Return a negative value
if the file does not exist, e.g. on kernels that predate it.
Factor the code to read an integer from a sysfs file out of
sysfs_val_get_int() so that it can be shared by both helpers.
This will be used in a subsequent commit to detect kernel warnings
triggered by tests.
Signed-off-by: David Matlack <dmatlack@google.com>
---
.../vfio/lib/include/libvfio/sysfs.h | 1 +
tools/testing/selftests/vfio/lib/sysfs.c | 28 ++++++++++++++++---
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
index ede114181cf8..5ef515eda686 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
@@ -9,5 +9,6 @@ void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
int sysfs_iommu_group_get(const char *bdf);
char *sysfs_driver_get(const char *bdf);
+int sysfs_kernel_warn_count_get(void);
#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H */
diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c
index 308da0526543..3845d81825e3 100644
--- a/tools/testing/selftests/vfio/lib/sysfs.c
+++ b/tools/testing/selftests/vfio/lib/sysfs.c
@@ -33,15 +33,12 @@ static void readlink_base(const char *path, const char *data_fmt, void *out_data
VFIO_ASSERT_EQ(ret, 1);
}
-static int sysfs_val_get_int(const char *component, const char *name,
- const char *file)
+static int sysfs_path_get_int(const char *path)
{
- char path[PATH_MAX];
char buf[32];
int ret;
int fd;
- snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
fd = open(path, O_RDONLY);
if (fd < 0)
return fd;
@@ -56,6 +53,15 @@ static int sysfs_val_get_int(const char *component, const char *name,
return ret;
}
+static int sysfs_val_get_int(const char *component, const char *name,
+ const char *file)
+{
+ char path[PATH_MAX];
+
+ snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
+ return sysfs_path_get_int(path);
+}
+
static int sysfs_val_set(const char *component, const char *name,
const char *file, const char *val)
{
@@ -170,3 +176,17 @@ char *sysfs_driver_get(const char *bdf)
return out_driver;
}
+
+/**
+ * sysfs_kernel_warn_count_get() - Get the number of kernel warnings since boot
+ *
+ * Read /sys/kernel/warn_count, which counts the warnings (e.g. WARN(), KASAN,
+ * and UBSAN reports) the kernel has encountered since boot.
+ *
+ * Return: The number of warnings, or a negative value if the kernel does not
+ * support /sys/kernel/warn_count.
+ */
+int sysfs_kernel_warn_count_get(void)
+{
+ return sysfs_path_get_int("/sys/kernel/warn_count");
+}
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
` (5 preceding siblings ...)
2026-11-19 0:01 ` [PATCH v2 6/8] vfio: selftests: Add sysfs_kernel_warn_count_get() David Matlack
@ 2026-11-19 0:01 ` David Matlack
2026-11-19 0:01 ` [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions David Matlack
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Add kmsg_open() and kmsg_print_warnings() to the library. kmsg_open()
opens /dev/kmsg positioned after the last message in the kernel log, and
kmsg_print_warnings() prints any kernel messages at KERN_WARNING or
higher priority that have been logged since.
Bound how much kmsg_print_warnings() reads and prints, since other
processes (e.g. tests running in parallel) may log a large number of
messages, and note in the output when messages were lost because the
kernel log wrapped. If /dev/kmsg cannot be opened, kmsg_open() prints a
message saying that kernel warnings will not be printed, so that callers
do not each need to.
This will be used in a subsequent commit to print kernel warnings to
the test log when a test triggers one, so the failure can be debugged
from the test log.
Signed-off-by: David Matlack <dmatlack@google.com>
---
.../selftests/vfio/lib/include/libvfio.h | 1 +
.../selftests/vfio/lib/include/libvfio/kmsg.h | 8 ++
tools/testing/selftests/vfio/lib/kmsg.c | 134 ++++++++++++++++++
tools/testing/selftests/vfio/lib/libvfio.mk | 1 +
4 files changed, 144 insertions(+)
create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio.h b/tools/testing/selftests/vfio/lib/include/libvfio.h
index 07862b470777..dca9a9af961e 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio.h
@@ -5,6 +5,7 @@
#include <libvfio/assert.h>
#include <libvfio/iommu.h>
#include <libvfio/iova_allocator.h>
+#include <libvfio/kmsg.h>
#include <libvfio/sysfs.h>
#include <libvfio/vfio_pci_device.h>
#include <libvfio/vfio_pci_driver.h>
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h b/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
new file mode 100644
index 000000000000..9b9e89129b8f
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+
+int kmsg_open(void);
+void kmsg_print_warnings(int fd);
+
+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H */
diff --git a/tools/testing/selftests/vfio/lib/kmsg.c b/tools/testing/selftests/vfio/lib/kmsg.c
new file mode 100644
index 000000000000..a25770d2247a
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/kmsg.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <syslog.h>
+#include <unistd.h>
+
+#include <libvfio.h>
+
+/* Large enough to hold any single /dev/kmsg record. */
+#define KMSG_RECORD_MAX 8192
+
+/* Limit how much is printed if the kernel logs a flood of warnings. */
+#define KMSG_PRINT_MAX 512
+
+/*
+ * Limit how many records are read, so that this terminates even if other
+ * processes are logging messages faster than they can be read.
+ */
+#define KMSG_READ_MAX 65536
+
+/**
+ * kmsg_open() - Open /dev/kmsg to read kernel messages logged from now on
+ *
+ * Open /dev/kmsg positioned after the last message currently in the kernel
+ * log, so that kmsg_print_warnings() only prints messages logged after this
+ * call. If /dev/kmsg cannot be opened, e.g. because the caller is not allowed
+ * to read the kernel log, print a message saying that kernel warnings will not
+ * be printed.
+ *
+ * Return: A file descriptor on success, or a negative errno on failure.
+ */
+int kmsg_open(void)
+{
+ int fd, ret;
+
+ fd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC);
+ if (fd < 0) {
+ ret = -errno;
+ goto out_err;
+ }
+
+ if (lseek(fd, 0, SEEK_END) < 0) {
+ ret = -errno;
+ close(fd);
+ goto out_err;
+ }
+
+ return fd;
+
+out_err:
+ fprintf(stderr,
+ "# kmsg: Failed to open /dev/kmsg (%s), kernel warnings will not be printed\n",
+ strerror(-ret));
+ return ret;
+}
+
+/**
+ * kmsg_print_warnings() - Print kernel warnings logged since kmsg_open()
+ * @fd: File descriptor returned by kmsg_open().
+ *
+ * Print all kernel messages at KERN_WARNING or higher priority that have been
+ * logged since kmsg_open() (or since the last call to this function), e.g. to
+ * provide context for a test failure caused by a kernel warning.
+ *
+ * Messages may be missing if the kernel log wrapped since they were logged,
+ * and output is limited if the kernel logged a large number of messages. A
+ * note is printed in either case.
+ */
+void kmsg_print_warnings(int fd)
+{
+ char buf[KMSG_RECORD_MAX];
+ bool truncated = true;
+ bool lost = false;
+ int printed = 0;
+ int omitted = 0;
+ int i;
+
+ for (i = 0; i < KMSG_READ_MAX; i++) {
+ char *text, *end;
+ ssize_t len;
+ int prio;
+
+ len = read(fd, buf, sizeof(buf) - 1);
+ if (len < 0 && errno == EPIPE) {
+ /* Messages were overwritten before being read. */
+ lost = true;
+ continue;
+ }
+
+ /* Stop at EAGAIN (no more messages) or on any other error. */
+ if (len <= 0) {
+ truncated = false;
+ break;
+ }
+
+ buf[len] = '\0';
+
+ /*
+ * Each record is "<prio>,<seq>,<ts>,<flags>;<message>\n",
+ * optionally followed by " KEY=VALUE\n" lines. See
+ * Documentation/ABI/testing/dev-kmsg.
+ */
+ text = strchr(buf, ';');
+ if (!text || sscanf(buf, "%d,", &prio) != 1)
+ continue;
+
+ text++;
+ end = strchr(text, '\n');
+ if (end)
+ *end = '\0';
+
+ if (LOG_FAC(prio) != LOG_FAC(LOG_KERN) || LOG_PRI(prio) > LOG_WARNING)
+ continue;
+
+ if (printed < KMSG_PRINT_MAX) {
+ fprintf(stderr, "# kmsg: %s\n", text);
+ printed++;
+ } else {
+ omitted++;
+ }
+ }
+
+ if (omitted)
+ fprintf(stderr, "# kmsg: (%d more messages omitted)\n", omitted);
+
+ if (lost)
+ fprintf(stderr, "# kmsg: (some messages were lost, see the kernel log)\n");
+
+ if (truncated)
+ fprintf(stderr, "# kmsg: (stopped reading, see the kernel log)\n");
+}
diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk
index eff95828719e..f8af0c14c3d0 100644
--- a/tools/testing/selftests/vfio/lib/libvfio.mk
+++ b/tools/testing/selftests/vfio/lib/libvfio.mk
@@ -5,6 +5,7 @@ LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
LIBVFIO_C := iommu.c
LIBVFIO_C += iova_allocator.c
+LIBVFIO_C += kmsg.c
LIBVFIO_C += libvfio.c
LIBVFIO_C += sysfs.c
LIBVFIO_C += vfio_pci_device.c
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 8/8] vfio: selftests: Add test for device power state transitions
2026-11-19 0:01 [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement David Matlack
` (6 preceding siblings ...)
2026-11-19 0:01 ` [PATCH v2 7/8] vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg David Matlack
@ 2026-11-19 0:01 ` David Matlack
7 siblings, 0 replies; 9+ messages in thread
From: David Matlack @ 2026-11-19 0:01 UTC (permalink / raw)
To: kvm, linux-kernel, linux-kselftest, stable
Cc: Abhishek Sahu, Alex Williamson, David Matlack,
Raghavendra Rao Ananta, Shuah Khan, Vipin Sharma
Add vfio_pci_power_state_test to exercise VFIO's handling of device
power state changes initiated by userspace through the PM capability in
config space.
The test covers:
- Closing a device in D3hot and then reopening it, which must leave
the device in D0.
- Enabling SR-IOV while the PF is in D3hot, which must return the PF to
D0 and prevent it from leaving D0 while VFs are enabled.
- Racing a close of the device while it is in D3hot against enabling
SR-IOV, both of which transition the device back to D0.
The SR-IOV tests are skipped if the device does not support SR-IOV,
already has VFs enabled, or SR-IOV cannot be configured on it (e.g.
vfio-pci with vfio_pci.enable_sriov=N). If an SR-IOV test case fails,
any VFs it left enabled are disabled so they don't affect later test
cases.
Fail the test if /sys/kernel/warn_count changes between fixture setup
and teardown, since bugs such as the close vs. SR-IOV enable race show
up only as kernel warnings. When that happens, print the warnings
logged to /dev/kmsg during the test so that the failure can be debugged
from the test log.
Signed-off-by: David Matlack <dmatlack@google.com>
---
tools/testing/selftests/vfio/Makefile | 1 +
.../vfio/vfio_pci_power_state_test.c | 256 ++++++++++++++++++
2 files changed, 257 insertions(+)
create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c
diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile
index 2c32c48db509..e2b152528560 100644
--- a/tools/testing/selftests/vfio/Makefile
+++ b/tools/testing/selftests/vfio/Makefile
@@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test
TEST_GEN_PROGS += vfio_pci_device_test
TEST_GEN_PROGS += vfio_pci_device_init_perf_test
TEST_GEN_PROGS += vfio_pci_driver_test
+TEST_GEN_PROGS += vfio_pci_power_state_test
TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
TEST_FILES += scripts/cleanup.sh
diff --git a/tools/testing/selftests/vfio/vfio_pci_power_state_test.c b/tools/testing/selftests/vfio/vfio_pci_power_state_test.c
new file mode 100644
index 000000000000..5b185c174036
--- /dev/null
+++ b/tools/testing/selftests/vfio/vfio_pci_power_state_test.c
@@ -0,0 +1,256 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <pthread.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <linux/pci_regs.h>
+
+#include <libvfio.h>
+
+#include "kselftest_harness.h"
+
+/* PCI_PM_CTRL_STATE_MASK values */
+#define D0 0
+#define D3HOT 3
+
+static const char *device_bdf;
+
+static u16 power_state_get(struct vfio_pci_device *device, u8 pm_cap)
+{
+ return vfio_pci_config_readw(device, pm_cap + PCI_PM_CTRL) &
+ PCI_PM_CTRL_STATE_MASK;
+}
+
+static void power_state_set(struct vfio_pci_device *device, u8 pm_cap,
+ u16 state)
+{
+ u16 pmcsr = vfio_pci_config_readw(device, pm_cap + PCI_PM_CTRL);
+
+ pmcsr &= ~PCI_PM_CTRL_STATE_MASK;
+ pmcsr |= state;
+
+ vfio_pci_config_writew(device, pm_cap + PCI_PM_CTRL, pmcsr);
+}
+
+/*
+ * Set by test cases that enable VFs, so that disable_vfs() can disable any VFs
+ * left enabled if the test case fails. Only set in the child process that runs
+ * the test case.
+ */
+static bool test_enabled_vfs;
+
+/*
+ * Disable VFs left enabled by a failed test case, so they don't affect later
+ * test cases. Called from fixture teardown, which runs when an ASSERT fails,
+ * and from an atexit() handler, which runs when a library helper exits the
+ * test process early (e.g. on a VFIO_ASSERT() failure).
+ */
+static void disable_vfs(void)
+{
+ if (test_enabled_vfs && sysfs_sriov_numvfs_get(device_bdf) > 0)
+ __sysfs_sriov_numvfs_set(device_bdf, 0);
+}
+
+/*
+ * Check whether the test can enable VFs on the device, briefly enabling one VF
+ * to find out. Returns the reason to skip the test, or NULL if it can run.
+ */
+static const char *sriov_skip_reason(void)
+{
+ int ret;
+
+ if (sysfs_sriov_totalvfs_get(device_bdf) <= 0)
+ return "Device does not support SR-IOV";
+
+ if (sysfs_sriov_numvfs_get(device_bdf) != 0)
+ return "SR-IOV is already enabled on the device";
+
+ /* From here on, any VFs on the device were enabled by this test. */
+ test_enabled_vfs = true;
+
+ /* -ENOENT means SR-IOV cannot be configured on the device. */
+ ret = __sysfs_sriov_numvfs_set(device_bdf, 1);
+ if (ret == -ENOENT)
+ return "Cannot enable SR-IOV on the device (vfio-pci requires enable_sriov=1)";
+
+ VFIO_ASSERT_EQ(ret, 0);
+ sysfs_sriov_numvfs_set(device_bdf, 0);
+ return NULL;
+}
+
+FIXTURE(vfio_pci_power_state_test) {
+ struct iommu *iommu;
+ struct vfio_pci_device *device;
+ u8 pm_cap;
+ int warn_count;
+ int kmsg_fd;
+};
+
+FIXTURE_SETUP(vfio_pci_power_state_test)
+{
+ /*
+ * Snapshot the kernel's warning count so that teardown can check that
+ * the test did not trigger any kernel warnings, which are otherwise
+ * invisible to the test. Open /dev/kmsg first so that any warning
+ * counted after the snapshot can be printed if the check fails.
+ */
+ self->kmsg_fd = kmsg_open();
+ self->warn_count = sysfs_kernel_warn_count_get();
+
+ self->iommu = iommu_init(default_iommu_mode);
+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
+
+ self->pm_cap = vfio_pci_find_capability(self->device, PCI_CAP_ID_PM);
+ if (!self->pm_cap)
+ SKIP(return, "Device does not have a PM capability\n");
+
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+FIXTURE_TEARDOWN(vfio_pci_power_state_test)
+{
+ disable_vfs();
+
+ if (self->device)
+ vfio_pci_device_cleanup(self->device);
+
+ iommu_cleanup(self->iommu);
+
+ if (self->warn_count >= 0) {
+ int warn_count = sysfs_kernel_warn_count_get();
+
+ if (warn_count != self->warn_count && self->kmsg_fd >= 0)
+ kmsg_print_warnings(self->kmsg_fd);
+
+ EXPECT_EQ(warn_count, self->warn_count);
+ }
+
+ if (self->kmsg_fd >= 0)
+ close(self->kmsg_fd);
+}
+
+/*
+ * Put the device into D3hot via config space, close it, and then reopen it.
+ * Closing the device while it is in D3hot must not trigger any kernel warnings
+ * (checked during fixture teardown), and the device must be in D0 when it is
+ * reopened.
+ *
+ * Note that this does not verify that VFIO returns the device to D0 during
+ * close, since that is not reliably observable from userspace: the device may
+ * runtime suspend after it is closed, and opening it also transitions it to
+ * D0.
+ */
+TEST_F(vfio_pci_power_state_test, reopen_after_close_in_d3hot)
+{
+ power_state_set(self->device, self->pm_cap, D3HOT);
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+ vfio_pci_device_cleanup(self->device);
+
+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+/*
+ * Put the device into D3hot via config space and then enable SR-IOV. VFIO
+ * must return the PF to D0 before enabling VFs, and must not allow the PF to
+ * leave D0 while VFs are enabled.
+ */
+TEST_F(vfio_pci_power_state_test, sriov_enable_in_d3hot)
+{
+ const char *skip_reason = sriov_skip_reason();
+
+ if (skip_reason)
+ SKIP(return, "%s", skip_reason);
+
+ power_state_set(self->device, self->pm_cap, D3HOT);
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+ sysfs_sriov_numvfs_set(device_bdf, 1);
+ EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+
+ power_state_set(self->device, self->pm_cap, D3HOT);
+ EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+
+ sysfs_sriov_numvfs_set(device_bdf, 0);
+
+ power_state_set(self->device, self->pm_cap, D3HOT);
+ EXPECT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+ power_state_set(self->device, self->pm_cap, D0);
+ EXPECT_EQ(power_state_get(self->device, self->pm_cap), D0);
+}
+
+struct sriov_enable_args {
+ pthread_barrier_t barrier;
+ int delay_us;
+};
+
+static void *sriov_enable_thread(void *arg)
+{
+ struct sriov_enable_args *args = arg;
+
+ pthread_barrier_wait(&args->barrier);
+ usleep(args->delay_us);
+ sysfs_sriov_numvfs_set(device_bdf, 1);
+ return NULL;
+}
+
+/*
+ * Race closing the device while it is in D3hot against enabling SR-IOV. Both
+ * paths transition the device back to D0, which must be serialized by VFIO,
+ * e.g. to avoid restoring the PF's BARs after VFs have been enabled or
+ * double-freeing the saved PM state.
+ *
+ * The D3hot->D0 transition during close includes a delay (10ms by default)
+ * before the device's state is restored. Use a barrier to start closing the
+ * device and enabling SR-IOV at the same time, and then delay enabling SR-IOV
+ * by a few milliseconds so that it lands inside that window. Vary the delay
+ * across iterations to account for differences in timing across systems and
+ * kernel configs.
+ */
+TEST_F(vfio_pci_power_state_test, close_in_d3hot_races_sriov_enable)
+{
+ struct sriov_enable_args args;
+ pthread_t thread;
+ const char *skip_reason;
+ int i;
+
+ skip_reason = sriov_skip_reason();
+ if (skip_reason)
+ SKIP(return, "%s", skip_reason);
+
+ ASSERT_EQ(pthread_barrier_init(&args.barrier, NULL, 2), 0);
+
+ /*
+ * Keep the number of iterations low since each one enables and
+ * disables SR-IOV, which can take over a second.
+ */
+ for (i = 0; i < 10; i++) {
+ args.delay_us = 2000 + i * 500;
+
+ power_state_set(self->device, self->pm_cap, D3HOT);
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D3HOT);
+
+ ASSERT_EQ(pthread_create(&thread, NULL, sriov_enable_thread, &args), 0);
+ pthread_barrier_wait(&args.barrier);
+ vfio_pci_device_cleanup(self->device);
+ self->device = NULL;
+ ASSERT_EQ(pthread_join(thread, NULL), 0);
+
+ sysfs_sriov_numvfs_set(device_bdf, 0);
+
+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
+ ASSERT_EQ(power_state_get(self->device, self->pm_cap), D0);
+ }
+
+ ASSERT_EQ(pthread_barrier_destroy(&args.barrier), 0);
+}
+
+int main(int argc, char *argv[])
+{
+ device_bdf = vfio_selftests_get_bdf(&argc, argv);
+ VFIO_ASSERT_EQ(atexit(disable_vfs), 0);
+
+ return test_harness_run(argc, argv);
+}
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread