[PATCH v2 0/2] vfio/nvgrace-gpu: Separate the fallback-mode open_device

[PATCH v2 0/2] vfio/nvgrace-gpu: Separate the fallback-mode open_device

[Andrea Parri]

In fallback mode, where the ACPI memory properties are missing or the
C2C link is down, nvgrace-gpu opens a device through
nvgrace_gpu_open_device() but closes it through
vfio_pci_core_close_device().  The open registers an empty usemem range
for poison handling that nothing unregisters, so a second open fails
with -EBUSY, and the first open after an unbind hits a KASAN
slab-use-after-free in register_pfn_address_space().

Patch 1 gives the fallback table its own open_device, as Alex suggested
on v1, and is marked for stable.  Patch 2 drops a usemem length check
in nvgrace_gpu_open_device() that only existed for fallback mode.

Tested on x86-64 under virtme-ng with KASAN, not on Grace hardware.  A
QEMU cxl-type3 function, bound through driver_override, takes the
fallback path and is opened through a noiommu VFIO cdev.  On v7.3-rc7
the second open fails with -EBUSY and the open after unbind and rebind
reports the use-after-free.  With patch 1, and with the whole series,
neither happens.  Patch 2 only affects full mode, which this setup
cannot reach.

Changes in v2:
- Replace the usemem.memlength guards with a core-only open_device for
  the fallback table (Alex)
- Add patch 2
- Fallback open no longer fails when BAR0 cannot be mapped; Ankit
  confirmed fallback mode does not need that check
- Rebase on v7.3-rc7

v1: https://lore.kernel.org/r/20260925161024.189771-1-parri.andrea@gmail.com/

Andrea Parri (2):
  vfio/nvgrace-gpu: Use a core-only open_device in fallback mode
  vfio/nvgrace-gpu: Drop the usemem length check in open_device

 drivers/vfio/pci/nvgrace-gpu/main.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)


base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.53.0

[PATCH v2 1/2] vfio/nvgrace-gpu: Use a core-only open_device in fallback mode

[Andrea Parri]

When the ACPI memory properties are missing or the C2C link is down, the
device is exposed through nvgrace_gpu_pci_core_ops.  That table opens
through nvgrace_gpu_open_device() but closes through
vfio_pci_core_close_device(), so open and close are not symmetric.

nvgrace_gpu_open_device() registers nvdev->usemem for poison handling
unconditionally.  In fallback mode usemem.memlength is zero, and
register_pfn_address_space() inserts a node covering [0, ULONG_MAX].
Nothing removes it: a second open fails with -EBUSY, and once the device
is unbound the node outlives the nvdev that contains it.  The next open
after a rebind reports:

  BUG: KASAN: slab-use-after-free in interval_tree_iter_first+0x209/0x310
  Read of size 8 at addr ffff88800894aa30 by task nvgrace_repro/203
  Call Trace:
   <TASK>
   dump_stack_lvl+0x5d/0x80
   print_report+0x153/0x4c6
   kasan_report+0xe4/0x1a0
   interval_tree_iter_first+0x209/0x310
   register_pfn_address_space+0x97/0x100
   nvgrace_gpu_open_device+0x21c/0x730
   vfio_df_open+0x213/0x520
   vfio_df_ioctl_bind_iommufd+0x3e8/0x6e0
   vfio_device_fops_unl_ioctl+0x782/0x1570
   __x64_sys_ioctl+0x13c/0x1c0
   do_syscall_64+0x106/0x630
   entry_SYSCALL_64_after_hwframe+0x77/0x7f
   </TASK>

  Allocated by task 188:
   ...
   _vfio_alloc_device+0x3a/0x500
   nvgrace_gpu_probe+0xcb/0x660
   local_pci_probe+0xcf/0x170
   ...

  Freed by task 188:
   ...
   kfree+0x20d/0x600
   device_release+0xcf/0x250
   kobject_put+0x152/0x2f0
   pci_device_remove+0xb0/0x1c0
   device_release_driver_internal+0x38d/0x540
   unbind_store+0xde/0x100
   ...

Give the fallback table its own open_device that only calls
vfio_pci_core_enable() and vfio_pci_core_finish_enable(), the
counterpart of vfio_pci_core_close_device().  Fallback mode then never
touches the nvgrace memory regions, as Alex suggested.

This should be safe because the fallback table uses the vfio-pci core
read, write and mmap handlers, so nothing there uses the state
nvgrace_gpu_open_device() sets up.  The one visible difference is that a
fallback open no longer fails when BAR0 cannot be mapped, which matches
vfio-pci.

Fixes: e5f19b619fa0 ("vfio/nvgrace-gpu: register device memory for poison handling")
Cc: stable@vger.kernel.org
Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
---
v2: Replace the usemem.memlength guards with a core-only open_device
    for the fallback table (Alex)
    Ankit confirmed fallback mode does not need the BAR0 check:
    https://lore.kernel.org/r/SAWPR12MB99926787B53C4FF469A6F3BE7AB09D2@SAWPR12MB999267.namprd12.prod.outlook.com/
v1: https://lore.kernel.org/r/20260925161024.189771-1-parri.andrea@gmail.com/

 drivers/vfio/pci/nvgrace-gpu/main.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c
index 963fd8ded20d1..208463c218088 100644
--- a/drivers/vfio/pci/nvgrace-gpu/main.c
+++ b/drivers/vfio/pci/nvgrace-gpu/main.c
@@ -1083,6 +1083,21 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = {
 	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
 };
 
+static int nvgrace_gpu_core_open_device(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+
+	return 0;
+}
+
 static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = {
 	.get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
 };
@@ -1091,7 +1106,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {
 	.name		= "nvgrace-gpu-vfio-pci-core",
 	.init		= vfio_pci_core_init_dev,
 	.release	= vfio_pci_core_release_dev,
-	.open_device	= nvgrace_gpu_open_device,
+	.open_device	= nvgrace_gpu_core_open_device,
 	.close_device	= vfio_pci_core_close_device,
 	.ioctl		= vfio_pci_core_ioctl,
 	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
-- 
2.53.0

[PATCH v2 2/2] vfio/nvgrace-gpu: Drop the usemem length check in open_device

[Andrea Parri]

nvgrace_gpu_open_device() sets up the fake BAR registers and remap_lock
only when usemem.memlength is nonzero.  The check exists for fallback
mode, which used to share this open path.  Fallback mode now has its own
open_device, so only devices with ACPI memory properties reach this one.

For those devices usemem.memlength is never zero: an empty region from
nvgrace_gpu_fetch_memory_property() selects fallback mode, and probe
fails when nvgrace_gpu_init_nvdev_struct() rounds usemem down to
nothing.  Do the setup unconditionally.

Assisted-by: LLM
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
---
v2: New patch

 drivers/vfio/pci/nvgrace-gpu/main.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c
index 208463c218088..8dc792b4fe7ac 100644
--- a/drivers/vfio/pci/nvgrace-gpu/main.c
+++ b/drivers/vfio/pci/nvgrace-gpu/main.c
@@ -185,10 +185,8 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
 	if (ret)
 		return ret;
 
-	if (nvdev->usemem.memlength) {
-		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
-		mutex_init(&nvdev->remap_lock);
-	}
+	nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+	mutex_init(&nvdev->remap_lock);
 
 	/*
 	 * GPU readiness is checked by reading the BAR0 registers.
-- 
2.53.0