mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver
@ 2026-11-19 21:50 Simon Song
  2026-11-19 21:50 ` [PATCH v3 1/9] rust: pci: add VFIO override device IDs Simon Song
                   ` (8 more replies)
  0 siblings, 9 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: Simon Song, alex, jgg, yishaih, skolothumtho, kevin.tian,
	airlied, simona, ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh,
	lossin, a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney,
	cjia, smitra, kjaju, alkumar, ankita, aniketa, kwankhede,
	targupta, linux-kernel, zhiwang, zhiw, boqun, daniel.almeida,
	tamird, work, jgg, bhelgaas, kwilczynski, dri-devel, peter

This is v3 of the Rust VFIO PCI abstractions and the NVIDIA VFIO variant
driver [1]. The driver uses nova-core's typed SR-IOV PF interface to
create, reset and release vGPU instances while VFIO provides userspace
access to the PCI device.

Patches 1-5 add the PCI helpers and callbacks the VFIO abstractions
need. Patch 6 adds the VFIO device handling common to all buses, and
patch 7 the VFIO PCI abstractions on top of it. Patch 8 publishes the
typed Nova PF interface, and patch 9 adds the NVIDIA VFIO variant
driver.

This series is based on drm-rust-next with the Rust PCI SR-IOV series
[2] applied. Patches 1-7 need nothing else. Patches 8 and 9 also depend
on Alok's vGPU manager series [3] and can follow once it has landed.

Runtime PM is not part of this series anymore. As Danilo suggested, it
will follow as a separate series based on Beata's generic Rust runtime
PM work [4], so that the rest can land independently. Until then,
vfio::pci::Registration keeps the device runtime active while it is
registered, so a VF bound to the driver is not runtime suspended.

Changes since v2:
- Drop runtime PM from this series (v2 patch 08 and the runtime PM
  parts of v2 patches 04, 09 and 12), see above.
- Put the common VFIO device handling first (v2 patch 10, now patch 6)
  and squash v2 patches 04 and 09 into one patch (patch 7) (Danilo).
- Drop the bind callback (v2 patch 06). A driver registers its VFIO
  device with vfio::pci::Registration::new(), for example from probe,
  and unregisters it by dropping the Registration (Danilo).
- Patch 1: say first what the function does in its documentation
  (Gary).
- Patch 2: return Result<u16> from vf_id(), and add a C helper so that
  it builds without CONFIG_PCI_IOV (Danilo).
- Patch 3: make DRIVER_MANAGED_DMA an unsafe marker, until the DMA
  setup tokens exist (Danilo).
- Patches 4 and 5: use Device::drvdata_borrow() and #[optional]; add
  the pci::ChannelState and pci::ErsResult types (Danilo).
- Patch 6: add vfio::DeviceBase and RawDevice; use GATs instead of
  ForLt/CovariantForLt and drop the covariance requirement on the
  registration data (Danilo).
- Patch 8: drop the CONFIG_PCI_IOV guards, use a pin-init
  self-reference for the GPU, simplify the SAFETY comment and use
  enable_all() (Danilo).
- Patch 9: rename the driver to nova-vgpu (Danilo), and adapt it to the
  changes above.

v2 patches 01, 02, 03, 05, 07, 10, 04+09, 11 and 12 are now patches
1 to 9.

[1] https://lore.kernel.org/nova-gpu/20261023154009.2246627-1-xinmengs@nvidia.com/
[2] https://lore.kernel.org/all/20261111223337.508989-1-xinmengs@nvidia.com/
[3] https://lore.kernel.org/all/cover.1793906667.git.alkumar@nvidia.com/
[4] https://lore.kernel.org/all/20261026120523.526942-2-beata.michalska@arm.com/

Danilo Krummrich (5):
  rust: pci: add VFIO override device IDs
  rust: pci: expose the VF index within its PF
  rust: pci: allow drivers to manage DMA ownership
  rust: vfio: add PCI variant driver abstractions
  vfio/nova-vgpu: add the Rust VFIO variant driver

Simon Song (2):
  rust: pci: add reset callbacks
  rust: pci: add an error_detected callback

Zhi Wang (2):
  rust: vfio: add common VFIO device handling
  gpu: nova-core: publish typed SR-IOV PF APIs

 drivers/gpu/Makefile                    |  15 +-
 drivers/gpu/nova-core/driver.rs         |  34 +-
 drivers/gpu/nova-core/nova_core.rs      |   3 +-
 drivers/gpu/nova-core/vgpu.rs           |   1 -
 drivers/gpu/nova-core/vgpu/vgpu_api.rs  | 178 ++++--
 drivers/vfio/pci/Kconfig                |   2 +
 drivers/vfio/pci/Makefile               |   2 +
 drivers/vfio/pci/nova-vgpu/Kconfig      |  20 +
 drivers/vfio/pci/nova-vgpu/Makefile     |   3 +
 drivers/vfio/pci/nova-vgpu/nova_vgpu.rs | 375 ++++++++++++
 rust/bindings/bindings_helper.h         |   4 +
 rust/helpers/pci.c                      |   5 +
 rust/kernel/lib.rs                      |   2 +
 rust/kernel/pci.rs                      | 194 ++++++
 rust/kernel/pci/iov.rs                  |  11 +
 rust/kernel/vfio.rs                     | 433 +++++++++++++
 rust/kernel/vfio/pci.rs                 | 777 ++++++++++++++++++++++++
 17 files changed, 2011 insertions(+), 48 deletions(-)
 create mode 100644 drivers/vfio/pci/nova-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nova-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nova-vgpu/nova_vgpu.rs
 create mode 100644 rust/kernel/vfio.rs
 create mode 100644 rust/kernel/vfio/pci.rs

-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 1/9] rust: pci: add VFIO override device IDs
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 2/9] rust: pci: expose the VF index within its PF Simon Song
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
	ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, zhiw, boqun, daniel.almeida, tamird, work,
	jgg, bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Danilo Krummrich <dakr@kernel.org>

VFIO variant drivers need override-only IDs and VFIO-specific module
aliases so userspace can select them for PCI assignment.

Add a DeviceId constructor equivalent to PCI_DRIVER_OVERRIDE_DEVICE_VFIO,
matching a vendor and device without restricting the PCI class.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 4887052d31b1..fa67f2fc26bf 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -209,6 +209,21 @@ macro_rules! module_pci_driver {
 impl DeviceId {
     const PCI_ANY_ID: u32 = !0;
 
+    /// Create a new `pci::DeviceId` for a VFIO driver from a vendor and device ID.
+    ///
+    /// The ID matches every device with this vendor and device ID, whatever its class, but only
+    /// once userspace has selected the driver for the device through `driver_override`. The
+    /// driver is never bound to a matching device automatically. The module alias generated for
+    /// the ID has the `vfio_pci:` prefix, which lets userspace find the VFIO driver for a device.
+    ///
+    /// Equivalent to C's `PCI_DRIVER_OVERRIDE_DEVICE_VFIO` macro.
+    #[inline]
+    pub const fn from_id_vfio_override(vendor: Vendor, device: u32) -> Self {
+        let mut id = Self::from_id(vendor, device);
+        id.0.override_only = bindings::PCI_ID_F_VFIO_DRIVER_OVERRIDE;
+        id
+    }
+
     /// Equivalent to C's `PCI_DEVICE` macro.
     ///
     /// Create a new `pci::DeviceId` from a vendor and device ID.
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 2/9] rust: pci: expose the VF index within its PF
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
  2026-11-19 21:50 ` [PATCH v3 1/9] rust: pci: add VFIO override device IDs Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 3/9] rust: pci: allow drivers to manage DMA ownership Simon Song
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
	ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, zhiw, boqun, daniel.almeida, tamird, work,
	jgg, bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Danilo Krummrich <dakr@kernel.org>

Nova identifies a vGPU instance by the VF's index within its PF. Expose
pci_iov_vf_id() through Device::vf_id() so a Rust VF driver can obtain
that index, with an error for devices that are not VFs.

Because enabling or disabling SR-IOV can update the PF's VF offset and
stride, from which the index is computed, restrict vf_id() to bound
devices, Device<device::Bound>.

Suggested-by: Danilo Krummrich <dakr@kernel.org>
Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/helpers/pci.c     |  5 +++++
 rust/kernel/pci/iov.rs | 11 +++++++++++
 2 files changed, 16 insertions(+)

diff --git a/rust/helpers/pci.c b/rust/helpers/pci.c
index 57c28ddeeb9b..1e677ebfe0d4 100644
--- a/rust/helpers/pci.c
+++ b/rust/helpers/pci.c
@@ -41,6 +41,11 @@ __rust_helper u32 rust_helper_pci_ext_cap_next(u32 header)
 }
 
 #ifndef CONFIG_PCI_IOV
+__rust_helper int rust_helper_pci_iov_vf_id(struct pci_dev *pdev)
+{
+	return pci_iov_vf_id(pdev);
+}
+
 __rust_helper int rust_helper_pci_enable_sriov(struct pci_dev *pdev,
 					       int nr_virtfn)
 {
diff --git a/rust/kernel/pci/iov.rs b/rust/kernel/pci/iov.rs
index d0db9d4a0f9b..5bb1340c9150 100644
--- a/rust/kernel/pci/iov.rs
+++ b/rust/kernel/pci/iov.rs
@@ -379,6 +379,17 @@ unsafe impl<F: ForLt> Send for VfRegistration<'_, F> where for<'a> F::Of<'a>: Se
 unsafe impl<F: ForLt> Sync for VfRegistration<'_, F> where for<'a> F::Of<'a>: Send + Sync {}
 
 impl Device<device::Bound> {
+    /// Return the zero-based VF index within its PF, or an error for a non-VF device.
+    ///
+    /// Returns [`ENOSYS`] when `CONFIG_PCI_IOV` is disabled.
+    #[inline]
+    pub fn vf_id(&self) -> Result<u16> {
+        // SAFETY: `self.as_raw()` points to a live PCI device; the helper checks VF membership.
+        let id = unsafe { bindings::pci_iov_vf_id(self.as_raw()) };
+        to_result(id)?;
+        Ok(id.try_into()?)
+    }
+
     /// Internal helper: reads the `vf_registration_data_rust` pointer from the
     /// PF, checks the `TypeId`, and returns a pinned reference.
     ///
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 3/9] rust: pci: allow drivers to manage DMA ownership
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
  2026-11-19 21:50 ` [PATCH v3 1/9] rust: pci: add VFIO override device IDs Simon Song
  2026-11-19 21:50 ` [PATCH v3 2/9] rust: pci: expose the VF index within its PF Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 4/9] rust: pci: add reset callbacks Simon Song
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
	ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, zhiw, boqun, daniel.almeida, tamird, work,
	jgg, bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Danilo Krummrich <dakr@kernel.org>

VFIO manages DMA ownership through the userspace IOMMU interface. Its
PCI variant drivers must opt out of the driver's default DMA domain.

Add Driver::DRIVER_MANAGED_DMA and pass it to the PCI core. Make it an
unsafe marker as suggested, until the DMA setup tokens exist.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index fa67f2fc26bf..6013c89bfb5e 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -105,6 +105,7 @@ unsafe fn register(
             (*pdrv.get()).probe = Some(Self::probe_callback);
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
+            (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA.is_some();
             if T::HAS_SRIOV_ENABLE {
                 (*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
             }
@@ -293,6 +294,24 @@ unsafe impl RawDeviceIdIndex for DeviceId {
     const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::pci_device_id, driver_data);
 }
 
+/// Marker for a driver that manages DMA ownership of its devices by itself.
+///
+/// The driver core does not claim the default DMA domain for the devices of such a driver. See
+/// [`Driver::DRIVER_MANAGED_DMA`].
+pub struct DriverManagedDma<D: ?Sized>(PhantomData<fn() -> *mut D>);
+
+impl<D: ?Sized> DriverManagedDma<D> {
+    /// Declares that driver `D` manages DMA ownership of its devices by itself.
+    ///
+    /// # Safety
+    ///
+    /// The caller must ensure that devices bound to `D` are not used for DMA allocations or
+    /// mappings through the kernel DMA API, directly or through helpers.
+    pub const unsafe fn new() -> Self {
+        Self(PhantomData)
+    }
+}
+
 /// `IdTable` type for PCI.
 pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
 
@@ -356,6 +375,10 @@ pub trait Driver {
     /// The table of device ids supported by the driver.
     const ID_TABLE: IdTable<Self::IdInfo>;
 
+    /// Set if the driver manages DMA ownership of its devices by itself, instead of using the
+    /// default DMA domain. Construct with [`DriverManagedDma::new`].
+    const DRIVER_MANAGED_DMA: Option<DriverManagedDma<Self>> = None;
+
     /// PCI driver probe.
     ///
     /// Called when a new pci device is added or discovered. Implementers should
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 4/9] rust: pci: add reset callbacks
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (2 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 3/9] rust: pci: allow drivers to manage DMA ownership Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 5/9] rust: pci: add an error_detected callback Simon Song
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: Simon Song, alex, jgg, yishaih, skolothumtho, kevin.tian,
	airlied, simona, ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh,
	lossin, a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney,
	cjia, smitra, kjaju, alkumar, ankita, aniketa, kwankhede,
	targupta, linux-kernel, zhiwang, zhiw, boqun, daniel.almeida,
	tamird, work, jgg, bhelgaas, kwilczynski, dri-devel, peter

Because a VFIO PCI variant driver may need to update its state when its
device is reset, it needs notifications before and after the reset.
The NVIDIA VFIO variant driver resets a VF's active vGPU instance.

Add reset_prepare() and reset_done() as optional callbacks to
pci::Driver.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 6013c89bfb5e..8fcec16e6ccf 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -106,6 +106,9 @@ unsafe fn register(
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
             (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA.is_some();
+            if T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
+                (*pdrv.get()).err_handler = &Self::ERROR_HANDLERS;
+            }
             if T::HAS_SRIOV_ENABLE {
                 (*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
             }
@@ -178,6 +181,54 @@ extern "C" fn remove_callback(pdev: *mut bindings::pci_dev) {
         // before `devres_release_all()` in `device_unbind_cleanup()` is sufficient.
         drop(data);
     }
+
+    extern "C" fn reset_prepare_callback(pdev: *mut bindings::pci_dev) {
+        // SAFETY: The PCI core only ever calls the reset_prepare callback with a valid pointer to
+        // a `struct pci_dev`, while holding the device lock.
+        //
+        // INVARIANT: `pdev` is valid for the duration of `reset_prepare_callback()`.
+        let pdev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+
+        // SAFETY: The PCI core only calls this callback for a device bound to this driver. The
+        // device lock excludes `probe_callback()` and `remove_callback()`, so the stored pointer
+        // has type `T::Data<'_>` and remains valid throughout this callback.
+        let data = unsafe { pdev.as_ref().drvdata_borrow::<T::Data<'_>>() };
+
+        T::reset_prepare(pdev, data);
+    }
+
+    extern "C" fn reset_done_callback(pdev: *mut bindings::pci_dev) {
+        // SAFETY: The PCI core only ever calls the reset_done callback with a valid pointer to a
+        // `struct pci_dev`, while holding the device lock.
+        //
+        // INVARIANT: `pdev` is valid for the duration of `reset_done_callback()`.
+        let pdev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+
+        // SAFETY: The PCI core only calls this callback for a device bound to this driver. The
+        // device lock excludes `probe_callback()` and `remove_callback()`, so the stored pointer
+        // has type `T::Data<'_>` and remains valid throughout this callback.
+        let data = unsafe { pdev.as_ref().drvdata_borrow::<T::Data<'_>>() };
+
+        T::reset_done(pdev, data);
+    }
+
+    const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
+        error_detected: None,
+        mmio_enabled: None,
+        slot_reset: None,
+        reset_prepare: if T::HAS_RESET_PREPARE {
+            Some(Self::reset_prepare_callback)
+        } else {
+            None
+        },
+        reset_done: if T::HAS_RESET_DONE {
+            Some(Self::reset_done_callback)
+        } else {
+            None
+        },
+        resume: None,
+        cor_error_detected: None,
+    };
 }
 
 /// Declares a kernel module that exposes a single PCI driver.
@@ -402,6 +453,14 @@ fn unbind<'bound>(dev: &'bound Device<device::Core<'_>>, this: Pin<&Self::Data<'
         let _ = (dev, this);
     }
 
+    /// Called before a PCI function reset, with the device lock held.
+    #[optional]
+    fn reset_prepare<'bound>(_dev: &Device<device::Core<'_>>, _this: Pin<&Self::Data<'bound>>);
+
+    /// Called after a PCI function reset, with the device lock held.
+    #[optional]
+    fn reset_done<'bound>(_dev: &Device<device::Core<'_>>, _this: Pin<&Self::Data<'bound>>);
+
     /// Enables Single Root I/O Virtualization (SR-IOV) for a PF.
     ///
     /// This callback is only invoked when `CONFIG_PCI_IOV` is enabled.
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 5/9] rust: pci: add an error_detected callback
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (3 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 4/9] rust: pci: add reset callbacks Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 6/9] rust: vfio: add common VFIO device handling Simon Song
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: Simon Song, alex, jgg, yishaih, skolothumtho, kevin.tian,
	airlied, simona, ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh,
	lossin, a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney,
	cjia, smitra, kjaju, alkumar, ankita, aniketa, kwankhede,
	targupta, linux-kernel, zhiwang, zhiw, boqun, daniel.almeida,
	tamird, work, jgg, bhelgaas, kwilczynski, dri-devel, peter

VFIO PCI variant drivers will use error_detected() to notify userspace
of PCI errors through vfio-pci-core.

Adding error_detected() to pci::Driver lets VFIO variants reuse the PCI
adapter's error callback dispatch.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 89 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 87 insertions(+), 2 deletions(-)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 8fcec16e6ccf..27ac8d8435f7 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -106,7 +106,7 @@ unsafe fn register(
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
             (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA.is_some();
-            if T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
+            if T::HAS_ERROR_DETECTED || T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
                 (*pdrv.get()).err_handler = &Self::ERROR_HANDLERS;
             }
             if T::HAS_SRIOV_ENABLE {
@@ -212,8 +212,36 @@ extern "C" fn reset_done_callback(pdev: *mut bindings::pci_dev) {
         T::reset_done(pdev, data);
     }
 
+    extern "C" fn error_detected_callback(
+        pdev: *mut bindings::pci_dev,
+        state: bindings::pci_channel_state_t,
+    ) -> bindings::pci_ers_result_t {
+        // SAFETY: PCI error recovery only ever calls the error_detected callback with a valid
+        // pointer to a `struct pci_dev`. AER (`report_error_detected()`) and EEH hold the device
+        // lock while calling it.
+        //
+        // INVARIANT: `pdev` is valid for the duration of `error_detected_callback()`.
+        let pdev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+
+        // SAFETY: PCI error recovery only calls this callback for a device bound to this driver.
+        // The device lock excludes `probe_callback()` and `remove_callback()`, so the stored
+        // pointer has type `T::Data<'_>` and remains valid throughout this callback.
+        let data = unsafe { pdev.as_ref().drvdata_borrow::<T::Data<'_>>() };
+
+        let Some(state) = ChannelState::from_raw(state) else {
+            // PCI error recovery only reports the states `ChannelState` represents.
+            return ErsResult::None.as_raw();
+        };
+
+        T::error_detected(pdev, data, state).as_raw()
+    }
+
     const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
-        error_detected: None,
+        error_detected: if T::HAS_ERROR_DETECTED {
+            Some(Self::error_detected_callback)
+        } else {
+            None
+        },
         mmio_enabled: None,
         slot_reset: None,
         reset_prepare: if T::HAS_RESET_PREPARE {
@@ -363,6 +391,55 @@ impl<D: ?Sized> DriverManagedDma<D> {
     }
 }
 
+/// The state of the I/O channel to a device, as reported by PCI error recovery.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ChannelState {
+    /// The I/O channel remains operational.
+    Normal,
+    /// I/O to the device is blocked.
+    Frozen,
+    /// The device has permanently failed.
+    PermanentFailure,
+}
+
+impl ChannelState {
+    fn from_raw(state: bindings::pci_channel_state_t) -> Option<Self> {
+        match state {
+            bindings::pci_channel_io_normal => Some(Self::Normal),
+            bindings::pci_channel_io_frozen => Some(Self::Frozen),
+            bindings::pci_channel_io_perm_failure => Some(Self::PermanentFailure),
+            _ => None,
+        }
+    }
+}
+
+/// The result a driver reports to PCI error recovery.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ErsResult {
+    /// The driver has no result to report.
+    None,
+    /// The driver can recover without a slot reset.
+    CanRecover,
+    /// The driver needs the slot to be reset.
+    NeedReset,
+    /// The device has failed completely and cannot be recovered.
+    Disconnect,
+    /// The driver has fully recovered and the device is operational.
+    Recovered,
+}
+
+impl ErsResult {
+    fn as_raw(self) -> bindings::pci_ers_result_t {
+        match self {
+            Self::None => bindings::pci_ers_result_PCI_ERS_RESULT_NONE,
+            Self::CanRecover => bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER,
+            Self::NeedReset => bindings::pci_ers_result_PCI_ERS_RESULT_NEED_RESET,
+            Self::Disconnect => bindings::pci_ers_result_PCI_ERS_RESULT_DISCONNECT,
+            Self::Recovered => bindings::pci_ers_result_PCI_ERS_RESULT_RECOVERED,
+        }
+    }
+}
+
 /// `IdTable` type for PCI.
 pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
 
@@ -461,6 +538,14 @@ fn unbind<'bound>(dev: &'bound Device<device::Core<'_>>, this: Pin<&Self::Data<'
     #[optional]
     fn reset_done<'bound>(_dev: &Device<device::Core<'_>>, _this: Pin<&Self::Data<'bound>>);
 
+    /// Handle a PCI error and return the PCI recovery result, with the device lock held.
+    #[optional]
+    fn error_detected<'bound>(
+        _dev: &Device<device::Core<'_>>,
+        _this: Pin<&Self::Data<'bound>>,
+        _state: ChannelState,
+    ) -> ErsResult;
+
     /// Enables Single Root I/O Virtualization (SR-IOV) for a PF.
     ///
     /// This callback is only invoked when `CONFIG_PCI_IOV` is enabled.
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 6/9] rust: vfio: add common VFIO device handling
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (4 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 5/9] rust: pci: add an error_detected callback Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 7/9] rust: vfio: add PCI variant driver abstractions Simon Song
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: Zhi Wang, alex, jgg, yishaih, skolothumtho, kevin.tian, airlied,
	simona, ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, boqun, daniel.almeida, tamird, work, jgg,
	bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Zhi Wang <zhiw@nvidia.com>

A VFIO PCI device contains a VFIO device. Expose that relationship in
Rust so common device handling is independent of PCI resources.

Add vfio::DeviceBase, a C VFIO device structure followed by pointers to
callback data, and vfio::Device, its alias for struct vfio_device. The
data is registration data and data kept from the first successful open
until the last close, which can borrow the former. Share reference
counting and callback-data lifetime handling in vfio.

The helpers are unused until the next patch adds the PCI abstraction.

Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/bindings/bindings_helper.h |   2 +
 rust/kernel/lib.rs              |   2 +
 rust/kernel/vfio.rs             | 430 ++++++++++++++++++++++++++++++++
 3 files changed, 434 insertions(+)
 create mode 100644 rust/kernel/vfio.rs

diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 3791af0aadc7..a5781bc1ea85 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -93,6 +93,7 @@
 #include <linux/task_work.h>
 #include <linux/tracepoint.h>
 #include <linux/usb.h>
+#include <linux/vfio.h>
 #include <linux/wait.h>
 #include <linux/workqueue.h>
 #include <linux/xarray.h>
@@ -115,6 +116,7 @@
 const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
 const size_t RUST_CONST_HELPER_ARCH_KMALLOC_MINALIGN = ARCH_KMALLOC_MINALIGN;
 const size_t RUST_CONST_HELPER_PAGE_SIZE = PAGE_SIZE;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_RESET = VFIO_DEVICE_RESET;
 const size_t RUST_CONST_HELPER_GENLMSG_DEFAULT_SIZE = GENLMSG_DEFAULT_SIZE;
 const gfp_t RUST_CONST_HELPER_GFP_ATOMIC = GFP_ATOMIC;
 const gfp_t RUST_CONST_HELPER_GFP_KERNEL = GFP_KERNEL;
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index 1ab41129d557..10a8e7de0d9c 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -147,6 +147,8 @@
 pub mod uaccess;
 #[cfg(CONFIG_USB = "y")]
 pub mod usb;
+#[cfg(CONFIG_VFIO)]
+pub mod vfio;
 pub mod workqueue;
 pub mod xarray;
 
diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
new file mode 100644
index 000000000000..bc50647a2437
--- /dev/null
+++ b/rust/kernel/vfio.rs
@@ -0,0 +1,430 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Virtual Function I/O (VFIO) devices and callback data.
+//!
+//! [`DeviceBase`] combines a C VFIO device structure with the data its callbacks use.
+//! [`Device`] is an alias of it for `struct vfio_device`.
+//!
+//! C header: [`include/linux/vfio.h`](srctree/include/linux/vfio.h)
+
+use crate::{
+    bindings,
+    prelude::*,
+    sync::aref::AlwaysRefCounted,
+    types::{
+        NotThreadSafe,
+        Opaque, //
+    },
+    uaccess::{
+        UserPtr,
+        UserSlice, //
+    }, //
+};
+use core::{
+    cell::UnsafeCell,
+    marker::PhantomData,
+    ptr::NonNull, //
+};
+
+/// Reset the VFIO device.
+pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
+
+/// The context in which a VFIO device reference is valid.
+///
+/// Callback views can only be borrowed from the corresponding VFIO trampoline.
+/// They cannot be refcounted or shared across threads.
+pub trait DeviceContext: private::Sealed {}
+
+/// A callback context in which VFIO keeps the device open.
+///
+/// Registration data and open data are available throughout such a callback.
+pub trait IoContext: DeviceContext {}
+
+/// An ordinary device reference, without callback-specific operations.
+pub struct Normal;
+
+/// The device is undergoing its exclusive first-open callback.
+pub struct Open;
+
+/// The device is borrowed for a VFIO ioctl callback on the current thread.
+pub struct Ioctl;
+
+/// The device is borrowed for a VFIO read callback on the current thread.
+pub struct Read;
+
+/// The device is borrowed for a VFIO write callback on the current thread.
+pub struct Write;
+
+/// The device is borrowed for a VFIO mmap callback on the current thread.
+pub struct Mmap;
+
+/// The device is borrowed for a VFIO region-info callback on the current thread.
+pub struct GetRegionInfo;
+
+mod private {
+    pub trait Sealed {}
+
+    impl Sealed for super::Normal {}
+    impl Sealed for super::Open {}
+    impl Sealed for super::Write {}
+    impl Sealed for super::Mmap {}
+    impl Sealed for super::Ioctl {}
+    impl Sealed for super::Read {}
+    impl Sealed for super::GetRegionInfo {}
+}
+
+impl DeviceContext for Normal {}
+impl DeviceContext for Open {}
+impl DeviceContext for Write {}
+impl DeviceContext for Mmap {}
+impl DeviceContext for Ioctl {}
+impl DeviceContext for Read {}
+impl DeviceContext for GetRegionInfo {}
+
+impl IoContext for Write {}
+impl IoContext for Mmap {}
+impl IoContext for Ioctl {}
+impl IoContext for Read {}
+impl IoContext for GetRegionInfo {}
+
+/// The data the driver of a [`Device`] shares with its callbacks.
+///
+/// Bus-specific devices take the data types from their own trait.
+pub trait Operations: Sized + Send + Sync + 'static {
+    /// Data owned by the registration and passed to VFIO callbacks.
+    ///
+    /// It can borrow resources of the driver's binding for `'a`. Per-device state lives here.
+    type RegistrationData<'a>: Send + Sync + 'a;
+
+    /// Data shared by callbacks from the first successful open until the last close.
+    ///
+    /// It can borrow the registration data, of lifetime `'data`, for `'a`.
+    type OpenData<'a, 'data: 'a>: Send + Sync + 'a;
+}
+
+/// Maps a C device structure to its embedded VFIO device.
+///
+/// # Safety
+///
+/// Implementations must return the address of the `vfio_device` at the beginning of `Self`,
+/// preserving the input pointer's provenance.
+pub unsafe trait RawDevice {
+    /// Returns the `vfio_device` embedded in `raw`.
+    ///
+    /// # Safety
+    ///
+    /// `raw` must point to an initialized, live `Self`.
+    unsafe fn raw_vfio_device(raw: *mut Self) -> *mut bindings::vfio_device;
+}
+
+// SAFETY: A `vfio_device` is its own VFIO device.
+unsafe impl RawDevice for bindings::vfio_device {
+    unsafe fn raw_vfio_device(raw: *mut Self) -> *mut bindings::vfio_device {
+        raw
+    }
+}
+
+/// Callback data stored after the C device structure.
+///
+/// `R` is the type of the registration data and `O` the type of the open data, both written
+/// with the placeholder lifetime `'static`, because the device has no lifetime to name.
+///
+/// # Invariants
+///
+/// `reg_data` is `None` until a registration publishes its data, and is not changed again. The
+/// data is pinned and remains valid while the device is registered. After unregistration or a
+/// failed registration attempt, the pointer may dangle; its pointee must never again be
+/// accessed through this device.
+/// `open_data` is null outside a successful first open, otherwise it owns a pinned allocation
+/// until the last close. The bus-specific layer drains callbacks before either is released.
+struct CallbackData<R, O> {
+    reg_data: UnsafeCell<Option<NonNull<R>>>,
+    open_data: UnsafeCell<*mut O>,
+}
+
+/// A C VFIO device structure `Raw`, followed by the callback data of the driver `T`.
+///
+/// `raw` is the first field, so the `vfio_device` at the beginning of `Raw` is also at the
+/// beginning of `Self`, as `vfio_alloc_device()` requires.
+///
+/// `R` and `O` are the types of the registration data and the open data of `T`, written with
+/// the placeholder lifetime `'static`. Use the alias [`Device`], which takes both from the
+/// operations trait of `T`.
+///
+/// # Invariants
+///
+/// - `R` and `O` are the registration data and open data types of `T` with all lifetimes
+///   replaced by `'static`. The data stored is only valid as described for `CallbackData`.
+/// - `raw` is initialized and remains alive while borrowed. The reference count of its embedded
+///   `vfio_device` owns the allocation.
+/// - The device is registered with VFIO only after its registration data has been published.
+/// - Callback contexts are borrowed only for the corresponding callback and thread; only
+///   [`Normal`] references can be reference-counted.
+#[repr(C)]
+pub struct DeviceBase<Raw, R, O, T, Ctx: DeviceContext = Normal> {
+    raw: Opaque<Raw>,
+    data: CallbackData<R, O>,
+    _context: PhantomData<(T, Ctx, NotThreadSafe)>,
+}
+
+/// A VFIO device, wrapping `struct vfio_device`.
+pub type Device<T, Ctx = Normal> = DeviceBase<
+    bindings::vfio_device,
+    <T as Operations>::RegistrationData<'static>,
+    <T as Operations>::OpenData<'static, 'static>,
+    T,
+    Ctx,
+>;
+
+#[expect(dead_code)]
+impl<Raw: RawDevice, R, O, T, Ctx: DeviceContext> DeviceBase<Raw, R, O, T, Ctx> {
+    /// Initializes the Rust fields of a newly allocated device.
+    ///
+    /// # Safety
+    ///
+    /// `this` must point to an allocation with the size and alignment of `Self`, whose `raw`
+    /// has been initialized. Nothing else may access the allocation during this call.
+    unsafe fn init(this: *mut Self) {
+        // SAFETY: The caller provides exclusive access to a suitable allocation.
+        unsafe {
+            (&raw mut (*this).data).write(CallbackData {
+                reg_data: UnsafeCell::new(None),
+                open_data: UnsafeCell::new(core::ptr::null_mut()),
+            });
+            (&raw mut (*this)._context).write(PhantomData);
+        }
+    }
+
+    /// Recovers `&Self` from the `vfio_device` VFIO passes to a callback.
+    ///
+    /// # Safety
+    ///
+    /// `vdev` must point to the `vfio_device` at the beginning of a `Self` that remains alive
+    /// for `'a`. The caller must provide the guarantees of `Ctx` throughout the borrow.
+    unsafe fn from_raw_vfio<'a>(vdev: *mut bindings::vfio_device) -> &'a Self {
+        // SAFETY: `Raw` begins with its `vfio_device` and is the first field of `Self`. The
+        // caller guarantees validity and the context.
+        unsafe { &*vdev.cast() }
+    }
+
+    fn as_raw(&self) -> *mut Raw {
+        self.raw.get()
+    }
+
+    fn as_raw_vfio(&self) -> *mut bindings::vfio_device {
+        // SAFETY: By the type invariants, `raw` is initialized and alive.
+        unsafe { Raw::raw_vfio_device(self.as_raw()) }
+    }
+}
+
+#[expect(dead_code)]
+impl<Raw: RawDevice, R, O, T> DeviceBase<Raw, R, O, T> {
+    /// Publishes the registration data passed to callbacks.
+    ///
+    /// Returns [`EBUSY`] if registration data has been published before, so that a device can
+    /// be registered at most once.
+    ///
+    /// # Safety
+    ///
+    /// The caller must serialize calls for this device. `data` must point to pinned data that
+    /// remains valid until the device has been unregistered or, if registration fails, is not
+    /// accessed through this device.
+    unsafe fn publish_registration_data(&self, data: NonNull<R>) -> Result {
+        // SAFETY: The caller serializes publication, and callbacks only read the slot.
+        if unsafe { (*self.data.reg_data.get()).is_some() } {
+            return Err(EBUSY);
+        }
+        // SAFETY: The slot is unpublished, so the device has not been registered and no
+        // callback can read the slot.
+        // INVARIANT: The slot is written once; the caller keeps `data` valid.
+        unsafe { *self.data.reg_data.get() = Some(data) };
+        Ok(())
+    }
+
+    /// Destroys the open data.
+    ///
+    /// # Safety
+    ///
+    /// Call once after a successful [`DeviceBase::open`], with all I/O callbacks drained and
+    /// the next first open excluded. Registration data and the resources the open data borrows
+    /// must remain alive.
+    unsafe fn close(&self) {
+        // SAFETY: The last close exclusively takes the allocation published by `open()`.
+        let raw =
+            unsafe { core::mem::replace(&mut *self.data.open_data.get(), core::ptr::null_mut()) };
+        // SAFETY: The allocation came from `KBox::into_raw()` and is destroyed without moving
+        // it. Lifetimes do not affect its layout.
+        unsafe { drop(KBox::from_raw(raw)) };
+    }
+}
+
+#[expect(dead_code)]
+impl<Raw: RawDevice, R, O, T> DeviceBase<Raw, R, O, T, Open> {
+    /// Creates the open data from the registration data.
+    ///
+    /// The [`Open`] context guarantees an exclusive first open of a registered device, whose
+    /// registration data remains valid until the last close.
+    ///
+    /// # Safety
+    ///
+    /// - No open data may exist.
+    /// - `OD` must be `O`, except for lifetimes.
+    /// - `R` names the registration data with the placeholder lifetime `'static`, and the
+    ///   caller chooses the lifetimes of `OD`; this function cannot tie either to the time the
+    ///   data is valid. `init` must therefore use the registration data parametrically, as a
+    ///   function generic over its lifetimes does: nothing borrowed from it may escape, other
+    ///   than into the returned open data.
+    /// - `init` must not capture references that are only valid for the callback and store
+    ///   them in the open data, which outlives the callback.
+    /// - The open data must be destroyed with [`DeviceBase::close`] at the last close.
+    unsafe fn open<'a, OD, I>(&'a self, init: impl FnOnce(&'a R) -> I) -> Result
+    where
+        I: PinInit<OD, Error>,
+    {
+        // SAFETY: By the type invariants, an `Open` reference only exists for a registered
+        // device, which has published registration data that nothing changes anymore. The
+        // data remains valid until the last close.
+        let reg_data = unsafe { (*self.data.reg_data.get()).unwrap_unchecked().as_ref() };
+
+        // Allocate before calling the driver so allocation failure cannot follow open.
+        let data = KBox::<OD>::new_uninit(GFP_KERNEL)?;
+        let data = data.write_pin_init(init(reg_data))?;
+
+        // SAFETY: Only the owning pointer moves; `close()` drops the allocation in place.
+        let raw = KBox::into_raw(unsafe { Pin::into_inner_unchecked(data) });
+        // SAFETY: The first open is exclusive, and no I/O callback runs before it returns.
+        // The caller guarantees that `OD` and `O` only differ in lifetimes, which do not
+        // affect the layout.
+        // INVARIANT: The allocation is pinned and owned by the slot until `close()`.
+        unsafe { *self.data.open_data.get() = raw.cast::<O>() };
+        Ok(())
+    }
+}
+
+#[expect(dead_code)]
+impl<Raw: RawDevice, R, O, T, Ctx: IoContext> DeviceBase<Raw, R, O, T, Ctx> {
+    /// Returns the registration data and the open data, for an I/O callback.
+    ///
+    /// # Safety
+    ///
+    /// `R` and `O` name the data with the placeholder lifetime `'static`, which is not the time
+    /// the data is valid. The caller must use both parametrically, as a function generic over
+    /// their lifetimes does: nothing borrowed from them may outlive the callback, and nothing
+    /// borrowed for the callback may be stored in them.
+    unsafe fn callback_data(&self) -> (&R, Pin<&O>) {
+        // SAFETY: By the type invariants, an `IoContext` reference only exists during an I/O
+        // callback on an open, registered device. VFIO excludes the last close and
+        // unregistration for its duration, so by the type invariants of `CallbackData` both
+        // slots hold valid data that nothing changes, and the open data is pinned.
+        unsafe {
+            let reg_data = (*self.data.reg_data.get()).unwrap_unchecked();
+            let open_data = *self.data.open_data.get();
+            (reg_data.as_ref(), Pin::new_unchecked(&*open_data))
+        }
+    }
+}
+
+// SAFETY: VFIO uses the embedded class device's refcount to own the full allocation.
+unsafe impl<Raw: RawDevice, R, O, T> AlwaysRefCounted for DeviceBase<Raw, R, O, T> {
+    fn inc_ref(&self) {
+        // SAFETY: A shared reference keeps the initialized VFIO device alive.
+        unsafe { bindings::get_device(&raw mut (*self.as_raw_vfio()).device) };
+    }
+
+    unsafe fn dec_ref(obj: NonNull<Self>) {
+        // SAFETY: The caller owns a reference, so `obj` points to a live device.
+        let raw = Opaque::cast_into(unsafe { &raw const (*obj.as_ptr()).raw });
+        // SAFETY: `raw` is initialized and alive, as above.
+        let vdev = unsafe { Raw::raw_vfio_device(raw) };
+        // SAFETY: The caller owns the VFIO reference being released.
+        unsafe { bindings::put_device(&raw mut (*vdev).device) };
+    }
+}
+
+// SAFETY: VFIO serializes open/close transitions, and callback data is `Send + Sync`.
+unsafe impl<Raw: RawDevice, R: Send + Sync, O: Send + Sync, T> Send for DeviceBase<Raw, R, O, T> {}
+
+// SAFETY: Shared access only changes the embedded class device's reference count. I/O callbacks
+// share `Sync` data; VFIO excludes them from mutations of `open_data` during first open and
+// last close, and registration data follows the registration lifetime. Mutable C state is
+// managed by the VFIO core.
+unsafe impl<Raw: RawDevice, R: Send + Sync, O: Send + Sync, T> Sync for DeviceBase<Raw, R, O, T> {}
+
+/// Capability buffer supplied by VFIO for a region-info callback.
+pub struct InfoCap<'a> {
+    #[expect(dead_code)]
+    raw: &'a mut bindings::vfio_info_cap,
+}
+
+/// Opaque wrapper around a `char __user *` buffer from a VFIO read or write callback.
+///
+/// Provides bounds-checked copying of kernel data into the user-space buffer.
+///
+/// This type cannot be constructed by driver code — it is only created by the
+/// callback trampoline.
+pub struct UserBuf {
+    ptr: UserPtr,
+    count: usize,
+}
+
+impl UserBuf {
+    /// Returns the buffer length in bytes.
+    pub fn len(&self) -> usize {
+        self.count
+    }
+
+    /// Returns `true` when the buffer is empty.
+    pub fn is_empty(&self) -> bool {
+        self.count == 0
+    }
+
+    /// Limit this callback's transfer to at most `count` bytes.
+    pub fn truncate(&mut self, count: usize) {
+        self.count = self.count.min(count);
+    }
+
+    /// Overwrite the part of `data` that overlaps a completed region read.
+    ///
+    /// `read_offset` is the starting region offset and `read_len` is the number of
+    /// bytes returned by the read. `data_offset` locates `data` within the same region.
+    pub fn write_overlapping(
+        &self,
+        read_offset: u64,
+        read_len: usize,
+        data_offset: u64,
+        data: &[u8],
+    ) -> Result {
+        if read_len > self.len() {
+            return Err(EINVAL);
+        }
+        let read_end = read_offset.checked_add(read_len as u64).ok_or(EOVERFLOW)?;
+        let data_end = data_offset
+            .checked_add(data.len() as u64)
+            .ok_or(EOVERFLOW)?;
+        let start = read_offset.max(data_offset);
+        let end = read_end.min(data_end);
+        if start >= end {
+            return Ok(());
+        }
+
+        // These differences are bounded by `read_len` and `data.len()`.
+        let buf_offset = (start - read_offset) as usize;
+        let data_start = (start - data_offset) as usize;
+        let data_end = (end - data_offset) as usize;
+        self.write_at(buf_offset, &data[data_start..data_end])
+    }
+
+    /// Copy `data` to the user buffer at byte offset `offset`.
+    ///
+    /// Returns [`EFAULT`] if the copy fails, [`EINVAL`] if the write would
+    /// exceed the buffer bounds.
+    fn write_at(&self, offset: usize, data: &[u8]) -> Result {
+        let end = offset.checked_add(data.len()).ok_or(EOVERFLOW)?;
+        if end > self.count {
+            return Err(EINVAL);
+        }
+        let dest = self.ptr.wrapping_byte_add(offset);
+        let mut writer = UserSlice::new(dest, data.len()).writer();
+        writer.write_slice(data)
+    }
+}
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 7/9] rust: vfio: add PCI variant driver abstractions
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (5 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 6/9] rust: vfio: add common VFIO device handling Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 8/9] gpu: nova-core: publish typed SR-IOV PF APIs Simon Song
  2026-11-19 21:50 ` [PATCH v3 9/9] vfio/nova-vgpu: add the Rust VFIO variant driver Simon Song
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
	ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, zhiw, boqun, daniel.almeida, tamird, work,
	jgg, bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Danilo Krummrich <dakr@kernel.org>

Add Rust abstractions for PCI drivers that extend vfio-pci-core with
device-specific operations.

Provide Operations, Device and Registration with separate registration
and first-open/last-close data. Allow open data to borrow registration
data, undo PCI enablement if open fails, and drop open data before
closing the core device.

Let a driver register its VFIO device with Registration::new(), for
example from its PCI probe callback, and unregister it by dropping the
Registration. Let drivers forward PCI reset and error callbacks through
the Registration. Let vfio-pci-core select its VGA arbitration
callbacks. Because runtime PM support follows in a separate patch
series, keep the device active while it is registered.

Restrict core I/O helpers to their corresponding callback contexts. Add
region mapping helpers, and wire up VFIO core support for device
features, matching, IOMMUFD and PASID. Allow passthrough devices to
export physical BARs as DMA buffers.

Add pci::ErsResult::from_raw() for the result of the VFIO PCI core's
error handler.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/bindings/bindings_helper.h |   2 +
 rust/kernel/pci.rs              |  12 +
 rust/kernel/vfio.rs             |  21 +-
 rust/kernel/vfio/pci.rs         | 777 ++++++++++++++++++++++++++++++++
 4 files changed, 803 insertions(+), 9 deletions(-)
 create mode 100644 rust/kernel/vfio/pci.rs

diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index a5781bc1ea85..b052f1e874e2 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -94,6 +94,7 @@
 #include <linux/tracepoint.h>
 #include <linux/usb.h>
 #include <linux/vfio.h>
+#include <linux/vfio_pci_core.h>
 #include <linux/wait.h>
 #include <linux/workqueue.h>
 #include <linux/xarray.h>
@@ -117,6 +118,7 @@ const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
 const size_t RUST_CONST_HELPER_ARCH_KMALLOC_MINALIGN = ARCH_KMALLOC_MINALIGN;
 const size_t RUST_CONST_HELPER_PAGE_SIZE = PAGE_SIZE;
 const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_RESET = VFIO_DEVICE_RESET;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_PCI_HOT_RESET = VFIO_DEVICE_PCI_HOT_RESET;
 const size_t RUST_CONST_HELPER_GENLMSG_DEFAULT_SIZE = GENLMSG_DEFAULT_SIZE;
 const gfp_t RUST_CONST_HELPER_GFP_ATOMIC = GFP_ATOMIC;
 const gfp_t RUST_CONST_HELPER_GFP_KERNEL = GFP_KERNEL;
diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 27ac8d8435f7..8a3fdcd1d059 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -429,6 +429,18 @@ pub enum ErsResult {
 }
 
 impl ErsResult {
+    #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
+    pub(crate) fn from_raw(result: bindings::pci_ers_result_t) -> Option<Self> {
+        match result {
+            bindings::pci_ers_result_PCI_ERS_RESULT_NONE => Some(Self::None),
+            bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER => Some(Self::CanRecover),
+            bindings::pci_ers_result_PCI_ERS_RESULT_NEED_RESET => Some(Self::NeedReset),
+            bindings::pci_ers_result_PCI_ERS_RESULT_DISCONNECT => Some(Self::Disconnect),
+            bindings::pci_ers_result_PCI_ERS_RESULT_RECOVERED => Some(Self::Recovered),
+            _ => None,
+        }
+    }
+
     fn as_raw(self) -> bindings::pci_ers_result_t {
         match self {
             Self::None => bindings::pci_ers_result_PCI_ERS_RESULT_NONE,
diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
index bc50647a2437..43e904e2c980 100644
--- a/rust/kernel/vfio.rs
+++ b/rust/kernel/vfio.rs
@@ -3,7 +3,7 @@
 //! Virtual Function I/O (VFIO) devices and callback data.
 //!
 //! [`DeviceBase`] combines a C VFIO device structure with the data its callbacks use.
-//! [`Device`] is an alias of it for `struct vfio_device`.
+//! [`Device`] and the bus-specific devices, such as `pci::Device`, are aliases of it.
 //!
 //! C header: [`include/linux/vfio.h`](srctree/include/linux/vfio.h)
 
@@ -26,6 +26,9 @@
     ptr::NonNull, //
 };
 
+#[cfg(CONFIG_VFIO_PCI_CORE)]
+pub mod pci;
+
 /// Reset the VFIO device.
 pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
 
@@ -89,7 +92,7 @@ impl IoContext for GetRegionInfo {}
 
 /// The data the driver of a [`Device`] shares with its callbacks.
 ///
-/// Bus-specific devices take the data types from their own trait.
+/// Bus-specific devices take the data types from their own trait, such as `pci::Operations`.
 pub trait Operations: Sized + Send + Sync + 'static {
     /// Data owned by the registration and passed to VFIO callbacks.
     ///
@@ -148,8 +151,8 @@ struct CallbackData<R, O> {
 /// beginning of `Self`, as `vfio_alloc_device()` requires.
 ///
 /// `R` and `O` are the types of the registration data and the open data of `T`, written with
-/// the placeholder lifetime `'static`. Use the alias [`Device`], which takes both from the
-/// operations trait of `T`.
+/// the placeholder lifetime `'static`. Use the aliases [`Device`] and `pci::Device`, which
+/// take both from the operations trait of `T`.
 ///
 /// # Invariants
 ///
@@ -176,7 +179,7 @@ pub struct DeviceBase<Raw, R, O, T, Ctx: DeviceContext = Normal> {
     Ctx,
 >;
 
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
 impl<Raw: RawDevice, R, O, T, Ctx: DeviceContext> DeviceBase<Raw, R, O, T, Ctx> {
     /// Initializes the Rust fields of a newly allocated device.
     ///
@@ -217,7 +220,7 @@ fn as_raw_vfio(&self) -> *mut bindings::vfio_device {
     }
 }
 
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
 impl<Raw: RawDevice, R, O, T> DeviceBase<Raw, R, O, T> {
     /// Publishes the registration data passed to callbacks.
     ///
@@ -258,7 +261,7 @@ unsafe fn close(&self) {
     }
 }
 
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
 impl<Raw: RawDevice, R, O, T> DeviceBase<Raw, R, O, T, Open> {
     /// Creates the open data from the registration data.
     ///
@@ -301,7 +304,7 @@ unsafe fn open<'a, OD, I>(&'a self, init: impl FnOnce(&'a R) -> I) -> Result
     }
 }
 
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
 impl<Raw: RawDevice, R, O, T, Ctx: IoContext> DeviceBase<Raw, R, O, T, Ctx> {
     /// Returns the registration data and the open data, for an I/O callback.
     ///
@@ -352,7 +355,7 @@ unsafe impl<Raw: RawDevice, R: Send + Sync, O: Send + Sync, T> Sync for DeviceBa
 
 /// Capability buffer supplied by VFIO for a region-info callback.
 pub struct InfoCap<'a> {
-    #[expect(dead_code)]
+    #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
     raw: &'a mut bindings::vfio_info_cap,
 }
 
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
new file mode 100644
index 000000000000..bde5ed139e5f
--- /dev/null
+++ b/rust/kernel/vfio/pci.rs
@@ -0,0 +1,777 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! VFIO PCI variant driver abstractions.
+//!
+//! Provides Rust abstractions for writing VFIO PCI variant drivers (also known
+//! as "VFIO PCI core" drivers). A [`Registration`] struct registers the VFIO
+//! device and owns its registration data, tying resource lifetimes to the PCI
+//! driver's binding scope.
+//!
+//! C header: [`include/linux/vfio_pci_core.h`](srctree/include/linux/vfio_pci_core.h)
+
+use super::{
+    DeviceBase,
+    InfoCap,
+    RawDevice,
+    UserBuf, //
+};
+pub use super::{
+    DeviceContext,
+    GetRegionInfo,
+    IoContext,
+    Ioctl,
+    Mmap,
+    Normal,
+    Open,
+    Read,
+    Write, //
+};
+
+use crate::{
+    alloc::allocator::Kmalloc,
+    bindings,
+    device,
+    error::{
+        from_err_ptr,
+        to_result, //
+    },
+    mm::virt::VmaRef,
+    pci,
+    prelude::*,
+    sync::aref::ARef,
+    types::NotThreadSafe,
+    uaccess::UserPtr, //
+};
+use core::{
+    alloc::Layout,
+    ptr::NonNull, //
+};
+
+/// Reset the PCI slot or bus containing a VFIO device.
+pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+
+/// Index of the PCI configuration-space region.
+pub const CONFIG_REGION_INDEX: u32 = bindings::VFIO_PCI_CONFIG_REGION_INDEX;
+
+/// The trait for VFIO PCI variant driver implementations.
+///
+/// Unoverridden callbacks delegate to the corresponding PCI core operation.
+///
+/// # Lifetime of the callback data
+///
+/// [`Registration`] owns the [`RegistrationData`](Operations::RegistrationData), which
+/// borrows from the PCI binding scope, and each callback borrows it. Callbacks are generic over
+/// the lifetime of the registration data, so they cannot store callback-local references in it.
+pub trait Operations: Sized + Send + Sync + 'static {
+    /// The name of the VFIO driver.
+    const NAME: &'static CStr;
+
+    /// Data owned by the [`Registration`] and passed to VFIO callbacks.
+    ///
+    /// It can borrow resources of the driver's binding for `'a`. Per-device state lives here.
+    type RegistrationData<'a>: Send + Sync + 'a;
+
+    /// Data shared by callbacks from the first successful open until the last close.
+    ///
+    /// It can borrow the registration data, of lifetime `'data`, for `'a`.
+    type OpenData<'a, 'data: 'a>: Send + Sync + 'a;
+
+    /// Called when the first file descriptor is opened for this device.
+    ///
+    /// `vfio_pci_core_enable()` has already succeeded; if this returns an
+    /// error, `vfio_pci_core_disable()` is called automatically. The returned
+    /// open data is dropped before PCI core close on the last close.
+    fn open_device<'a, 'data: 'a>(
+        dev: &'a Device<Self, Open>,
+        reg_data: &'a Self::RegistrationData<'data>,
+    ) -> impl PinInit<Self::OpenData<'a, 'data>, Error> + 'a;
+
+    /// Handle a VFIO ioctl.
+    ///
+    /// The default `vfio_pci_core_ioctl()` is available via
+    /// [`Device::core_ioctl()`] for delegation.
+    fn ioctl<'data>(
+        dev: &Device<Self, Ioctl>,
+        reg_data: &Self::RegistrationData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        cmd: u32,
+        arg: usize,
+    ) -> Result<isize>;
+
+    /// Read from the device.
+    ///
+    /// `buf` is a user-space buffer provided by the VFIO core. Use
+    /// [`Device::core_read()`] to delegate the default read, and
+    /// [`UserBuf::write_overlapping()`] to override specific bytes afterwards.
+    fn read<'data>(
+        dev: &Device<Self, Read>,
+        reg_data: &Self::RegistrationData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        buf: &mut UserBuf,
+        ppos: &mut Position<'_>,
+    ) -> Result<isize>;
+
+    /// Write to the device, delegating to [`Device::core_write()`] if appropriate.
+    fn write<'data>(
+        dev: &Device<Self, Write>,
+        _reg_data: &Self::RegistrationData<'data>,
+        _open_data: Pin<&Self::OpenData<'_, 'data>>,
+        buf: &mut UserBuf,
+        ppos: &mut Position<'_>,
+    ) -> Result<isize> {
+        dev.core_write(buf, ppos)
+    }
+
+    /// Map a device region, delegating to [`Device::core_mmap()`] if appropriate.
+    fn mmap<'data>(
+        dev: &Device<Self, Mmap>,
+        _reg_data: &Self::RegistrationData<'data>,
+        _open_data: Pin<&Self::OpenData<'_, 'data>>,
+        mapping: &mut Mapping<'_>,
+    ) -> Result {
+        dev.core_mmap(mapping)
+    }
+
+    /// Prepare for PCI reset, including resets initiated during enable or close.
+    fn reset_prepare(_reg_data: &Self::RegistrationData<'_>) {}
+
+    /// Finish PCI reset; errors are reported through the VFIO error interrupt.
+    fn reset_done(_reg_data: &Self::RegistrationData<'_>) -> Result {
+        Ok(())
+    }
+
+    /// Fill in region info capabilities.
+    ///
+    /// The default `vfio_pci_ioctl_get_region_info()` is available via
+    /// [`Device::core_get_region_info()`] for delegation.
+    fn get_region_info<'data>(
+        dev: &Device<Self, GetRegionInfo>,
+        reg_data: &Self::RegistrationData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut InfoCap<'_>,
+    ) -> Result;
+}
+
+/// A VFIO PCI file position, encoding a region index and an offset within that region.
+pub struct Position<'a> {
+    raw: &'a mut i64,
+}
+
+impl Position<'_> {
+    /// Commit a temporary position only after the operation succeeds.
+    ///
+    /// This preserves the original position if a read succeeds but a subsequent
+    /// user copy fails.
+    pub fn with_temporary<R>(
+        &mut self,
+        operation: impl FnOnce(&mut Position<'_>) -> Result<R>,
+    ) -> Result<R> {
+        let mut next = *self.raw;
+        let result = operation(&mut Position { raw: &mut next })?;
+        *self.raw = next;
+        Ok(result)
+    }
+
+    /// Returns the PCI region index.
+    pub fn region_index(&self) -> u32 {
+        ((*self.raw as u64) >> bindings::VFIO_PCI_OFFSET_SHIFT) as u32
+    }
+
+    /// Returns the byte offset within the current region.
+    pub fn region_offset(&self) -> u64 {
+        (*self.raw as u64) & ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1)
+    }
+}
+
+/// Mapping metadata supplied by VFIO while the current thread holds the mmap lock.
+///
+/// The underlying VMA can only be passed back to the core mmap operation.
+pub struct Mapping<'a> {
+    vma: &'a VmaRef,
+    _not_thread_safe: NotThreadSafe,
+}
+
+impl Mapping<'_> {
+    fn page_offset(&self) -> u64 {
+        // SAFETY: The mmap callback holds the mmap lock for this VMA.
+        unsafe { (*self.vma.as_ptr()).vm_pgoff as u64 }
+    }
+
+    /// Returns the PCI region index encoded in the mapping offset.
+    pub fn region_index(&self) -> u32 {
+        (self.page_offset() >> (bindings::VFIO_PCI_OFFSET_SHIFT as usize - crate::page::PAGE_SHIFT))
+            as u32
+    }
+
+    /// Returns the byte offset within the PCI region.
+    fn region_offset(&self) -> u64 {
+        let mask = ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1) >> crate::page::PAGE_SHIFT;
+        (self.page_offset() & mask) << crate::page::PAGE_SHIFT
+    }
+
+    /// Returns the requested mapping length in bytes.
+    fn size(&self) -> Result<u64> {
+        self.vma
+            .end()
+            .checked_sub(self.vma.start())
+            .map(|size| size as u64)
+            .ok_or(EOVERFLOW)
+    }
+
+    /// Returns the exclusive end of the requested mapping within the PCI region.
+    pub fn region_end(&self) -> Result<u64> {
+        self.region_offset()
+            .checked_add(self.size()?)
+            .ok_or(EOVERFLOW)
+    }
+}
+
+/// A VFIO PCI variant device, wrapping `struct vfio_pci_core_device`.
+///
+/// Callback contexts expose only their matching core helper and cannot be
+/// refcounted or shared across threads.
+///
+/// # Invariants
+///
+/// In addition to the invariants of [`DeviceBase`]:
+///
+/// - The wrapped `vfio_pci_core_device` was initialized by `vfio_pci_core_init_dev()`.
+/// - The callback data follows the VFIO registration and first-open/last-close lifetimes.
+///   Open data is destroyed before PCI core close, and registration data is
+///   released after PCI core unregistration has drained callbacks.
+pub type Device<T, Ctx = Normal> = DeviceBase<
+    bindings::vfio_pci_core_device,
+    <T as Operations>::RegistrationData<'static>,
+    <T as Operations>::OpenData<'static, 'static>,
+    T,
+    Ctx,
+>;
+
+static_assert!(core::mem::offset_of!(bindings::vfio_pci_core_device, vdev) == 0);
+
+// SAFETY: `vdev` is the first field of `vfio_pci_core_device`, as asserted above, and the
+// returned pointer is derived from `raw`.
+unsafe impl RawDevice for bindings::vfio_pci_core_device {
+    unsafe fn raw_vfio_device(raw: *mut Self) -> *mut bindings::vfio_device {
+        // SAFETY: The caller guarantees that `raw` points to a live `Self`.
+        unsafe { &raw mut (*raw).vdev }
+    }
+}
+
+impl<T: Operations> Device<T> {
+    /// Allocate a VFIO PCI device for subsequent registration.
+    pub fn new(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+        Self::allocate(pdev, false)
+    }
+
+    /// Allocate a device whose physical BARs may also be exported as DMA buffers.
+    pub fn new_passthrough(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+        Self::allocate(pdev, true)
+    }
+
+    fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<ARef<Self>> {
+        let size = Kmalloc::aligned_layout(Layout::new::<Self>()).size();
+
+        // SAFETY: The bound PCI device and static ops are valid. The allocation
+        // includes the full Rust wrapper with kmalloc-compatible size and alignment.
+        let raw = from_err_ptr(unsafe {
+            bindings::_vfio_alloc_device(size, pdev.as_ref().as_raw(), &Self::OPS)
+        })?;
+        let this = NonNull::new(raw.cast::<Self>()).ok_or(ENOMEM)?;
+
+        // SAFETY: `vfio_pci_core_init_dev()` initialized the C device in this new allocation,
+        // which nothing else can access yet.
+        unsafe { Self::init(this.as_ptr()) };
+
+        if passthrough {
+            // SAFETY: No registration or callbacks can observe this allocation yet.
+            unsafe { (*this.as_ref().as_raw()).pci_ops = &Self::PASSTHROUGH_OPS };
+        }
+
+        // SAFETY: `this` owns the initial VFIO device reference.
+        Ok(unsafe { ARef::from_raw(this) })
+    }
+}
+
+impl<T: Operations, Ctx: IoContext> Device<T, Ctx> {
+    /// Accesses the registration data and the open data from an I/O callback.
+    ///
+    /// The data lifetimes stay independent of the callback borrow, so that borrowed data
+    /// cannot be stored back into the registration data or the open data.
+    fn callback_data_with<V, F>(&self, f: F) -> V
+    where
+        F: for<'a, 'data> FnOnce(&T::RegistrationData<'data>, Pin<&T::OpenData<'a, 'data>>) -> V,
+    {
+        // SAFETY: `f` is generic over the lifetimes of the data, so it uses the data
+        // parametrically.
+        let (reg_data, open_data) = unsafe { self.callback_data() };
+        f(reg_data, open_data)
+    }
+}
+
+impl<T: Operations> Device<T, Open> {
+    /// Set the device ID exposed through VFIO's virtual PCI configuration space.
+    pub fn set_device_id(&self, device_id: u16) {
+        // SAFETY: core_enable allocated vconfig; the exclusive first-open callback
+        // runs before finish_enable publishes the configuration to userspace.
+        unsafe {
+            (*self.as_raw())
+                .vconfig
+                .add(bindings::PCI_DEVICE_ID as usize)
+                .cast::<u16>()
+                .write_unaligned(device_id.to_le());
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Ioctl> {
+    /// Delegate to `vfio_pci_core_ioctl()`.
+    pub fn core_ioctl(&self, cmd: u32, arg: usize) -> Result<isize> {
+        // SAFETY: The Ioctl context keeps this device open on the callback thread.
+        let ret = unsafe { bindings::vfio_pci_core_ioctl(self.as_raw_vfio(), cmd, arg) };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Read> {
+    /// Delegate to `vfio_pci_core_read()`.
+    ///
+    /// The VFIO core fills the user-space buffer with the default PCI data
+    /// for the region identified by `ppos` and advances the position by the bytes read.
+    pub fn core_read(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+        // SAFETY: The Read context keeps this device open on the callback thread;
+        // `buf.ptr` is a user-space pointer supplied by VFIO.
+        let ret = unsafe {
+            bindings::vfio_pci_core_read(
+                self.as_raw_vfio(),
+                buf.ptr.as_mut_ptr().cast(),
+                buf.count,
+                ppos.raw,
+            )
+        };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Write> {
+    /// Delegate to `vfio_pci_core_write()` for this callback's bounded buffer.
+    pub fn core_write(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+        // SAFETY: The Write context keeps the device open on the callback thread;
+        // the buffer is a userspace pointer supplied by VFIO.
+        let ret = unsafe {
+            bindings::vfio_pci_core_write(
+                self.as_raw_vfio(),
+                buf.ptr.as_const_ptr().cast(),
+                buf.count,
+                ppos.raw,
+            )
+        };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Mmap> {
+    /// Delegate to `vfio_pci_core_mmap()` for the current callback's VMA.
+    pub fn core_mmap(&self, mapping: &mut Mapping<'_>) -> Result {
+        // SAFETY: The Mmap context and Mapping keep the open device and locked VMA
+        // on the callback thread. The VMA cannot be replaced by driver code.
+        to_result(unsafe { bindings::vfio_pci_core_mmap(self.as_raw_vfio(), mapping.vma.as_ptr()) })
+    }
+}
+
+impl<T: Operations> Device<T, GetRegionInfo> {
+    /// Delegate to `vfio_pci_ioctl_get_region_info()`.
+    pub fn core_get_region_info(
+        &self,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut InfoCap<'_>,
+    ) -> Result {
+        // SAFETY: GetRegionInfo keeps the device open on the callback thread;
+        // `InfoCap` preserves the validity of VFIO's capability buffer.
+        to_result(unsafe {
+            bindings::vfio_pci_ioctl_get_region_info(self.as_raw_vfio(), info, caps.raw)
+        })
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` in VFIO's first-open context.
+/// VFIO must exclude other opens, closes and I/O callbacks for this call.
+unsafe extern "C" fn open_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) -> c_int {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the first-open context.
+    let dev = unsafe { Device::<T, Open>::from_raw_vfio(vdev) };
+
+    // SAFETY: Enable the PCI-core side first.
+    let ret = unsafe { bindings::vfio_pci_core_enable(dev.as_raw()) };
+    if ret != 0 {
+        return ret;
+    }
+
+    // SAFETY: The previous last close destroyed any earlier open data. The open data type
+    // passed is `T::OpenData`, which only differs in lifetimes from the one of `Device<T>`.
+    // `T::open_device()` is generic over the lifetimes of its arguments, and the closure
+    // captures nothing but `dev`, which it passes on with the same lifetime as the
+    // registration data. `close_device_cb()` destroys the open data at the last close.
+    let result = unsafe {
+        dev.open::<T::OpenData<'_, 'static>, _>(|reg_data| T::open_device(dev, reg_data))
+    };
+    match result {
+        Ok(()) => {
+            // SAFETY: open succeeded.
+            unsafe { bindings::vfio_pci_core_finish_enable(dev.as_raw()) };
+            0
+        }
+        Err(e) => {
+            // SAFETY: Undo the enable on failure.
+            unsafe { bindings::vfio_pci_core_disable(dev.as_raw()) };
+            e.to_errno()
+        }
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` whose first open succeeded.
+/// Call exactly once for the last close, after draining I/O callbacks and while
+/// excluding the next first open.
+unsafe extern "C" fn close_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) {
+    // SAFETY: `vdev` belongs to a live `Device<T>`.
+    let dev = unsafe { Device::<T>::from_raw_vfio(vdev) };
+    // SAFETY: Last close drains I/O callbacks and excludes the next first open.
+    // Registration data and PCI resources remain valid while open data is destroyed.
+    unsafe { dev.close() };
+    // SAFETY: Matches the enable in open_device_cb.
+    unsafe { bindings::vfio_pci_core_close_device(vdev) };
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's ioctl callback
+/// context. Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn ioctl_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    cmd: core::ffi::c_uint,
+    arg: usize,
+) -> isize {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the ioctl context.
+    let dev = unsafe { Device::<T, Ioctl>::from_raw_vfio(vdev) };
+    match dev.callback_data_with(|rd, od| T::ioctl(dev, rd, od, cmd, arg)) {
+        Ok(v) => v,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's read context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn read_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    buf: *mut u8,
+    count: usize,
+    ppos: *mut i64,
+) -> isize {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the read context.
+    let dev = unsafe { Device::<T, Read>::from_raw_vfio(vdev) };
+    let mut ubuf = UserBuf {
+        ptr: UserPtr::from_ptr(buf.cast()),
+        count,
+    };
+    let mut pos = Position {
+        // SAFETY: `ppos` is a valid kernel pointer provided by the VFIO core.
+        raw: unsafe { &mut *ppos },
+    };
+    match dev.callback_data_with(|rd, od| T::read(dev, rd, od, &mut ubuf, &mut pos)) {
+        Ok(n) => n,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's region-info
+/// context. `info` and `caps` must be valid, disjoint, exclusively accessible
+/// objects; the capability buffer must satisfy the VFIO core's invariants.
+/// Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn get_region_info_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    info: *mut bindings::vfio_region_info,
+    caps: *mut bindings::vfio_info_cap,
+) -> c_int {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the region-info context.
+    let dev = unsafe { Device::<T, GetRegionInfo>::from_raw_vfio(vdev) };
+    // SAFETY: `info` is a valid pointer provided by the VFIO core.
+    let info = unsafe { &mut *info };
+    let mut caps = InfoCap {
+        // SAFETY: `caps` is a valid pointer provided by the VFIO core.
+        raw: unsafe { &mut *caps },
+    };
+    match dev.callback_data_with(|rd, od| T::get_region_info(dev, rd, od, info, &mut caps)) {
+        Ok(()) => 0,
+        Err(e) => e.to_errno(),
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's write context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn write_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    buf: *const u8,
+    count: usize,
+    ppos: *mut i64,
+) -> isize {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the write context.
+    let dev = unsafe { Device::<T, Write>::from_raw_vfio(vdev) };
+    let mut ubuf = UserBuf {
+        ptr: UserPtr::from_ptr(buf.cast_mut().cast()),
+        count,
+    };
+    let mut pos = Position {
+        // SAFETY: VFIO holds the file-position lock when required.
+        raw: unsafe { &mut *ppos },
+    };
+    match dev.callback_data_with(|rd, od| T::write(dev, rd, od, &mut ubuf, &mut pos)) {
+        Ok(n) => n,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's mmap context.
+/// `vma` must be the valid VMA being mapped, with the mmap write lock held on
+/// this thread. Registration and open data must remain alive until return.
+unsafe extern "C" fn mmap_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    vma: *mut bindings::vm_area_struct,
+) -> c_int {
+    // SAFETY: `vdev` belongs to a live `Device<T>`; VFIO provides the mmap context.
+    let dev = unsafe { Device::<T, Mmap>::from_raw_vfio(vdev) };
+    let mut mapping = Mapping {
+        // SAFETY: The mmap callback holds the mmap lock for this VMA.
+        vma: unsafe { VmaRef::from_raw(vma) },
+        _not_thread_safe: NotThreadSafe,
+    };
+    match dev.callback_data_with(|rd, od| T::mmap(dev, rd, od, &mut mapping)) {
+        Ok(()) => 0,
+        Err(e) => e.to_errno(),
+    }
+}
+
+/// Build the `vfio_device_ops` vtable for a variant driver `T`.
+///
+/// Open and I/O callbacks dispatch to `T`'s trait implementation.
+/// All other callbacks use the `vfio_pci_core_*` defaults.
+macro_rules! build_ops {
+    ($T:ty) => {
+        bindings::vfio_device_ops {
+            #[allow(clippy::disallowed_methods)]
+            name: <$T>::NAME.as_ptr().cast_mut().cast(),
+            init: Some(bindings::vfio_pci_core_init_dev),
+            release: Some(bindings::vfio_pci_core_release_dev),
+            open_device: Some(open_device_cb::<$T>),
+            close_device: Some(close_device_cb::<$T>),
+            ioctl: Some(ioctl_cb::<$T>),
+            read: Some(read_cb::<$T>),
+            write: Some(write_cb::<$T>),
+            mmap: Some(mmap_cb::<$T>),
+            request: Some(bindings::vfio_pci_core_request),
+            get_region_info_caps: Some(get_region_info_cb::<$T>),
+            match_: Some(bindings::vfio_pci_core_match),
+            match_token_uuid: Some(bindings::vfio_pci_core_match_token_uuid),
+            device_feature: Some(bindings::vfio_pci_core_ioctl_feature),
+            #[cfg(CONFIG_IOMMUFD)]
+            bind_iommufd: Some(bindings::vfio_iommufd_physical_bind),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            bind_iommufd: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            unbind_iommufd: Some(bindings::vfio_iommufd_physical_unbind),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            unbind_iommufd: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            attach_ioas: Some(bindings::vfio_iommufd_physical_attach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            attach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            detach_ioas: Some(bindings::vfio_iommufd_physical_detach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            detach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            pasid_attach_ioas: Some(bindings::vfio_iommufd_physical_pasid_attach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            pasid_attach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            pasid_detach_ioas: Some(bindings::vfio_iommufd_physical_pasid_detach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            pasid_detach_ioas: None,
+            dma_unmap: None,
+        }
+    };
+}
+
+/// The registration of a VFIO PCI variant device.
+///
+/// Owns both the VFIO device reference and the registration data, tying
+/// resource lifetimes to the PCI binding scope `'a`. The device is registered
+/// on construction and unregistered when the `Registration` is dropped.
+pub struct Registration<'a, T: Operations> {
+    dev: ARef<Device<T>>,
+    reg_data: Pin<KBox<T::RegistrationData<'a>>>,
+    // Released after `Drop::drop()` has unregistered `dev`.
+    _active: RuntimeActive,
+}
+
+/// Keeps the PCI device of a registered VFIO device runtime active.
+///
+/// The VFIO PCI core allows runtime suspend of the PCI device while its VFIO device is
+/// registered, and expects the PCI driver's runtime PM callbacks to call into the VFIO PCI
+/// core. These abstractions do not provide such callbacks yet, so the device must not be
+/// runtime suspended.
+///
+/// # Invariants
+///
+/// This type owns a runtime PM usage reference on the device.
+// TODO: Do not hold this reference for drivers that provide runtime PM callbacks, once these
+// abstractions support them.
+struct RuntimeActive(#[cfg_attr(not(CONFIG_PM), expect(dead_code))] ARef<device::Device>);
+
+impl RuntimeActive {
+    fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
+        // SAFETY: `dev` is a valid device. This is `pm_runtime_get_sync()`, which takes a usage
+        // reference even if it fails.
+        #[cfg(CONFIG_PM)]
+        let ret =
+            unsafe { bindings::__pm_runtime_resume(dev.as_raw(), bindings::RPM_GET_PUT as c_int) };
+        #[cfg(not(CONFIG_PM))]
+        let ret = 0;
+        // INVARIANT: `this` owns the usage reference, and drops it if the resume failed.
+        let this = Self(dev.into());
+        to_result(ret)?;
+        Ok(this)
+    }
+}
+
+impl Drop for RuntimeActive {
+    fn drop(&mut self) {
+        // SAFETY: `self.0` is a valid device, and `new()` took the usage reference dropped
+        // here. This is `pm_runtime_put()`.
+        #[cfg(CONFIG_PM)]
+        unsafe {
+            bindings::__pm_runtime_idle(
+                self.0.as_raw(),
+                (bindings::RPM_GET_PUT | bindings::RPM_ASYNC) as c_int,
+            )
+        };
+    }
+}
+
+impl<T: Operations> Device<T> {
+    const OPS: bindings::vfio_device_ops = build_ops!(T);
+
+    const PASSTHROUGH_OPS: bindings::vfio_pci_device_ops = bindings::vfio_pci_device_ops {
+        #[cfg(CONFIG_VFIO_PCI_DMABUF)]
+        get_dmabuf_phys: Some(bindings::vfio_pci_core_get_dmabuf_phys),
+        #[cfg(not(CONFIG_VFIO_PCI_DMABUF))]
+        get_dmabuf_phys: None,
+    };
+}
+
+impl<'a, T: Operations> Registration<'a, T> {
+    /// Register a previously allocated VFIO PCI device with its callback data.
+    ///
+    /// `dev` must have been allocated for `pdev`, and can be registered only once. `D` is the
+    /// PCI driver bound to `pdev`.
+    ///
+    /// # Safety
+    ///
+    /// The returned `Registration` must be stored in the bus device private data of `D`, the
+    /// driver bound to `pdev`, and must not be leaked: it has to be dropped while that driver
+    /// is being unbound from `pdev`, or when its probe fails.
+    pub unsafe fn new<D: pci::Driver>(
+        pdev: &'a pci::Device<device::Core<'_>>,
+        dev: &Device<T>,
+        reg_data: impl PinInit<T::RegistrationData<'a>, Error>,
+    ) -> Result<Self> {
+        build_assert!(
+            D::DRIVER_MANAGED_DMA.is_some(),
+            "VFIO drivers must manage DMA ownership"
+        );
+        build_assert!(
+            !D::HAS_SRIOV_ENABLE && !D::HAS_SRIOV_DISABLE,
+            "VFIO PCI variant drivers do not support SR-IOV configuration"
+        );
+
+        // SAFETY: `dev` holds a live VFIO allocation; only compare its parent pointer.
+        if unsafe { (*dev.as_raw_vfio()).dev } != pdev.as_ref().as_raw() {
+            return Err(EINVAL);
+        }
+
+        let reg_data: Pin<KBox<T::RegistrationData<'a>>> = KBox::pin_init(reg_data, GFP_KERNEL)?;
+        let ptr: NonNull<T::RegistrationData<'static>> =
+            NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
+
+        // SAFETY: `dev` belongs to `pdev`, whose device lock serializes publication. The
+        // returned `Registration` keeps the pinned data alive until it has unregistered `dev`;
+        // if registration fails below, `dev` remains unregistered and never accesses the data.
+        unsafe { dev.publish_registration_data(ptr) }?;
+
+        // Registering `dev` allows runtime suspend of `pdev`; see `RuntimeActive`.
+        let active = RuntimeActive::new(pdev.as_ref())?;
+
+        // SAFETY: `dev` is initialized, has published its registration data and has not been
+        // registered before. The caller holds the PCI device lock of its parent.
+        to_result(unsafe { bindings::vfio_pci_core_register_device(dev.as_raw()) })?;
+
+        Ok(Self {
+            dev: dev.into(),
+            reg_data,
+            _active: active,
+        })
+    }
+
+    /// Prepare the variant driver's state for a PCI reset.
+    pub fn reset_prepare(&self) {
+        T::reset_prepare(&self.reg_data);
+    }
+
+    /// Complete the variant driver's state update after a PCI reset.
+    pub fn reset_done(&self) -> Result {
+        T::reset_done(&self.reg_data)
+    }
+
+    /// Signal the VFIO error interrupt and return the PCI recovery result.
+    pub fn error_detected(&self) -> pci::ErsResult {
+        // SAFETY: This owner keeps the initialized VFIO core device alive.
+        let result = unsafe { bindings::vfio_pci_core_aer_err_detected(self.dev.as_raw()) };
+        // The VFIO PCI core only returns results `pci::ErsResult` represents.
+        pci::ErsResult::from_raw(result).unwrap_or(pci::ErsResult::None)
+    }
+}
+
+impl<T: Operations> Drop for Registration<'_, T> {
+    fn drop(&mut self) {
+        // SAFETY: `new()` registered the device. By its safety requirements, this runs with the
+        // PCI device lock held, either from the PCI remove callback or while probe fails.
+        // Unregistration drains the VFIO callbacks, the only users of the registration data
+        // besides this owner, before the data is released.
+        unsafe { bindings::vfio_pci_core_unregister_device(self.dev.as_raw()) };
+    }
+}
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 8/9] gpu: nova-core: publish typed SR-IOV PF APIs
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (6 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 7/9] rust: vfio: add PCI variant driver abstractions Simon Song
@ 2026-11-19 21:50 ` Simon Song
  2026-11-19 21:50 ` [PATCH v3 9/9] vfio/nova-vgpu: add the Rust VFIO variant driver Simon Song
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: Zhi Wang, alex, jgg, yishaih, skolothumtho, kevin.tian, airlied,
	simona, ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, boqun, daniel.almeida, tamird, work, jgg,
	bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Zhi Wang <zhiw@nvidia.com>

Register NovaCoreVfApi as typed SR-IOV PF data once the GPU is initialized.

Enable and disable VFs through the sriov_enable() and sriov_disable()
callbacks. The VF registration removes VFs before dropping its PF API
data. Declare the registration before the GPU so its borrowed services
remain valid through VF teardown.

Expose a VF-side handle that borrows these services for each operation.
Return an owned instance on open and close it on drop while the VF
remains bound.

Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 drivers/gpu/nova-core/driver.rs        |  34 ++++-
 drivers/gpu/nova-core/nova_core.rs     |   3 +-
 drivers/gpu/nova-core/vgpu.rs          |   1 -
 drivers/gpu/nova-core/vgpu/vgpu_api.rs | 178 +++++++++++++++++++------
 4 files changed, 171 insertions(+), 45 deletions(-)

diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index 1785a5294823..a306e4233df6 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -29,7 +29,8 @@
         NovaCoreFwCtl,
         NovaCoreFwCtlData, //
     },
-    gpu::Gpu, //
+    gpu::Gpu,
+    vgpu::vgpu_api::NovaCoreVfApi, //
 };
 
 /// Counter for generating unique auxiliary device IDs.
@@ -41,6 +42,10 @@ pub(crate) struct NovaCore<'bound> {
     _reg: auxiliary::Registration<'bound, ForLt!(NovaCoreApi<'_>)>,
     /// Unregisters fwctl and drains its callbacks before GPU resources are dropped.
     _fwctl: fwctl::Registration<'bound, NovaCoreFwCtl>,
+    #[allow(clippy::type_complexity)]
+    #[pin]
+    #[uses('gpu: invariant)]
+    _vf_registration: pci::VfRegistration<'gpu, ForLt!(NovaCoreVfApi<'_>)>,
     #[pin]
     pub(crate) gpu: Gpu<'bound>,
     bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -112,6 +117,10 @@ fn probe<'bound>(
 
         try_pin_init!(NovaCore {
             _: {
+                if pdev.is_virtfn() {
+                    return Err(ENODEV);
+                }
+
                 pdev.enable_device_mem()?;
                 pdev.set_master();
             },
@@ -150,6 +159,12 @@ fn probe<'bound>(
                 Ok(())
             }),
 
+            // SAFETY: The registration is stored in `NovaCore`, this driver's bus device
+            // private data.
+            _vf_registration <- unsafe {
+                pci::VfRegistration::new(pdev, NovaCoreVfApi::new(pdev, gpu))
+            },
+
             _reg: {
                 // TODO: Use `&gpu` self-referential pin-init syntax once available.
                 //
@@ -198,4 +213,21 @@ fn probe<'bound>(
             },
         })
     }
+
+    fn sriov_enable<'a>(
+        _dev: &'a pci::Device<Core<'_>>,
+        _this: Pin<&Self::Data<'a>>,
+        token: pci::SriovEnable<'a>,
+    ) -> Result<pci::SriovEnabled<'a>> {
+        token.enable_all()
+    }
+
+    fn sriov_disable<'a>(
+        _dev: &'a pci::Device<Core<'_>>,
+        _this: Pin<&Self::Data<'a>>,
+        token: pci::SriovDisable<'a>,
+    ) -> Result {
+        token.disable();
+        Ok(())
+    }
 }
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 5f09b69f3b54..a840b663cf73 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -31,9 +31,10 @@
 mod vbios;
 mod vgpu;
 
-#[cfg(CONFIG_PCI_IOV)]
 pub use vgpu::vgpu_api::{
     NovaCoreVfApi,
+    NovaCoreVfApiHandle,
+    VgpuInstance,
     VgpuTypeInfo, //
 };
 
diff --git a/drivers/gpu/nova-core/vgpu.rs b/drivers/gpu/nova-core/vgpu.rs
index f198fa218297..38dcd68f52e7 100644
--- a/drivers/gpu/nova-core/vgpu.rs
+++ b/drivers/gpu/nova-core/vgpu.rs
@@ -42,7 +42,6 @@
 mod instance;
 mod log;
 mod scrubber;
-#[cfg_attr(not(CONFIG_PCI_IOV), expect(dead_code, unreachable_pub))]
 pub(crate) mod vgpu_api;
 mod vram;
 
diff --git a/drivers/gpu/nova-core/vgpu/vgpu_api.rs b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
index f9f3e527bdc0..cfac0ca10c1e 100644
--- a/drivers/gpu/nova-core/vgpu/vgpu_api.rs
+++ b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
@@ -8,7 +8,8 @@
 use kernel::{
     device,
     pci,
-    prelude::*, //
+    prelude::*,
+    types::ForLt, //
 };
 
 use crate::{
@@ -52,29 +53,41 @@ fn from_vgpu_type(vgpu_type: &VgpuType) -> Self {
 }
 
 /// PF-owned operations available while a VF driver is bound.
-pub struct NovaCoreVfApi<'gpu> {
+#[pin_data]
+pub struct NovaCoreVfApi<'gpu>
+where
+    // The GPU resources are invariant in their lifetime, which outlives this API borrow.
+    exists<'gpu_resources>: 'gpu,
+{
     pdev: &'gpu pci::Device<device::Bound>,
-    cmdq: &'gpu Cmdq<'gpu>,
-    bar: Bar0<'gpu>,
-    vgpu: Option<&'gpu VgpuManager<'gpu>>,
+    #[uses('gpu_resources: invariant)]
+    cmdq: &'gpu Cmdq<'gpu_resources>,
+    #[uses('gpu_resources)]
+    bar: Bar0<'gpu_resources>,
+    #[uses('gpu_resources: invariant)]
+    vgpu: Option<&'gpu VgpuManager<'gpu_resources>>,
     total_vfs: Option<NonZero<u16>>,
 }
 
 impl<'gpu> NovaCoreVfApi<'gpu> {
-    #[expect(dead_code)]
-    pub(crate) fn new(gpu: &'gpu Gpu<'gpu>, pdev: &'gpu pci::Device<device::Bound>) -> Self {
-        Self {
+    /// Creates the PF services that VF drivers access through the VF registration.
+    pub(crate) fn new<'res: 'gpu>(
+        pdev: &'gpu pci::Device<device::Bound>,
+        gpu: Pin<&'gpu Gpu<'res>>,
+    ) -> impl PinInit<Self, Error> + use<'gpu, 'res> {
+        let gpu = gpu.get_ref();
+        try_pin_init!(Self {
             pdev,
             cmdq: gpu.cmdq(),
             bar: gpu.bar0(),
             vgpu: gpu.vgpu_manager(),
             total_vfs: gpu.vgpu_total_vfs(),
-        }
+        })
     }
 
     /// Returns whether the PF booted with vGPU support enabled.
     pub fn is_available(&self) -> bool {
-        self.vgpu.is_some()
+        self.with_vgpu(|vgpu| vgpu.is_some())
     }
 
     fn gfid(&self, gfid: u32) -> Result<Gfid> {
@@ -84,11 +97,29 @@ fn gfid(&self, gfid: u32) -> Result<Gfid> {
 }
 
 impl NovaCoreVfApi<'_> {
+    /// Obtains the enabled PF services for a bound VF.
+    pub fn handle(vf: &pci::Device<device::Bound>) -> Result<NovaCoreVfApiHandle<'_>> {
+        let handle = NovaCoreVfApiHandle { vf };
+        handle.with(|api| {
+            if api.is_available() {
+                Ok(())
+            } else {
+                Err(ENODEV)
+            }
+        })?;
+        Ok(handle)
+    }
+
     /// Creates and boots an instance for a one-based VF ID.
     ///
     /// `sbdf` encodes the VF address as `(segment << 16) | (bus << 8) | devfn`.
     /// `vm_pid` identifies the VM process's thread group. This call may sleep.
-    pub fn open_instance(&self, gfid: u32, sbdf: u32, vm_pid: u32) -> Result<VgpuTypeInfo> {
+    pub fn open_instance(
+        self: Pin<&Self>,
+        gfid: u32,
+        sbdf: u32,
+        vm_pid: u32,
+    ) -> Result<VgpuTypeInfo> {
         let dev = self.pdev.as_ref();
         let gfid = self.gfid(gfid)?;
         let dbdf = Dbdf::from_raw(sbdf);
@@ -100,31 +131,33 @@ pub fn open_instance(&self, gfid: u32, sbdf: u32, vm_pid: u32) -> Result<VgpuTyp
             dbdf.into_raw()
         );
 
-        let bar = self.bar;
-        let cmdq = self.cmdq;
-        let vgpu = self.vgpu.ok_or(ENODEV)?;
-
-        let type_id = query_assigned_vf_type(dev, cmdq, dbdf)?;
-        dev_dbg!(
-            dev,
-            "vgpu_open: gfid={} assigned type_id={}\n",
-            gfid.get(),
-            type_id
-        );
-
-        let vgpu_type = query_vgpu_type(dev, cmdq, type_id)?;
-        dev_dbg!(
-            dev,
-            "vgpu_open: gfid={} vgpu_type={} fb_length={:#x}\n",
-            gfid.get(),
-            vgpu_type.vgpu_type_id,
-            vgpu_type.fb_length
-        );
-
-        let type_info = VgpuTypeInfo::from_vgpu_type(&vgpu_type);
-        vgpu.create_instance(bar, InstanceInfo::new(gfid, dbdf, vgpu_type, vm_pid))?;
-
-        Ok(type_info)
+        self.with_project_ref(|this| {
+            let bar = *this.bar;
+            let cmdq = *this.cmdq;
+            let vgpu = this.vgpu.ok_or(ENODEV)?;
+
+            let type_id = query_assigned_vf_type(dev, cmdq, dbdf)?;
+            dev_dbg!(
+                dev,
+                "vgpu_open: gfid={} assigned type_id={}\n",
+                gfid.get(),
+                type_id
+            );
+
+            let vgpu_type = query_vgpu_type(dev, cmdq, type_id)?;
+            dev_dbg!(
+                dev,
+                "vgpu_open: gfid={} vgpu_type={} fb_length={:#x}\n",
+                gfid.get(),
+                vgpu_type.vgpu_type_id,
+                vgpu_type.fb_length
+            );
+
+            let type_info = VgpuTypeInfo::from_vgpu_type(&vgpu_type);
+            vgpu.create_instance(bar, InstanceInfo::new(gfid, dbdf, vgpu_type, vm_pid))?;
+
+            Ok(type_info)
+        })
     }
 
     /// Tears down an instance, retaining resources if firmware cleanup fails.
@@ -136,11 +169,13 @@ pub fn close_instance(&self, gfid: u32) -> Result {
 
         dev_dbg!(dev, "vgpu_close: gfid={}\n", gfid.get());
 
-        let result = self.vgpu.ok_or(ENODEV)?.close_instance(gfid);
-        if let Err(error) = result {
-            dev_err!(dev, "vgpu_close: gfid={} failed: {:?}\n", gfid.get(), error);
-        }
-        result
+        self.with_vgpu(|vgpu| {
+            let result = vgpu.ok_or(ENODEV)?.close_instance(gfid);
+            if let Err(error) = result {
+                dev_err!(dev, "vgpu_close: gfid={} failed: {:?}\n", gfid.get(), error);
+            }
+            result
+        })
     }
 
     /// Resets an instance and scrubs its guest VRAM. This call may sleep.
@@ -150,9 +185,68 @@ pub fn reset_instance(&self, gfid: u32) -> Result {
 
         dev_dbg!(dev, "vgpu_reset: gfid={}\n", gfid.get());
 
-        self.vgpu.ok_or(ENODEV)?.reset_instance(gfid)?;
+        self.with_vgpu(|vgpu| vgpu.ok_or(ENODEV)?.reset_instance(gfid))?;
 
         dev_dbg!(dev, "vgpu_reset: gfid={} done\n", gfid.get());
         Ok(())
     }
 }
+
+/// Access to PF services for the lifetime of a VF driver binding.
+///
+/// The PF's registration keeps these services available until VF removal
+/// completes and is dropped before the GPU resources they borrow.
+pub struct NovaCoreVfApiHandle<'vf> {
+    vf: &'vf pci::Device<device::Bound>,
+}
+
+impl<'vf> NovaCoreVfApiHandle<'vf> {
+    /// Borrows the typed PF services for a single operation.
+    fn with<R>(
+        &self,
+        f: impl for<'borrow, 'data> FnOnce(Pin<&'borrow NovaCoreVfApi<'data>>) -> Result<R>,
+    ) -> Result<R> {
+        self.vf
+            .vf_registration_data_with::<ForLt!(NovaCoreVfApi<'_>), _>(f)?
+    }
+
+    /// Creates and boots an instance that closes when dropped.
+    ///
+    /// The arguments have the same meaning as in [`NovaCoreVfApi::open_instance`].
+    pub fn open(&self, gfid: u32, dbdf: u32, vm_pid: u32) -> Result<VgpuInstance<'vf>> {
+        let type_info = self.with(|api| api.open_instance(gfid, dbdf, vm_pid))?;
+        Ok(VgpuInstance {
+            api: Self { vf: self.vf },
+            gfid,
+            type_info,
+        })
+    }
+
+    /// Resets an active instance and scrubs its guest VRAM.
+    pub fn reset(&self, gfid: u32) -> Result {
+        self.with(|api| api.reset_instance(gfid))
+    }
+}
+
+/// An active instance whose teardown runs while its VF driver remains bound.
+///
+/// Dropping this guard may sleep while firmware teardown completes.
+pub struct VgpuInstance<'vf> {
+    api: NovaCoreVfApiHandle<'vf>,
+    gfid: u32,
+    type_info: VgpuTypeInfo,
+}
+
+impl VgpuInstance<'_> {
+    /// Returns the assigned PCI IDs and BAR1 aperture size.
+    #[inline]
+    pub fn type_info(&self) -> &VgpuTypeInfo {
+        &self.type_info
+    }
+}
+
+impl Drop for VgpuInstance<'_> {
+    fn drop(&mut self) {
+        let _ = self.api.with(|api| api.close_instance(self.gfid));
+    }
+}
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3 9/9] vfio/nova-vgpu: add the Rust VFIO variant driver
  2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
                   ` (7 preceding siblings ...)
  2026-11-19 21:50 ` [PATCH v3 8/9] gpu: nova-core: publish typed SR-IOV PF APIs Simon Song
@ 2026-11-19 21:50 ` Simon Song
  8 siblings, 0 replies; 10+ messages in thread
From: Simon Song @ 2026-11-19 21:50 UTC (permalink / raw)
  To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
  Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
	ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
	a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
	smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
	linux-kernel, zhiwang, zhiw, boqun, daniel.almeida, tamird, work,
	jgg, bhelgaas, kwilczynski, dri-devel, peter, xinmengs

From: Danilo Krummrich <dakr@kernel.org>

NVIDIA vGPU VFs need nova-core to manage their instances when assigned
to userspace through VFIO.

Add a Rust VFIO PCI variant driver that uses the typed SR-IOV PF APIs
to create an instance on first open and close it on last close. Expose
the assigned device and subsystem IDs, enforce the assigned BAR1
aperture, and reset the active vGPU instance on PCI reset.

Match NVIDIA devices through a VFIO override-only entry. Devices
without an available Nova vGPU interface use ordinary PCI passthrough.

Register the VFIO device from probe and keep the Registration in the
PCI driver data, so that unbind unregisters it. Extend the GPU build
rules to make nova-core crate metadata available to the VFIO driver and
export the Rust symbols it references.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 drivers/gpu/Makefile                    |  15 +-
 drivers/vfio/pci/Kconfig                |   2 +
 drivers/vfio/pci/Makefile               |   2 +
 drivers/vfio/pci/nova-vgpu/Kconfig      |  20 ++
 drivers/vfio/pci/nova-vgpu/Makefile     |   3 +
 drivers/vfio/pci/nova-vgpu/nova_vgpu.rs | 375 ++++++++++++++++++++++++
 6 files changed, 414 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/pci/nova-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nova-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nova-vgpu/nova_vgpu.rs

diff --git a/drivers/gpu/Makefile b/drivers/gpu/Makefile
index e372fc02139f..5fd610a172ad 100644
--- a/drivers/gpu/Makefile
+++ b/drivers/gpu/Makefile
@@ -19,8 +19,9 @@ nova-core-y := nova-core/nova_core.o nova-core/nova_core_exports.o
 obj-$(CONFIG_DRM_NOVA) += nova-drm.o
 nova-drm-y := drm/nova/nova.o
 
-# Export Rust symbols from nova-core only if nova-drm actually references them.
-nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o)
+# Export Rust symbols from nova-core only if dependent modules reference them.
+nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o) \
+	$(if $(CONFIG_NOVA_VGPU_VFIO_PCI),$(obj)/../vfio/pci/nova-vgpu/nova_vgpu.o)
 
 rust_needed_exports = \
 	{ $(if $(strip $(2)),$(NM) -u $(2);,) echo "__DEFINED_RUST_SYMBOLS__"; \
@@ -57,10 +58,18 @@ $(obj)/nova-core/nova_core_exports.o: private cmd_gensymtypes_c = \
 		$(obj)/nova-core/nova_core.o
 endif
 
-# Output nova-core's crate metadata for use by nova-drm at compile time.
+# Output nova-core's crate metadata for its Rust consumers at compile time.
 RUSTFLAGS_nova-core/nova_core.o += \
 	--emit=metadata=$(objtree)/$(obj)/nova-core/libnova_core.rmeta
 
 # Allow nova-drm to import nova-core's types.
 $(obj)/drm/nova/nova.o: $(obj)/nova-core/nova_core.o
 RUSTFLAGS_drm/nova/nova.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
+
+# Build the NVIDIA VFIO driver here so metadata and generated exports share
+# the same dependency graph as nova-drm.
+obj-$(CONFIG_NOVA_VGPU_VFIO_PCI) += nova-vgpu-vfio-pci.o
+nova-vgpu-vfio-pci-y := ../vfio/pci/nova-vgpu/nova_vgpu.o
+
+$(obj)/../vfio/pci/nova-vgpu/nova_vgpu.o: $(obj)/nova-core/nova_core.o
+RUSTFLAGS_../vfio/pci/nova-vgpu/nova_vgpu.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..dd9b755262a4 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -74,4 +74,6 @@ source "drivers/vfio/pci/qat/Kconfig"
 
 source "drivers/vfio/pci/xe/Kconfig"
 
+source "drivers/vfio/pci/nova-vgpu/Kconfig"
+
 endmenu
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..1d9fb11eb3c2 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -24,3 +24,5 @@ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
 obj-$(CONFIG_QAT_VFIO_PCI) += qat/
 
 obj-$(CONFIG_XE_VFIO_PCI) += xe/
+
+# nova-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
diff --git a/drivers/vfio/pci/nova-vgpu/Kconfig b/drivers/vfio/pci/nova-vgpu/Kconfig
new file mode 100644
index 000000000000..228f6ebc108c
--- /dev/null
+++ b/drivers/vfio/pci/nova-vgpu/Kconfig
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NOVA_VGPU_VFIO_PCI
+	tristate "VFIO support for the NVIDIA vGPU"
+	depends on NOVA_CORE && PCI_IOV && RUST
+	select VFIO_PCI_CORE
+	help
+	  This option enables VFIO (Virtual Function I/O) support for
+	  NVIDIA virtual GPUs (vGPU). It allows the assignment of a virtual
+	  GPU instance to userspace applications via VFIO, typically used
+	  with hypervisors such as KVM and device emulators like QEMU.
+
+	  Devices without an available Nova vGPU interface use ordinary PCI
+	  passthrough. Both paths use the VFIO PCI core defaults; vfio-pci
+	  module options do not apply.
+
+	  The NVIDIA vGPU allows a physical GPU to be partitioned into
+	  multiple virtual GPUs, each of which can be passed to a virtual
+	  machine as a PCI device using the standard VFIO infrastructure.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/nova-vgpu/Makefile b/drivers/vfio/pci/nova-vgpu/Makefile
new file mode 100644
index 000000000000..0875c782602e
--- /dev/null
+++ b/drivers/vfio/pci/nova-vgpu/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+# nova-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
+# nova_vgpu.o (rust-analyzer marker - DO NOT REMOVE).
diff --git a/drivers/vfio/pci/nova-vgpu/nova_vgpu.rs b/drivers/vfio/pci/nova-vgpu/nova_vgpu.rs
new file mode 100644
index 000000000000..20553b7b4e2e
--- /dev/null
+++ b/drivers/vfio/pci/nova-vgpu/nova_vgpu.rs
@@ -0,0 +1,375 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! NVIDIA vGPU VFIO variant driver.
+//!
+//! Nova VFs own an instance between the first VFIO open and last close. Other
+//! matching devices use ordinary PCI passthrough.
+
+use kernel::{
+    bindings,
+    device::{
+        Bound,
+        Core, //
+    },
+    io::resource::Flags,
+    pci,
+    prelude::*,
+    sync::{
+        CondVar,
+        Mutex,
+        MutexGuard, //
+    },
+    vfio::{
+        self,
+        pci::{
+            GetRegionInfo,
+            Ioctl,
+            Mapping,
+            Mmap,
+            Open,
+            Position,
+            Read,
+            Write, //
+        }, //
+    }, //
+};
+use nova_core::{
+    NovaCoreVfApi,
+    NovaCoreVfApiHandle,
+    VgpuInstance, //
+};
+
+struct NovaVgpuOps;
+
+struct VgpuRegistration<'a> {
+    api: NovaCoreVfApiHandle<'a>,
+    gfid: u32,
+}
+
+#[derive(Default)]
+struct InstanceState {
+    active: bool,
+    resetting: bool,
+    reset_error: Option<Error>,
+}
+
+#[pin_data]
+struct NovaVgpuRegData<'a> {
+    pdev: &'a pci::Device<Bound>,
+    vgpu: Option<VgpuRegistration<'a>>,
+    fb_bar: u32,
+    #[pin]
+    state: Mutex<InstanceState>,
+    #[pin]
+    reset_done: CondVar,
+}
+
+impl NovaVgpuRegData<'_> {
+    fn lock_instance(&self) -> MutexGuard<'_, InstanceState> {
+        let mut state = self.state.lock();
+        while state.resetting {
+            self.reset_done.wait(&mut state);
+        }
+        state
+    }
+}
+
+struct NovaVgpuOpenData<'a, 'data> {
+    registration: &'a NovaVgpuRegData<'data>,
+    instance: Option<VgpuInstance<'data>>,
+}
+
+impl<'a, 'data> NovaVgpuOpenData<'a, 'data> {
+    fn new(
+        dev: &vfio::pci::Device<NovaVgpuOps, Open>,
+        rd: &'a NovaVgpuRegData<'data>,
+    ) -> Result<Self> {
+        let mut state = rd.lock_instance();
+        let instance = match &rd.vgpu {
+            Some(vgpu) => {
+                // The firmware DBDF format has a 16-bit PCI segment field.
+                let segment = u16::try_from(rd.pdev.domain_nr()).map_err(|_| EOVERFLOW)?;
+                let dbdf = (u32::from(segment) << 16) | u32::from(rd.pdev.dev_id());
+                let vm_pid = kernel::current!().tgid().try_into()?;
+                let instance = vgpu.api.open(vgpu.gfid, dbdf, vm_pid)?;
+                let info = instance.type_info();
+                dev.set_device_id(info.pci_dev_id as u16);
+                state.active = true;
+                state.reset_error = None;
+                Some(instance)
+            }
+            None => None,
+        };
+        Ok(Self {
+            registration: rd,
+            instance,
+        })
+    }
+
+    fn bar1_size(&self) -> Result<Option<u64>> {
+        let Some(instance) = &self.instance else {
+            return Ok(None);
+        };
+        let physical = self
+            .registration
+            .pdev
+            .resource_len(self.registration.fb_bar)?;
+        let size = instance
+            .type_info()
+            .bar1_length
+            .checked_mul(1 << 20)
+            .ok_or(EOVERFLOW)?;
+        Ok(Some(if size == 0 {
+            physical
+        } else {
+            size.min(physical)
+        }))
+    }
+
+    fn limit_bar1(&self, buf: &mut vfio::UserBuf, position: &Position<'_>) -> Result {
+        if buf.is_empty() || position.region_index() != self.registration.fb_bar {
+            return Ok(());
+        }
+        if let Some(size) = self.bar1_size()? {
+            let offset = position.region_offset();
+            if offset >= size {
+                return Err(EINVAL);
+            }
+            if size - offset < buf.len() as u64 {
+                buf.truncate((size - offset) as usize);
+            }
+        }
+        Ok(())
+    }
+}
+
+impl Drop for NovaVgpuOpenData<'_, '_> {
+    fn drop(&mut self) {
+        let mut state = self.registration.lock_instance();
+        state.active = false;
+        drop(self.instance.take());
+    }
+}
+
+impl vfio::pci::Operations for NovaVgpuOps {
+    const NAME: &'static CStr = c"nova-vgpu-vfio-pci";
+    type RegistrationData<'a> = NovaVgpuRegData<'a>;
+    type OpenData<'a, 'data: 'a> = NovaVgpuOpenData<'a, 'data>;
+
+    fn open_device<'a, 'data: 'a>(
+        dev: &'a vfio::pci::Device<Self, Open>,
+        rd: &'a NovaVgpuRegData<'data>,
+    ) -> impl PinInit<Self::OpenData<'a, 'data>, Error> + 'a {
+        NovaVgpuOpenData::new(dev, rd)
+    }
+
+    fn ioctl<'data>(
+        dev: &vfio::pci::Device<Self, Ioctl>,
+        rd: &NovaVgpuRegData<'data>,
+        _open_data: Pin<&Self::OpenData<'_, 'data>>,
+        cmd: u32,
+        arg: usize,
+    ) -> Result<isize> {
+        let reset = cmd == vfio::DEVICE_RESET || cmd == vfio::pci::DEVICE_PCI_HOT_RESET;
+        if reset {
+            if let Some(error) = rd.state.lock().reset_error {
+                return Err(error);
+            }
+        }
+        let result = dev.core_ioctl(cmd, arg)?;
+        if reset {
+            if let Some(error) = rd.state.lock().reset_error {
+                return Err(error);
+            }
+        }
+        Ok(result)
+    }
+
+    fn read<'data>(
+        dev: &vfio::pci::Device<Self, Read>,
+        _rd: &NovaVgpuRegData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        buf: &mut vfio::UserBuf,
+        position: &mut Position<'_>,
+    ) -> Result<isize> {
+        if let Some(instance) = &open_data.instance {
+            if position.region_index() == vfio::pci::CONFIG_REGION_INDEX {
+                let offset = position.region_offset();
+                return position.with_temporary(|next| {
+                    let count = dev.core_read(buf, next)?;
+                    buf.write_overlapping(
+                        offset,
+                        count.try_into()?,
+                        u64::from(bindings::PCI_SUBSYSTEM_ID),
+                        &(instance.type_info().pci_subsys_id as u16).to_le_bytes(),
+                    )?;
+                    Ok(count)
+                });
+            }
+        }
+        open_data.limit_bar1(buf, position)?;
+        dev.core_read(buf, position)
+    }
+
+    fn write<'data>(
+        dev: &vfio::pci::Device<Self, Write>,
+        _rd: &NovaVgpuRegData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        buf: &mut vfio::UserBuf,
+        position: &mut Position<'_>,
+    ) -> Result<isize> {
+        open_data.limit_bar1(buf, position)?;
+        dev.core_write(buf, position)
+    }
+
+    fn mmap<'data>(
+        dev: &vfio::pci::Device<Self, Mmap>,
+        rd: &NovaVgpuRegData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        mapping: &mut Mapping<'_>,
+    ) -> Result {
+        if mapping.region_index() == rd.fb_bar {
+            if let Some(size) = open_data.bar1_size()? {
+                if mapping.region_end()? > size {
+                    return Err(EINVAL);
+                }
+            }
+        }
+        dev.core_mmap(mapping)
+    }
+
+    fn get_region_info<'data>(
+        dev: &vfio::pci::Device<Self, GetRegionInfo>,
+        rd: &NovaVgpuRegData<'data>,
+        open_data: Pin<&Self::OpenData<'_, 'data>>,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut vfio::InfoCap<'_>,
+    ) -> Result {
+        dev.core_get_region_info(info, caps)?;
+        if info.index == rd.fb_bar && info.size != 0 {
+            if let Some(size) = open_data.bar1_size()? {
+                info.size = info.size.min(size);
+            }
+        }
+        Ok(())
+    }
+
+    fn reset_prepare(rd: &NovaVgpuRegData<'_>) {
+        // PCI holds the device lock, and VFIO may hold its memory lock. Open and
+        // close release this mutex before entering the VFIO core.
+        let mut state = rd.state.lock();
+        state.resetting = true;
+        if state.active {
+            if let Some(vgpu) = &rd.vgpu {
+                if let Err(error) = vgpu.api.reset(vgpu.gfid) {
+                    state.reset_error.get_or_insert(error);
+                }
+            }
+        }
+    }
+
+    fn reset_done(rd: &NovaVgpuRegData<'_>) -> Result {
+        let mut state = rd.state.lock();
+        let error = state.reset_error;
+        state.resetting = false;
+        rd.reset_done.notify_all();
+        error.map_or(Ok(()), Err)
+    }
+}
+
+struct NovaVgpuDriver;
+
+kernel::pci_device_table!(
+    PCI_TABLE,
+    <NovaVgpuDriver as pci::Driver>::IdInfo,
+    [(
+        pci::DeviceId::from_id_vfio_override(pci::Vendor::NVIDIA, bindings::PCI_ANY_ID as u32),
+        (),
+    ),]
+);
+
+#[pin_data]
+struct NovaVgpuData<'bound> {
+    reg: vfio::pci::Registration<'bound, NovaVgpuOps>,
+}
+
+#[vtable]
+impl pci::Driver for NovaVgpuDriver {
+    type IdInfo = ();
+    type Data<'bound> = NovaVgpuData<'bound>;
+    const ID_TABLE: pci::IdTable<Self::IdInfo> = &PCI_TABLE;
+    // SAFETY: VFIO hands the DMA ownership of the device to userspace. This driver never uses
+    // the kernel DMA API for its devices, directly or through helpers.
+    const DRIVER_MANAGED_DMA: Option<pci::DriverManagedDma<Self>> =
+        Some(unsafe { pci::DriverManagedDma::new() });
+
+    fn probe<'bound>(
+        pdev: &'bound pci::Device<Core<'_>>,
+        _info: Option<&'bound Self::IdInfo>,
+    ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+        try_pin_init!(NovaVgpuData {
+            reg: {
+                let vgpu = match NovaCoreVfApi::handle(pdev) {
+                    Ok(api) => Some(VgpuRegistration {
+                        api,
+                        gfid: u32::from(pdev.vf_id()?) + 1,
+                    }),
+                    Err(_) => None,
+                };
+                let dev = if vgpu.is_some() {
+                    vfio::pci::Device::<NovaVgpuOps>::new(pdev)?
+                } else {
+                    vfio::pci::Device::<NovaVgpuOps>::new_passthrough(pdev)?
+                };
+                let fb_bar = if pdev.resource_flags(0)?.contains(Flags::IORESOURCE_MEM_64) {
+                    2
+                } else {
+                    1
+                };
+                // SAFETY: The registration is stored in `NovaVgpuData`, this driver's bus
+                // device private data, which is never leaked.
+                unsafe {
+                    vfio::pci::Registration::new::<Self>(
+                        pdev,
+                        &dev,
+                        try_pin_init!(NovaVgpuRegData {
+                            pdev,
+                            vgpu,
+                            fb_bar,
+                            state <- kernel::new_mutex!(InstanceState::default()),
+                            reset_done <- kernel::new_condvar!(),
+                        }),
+                    )?
+                }
+            },
+        })
+    }
+
+    fn reset_prepare<'bound>(_pdev: &pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) {
+        data.reg.reset_prepare();
+    }
+
+    fn reset_done<'bound>(pdev: &pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) {
+        if let Err(error) = data.reg.reset_done() {
+            dev_err!(pdev, "VFIO device reset failed: {:?}\n", error);
+            data.reg.error_detected();
+        }
+    }
+
+    fn error_detected<'bound>(
+        _pdev: &pci::Device<Core<'_>>,
+        data: Pin<&Self::Data<'bound>>,
+        _state: pci::ChannelState,
+    ) -> pci::ErsResult {
+        data.reg.error_detected()
+    }
+}
+
+kernel::module_pci_driver! {
+    type: NovaVgpuDriver,
+    name: "nova-vgpu-vfio-pci",
+    authors: ["NVIDIA"],
+    description: "NVIDIA vGPU VFIO variant driver",
+    license: "GPL v2",
+}
-- 
2.43.7


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-11-19 21:54 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:50 [PATCH v3 0/9] Rust VFIO PCI support and NVIDIA vGPU driver Simon Song
2026-11-19 21:50 ` [PATCH v3 1/9] rust: pci: add VFIO override device IDs Simon Song
2026-11-19 21:50 ` [PATCH v3 2/9] rust: pci: expose the VF index within its PF Simon Song
2026-11-19 21:50 ` [PATCH v3 3/9] rust: pci: allow drivers to manage DMA ownership Simon Song
2026-11-19 21:50 ` [PATCH v3 4/9] rust: pci: add reset callbacks Simon Song
2026-11-19 21:50 ` [PATCH v3 5/9] rust: pci: add an error_detected callback Simon Song
2026-11-19 21:50 ` [PATCH v3 6/9] rust: vfio: add common VFIO device handling Simon Song
2026-11-19 21:50 ` [PATCH v3 7/9] rust: vfio: add PCI variant driver abstractions Simon Song
2026-11-19 21:50 ` [PATCH v3 8/9] gpu: nova-core: publish typed SR-IOV PF APIs Simon Song
2026-11-19 21:50 ` [PATCH v3 9/9] vfio/nova-vgpu: add the Rust VFIO variant driver Simon Song

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®