[PATCH v2 00/12] Rust VFIO PCI support and NVIDIA vGPU driver

[PATCH v2 00/12] Rust VFIO PCI support and NVIDIA vGPU driver

[Simon Song]

This updates Zhi Wang's Rust VFIO PCI series [1]. The NVIDIA variant
uses nova-core's typed SR-IOV PF interface to create, reset and release
vGPU instances while VFIO provides userspace access to the PCI device.

The base includes Zhi's BAR1 self-test fix:

- commit 317e76cc60b7 ("gpu: nova-core: mm: keep BAR1 self-test page
  tables alive") [2]

It also includes both prerequisites cherry-picked from Alex Williamson's
VFIO tree:

- commit f1b563d3e3cb ("Merge tag 'topic/pci-vgaarb-rework-2026-10-01'
  of https://gitlab.freedesktop.org/drm/kernel into v7.4/vfio/next") [3]
- commit 6a5f427bee12 ("vfio/pci: Remove the core dependency on driver
  data") [4]

These allow Rust VFIO PCI drivers to use their own typed PCI driver data.

Patches 1-4 provide the PCI helpers and VFIO PCI abstractions. Patches
5-8 add reset, bind, AER and runtime-PM support to the shared PCI adapter.
Patch 9 migrates VFIO to that adapter. Patch 10 separates common VFIO
handling from PCI, patch 11 publishes the typed Nova PF interface, and
patch 12 adds the NVIDIA VFIO variant driver.

Changes since v1:
- Patch 2: restrict vf_id() to Device<device::Bound>, following Danilo's
  review, so VF index lookup requires a bound PCI device.
- Patch 4: adapt to the updated VFIO core driver-data APIs. Pass only the
  core device to vfio_pci_core_register_device() and
  vfio_pci_core_aer_err_detected(), use VFIO core's VGA callback, and
  guard runtime-PM helpers with CONFIG_PM.
- New patches 5-8: add bind, reset, AER and runtime-PM callbacks to
  pci::Driver so VFIO variants can reuse the PCI adapter. Run bind()
  after driver data is installed, and require Sync only for runtime-PM
  driver data.
- New patch 9: remove vfio::pci::Adapter and expose VFIO registration
  through the shared PCI adapter, with a build-time check that VFIO
  drivers install PM callbacks when CONFIG_PM is enabled.
- Patch 12 (v1 patch 7): register the NVIDIA driver through pci::Adapter
  and delegate bind, unbind, reset, AER and runtime PM to its VFIO
  registration.

Integration branch:
https://github.com/xinmengsong/nova-core/tree/nova-vgpu-wip-oct-18

[1]
https://lore.kernel.org/nova-gpu/cover.1790764573.git.zhiw@nvidia.com/
[2]
https://github.com/zhiwang-nvidia/nova-core/commit/317e76cc60b763a03664cbf8244f4de076e58aa7
[3]
https://github.com/awilliam/linux-vfio/commit/f1b563d3e3cb0f9a97ef77f538ffc0e75e79a628
[4]
https://github.com/awilliam/linux-vfio/commit/6a5f427bee12c82beaf13efbd133416da1541c39

Danilo Krummrich (5):
  rust: pci: add VFIO override device IDs
  rust: pci: expose the VF index within its PF
  rust: pci: allow drivers to manage DMA ownership
  rust: vfio: add PCI variant driver abstractions
  vfio/nvidia-vgpu: add the Rust VFIO variant driver

Simon Song (5):
  rust: pci: add reset callbacks
  rust: pci: add a bind callback after installing driver data
  rust: pci: add an error_detected callback
  rust: pci: add runtime PM callbacks
  rust: vfio: use the PCI adapter for variant drivers

Zhi Wang (2):
  rust: vfio: separate common device handling from PCI
  gpu: nova-core: publish typed SR-IOV PF APIs

 drivers/gpu/Makefile                        |  15 +-
 drivers/gpu/nova-core/driver.rs             |  47 ++
 drivers/gpu/nova-core/nova_core.rs          |   2 +
 drivers/gpu/nova-core/vgpu/vgpu_api.rs      |  82 +-
 drivers/vfio/pci/Kconfig                    |   2 +
 drivers/vfio/pci/Makefile                   |   2 +
 drivers/vfio/pci/nvidia-vgpu/Kconfig        |  20 +
 drivers/vfio/pci/nvidia-vgpu/Makefile       |   3 +
 drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs | 402 +++++++++
 rust/bindings/bindings_helper.h             |   4 +
 rust/kernel/lib.rs                          |   2 +
 rust/kernel/pci.rs                          | 229 ++++++
 rust/kernel/pci/iov.rs                      |   8 +
 rust/kernel/vfio.rs                         | 290 +++++++
 rust/kernel/vfio/pci.rs                     | 849 ++++++++++++++++++++
 15 files changed, 1953 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
 create mode 100644 rust/kernel/vfio.rs
 create mode 100644 rust/kernel/vfio/pci.rs

base-commit: 07484ea6c8335226301113bc86e7c11b52312f51
-- 
2.43.7

[PATCH v2 01/12] rust: pci: add VFIO override device IDs

[Simon Song]

From: Danilo Krummrich <dakr@kernel.org>

VFIO variant drivers need override-only IDs and VFIO-specific module
aliases so userspace can select them for PCI assignment.

Add a DeviceId constructor equivalent to PCI_DRIVER_OVERRIDE_DEVICE_VFIO,
matching a vendor and device without restricting the PCI class.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index d83492e06ce..10864ffda4e 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -206,6 +206,13 @@ macro_rules! module_pci_driver {
 impl DeviceId {
     const PCI_ANY_ID: u32 = !0;
 
+    /// Equivalent to C's `PCI_DRIVER_OVERRIDE_DEVICE_VFIO` macro.
+    pub const fn from_id_vfio_override(vendor: Vendor, device: u32) -> Self {
+        let mut id = Self::from_id(vendor, device);
+        id.0.override_only = bindings::PCI_ID_F_VFIO_DRIVER_OVERRIDE;
+        id
+    }
+
     /// Equivalent to C's `PCI_DEVICE` macro.
     ///
     /// Create a new `pci::DeviceId` from a vendor and device ID.
-- 
2.43.7

[PATCH v2 02/12] rust: pci: expose the VF index within its PF

[Simon Song]

From: Danilo Krummrich <dakr@kernel.org>

Nova identifies a vGPU instance by the VF's index within its PF. Expose
pci_iov_vf_id() through Device::vf_id() so a Rust VF driver can obtain
that index, with an error for devices that are not VFs.

Restrict vf_id() to Device<device::Bound>

Suggested-by: Danilo Krummrich <dakr@kernel.org>
Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci/iov.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/rust/kernel/pci/iov.rs b/rust/kernel/pci/iov.rs
index 9d9bd3ac702..76b2e395ca3 100644
--- a/rust/kernel/pci/iov.rs
+++ b/rust/kernel/pci/iov.rs
@@ -329,6 +329,14 @@ unsafe impl<F: ForLt> Send for VfRegistration<'_, F> where for<'a> F::Of<'a>: Se
 unsafe impl<F: ForLt> Sync for VfRegistration<'_, F> where for<'a> F::Of<'a>: Send + Sync {}
 
 impl Device<device::Bound> {
+    /// Return the zero-based VF index within its PF, or an error for a non-VF device.
+    pub fn vf_id(&self) -> Result<u32> {
+        // SAFETY: `self.as_raw()` points to a live PCI device; the helper checks VF membership.
+        let id = unsafe { bindings::pci_iov_vf_id(self.as_raw()) };
+        to_result(id)?;
+        Ok(id as u32)
+    }
+
     /// Internal helper: reads the `vf_registration_data_rust` pointer from the
     /// PF, checks the `TypeId`, and returns a pinned reference.
     ///
-- 
2.43.7

[PATCH v2 03/12] rust: pci: allow drivers to manage DMA ownership

[Simon Song]

From: Danilo Krummrich <dakr@kernel.org>

VFIO manages DMA ownership through the userspace IOMMU interface. Its
PCI variant drivers must opt out of the driver's default DMA domain.

Add Driver::DRIVER_MANAGED_DMA and pass it to the PCI core. Existing
Rust PCI drivers keep the default setting.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 10864ffda4e..a156927e23d 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -108,6 +108,7 @@ unsafe fn register(
             (*pdrv.get()).probe = Some(Self::probe_callback);
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
+            (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA;
             #[cfg(CONFIG_PCI_IOV)]
             if T::HAS_SRIOV_ENABLE {
                 (*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
@@ -345,6 +346,9 @@ pub trait Driver {
     /// The table of device ids supported by the driver.
     const ID_TABLE: IdTable<Self::IdInfo>;
 
+    /// Whether the driver manages DMA ownership instead of using the default DMA domain.
+    const DRIVER_MANAGED_DMA: bool = false;
+
     /// PCI driver probe.
     ///
     /// Called when a new pci device is added or discovered. Implementers should
-- 
2.43.7

[PATCH v2 04/12] rust: vfio: add PCI variant driver abstractions

[Simon Song]

From: Danilo Krummrich <dakr@kernel.org>

Add Rust abstractions for PCI drivers that extend vfio-pci-core with
device-specific operations.

Provide Operations, Device and Registration with separate registration
and first-open/last-close data. Allow open data to borrow registration
data, undo PCI enablement if open fails, and drop open data before
closing the core device.

Use a PCI adapter to register VFIO after publishing initialized driver
data and unregister it before unbind and data destruction. Route runtime
PM, AER and reset callbacks through the driver's typed registration.
Let vfio-pci-core select its VGA arbitration callbacks.

Restrict core I/O helpers to their corresponding callback contexts.
Require covariant registration data so callback borrows can safely
shorten its erased binding lifetime. Add bounded user-buffer and region
mapping helpers, and wire up VFIO core support for device features,
matching, IOMMUFD and PASID. Allow passthrough devices to export physical
BARs as DMA buffers.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/bindings/bindings_helper.h |    4 +
 rust/kernel/lib.rs              |    2 +
 rust/kernel/vfio.rs             |  100 +++
 rust/kernel/vfio/pci.rs         | 1095 +++++++++++++++++++++++++++++++
 4 files changed, 1201 insertions(+)
 create mode 100644 rust/kernel/vfio.rs
 create mode 100644 rust/kernel/vfio/pci.rs

diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 930e63290cd..25ae5d30a93 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -93,6 +93,8 @@
 #include <linux/task_work.h>
 #include <linux/tracepoint.h>
 #include <linux/usb.h>
+#include <linux/vfio.h>
+#include <linux/vfio_pci_core.h>
 #include <linux/wait.h>
 #include <linux/workqueue.h>
 #include <linux/xarray.h>
@@ -115,6 +117,8 @@
 const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
 const size_t RUST_CONST_HELPER_ARCH_KMALLOC_MINALIGN = ARCH_KMALLOC_MINALIGN;
 const size_t RUST_CONST_HELPER_PAGE_SIZE = PAGE_SIZE;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_RESET = VFIO_DEVICE_RESET;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_PCI_HOT_RESET = VFIO_DEVICE_PCI_HOT_RESET;
 const size_t RUST_CONST_HELPER_GENLMSG_DEFAULT_SIZE = GENLMSG_DEFAULT_SIZE;
 const gfp_t RUST_CONST_HELPER_GFP_ATOMIC = GFP_ATOMIC;
 const gfp_t RUST_CONST_HELPER_GFP_KERNEL = GFP_KERNEL;
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index f9ef36217bb..261a411ab50 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -146,6 +146,8 @@
 pub mod uaccess;
 #[cfg(CONFIG_USB = "y")]
 pub mod usb;
+#[cfg(CONFIG_VFIO)]
+pub mod vfio;
 pub mod workqueue;
 pub mod xarray;
 
diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
new file mode 100644
index 00000000000..ed93066e2da
--- /dev/null
+++ b/rust/kernel/vfio.rs
@@ -0,0 +1,100 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Virtual Function I/O (VFIO) abstractions.
+
+use crate::{
+    bindings,
+    prelude::*,
+    uaccess::{
+        UserPtr,
+        UserSlice, //
+    },
+};
+
+#[cfg(CONFIG_VFIO_PCI_CORE)]
+pub mod pci;
+
+/// Reset the VFIO device.
+pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
+
+/// Reset the PCI slot or bus containing a VFIO device.
+pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+
+/// Capability buffer supplied by VFIO for a region-info callback.
+#[cfg(CONFIG_VFIO_PCI_CORE)]
+pub struct InfoCap<'a> {
+    raw: &'a mut bindings::vfio_info_cap,
+}
+
+/// Opaque wrapper around a `char __user *` buffer from a VFIO read or write callback.
+///
+/// Provides bounds-checked copying of kernel data into the user-space buffer.
+///
+/// This type cannot be constructed by driver code — it is only created by the
+/// callback trampoline.
+pub struct UserBuf {
+    ptr: UserPtr,
+    count: usize,
+}
+
+impl UserBuf {
+    /// Returns the buffer length in bytes.
+    pub fn len(&self) -> usize {
+        self.count
+    }
+
+    /// Returns `true` when the buffer is empty.
+    pub fn is_empty(&self) -> bool {
+        self.count == 0
+    }
+
+    /// Limit this callback's transfer to at most `count` bytes.
+    pub fn truncate(&mut self, count: usize) {
+        self.count = self.count.min(count);
+    }
+
+    /// Overwrite the part of `data` that overlaps a completed region read.
+    ///
+    /// `read_offset` is the starting region offset and `read_len` is the number of
+    /// bytes returned by the read. `data_offset` locates `data` within the same region.
+    pub fn write_overlapping(
+        &self,
+        read_offset: u64,
+        read_len: usize,
+        data_offset: u64,
+        data: &[u8],
+    ) -> Result {
+        if read_len > self.len() {
+            return Err(EINVAL);
+        }
+        let read_end = read_offset.checked_add(read_len as u64).ok_or(EOVERFLOW)?;
+        let data_end = data_offset
+            .checked_add(data.len() as u64)
+            .ok_or(EOVERFLOW)?;
+        let start = read_offset.max(data_offset);
+        let end = read_end.min(data_end);
+        if start >= end {
+            return Ok(());
+        }
+
+        // These differences are bounded by `read_len` and `data.len()`.
+        let buf_offset = (start - read_offset) as usize;
+        let data_start = (start - data_offset) as usize;
+        let data_end = (end - data_offset) as usize;
+        self.write_at(buf_offset, &data[data_start..data_end])
+    }
+
+    /// Copy `data` to the user buffer at byte offset `offset`.
+    ///
+    /// Returns [`EFAULT`] if the copy fails, [`EINVAL`] if the write would
+    /// exceed the buffer bounds.
+    fn write_at(&self, offset: usize, data: &[u8]) -> Result {
+        let end = offset.checked_add(data.len()).ok_or(EOVERFLOW)?;
+        if end > self.count {
+            return Err(EINVAL);
+        }
+        let dest = self.ptr.wrapping_byte_add(offset);
+        let mut writer = UserSlice::new(dest, data.len()).writer();
+        writer.write_slice(data)
+    }
+}
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
new file mode 100644
index 00000000000..43c14b4e3a1
--- /dev/null
+++ b/rust/kernel/vfio/pci.rs
@@ -0,0 +1,1095 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! VFIO PCI variant driver abstractions.
+//!
+//! Provides Rust abstractions for writing VFIO PCI variant drivers (also known
+//! as "VFIO PCI core" drivers). A [`Registration`] struct owns the VFIO device
+//! and its associated registration data, tying resource lifetimes to the PCI
+//! driver's binding scope.
+//!
+//! C header: [`include/linux/vfio_pci_core.h`](srctree/include/linux/vfio_pci_core.h)
+
+use super::{
+    InfoCap,
+    UserBuf, //
+};
+
+use crate::{
+    alloc::allocator::Kmalloc,
+    bindings,
+    device,
+    device_id::RawDeviceIdIndex,
+    driver,
+    error::{
+        from_err_ptr,
+        from_result,
+        to_result, //
+    },
+    mm::virt::VmaRef,
+    pci,
+    prelude::*,
+    sync::aref::{
+        ARef,
+        AlwaysRefCounted, //
+    },
+    types::{
+        CovariantForLt,
+        ForLt,
+        NotThreadSafe,
+        Opaque, //
+    },
+    uaccess::UserPtr,
+    ThisModule, //
+};
+use core::{
+    alloc::Layout,
+    cell::UnsafeCell,
+    marker::PhantomData,
+    ptr::NonNull, //
+};
+
+/// Index of the PCI configuration-space region.
+pub const CONFIG_REGION_INDEX: u32 = bindings::VFIO_PCI_CONFIG_REGION_INDEX;
+
+/// The context in which a VFIO PCI device reference is valid.
+///
+/// Callback views can only be borrowed from the corresponding VFIO trampoline.
+/// They cannot be refcounted or shared across threads.
+pub trait DeviceContext: private::Sealed {}
+
+/// An ordinary device reference, without callback-specific operations.
+pub struct Normal;
+
+/// The device is enabled but has not yet completed its first-open callback.
+pub struct Open;
+
+/// The device is borrowed for a VFIO ioctl callback on the current thread.
+pub struct Ioctl;
+
+/// The device is borrowed for a VFIO read callback on the current thread.
+pub struct Read;
+
+/// The device is borrowed for a VFIO write callback on the current thread.
+pub struct Write;
+
+/// The device is borrowed for a VFIO mmap callback on the current thread.
+pub struct Mmap;
+
+/// The device is borrowed for a VFIO region-info callback on the current thread.
+pub struct GetRegionInfo;
+
+mod private {
+    pub trait Sealed {}
+
+    impl Sealed for super::Normal {}
+    impl Sealed for super::Open {}
+    impl Sealed for super::Write {}
+    impl Sealed for super::Mmap {}
+    impl Sealed for super::Ioctl {}
+    impl Sealed for super::Read {}
+    impl Sealed for super::GetRegionInfo {}
+}
+
+impl DeviceContext for Normal {}
+impl DeviceContext for Open {}
+impl DeviceContext for Write {}
+impl DeviceContext for Mmap {}
+impl DeviceContext for Ioctl {}
+impl DeviceContext for Read {}
+impl DeviceContext for GetRegionInfo {}
+
+/// Connect a PCI driver's private data to its VFIO registration.
+///
+/// # Safety
+///
+/// `registration` must always select this PCI device's sole registration and must not sleep.
+/// The driver must use [`Adapter`], and its `probe` and `unbind` callbacks must not enable SR-IOV.
+pub unsafe trait Driver: for<'a> pci::Driver<Data<'a>: Sync> + Sized {
+    /// The VFIO operations implemented by this driver.
+    type Operations: Operations;
+
+    /// Select the registration without blocking or changing its identity.
+    fn registration<'a, 'bound>(
+        data: Pin<&'a Self::Data<'bound>>,
+    ) -> &'a Registration<'bound, Self::Operations>;
+}
+
+/// Register a VFIO PCI variant driver and coordinate its callback lifetimes.
+///
+/// The adapter publishes complete private data before registering VFIO, and unregisters VFIO
+/// before calling the driver's optional `unbind` callback or removing private data. SR-IOV
+/// enable and disable callbacks are not supported by this adapter.
+pub struct Adapter<D: Driver>(D);
+
+// SAFETY: The C PCI driver embeds `device_driver` at the stated offset; its private data is
+// the pinned `D::Data` installed by this adapter's probe callback.
+unsafe impl<D: Driver> driver::DriverLayout for Adapter<D> {
+    type DriverType = bindings::pci_driver;
+    type DriverData<'bound> = D::Data<'bound>;
+    const DEVICE_DRIVER_OFFSET: usize = core::mem::offset_of!(Self::DriverType, driver);
+}
+
+// SAFETY: The driver core unregisters only a successfully registered, still-live PCI driver.
+unsafe impl<D: Driver> driver::RegistrationOps for Adapter<D> {
+    unsafe fn register(
+        pdrv: &Opaque<Self::DriverType>,
+        name: &'static CStr,
+        module: &'static ThisModule,
+    ) -> Result {
+        build_assert!(
+            D::DRIVER_MANAGED_DMA,
+            "VFIO drivers must manage DMA ownership"
+        );
+        #[cfg(CONFIG_PCI_IOV)]
+        build_assert!(
+            !D::HAS_SRIOV_ENABLE && !D::HAS_SRIOV_DISABLE,
+            "VFIO PCI variant drivers do not support SR-IOV configuration"
+        );
+
+        // SAFETY: The driver is being initialized and is not yet visible to the PCI core.
+        unsafe {
+            (*pdrv.get()).name = name.as_char_ptr();
+            (*pdrv.get()).probe = Some(Self::probe_callback);
+            (*pdrv.get()).remove = Some(Self::remove_callback);
+            (*pdrv.get()).id_table = D::ID_TABLE.as_ptr();
+            (*pdrv.get()).driver_managed_dma = D::DRIVER_MANAGED_DMA;
+            (*pdrv.get()).err_handler = &PciCallbacks::<D>::ERROR_HANDLERS;
+            (*pdrv.get()).driver.pm = &PciCallbacks::<D>::PM_OPS;
+        }
+
+        // SAFETY: The caller keeps this initialized driver pinned until unregistration.
+        to_result(unsafe {
+            bindings::__pci_register_driver(pdrv.get(), module.as_ptr(), name.as_char_ptr())
+        })
+    }
+
+    unsafe fn unregister(pdrv: &Opaque<Self::DriverType>) {
+        // SAFETY: The registration contract guarantees a live, registered PCI driver.
+        unsafe { bindings::pci_unregister_driver(pdrv.get()) };
+    }
+}
+
+impl<D: Driver> Adapter<D> {
+    extern "C" fn probe_callback(
+        pdev: *mut bindings::pci_dev,
+        id: *const bindings::pci_device_id,
+    ) -> c_int {
+        // SAFETY: PCI supplies a live device and holds its lock throughout probe.
+        let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
+        // SAFETY: `DeviceId` is a transparent wrapper with no additional invariants.
+        let id = unsafe { &*id.cast::<pci::DeviceId>() };
+        // SAFETY: The ID comes from `D::ID_TABLE`, a dynamic ID with matching driver_data,
+        // or the wildcard ID whose driver_data is zero.
+        let info = unsafe { id.info_unchecked_opt::<D::IdInfo>() };
+
+        from_result(|| {
+            pdev.as_ref().set_drvdata(D::probe(pdev, info))?;
+            // SAFETY: The fully initialized, pinned `D::Data` was just installed above.
+            let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
+            // SAFETY: `D` selects this device's sole registration. Private data is installed
+            // before VFIO can enable runtime PM.
+            if let Err(error) = unsafe { D::registration(data).register() } {
+                // SAFETY: Failed VFIO registration has unwound its callbacks. In particular,
+                // rejecting a PF with enabled VFs must not disable those VFs during rollback.
+                drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
+                return Err(error);
+            }
+            Ok(0)
+        })
+    }
+
+    extern "C" fn remove_callback(pdev: *mut bindings::pci_dev) {
+        // SAFETY: PCI supplies a live device under its lock, after quiescing runtime PM.
+        let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
+        // SAFETY: A successful probe installed this device's complete private data.
+        let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
+        // SAFETY: Probe registered this object. Private data and all its fields remain live
+        // while VFIO unregister drains VFIO, VGA and PM callbacks.
+        unsafe { D::registration(data).unregister() };
+        D::unbind(pdev, data);
+        // SAFETY: Callbacks have drained, so private data can now be removed and destroyed.
+        drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
+    }
+}
+
+/// The trait for VFIO PCI variant driver implementations.
+///
+/// Unoverridden callbacks delegate to the corresponding PCI core operation.
+///
+/// # Lifetime of `RegistrationData`
+///
+/// `RegistrationData` describes a covariant type family borrowing from the PCI
+/// binding scope. [`Registration`] owns the data, and each callback borrows it.
+/// Covariance permits shortening the binding lifetime to the callback's borrow
+/// without allowing callback-local references to be stored in registration data.
+pub trait Operations: Sized + Send + Sync + 'static {
+    /// The name of the VFIO driver.
+    const NAME: &'static CStr;
+
+    /// Data owned by the [`Registration`] and passed to VFIO callbacks.
+    ///
+    /// Per-device state (e.g. the GFID, cached type info) lives here.
+    type RegistrationData: for<'a> CovariantForLt<Of<'a>: Send + Sync> + 'static;
+
+    /// Data shared by callbacks from the first successful open until the last close.
+    /// May borrow registration data; dropped before PCI core close on the last close.
+    type OpenData<'a>: Send + Sync + 'a;
+
+    /// Called when the first file descriptor is opened for this device.
+    ///
+    /// `vfio_pci_core_enable()` has already succeeded; if this returns an
+    /// error, `vfio_pci_core_disable()` is called automatically.
+    fn open_device<'a>(
+        dev: &'a Device<Self, Open>,
+        reg_data: &'a <Self::RegistrationData as ForLt>::Of<'a>,
+    ) -> impl PinInit<Self::OpenData<'a>, Error> + 'a;
+
+    /// Handle a VFIO ioctl.
+    ///
+    /// The default `vfio_pci_core_ioctl()` is available via
+    /// [`Device::core_ioctl()`] for delegation.
+    fn ioctl<'a>(
+        dev: &Device<Self, Ioctl>,
+        reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        cmd: u32,
+        arg: usize,
+    ) -> Result<isize>;
+
+    /// Read from the device.
+    ///
+    /// `buf` is a user-space buffer provided by the VFIO core. Use
+    /// [`Device::core_read()`] to delegate the default read, and
+    /// [`UserBuf::write_overlapping()`] to override specific bytes afterwards.
+    fn read<'a>(
+        dev: &Device<Self, Read>,
+        reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        buf: &mut UserBuf,
+        ppos: &mut Position<'_>,
+    ) -> Result<isize>;
+
+    /// Write to the device, delegating to [`Device::core_write()`] if appropriate.
+    fn write<'a>(
+        dev: &Device<Self, Write>,
+        _reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+        _open_data: Pin<&Self::OpenData<'a>>,
+        buf: &mut UserBuf,
+        ppos: &mut Position<'_>,
+    ) -> Result<isize> {
+        dev.core_write(buf, ppos)
+    }
+
+    /// Map a device region, delegating to [`Device::core_mmap()`] if appropriate.
+    fn mmap<'a>(
+        dev: &Device<Self, Mmap>,
+        _reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+        _open_data: Pin<&Self::OpenData<'a>>,
+        mapping: &mut Mapping<'_>,
+    ) -> Result {
+        dev.core_mmap(mapping)
+    }
+
+    /// Prepare for PCI reset, including resets initiated during enable or close.
+    fn reset_prepare(_reg_data: &<Self::RegistrationData as ForLt>::Of<'_>) {}
+
+    /// Finish PCI reset; errors are reported through the VFIO error interrupt.
+    fn reset_done(_reg_data: &<Self::RegistrationData as ForLt>::Of<'_>) -> Result {
+        Ok(())
+    }
+
+    /// Fill in region info capabilities.
+    ///
+    /// The default `vfio_pci_ioctl_get_region_info()` is available via
+    /// [`Device::core_get_region_info()`] for delegation.
+    fn get_region_info<'a>(
+        dev: &Device<Self, GetRegionInfo>,
+        reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut InfoCap<'_>,
+    ) -> Result;
+}
+
+/// A VFIO PCI file position, encoding a region index and an offset within that region.
+pub struct Position<'a> {
+    raw: &'a mut i64,
+}
+
+impl Position<'_> {
+    /// Commit a temporary position only after the operation succeeds.
+    ///
+    /// This preserves the original position if a read succeeds but a subsequent
+    /// user copy fails.
+    pub fn with_temporary<R>(
+        &mut self,
+        operation: impl FnOnce(&mut Position<'_>) -> Result<R>,
+    ) -> Result<R> {
+        let mut next = *self.raw;
+        let result = operation(&mut Position { raw: &mut next })?;
+        *self.raw = next;
+        Ok(result)
+    }
+
+    /// Returns the PCI region index.
+    pub fn region_index(&self) -> u32 {
+        ((*self.raw as u64) >> bindings::VFIO_PCI_OFFSET_SHIFT) as u32
+    }
+
+    /// Returns the byte offset within the current region.
+    pub fn region_offset(&self) -> u64 {
+        (*self.raw as u64) & ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1)
+    }
+}
+
+/// Mapping metadata supplied by VFIO while the current thread holds the mmap lock.
+///
+/// The underlying VMA can only be passed back to the core mmap operation.
+pub struct Mapping<'a> {
+    vma: &'a VmaRef,
+    _not_thread_safe: NotThreadSafe,
+}
+
+impl Mapping<'_> {
+    fn page_offset(&self) -> u64 {
+        // SAFETY: The mmap callback holds the mmap lock for this VMA.
+        unsafe { (*self.vma.as_ptr()).vm_pgoff as u64 }
+    }
+
+    /// Returns the PCI region index encoded in the mapping offset.
+    pub fn region_index(&self) -> u32 {
+        (self.page_offset() >> (bindings::VFIO_PCI_OFFSET_SHIFT as usize - crate::page::PAGE_SHIFT))
+            as u32
+    }
+
+    /// Returns the byte offset within the PCI region.
+    fn region_offset(&self) -> u64 {
+        let mask = ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1) >> crate::page::PAGE_SHIFT;
+        (self.page_offset() & mask) << crate::page::PAGE_SHIFT
+    }
+
+    /// Returns the requested mapping length in bytes.
+    fn size(&self) -> Result<u64> {
+        self.vma
+            .end()
+            .checked_sub(self.vma.start())
+            .map(|size| size as u64)
+            .ok_or(EOVERFLOW)
+    }
+
+    /// Returns the exclusive end of the requested mapping within the PCI region.
+    pub fn region_end(&self) -> Result<u64> {
+        self.region_offset()
+            .checked_add(self.size()?)
+            .ok_or(EOVERFLOW)
+    }
+}
+
+/// A VFIO PCI variant device, wrapping `struct vfio_pci_core_device`.
+///
+/// The layout is `#[repr(C)]` with `core_device` first, so the
+/// `vfio_device` embedded at offset 0 of `vfio_pci_core_device` is also at
+/// offset 0 of `Self`, matching the `vfio_alloc_device` requirement.
+/// Callback contexts expose only their matching core helper and cannot be
+/// refcounted or shared across threads.
+///
+/// # Invariants
+///
+/// - Callback contexts are only borrowed for the corresponding callback and thread.
+/// - `core_device` was initialized by `vfio_pci_core_init_dev()`.
+/// - The VFIO device refcount owns the allocation lifetime.
+/// - `reg_data` is dangling without an owner or points to a valid
+///   `<T::RegistrationData as ForLt>::Of<'_>` owned by the enclosing [`Registration`].
+/// - `open_data` is null outside a successful open, otherwise it owns a pinned
+///   `KBox<T::OpenData<'_>>` until last close. Only open/close mutate it; VFIO
+///   prevents I/O callbacks from racing those transitions. The data is dropped
+///   before PCI core close and before registration data is freed.
+#[repr(C)]
+pub struct Device<T: Operations, Ctx: DeviceContext = Normal> {
+    core_device: Opaque<bindings::vfio_pci_core_device>,
+    reg_data: UnsafeCell<NonNull<<T::RegistrationData as ForLt>::Of<'static>>>,
+    open_data: UnsafeCell<*mut T::OpenData<'static>>,
+    _context: PhantomData<(Ctx, NotThreadSafe)>,
+}
+
+impl<T: Operations> Device<T> {
+    /// Allocate a VFIO PCI device for subsequent registration.
+    pub fn new(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+        Self::allocate(pdev, false)
+    }
+
+    /// Allocate a device whose physical BARs may also be exported as DMA buffers.
+    pub fn new_passthrough(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+        Self::allocate(pdev, true)
+    }
+
+    fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<ARef<Self>> {
+        const_assert!(core::mem::offset_of!(Self, core_device) == 0);
+        let size = Kmalloc::aligned_layout(Layout::new::<Self>()).size();
+
+        // SAFETY: The bound PCI device and static ops are valid. The allocation
+        // includes the full Rust wrapper with kmalloc-compatible size and alignment.
+        let raw = from_err_ptr(unsafe {
+            bindings::_vfio_alloc_device(size, pdev.as_ref().as_raw(), &Self::OPS)
+        })?;
+        let this = NonNull::new(raw.cast::<Self>()).ok_or(ENOMEM)?;
+
+        // SAFETY: Initialise the Rust field in the newly allocated device.
+        unsafe {
+            (&raw mut (*this.as_ptr()).reg_data).write(UnsafeCell::new(NonNull::dangling()));
+            (&raw mut (*this.as_ptr()).open_data).write(UnsafeCell::new(core::ptr::null_mut()));
+            (&raw mut (*this.as_ptr())._context).write(PhantomData);
+        }
+
+        if passthrough {
+            // SAFETY: No registration or callbacks can observe this allocation yet.
+            unsafe { (*this.as_ref().core_device()).pci_ops = &Self::PASSTHROUGH_OPS };
+        }
+
+        // SAFETY: `this` owns the initial VFIO device reference.
+        Ok(unsafe { ARef::from_raw(this) })
+    }
+
+    /// Access the registration data through a closure with an HRTB lifetime.
+    ///
+    /// The closure's `for<'a>` bound prevents the caller from smuggling
+    /// references with a concrete short lifetime out of the closure.
+    ///
+    /// # Safety
+    ///
+    /// Registration data must be published and remain valid through this call.
+    unsafe fn registration_data_with<R>(
+        &self,
+        f: impl for<'a> FnOnce(&'a <T::RegistrationData as ForLt>::Of<'a>) -> R,
+    ) -> R {
+        // SAFETY: Registration keeps the allocation and its borrowed resources
+        // alive until unregistration completes. Covariance permits shortening
+        // the erased binding lifetime to this callback's borrow.
+        let reg_data: &<T::RegistrationData as ForLt>::Of<'_> = unsafe {
+            (*self.reg_data.get())
+                .cast::<<T::RegistrationData as ForLt>::Of<'_>>()
+                .as_ref()
+        };
+        f(reg_data)
+    }
+
+    /// # Safety
+    ///
+    /// The caller must be an I/O callback while VFIO keeps the device open.
+    unsafe fn callback_data_with<R>(
+        &self,
+        f: impl for<'borrow, 'data> FnOnce(
+            &'borrow <T::RegistrationData as ForLt>::Of<'data>,
+            Pin<&'borrow T::OpenData<'data>>,
+        ) -> R,
+    ) -> R {
+        // SAFETY: VFIO excludes open/close and unregister while this callback runs.
+        unsafe {
+            self.registration_data_with(|reg_data| {
+                let ptr = (*self.open_data.get()).cast::<T::OpenData<'_>>();
+                // The data lifetime stays independent of the callback borrow,
+                // so borrowed OpenData fields cannot be stored back into OpenData.
+                f(reg_data, Pin::new_unchecked(&*ptr))
+            })
+        }
+    }
+
+    /// # Safety
+    /// The returned view must only be used during the callback represented by `Ctx`.
+    unsafe fn with_context<Ctx: DeviceContext>(&self) -> &Device<T, Ctx> {
+        // SAFETY: All contexts have the same layout; the caller guarantees the context.
+        unsafe { &*core::ptr::from_ref(self).cast() }
+    }
+
+    /// Recover `&Self` from a raw `*mut vfio_device` in a callback.
+    ///
+    /// # Safety
+    ///
+    /// `vdev` must point to the `vfio_device` at the start of a
+    /// `Device<T>` allocated by `_vfio_alloc_device()` and remain valid for `'a`.
+    unsafe fn from_vfio_device<'a>(vdev: *mut bindings::vfio_device) -> &'a Self {
+        // SAFETY: `vfio_device` is at offset 0 of `vfio_pci_core_device`,
+        // which is at offset 0 of `Device<T>`, so the pointer cast is
+        // valid.
+        unsafe { &*(vdev.cast()) }
+    }
+}
+
+impl<T: Operations, Ctx: DeviceContext> Device<T, Ctx> {
+    fn core_device(&self) -> *mut bindings::vfio_pci_core_device {
+        self.core_device.get()
+    }
+
+    fn vfio_device(&self) -> *mut bindings::vfio_device {
+        self.core_device.get().cast()
+    }
+}
+
+impl<T: Operations> Device<T, Open> {
+    /// Set the device ID exposed through VFIO's virtual PCI configuration space.
+    pub fn set_device_id(&self, device_id: u16) {
+        // SAFETY: core_enable allocated vconfig; the exclusive first-open callback
+        // runs before finish_enable publishes the configuration to userspace.
+        unsafe {
+            (*self.core_device())
+                .vconfig
+                .add(bindings::PCI_DEVICE_ID as usize)
+                .cast::<u16>()
+                .write_unaligned(device_id.to_le());
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Ioctl> {
+    /// Delegate to `vfio_pci_core_ioctl()`.
+    pub fn core_ioctl(&self, cmd: u32, arg: usize) -> Result<isize> {
+        // SAFETY: The Ioctl context keeps this device open on the callback thread.
+        let ret = unsafe { bindings::vfio_pci_core_ioctl(self.vfio_device(), cmd, arg) };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Read> {
+    /// Delegate to `vfio_pci_core_read()`.
+    ///
+    /// The VFIO core fills the user-space buffer with the default PCI data
+    /// for the region identified by `ppos` and advances the position by the bytes read.
+    pub fn core_read(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+        // SAFETY: The Read context keeps this device open on the callback thread;
+        // `buf.ptr` is a user-space pointer supplied by VFIO.
+        let ret = unsafe {
+            bindings::vfio_pci_core_read(
+                self.vfio_device(),
+                buf.ptr.as_mut_ptr().cast(),
+                buf.count,
+                ppos.raw,
+            )
+        };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Write> {
+    /// Delegate to `vfio_pci_core_write()` for this callback's bounded buffer.
+    pub fn core_write(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+        // SAFETY: The Write context keeps the device open on the callback thread;
+        // the buffer is a userspace pointer supplied by VFIO.
+        let ret = unsafe {
+            bindings::vfio_pci_core_write(
+                self.vfio_device(),
+                buf.ptr.as_const_ptr().cast(),
+                buf.count,
+                ppos.raw,
+            )
+        };
+        if ret < 0 {
+            Err(Error::from_errno(ret as i32))
+        } else {
+            Ok(ret)
+        }
+    }
+}
+
+impl<T: Operations> Device<T, Mmap> {
+    /// Delegate to `vfio_pci_core_mmap()` for the current callback's VMA.
+    pub fn core_mmap(&self, mapping: &mut Mapping<'_>) -> Result {
+        // SAFETY: The Mmap context and Mapping keep the open device and locked VMA
+        // on the callback thread. The VMA cannot be replaced by driver code.
+        to_result(unsafe { bindings::vfio_pci_core_mmap(self.vfio_device(), mapping.vma.as_ptr()) })
+    }
+}
+
+impl<T: Operations> Device<T, GetRegionInfo> {
+    /// Delegate to `vfio_pci_ioctl_get_region_info()`.
+    pub fn core_get_region_info(
+        &self,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut InfoCap<'_>,
+    ) -> Result {
+        // SAFETY: GetRegionInfo keeps the device open on the callback thread;
+        // `InfoCap` preserves the validity of VFIO's capability buffer.
+        to_result(unsafe {
+            bindings::vfio_pci_ioctl_get_region_info(self.vfio_device(), info, caps.raw)
+        })
+    }
+}
+
+// SAFETY: The embedded device reference count owns the VFIO allocation.
+unsafe impl<T: Operations> AlwaysRefCounted for Device<T> {
+    fn inc_ref(&self) {
+        // SAFETY: `self` holds a live VFIO device reference.
+        unsafe { bindings::get_device(&raw mut (*self.vfio_device()).device) };
+    }
+
+    unsafe fn dec_ref(obj: NonNull<Self>) {
+        // SAFETY: The caller owns the reference being released.
+        unsafe { bindings::put_device(&raw mut (*obj.as_ref().vfio_device()).device) };
+    }
+}
+
+// SAFETY: VFIO serializes open/close transitions, and callback data is Send + Sync.
+unsafe impl<T: Operations> Send for Device<T> {}
+
+// SAFETY: I/O callbacks share Sync data. VFIO excludes them from mutations of
+// `open_data` during first open and last close; registration data follows the
+// registration lifetime. Mutable C state is managed by the VFIO core.
+unsafe impl<T: Operations> Sync for Device<T> {}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` in VFIO's first-open context.
+/// VFIO must exclude other opens, closes and I/O callbacks for this call.
+unsafe extern "C" fn open_device_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+) -> core::ffi::c_int {
+    // SAFETY: `vdev` is valid; set by vfio_alloc_device.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+
+    // SAFETY: Enable the PCI-core side first.
+    let ret = unsafe { bindings::vfio_pci_core_enable(dev.core_device()) };
+    if ret != 0 {
+        return ret;
+    }
+
+    // SAFETY: Registration data is initialized before the VFIO device is published.
+    let result = unsafe {
+        dev.registration_data_with(|rd| {
+            // Allocate before calling the driver so allocation failure cannot follow open.
+            let data = KBox::<T::OpenData<'_>>::new_uninit(GFP_KERNEL)?;
+            let data = data.write_pin_init(T::open_device(dev.with_context::<Open>(), rd))?;
+
+            // SAFETY: Only the owning pointer is moved; the allocation remains pinned.
+            let raw =
+                KBox::into_raw(Pin::into_inner_unchecked(data)).cast::<T::OpenData<'static>>();
+            // SAFETY: First open is exclusive with I/O callbacks and last close.
+            // Lifetimes do not affect layout; the device owns the allocation until close.
+            *dev.open_data.get() = raw;
+            Ok::<(), Error>(())
+        })
+    };
+    match result {
+        Ok(()) => {
+            // SAFETY: open succeeded.
+            unsafe { bindings::vfio_pci_core_finish_enable(dev.core_device()) };
+            0
+        }
+        Err(e) => {
+            // SAFETY: Undo the enable on failure.
+            unsafe { bindings::vfio_pci_core_disable(dev.core_device()) };
+            e.to_errno()
+        }
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` whose first open succeeded.
+/// Call exactly once for the last close, after draining I/O callbacks and while
+/// excluding the next first open.
+unsafe extern "C" fn close_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) {
+    // SAFETY: `vdev` is valid.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    // SAFETY: Last close is exclusive with I/O callbacks and the next first open.
+    let raw = unsafe { core::mem::replace(&mut *dev.open_data.get(), core::ptr::null_mut()) };
+    // Run the driver's destructor while registration data and PCI resources are valid.
+    // SAFETY: Successful open transferred this allocation with `KBox::into_raw`.
+    // Last close takes ownership exactly once and drops it without moving the pointee.
+    unsafe { drop(KBox::from_raw(raw)) };
+    // SAFETY: Matches the enable in open_device_cb.
+    unsafe { bindings::vfio_pci_core_close_device(vdev) };
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's ioctl callback
+/// context. Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn ioctl_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    cmd: core::ffi::c_uint,
+    arg: usize,
+) -> isize {
+    // SAFETY: `vdev` is valid.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+    match unsafe {
+        dev.callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
+    } {
+        Ok(v) => v,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's read context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn read_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    buf: *mut u8,
+    count: usize,
+    ppos: *mut i64,
+) -> isize {
+    // SAFETY: `vdev` is valid. `buf` is a valid user-space pointer provided by
+    // the VFIO core. `ppos` is a valid kernel pointer.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    let mut ubuf = UserBuf {
+        ptr: UserPtr::from_ptr(buf.cast()),
+        count,
+    };
+    let mut pos = Position {
+        // SAFETY: `ppos` is a valid kernel pointer provided by the VFIO core.
+        raw: unsafe { &mut *ppos },
+    };
+    // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+    match unsafe {
+        dev.callback_data_with(|rd, od| {
+            T::read(dev.with_context::<Read>(), rd, od, &mut ubuf, &mut pos)
+        })
+    } {
+        Ok(n) => n,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's region-info
+/// context. `info` and `caps` must be valid, disjoint, exclusively accessible
+/// objects; the capability buffer must satisfy the VFIO core's invariants.
+/// Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn get_region_info_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    info: *mut bindings::vfio_region_info,
+    caps: *mut bindings::vfio_info_cap,
+) -> core::ffi::c_int {
+    // SAFETY: `vdev`, `info`, and `caps` are valid kernel pointers provided
+    // by the VFIO core.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    // SAFETY: `info` is a valid pointer provided by the VFIO core.
+    let info = unsafe { &mut *info };
+    let mut caps = InfoCap {
+        // SAFETY: `caps` is a valid pointer provided by the VFIO core.
+        raw: unsafe { &mut *caps },
+    };
+    // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+    match unsafe {
+        dev.callback_data_with(|rd, od| {
+            T::get_region_info(dev.with_context::<GetRegionInfo>(), rd, od, info, &mut caps)
+        })
+    } {
+        Ok(()) => 0,
+        Err(e) => e.to_errno(),
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's write context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn write_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    buf: *const u8,
+    count: usize,
+    ppos: *mut i64,
+) -> isize {
+    // SAFETY: VFIO supplies an open device and a valid file position pointer.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    let mut ubuf = UserBuf {
+        ptr: UserPtr::from_ptr(buf.cast_mut().cast()),
+        count,
+    };
+    let mut pos = Position {
+        // SAFETY: VFIO holds the file-position lock when required.
+        raw: unsafe { &mut *ppos },
+    };
+    // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
+    match unsafe {
+        dev.callback_data_with(|rd, od| {
+            T::write(dev.with_context::<Write>(), rd, od, &mut ubuf, &mut pos)
+        })
+    } {
+        Ok(n) => n,
+        Err(e) => e.to_errno() as isize,
+    }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's mmap context.
+/// `vma` must be the valid VMA being mapped, with the mmap write lock held on
+/// this thread. Registration and open data must remain alive until return.
+unsafe extern "C" fn mmap_cb<T: Operations>(
+    vdev: *mut bindings::vfio_device,
+    vma: *mut bindings::vm_area_struct,
+) -> core::ffi::c_int {
+    // SAFETY: VFIO supplies an open device for this mmap callback.
+    let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+    let mut mapping = Mapping {
+        // SAFETY: The mmap callback holds the mmap lock for this VMA.
+        vma: unsafe { VmaRef::from_raw(vma) },
+        _not_thread_safe: NotThreadSafe,
+    };
+    // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
+    match unsafe {
+        dev.callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
+    } {
+        Ok(()) => 0,
+        Err(e) => e.to_errno(),
+    }
+}
+
+/// Callbacks that recover the VFIO device through its PCI driver's private data.
+struct PciCallbacks<D>(PhantomData<D>);
+
+impl<D: Driver> PciCallbacks<D> {
+    /// # Safety
+    ///
+    /// `dev` must belong to `D`. Its private data must remain installed and valid for `'a`,
+    /// or not yet be installed. The caller must exclude its removal; runtime PM callbacks
+    /// use their own synchronization rather than the PCI device lock.
+    unsafe fn registration<'a>(
+        dev: *mut bindings::device,
+    ) -> Option<&'a Registration<'a, D::Operations>> {
+        // SAFETY: The callback supplies a live device bound to `D`.
+        let data = unsafe { bindings::dev_get_drvdata(dev) };
+        if data.is_null() {
+            return None;
+        }
+        // SAFETY: PCI published a fully initialized, pinned `D::Data`. The caller prevents
+        // its removal, and `D::Data: Sync` permits shared runtime PM callback borrows.
+        let data = unsafe { Pin::new_unchecked(&*data.cast::<D::Data<'_>>()) };
+        Some(D::registration(data))
+    }
+
+    unsafe extern "C" fn reset_prepare(pdev: *mut bindings::pci_dev) {
+        // SAFETY: PCI holds the device lock across reset and registration teardown.
+        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+            return;
+        };
+        // SAFETY: The registration owns callback data throughout bind and unbind.
+        unsafe { reg.dev.registration_data_with(D::Operations::reset_prepare) };
+    }
+
+    unsafe extern "C" fn reset_done(pdev: *mut bindings::pci_dev) {
+        // SAFETY: PCI holds the device lock across reset and registration teardown.
+        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+            return;
+        };
+        // SAFETY: The registration owns callback data throughout bind and unbind.
+        let result = unsafe { reg.dev.registration_data_with(D::Operations::reset_done) };
+        if let Err(error) = result {
+            // SAFETY: The PCI callback keeps its embedded device alive.
+            let dev = unsafe { device::Device::<device::Normal>::from_raw(&raw mut (*pdev).dev) };
+            dev_err!(dev, "VFIO device reset failed: {:?}\n", error);
+            // SAFETY: The initialized VFIO device remains alive for this callback.
+            unsafe {
+                bindings::vfio_pci_core_aer_err_detected(reg.dev.core_device());
+            }
+        }
+    }
+
+    unsafe extern "C" fn error_detected(
+        pdev: *mut bindings::pci_dev,
+        _state: bindings::pci_channel_state_t,
+    ) -> bindings::pci_ers_result_t {
+        // SAFETY: PCI serializes error callbacks with binding and unbinding.
+        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+            return bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER;
+        };
+        // SAFETY: The registered device owns its error notification state.
+        unsafe { bindings::vfio_pci_core_aer_err_detected(reg.dev.core_device()) }
+    }
+
+    #[cfg(CONFIG_PM)]
+    unsafe extern "C" fn runtime_suspend(dev: *mut bindings::device) -> c_int {
+        // SAFETY: Registration enables PM only after private data is installed. PCI remove
+        // and VFIO unregister quiesce runtime PM before private data can be removed.
+        let Some(reg) = (unsafe { Self::registration(dev) }) else {
+            return EBUSY.to_errno();
+        };
+        // SAFETY: This is the runtime-suspend callback for the initialized core device.
+        unsafe { bindings::vfio_pci_core_runtime_suspend(reg.dev.core_device()) }
+    }
+
+    #[cfg(CONFIG_PM)]
+    unsafe extern "C" fn runtime_resume(dev: *mut bindings::device) -> c_int {
+        // SAFETY: Private data remains installed until VFIO unregister has resumed the
+        // device and quiesced PM, including registration failure.
+        let Some(reg) = (unsafe { Self::registration(dev) }) else {
+            return ENODEV.to_errno();
+        };
+        // SAFETY: This is the runtime-resume callback for the initialized core device.
+        unsafe { bindings::vfio_pci_core_runtime_resume(reg.dev.core_device()) }
+    }
+
+    const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
+        error_detected: Some(Self::error_detected),
+        mmio_enabled: None,
+        slot_reset: None,
+        reset_prepare: Some(Self::reset_prepare),
+        reset_done: Some(Self::reset_done),
+        resume: None,
+        cor_error_detected: None,
+    };
+
+    const PM_OPS: bindings::dev_pm_ops = bindings::dev_pm_ops {
+        #[cfg(CONFIG_PM)]
+        runtime_suspend: Some(Self::runtime_suspend),
+        #[cfg(CONFIG_PM)]
+        runtime_resume: Some(Self::runtime_resume),
+        ..pin_init::zeroed()
+    };
+}
+
+/// Build the `vfio_device_ops` vtable for a variant driver `T`.
+///
+/// Open and I/O callbacks dispatch to `T`'s trait implementation.
+/// All other callbacks use the `vfio_pci_core_*` defaults.
+macro_rules! build_ops {
+    ($T:ty) => {
+        bindings::vfio_device_ops {
+            #[allow(clippy::disallowed_methods)]
+            name: <$T>::NAME.as_ptr().cast_mut().cast(),
+            init: Some(bindings::vfio_pci_core_init_dev),
+            release: Some(bindings::vfio_pci_core_release_dev),
+            open_device: Some(open_device_cb::<$T>),
+            close_device: Some(close_device_cb::<$T>),
+            ioctl: Some(ioctl_cb::<$T>),
+            read: Some(read_cb::<$T>),
+            write: Some(write_cb::<$T>),
+            mmap: Some(mmap_cb::<$T>),
+            request: Some(bindings::vfio_pci_core_request),
+            get_region_info_caps: Some(get_region_info_cb::<$T>),
+            match_: Some(bindings::vfio_pci_core_match),
+            match_token_uuid: Some(bindings::vfio_pci_core_match_token_uuid),
+            device_feature: Some(bindings::vfio_pci_core_ioctl_feature),
+            #[cfg(CONFIG_IOMMUFD)]
+            bind_iommufd: Some(bindings::vfio_iommufd_physical_bind),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            bind_iommufd: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            unbind_iommufd: Some(bindings::vfio_iommufd_physical_unbind),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            unbind_iommufd: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            attach_ioas: Some(bindings::vfio_iommufd_physical_attach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            attach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            detach_ioas: Some(bindings::vfio_iommufd_physical_detach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            detach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            pasid_attach_ioas: Some(bindings::vfio_iommufd_physical_pasid_attach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            pasid_attach_ioas: None,
+            #[cfg(CONFIG_IOMMUFD)]
+            pasid_detach_ioas: Some(bindings::vfio_iommufd_physical_pasid_detach_ioas),
+            #[cfg(not(CONFIG_IOMMUFD))]
+            pasid_detach_ioas: None,
+            dma_unmap: None,
+        }
+    };
+}
+
+/// The registration of a VFIO PCI variant device.
+///
+/// Owns both the VFIO device reference and the registration data, tying
+/// resource lifetimes to the PCI binding scope `'a`.
+///
+/// Prepare this owner during probe. [`Adapter`] registers it after private data is installed
+/// and unregisters it before private data is removed. Dropping the owner releases the VFIO
+/// allocation and registration data after callbacks have drained.
+pub struct Registration<'a, T: Operations> {
+    dev: ARef<Device<T>>,
+    _reg_data: Pin<KBox<<T::RegistrationData as ForLt>::Of<'a>>>,
+}
+
+impl<T: Operations> Device<T> {
+    const OPS: bindings::vfio_device_ops = build_ops!(T);
+
+    const PASSTHROUGH_OPS: bindings::vfio_pci_device_ops = bindings::vfio_pci_device_ops {
+        #[cfg(CONFIG_VFIO_PCI_DMABUF)]
+        get_dmabuf_phys: Some(bindings::vfio_pci_core_get_dmabuf_phys),
+        #[cfg(not(CONFIG_VFIO_PCI_DMABUF))]
+        get_dmabuf_phys: None,
+    };
+}
+
+impl<'a, T: Operations> Registration<'a, T> {
+    /// Prepare a previously allocated VFIO PCI device and its callback data.
+    ///
+    /// # Safety
+    ///
+    /// The device must have been allocated during this parent binding and never registered.
+    /// No other `Registration` may own or publish callback data for this device until this
+    /// owner is dropped.
+    /// The caller must hold the PCI device lock. The owner must be stored in the private data
+    /// of a driver registered through [`Adapter`] and selected by [`Driver::registration`].
+    pub unsafe fn new(
+        pdev: &'a pci::Device<device::Core<'_>>,
+        dev: &Device<T>,
+        reg_data: impl PinInit<<T::RegistrationData as ForLt>::Of<'a>, Error>,
+    ) -> Result<Self> {
+        // SAFETY: `dev` holds a live VFIO allocation; only compare its parent pointer.
+        if unsafe { (*dev.vfio_device()).dev } != pdev.as_ref().as_raw() {
+            return Err(EINVAL);
+        }
+
+        let reg_data: Pin<KBox<<T::RegistrationData as ForLt>::Of<'a>>> =
+            KBox::pin_init(reg_data, GFP_KERNEL)?;
+        let ptr: NonNull<<T::RegistrationData as ForLt>::Of<'static>> =
+            NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
+
+        // SAFETY: No concurrent registration or callbacks; publish before registering.
+        unsafe { *dev.reg_data.get() = ptr };
+
+        Ok(Self {
+            dev: dev.into(),
+            _reg_data: reg_data,
+        })
+    }
+
+    /// Register after the PCI adapter has installed the driver's private data.
+    ///
+    /// # Safety
+    ///
+    /// Call once under the PCI device lock, after installing complete driver data whose
+    /// [`Driver::registration`] selector returns `self`. On success, unregister before
+    /// private data is removed or destroyed.
+    unsafe fn register(&self) -> Result {
+        // SAFETY: The caller has published complete private data under the PCI device lock;
+        // it remains accessible during runtime PM setup.
+        to_result(unsafe { bindings::vfio_pci_core_register_device(self.dev.core_device()) })
+    }
+
+    /// Unregister while callbacks can still recover the PCI driver's private data.
+    ///
+    /// # Safety
+    ///
+    /// Call once after successful registration, with the PCI device lock held and private
+    /// data still installed. No private-data fields may have been destroyed.
+    unsafe fn unregister(&self) {
+        // SAFETY: The caller keeps private data installed while unregistration drains VFIO,
+        // VGA and runtime PM callbacks. No field of the enclosing data is being destroyed.
+        unsafe { bindings::vfio_pci_core_unregister_device(self.dev.core_device()) };
+    }
+}
+
+impl<T: Operations> Drop for Registration<'_, T> {
+    fn drop(&mut self) {
+        // SAFETY: The adapter unregisters before removing private data. Failed or unattempted
+        // registration also leaves no callbacks behind.
+        unsafe { *self.dev.reg_data.get() = NonNull::dangling() };
+    }
+}
-- 
2.43.7

[PATCH v2 05/12] rust: pci: add reset callbacks

[Simon Song]

VFIO PCI variant drivers will use reset_prepare() and reset_done() to
forward PCI reset notifications to their VFIO registration.

Adding these callbacks to pci::Driver lets VFIO variants reuse the PCI
adapter's reset callback dispatch. A following patch removes the
duplicate vfio::pci::Adapter.

Because there is no gurantee that drvdata must be valid when reset is
issued, using callback_data_with which contains the helper to null-check
drv_data, this could occur during probe if reset is called before
set_drvdata

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs      | 70 +++++++++++++++++++++++++++++++++++++++++
 rust/kernel/vfio/pci.rs | 12 +++++++
 2 files changed, 82 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index a156927e23d..adc7e92d44e 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -109,6 +109,9 @@ unsafe fn register(
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
             (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA;
+            if T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
+                (*pdrv.get()).err_handler = &Self::ERROR_HANDLERS;
+            }
             #[cfg(CONFIG_PCI_IOV)]
             if T::HAS_SRIOV_ENABLE {
                 (*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
@@ -175,6 +178,61 @@ extern "C" fn remove_callback(pdev: *mut bindings::pci_dev) {
         // Drop private data before returning to PCI core, while its removal context is valid.
         drop(data);
     }
+
+    /// # Safety
+    ///
+    /// `dev` must belong to `T`. Any installed driver data must remain installed until this
+    /// call returns. The data lifetime is independent of the callback borrow.
+    unsafe fn callback_data_with<R>(
+        dev: *mut bindings::device,
+        f: impl for<'borrow, 'data> FnOnce(Pin<&'borrow T::Data<'data>>) -> R,
+    ) -> Option<R> {
+        // SAFETY: The callback supplies a device bound to `T`.
+        let data = unsafe { bindings::dev_get_drvdata(dev) };
+        if data.is_null() {
+            return None;
+        }
+        // SAFETY: PCI installed pinned `T::Data`; the caller excludes its removal.
+        Some(f(unsafe {
+            Pin::new_unchecked(&*data.cast::<T::Data<'_>>())
+        }))
+    }
+
+    extern "C" fn reset_prepare_callback(pdev: *mut bindings::pci_dev) {
+        // SAFETY: PCI holds the device lock throughout reset.
+        let dev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+        // SAFETY: The device lock prevents removal of installed driver data.
+        let _ = unsafe {
+            Self::callback_data_with(&raw mut (*pdev).dev, |data| T::reset_prepare(dev, data))
+        };
+    }
+
+    extern "C" fn reset_done_callback(pdev: *mut bindings::pci_dev) {
+        // SAFETY: PCI holds the device lock throughout reset.
+        let dev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+        // SAFETY: The device lock prevents removal of installed driver data.
+        let _ = unsafe {
+            Self::callback_data_with(&raw mut (*pdev).dev, |data| T::reset_done(dev, data))
+        };
+    }
+
+    const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
+        error_detected: None,
+        mmio_enabled: None,
+        slot_reset: None,
+        reset_prepare: if T::HAS_RESET_PREPARE {
+            Some(Self::reset_prepare_callback)
+        } else {
+            None
+        },
+        reset_done: if T::HAS_RESET_DONE {
+            Some(Self::reset_done_callback)
+        } else {
+            None
+        },
+        resume: None,
+        cor_error_detected: None,
+    };
 }
 
 /// Declares a kernel module that exposes a single PCI driver.
@@ -375,6 +433,18 @@ fn unbind<'bound>(dev: &'bound Device<device::Core<'_>>, this: Pin<&Self::Data<'
         let _ = (dev, this);
     }
 
+    /// Called before a PCI function reset, with the device lock held.
+    fn reset_prepare<'bound>(dev: &Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>) {
+        let _ = (dev, this);
+        build_error!(crate::error::VTABLE_DEFAULT_ERROR)
+    }
+
+    /// Called after a PCI function reset, with the device lock held.
+    fn reset_done<'bound>(dev: &Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>) {
+        let _ = (dev, this);
+        build_error!(crate::error::VTABLE_DEFAULT_ERROR)
+    }
+
     /// Enables Single Root I/O Virtualization (SR-IOV) for a PF.
     ///
     /// Called when userspace writes a nonzero VF count to `sriov_numvfs`. The token carries the
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
index 43c14b4e3a1..db30dbce6a8 100644
--- a/rust/kernel/vfio/pci.rs
+++ b/rust/kernel/vfio/pci.rs
@@ -1084,6 +1084,18 @@ unsafe fn unregister(&self) {
         // VGA and runtime PM callbacks. No field of the enclosing data is being destroyed.
         unsafe { bindings::vfio_pci_core_unregister_device(self.dev.core_device()) };
     }
+
+    /// Prepare the variant driver's state for a PCI reset.
+    pub fn reset_prepare(&self) {
+        // SAFETY: This owner keeps the pinned registration data alive.
+        unsafe { self.dev.registration_data_with(T::reset_prepare) };
+    }
+
+    /// Complete the variant driver's state update after a PCI reset.
+    pub fn reset_done(&self) -> Result {
+        // SAFETY: This owner keeps the pinned registration data alive.
+        unsafe { self.dev.registration_data_with(T::reset_done) }
+    }
 }
 
 impl<T: Operations> Drop for Registration<'_, T> {
-- 
2.43.7

[PATCH v2 06/12] rust: pci: add a bind callback after installing driver data

[Simon Song]

VFIO PCI variant drivers will use bind() to register their VFIO devices
after PCI driver data is installed.

Adding bind() to pci::Driver lets VFIO variants reuse the PCI adapter's
probe and remove paths. A following patch removes the duplicate
vfio::pci::Adapter.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index adc7e92d44e..1951b3c8a5c 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -154,6 +154,13 @@ extern "C" fn probe_callback(
             let data = T::probe(pdev, info);
 
             pdev.as_ref().set_drvdata(data)?;
+            // SAFETY: `set_drvdata` installed the pinned `T::Data` above.
+            let data = unsafe { pdev.as_ref().drvdata_borrow::<T::Data<'_>>() };
+            if let Err(error) = T::bind(pdev, data) {
+                // SAFETY: Failed bind leaves no callbacks borrowing the installed data.
+                drop(unsafe { pdev.as_ref().drvdata_obtain::<T::Data<'_>>() });
+                return Err(error);
+            }
             Ok(0)
         })
     }
@@ -416,6 +423,15 @@ fn probe<'bound>(
         id_info: Option<&'bound Self::IdInfo>,
     ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound;
 
+    /// Complete probe after the pinned driver data has been installed.
+    ///
+    /// Called with the PCI device lock held. On error, the data is dropped without calling
+    /// [`Self::unbind`]. Any callbacks started here must be stopped before returning an error.
+    fn bind<'bound>(dev: &Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>) -> Result {
+        let _ = (dev, this);
+        Ok(())
+    }
+
     /// PCI driver unbind.
     ///
     /// Called when a [`Device`] is unbound from its bound [`Driver`]. Implementing this callback
-- 
2.43.7

[PATCH v2 07/12] rust: pci: add an error_detected callback

[Simon Song]

VFIO PCI variant drivers will use error_detected() to notify userspace
of PCI errors through vfio-pci-core.

Adding error_detected() to pci::Driver lets VFIO variants reuse the PCI
adapter's error callback dispatch. A following patch removes the
duplicate vfio::pci::Adapter.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs      | 33 +++++++++++++++++++++++++++++++--
 rust/kernel/vfio/pci.rs |  6 ++++++
 2 files changed, 37 insertions(+), 2 deletions(-)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 1951b3c8a5c..ddc28099087 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -109,7 +109,7 @@ unsafe fn register(
             (*pdrv.get()).remove = Some(Self::remove_callback);
             (*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
             (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA;
-            if T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
+            if T::HAS_ERROR_DETECTED || T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
                 (*pdrv.get()).err_handler = &Self::ERROR_HANDLERS;
             }
             #[cfg(CONFIG_PCI_IOV)]
@@ -223,8 +223,27 @@ extern "C" fn reset_done_callback(pdev: *mut bindings::pci_dev) {
         };
     }
 
+    extern "C" fn error_detected_callback(
+        pdev: *mut bindings::pci_dev,
+        state: bindings::pci_channel_state_t,
+    ) -> bindings::pci_ers_result_t {
+        // SAFETY: PCI error recovery holds the device lock.
+        let dev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
+        // SAFETY: The device lock prevents removal of installed driver data.
+        unsafe {
+            Self::callback_data_with(&raw mut (*pdev).dev, |data| {
+                T::error_detected(dev, data, state)
+            })
+        }
+        .unwrap_or(bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER)
+    }
+
     const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
-        error_detected: None,
+        error_detected: if T::HAS_ERROR_DETECTED {
+            Some(Self::error_detected_callback)
+        } else {
+            None
+        },
         mmio_enabled: None,
         slot_reset: None,
         reset_prepare: if T::HAS_RESET_PREPARE {
@@ -461,6 +480,16 @@ fn reset_done<'bound>(dev: &Device<device::Core<'_>>, this: Pin<&Self::Data<'bou
         build_error!(crate::error::VTABLE_DEFAULT_ERROR)
     }
 
+    /// Handle a PCI error and return the PCI recovery result, with the device lock held.
+    fn error_detected<'bound>(
+        dev: &Device<device::Core<'_>>,
+        this: Pin<&Self::Data<'bound>>,
+        state: bindings::pci_channel_state_t,
+    ) -> bindings::pci_ers_result_t {
+        let _ = (dev, this, state);
+        build_error!(crate::error::VTABLE_DEFAULT_ERROR)
+    }
+
     /// Enables Single Root I/O Virtualization (SR-IOV) for a PF.
     ///
     /// Called when userspace writes a nonzero VF count to `sriov_numvfs`. The token carries the
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
index db30dbce6a8..0fbc6a920dc 100644
--- a/rust/kernel/vfio/pci.rs
+++ b/rust/kernel/vfio/pci.rs
@@ -1096,6 +1096,12 @@ pub fn reset_done(&self) -> Result {
         // SAFETY: This owner keeps the pinned registration data alive.
         unsafe { self.dev.registration_data_with(T::reset_done) }
     }
+
+    /// Signal the VFIO error interrupt and return the PCI recovery result.
+    pub fn error_detected(&self) -> bindings::pci_ers_result_t {
+        // SAFETY: This owner keeps the initialized VFIO core device alive.
+        unsafe { bindings::vfio_pci_core_aer_err_detected(self.dev.core_device()) }
+    }
 }
 
 impl<T: Operations> Drop for Registration<'_, T> {
-- 
2.43.7

[PATCH v2 08/12] rust: pci: add runtime PM callbacks

[Simon Song]

VFIO PCI variant drivers will use runtime_suspend() and runtime_resume()
to forward runtime PM callbacks to vfio-pci-core.

Adding these callbacks to pci::Driver lets VFIO variants reuse the PCI
adapter's runtime PM callback dispatch. A following patch removes the
duplicate vfio::pci::Adapter.

Require Sync driver data only for runtime PM, whose callbacks run
without the PCI device lock.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/pci.rs | 103 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index ddc28099087..136a817a833 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -112,6 +112,10 @@ unsafe fn register(
             if T::HAS_ERROR_DETECTED || T::HAS_RESET_PREPARE || T::HAS_RESET_DONE {
                 (*pdrv.get()).err_handler = &Self::ERROR_HANDLERS;
             }
+            #[cfg(CONFIG_PM)]
+            if let Some(ops) = T::PM_OPS {
+                (*pdrv.get()).driver.pm = ops.raw;
+            }
             #[cfg(CONFIG_PCI_IOV)]
             if T::HAS_SRIOV_ENABLE {
                 (*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
@@ -261,6 +265,72 @@ extern "C" fn error_detected_callback(
     };
 }
 
+#[cfg(CONFIG_PM)]
+impl<T: Driver> Adapter<T>
+where
+    for<'a> T::Data<'a>: Sync,
+{
+    extern "C" fn runtime_suspend_callback(dev: *mut bindings::device) -> c_int {
+        // SAFETY: PCI supplies its embedded device; runtime PM excludes driver-data removal.
+        let pdev =
+            unsafe { &*container_of!(dev, bindings::pci_dev, dev).cast::<Device<device::Bound>>() };
+        // SAFETY: Runtime PM keeps installed data alive; `T::Data: Sync` permits shared borrows.
+        let result =
+            unsafe { Self::callback_data_with(dev, |data| T::runtime_suspend(pdev, data)) }
+                .unwrap_or(Err(EBUSY));
+        match result {
+            Ok(()) => 0,
+            Err(error) => error.to_errno(),
+        }
+    }
+
+    extern "C" fn runtime_resume_callback(dev: *mut bindings::device) -> c_int {
+        // SAFETY: PCI supplies its embedded device; runtime PM excludes driver-data removal.
+        let pdev =
+            unsafe { &*container_of!(dev, bindings::pci_dev, dev).cast::<Device<device::Bound>>() };
+        // SAFETY: Runtime PM keeps installed data alive; `T::Data: Sync` permits shared borrows.
+        let result = unsafe { Self::callback_data_with(dev, |data| T::runtime_resume(pdev, data)) }
+            .unwrap_or(Err(ENODEV));
+        match result {
+            Ok(()) => 0,
+            Err(error) => error.to_errno(),
+        }
+    }
+
+    const PM_OPS: bindings::dev_pm_ops = bindings::dev_pm_ops {
+        runtime_suspend: Some(Self::runtime_suspend_callback),
+        runtime_resume: Some(Self::runtime_resume_callback),
+        ..pin_init::zeroed()
+    };
+}
+
+/// Runtime PM callbacks for PCI driver `T`, installed through [`Driver::PM_OPS`].
+#[cfg(CONFIG_PM)]
+pub struct PmOps<T: ?Sized> {
+    raw: &'static bindings::dev_pm_ops,
+    // Invariant in `T`, so subtyping can't pass one driver's callbacks off as another's.
+    _driver: PhantomData<fn(&T) -> &T>,
+}
+
+#[cfg(CONFIG_PM)]
+impl<T: Driver> PmOps<T>
+where
+    for<'a> T::Data<'a>: Sync,
+{
+    /// Returns the runtime PM callbacks for `T`, which must override both PM methods.
+    #[expect(clippy::new_without_default)]
+    pub const fn new() -> Self {
+        const_assert!(
+            T::HAS_RUNTIME_SUSPEND && T::HAS_RUNTIME_RESUME,
+            "PCI runtime PM requires both runtime_suspend and runtime_resume"
+        );
+        Self {
+            raw: &Adapter::<T>::PM_OPS,
+            _driver: PhantomData,
+        }
+    }
+}
+
 /// Declares a kernel module that exposes a single PCI driver.
 ///
 /// # Examples
@@ -425,6 +495,9 @@ pub trait Driver {
     type IdInfo: 'static;
 
     /// The type of the driver's bus device private data.
+    ///
+    /// Callbacks access this data under the PCI device lock, except runtime PM callbacks,
+    /// which additionally require [`Sync`].
     type Data<'bound>: Send + 'bound;
 
     /// The table of device ids supported by the driver.
@@ -462,6 +535,9 @@ fn bind<'bound>(dev: &Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>)
     ///
     /// Otherwise, release operations for driver resources should be performed in `Drop`.
     ///
+    /// PCI has drained runtime-PM work before this callback. Driver data remains installed
+    /// until this callback returns.
+    ///
     /// For a PF with enabled VFs, `VfRegistration` disables SR-IOV when it is dropped. This
     /// callback must leave resources accessed by VF drivers available until then.
     fn unbind<'bound>(dev: &'bound Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>) {
@@ -490,6 +566,33 @@ fn error_detected<'bound>(
         build_error!(crate::error::VTABLE_DEFAULT_ERROR)
     }
 
+    /// Prepare for runtime suspend, without holding the PCI device lock.
+    #[cfg(CONFIG_PM)]
+    fn runtime_suspend<'bound>(
+        dev: &Device<device::Bound>,
+        this: Pin<&Self::Data<'bound>>,
+    ) -> Result
+    where
+        for<'a> Self::Data<'a>: Sync,
+    {
+        let _ = (dev, this);
+        build_error!(crate::error::VTABLE_DEFAULT_ERROR)
+    }
+
+    /// Finish runtime resume, without holding the PCI device lock.
+    #[cfg(CONFIG_PM)]
+    fn runtime_resume<'bound>(dev: &Device<device::Bound>, this: Pin<&Self::Data<'bound>>) -> Result
+    where
+        for<'a> Self::Data<'a>: Sync,
+    {
+        let _ = (dev, this);
+        build_error!(crate::error::VTABLE_DEFAULT_ERROR)
+    }
+
+    /// Runtime PM callbacks, if any. Construct with [`PmOps::new`].
+    #[cfg(CONFIG_PM)]
+    const PM_OPS: Option<PmOps<Self>> = None;
+
     /// Enables Single Root I/O Virtualization (SR-IOV) for a PF.
     ///
     /// Called when userspace writes a nonzero VF count to `sriov_numvfs`. The token carries the
-- 
2.43.7

[PATCH v2 09/12] rust: vfio: use the PCI adapter for variant drivers

[Simon Song]

pci::Driver now provides reset, bind, error, pm callbacks, vfio::pci
now can reuse pci adapter's callback instead of maintain its own
separate adapter

Remove vfio::pci::Adapter and its callback dispatch. Expose VFIO
registration and runtime PM operations for drivers using pci::Adapter.

Move the DMA ownership and SR-IOV checks to registration, and require
VFIO drivers to install PM callbacks when CONFIG_PM is enabled.

Suggested-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/vfio/pci.rs | 287 +++++++---------------------------------
 1 file changed, 51 insertions(+), 236 deletions(-)

diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
index 0fbc6a920dc..f3e68a3d18c 100644
--- a/rust/kernel/vfio/pci.rs
+++ b/rust/kernel/vfio/pci.rs
@@ -18,11 +18,8 @@
     alloc::allocator::Kmalloc,
     bindings,
     device,
-    device_id::RawDeviceIdIndex,
-    driver,
     error::{
         from_err_ptr,
-        from_result,
         to_result, //
     },
     mm::virt::VmaRef,
@@ -38,8 +35,7 @@
         NotThreadSafe,
         Opaque, //
     },
-    uaccess::UserPtr,
-    ThisModule, //
+    uaccess::UserPtr, //
 };
 use core::{
     alloc::Layout,
@@ -98,120 +94,6 @@ impl DeviceContext for Ioctl {}
 impl DeviceContext for Read {}
 impl DeviceContext for GetRegionInfo {}
 
-/// Connect a PCI driver's private data to its VFIO registration.
-///
-/// # Safety
-///
-/// `registration` must always select this PCI device's sole registration and must not sleep.
-/// The driver must use [`Adapter`], and its `probe` and `unbind` callbacks must not enable SR-IOV.
-pub unsafe trait Driver: for<'a> pci::Driver<Data<'a>: Sync> + Sized {
-    /// The VFIO operations implemented by this driver.
-    type Operations: Operations;
-
-    /// Select the registration without blocking or changing its identity.
-    fn registration<'a, 'bound>(
-        data: Pin<&'a Self::Data<'bound>>,
-    ) -> &'a Registration<'bound, Self::Operations>;
-}
-
-/// Register a VFIO PCI variant driver and coordinate its callback lifetimes.
-///
-/// The adapter publishes complete private data before registering VFIO, and unregisters VFIO
-/// before calling the driver's optional `unbind` callback or removing private data. SR-IOV
-/// enable and disable callbacks are not supported by this adapter.
-pub struct Adapter<D: Driver>(D);
-
-// SAFETY: The C PCI driver embeds `device_driver` at the stated offset; its private data is
-// the pinned `D::Data` installed by this adapter's probe callback.
-unsafe impl<D: Driver> driver::DriverLayout for Adapter<D> {
-    type DriverType = bindings::pci_driver;
-    type DriverData<'bound> = D::Data<'bound>;
-    const DEVICE_DRIVER_OFFSET: usize = core::mem::offset_of!(Self::DriverType, driver);
-}
-
-// SAFETY: The driver core unregisters only a successfully registered, still-live PCI driver.
-unsafe impl<D: Driver> driver::RegistrationOps for Adapter<D> {
-    unsafe fn register(
-        pdrv: &Opaque<Self::DriverType>,
-        name: &'static CStr,
-        module: &'static ThisModule,
-    ) -> Result {
-        build_assert!(
-            D::DRIVER_MANAGED_DMA,
-            "VFIO drivers must manage DMA ownership"
-        );
-        #[cfg(CONFIG_PCI_IOV)]
-        build_assert!(
-            !D::HAS_SRIOV_ENABLE && !D::HAS_SRIOV_DISABLE,
-            "VFIO PCI variant drivers do not support SR-IOV configuration"
-        );
-
-        // SAFETY: The driver is being initialized and is not yet visible to the PCI core.
-        unsafe {
-            (*pdrv.get()).name = name.as_char_ptr();
-            (*pdrv.get()).probe = Some(Self::probe_callback);
-            (*pdrv.get()).remove = Some(Self::remove_callback);
-            (*pdrv.get()).id_table = D::ID_TABLE.as_ptr();
-            (*pdrv.get()).driver_managed_dma = D::DRIVER_MANAGED_DMA;
-            (*pdrv.get()).err_handler = &PciCallbacks::<D>::ERROR_HANDLERS;
-            (*pdrv.get()).driver.pm = &PciCallbacks::<D>::PM_OPS;
-        }
-
-        // SAFETY: The caller keeps this initialized driver pinned until unregistration.
-        to_result(unsafe {
-            bindings::__pci_register_driver(pdrv.get(), module.as_ptr(), name.as_char_ptr())
-        })
-    }
-
-    unsafe fn unregister(pdrv: &Opaque<Self::DriverType>) {
-        // SAFETY: The registration contract guarantees a live, registered PCI driver.
-        unsafe { bindings::pci_unregister_driver(pdrv.get()) };
-    }
-}
-
-impl<D: Driver> Adapter<D> {
-    extern "C" fn probe_callback(
-        pdev: *mut bindings::pci_dev,
-        id: *const bindings::pci_device_id,
-    ) -> c_int {
-        // SAFETY: PCI supplies a live device and holds its lock throughout probe.
-        let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
-        // SAFETY: `DeviceId` is a transparent wrapper with no additional invariants.
-        let id = unsafe { &*id.cast::<pci::DeviceId>() };
-        // SAFETY: The ID comes from `D::ID_TABLE`, a dynamic ID with matching driver_data,
-        // or the wildcard ID whose driver_data is zero.
-        let info = unsafe { id.info_unchecked_opt::<D::IdInfo>() };
-
-        from_result(|| {
-            pdev.as_ref().set_drvdata(D::probe(pdev, info))?;
-            // SAFETY: The fully initialized, pinned `D::Data` was just installed above.
-            let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
-            // SAFETY: `D` selects this device's sole registration. Private data is installed
-            // before VFIO can enable runtime PM.
-            if let Err(error) = unsafe { D::registration(data).register() } {
-                // SAFETY: Failed VFIO registration has unwound its callbacks. In particular,
-                // rejecting a PF with enabled VFs must not disable those VFs during rollback.
-                drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
-                return Err(error);
-            }
-            Ok(0)
-        })
-    }
-
-    extern "C" fn remove_callback(pdev: *mut bindings::pci_dev) {
-        // SAFETY: PCI supplies a live device under its lock, after quiescing runtime PM.
-        let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
-        // SAFETY: A successful probe installed this device's complete private data.
-        let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
-        // SAFETY: Probe registered this object. Private data and all its fields remain live
-        // while VFIO unregister drains VFIO, VGA and PM callbacks.
-        unsafe { D::registration(data).unregister() };
-        D::unbind(pdev, data);
-        // SAFETY: Callbacks have drained, so private data can now be removed and destroyed.
-        drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
-    }
-}
-
 /// The trait for VFIO PCI variant driver implementations.
 ///
 /// Unoverridden callbacks delegate to the corresponding PCI core operation.
@@ -848,109 +730,6 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     }
 }
 
-/// Callbacks that recover the VFIO device through its PCI driver's private data.
-struct PciCallbacks<D>(PhantomData<D>);
-
-impl<D: Driver> PciCallbacks<D> {
-    /// # Safety
-    ///
-    /// `dev` must belong to `D`. Its private data must remain installed and valid for `'a`,
-    /// or not yet be installed. The caller must exclude its removal; runtime PM callbacks
-    /// use their own synchronization rather than the PCI device lock.
-    unsafe fn registration<'a>(
-        dev: *mut bindings::device,
-    ) -> Option<&'a Registration<'a, D::Operations>> {
-        // SAFETY: The callback supplies a live device bound to `D`.
-        let data = unsafe { bindings::dev_get_drvdata(dev) };
-        if data.is_null() {
-            return None;
-        }
-        // SAFETY: PCI published a fully initialized, pinned `D::Data`. The caller prevents
-        // its removal, and `D::Data: Sync` permits shared runtime PM callback borrows.
-        let data = unsafe { Pin::new_unchecked(&*data.cast::<D::Data<'_>>()) };
-        Some(D::registration(data))
-    }
-
-    unsafe extern "C" fn reset_prepare(pdev: *mut bindings::pci_dev) {
-        // SAFETY: PCI holds the device lock across reset and registration teardown.
-        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
-            return;
-        };
-        // SAFETY: The registration owns callback data throughout bind and unbind.
-        unsafe { reg.dev.registration_data_with(D::Operations::reset_prepare) };
-    }
-
-    unsafe extern "C" fn reset_done(pdev: *mut bindings::pci_dev) {
-        // SAFETY: PCI holds the device lock across reset and registration teardown.
-        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
-            return;
-        };
-        // SAFETY: The registration owns callback data throughout bind and unbind.
-        let result = unsafe { reg.dev.registration_data_with(D::Operations::reset_done) };
-        if let Err(error) = result {
-            // SAFETY: The PCI callback keeps its embedded device alive.
-            let dev = unsafe { device::Device::<device::Normal>::from_raw(&raw mut (*pdev).dev) };
-            dev_err!(dev, "VFIO device reset failed: {:?}\n", error);
-            // SAFETY: The initialized VFIO device remains alive for this callback.
-            unsafe {
-                bindings::vfio_pci_core_aer_err_detected(reg.dev.core_device());
-            }
-        }
-    }
-
-    unsafe extern "C" fn error_detected(
-        pdev: *mut bindings::pci_dev,
-        _state: bindings::pci_channel_state_t,
-    ) -> bindings::pci_ers_result_t {
-        // SAFETY: PCI serializes error callbacks with binding and unbinding.
-        let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
-            return bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER;
-        };
-        // SAFETY: The registered device owns its error notification state.
-        unsafe { bindings::vfio_pci_core_aer_err_detected(reg.dev.core_device()) }
-    }
-
-    #[cfg(CONFIG_PM)]
-    unsafe extern "C" fn runtime_suspend(dev: *mut bindings::device) -> c_int {
-        // SAFETY: Registration enables PM only after private data is installed. PCI remove
-        // and VFIO unregister quiesce runtime PM before private data can be removed.
-        let Some(reg) = (unsafe { Self::registration(dev) }) else {
-            return EBUSY.to_errno();
-        };
-        // SAFETY: This is the runtime-suspend callback for the initialized core device.
-        unsafe { bindings::vfio_pci_core_runtime_suspend(reg.dev.core_device()) }
-    }
-
-    #[cfg(CONFIG_PM)]
-    unsafe extern "C" fn runtime_resume(dev: *mut bindings::device) -> c_int {
-        // SAFETY: Private data remains installed until VFIO unregister has resumed the
-        // device and quiesced PM, including registration failure.
-        let Some(reg) = (unsafe { Self::registration(dev) }) else {
-            return ENODEV.to_errno();
-        };
-        // SAFETY: This is the runtime-resume callback for the initialized core device.
-        unsafe { bindings::vfio_pci_core_runtime_resume(reg.dev.core_device()) }
-    }
-
-    const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
-        error_detected: Some(Self::error_detected),
-        mmio_enabled: None,
-        slot_reset: None,
-        reset_prepare: Some(Self::reset_prepare),
-        reset_done: Some(Self::reset_done),
-        resume: None,
-        cor_error_detected: None,
-    };
-
-    const PM_OPS: bindings::dev_pm_ops = bindings::dev_pm_ops {
-        #[cfg(CONFIG_PM)]
-        runtime_suspend: Some(Self::runtime_suspend),
-        #[cfg(CONFIG_PM)]
-        runtime_resume: Some(Self::runtime_resume),
-        ..pin_init::zeroed()
-    };
-}
-
 /// Build the `vfio_device_ops` vtable for a variant driver `T`.
 ///
 /// Open and I/O callbacks dispatch to `T`'s trait implementation.
@@ -1007,9 +786,8 @@ macro_rules! build_ops {
 /// Owns both the VFIO device reference and the registration data, tying
 /// resource lifetimes to the PCI binding scope `'a`.
 ///
-/// Prepare this owner during probe. [`Adapter`] registers it after private data is installed
-/// and unregisters it before private data is removed. Dropping the owner releases the VFIO
-/// allocation and registration data after callbacks have drained.
+/// Prepare this owner during probe, register it in [`pci::Driver::bind`], and unregister it
+/// in [`pci::Driver::unbind`] before releasing any driver data.
 pub struct Registration<'a, T: Operations> {
     dev: ARef<Device<T>>,
     _reg_data: Pin<KBox<<T::RegistrationData as ForLt>::Of<'a>>>,
@@ -1034,8 +812,8 @@ impl<'a, T: Operations> Registration<'a, T> {
     /// The device must have been allocated during this parent binding and never registered.
     /// No other `Registration` may own or publish callback data for this device until this
     /// owner is dropped.
-    /// The caller must hold the PCI device lock. The owner must be stored in the private data
-    /// of a driver registered through [`Adapter`] and selected by [`Driver::registration`].
+    /// The caller must hold the PCI device lock. Store the owner in the driver's private data
+    /// and keep it alive until unregistration has completed.
     pub unsafe fn new(
         pdev: &'a pci::Device<device::Core<'_>>,
         dev: &Device<T>,
@@ -1064,10 +842,25 @@ pub unsafe fn new(
     ///
     /// # Safety
     ///
-    /// Call once under the PCI device lock, after installing complete driver data whose
-    /// [`Driver::registration`] selector returns `self`. On success, unregister before
-    /// private data is removed or destroyed.
-    unsafe fn register(&self) -> Result {
+    /// Call once from the bound PCI driver `D`'s `bind` callback, with `self` in its installed
+    /// driver data. Its PCI reset, error and runtime-PM callbacks must delegate to this owner.
+    /// After success, unregister before any later bind error or driver-data teardown.
+    pub unsafe fn register<D: pci::Driver>(&self) -> Result {
+        build_assert!(
+            D::DRIVER_MANAGED_DMA,
+            "VFIO drivers must manage DMA ownership"
+        );
+        #[cfg(CONFIG_PM)]
+        build_assert!(
+            D::PM_OPS.is_some(),
+            "VFIO drivers must install runtime PM callbacks"
+        );
+        #[cfg(CONFIG_PCI_IOV)]
+        build_assert!(
+            !D::HAS_SRIOV_ENABLE && !D::HAS_SRIOV_DISABLE,
+            "VFIO PCI variant drivers do not support SR-IOV configuration"
+        );
+
         // SAFETY: The caller has published complete private data under the PCI device lock;
         // it remains accessible during runtime PM setup.
         to_result(unsafe { bindings::vfio_pci_core_register_device(self.dev.core_device()) })
@@ -1077,9 +870,10 @@ unsafe fn register(&self) -> Result {
     ///
     /// # Safety
     ///
-    /// Call once after successful registration, with the PCI device lock held and private
-    /// data still installed. No private-data fields may have been destroyed.
-    unsafe fn unregister(&self) {
+    /// Call once after successful registration, with the PCI device lock held and all driver
+    /// data still installed. Do not call after registration failed: that can disable VFs
+    /// belonging to a PF which VFIO refused to bind.
+    pub unsafe fn unregister(&self) {
         // SAFETY: The caller keeps private data installed while unregistration drains VFIO,
         // VGA and runtime PM callbacks. No field of the enclosing data is being destroyed.
         unsafe { bindings::vfio_pci_core_unregister_device(self.dev.core_device()) };
@@ -1102,12 +896,33 @@ pub fn error_detected(&self) -> bindings::pci_ers_result_t {
         // SAFETY: This owner keeps the initialized VFIO core device alive.
         unsafe { bindings::vfio_pci_core_aer_err_detected(self.dev.core_device()) }
     }
+
+    /// Prepare the VFIO core device for runtime suspend.
+    ///
+    /// # Safety
+    ///
+    /// Only call from the parent PCI driver's runtime-suspend callback.
+    #[cfg(CONFIG_PM)]
+    pub unsafe fn runtime_suspend(&self) -> Result {
+        // SAFETY: The caller supplies the runtime-suspend callback context.
+        to_result(unsafe { bindings::vfio_pci_core_runtime_suspend(self.dev.core_device()) })
+    }
+
+    /// Complete the VFIO core device's runtime resume.
+    ///
+    /// # Safety
+    ///
+    /// Only call from the parent PCI driver's runtime-resume callback.
+    #[cfg(CONFIG_PM)]
+    pub unsafe fn runtime_resume(&self) -> Result {
+        // SAFETY: The caller supplies the runtime-resume callback context.
+        to_result(unsafe { bindings::vfio_pci_core_runtime_resume(self.dev.core_device()) })
+    }
 }
 
 impl<T: Operations> Drop for Registration<'_, T> {
     fn drop(&mut self) {
-        // SAFETY: The adapter unregisters before removing private data. Failed or unattempted
-        // registration also leaves no callbacks behind.
+        // SAFETY: The driver unregistered in `unbind`, or registration did not succeed.
         unsafe { *self.dev.reg_data.get() = NonNull::dangling() };
     }
 }
-- 
2.43.7

[PATCH v2 10/12] rust: vfio: separate common device handling from PCI

[Simon Song]

From: Zhi Wang <zhiw@nvidia.com>

A VFIO PCI device contains a VFIO device. Expose that relationship in
Rust so common device handling is independent of PCI resources.

Add vfio::Device and have vfio::pci::Device borrow its embedded device
through AsRef, preserving the callback context. Share reference counting
and callback-data lifetime handling in vfio. Keep driver operations,
allocation, registration and the PCI enable/close sequence in vfio::pci.

No functional change is intended.

Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 rust/kernel/vfio.rs     | 200 +++++++++++++++++++++++++++++++++++++++-
 rust/kernel/vfio/pci.rs | 199 ++++++++++++---------------------------
 2 files changed, 255 insertions(+), 144 deletions(-)

diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
index ed93066e2da..a0bfbc7c59a 100644
--- a/rust/kernel/vfio.rs
+++ b/rust/kernel/vfio.rs
@@ -1,14 +1,29 @@
 // SPDX-License-Identifier: GPL-2.0
 
-//! Virtual Function I/O (VFIO) abstractions.
+//! Virtual Function I/O (VFIO) devices and callback data.
+//!
+//! Bus-specific device types borrow their embedded VFIO device through [`AsRef`].
+//! C header: [`include/linux/vfio.h`](srctree/include/linux/vfio.h)
 
 use crate::{
     bindings,
     prelude::*,
+    sync::aref::AlwaysRefCounted,
+    types::{
+        CovariantForLt,
+        ForLt,
+        NotThreadSafe,
+        Opaque, //
+    },
     uaccess::{
         UserPtr,
         UserSlice, //
-    },
+    }, //
+};
+use core::{
+    cell::UnsafeCell,
+    marker::PhantomData,
+    ptr::NonNull, //
 };
 
 #[cfg(CONFIG_VFIO_PCI_CORE)]
@@ -17,12 +32,187 @@
 /// Reset the VFIO device.
 pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
 
-/// Reset the PCI slot or bus containing a VFIO device.
-pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+/// The context in which a VFIO device reference is valid.
+///
+/// Callback views can only be borrowed from the corresponding VFIO trampoline.
+/// They cannot be refcounted or shared across threads.
+pub trait DeviceContext: private::Sealed {}
+
+/// An ordinary device reference, without callback-specific operations.
+pub struct Normal;
+
+/// The device is undergoing its exclusive first-open callback.
+pub struct Open;
+
+/// The device is borrowed for a VFIO ioctl callback on the current thread.
+pub struct Ioctl;
+
+/// The device is borrowed for a VFIO read callback on the current thread.
+pub struct Read;
+
+/// The device is borrowed for a VFIO write callback on the current thread.
+pub struct Write;
+
+/// The device is borrowed for a VFIO mmap callback on the current thread.
+pub struct Mmap;
+
+/// The device is borrowed for a VFIO region-info callback on the current thread.
+pub struct GetRegionInfo;
+
+mod private {
+    pub trait Sealed {}
+
+    impl Sealed for super::Normal {}
+    impl Sealed for super::Open {}
+    impl Sealed for super::Write {}
+    impl Sealed for super::Mmap {}
+    impl Sealed for super::Ioctl {}
+    impl Sealed for super::Read {}
+    impl Sealed for super::GetRegionInfo {}
+}
+
+impl DeviceContext for Normal {}
+impl DeviceContext for Open {}
+impl DeviceContext for Write {}
+impl DeviceContext for Mmap {}
+impl DeviceContext for Ioctl {}
+impl DeviceContext for Read {}
+impl DeviceContext for GetRegionInfo {}
+
+/// A VFIO device, independent of its bus-specific wrapper.
+///
+/// # Invariants
+///
+/// The underlying `vfio_device` is initialized and remains alive while borrowed.
+/// Callback contexts are borrowed only for the corresponding callback and thread;
+/// only [`Normal`] references can be reference-counted.
+#[repr(transparent)]
+pub struct Device<Ctx: DeviceContext = Normal> {
+    raw: Opaque<bindings::vfio_device>,
+    _context: PhantomData<(Ctx, NotThreadSafe)>,
+}
+
+impl<Ctx: DeviceContext> Device<Ctx> {
+    /// # Safety
+    ///
+    /// `raw` must point to an initialized `vfio_device` that remains alive for
+    /// `'a`. The caller must provide the guarantees of `Ctx` throughout the borrow.
+    #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
+    unsafe fn from_raw<'a>(raw: *mut bindings::vfio_device) -> &'a Self {
+        // SAFETY: Self is a transparent wrapper; the caller guarantees validity and context.
+        unsafe { &*raw.cast() }
+    }
+
+    fn as_raw(&self) -> *mut bindings::vfio_device {
+        self.raw.get()
+    }
+}
+
+// SAFETY: VFIO uses the embedded class device's refcount to own the full allocation.
+unsafe impl AlwaysRefCounted for Device {
+    fn inc_ref(&self) {
+        // SAFETY: A shared reference keeps the initialized VFIO device alive.
+        unsafe { bindings::get_device(&raw mut (*self.as_raw()).device) };
+    }
+
+    unsafe fn dec_ref(obj: NonNull<Self>) {
+        // SAFETY: The caller owns the VFIO reference being released.
+        unsafe { bindings::put_device(&raw mut (*obj.as_ref().as_raw()).device) };
+    }
+}
+
+// SAFETY: Ordinary VFIO references expose no unsynchronized callback operations.
+unsafe impl Send for Device {}
+// SAFETY: Shared access only changes the embedded class device's reference count.
+unsafe impl Sync for Device {}
+
+/// Callback data stored after the bus-specific C device structure.
+///
+/// # Invariants
+///
+/// `reg_data` is dangling when unpublished; otherwise it borrows pinned data kept
+/// alive until callback access has ended. `open_data` is null outside a successful
+/// first open, otherwise owns a pinned allocation until the last close.
+/// The bus-specific layer drains callbacks before releasing either allocation.
+struct CallbackData<R: CovariantForLt, O: ForLt> {
+    reg_data: UnsafeCell<NonNull<R::Of<'static>>>,
+    open_data: UnsafeCell<*mut O::Of<'static>>,
+}
+
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
+impl<R: CovariantForLt, O: ForLt> CallbackData<R, O> {
+    fn new() -> Self {
+        Self {
+            reg_data: UnsafeCell::new(NonNull::dangling()),
+            open_data: UnsafeCell::new(core::ptr::null_mut()),
+        }
+    }
+
+    /// Access registration data without allowing its erased lifetime to escape.
+    ///
+    /// # Safety
+    ///
+    /// Registration data must be published and remain valid through this call.
+    /// The caller must exclude concurrent changes to `reg_data`.
+    unsafe fn registration_data_with<V>(&self, f: impl for<'a> FnOnce(&'a R::Of<'a>) -> V) -> V {
+        // SAFETY: The caller keeps registration data alive and excludes mutation.
+        // Covariance permits shortening its erased lifetime to this borrow.
+        let reg_data = unsafe { (*self.reg_data.get()).cast::<R::Of<'_>>().as_ref() };
+        f(reg_data)
+    }
+
+    /// # Safety
+    ///
+    /// The caller must be an I/O callback while VFIO keeps the device open.
+    /// Registration and open data must remain valid, with their pointer slots
+    /// unchanged throughout the call.
+    unsafe fn callback_data_with<V>(
+        &self,
+        f: impl for<'borrow, 'data> FnOnce(&'borrow R::Of<'data>, Pin<&'borrow O::Of<'data>>) -> V,
+    ) -> V {
+        // SAFETY: The caller excludes open/close and unregister while borrowing the data.
+        unsafe {
+            self.registration_data_with(|reg_data| {
+                let ptr = (*self.open_data.get()).cast::<O::Of<'_>>();
+                // The data lifetime stays independent of the callback borrow,
+                // so borrowed OpenData fields cannot be stored back into OpenData.
+                f(reg_data, Pin::new_unchecked(&*ptr))
+            })
+        }
+    }
+
+    /// # Safety
+    ///
+    /// Call only during exclusive first open, with no existing open data.
+    /// All resources borrowed for `'a` must remain valid until `close()` returns.
+    unsafe fn open<'a, I: PinInit<O::Of<'a>, Error>>(&self, init: impl FnOnce() -> I) -> Result {
+        // Allocate before calling the driver so allocation failure cannot follow open.
+        let data = KBox::<O::Of<'a>>::new_uninit(GFP_KERNEL)?;
+        let data = data.write_pin_init(init())?;
+
+        // SAFETY: Only the owning pointer moves; close drops the allocation in place.
+        let raw = KBox::into_raw(unsafe { Pin::into_inner_unchecked(data) });
+        // SAFETY: First open is exclusive; the caller keeps borrowed resources alive
+        // through close. Lifetimes do not affect the allocation's layout.
+        unsafe { *self.open_data.get() = raw.cast::<O::Of<'static>>() };
+        Ok(())
+    }
+
+    /// # Safety
+    ///
+    /// Call once after a successful open, with all callbacks drained and the next
+    /// open excluded. Registration data and borrowed resources must remain alive.
+    unsafe fn close(&self) {
+        // SAFETY: Last close exclusively takes the allocation published by open.
+        let raw = unsafe { core::mem::replace(&mut *self.open_data.get(), core::ptr::null_mut()) };
+        // SAFETY: The allocation came from KBox::into_raw and is destroyed without moving it.
+        unsafe { drop(KBox::from_raw(raw)) };
+    }
+}
 
 /// Capability buffer supplied by VFIO for a region-info callback.
-#[cfg(CONFIG_VFIO_PCI_CORE)]
 pub struct InfoCap<'a> {
+    #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
     raw: &'a mut bindings::vfio_info_cap,
 }
 
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
index f3e68a3d18c..a27b1a1774c 100644
--- a/rust/kernel/vfio/pci.rs
+++ b/rust/kernel/vfio/pci.rs
@@ -10,9 +10,20 @@
 //! C header: [`include/linux/vfio_pci_core.h`](srctree/include/linux/vfio_pci_core.h)
 
 use super::{
+    CallbackData,
     InfoCap,
     UserBuf, //
 };
+pub use super::{
+    DeviceContext,
+    GetRegionInfo,
+    Ioctl,
+    Mmap,
+    Normal,
+    Open,
+    Read,
+    Write, //
+};
 
 use crate::{
     alloc::allocator::Kmalloc,
@@ -39,61 +50,16 @@
 };
 use core::{
     alloc::Layout,
-    cell::UnsafeCell,
     marker::PhantomData,
     ptr::NonNull, //
 };
 
+/// Reset the PCI slot or bus containing a VFIO device.
+pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+
 /// Index of the PCI configuration-space region.
 pub const CONFIG_REGION_INDEX: u32 = bindings::VFIO_PCI_CONFIG_REGION_INDEX;
 
-/// The context in which a VFIO PCI device reference is valid.
-///
-/// Callback views can only be borrowed from the corresponding VFIO trampoline.
-/// They cannot be refcounted or shared across threads.
-pub trait DeviceContext: private::Sealed {}
-
-/// An ordinary device reference, without callback-specific operations.
-pub struct Normal;
-
-/// The device is enabled but has not yet completed its first-open callback.
-pub struct Open;
-
-/// The device is borrowed for a VFIO ioctl callback on the current thread.
-pub struct Ioctl;
-
-/// The device is borrowed for a VFIO read callback on the current thread.
-pub struct Read;
-
-/// The device is borrowed for a VFIO write callback on the current thread.
-pub struct Write;
-
-/// The device is borrowed for a VFIO mmap callback on the current thread.
-pub struct Mmap;
-
-/// The device is borrowed for a VFIO region-info callback on the current thread.
-pub struct GetRegionInfo;
-
-mod private {
-    pub trait Sealed {}
-
-    impl Sealed for super::Normal {}
-    impl Sealed for super::Open {}
-    impl Sealed for super::Write {}
-    impl Sealed for super::Mmap {}
-    impl Sealed for super::Ioctl {}
-    impl Sealed for super::Read {}
-    impl Sealed for super::GetRegionInfo {}
-}
-
-impl DeviceContext for Normal {}
-impl DeviceContext for Open {}
-impl DeviceContext for Write {}
-impl DeviceContext for Mmap {}
-impl DeviceContext for Ioctl {}
-impl DeviceContext for Read {}
-impl DeviceContext for GetRegionInfo {}
-
 /// The trait for VFIO PCI variant driver implementations.
 ///
 /// Unoverridden callbacks delegate to the corresponding PCI core operation.
@@ -280,20 +246,22 @@ pub fn region_end(&self) -> Result<u64> {
 /// - Callback contexts are only borrowed for the corresponding callback and thread.
 /// - `core_device` was initialized by `vfio_pci_core_init_dev()`.
 /// - The VFIO device refcount owns the allocation lifetime.
-/// - `reg_data` is dangling without an owner or points to a valid
-///   `<T::RegistrationData as ForLt>::Of<'_>` owned by the enclosing [`Registration`].
-/// - `open_data` is null outside a successful open, otherwise it owns a pinned
-///   `KBox<T::OpenData<'_>>` until last close. Only open/close mutate it; VFIO
-///   prevents I/O callbacks from racing those transitions. The data is dropped
-///   before PCI core close and before registration data is freed.
+/// - `data` follows the VFIO registration and first-open/last-close lifetimes.
+///   Open data is destroyed before PCI core close, and registration data is
+///   released after PCI core unregistration has drained callbacks.
 #[repr(C)]
 pub struct Device<T: Operations, Ctx: DeviceContext = Normal> {
     core_device: Opaque<bindings::vfio_pci_core_device>,
-    reg_data: UnsafeCell<NonNull<<T::RegistrationData as ForLt>::Of<'static>>>,
-    open_data: UnsafeCell<*mut T::OpenData<'static>>,
+    data: CallbackData<T::RegistrationData, OpenDataFamily<T>>,
     _context: PhantomData<(Ctx, NotThreadSafe)>,
 }
 
+struct OpenDataFamily<T>(PhantomData<T>);
+
+impl<T: Operations> ForLt for OpenDataFamily<T> {
+    type Of<'a> = T::OpenData<'a>;
+}
+
 impl<T: Operations> Device<T> {
     /// Allocate a VFIO PCI device for subsequent registration.
     pub fn new(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
@@ -307,6 +275,7 @@ pub fn new_passthrough(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self
 
     fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<ARef<Self>> {
         const_assert!(core::mem::offset_of!(Self, core_device) == 0);
+        const_assert!(core::mem::offset_of!(bindings::vfio_pci_core_device, vdev) == 0);
         let size = Kmalloc::aligned_layout(Layout::new::<Self>()).size();
 
         // SAFETY: The bound PCI device and static ops are valid. The allocation
@@ -318,8 +287,7 @@ fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<A
 
         // SAFETY: Initialise the Rust field in the newly allocated device.
         unsafe {
-            (&raw mut (*this.as_ptr()).reg_data).write(UnsafeCell::new(NonNull::dangling()));
-            (&raw mut (*this.as_ptr()).open_data).write(UnsafeCell::new(core::ptr::null_mut()));
+            (&raw mut (*this.as_ptr()).data).write(CallbackData::new());
             (&raw mut (*this.as_ptr())._context).write(PhantomData);
         }
 
@@ -332,50 +300,6 @@ fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<A
         Ok(unsafe { ARef::from_raw(this) })
     }
 
-    /// Access the registration data through a closure with an HRTB lifetime.
-    ///
-    /// The closure's `for<'a>` bound prevents the caller from smuggling
-    /// references with a concrete short lifetime out of the closure.
-    ///
-    /// # Safety
-    ///
-    /// Registration data must be published and remain valid through this call.
-    unsafe fn registration_data_with<R>(
-        &self,
-        f: impl for<'a> FnOnce(&'a <T::RegistrationData as ForLt>::Of<'a>) -> R,
-    ) -> R {
-        // SAFETY: Registration keeps the allocation and its borrowed resources
-        // alive until unregistration completes. Covariance permits shortening
-        // the erased binding lifetime to this callback's borrow.
-        let reg_data: &<T::RegistrationData as ForLt>::Of<'_> = unsafe {
-            (*self.reg_data.get())
-                .cast::<<T::RegistrationData as ForLt>::Of<'_>>()
-                .as_ref()
-        };
-        f(reg_data)
-    }
-
-    /// # Safety
-    ///
-    /// The caller must be an I/O callback while VFIO keeps the device open.
-    unsafe fn callback_data_with<R>(
-        &self,
-        f: impl for<'borrow, 'data> FnOnce(
-            &'borrow <T::RegistrationData as ForLt>::Of<'data>,
-            Pin<&'borrow T::OpenData<'data>>,
-        ) -> R,
-    ) -> R {
-        // SAFETY: VFIO excludes open/close and unregister while this callback runs.
-        unsafe {
-            self.registration_data_with(|reg_data| {
-                let ptr = (*self.open_data.get()).cast::<T::OpenData<'_>>();
-                // The data lifetime stays independent of the callback borrow,
-                // so borrowed OpenData fields cannot be stored back into OpenData.
-                f(reg_data, Pin::new_unchecked(&*ptr))
-            })
-        }
-    }
-
     /// # Safety
     /// The returned view must only be used during the callback represented by `Ctx`.
     unsafe fn with_context<Ctx: DeviceContext>(&self) -> &Device<T, Ctx> {
@@ -403,7 +327,15 @@ fn core_device(&self) -> *mut bindings::vfio_pci_core_device {
     }
 
     fn vfio_device(&self) -> *mut bindings::vfio_device {
-        self.core_device.get().cast()
+        self.as_ref().as_raw()
+    }
+}
+
+impl<T: Operations, Ctx: DeviceContext> AsRef<super::Device<Ctx>> for Device<T, Ctx> {
+    fn as_ref(&self) -> &super::Device<Ctx> {
+        // SAFETY: The PCI core device embeds an initialized vfio_device. The borrow
+        // covers the same object and preserves the callback context and its lifetime.
+        unsafe { super::Device::from_raw(&raw mut (*self.core_device()).vdev) }
     }
 }
 
@@ -507,13 +439,14 @@ pub fn core_get_region_info(
 // SAFETY: The embedded device reference count owns the VFIO allocation.
 unsafe impl<T: Operations> AlwaysRefCounted for Device<T> {
     fn inc_ref(&self) {
-        // SAFETY: `self` holds a live VFIO device reference.
-        unsafe { bindings::get_device(&raw mut (*self.vfio_device()).device) };
+        self.as_ref().inc_ref();
     }
 
     unsafe fn dec_ref(obj: NonNull<Self>) {
-        // SAFETY: The caller owns the reference being released.
-        unsafe { bindings::put_device(&raw mut (*obj.as_ref().vfio_device()).device) };
+        // SAFETY: The caller owns a reference to this live PCI wrapper.
+        let dev = unsafe { obj.as_ref() }.as_ref();
+        // SAFETY: The embedded VFIO device owns the same allocation and reference.
+        unsafe { super::Device::dec_ref(NonNull::from(dev)) };
     }
 }
 
@@ -529,9 +462,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
 ///
 /// `vdev` must belong to a registered `Device<T>` in VFIO's first-open context.
 /// VFIO must exclude other opens, closes and I/O callbacks for this call.
-unsafe extern "C" fn open_device_cb<T: Operations>(
-    vdev: *mut bindings::vfio_device,
-) -> core::ffi::c_int {
+unsafe extern "C" fn open_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) -> c_int {
     // SAFETY: `vdev` is valid; set by vfio_alloc_device.
     let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
 
@@ -543,18 +474,9 @@ unsafe impl<T: Operations> Sync for Device<T> {}
 
     // SAFETY: Registration data is initialized before the VFIO device is published.
     let result = unsafe {
-        dev.registration_data_with(|rd| {
-            // Allocate before calling the driver so allocation failure cannot follow open.
-            let data = KBox::<T::OpenData<'_>>::new_uninit(GFP_KERNEL)?;
-            let data = data.write_pin_init(T::open_device(dev.with_context::<Open>(), rd))?;
-
-            // SAFETY: Only the owning pointer is moved; the allocation remains pinned.
-            let raw =
-                KBox::into_raw(Pin::into_inner_unchecked(data)).cast::<T::OpenData<'static>>();
-            // SAFETY: First open is exclusive with I/O callbacks and last close.
-            // Lifetimes do not affect layout; the device owns the allocation until close.
-            *dev.open_data.get() = raw;
-            Ok::<(), Error>(())
+        dev.data.registration_data_with(|rd| {
+            dev.data
+                .open(|| T::open_device(dev.with_context::<Open>(), rd))
         })
     };
     match result {
@@ -579,12 +501,9 @@ unsafe impl<T: Operations> Sync for Device<T> {}
 unsafe extern "C" fn close_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) {
     // SAFETY: `vdev` is valid.
     let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
-    // SAFETY: Last close is exclusive with I/O callbacks and the next first open.
-    let raw = unsafe { core::mem::replace(&mut *dev.open_data.get(), core::ptr::null_mut()) };
-    // Run the driver's destructor while registration data and PCI resources are valid.
-    // SAFETY: Successful open transferred this allocation with `KBox::into_raw`.
-    // Last close takes ownership exactly once and drops it without moving the pointee.
-    unsafe { drop(KBox::from_raw(raw)) };
+    // SAFETY: Last close drains I/O callbacks and excludes the next first open.
+    // Registration data and PCI resources remain valid while open data is destroyed.
+    unsafe { dev.data.close() };
     // SAFETY: Matches the enable in open_device_cb.
     unsafe { bindings::vfio_pci_core_close_device(vdev) };
 }
@@ -602,7 +521,8 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
     // SAFETY: VFIO invokes this callback for an open device with valid arguments.
     match unsafe {
-        dev.callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
+        dev.data
+            .callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
     } {
         Ok(v) => v,
         Err(e) => e.to_errno() as isize,
@@ -633,7 +553,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     };
     // SAFETY: VFIO invokes this callback for an open device with valid arguments.
     match unsafe {
-        dev.callback_data_with(|rd, od| {
+        dev.data.callback_data_with(|rd, od| {
             T::read(dev.with_context::<Read>(), rd, od, &mut ubuf, &mut pos)
         })
     } {
@@ -652,7 +572,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     vdev: *mut bindings::vfio_device,
     info: *mut bindings::vfio_region_info,
     caps: *mut bindings::vfio_info_cap,
-) -> core::ffi::c_int {
+) -> c_int {
     // SAFETY: `vdev`, `info`, and `caps` are valid kernel pointers provided
     // by the VFIO core.
     let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
@@ -664,7 +584,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     };
     // SAFETY: VFIO invokes this callback for an open device with valid arguments.
     match unsafe {
-        dev.callback_data_with(|rd, od| {
+        dev.data.callback_data_with(|rd, od| {
             T::get_region_info(dev.with_context::<GetRegionInfo>(), rd, od, info, &mut caps)
         })
     } {
@@ -696,7 +616,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     };
     // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
     match unsafe {
-        dev.callback_data_with(|rd, od| {
+        dev.data.callback_data_with(|rd, od| {
             T::write(dev.with_context::<Write>(), rd, od, &mut ubuf, &mut pos)
         })
     } {
@@ -713,7 +633,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
 unsafe extern "C" fn mmap_cb<T: Operations>(
     vdev: *mut bindings::vfio_device,
     vma: *mut bindings::vm_area_struct,
-) -> core::ffi::c_int {
+) -> c_int {
     // SAFETY: VFIO supplies an open device for this mmap callback.
     let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
     let mut mapping = Mapping {
@@ -723,7 +643,8 @@ unsafe impl<T: Operations> Sync for Device<T> {}
     };
     // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
     match unsafe {
-        dev.callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
+        dev.data
+            .callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
     } {
         Ok(()) => 0,
         Err(e) => e.to_errno(),
@@ -830,7 +751,7 @@ pub unsafe fn new(
             NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
 
         // SAFETY: No concurrent registration or callbacks; publish before registering.
-        unsafe { *dev.reg_data.get() = ptr };
+        unsafe { *dev.data.reg_data.get() = ptr };
 
         Ok(Self {
             dev: dev.into(),
@@ -882,13 +803,13 @@ pub unsafe fn unregister(&self) {
     /// Prepare the variant driver's state for a PCI reset.
     pub fn reset_prepare(&self) {
         // SAFETY: This owner keeps the pinned registration data alive.
-        unsafe { self.dev.registration_data_with(T::reset_prepare) };
+        unsafe { self.dev.data.registration_data_with(T::reset_prepare) };
     }
 
     /// Complete the variant driver's state update after a PCI reset.
     pub fn reset_done(&self) -> Result {
         // SAFETY: This owner keeps the pinned registration data alive.
-        unsafe { self.dev.registration_data_with(T::reset_done) }
+        unsafe { self.dev.data.registration_data_with(T::reset_done) }
     }
 
     /// Signal the VFIO error interrupt and return the PCI recovery result.
@@ -923,6 +844,6 @@ pub unsafe fn runtime_resume(&self) -> Result {
 impl<T: Operations> Drop for Registration<'_, T> {
     fn drop(&mut self) {
         // SAFETY: The driver unregistered in `unbind`, or registration did not succeed.
-        unsafe { *self.dev.reg_data.get() = NonNull::dangling() };
+        unsafe { *self.dev.data.reg_data.get() = NonNull::dangling() };
     }
 }
-- 
2.43.7

[PATCH v2 11/12] gpu: nova-core: publish typed SR-IOV PF APIs

[Simon Song]

From: Zhi Wang <zhiw@nvidia.com>

Register NovaCoreVfApi as typed SR-IOV PF data once the GPU is initialized.

Enable and disable VFs through sriov_configure(). The Rust PCI adapter
removes VFs before dropping PF driver data. Declare the registration
before the GPU so its borrowed services remain valid through VF teardown.

Expose a VF-side handle that borrows these services for each operation.
Return an owned instance on open and close it on drop while the VF
remains bound.

Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 drivers/gpu/nova-core/driver.rs        | 47 +++++++++++++++
 drivers/gpu/nova-core/nova_core.rs     |  2 +
 drivers/gpu/nova-core/vgpu/vgpu_api.rs | 82 +++++++++++++++++++++++++-
 3 files changed, 130 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index dc6febeee65..99beab18987 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -35,6 +35,12 @@
     }, //
 };
 
+#[cfg(CONFIG_PCI_IOV)]
+use kernel::types::ForLt;
+
+#[cfg(CONFIG_PCI_IOV)]
+use crate::vgpu::vgpu_api::NovaCoreVfApi;
+
 /// Counter for generating unique auxiliary device IDs.
 static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
 
@@ -42,6 +48,10 @@
 pub(crate) struct NovaCore<'bound> {
     /// Firmware-control registration.
     _fwctl: fwctl::Registration<'bound, NovaCoreFwCtl>,
+    #[cfg(CONFIG_PCI_IOV)]
+    #[allow(clippy::type_complexity)]
+    #[pin]
+    _vf_registration: pci::VfRegistration<'bound, ForLt!(NovaCoreVfApi<'_>)>,
     #[pin]
     pub(crate) gpu: Gpu<'bound>,
     bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -114,6 +124,11 @@ fn probe<'bound>(
         pin_init::pin_init_scope(move || {
             dev_dbg!(pdev, "Probe Nova Core GPU driver.\n");
 
+            #[cfg(CONFIG_PCI_IOV)]
+            if pdev.is_virtfn() {
+                return Err(ENODEV);
+            }
+
             pdev.enable_device_mem()?;
             pdev.set_master();
 
@@ -144,6 +159,18 @@ fn probe<'bound>(
                 // Run optional GPU selftests.
                 #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
                 _: { gpu.run_selftests(pdev) },
+                #[cfg(CONFIG_PCI_IOV)]
+                _vf_registration <- {
+                    // SAFETY: `gpu` is initialized at its pinned address and
+                    // outlives the registration and its borrowed API data.
+                    let gpu = unsafe { &(*this.as_ptr()).gpu };
+                    let api = NovaCoreVfApi::new(gpu, pdev);
+                    // SAFETY: Probe has exclusive access to the registration
+                    // slot and no VFs are enabled before successful probe. The
+                    // registration keeps PF services available until VF removal
+                    // completes and is dropped before its borrowed GPU resources.
+                    unsafe { pci::VfRegistration::new(pdev, Ok(api)) }
+                },
                 _reg: auxiliary::Registration::new(
                     pdev.as_ref(),
                     c"nova-drm",
@@ -176,4 +203,24 @@ fn probe<'bound>(
             }))
         })
     }
+
+    #[cfg(CONFIG_PCI_IOV)]
+    fn sriov_enable<'bound, 'callback>(
+        _dev: &'bound pci::Device<Core<'_>>,
+        _this: Pin<&Self::Data<'bound>>,
+        token: pci::SriovEnable<'callback>,
+    ) -> Result<pci::SriovEnabled<'callback>> {
+        let num_vfs = token.num_vfs();
+        token.enable(num_vfs)
+    }
+
+    #[cfg(CONFIG_PCI_IOV)]
+    fn sriov_disable<'bound>(
+        _dev: &'bound pci::Device<Core<'_>>,
+        _this: Pin<&Self::Data<'bound>>,
+        token: pci::SriovDisable<'_>,
+    ) -> Result {
+        token.disable();
+        Ok(())
+    }
 }
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 619f32c7941..ee511bdf354 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -33,6 +33,8 @@
 #[cfg(CONFIG_PCI_IOV)]
 pub use vgpu::vgpu_api::{
     NovaCoreVfApi,
+    NovaCoreVfApiHandle,
+    VgpuInstance,
     VgpuTypeInfo, //
 };
 
diff --git a/drivers/gpu/nova-core/vgpu/vgpu_api.rs b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
index f9f3e527bdc..a440108d2c2 100644
--- a/drivers/gpu/nova-core/vgpu/vgpu_api.rs
+++ b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
@@ -11,6 +11,9 @@
     prelude::*, //
 };
 
+#[cfg(CONFIG_PCI_IOV)]
+use kernel::types::ForLt;
+
 use crate::{
     driver::Bar0,
     gpu::Gpu,
@@ -61,7 +64,6 @@ pub struct NovaCoreVfApi<'gpu> {
 }
 
 impl<'gpu> NovaCoreVfApi<'gpu> {
-    #[expect(dead_code)]
     pub(crate) fn new(gpu: &'gpu Gpu<'gpu>, pdev: &'gpu pci::Device<device::Bound>) -> Self {
         Self {
             pdev,
@@ -84,6 +86,20 @@ fn gfid(&self, gfid: u32) -> Result<Gfid> {
 }
 
 impl NovaCoreVfApi<'_> {
+    /// Obtains the enabled PF services for a bound VF.
+    #[cfg(CONFIG_PCI_IOV)]
+    pub fn handle(vf: &pci::Device<device::Bound>) -> Result<NovaCoreVfApiHandle<'_>> {
+        let handle = NovaCoreVfApiHandle { vf };
+        handle.with(|api| {
+            if api.is_available() {
+                Ok(())
+            } else {
+                Err(ENODEV)
+            }
+        })?;
+        Ok(handle)
+    }
+
     /// Creates and boots an instance for a one-based VF ID.
     ///
     /// `sbdf` encodes the VF address as `(segment << 16) | (bus << 8) | devfn`.
@@ -156,3 +172,67 @@ pub fn reset_instance(&self, gfid: u32) -> Result {
         Ok(())
     }
 }
+
+/// Access to PF services for the lifetime of a VF driver binding.
+///
+/// The PF's registration keeps these services available until VF removal
+/// completes and is dropped before the GPU resources they borrow.
+#[cfg(CONFIG_PCI_IOV)]
+pub struct NovaCoreVfApiHandle<'vf> {
+    vf: &'vf pci::Device<device::Bound>,
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl<'vf> NovaCoreVfApiHandle<'vf> {
+    /// Borrows the typed PF services for a single operation.
+    fn with<R>(
+        &self,
+        f: impl for<'borrow, 'data> FnOnce(Pin<&'borrow NovaCoreVfApi<'data>>) -> Result<R>,
+    ) -> Result<R> {
+        self.vf
+            .vf_registration_data_with::<ForLt!(NovaCoreVfApi<'_>), _>(f)?
+    }
+
+    /// Creates and boots an instance that closes when dropped.
+    ///
+    /// The arguments have the same meaning as in [`NovaCoreVfApi::open_instance`].
+    pub fn open(&self, gfid: u32, dbdf: u32, vm_pid: u32) -> Result<VgpuInstance<'vf>> {
+        let type_info = self.with(|api| api.open_instance(gfid, dbdf, vm_pid))?;
+        Ok(VgpuInstance {
+            api: Self { vf: self.vf },
+            gfid,
+            type_info,
+        })
+    }
+
+    /// Resets an active instance and scrubs its guest VRAM.
+    pub fn reset(&self, gfid: u32) -> Result {
+        self.with(|api| api.reset_instance(gfid))
+    }
+}
+
+/// An active instance whose teardown runs while its VF driver remains bound.
+///
+/// Dropping this guard may sleep while firmware teardown completes.
+#[cfg(CONFIG_PCI_IOV)]
+pub struct VgpuInstance<'vf> {
+    api: NovaCoreVfApiHandle<'vf>,
+    gfid: u32,
+    type_info: VgpuTypeInfo,
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl VgpuInstance<'_> {
+    /// Returns the assigned PCI IDs and BAR1 aperture size.
+    #[inline]
+    pub fn type_info(&self) -> &VgpuTypeInfo {
+        &self.type_info
+    }
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl Drop for VgpuInstance<'_> {
+    fn drop(&mut self) {
+        let _ = self.api.with(|api| api.close_instance(self.gfid));
+    }
+}
-- 
2.43.7

[PATCH v2 12/12] vfio/nvidia-vgpu: add the Rust VFIO variant driver

[Simon Song]

From: Danilo Krummrich <dakr@kernel.org>

NVIDIA vGPU VFs need nova-core to manage their instances when assigned
to userspace through VFIO.

Add a Rust VFIO PCI variant driver that uses the typed SR-IOV PF APIs
to create an instance on first open and close it on last close. Expose
the assigned device and subsystem IDs, enforce the assigned BAR1
aperture, and coordinate firmware reset with PCI reset.

Match NVIDIA devices through a VFIO override-only entry. Devices
without an available Nova vGPU interface use ordinary PCI passthrough.

Use the PCI adapter to keep callback data available throughout
registration and removal. Extend the GPU build rules to make nova-core
crate metadata available to the VFIO driver and export the Rust symbols
it references.

Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
Signed-off-by: Simon Song <xinmengs@nvidia.com>
---
 drivers/gpu/Makefile                        |  15 +-
 drivers/vfio/pci/Kconfig                    |   2 +
 drivers/vfio/pci/Makefile                   |   2 +
 drivers/vfio/pci/nvidia-vgpu/Kconfig        |  20 +
 drivers/vfio/pci/nvidia-vgpu/Makefile       |   3 +
 drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs | 402 ++++++++++++++++++++
 6 files changed, 441 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs

diff --git a/drivers/gpu/Makefile b/drivers/gpu/Makefile
index e372fc02139..1b8a907e716 100644
--- a/drivers/gpu/Makefile
+++ b/drivers/gpu/Makefile
@@ -19,8 +19,9 @@ nova-core-y := nova-core/nova_core.o nova-core/nova_core_exports.o
 obj-$(CONFIG_DRM_NOVA) += nova-drm.o
 nova-drm-y := drm/nova/nova.o
 
-# Export Rust symbols from nova-core only if nova-drm actually references them.
-nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o)
+# Export Rust symbols from nova-core only if dependent modules reference them.
+nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o) \
+	$(if $(CONFIG_NVIDIA_VGPU_VFIO_PCI),$(obj)/../vfio/pci/nvidia-vgpu/nvidia_vgpu.o)
 
 rust_needed_exports = \
 	{ $(if $(strip $(2)),$(NM) -u $(2);,) echo "__DEFINED_RUST_SYMBOLS__"; \
@@ -57,10 +58,18 @@ $(obj)/nova-core/nova_core_exports.o: private cmd_gensymtypes_c = \
 		$(obj)/nova-core/nova_core.o
 endif
 
-# Output nova-core's crate metadata for use by nova-drm at compile time.
+# Output nova-core's crate metadata for its Rust consumers at compile time.
 RUSTFLAGS_nova-core/nova_core.o += \
 	--emit=metadata=$(objtree)/$(obj)/nova-core/libnova_core.rmeta
 
 # Allow nova-drm to import nova-core's types.
 $(obj)/drm/nova/nova.o: $(obj)/nova-core/nova_core.o
 RUSTFLAGS_drm/nova/nova.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
+
+# Build the NVIDIA VFIO driver here so metadata and generated exports share
+# the same dependency graph as nova-drm.
+obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu-vfio-pci.o
+nvidia-vgpu-vfio-pci-y := ../vfio/pci/nvidia-vgpu/nvidia_vgpu.o
+
+$(obj)/../vfio/pci/nvidia-vgpu/nvidia_vgpu.o: $(obj)/nova-core/nova_core.o
+RUSTFLAGS_../vfio/pci/nvidia-vgpu/nvidia_vgpu.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185..b48d8d1af42 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -74,4 +74,6 @@ source "drivers/vfio/pci/qat/Kconfig"
 
 source "drivers/vfio/pci/xe/Kconfig"
 
+source "drivers/vfio/pci/nvidia-vgpu/Kconfig"
+
 endmenu
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241..58c88304a18 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -24,3 +24,5 @@ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
 obj-$(CONFIG_QAT_VFIO_PCI) += qat/
 
 obj-$(CONFIG_XE_VFIO_PCI) += xe/
+
+# nvidia-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
diff --git a/drivers/vfio/pci/nvidia-vgpu/Kconfig b/drivers/vfio/pci/nvidia-vgpu/Kconfig
new file mode 100644
index 00000000000..1363f84163e
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Kconfig
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVIDIA_VGPU_VFIO_PCI
+	tristate "VFIO support for the NVIDIA vGPU"
+	depends on NOVA_CORE && PCI_IOV && RUST
+	select VFIO_PCI_CORE
+	help
+	  This option enables VFIO (Virtual Function I/O) support for
+	  NVIDIA virtual GPUs (vGPU). It allows the assignment of a virtual
+	  GPU instance to userspace applications via VFIO, typically used
+	  with hypervisors such as KVM and device emulators like QEMU.
+
+	  Devices without an available Nova vGPU interface use ordinary PCI
+	  passthrough. Both paths use the VFIO PCI core defaults; vfio-pci
+	  module options do not apply.
+
+	  The NVIDIA vGPU allows a physical GPU to be partitioned into
+	  multiple virtual GPUs, each of which can be passed to a virtual
+	  machine as a PCI device using the standard VFIO infrastructure.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile
new file mode 100644
index 00000000000..bce0133562f
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+# nvidia-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
+# nvidia_vgpu.o (rust-analyzer marker - DO NOT REMOVE).
diff --git a/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs b/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
new file mode 100644
index 00000000000..f16d6cf6a4a
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
@@ -0,0 +1,402 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! NVIDIA vGPU VFIO variant driver.
+//!
+//! Nova VFs own an instance between the first VFIO open and last close. Other
+//! matching devices use ordinary PCI passthrough.
+
+use kernel::{
+    bindings,
+    device::{
+        Bound,
+        Core, //
+    },
+    io::resource::Flags,
+    pci,
+    prelude::*,
+    sync::{
+        CondVar,
+        Mutex,
+        MutexGuard, //
+    },
+    types::CovariantForLt,
+    vfio::{
+        self,
+        pci::{
+            GetRegionInfo,
+            Ioctl,
+            Mapping,
+            Mmap,
+            Open,
+            Position,
+            Read,
+            Write, //
+        }, //
+    }, //
+};
+use nova_core::{
+    NovaCoreVfApi,
+    NovaCoreVfApiHandle,
+    VgpuInstance, //
+};
+
+struct NvidiaVgpuOps;
+
+struct VgpuRegistration<'a> {
+    api: NovaCoreVfApiHandle<'a>,
+    gfid: u32,
+}
+
+#[derive(Default)]
+struct InstanceState {
+    active: bool,
+    resetting: bool,
+    reset_error: Option<Error>,
+}
+
+#[pin_data]
+struct NvidiaVgpuRegData<'a> {
+    pdev: &'a pci::Device<Bound>,
+    vgpu: Option<VgpuRegistration<'a>>,
+    fb_bar: u32,
+    #[pin]
+    state: Mutex<InstanceState>,
+    #[pin]
+    reset_done: CondVar,
+}
+
+impl NvidiaVgpuRegData<'_> {
+    fn lock_instance(&self) -> MutexGuard<'_, InstanceState> {
+        let mut state = self.state.lock();
+        while state.resetting {
+            self.reset_done.wait(&mut state);
+        }
+        state
+    }
+}
+
+struct NvidiaVgpuOpenData<'a> {
+    registration: &'a NvidiaVgpuRegData<'a>,
+    instance: Option<VgpuInstance<'a>>,
+}
+
+impl<'a> NvidiaVgpuOpenData<'a> {
+    fn new(
+        dev: &vfio::pci::Device<NvidiaVgpuOps, Open>,
+        rd: &'a NvidiaVgpuRegData<'a>,
+    ) -> Result<Self> {
+        let mut state = rd.lock_instance();
+        let instance = match &rd.vgpu {
+            Some(vgpu) => {
+                // The firmware DBDF format has a 16-bit PCI segment field.
+                let segment = u16::try_from(rd.pdev.domain_nr()).map_err(|_| EOVERFLOW)?;
+                let dbdf = (u32::from(segment) << 16) | u32::from(rd.pdev.dev_id());
+                let vm_pid = kernel::current!().tgid().try_into()?;
+                let instance = vgpu.api.open(vgpu.gfid, dbdf, vm_pid)?;
+                let info = instance.type_info();
+                dev.set_device_id(info.pci_dev_id as u16);
+                state.active = true;
+                state.reset_error = None;
+                Some(instance)
+            }
+            None => None,
+        };
+        Ok(Self {
+            registration: rd,
+            instance,
+        })
+    }
+
+    fn bar1_size(&self) -> Result<Option<u64>> {
+        let Some(instance) = &self.instance else {
+            return Ok(None);
+        };
+        let physical = self
+            .registration
+            .pdev
+            .resource_len(self.registration.fb_bar)?;
+        let size = instance
+            .type_info()
+            .bar1_length
+            .checked_mul(1 << 20)
+            .ok_or(EOVERFLOW)?;
+        Ok(Some(if size == 0 {
+            physical
+        } else {
+            size.min(physical)
+        }))
+    }
+
+    fn limit_bar1(&self, buf: &mut vfio::UserBuf, position: &Position<'_>) -> Result {
+        if buf.is_empty() || position.region_index() != self.registration.fb_bar {
+            return Ok(());
+        }
+        if let Some(size) = self.bar1_size()? {
+            let offset = position.region_offset();
+            if offset >= size {
+                return Err(EINVAL);
+            }
+            if size - offset < buf.len() as u64 {
+                buf.truncate((size - offset) as usize);
+            }
+        }
+        Ok(())
+    }
+}
+
+impl Drop for NvidiaVgpuOpenData<'_> {
+    fn drop(&mut self) {
+        let mut state = self.registration.lock_instance();
+        state.active = false;
+        drop(self.instance.take());
+    }
+}
+
+impl vfio::pci::Operations for NvidiaVgpuOps {
+    const NAME: &'static CStr = c"nvidia-vgpu-vfio-pci";
+    type RegistrationData = CovariantForLt!(NvidiaVgpuRegData<'_>);
+    type OpenData<'a> = NvidiaVgpuOpenData<'a>;
+
+    fn open_device<'a>(
+        dev: &'a vfio::pci::Device<Self, Open>,
+        rd: &'a NvidiaVgpuRegData<'a>,
+    ) -> impl PinInit<Self::OpenData<'a>, Error> + 'a {
+        NvidiaVgpuOpenData::new(dev, rd)
+    }
+
+    fn ioctl<'a>(
+        dev: &vfio::pci::Device<Self, Ioctl>,
+        rd: &NvidiaVgpuRegData<'a>,
+        _open_data: Pin<&Self::OpenData<'a>>,
+        cmd: u32,
+        arg: usize,
+    ) -> Result<isize> {
+        let reset = cmd == vfio::DEVICE_RESET || cmd == vfio::pci::DEVICE_PCI_HOT_RESET;
+        if reset {
+            if let Some(error) = rd.state.lock().reset_error {
+                return Err(error);
+            }
+        }
+        let result = dev.core_ioctl(cmd, arg)?;
+        if reset {
+            if let Some(error) = rd.state.lock().reset_error {
+                return Err(error);
+            }
+        }
+        Ok(result)
+    }
+
+    fn read<'a>(
+        dev: &vfio::pci::Device<Self, Read>,
+        _rd: &NvidiaVgpuRegData<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        buf: &mut vfio::UserBuf,
+        position: &mut Position<'_>,
+    ) -> Result<isize> {
+        if let Some(instance) = &open_data.instance {
+            if position.region_index() == vfio::pci::CONFIG_REGION_INDEX {
+                let offset = position.region_offset();
+                return position.with_temporary(|next| {
+                    let count = dev.core_read(buf, next)?;
+                    buf.write_overlapping(
+                        offset,
+                        count.try_into()?,
+                        u64::from(bindings::PCI_SUBSYSTEM_ID),
+                        &(instance.type_info().pci_subsys_id as u16).to_le_bytes(),
+                    )?;
+                    Ok(count)
+                });
+            }
+        }
+        open_data.limit_bar1(buf, position)?;
+        dev.core_read(buf, position)
+    }
+
+    fn write<'a>(
+        dev: &vfio::pci::Device<Self, Write>,
+        _rd: &NvidiaVgpuRegData<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        buf: &mut vfio::UserBuf,
+        position: &mut Position<'_>,
+    ) -> Result<isize> {
+        open_data.limit_bar1(buf, position)?;
+        dev.core_write(buf, position)
+    }
+
+    fn mmap<'a>(
+        dev: &vfio::pci::Device<Self, Mmap>,
+        rd: &NvidiaVgpuRegData<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        mapping: &mut Mapping<'_>,
+    ) -> Result {
+        if mapping.region_index() == rd.fb_bar {
+            if let Some(size) = open_data.bar1_size()? {
+                if mapping.region_end()? > size {
+                    return Err(EINVAL);
+                }
+            }
+        }
+        dev.core_mmap(mapping)
+    }
+
+    fn get_region_info<'a>(
+        dev: &vfio::pci::Device<Self, GetRegionInfo>,
+        rd: &NvidiaVgpuRegData<'a>,
+        open_data: Pin<&Self::OpenData<'a>>,
+        info: &mut bindings::vfio_region_info,
+        caps: &mut vfio::InfoCap<'_>,
+    ) -> Result {
+        dev.core_get_region_info(info, caps)?;
+        if info.index == rd.fb_bar && info.size != 0 {
+            if let Some(size) = open_data.bar1_size()? {
+                info.size = info.size.min(size);
+            }
+        }
+        Ok(())
+    }
+
+    fn reset_prepare(rd: &NvidiaVgpuRegData<'_>) {
+        // PCI holds the device lock, and VFIO may hold its memory lock. Open and
+        // close release this mutex before entering the VFIO core.
+        let mut state = rd.state.lock();
+        state.resetting = true;
+        if state.active {
+            if let Some(vgpu) = &rd.vgpu {
+                if let Err(error) = vgpu.api.reset(vgpu.gfid) {
+                    state.reset_error.get_or_insert(error);
+                }
+            }
+        }
+    }
+
+    fn reset_done(rd: &NvidiaVgpuRegData<'_>) -> Result {
+        let mut state = rd.state.lock();
+        let error = state.reset_error;
+        state.resetting = false;
+        rd.reset_done.notify_all();
+        error.map_or(Ok(()), Err)
+    }
+}
+
+struct NvidiaVgpuDriver;
+
+kernel::pci_device_table!(
+    PCI_TABLE,
+    <NvidiaVgpuDriver as pci::Driver>::IdInfo,
+    [(
+        pci::DeviceId::from_id_vfio_override(pci::Vendor::NVIDIA, bindings::PCI_ANY_ID as u32),
+        (),
+    ),]
+);
+
+#[pin_data]
+struct NvidiaVgpuData<'bound> {
+    reg: vfio::pci::Registration<'bound, NvidiaVgpuOps>,
+}
+
+#[vtable]
+impl pci::Driver for NvidiaVgpuDriver {
+    type IdInfo = ();
+    type Data<'bound> = NvidiaVgpuData<'bound>;
+    const ID_TABLE: pci::IdTable<Self::IdInfo> = &PCI_TABLE;
+    const DRIVER_MANAGED_DMA: bool = true;
+    #[cfg(CONFIG_PM)]
+    const PM_OPS: Option<pci::PmOps<Self>> = Some(pci::PmOps::new());
+
+    fn probe<'bound>(
+        pdev: &'bound pci::Device<Core<'_>>,
+        _info: Option<&'bound Self::IdInfo>,
+    ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+        try_pin_init!(NvidiaVgpuData {
+            reg: {
+                let vgpu = match NovaCoreVfApi::handle(pdev) {
+                    Ok(api) => Some(VgpuRegistration {
+                        api,
+                        gfid: pdev.vf_id()?.checked_add(1).ok_or(EOVERFLOW)?,
+                    }),
+                    Err(_) => None,
+                };
+                let dev = if vgpu.is_some() {
+                    vfio::pci::Device::<NvidiaVgpuOps>::new(pdev)?
+                } else {
+                    vfio::pci::Device::<NvidiaVgpuOps>::new_passthrough(pdev)?
+                };
+                let fb_bar = if pdev.resource_flags(0)?.contains(Flags::IORESOURCE_MEM_64) {
+                    2
+                } else {
+                    1
+                };
+                // SAFETY: This probe owns the unregistered device; bind/unbind handle registration.
+                unsafe {
+                    vfio::pci::Registration::new(
+                        pdev,
+                        &dev,
+                        try_pin_init!(NvidiaVgpuRegData {
+                            pdev,
+                            vgpu,
+                            fb_bar,
+                            state <- kernel::new_mutex!(InstanceState::default()),
+                            reset_done <- kernel::new_condvar!(),
+                        }),
+                    )?
+                }
+            },
+        })
+    }
+
+    fn bind<'bound>(_pdev: &pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) -> Result {
+        // SAFETY: PCI installed the complete driver data; `unbind` unregisters it before removal.
+        unsafe { data.reg.register::<Self>() }
+    }
+
+    fn unbind<'bound>(_pdev: &'bound pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) {
+        // SAFETY: Bind succeeded, and PCI has not removed or destroyed any driver data.
+        unsafe { data.reg.unregister() };
+    }
+
+    fn reset_prepare<'bound>(_pdev: &pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) {
+        data.reg.reset_prepare();
+    }
+
+    fn reset_done<'bound>(pdev: &pci::Device<Core<'_>>, data: Pin<&Self::Data<'bound>>) {
+        if let Err(error) = data.reg.reset_done() {
+            dev_err!(pdev, "VFIO device reset failed: {:?}\n", error);
+            data.reg.error_detected();
+        }
+    }
+
+    fn error_detected<'bound>(
+        _pdev: &pci::Device<Core<'_>>,
+        data: Pin<&Self::Data<'bound>>,
+        _state: bindings::pci_channel_state_t,
+    ) -> bindings::pci_ers_result_t {
+        data.reg.error_detected()
+    }
+
+    #[cfg(CONFIG_PM)]
+    fn runtime_suspend<'bound>(
+        _pdev: &pci::Device<Bound>,
+        data: Pin<&Self::Data<'bound>>,
+    ) -> Result {
+        // SAFETY: Called from this device's PCI runtime-suspend callback.
+        unsafe { data.reg.runtime_suspend() }
+    }
+
+    #[cfg(CONFIG_PM)]
+    fn runtime_resume<'bound>(
+        _pdev: &pci::Device<Bound>,
+        data: Pin<&Self::Data<'bound>>,
+    ) -> Result {
+        // SAFETY: Called from this device's PCI runtime-resume callback.
+        unsafe { data.reg.runtime_resume() }
+    }
+}
+
+kernel::module_pci_driver! {
+    type: NvidiaVgpuDriver,
+    name: "nvidia-vgpu-vfio-pci",
+    authors: ["NVIDIA"],
+    description: "NVIDIA vGPU VFIO variant driver",
+    license: "GPL v2",
+}
-- 
2.43.7

Re: [PATCH v2 01/12] rust: pci: add VFIO override device IDs

[Gary Guo]

On Fri Oct 23, 2026 at 4:39 PM BST, Simon Song wrote:
> From: Danilo Krummrich <dakr@kernel.org>
>
> VFIO variant drivers need override-only IDs and VFIO-specific module
> aliases so userspace can select them for PCI assignment.
>
> Add a DeviceId constructor equivalent to PCI_DRIVER_OVERRIDE_DEVICE_VFIO,
> matching a vendor and device without restricting the PCI class.
>
> Signed-off-by: Danilo Krummrich <dakr@kernel.org>
> Signed-off-by: Zhi Wang <zhiw@nvidia.com>
> Signed-off-by: Simon Song <xinmengs@nvidia.com>
> ---
>  rust/kernel/pci.rs | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
> index d83492e06ce..10864ffda4e 100644
> --- a/rust/kernel/pci.rs
> +++ b/rust/kernel/pci.rs
> @@ -206,6 +206,13 @@ macro_rules! module_pci_driver {
>  impl DeviceId {
>      const PCI_ANY_ID: u32 = !0;
>  
> +    /// Equivalent to C's `PCI_DRIVER_OVERRIDE_DEVICE_VFIO` macro.

The documentation should primary describe what the function is.

The C equivalence is good to have but should be secondary info.

Best,
Gary

> +    pub const fn from_id_vfio_override(vendor: Vendor, device: u32) -> Self {
> +        let mut id = Self::from_id(vendor, device);
> +        id.0.override_only = bindings::PCI_ID_F_VFIO_DRIVER_OVERRIDE;
> +        id
> +    }
> +
>      /// Equivalent to C's `PCI_DEVICE` macro.
>      ///
>      /// Create a new `pci::DeviceId` from a vendor and device ID.