* [PATCH 1/7] rust: pci: add VFIO override device IDs
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 10:57 ` [PATCH 2/7] rust: pci: expose the VF index within its PF Zhi Wang
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
From: Danilo Krummrich <dakr@kernel.org>
VFIO variant drivers need override-only IDs and VFIO-specific module
aliases so userspace can select them for PCI assignment.
Add a DeviceId constructor equivalent to PCI_DRIVER_OVERRIDE_DEVICE_VFIO,
matching a vendor and device without restricting the PCI class.
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
rust/kernel/pci.rs | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index d83492e06cee..10864ffda4ea 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -206,6 +206,13 @@ macro_rules! module_pci_driver {
impl DeviceId {
const PCI_ANY_ID: u32 = !0;
+ /// Equivalent to C's `PCI_DRIVER_OVERRIDE_DEVICE_VFIO` macro.
+ pub const fn from_id_vfio_override(vendor: Vendor, device: u32) -> Self {
+ let mut id = Self::from_id(vendor, device);
+ id.0.override_only = bindings::PCI_ID_F_VFIO_DRIVER_OVERRIDE;
+ id
+ }
+
/// Equivalent to C's `PCI_DEVICE` macro.
///
/// Create a new `pci::DeviceId` from a vendor and device ID.
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 2/7] rust: pci: expose the VF index within its PF
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
2026-09-30 10:57 ` [PATCH 1/7] rust: pci: add VFIO override device IDs Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 15:48 ` Danilo Krummrich
2026-09-30 10:57 ` [PATCH 3/7] rust: pci: allow drivers to manage DMA ownership Zhi Wang
` (4 subsequent siblings)
6 siblings, 1 reply; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
From: Danilo Krummrich <dakr@kernel.org>
Nova identifies a vGPU instance by the VF's index within its PF. Expose
pci_iov_vf_id() through Device::vf_id() so a Rust VF driver can obtain
that index, with an error for devices that are not VFs.
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
rust/kernel/pci/iov.rs | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/rust/kernel/pci/iov.rs b/rust/kernel/pci/iov.rs
index 9d9bd3ac7025..fac134d1ebab 100644
--- a/rust/kernel/pci/iov.rs
+++ b/rust/kernel/pci/iov.rs
@@ -40,6 +40,14 @@ pub fn is_virtfn(&self) -> bool {
// SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`.
unsafe { (*self.as_raw()).is_virtfn() != 0 }
}
+
+ /// Return the zero-based VF index within its PF, or an error for a non-VF device.
+ pub fn vf_id(&self) -> Result<u32> {
+ // SAFETY: `self.as_raw()` points to a live PCI device; the helper checks VF membership.
+ let id = unsafe { bindings::pci_iov_vf_id(self.as_raw()) };
+ to_result(id)?;
+ Ok(id as u32)
+ }
}
impl Device<device::Core<'_>> {
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH 2/7] rust: pci: expose the VF index within its PF
2026-09-30 10:57 ` [PATCH 2/7] rust: pci: expose the VF index within its PF Zhi Wang
@ 2026-09-30 15:48 ` Danilo Krummrich
0 siblings, 0 replies; 9+ messages in thread
From: Danilo Krummrich @ 2026-09-30 15:48 UTC (permalink / raw)
To: Zhi Wang
Cc: acourbot, kvm, rust-for-linux, nova-gpu, linux-pci, alex, jgg,
yishaih, skolothumtho, kevin.tian, airlied, simona, ojeda,
alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin, a.hindborg,
aliceryhl, tmgross, jhubbard, ecourtney, cjia, smitra, kjaju,
alkumar, ankita, aniketa, kwankhede, targupta, linux-kernel,
zhiwang, Boqun Feng, Daniel Almeida, Tamir Duberstein,
Onur Özkan, Jason Gunthorpe, Bjorn Helgaas,
Krzysztof Wilczyński, dri-devel, Peter Colberg, Simon Song
On Wed Sep 30, 2026 at 12:57 PM CEST, Zhi Wang wrote:
> @@ -40,6 +40,14 @@ pub fn is_virtfn(&self) -> bool {
> // SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`.
> unsafe { (*self.as_raw()).is_virtfn() != 0 }
> }
> +
> + /// Return the zero-based VF index within its PF, or an error for a non-VF device.
> + pub fn vf_id(&self) -> Result<u32> {
> + // SAFETY: `self.as_raw()` points to a live PCI device; the helper checks VF membership.
> + let id = unsafe { bindings::pci_iov_vf_id(self.as_raw()) };
> + to_result(id)?;
> + Ok(id as u32)
> + }
In my PoC I had this implemented for Device<Bound>, but here it is implemented
for just Device I think? It's not visible from the context, but IIRC you moved
is_virtfn() in an impl Device block (which is correct).
So, if this is correct, please move it under Device<Bound>, as offset and stride
are not guaranteed to not be concurrently modified otherwise.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/7] rust: pci: allow drivers to manage DMA ownership
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
2026-09-30 10:57 ` [PATCH 1/7] rust: pci: add VFIO override device IDs Zhi Wang
2026-09-30 10:57 ` [PATCH 2/7] rust: pci: expose the VF index within its PF Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 10:57 ` [PATCH 4/7] rust: vfio: add PCI variant driver abstractions Zhi Wang
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
From: Danilo Krummrich <dakr@kernel.org>
VFIO manages DMA ownership through the userspace IOMMU interface. Its
PCI variant drivers must opt out of the driver's default DMA domain.
Add Driver::DRIVER_MANAGED_DMA and pass it to the PCI core. Existing
Rust PCI drivers keep the default setting.
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
rust/kernel/pci.rs | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 10864ffda4ea..a156927e23d5 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -108,6 +108,7 @@ unsafe fn register(
(*pdrv.get()).probe = Some(Self::probe_callback);
(*pdrv.get()).remove = Some(Self::remove_callback);
(*pdrv.get()).id_table = T::ID_TABLE.as_ptr();
+ (*pdrv.get()).driver_managed_dma = T::DRIVER_MANAGED_DMA;
#[cfg(CONFIG_PCI_IOV)]
if T::HAS_SRIOV_ENABLE {
(*pdrv.get()).sriov_configure = Some(Self::sriov_configure_callback);
@@ -345,6 +346,9 @@ pub trait Driver {
/// The table of device ids supported by the driver.
const ID_TABLE: IdTable<Self::IdInfo>;
+ /// Whether the driver manages DMA ownership instead of using the default DMA domain.
+ const DRIVER_MANAGED_DMA: bool = false;
+
/// PCI driver probe.
///
/// Called when a new pci device is added or discovered. Implementers should
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 4/7] rust: vfio: add PCI variant driver abstractions
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
` (2 preceding siblings ...)
2026-09-30 10:57 ` [PATCH 3/7] rust: pci: allow drivers to manage DMA ownership Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 10:57 ` [PATCH 5/7] rust: vfio: separate common device handling from PCI Zhi Wang
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
From: Danilo Krummrich <dakr@kernel.org>
Add Rust abstractions for PCI drivers that extend vfio-pci-core with
device-specific operations.
Provide Operations, Device and Registration with separate registration
and first-open/last-close data. Allow open data to borrow registration
data, undo PCI enablement if open fails, and drop open data before
closing the core device.
Use a PCI adapter to register VFIO after publishing initialized driver
data and unregister it before unbind and data destruction. Route runtime
PM, AER, reset and VGA callbacks through the driver's typed registration.
Restrict core I/O helpers to their corresponding callback contexts.
Require covariant registration data so callback borrows can safely
shorten its erased binding lifetime. Add bounded user-buffer and region
mapping helpers, and wire up VFIO core support for device features,
matching, IOMMUFD and PASID. Allow passthrough devices to export physical
BARs as DMA buffers.
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
rust/bindings/bindings_helper.h | 4 +
rust/kernel/lib.rs | 2 +
rust/kernel/vfio.rs | 100 +++
rust/kernel/vfio/pci.rs | 1109 +++++++++++++++++++++++++++++++
4 files changed, 1215 insertions(+)
create mode 100644 rust/kernel/vfio.rs
create mode 100644 rust/kernel/vfio/pci.rs
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 930e63290cdd..25ae5d30a93f 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -93,6 +93,8 @@
#include <linux/task_work.h>
#include <linux/tracepoint.h>
#include <linux/usb.h>
+#include <linux/vfio.h>
+#include <linux/vfio_pci_core.h>
#include <linux/wait.h>
#include <linux/workqueue.h>
#include <linux/xarray.h>
@@ -115,6 +117,8 @@
const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
const size_t RUST_CONST_HELPER_ARCH_KMALLOC_MINALIGN = ARCH_KMALLOC_MINALIGN;
const size_t RUST_CONST_HELPER_PAGE_SIZE = PAGE_SIZE;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_RESET = VFIO_DEVICE_RESET;
+const unsigned int RUST_CONST_HELPER_VFIO_DEVICE_PCI_HOT_RESET = VFIO_DEVICE_PCI_HOT_RESET;
const size_t RUST_CONST_HELPER_GENLMSG_DEFAULT_SIZE = GENLMSG_DEFAULT_SIZE;
const gfp_t RUST_CONST_HELPER_GFP_ATOMIC = GFP_ATOMIC;
const gfp_t RUST_CONST_HELPER_GFP_KERNEL = GFP_KERNEL;
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index f9ef36217bb5..261a411ab506 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -146,6 +146,8 @@
pub mod uaccess;
#[cfg(CONFIG_USB = "y")]
pub mod usb;
+#[cfg(CONFIG_VFIO)]
+pub mod vfio;
pub mod workqueue;
pub mod xarray;
diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
new file mode 100644
index 000000000000..ed93066e2da5
--- /dev/null
+++ b/rust/kernel/vfio.rs
@@ -0,0 +1,100 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Virtual Function I/O (VFIO) abstractions.
+
+use crate::{
+ bindings,
+ prelude::*,
+ uaccess::{
+ UserPtr,
+ UserSlice, //
+ },
+};
+
+#[cfg(CONFIG_VFIO_PCI_CORE)]
+pub mod pci;
+
+/// Reset the VFIO device.
+pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
+
+/// Reset the PCI slot or bus containing a VFIO device.
+pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+
+/// Capability buffer supplied by VFIO for a region-info callback.
+#[cfg(CONFIG_VFIO_PCI_CORE)]
+pub struct InfoCap<'a> {
+ raw: &'a mut bindings::vfio_info_cap,
+}
+
+/// Opaque wrapper around a `char __user *` buffer from a VFIO read or write callback.
+///
+/// Provides bounds-checked copying of kernel data into the user-space buffer.
+///
+/// This type cannot be constructed by driver code — it is only created by the
+/// callback trampoline.
+pub struct UserBuf {
+ ptr: UserPtr,
+ count: usize,
+}
+
+impl UserBuf {
+ /// Returns the buffer length in bytes.
+ pub fn len(&self) -> usize {
+ self.count
+ }
+
+ /// Returns `true` when the buffer is empty.
+ pub fn is_empty(&self) -> bool {
+ self.count == 0
+ }
+
+ /// Limit this callback's transfer to at most `count` bytes.
+ pub fn truncate(&mut self, count: usize) {
+ self.count = self.count.min(count);
+ }
+
+ /// Overwrite the part of `data` that overlaps a completed region read.
+ ///
+ /// `read_offset` is the starting region offset and `read_len` is the number of
+ /// bytes returned by the read. `data_offset` locates `data` within the same region.
+ pub fn write_overlapping(
+ &self,
+ read_offset: u64,
+ read_len: usize,
+ data_offset: u64,
+ data: &[u8],
+ ) -> Result {
+ if read_len > self.len() {
+ return Err(EINVAL);
+ }
+ let read_end = read_offset.checked_add(read_len as u64).ok_or(EOVERFLOW)?;
+ let data_end = data_offset
+ .checked_add(data.len() as u64)
+ .ok_or(EOVERFLOW)?;
+ let start = read_offset.max(data_offset);
+ let end = read_end.min(data_end);
+ if start >= end {
+ return Ok(());
+ }
+
+ // These differences are bounded by `read_len` and `data.len()`.
+ let buf_offset = (start - read_offset) as usize;
+ let data_start = (start - data_offset) as usize;
+ let data_end = (end - data_offset) as usize;
+ self.write_at(buf_offset, &data[data_start..data_end])
+ }
+
+ /// Copy `data` to the user buffer at byte offset `offset`.
+ ///
+ /// Returns [`EFAULT`] if the copy fails, [`EINVAL`] if the write would
+ /// exceed the buffer bounds.
+ fn write_at(&self, offset: usize, data: &[u8]) -> Result {
+ let end = offset.checked_add(data.len()).ok_or(EOVERFLOW)?;
+ if end > self.count {
+ return Err(EINVAL);
+ }
+ let dest = self.ptr.wrapping_byte_add(offset);
+ let mut writer = UserSlice::new(dest, data.len()).writer();
+ writer.write_slice(data)
+ }
+}
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
new file mode 100644
index 000000000000..0fca288fe6ae
--- /dev/null
+++ b/rust/kernel/vfio/pci.rs
@@ -0,0 +1,1109 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! VFIO PCI variant driver abstractions.
+//!
+//! Provides Rust abstractions for writing VFIO PCI variant drivers (also known
+//! as "VFIO PCI core" drivers). A [`Registration`] struct owns the VFIO device
+//! and its associated registration data, tying resource lifetimes to the PCI
+//! driver's binding scope.
+//!
+//! C header: [`include/linux/vfio_pci_core.h`](srctree/include/linux/vfio_pci_core.h)
+
+use super::{
+ InfoCap,
+ UserBuf, //
+};
+
+use crate::{
+ alloc::allocator::Kmalloc,
+ bindings,
+ device,
+ device_id::RawDeviceIdIndex,
+ driver,
+ error::{
+ from_err_ptr,
+ from_result,
+ to_result, //
+ },
+ mm::virt::VmaRef,
+ pci,
+ prelude::*,
+ sync::aref::{
+ ARef,
+ AlwaysRefCounted, //
+ },
+ types::{
+ CovariantForLt,
+ ForLt,
+ NotThreadSafe,
+ Opaque, //
+ },
+ uaccess::UserPtr,
+ ThisModule, //
+};
+use core::{
+ alloc::Layout,
+ cell::UnsafeCell,
+ marker::PhantomData,
+ ptr::NonNull, //
+};
+
+/// Index of the PCI configuration-space region.
+pub const CONFIG_REGION_INDEX: u32 = bindings::VFIO_PCI_CONFIG_REGION_INDEX;
+
+/// The context in which a VFIO PCI device reference is valid.
+///
+/// Callback views can only be borrowed from the corresponding VFIO trampoline.
+/// They cannot be refcounted or shared across threads.
+pub trait DeviceContext: private::Sealed {}
+
+/// An ordinary device reference, without callback-specific operations.
+pub struct Normal;
+
+/// The device is enabled but has not yet completed its first-open callback.
+pub struct Open;
+
+/// The device is borrowed for a VFIO ioctl callback on the current thread.
+pub struct Ioctl;
+
+/// The device is borrowed for a VFIO read callback on the current thread.
+pub struct Read;
+
+/// The device is borrowed for a VFIO write callback on the current thread.
+pub struct Write;
+
+/// The device is borrowed for a VFIO mmap callback on the current thread.
+pub struct Mmap;
+
+/// The device is borrowed for a VFIO region-info callback on the current thread.
+pub struct GetRegionInfo;
+
+mod private {
+ pub trait Sealed {}
+
+ impl Sealed for super::Normal {}
+ impl Sealed for super::Open {}
+ impl Sealed for super::Write {}
+ impl Sealed for super::Mmap {}
+ impl Sealed for super::Ioctl {}
+ impl Sealed for super::Read {}
+ impl Sealed for super::GetRegionInfo {}
+}
+
+impl DeviceContext for Normal {}
+impl DeviceContext for Open {}
+impl DeviceContext for Write {}
+impl DeviceContext for Mmap {}
+impl DeviceContext for Ioctl {}
+impl DeviceContext for Read {}
+impl DeviceContext for GetRegionInfo {}
+
+/// Connect a PCI driver's private data to its VFIO registration.
+///
+/// # Safety
+///
+/// `registration` must always select this PCI device's sole registration and must not sleep;
+/// VGA arbitration may invoke it with interrupts disabled. The driver must use [`Adapter`],
+/// and its `probe` and `unbind` callbacks must not enable SR-IOV.
+pub unsafe trait Driver: for<'a> pci::Driver<Data<'a>: Sync> + Sized {
+ /// The VFIO operations implemented by this driver.
+ type Operations: Operations;
+
+ /// Select the registration without blocking or changing its identity.
+ fn registration<'a, 'bound>(
+ data: Pin<&'a Self::Data<'bound>>,
+ ) -> &'a Registration<'bound, Self::Operations>;
+}
+
+/// Register a VFIO PCI variant driver and coordinate its callback lifetimes.
+///
+/// The adapter publishes complete private data before registering VFIO, and unregisters VFIO
+/// before calling the driver's optional `unbind` callback or removing private data. SR-IOV
+/// enable and disable callbacks are not supported by this adapter.
+pub struct Adapter<D: Driver>(D);
+
+// SAFETY: The C PCI driver embeds `device_driver` at the stated offset; its private data is
+// the pinned `D::Data` installed by this adapter's probe callback.
+unsafe impl<D: Driver> driver::DriverLayout for Adapter<D> {
+ type DriverType = bindings::pci_driver;
+ type DriverData<'bound> = D::Data<'bound>;
+ const DEVICE_DRIVER_OFFSET: usize = core::mem::offset_of!(Self::DriverType, driver);
+}
+
+// SAFETY: The driver core unregisters only a successfully registered, still-live PCI driver.
+unsafe impl<D: Driver> driver::RegistrationOps for Adapter<D> {
+ unsafe fn register(
+ pdrv: &Opaque<Self::DriverType>,
+ name: &'static CStr,
+ module: &'static ThisModule,
+ ) -> Result {
+ build_assert!(
+ D::DRIVER_MANAGED_DMA,
+ "VFIO drivers must manage DMA ownership"
+ );
+ #[cfg(CONFIG_PCI_IOV)]
+ build_assert!(
+ !D::HAS_SRIOV_ENABLE && !D::HAS_SRIOV_DISABLE,
+ "VFIO PCI variant drivers do not support SR-IOV configuration"
+ );
+
+ // SAFETY: The driver is being initialized and is not yet visible to the PCI core.
+ unsafe {
+ (*pdrv.get()).name = name.as_char_ptr();
+ (*pdrv.get()).probe = Some(Self::probe_callback);
+ (*pdrv.get()).remove = Some(Self::remove_callback);
+ (*pdrv.get()).id_table = D::ID_TABLE.as_ptr();
+ (*pdrv.get()).driver_managed_dma = D::DRIVER_MANAGED_DMA;
+ (*pdrv.get()).err_handler = &PciCallbacks::<D>::ERROR_HANDLERS;
+ (*pdrv.get()).driver.pm = &PciCallbacks::<D>::PM_OPS;
+ }
+
+ // SAFETY: The caller keeps this initialized driver pinned until unregistration.
+ to_result(unsafe {
+ bindings::__pci_register_driver(pdrv.get(), module.as_ptr(), name.as_char_ptr())
+ })
+ }
+
+ unsafe fn unregister(pdrv: &Opaque<Self::DriverType>) {
+ // SAFETY: The registration contract guarantees a live, registered PCI driver.
+ unsafe { bindings::pci_unregister_driver(pdrv.get()) };
+ }
+}
+
+impl<D: Driver> Adapter<D> {
+ extern "C" fn probe_callback(
+ pdev: *mut bindings::pci_dev,
+ id: *const bindings::pci_device_id,
+ ) -> c_int {
+ // SAFETY: PCI supplies a live device and holds its lock throughout probe.
+ let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
+ // SAFETY: `DeviceId` is a transparent wrapper with no additional invariants.
+ let id = unsafe { &*id.cast::<pci::DeviceId>() };
+ // SAFETY: The ID comes from `D::ID_TABLE`, a dynamic ID with matching driver_data,
+ // or the wildcard ID whose driver_data is zero.
+ let info = unsafe { id.info_unchecked_opt::<D::IdInfo>() };
+
+ from_result(|| {
+ pdev.as_ref().set_drvdata(D::probe(pdev, info))?;
+ // SAFETY: The fully initialized, pinned `D::Data` was just installed above.
+ let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
+ // SAFETY: `D` selects this device's sole registration. Private data is installed
+ // before VFIO can invoke VGA callbacks or enable runtime PM.
+ if let Err(error) = unsafe { D::registration(data).register::<D>() } {
+ // SAFETY: Failed VFIO registration has unwound its callbacks. In particular,
+ // rejecting a PF with enabled VFs must not disable those VFs during rollback.
+ drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
+ return Err(error);
+ }
+ Ok(0)
+ })
+ }
+
+ extern "C" fn remove_callback(pdev: *mut bindings::pci_dev) {
+ // SAFETY: PCI supplies a live device under its lock, after quiescing runtime PM.
+ let pdev = unsafe { &*pdev.cast::<pci::Device<device::CoreInternal<'_>>>() };
+ // SAFETY: A successful probe installed this device's complete private data.
+ let data = unsafe { pdev.as_ref().drvdata_borrow::<D::Data<'_>>() };
+ // SAFETY: Probe registered this object. Private data and all its fields remain live
+ // while VFIO unregister drains VFIO, VGA and PM callbacks.
+ unsafe { D::registration(data).unregister() };
+ D::unbind(pdev, data);
+ // SAFETY: Callbacks have drained, so private data can now be removed and destroyed.
+ drop(unsafe { pdev.as_ref().drvdata_obtain::<D::Data<'_>>() });
+ }
+}
+
+/// The trait for VFIO PCI variant driver implementations.
+///
+/// Unoverridden callbacks delegate to the corresponding PCI core operation.
+///
+/// # Lifetime of `RegistrationData`
+///
+/// `RegistrationData` describes a covariant type family borrowing from the PCI
+/// binding scope. [`Registration`] owns the data, and each callback borrows it.
+/// Covariance permits shortening the binding lifetime to the callback's borrow
+/// without allowing callback-local references to be stored in registration data.
+pub trait Operations: Sized + Send + Sync + 'static {
+ /// The name of the VFIO driver.
+ const NAME: &'static CStr;
+
+ /// Data owned by the [`Registration`] and passed to VFIO callbacks.
+ ///
+ /// Per-device state (e.g. the GFID, cached type info) lives here.
+ type RegistrationData: for<'a> CovariantForLt<Of<'a>: Send + Sync> + 'static;
+
+ /// Data shared by callbacks from the first successful open until the last close.
+ /// May borrow registration data; dropped before PCI core close on the last close.
+ type OpenData<'a>: Send + Sync + 'a;
+
+ /// Called when the first file descriptor is opened for this device.
+ ///
+ /// `vfio_pci_core_enable()` has already succeeded; if this returns an
+ /// error, `vfio_pci_core_disable()` is called automatically.
+ fn open_device<'a>(
+ dev: &'a Device<Self, Open>,
+ reg_data: &'a <Self::RegistrationData as ForLt>::Of<'a>,
+ ) -> impl PinInit<Self::OpenData<'a>, Error> + 'a;
+
+ /// Handle a VFIO ioctl.
+ ///
+ /// The default `vfio_pci_core_ioctl()` is available via
+ /// [`Device::core_ioctl()`] for delegation.
+ fn ioctl<'a>(
+ dev: &Device<Self, Ioctl>,
+ reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ cmd: u32,
+ arg: usize,
+ ) -> Result<isize>;
+
+ /// Read from the device.
+ ///
+ /// `buf` is a user-space buffer provided by the VFIO core. Use
+ /// [`Device::core_read()`] to delegate the default read, and
+ /// [`UserBuf::write_overlapping()`] to override specific bytes afterwards.
+ fn read<'a>(
+ dev: &Device<Self, Read>,
+ reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ buf: &mut UserBuf,
+ ppos: &mut Position<'_>,
+ ) -> Result<isize>;
+
+ /// Write to the device, delegating to [`Device::core_write()`] if appropriate.
+ fn write<'a>(
+ dev: &Device<Self, Write>,
+ _reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+ _open_data: Pin<&Self::OpenData<'a>>,
+ buf: &mut UserBuf,
+ ppos: &mut Position<'_>,
+ ) -> Result<isize> {
+ dev.core_write(buf, ppos)
+ }
+
+ /// Map a device region, delegating to [`Device::core_mmap()`] if appropriate.
+ fn mmap<'a>(
+ dev: &Device<Self, Mmap>,
+ _reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+ _open_data: Pin<&Self::OpenData<'a>>,
+ mapping: &mut Mapping<'_>,
+ ) -> Result {
+ dev.core_mmap(mapping)
+ }
+
+ /// Prepare for PCI reset, including resets initiated during enable or close.
+ fn reset_prepare(_reg_data: &<Self::RegistrationData as ForLt>::Of<'_>) {}
+
+ /// Finish PCI reset; errors are reported through the VFIO error interrupt.
+ fn reset_done(_reg_data: &<Self::RegistrationData as ForLt>::Of<'_>) -> Result {
+ Ok(())
+ }
+
+ /// Fill in region info capabilities.
+ ///
+ /// The default `vfio_pci_ioctl_get_region_info()` is available via
+ /// [`Device::core_get_region_info()`] for delegation.
+ fn get_region_info<'a>(
+ dev: &Device<Self, GetRegionInfo>,
+ reg_data: &<Self::RegistrationData as ForLt>::Of<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ info: &mut bindings::vfio_region_info,
+ caps: &mut InfoCap<'_>,
+ ) -> Result;
+}
+
+/// A VFIO PCI file position, encoding a region index and an offset within that region.
+pub struct Position<'a> {
+ raw: &'a mut i64,
+}
+
+impl Position<'_> {
+ /// Commit a temporary position only after the operation succeeds.
+ ///
+ /// This preserves the original position if a read succeeds but a subsequent
+ /// user copy fails.
+ pub fn with_temporary<R>(
+ &mut self,
+ operation: impl FnOnce(&mut Position<'_>) -> Result<R>,
+ ) -> Result<R> {
+ let mut next = *self.raw;
+ let result = operation(&mut Position { raw: &mut next })?;
+ *self.raw = next;
+ Ok(result)
+ }
+
+ /// Returns the PCI region index.
+ pub fn region_index(&self) -> u32 {
+ ((*self.raw as u64) >> bindings::VFIO_PCI_OFFSET_SHIFT) as u32
+ }
+
+ /// Returns the byte offset within the current region.
+ pub fn region_offset(&self) -> u64 {
+ (*self.raw as u64) & ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1)
+ }
+}
+
+/// Mapping metadata supplied by VFIO while the current thread holds the mmap lock.
+///
+/// The underlying VMA can only be passed back to the core mmap operation.
+pub struct Mapping<'a> {
+ vma: &'a VmaRef,
+ _not_thread_safe: NotThreadSafe,
+}
+
+impl Mapping<'_> {
+ fn page_offset(&self) -> u64 {
+ // SAFETY: The mmap callback holds the mmap lock for this VMA.
+ unsafe { (*self.vma.as_ptr()).vm_pgoff as u64 }
+ }
+
+ /// Returns the PCI region index encoded in the mapping offset.
+ pub fn region_index(&self) -> u32 {
+ (self.page_offset() >> (bindings::VFIO_PCI_OFFSET_SHIFT as usize - crate::page::PAGE_SHIFT))
+ as u32
+ }
+
+ /// Returns the byte offset within the PCI region.
+ fn region_offset(&self) -> u64 {
+ let mask = ((1u64 << bindings::VFIO_PCI_OFFSET_SHIFT) - 1) >> crate::page::PAGE_SHIFT;
+ (self.page_offset() & mask) << crate::page::PAGE_SHIFT
+ }
+
+ /// Returns the requested mapping length in bytes.
+ fn size(&self) -> Result<u64> {
+ self.vma
+ .end()
+ .checked_sub(self.vma.start())
+ .map(|size| size as u64)
+ .ok_or(EOVERFLOW)
+ }
+
+ /// Returns the exclusive end of the requested mapping within the PCI region.
+ pub fn region_end(&self) -> Result<u64> {
+ self.region_offset()
+ .checked_add(self.size()?)
+ .ok_or(EOVERFLOW)
+ }
+}
+
+/// A VFIO PCI variant device, wrapping `struct vfio_pci_core_device`.
+///
+/// The layout is `#[repr(C)]` with `core_device` first, so the
+/// `vfio_device` embedded at offset 0 of `vfio_pci_core_device` is also at
+/// offset 0 of `Self`, matching the `vfio_alloc_device` requirement.
+/// Callback contexts expose only their matching core helper and cannot be
+/// refcounted or shared across threads.
+///
+/// # Invariants
+///
+/// - Callback contexts are only borrowed for the corresponding callback and thread.
+/// - `core_device` was initialized by `vfio_pci_core_init_dev()`.
+/// - The VFIO device refcount owns the allocation lifetime.
+/// - `reg_data` is dangling without an owner or points to a valid
+/// `<T::RegistrationData as ForLt>::Of<'_>` owned by the enclosing [`Registration`].
+/// - `open_data` is null outside a successful open, otherwise it owns a pinned
+/// `KBox<T::OpenData<'_>>` until last close. Only open/close mutate it; VFIO
+/// prevents I/O callbacks from racing those transitions. The data is dropped
+/// before PCI core close and before registration data is freed.
+#[repr(C)]
+pub struct Device<T: Operations, Ctx: DeviceContext = Normal> {
+ core_device: Opaque<bindings::vfio_pci_core_device>,
+ reg_data: UnsafeCell<NonNull<<T::RegistrationData as ForLt>::Of<'static>>>,
+ open_data: UnsafeCell<*mut T::OpenData<'static>>,
+ _context: PhantomData<(Ctx, NotThreadSafe)>,
+}
+
+impl<T: Operations> Device<T> {
+ /// Allocate a VFIO PCI device for subsequent registration.
+ pub fn new(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+ Self::allocate(pdev, false)
+ }
+
+ /// Allocate a device whose physical BARs may also be exported as DMA buffers.
+ pub fn new_passthrough(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
+ Self::allocate(pdev, true)
+ }
+
+ fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<ARef<Self>> {
+ const_assert!(core::mem::offset_of!(Self, core_device) == 0);
+ let size = Kmalloc::aligned_layout(Layout::new::<Self>()).size();
+
+ // SAFETY: The bound PCI device and static ops are valid. The allocation
+ // includes the full Rust wrapper with kmalloc-compatible size and alignment.
+ let raw = from_err_ptr(unsafe {
+ bindings::_vfio_alloc_device(size, pdev.as_ref().as_raw(), &Self::OPS)
+ })?;
+ let this = NonNull::new(raw.cast::<Self>()).ok_or(ENOMEM)?;
+
+ // SAFETY: Initialise the Rust field in the newly allocated device.
+ unsafe {
+ (&raw mut (*this.as_ptr()).reg_data).write(UnsafeCell::new(NonNull::dangling()));
+ (&raw mut (*this.as_ptr()).open_data).write(UnsafeCell::new(core::ptr::null_mut()));
+ (&raw mut (*this.as_ptr())._context).write(PhantomData);
+ }
+
+ if passthrough {
+ // SAFETY: No registration or callbacks can observe this allocation yet.
+ unsafe { (*this.as_ref().core_device()).pci_ops = &Self::PASSTHROUGH_OPS };
+ }
+
+ // SAFETY: `this` owns the initial VFIO device reference.
+ Ok(unsafe { ARef::from_raw(this) })
+ }
+
+ /// Access the registration data through a closure with an HRTB lifetime.
+ ///
+ /// The closure's `for<'a>` bound prevents the caller from smuggling
+ /// references with a concrete short lifetime out of the closure.
+ ///
+ /// # Safety
+ ///
+ /// Registration data must be published and remain valid through this call.
+ unsafe fn registration_data_with<R>(
+ &self,
+ f: impl for<'a> FnOnce(&'a <T::RegistrationData as ForLt>::Of<'a>) -> R,
+ ) -> R {
+ // SAFETY: Registration keeps the allocation and its borrowed resources
+ // alive until unregistration completes. Covariance permits shortening
+ // the erased binding lifetime to this callback's borrow.
+ let reg_data: &<T::RegistrationData as ForLt>::Of<'_> = unsafe {
+ (*self.reg_data.get())
+ .cast::<<T::RegistrationData as ForLt>::Of<'_>>()
+ .as_ref()
+ };
+ f(reg_data)
+ }
+
+ /// # Safety
+ ///
+ /// The caller must be an I/O callback while VFIO keeps the device open.
+ unsafe fn callback_data_with<R>(
+ &self,
+ f: impl for<'borrow, 'data> FnOnce(
+ &'borrow <T::RegistrationData as ForLt>::Of<'data>,
+ Pin<&'borrow T::OpenData<'data>>,
+ ) -> R,
+ ) -> R {
+ // SAFETY: VFIO excludes open/close and unregister while this callback runs.
+ unsafe {
+ self.registration_data_with(|reg_data| {
+ let ptr = (*self.open_data.get()).cast::<T::OpenData<'_>>();
+ // The data lifetime stays independent of the callback borrow,
+ // so borrowed OpenData fields cannot be stored back into OpenData.
+ f(reg_data, Pin::new_unchecked(&*ptr))
+ })
+ }
+ }
+
+ /// # Safety
+ /// The returned view must only be used during the callback represented by `Ctx`.
+ unsafe fn with_context<Ctx: DeviceContext>(&self) -> &Device<T, Ctx> {
+ // SAFETY: All contexts have the same layout; the caller guarantees the context.
+ unsafe { &*core::ptr::from_ref(self).cast() }
+ }
+
+ /// Recover `&Self` from a raw `*mut vfio_device` in a callback.
+ ///
+ /// # Safety
+ ///
+ /// `vdev` must point to the `vfio_device` at the start of a
+ /// `Device<T>` allocated by `_vfio_alloc_device()` and remain valid for `'a`.
+ unsafe fn from_vfio_device<'a>(vdev: *mut bindings::vfio_device) -> &'a Self {
+ // SAFETY: `vfio_device` is at offset 0 of `vfio_pci_core_device`,
+ // which is at offset 0 of `Device<T>`, so the pointer cast is
+ // valid.
+ unsafe { &*(vdev.cast()) }
+ }
+}
+
+impl<T: Operations, Ctx: DeviceContext> Device<T, Ctx> {
+ fn core_device(&self) -> *mut bindings::vfio_pci_core_device {
+ self.core_device.get()
+ }
+
+ fn vfio_device(&self) -> *mut bindings::vfio_device {
+ self.core_device.get().cast()
+ }
+}
+
+impl<T: Operations> Device<T, Open> {
+ /// Set the device ID exposed through VFIO's virtual PCI configuration space.
+ pub fn set_device_id(&self, device_id: u16) {
+ // SAFETY: core_enable allocated vconfig; the exclusive first-open callback
+ // runs before finish_enable publishes the configuration to userspace.
+ unsafe {
+ (*self.core_device())
+ .vconfig
+ .add(bindings::PCI_DEVICE_ID as usize)
+ .cast::<u16>()
+ .write_unaligned(device_id.to_le());
+ }
+ }
+}
+
+impl<T: Operations> Device<T, Ioctl> {
+ /// Delegate to `vfio_pci_core_ioctl()`.
+ pub fn core_ioctl(&self, cmd: u32, arg: usize) -> Result<isize> {
+ // SAFETY: The Ioctl context keeps this device open on the callback thread.
+ let ret = unsafe { bindings::vfio_pci_core_ioctl(self.vfio_device(), cmd, arg) };
+ if ret < 0 {
+ Err(Error::from_errno(ret as i32))
+ } else {
+ Ok(ret)
+ }
+ }
+}
+
+impl<T: Operations> Device<T, Read> {
+ /// Delegate to `vfio_pci_core_read()`.
+ ///
+ /// The VFIO core fills the user-space buffer with the default PCI data
+ /// for the region identified by `ppos` and advances the position by the bytes read.
+ pub fn core_read(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+ // SAFETY: The Read context keeps this device open on the callback thread;
+ // `buf.ptr` is a user-space pointer supplied by VFIO.
+ let ret = unsafe {
+ bindings::vfio_pci_core_read(
+ self.vfio_device(),
+ buf.ptr.as_mut_ptr().cast(),
+ buf.count,
+ ppos.raw,
+ )
+ };
+ if ret < 0 {
+ Err(Error::from_errno(ret as i32))
+ } else {
+ Ok(ret)
+ }
+ }
+}
+
+impl<T: Operations> Device<T, Write> {
+ /// Delegate to `vfio_pci_core_write()` for this callback's bounded buffer.
+ pub fn core_write(&self, buf: &UserBuf, ppos: &mut Position<'_>) -> Result<isize> {
+ // SAFETY: The Write context keeps the device open on the callback thread;
+ // the buffer is a userspace pointer supplied by VFIO.
+ let ret = unsafe {
+ bindings::vfio_pci_core_write(
+ self.vfio_device(),
+ buf.ptr.as_const_ptr().cast(),
+ buf.count,
+ ppos.raw,
+ )
+ };
+ if ret < 0 {
+ Err(Error::from_errno(ret as i32))
+ } else {
+ Ok(ret)
+ }
+ }
+}
+
+impl<T: Operations> Device<T, Mmap> {
+ /// Delegate to `vfio_pci_core_mmap()` for the current callback's VMA.
+ pub fn core_mmap(&self, mapping: &mut Mapping<'_>) -> Result {
+ // SAFETY: The Mmap context and Mapping keep the open device and locked VMA
+ // on the callback thread. The VMA cannot be replaced by driver code.
+ to_result(unsafe { bindings::vfio_pci_core_mmap(self.vfio_device(), mapping.vma.as_ptr()) })
+ }
+}
+
+impl<T: Operations> Device<T, GetRegionInfo> {
+ /// Delegate to `vfio_pci_ioctl_get_region_info()`.
+ pub fn core_get_region_info(
+ &self,
+ info: &mut bindings::vfio_region_info,
+ caps: &mut InfoCap<'_>,
+ ) -> Result {
+ // SAFETY: GetRegionInfo keeps the device open on the callback thread;
+ // `InfoCap` preserves the validity of VFIO's capability buffer.
+ to_result(unsafe {
+ bindings::vfio_pci_ioctl_get_region_info(self.vfio_device(), info, caps.raw)
+ })
+ }
+}
+
+// SAFETY: The embedded device reference count owns the VFIO allocation.
+unsafe impl<T: Operations> AlwaysRefCounted for Device<T> {
+ fn inc_ref(&self) {
+ // SAFETY: `self` holds a live VFIO device reference.
+ unsafe { bindings::get_device(&raw mut (*self.vfio_device()).device) };
+ }
+
+ unsafe fn dec_ref(obj: NonNull<Self>) {
+ // SAFETY: The caller owns the reference being released.
+ unsafe { bindings::put_device(&raw mut (*obj.as_ref().vfio_device()).device) };
+ }
+}
+
+// SAFETY: VFIO serializes open/close transitions, and callback data is Send + Sync.
+unsafe impl<T: Operations> Send for Device<T> {}
+
+// SAFETY: I/O callbacks share Sync data. VFIO excludes them from mutations of
+// `open_data` during first open and last close; registration data follows the
+// registration lifetime. Mutable C state is managed by the VFIO core.
+unsafe impl<T: Operations> Sync for Device<T> {}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` in VFIO's first-open context.
+/// VFIO must exclude other opens, closes and I/O callbacks for this call.
+unsafe extern "C" fn open_device_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+) -> core::ffi::c_int {
+ // SAFETY: `vdev` is valid; set by vfio_alloc_device.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+
+ // SAFETY: Enable the PCI-core side first.
+ let ret = unsafe { bindings::vfio_pci_core_enable(dev.core_device()) };
+ if ret != 0 {
+ return ret;
+ }
+
+ // SAFETY: Registration data is initialized before the VFIO device is published.
+ let result = unsafe {
+ dev.registration_data_with(|rd| {
+ // Allocate before calling the driver so allocation failure cannot follow open.
+ let data = KBox::<T::OpenData<'_>>::new_uninit(GFP_KERNEL)?;
+ let data = data.write_pin_init(T::open_device(dev.with_context::<Open>(), rd))?;
+
+ // SAFETY: Only the owning pointer is moved; the allocation remains pinned.
+ let raw =
+ KBox::into_raw(Pin::into_inner_unchecked(data)).cast::<T::OpenData<'static>>();
+ // SAFETY: First open is exclusive with I/O callbacks and last close.
+ // Lifetimes do not affect layout; the device owns the allocation until close.
+ *dev.open_data.get() = raw;
+ Ok::<(), Error>(())
+ })
+ };
+ match result {
+ Ok(()) => {
+ // SAFETY: open succeeded.
+ unsafe { bindings::vfio_pci_core_finish_enable(dev.core_device()) };
+ 0
+ }
+ Err(e) => {
+ // SAFETY: Undo the enable on failure.
+ unsafe { bindings::vfio_pci_core_disable(dev.core_device()) };
+ e.to_errno()
+ }
+ }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered `Device<T>` whose first open succeeded.
+/// Call exactly once for the last close, after draining I/O callbacks and while
+/// excluding the next first open.
+unsafe extern "C" fn close_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) {
+ // SAFETY: `vdev` is valid.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ // SAFETY: Last close is exclusive with I/O callbacks and the next first open.
+ let raw = unsafe { core::mem::replace(&mut *dev.open_data.get(), core::ptr::null_mut()) };
+ // Run the driver's destructor while registration data and PCI resources are valid.
+ // SAFETY: Successful open transferred this allocation with `KBox::into_raw`.
+ // Last close takes ownership exactly once and drops it without moving the pointee.
+ unsafe { drop(KBox::from_raw(raw)) };
+ // SAFETY: Matches the enable in open_device_cb.
+ unsafe { bindings::vfio_pci_core_close_device(vdev) };
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's ioctl callback
+/// context. Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn ioctl_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+ cmd: core::ffi::c_uint,
+ arg: usize,
+) -> isize {
+ // SAFETY: `vdev` is valid.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+ match unsafe {
+ dev.callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
+ } {
+ Ok(v) => v,
+ Err(e) => e.to_errno() as isize,
+ }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's read context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn read_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+ buf: *mut u8,
+ count: usize,
+ ppos: *mut i64,
+) -> isize {
+ // SAFETY: `vdev` is valid. `buf` is a valid user-space pointer provided by
+ // the VFIO core. `ppos` is a valid kernel pointer.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ let mut ubuf = UserBuf {
+ ptr: UserPtr::from_ptr(buf.cast()),
+ count,
+ };
+ let mut pos = Position {
+ // SAFETY: `ppos` is a valid kernel pointer provided by the VFIO core.
+ raw: unsafe { &mut *ppos },
+ };
+ // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+ match unsafe {
+ dev.callback_data_with(|rd, od| {
+ T::read(dev.with_context::<Read>(), rd, od, &mut ubuf, &mut pos)
+ })
+ } {
+ Ok(n) => n,
+ Err(e) => e.to_errno() as isize,
+ }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's region-info
+/// context. `info` and `caps` must be valid, disjoint, exclusively accessible
+/// objects; the capability buffer must satisfy the VFIO core's invariants.
+/// Registration and open data must remain alive until the call returns.
+unsafe extern "C" fn get_region_info_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+ info: *mut bindings::vfio_region_info,
+ caps: *mut bindings::vfio_info_cap,
+) -> core::ffi::c_int {
+ // SAFETY: `vdev`, `info`, and `caps` are valid kernel pointers provided
+ // by the VFIO core.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ // SAFETY: `info` is a valid pointer provided by the VFIO core.
+ let info = unsafe { &mut *info };
+ let mut caps = InfoCap {
+ // SAFETY: `caps` is a valid pointer provided by the VFIO core.
+ raw: unsafe { &mut *caps },
+ };
+ // SAFETY: VFIO invokes this callback for an open device with valid arguments.
+ match unsafe {
+ dev.callback_data_with(|rd, od| {
+ T::get_region_info(dev.with_context::<GetRegionInfo>(), rd, od, info, &mut caps)
+ })
+ } {
+ Ok(()) => 0,
+ Err(e) => e.to_errno(),
+ }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's write context.
+/// `buf` must be a userspace address and `ppos` must be exclusively accessible
+/// for this call. Registration and open data must remain alive until it returns.
+unsafe extern "C" fn write_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+ buf: *const u8,
+ count: usize,
+ ppos: *mut i64,
+) -> isize {
+ // SAFETY: VFIO supplies an open device and a valid file position pointer.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ let mut ubuf = UserBuf {
+ ptr: UserPtr::from_ptr(buf.cast_mut().cast()),
+ count,
+ };
+ let mut pos = Position {
+ // SAFETY: VFIO holds the file-position lock when required.
+ raw: unsafe { &mut *ppos },
+ };
+ // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
+ match unsafe {
+ dev.callback_data_with(|rd, od| {
+ T::write(dev.with_context::<Write>(), rd, od, &mut ubuf, &mut pos)
+ })
+ } {
+ Ok(n) => n,
+ Err(e) => e.to_errno() as isize,
+ }
+}
+
+/// # Safety
+///
+/// `vdev` must belong to a registered, open `Device<T>` in VFIO's mmap context.
+/// `vma` must be the valid VMA being mapped, with the mmap write lock held on
+/// this thread. Registration and open data must remain alive until return.
+unsafe extern "C" fn mmap_cb<T: Operations>(
+ vdev: *mut bindings::vfio_device,
+ vma: *mut bindings::vm_area_struct,
+) -> core::ffi::c_int {
+ // SAFETY: VFIO supplies an open device for this mmap callback.
+ let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
+ let mut mapping = Mapping {
+ // SAFETY: The mmap callback holds the mmap lock for this VMA.
+ vma: unsafe { VmaRef::from_raw(vma) },
+ _not_thread_safe: NotThreadSafe,
+ };
+ // SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
+ match unsafe {
+ dev.callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
+ } {
+ Ok(()) => 0,
+ Err(e) => e.to_errno(),
+ }
+}
+
+/// Callbacks that recover the VFIO device through its PCI driver's private data.
+struct PciCallbacks<D>(PhantomData<D>);
+
+impl<D: Driver> PciCallbacks<D> {
+ /// # Safety
+ ///
+ /// `dev` must belong to `D`. Its private data must remain installed and valid for `'a`,
+ /// or not yet be installed. The caller must exclude its removal; runtime PM and VGA
+ /// callbacks use their own synchronization rather than the PCI device lock.
+ unsafe fn registration<'a>(
+ dev: *mut bindings::device,
+ ) -> Option<&'a Registration<'a, D::Operations>> {
+ // SAFETY: The callback supplies a live device bound to `D`.
+ let data = unsafe { bindings::dev_get_drvdata(dev) };
+ if data.is_null() {
+ return None;
+ }
+ // SAFETY: PCI published a fully initialized, pinned `D::Data`. The caller prevents
+ // its removal, and `D::Data: Sync` permits the shared PM and VGA callback borrows.
+ let data = unsafe { Pin::new_unchecked(&*data.cast::<D::Data<'_>>()) };
+ Some(D::registration(data))
+ }
+
+ unsafe extern "C" fn reset_prepare(pdev: *mut bindings::pci_dev) {
+ // SAFETY: PCI holds the device lock across reset and registration teardown.
+ let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+ return;
+ };
+ // SAFETY: The registration owns callback data throughout bind and unbind.
+ unsafe { reg.dev.registration_data_with(D::Operations::reset_prepare) };
+ }
+
+ unsafe extern "C" fn reset_done(pdev: *mut bindings::pci_dev) {
+ // SAFETY: PCI holds the device lock across reset and registration teardown.
+ let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+ return;
+ };
+ // SAFETY: The registration owns callback data throughout bind and unbind.
+ let result = unsafe { reg.dev.registration_data_with(D::Operations::reset_done) };
+ if let Err(error) = result {
+ // SAFETY: The PCI callback keeps its embedded device alive.
+ let dev = unsafe { device::Device::<device::Normal>::from_raw(&raw mut (*pdev).dev) };
+ dev_err!(dev, "VFIO device reset failed: {:?}\n", error);
+ // SAFETY: The initialized VFIO device remains alive for this callback.
+ unsafe {
+ bindings::vfio_pci_core_aer_err_detected(
+ reg.dev.core_device(),
+ bindings::pci_channel_io_normal,
+ );
+ }
+ }
+ }
+
+ unsafe extern "C" fn error_detected(
+ pdev: *mut bindings::pci_dev,
+ state: bindings::pci_channel_state_t,
+ ) -> bindings::pci_ers_result_t {
+ // SAFETY: PCI serializes error callbacks with binding and unbinding.
+ let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+ return bindings::pci_ers_result_PCI_ERS_RESULT_CAN_RECOVER;
+ };
+ // SAFETY: The registered device owns its error notification state.
+ unsafe { bindings::vfio_pci_core_aer_err_detected(reg.dev.core_device(), state) }
+ }
+
+ unsafe extern "C" fn runtime_suspend(dev: *mut bindings::device) -> c_int {
+ // SAFETY: Registration enables PM only after private data is installed. PCI remove
+ // and VFIO unregister quiesce runtime PM before private data can be removed.
+ let Some(reg) = (unsafe { Self::registration(dev) }) else {
+ return EBUSY.to_errno();
+ };
+ // SAFETY: This is the runtime-suspend callback for the initialized core device.
+ unsafe { bindings::vfio_pci_core_runtime_suspend(reg.dev.core_device()) }
+ }
+
+ unsafe extern "C" fn runtime_resume(dev: *mut bindings::device) -> c_int {
+ // SAFETY: Private data remains installed until VFIO unregister has resumed the
+ // device and quiesced PM, including registration failure.
+ let Some(reg) = (unsafe { Self::registration(dev) }) else {
+ return ENODEV.to_errno();
+ };
+ // SAFETY: This is the runtime-resume callback for the initialized core device.
+ unsafe { bindings::vfio_pci_core_runtime_resume(reg.dev.core_device()) }
+ }
+
+ unsafe extern "C" fn vga_set_decode(pdev: *mut bindings::pci_dev, single_vga: bool) -> c_uint {
+ // SAFETY: VFIO registers VGA callbacks after private data is installed and removes
+ // them synchronously before unbind returns. The selector does not sleep.
+ let Some(reg) = (unsafe { Self::registration(&raw mut (*pdev).dev) }) else {
+ return 0;
+ };
+ // SAFETY: VGA arbitration keeps this callback's initialized core device alive.
+ unsafe { bindings::vfio_pci_core_vga_set_decode(reg.dev.core_device(), single_vga) }
+ }
+
+ const ERROR_HANDLERS: bindings::pci_error_handlers = bindings::pci_error_handlers {
+ error_detected: Some(Self::error_detected),
+ mmio_enabled: None,
+ slot_reset: None,
+ reset_prepare: Some(Self::reset_prepare),
+ reset_done: Some(Self::reset_done),
+ resume: None,
+ cor_error_detected: None,
+ };
+
+ const PM_OPS: bindings::dev_pm_ops = bindings::dev_pm_ops {
+ runtime_suspend: Some(Self::runtime_suspend),
+ runtime_resume: Some(Self::runtime_resume),
+ ..pin_init::zeroed()
+ };
+}
+
+/// Build the `vfio_device_ops` vtable for a variant driver `T`.
+///
+/// Open and I/O callbacks dispatch to `T`'s trait implementation.
+/// All other callbacks use the `vfio_pci_core_*` defaults.
+macro_rules! build_ops {
+ ($T:ty) => {
+ bindings::vfio_device_ops {
+ #[allow(clippy::disallowed_methods)]
+ name: <$T>::NAME.as_ptr().cast_mut().cast(),
+ init: Some(bindings::vfio_pci_core_init_dev),
+ release: Some(bindings::vfio_pci_core_release_dev),
+ open_device: Some(open_device_cb::<$T>),
+ close_device: Some(close_device_cb::<$T>),
+ ioctl: Some(ioctl_cb::<$T>),
+ read: Some(read_cb::<$T>),
+ write: Some(write_cb::<$T>),
+ mmap: Some(mmap_cb::<$T>),
+ request: Some(bindings::vfio_pci_core_request),
+ get_region_info_caps: Some(get_region_info_cb::<$T>),
+ match_: Some(bindings::vfio_pci_core_match),
+ match_token_uuid: Some(bindings::vfio_pci_core_match_token_uuid),
+ device_feature: Some(bindings::vfio_pci_core_ioctl_feature),
+ #[cfg(CONFIG_IOMMUFD)]
+ bind_iommufd: Some(bindings::vfio_iommufd_physical_bind),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ bind_iommufd: None,
+ #[cfg(CONFIG_IOMMUFD)]
+ unbind_iommufd: Some(bindings::vfio_iommufd_physical_unbind),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ unbind_iommufd: None,
+ #[cfg(CONFIG_IOMMUFD)]
+ attach_ioas: Some(bindings::vfio_iommufd_physical_attach_ioas),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ attach_ioas: None,
+ #[cfg(CONFIG_IOMMUFD)]
+ detach_ioas: Some(bindings::vfio_iommufd_physical_detach_ioas),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ detach_ioas: None,
+ #[cfg(CONFIG_IOMMUFD)]
+ pasid_attach_ioas: Some(bindings::vfio_iommufd_physical_pasid_attach_ioas),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ pasid_attach_ioas: None,
+ #[cfg(CONFIG_IOMMUFD)]
+ pasid_detach_ioas: Some(bindings::vfio_iommufd_physical_pasid_detach_ioas),
+ #[cfg(not(CONFIG_IOMMUFD))]
+ pasid_detach_ioas: None,
+ dma_unmap: None,
+ }
+ };
+}
+
+/// The registration of a VFIO PCI variant device.
+///
+/// Owns both the VFIO device reference and the registration data, tying
+/// resource lifetimes to the PCI binding scope `'a`.
+///
+/// Prepare this owner during probe. [`Adapter`] registers it after private data is installed
+/// and unregisters it before private data is removed. Dropping the owner releases the VFIO
+/// allocation and registration data after callbacks have drained.
+pub struct Registration<'a, T: Operations> {
+ dev: ARef<Device<T>>,
+ _reg_data: Pin<KBox<<T::RegistrationData as ForLt>::Of<'a>>>,
+}
+
+impl<T: Operations> Device<T> {
+ const OPS: bindings::vfio_device_ops = build_ops!(T);
+
+ const PASSTHROUGH_OPS: bindings::vfio_pci_device_ops = bindings::vfio_pci_device_ops {
+ #[cfg(CONFIG_VFIO_PCI_DMABUF)]
+ get_dmabuf_phys: Some(bindings::vfio_pci_core_get_dmabuf_phys),
+ #[cfg(not(CONFIG_VFIO_PCI_DMABUF))]
+ get_dmabuf_phys: None,
+ };
+}
+
+impl<'a, T: Operations> Registration<'a, T> {
+ /// Prepare a previously allocated VFIO PCI device and its callback data.
+ ///
+ /// # Safety
+ ///
+ /// The device must have been allocated during this parent binding and never registered.
+ /// No other `Registration` may own or publish callback data for this device until this
+ /// owner is dropped.
+ /// The caller must hold the PCI device lock. The owner must be stored in the private data
+ /// of a driver registered through [`Adapter`] and selected by [`Driver::registration`].
+ pub unsafe fn new(
+ pdev: &'a pci::Device<device::Core<'_>>,
+ dev: &Device<T>,
+ reg_data: impl PinInit<<T::RegistrationData as ForLt>::Of<'a>, Error>,
+ ) -> Result<Self> {
+ // SAFETY: `dev` holds a live VFIO allocation; only compare its parent pointer.
+ if unsafe { (*dev.vfio_device()).dev } != pdev.as_ref().as_raw() {
+ return Err(EINVAL);
+ }
+
+ let reg_data: Pin<KBox<<T::RegistrationData as ForLt>::Of<'a>>> =
+ KBox::pin_init(reg_data, GFP_KERNEL)?;
+ let ptr: NonNull<<T::RegistrationData as ForLt>::Of<'static>> =
+ NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
+
+ // SAFETY: No concurrent registration or callbacks; publish before registering.
+ unsafe { *dev.reg_data.get() = ptr };
+
+ Ok(Self {
+ dev: dev.into(),
+ _reg_data: reg_data,
+ })
+ }
+
+ /// Register after the PCI adapter has installed the driver's private data.
+ ///
+ /// # Safety
+ ///
+ /// Call once under the PCI device lock, after installing complete `D::Data` whose selector
+ /// returns `self`. On success, unregister before private data is removed or destroyed.
+ unsafe fn register<D: Driver<Operations = T>>(&self) -> Result {
+ // SAFETY: The caller has published complete private data under the PCI device lock;
+ // it remains accessible during the synchronous VGA callback and runtime PM setup.
+ to_result(unsafe {
+ bindings::vfio_pci_core_register_device(
+ self.dev.core_device(),
+ Some(PciCallbacks::<D>::vga_set_decode),
+ )
+ })
+ }
+
+ /// Unregister while callbacks can still recover the PCI driver's private data.
+ ///
+ /// # Safety
+ ///
+ /// Call once after successful registration, with the PCI device lock held and private
+ /// data still installed. No private-data fields may have been destroyed.
+ unsafe fn unregister(&self) {
+ // SAFETY: The caller keeps private data installed while unregistration drains VFIO,
+ // VGA and runtime PM callbacks. No field of the enclosing data is being destroyed.
+ unsafe { bindings::vfio_pci_core_unregister_device(self.dev.core_device()) };
+ }
+}
+
+impl<T: Operations> Drop for Registration<'_, T> {
+ fn drop(&mut self) {
+ // SAFETY: The adapter unregisters before removing private data. Failed or unattempted
+ // registration also leaves no callbacks behind.
+ unsafe { *self.dev.reg_data.get() = NonNull::dangling() };
+ }
+}
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 5/7] rust: vfio: separate common device handling from PCI
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
` (3 preceding siblings ...)
2026-09-30 10:57 ` [PATCH 4/7] rust: vfio: add PCI variant driver abstractions Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 10:57 ` [PATCH 6/7] gpu: nova-core: publish typed SR-IOV PF APIs Zhi Wang
2026-09-30 10:57 ` [PATCH 7/7] vfio/nvidia-vgpu: add the Rust VFIO variant driver Zhi Wang
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
A VFIO PCI device contains a VFIO device. Expose that relationship in
Rust so common device handling is independent of PCI resources.
Add vfio::Device and have vfio::pci::Device borrow its embedded device
through AsRef, preserving the callback context. Share reference counting
and callback-data lifetime handling in vfio. Keep driver operations,
allocation, registration and the PCI enable/close sequence in vfio::pci.
No functional change is intended.
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
rust/kernel/vfio.rs | 200 +++++++++++++++++++++++++++++++++++++-
rust/kernel/vfio/pci.rs | 207 +++++++++++++---------------------------
2 files changed, 263 insertions(+), 144 deletions(-)
diff --git a/rust/kernel/vfio.rs b/rust/kernel/vfio.rs
index ed93066e2da5..a0bfbc7c59a0 100644
--- a/rust/kernel/vfio.rs
+++ b/rust/kernel/vfio.rs
@@ -1,14 +1,29 @@
// SPDX-License-Identifier: GPL-2.0
-//! Virtual Function I/O (VFIO) abstractions.
+//! Virtual Function I/O (VFIO) devices and callback data.
+//!
+//! Bus-specific device types borrow their embedded VFIO device through [`AsRef`].
+//! C header: [`include/linux/vfio.h`](srctree/include/linux/vfio.h)
use crate::{
bindings,
prelude::*,
+ sync::aref::AlwaysRefCounted,
+ types::{
+ CovariantForLt,
+ ForLt,
+ NotThreadSafe,
+ Opaque, //
+ },
uaccess::{
UserPtr,
UserSlice, //
- },
+ }, //
+};
+use core::{
+ cell::UnsafeCell,
+ marker::PhantomData,
+ ptr::NonNull, //
};
#[cfg(CONFIG_VFIO_PCI_CORE)]
@@ -17,12 +32,187 @@
/// Reset the VFIO device.
pub const DEVICE_RESET: u32 = bindings::VFIO_DEVICE_RESET;
-/// Reset the PCI slot or bus containing a VFIO device.
-pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+/// The context in which a VFIO device reference is valid.
+///
+/// Callback views can only be borrowed from the corresponding VFIO trampoline.
+/// They cannot be refcounted or shared across threads.
+pub trait DeviceContext: private::Sealed {}
+
+/// An ordinary device reference, without callback-specific operations.
+pub struct Normal;
+
+/// The device is undergoing its exclusive first-open callback.
+pub struct Open;
+
+/// The device is borrowed for a VFIO ioctl callback on the current thread.
+pub struct Ioctl;
+
+/// The device is borrowed for a VFIO read callback on the current thread.
+pub struct Read;
+
+/// The device is borrowed for a VFIO write callback on the current thread.
+pub struct Write;
+
+/// The device is borrowed for a VFIO mmap callback on the current thread.
+pub struct Mmap;
+
+/// The device is borrowed for a VFIO region-info callback on the current thread.
+pub struct GetRegionInfo;
+
+mod private {
+ pub trait Sealed {}
+
+ impl Sealed for super::Normal {}
+ impl Sealed for super::Open {}
+ impl Sealed for super::Write {}
+ impl Sealed for super::Mmap {}
+ impl Sealed for super::Ioctl {}
+ impl Sealed for super::Read {}
+ impl Sealed for super::GetRegionInfo {}
+}
+
+impl DeviceContext for Normal {}
+impl DeviceContext for Open {}
+impl DeviceContext for Write {}
+impl DeviceContext for Mmap {}
+impl DeviceContext for Ioctl {}
+impl DeviceContext for Read {}
+impl DeviceContext for GetRegionInfo {}
+
+/// A VFIO device, independent of its bus-specific wrapper.
+///
+/// # Invariants
+///
+/// The underlying `vfio_device` is initialized and remains alive while borrowed.
+/// Callback contexts are borrowed only for the corresponding callback and thread;
+/// only [`Normal`] references can be reference-counted.
+#[repr(transparent)]
+pub struct Device<Ctx: DeviceContext = Normal> {
+ raw: Opaque<bindings::vfio_device>,
+ _context: PhantomData<(Ctx, NotThreadSafe)>,
+}
+
+impl<Ctx: DeviceContext> Device<Ctx> {
+ /// # Safety
+ ///
+ /// `raw` must point to an initialized `vfio_device` that remains alive for
+ /// `'a`. The caller must provide the guarantees of `Ctx` throughout the borrow.
+ #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
+ unsafe fn from_raw<'a>(raw: *mut bindings::vfio_device) -> &'a Self {
+ // SAFETY: Self is a transparent wrapper; the caller guarantees validity and context.
+ unsafe { &*raw.cast() }
+ }
+
+ fn as_raw(&self) -> *mut bindings::vfio_device {
+ self.raw.get()
+ }
+}
+
+// SAFETY: VFIO uses the embedded class device's refcount to own the full allocation.
+unsafe impl AlwaysRefCounted for Device {
+ fn inc_ref(&self) {
+ // SAFETY: A shared reference keeps the initialized VFIO device alive.
+ unsafe { bindings::get_device(&raw mut (*self.as_raw()).device) };
+ }
+
+ unsafe fn dec_ref(obj: NonNull<Self>) {
+ // SAFETY: The caller owns the VFIO reference being released.
+ unsafe { bindings::put_device(&raw mut (*obj.as_ref().as_raw()).device) };
+ }
+}
+
+// SAFETY: Ordinary VFIO references expose no unsynchronized callback operations.
+unsafe impl Send for Device {}
+// SAFETY: Shared access only changes the embedded class device's reference count.
+unsafe impl Sync for Device {}
+
+/// Callback data stored after the bus-specific C device structure.
+///
+/// # Invariants
+///
+/// `reg_data` is dangling when unpublished; otherwise it borrows pinned data kept
+/// alive until callback access has ended. `open_data` is null outside a successful
+/// first open, otherwise owns a pinned allocation until the last close.
+/// The bus-specific layer drains callbacks before releasing either allocation.
+struct CallbackData<R: CovariantForLt, O: ForLt> {
+ reg_data: UnsafeCell<NonNull<R::Of<'static>>>,
+ open_data: UnsafeCell<*mut O::Of<'static>>,
+}
+
+#[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
+impl<R: CovariantForLt, O: ForLt> CallbackData<R, O> {
+ fn new() -> Self {
+ Self {
+ reg_data: UnsafeCell::new(NonNull::dangling()),
+ open_data: UnsafeCell::new(core::ptr::null_mut()),
+ }
+ }
+
+ /// Access registration data without allowing its erased lifetime to escape.
+ ///
+ /// # Safety
+ ///
+ /// Registration data must be published and remain valid through this call.
+ /// The caller must exclude concurrent changes to `reg_data`.
+ unsafe fn registration_data_with<V>(&self, f: impl for<'a> FnOnce(&'a R::Of<'a>) -> V) -> V {
+ // SAFETY: The caller keeps registration data alive and excludes mutation.
+ // Covariance permits shortening its erased lifetime to this borrow.
+ let reg_data = unsafe { (*self.reg_data.get()).cast::<R::Of<'_>>().as_ref() };
+ f(reg_data)
+ }
+
+ /// # Safety
+ ///
+ /// The caller must be an I/O callback while VFIO keeps the device open.
+ /// Registration and open data must remain valid, with their pointer slots
+ /// unchanged throughout the call.
+ unsafe fn callback_data_with<V>(
+ &self,
+ f: impl for<'borrow, 'data> FnOnce(&'borrow R::Of<'data>, Pin<&'borrow O::Of<'data>>) -> V,
+ ) -> V {
+ // SAFETY: The caller excludes open/close and unregister while borrowing the data.
+ unsafe {
+ self.registration_data_with(|reg_data| {
+ let ptr = (*self.open_data.get()).cast::<O::Of<'_>>();
+ // The data lifetime stays independent of the callback borrow,
+ // so borrowed OpenData fields cannot be stored back into OpenData.
+ f(reg_data, Pin::new_unchecked(&*ptr))
+ })
+ }
+ }
+
+ /// # Safety
+ ///
+ /// Call only during exclusive first open, with no existing open data.
+ /// All resources borrowed for `'a` must remain valid until `close()` returns.
+ unsafe fn open<'a, I: PinInit<O::Of<'a>, Error>>(&self, init: impl FnOnce() -> I) -> Result {
+ // Allocate before calling the driver so allocation failure cannot follow open.
+ let data = KBox::<O::Of<'a>>::new_uninit(GFP_KERNEL)?;
+ let data = data.write_pin_init(init())?;
+
+ // SAFETY: Only the owning pointer moves; close drops the allocation in place.
+ let raw = KBox::into_raw(unsafe { Pin::into_inner_unchecked(data) });
+ // SAFETY: First open is exclusive; the caller keeps borrowed resources alive
+ // through close. Lifetimes do not affect the allocation's layout.
+ unsafe { *self.open_data.get() = raw.cast::<O::Of<'static>>() };
+ Ok(())
+ }
+
+ /// # Safety
+ ///
+ /// Call once after a successful open, with all callbacks drained and the next
+ /// open excluded. Registration data and borrowed resources must remain alive.
+ unsafe fn close(&self) {
+ // SAFETY: Last close exclusively takes the allocation published by open.
+ let raw = unsafe { core::mem::replace(&mut *self.open_data.get(), core::ptr::null_mut()) };
+ // SAFETY: The allocation came from KBox::into_raw and is destroyed without moving it.
+ unsafe { drop(KBox::from_raw(raw)) };
+ }
+}
/// Capability buffer supplied by VFIO for a region-info callback.
-#[cfg(CONFIG_VFIO_PCI_CORE)]
pub struct InfoCap<'a> {
+ #[cfg_attr(not(CONFIG_VFIO_PCI_CORE), expect(dead_code))]
raw: &'a mut bindings::vfio_info_cap,
}
diff --git a/rust/kernel/vfio/pci.rs b/rust/kernel/vfio/pci.rs
index 0fca288fe6ae..20a89a113751 100644
--- a/rust/kernel/vfio/pci.rs
+++ b/rust/kernel/vfio/pci.rs
@@ -10,9 +10,20 @@
//! C header: [`include/linux/vfio_pci_core.h`](srctree/include/linux/vfio_pci_core.h)
use super::{
+ CallbackData,
InfoCap,
UserBuf, //
};
+pub use super::{
+ DeviceContext,
+ GetRegionInfo,
+ Ioctl,
+ Mmap,
+ Normal,
+ Open,
+ Read,
+ Write, //
+};
use crate::{
alloc::allocator::Kmalloc,
@@ -43,61 +54,16 @@
};
use core::{
alloc::Layout,
- cell::UnsafeCell,
marker::PhantomData,
ptr::NonNull, //
};
+/// Reset the PCI slot or bus containing a VFIO device.
+pub const DEVICE_PCI_HOT_RESET: u32 = bindings::VFIO_DEVICE_PCI_HOT_RESET;
+
/// Index of the PCI configuration-space region.
pub const CONFIG_REGION_INDEX: u32 = bindings::VFIO_PCI_CONFIG_REGION_INDEX;
-/// The context in which a VFIO PCI device reference is valid.
-///
-/// Callback views can only be borrowed from the corresponding VFIO trampoline.
-/// They cannot be refcounted or shared across threads.
-pub trait DeviceContext: private::Sealed {}
-
-/// An ordinary device reference, without callback-specific operations.
-pub struct Normal;
-
-/// The device is enabled but has not yet completed its first-open callback.
-pub struct Open;
-
-/// The device is borrowed for a VFIO ioctl callback on the current thread.
-pub struct Ioctl;
-
-/// The device is borrowed for a VFIO read callback on the current thread.
-pub struct Read;
-
-/// The device is borrowed for a VFIO write callback on the current thread.
-pub struct Write;
-
-/// The device is borrowed for a VFIO mmap callback on the current thread.
-pub struct Mmap;
-
-/// The device is borrowed for a VFIO region-info callback on the current thread.
-pub struct GetRegionInfo;
-
-mod private {
- pub trait Sealed {}
-
- impl Sealed for super::Normal {}
- impl Sealed for super::Open {}
- impl Sealed for super::Write {}
- impl Sealed for super::Mmap {}
- impl Sealed for super::Ioctl {}
- impl Sealed for super::Read {}
- impl Sealed for super::GetRegionInfo {}
-}
-
-impl DeviceContext for Normal {}
-impl DeviceContext for Open {}
-impl DeviceContext for Write {}
-impl DeviceContext for Mmap {}
-impl DeviceContext for Ioctl {}
-impl DeviceContext for Read {}
-impl DeviceContext for GetRegionInfo {}
-
/// Connect a PCI driver's private data to its VFIO registration.
///
/// # Safety
@@ -399,20 +365,22 @@ pub fn region_end(&self) -> Result<u64> {
/// - Callback contexts are only borrowed for the corresponding callback and thread.
/// - `core_device` was initialized by `vfio_pci_core_init_dev()`.
/// - The VFIO device refcount owns the allocation lifetime.
-/// - `reg_data` is dangling without an owner or points to a valid
-/// `<T::RegistrationData as ForLt>::Of<'_>` owned by the enclosing [`Registration`].
-/// - `open_data` is null outside a successful open, otherwise it owns a pinned
-/// `KBox<T::OpenData<'_>>` until last close. Only open/close mutate it; VFIO
-/// prevents I/O callbacks from racing those transitions. The data is dropped
-/// before PCI core close and before registration data is freed.
+/// - `data` follows the VFIO registration and first-open/last-close lifetimes.
+/// Open data is destroyed before PCI core close, and registration data is
+/// released after PCI core unregistration has drained callbacks.
#[repr(C)]
pub struct Device<T: Operations, Ctx: DeviceContext = Normal> {
core_device: Opaque<bindings::vfio_pci_core_device>,
- reg_data: UnsafeCell<NonNull<<T::RegistrationData as ForLt>::Of<'static>>>,
- open_data: UnsafeCell<*mut T::OpenData<'static>>,
+ data: CallbackData<T::RegistrationData, OpenDataFamily<T>>,
_context: PhantomData<(Ctx, NotThreadSafe)>,
}
+struct OpenDataFamily<T>(PhantomData<T>);
+
+impl<T: Operations> ForLt for OpenDataFamily<T> {
+ type Of<'a> = T::OpenData<'a>;
+}
+
impl<T: Operations> Device<T> {
/// Allocate a VFIO PCI device for subsequent registration.
pub fn new(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self>> {
@@ -426,6 +394,7 @@ pub fn new_passthrough(pdev: &pci::Device<device::Core<'_>>) -> Result<ARef<Self
fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<ARef<Self>> {
const_assert!(core::mem::offset_of!(Self, core_device) == 0);
+ const_assert!(core::mem::offset_of!(bindings::vfio_pci_core_device, vdev) == 0);
let size = Kmalloc::aligned_layout(Layout::new::<Self>()).size();
// SAFETY: The bound PCI device and static ops are valid. The allocation
@@ -437,8 +406,7 @@ fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<A
// SAFETY: Initialise the Rust field in the newly allocated device.
unsafe {
- (&raw mut (*this.as_ptr()).reg_data).write(UnsafeCell::new(NonNull::dangling()));
- (&raw mut (*this.as_ptr()).open_data).write(UnsafeCell::new(core::ptr::null_mut()));
+ (&raw mut (*this.as_ptr()).data).write(CallbackData::new());
(&raw mut (*this.as_ptr())._context).write(PhantomData);
}
@@ -451,50 +419,6 @@ fn allocate(pdev: &pci::Device<device::Core<'_>>, passthrough: bool) -> Result<A
Ok(unsafe { ARef::from_raw(this) })
}
- /// Access the registration data through a closure with an HRTB lifetime.
- ///
- /// The closure's `for<'a>` bound prevents the caller from smuggling
- /// references with a concrete short lifetime out of the closure.
- ///
- /// # Safety
- ///
- /// Registration data must be published and remain valid through this call.
- unsafe fn registration_data_with<R>(
- &self,
- f: impl for<'a> FnOnce(&'a <T::RegistrationData as ForLt>::Of<'a>) -> R,
- ) -> R {
- // SAFETY: Registration keeps the allocation and its borrowed resources
- // alive until unregistration completes. Covariance permits shortening
- // the erased binding lifetime to this callback's borrow.
- let reg_data: &<T::RegistrationData as ForLt>::Of<'_> = unsafe {
- (*self.reg_data.get())
- .cast::<<T::RegistrationData as ForLt>::Of<'_>>()
- .as_ref()
- };
- f(reg_data)
- }
-
- /// # Safety
- ///
- /// The caller must be an I/O callback while VFIO keeps the device open.
- unsafe fn callback_data_with<R>(
- &self,
- f: impl for<'borrow, 'data> FnOnce(
- &'borrow <T::RegistrationData as ForLt>::Of<'data>,
- Pin<&'borrow T::OpenData<'data>>,
- ) -> R,
- ) -> R {
- // SAFETY: VFIO excludes open/close and unregister while this callback runs.
- unsafe {
- self.registration_data_with(|reg_data| {
- let ptr = (*self.open_data.get()).cast::<T::OpenData<'_>>();
- // The data lifetime stays independent of the callback borrow,
- // so borrowed OpenData fields cannot be stored back into OpenData.
- f(reg_data, Pin::new_unchecked(&*ptr))
- })
- }
- }
-
/// # Safety
/// The returned view must only be used during the callback represented by `Ctx`.
unsafe fn with_context<Ctx: DeviceContext>(&self) -> &Device<T, Ctx> {
@@ -522,7 +446,15 @@ fn core_device(&self) -> *mut bindings::vfio_pci_core_device {
}
fn vfio_device(&self) -> *mut bindings::vfio_device {
- self.core_device.get().cast()
+ self.as_ref().as_raw()
+ }
+}
+
+impl<T: Operations, Ctx: DeviceContext> AsRef<super::Device<Ctx>> for Device<T, Ctx> {
+ fn as_ref(&self) -> &super::Device<Ctx> {
+ // SAFETY: The PCI core device embeds an initialized vfio_device. The borrow
+ // covers the same object and preserves the callback context and its lifetime.
+ unsafe { super::Device::from_raw(&raw mut (*self.core_device()).vdev) }
}
}
@@ -626,13 +558,14 @@ pub fn core_get_region_info(
// SAFETY: The embedded device reference count owns the VFIO allocation.
unsafe impl<T: Operations> AlwaysRefCounted for Device<T> {
fn inc_ref(&self) {
- // SAFETY: `self` holds a live VFIO device reference.
- unsafe { bindings::get_device(&raw mut (*self.vfio_device()).device) };
+ self.as_ref().inc_ref();
}
unsafe fn dec_ref(obj: NonNull<Self>) {
- // SAFETY: The caller owns the reference being released.
- unsafe { bindings::put_device(&raw mut (*obj.as_ref().vfio_device()).device) };
+ // SAFETY: The caller owns a reference to this live PCI wrapper.
+ let dev = unsafe { obj.as_ref() }.as_ref();
+ // SAFETY: The embedded VFIO device owns the same allocation and reference.
+ unsafe { super::Device::dec_ref(NonNull::from(dev)) };
}
}
@@ -648,9 +581,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
///
/// `vdev` must belong to a registered `Device<T>` in VFIO's first-open context.
/// VFIO must exclude other opens, closes and I/O callbacks for this call.
-unsafe extern "C" fn open_device_cb<T: Operations>(
- vdev: *mut bindings::vfio_device,
-) -> core::ffi::c_int {
+unsafe extern "C" fn open_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) -> c_int {
// SAFETY: `vdev` is valid; set by vfio_alloc_device.
let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
@@ -662,18 +593,9 @@ unsafe impl<T: Operations> Sync for Device<T> {}
// SAFETY: Registration data is initialized before the VFIO device is published.
let result = unsafe {
- dev.registration_data_with(|rd| {
- // Allocate before calling the driver so allocation failure cannot follow open.
- let data = KBox::<T::OpenData<'_>>::new_uninit(GFP_KERNEL)?;
- let data = data.write_pin_init(T::open_device(dev.with_context::<Open>(), rd))?;
-
- // SAFETY: Only the owning pointer is moved; the allocation remains pinned.
- let raw =
- KBox::into_raw(Pin::into_inner_unchecked(data)).cast::<T::OpenData<'static>>();
- // SAFETY: First open is exclusive with I/O callbacks and last close.
- // Lifetimes do not affect layout; the device owns the allocation until close.
- *dev.open_data.get() = raw;
- Ok::<(), Error>(())
+ dev.data.registration_data_with(|rd| {
+ dev.data
+ .open(|| T::open_device(dev.with_context::<Open>(), rd))
})
};
match result {
@@ -698,12 +620,9 @@ unsafe impl<T: Operations> Sync for Device<T> {}
unsafe extern "C" fn close_device_cb<T: Operations>(vdev: *mut bindings::vfio_device) {
// SAFETY: `vdev` is valid.
let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
- // SAFETY: Last close is exclusive with I/O callbacks and the next first open.
- let raw = unsafe { core::mem::replace(&mut *dev.open_data.get(), core::ptr::null_mut()) };
- // Run the driver's destructor while registration data and PCI resources are valid.
- // SAFETY: Successful open transferred this allocation with `KBox::into_raw`.
- // Last close takes ownership exactly once and drops it without moving the pointee.
- unsafe { drop(KBox::from_raw(raw)) };
+ // SAFETY: Last close drains I/O callbacks and excludes the next first open.
+ // Registration data and PCI resources remain valid while open data is destroyed.
+ unsafe { dev.data.close() };
// SAFETY: Matches the enable in open_device_cb.
unsafe { bindings::vfio_pci_core_close_device(vdev) };
}
@@ -721,7 +640,8 @@ unsafe impl<T: Operations> Sync for Device<T> {}
let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
// SAFETY: VFIO invokes this callback for an open device with valid arguments.
match unsafe {
- dev.callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
+ dev.data
+ .callback_data_with(|rd, od| T::ioctl(dev.with_context::<Ioctl>(), rd, od, cmd, arg))
} {
Ok(v) => v,
Err(e) => e.to_errno() as isize,
@@ -752,7 +672,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
};
// SAFETY: VFIO invokes this callback for an open device with valid arguments.
match unsafe {
- dev.callback_data_with(|rd, od| {
+ dev.data.callback_data_with(|rd, od| {
T::read(dev.with_context::<Read>(), rd, od, &mut ubuf, &mut pos)
})
} {
@@ -771,7 +691,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
vdev: *mut bindings::vfio_device,
info: *mut bindings::vfio_region_info,
caps: *mut bindings::vfio_info_cap,
-) -> core::ffi::c_int {
+) -> c_int {
// SAFETY: `vdev`, `info`, and `caps` are valid kernel pointers provided
// by the VFIO core.
let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
@@ -783,7 +703,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
};
// SAFETY: VFIO invokes this callback for an open device with valid arguments.
match unsafe {
- dev.callback_data_with(|rd, od| {
+ dev.data.callback_data_with(|rd, od| {
T::get_region_info(dev.with_context::<GetRegionInfo>(), rd, od, info, &mut caps)
})
} {
@@ -815,7 +735,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
};
// SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
match unsafe {
- dev.callback_data_with(|rd, od| {
+ dev.data.callback_data_with(|rd, od| {
T::write(dev.with_context::<Write>(), rd, od, &mut ubuf, &mut pos)
})
} {
@@ -832,7 +752,7 @@ unsafe impl<T: Operations> Sync for Device<T> {}
unsafe extern "C" fn mmap_cb<T: Operations>(
vdev: *mut bindings::vfio_device,
vma: *mut bindings::vm_area_struct,
-) -> core::ffi::c_int {
+) -> c_int {
// SAFETY: VFIO supplies an open device for this mmap callback.
let dev = unsafe { Device::<T>::from_vfio_device(vdev) };
let mut mapping = Mapping {
@@ -842,7 +762,8 @@ unsafe impl<T: Operations> Sync for Device<T> {}
};
// SAFETY: This callback runs while VFIO keeps both callback data allocations alive.
match unsafe {
- dev.callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
+ dev.data
+ .callback_data_with(|rd, od| T::mmap(dev.with_context::<Mmap>(), rd, od, &mut mapping))
} {
Ok(()) => 0,
Err(e) => e.to_errno(),
@@ -878,7 +799,11 @@ unsafe fn registration<'a>(
return;
};
// SAFETY: The registration owns callback data throughout bind and unbind.
- unsafe { reg.dev.registration_data_with(D::Operations::reset_prepare) };
+ unsafe {
+ reg.dev
+ .data
+ .registration_data_with(D::Operations::reset_prepare)
+ };
}
unsafe extern "C" fn reset_done(pdev: *mut bindings::pci_dev) {
@@ -887,7 +812,11 @@ unsafe fn registration<'a>(
return;
};
// SAFETY: The registration owns callback data throughout bind and unbind.
- let result = unsafe { reg.dev.registration_data_with(D::Operations::reset_done) };
+ let result = unsafe {
+ reg.dev
+ .data
+ .registration_data_with(D::Operations::reset_done)
+ };
if let Err(error) = result {
// SAFETY: The PCI callback keeps its embedded device alive.
let dev = unsafe { device::Device::<device::Normal>::from_raw(&raw mut (*pdev).dev) };
@@ -1062,7 +991,7 @@ pub unsafe fn new(
NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
// SAFETY: No concurrent registration or callbacks; publish before registering.
- unsafe { *dev.reg_data.get() = ptr };
+ unsafe { *dev.data.reg_data.get() = ptr };
Ok(Self {
dev: dev.into(),
@@ -1104,6 +1033,6 @@ impl<T: Operations> Drop for Registration<'_, T> {
fn drop(&mut self) {
// SAFETY: The adapter unregisters before removing private data. Failed or unattempted
// registration also leaves no callbacks behind.
- unsafe { *self.dev.reg_data.get() = NonNull::dangling() };
+ unsafe { *self.dev.data.reg_data.get() = NonNull::dangling() };
}
}
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 6/7] gpu: nova-core: publish typed SR-IOV PF APIs
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
` (4 preceding siblings ...)
2026-09-30 10:57 ` [PATCH 5/7] rust: vfio: separate common device handling from PCI Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
2026-09-30 10:57 ` [PATCH 7/7] vfio/nvidia-vgpu: add the Rust VFIO variant driver Zhi Wang
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
Register NovaCoreVfApi as typed SR-IOV PF data once the GPU is initialized.
Enable and disable VFs through sriov_configure(). The Rust PCI adapter
removes VFs before dropping PF driver data. Declare the registration
before the GPU so its borrowed services remain valid through VF teardown.
Expose a VF-side handle that borrows these services for each operation.
Return an owned instance on open and close it on drop while the VF
remains bound.
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
drivers/gpu/nova-core/driver.rs | 47 +++++++++++++++
drivers/gpu/nova-core/nova_core.rs | 2 +
drivers/gpu/nova-core/vgpu/vgpu_api.rs | 82 +++++++++++++++++++++++++-
3 files changed, 130 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index dc6febeee656..99beab189879 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -35,6 +35,12 @@
}, //
};
+#[cfg(CONFIG_PCI_IOV)]
+use kernel::types::ForLt;
+
+#[cfg(CONFIG_PCI_IOV)]
+use crate::vgpu::vgpu_api::NovaCoreVfApi;
+
/// Counter for generating unique auxiliary device IDs.
static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
@@ -42,6 +48,10 @@
pub(crate) struct NovaCore<'bound> {
/// Firmware-control registration.
_fwctl: fwctl::Registration<'bound, NovaCoreFwCtl>,
+ #[cfg(CONFIG_PCI_IOV)]
+ #[allow(clippy::type_complexity)]
+ #[pin]
+ _vf_registration: pci::VfRegistration<'bound, ForLt!(NovaCoreVfApi<'_>)>,
#[pin]
pub(crate) gpu: Gpu<'bound>,
bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -114,6 +124,11 @@ fn probe<'bound>(
pin_init::pin_init_scope(move || {
dev_dbg!(pdev, "Probe Nova Core GPU driver.\n");
+ #[cfg(CONFIG_PCI_IOV)]
+ if pdev.is_virtfn() {
+ return Err(ENODEV);
+ }
+
pdev.enable_device_mem()?;
pdev.set_master();
@@ -144,6 +159,18 @@ fn probe<'bound>(
// Run optional GPU selftests.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
_: { gpu.run_selftests(pdev) },
+ #[cfg(CONFIG_PCI_IOV)]
+ _vf_registration <- {
+ // SAFETY: `gpu` is initialized at its pinned address and
+ // outlives the registration and its borrowed API data.
+ let gpu = unsafe { &(*this.as_ptr()).gpu };
+ let api = NovaCoreVfApi::new(gpu, pdev);
+ // SAFETY: Probe has exclusive access to the registration
+ // slot and no VFs are enabled before successful probe. The
+ // registration keeps PF services available until VF removal
+ // completes and is dropped before its borrowed GPU resources.
+ unsafe { pci::VfRegistration::new(pdev, Ok(api)) }
+ },
_reg: auxiliary::Registration::new(
pdev.as_ref(),
c"nova-drm",
@@ -176,4 +203,24 @@ fn probe<'bound>(
}))
})
}
+
+ #[cfg(CONFIG_PCI_IOV)]
+ fn sriov_enable<'bound, 'callback>(
+ _dev: &'bound pci::Device<Core<'_>>,
+ _this: Pin<&Self::Data<'bound>>,
+ token: pci::SriovEnable<'callback>,
+ ) -> Result<pci::SriovEnabled<'callback>> {
+ let num_vfs = token.num_vfs();
+ token.enable(num_vfs)
+ }
+
+ #[cfg(CONFIG_PCI_IOV)]
+ fn sriov_disable<'bound>(
+ _dev: &'bound pci::Device<Core<'_>>,
+ _this: Pin<&Self::Data<'bound>>,
+ token: pci::SriovDisable<'_>,
+ ) -> Result {
+ token.disable();
+ Ok(())
+ }
}
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 619f32c79415..ee511bdf3544 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -33,6 +33,8 @@
#[cfg(CONFIG_PCI_IOV)]
pub use vgpu::vgpu_api::{
NovaCoreVfApi,
+ NovaCoreVfApiHandle,
+ VgpuInstance,
VgpuTypeInfo, //
};
diff --git a/drivers/gpu/nova-core/vgpu/vgpu_api.rs b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
index f9f3e527bdc0..a440108d2c26 100644
--- a/drivers/gpu/nova-core/vgpu/vgpu_api.rs
+++ b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
@@ -11,6 +11,9 @@
prelude::*, //
};
+#[cfg(CONFIG_PCI_IOV)]
+use kernel::types::ForLt;
+
use crate::{
driver::Bar0,
gpu::Gpu,
@@ -61,7 +64,6 @@ pub struct NovaCoreVfApi<'gpu> {
}
impl<'gpu> NovaCoreVfApi<'gpu> {
- #[expect(dead_code)]
pub(crate) fn new(gpu: &'gpu Gpu<'gpu>, pdev: &'gpu pci::Device<device::Bound>) -> Self {
Self {
pdev,
@@ -84,6 +86,20 @@ fn gfid(&self, gfid: u32) -> Result<Gfid> {
}
impl NovaCoreVfApi<'_> {
+ /// Obtains the enabled PF services for a bound VF.
+ #[cfg(CONFIG_PCI_IOV)]
+ pub fn handle(vf: &pci::Device<device::Bound>) -> Result<NovaCoreVfApiHandle<'_>> {
+ let handle = NovaCoreVfApiHandle { vf };
+ handle.with(|api| {
+ if api.is_available() {
+ Ok(())
+ } else {
+ Err(ENODEV)
+ }
+ })?;
+ Ok(handle)
+ }
+
/// Creates and boots an instance for a one-based VF ID.
///
/// `sbdf` encodes the VF address as `(segment << 16) | (bus << 8) | devfn`.
@@ -156,3 +172,67 @@ pub fn reset_instance(&self, gfid: u32) -> Result {
Ok(())
}
}
+
+/// Access to PF services for the lifetime of a VF driver binding.
+///
+/// The PF's registration keeps these services available until VF removal
+/// completes and is dropped before the GPU resources they borrow.
+#[cfg(CONFIG_PCI_IOV)]
+pub struct NovaCoreVfApiHandle<'vf> {
+ vf: &'vf pci::Device<device::Bound>,
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl<'vf> NovaCoreVfApiHandle<'vf> {
+ /// Borrows the typed PF services for a single operation.
+ fn with<R>(
+ &self,
+ f: impl for<'borrow, 'data> FnOnce(Pin<&'borrow NovaCoreVfApi<'data>>) -> Result<R>,
+ ) -> Result<R> {
+ self.vf
+ .vf_registration_data_with::<ForLt!(NovaCoreVfApi<'_>), _>(f)?
+ }
+
+ /// Creates and boots an instance that closes when dropped.
+ ///
+ /// The arguments have the same meaning as in [`NovaCoreVfApi::open_instance`].
+ pub fn open(&self, gfid: u32, dbdf: u32, vm_pid: u32) -> Result<VgpuInstance<'vf>> {
+ let type_info = self.with(|api| api.open_instance(gfid, dbdf, vm_pid))?;
+ Ok(VgpuInstance {
+ api: Self { vf: self.vf },
+ gfid,
+ type_info,
+ })
+ }
+
+ /// Resets an active instance and scrubs its guest VRAM.
+ pub fn reset(&self, gfid: u32) -> Result {
+ self.with(|api| api.reset_instance(gfid))
+ }
+}
+
+/// An active instance whose teardown runs while its VF driver remains bound.
+///
+/// Dropping this guard may sleep while firmware teardown completes.
+#[cfg(CONFIG_PCI_IOV)]
+pub struct VgpuInstance<'vf> {
+ api: NovaCoreVfApiHandle<'vf>,
+ gfid: u32,
+ type_info: VgpuTypeInfo,
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl VgpuInstance<'_> {
+ /// Returns the assigned PCI IDs and BAR1 aperture size.
+ #[inline]
+ pub fn type_info(&self) -> &VgpuTypeInfo {
+ &self.type_info
+ }
+}
+
+#[cfg(CONFIG_PCI_IOV)]
+impl Drop for VgpuInstance<'_> {
+ fn drop(&mut self) {
+ let _ = self.api.with(|api| api.close_instance(self.gfid));
+ }
+}
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 7/7] vfio/nvidia-vgpu: add the Rust VFIO variant driver
2026-09-30 10:57 [PATCH 0/7] Rust VFIO PCI support and NVIDIA vGPU driver Zhi Wang
` (5 preceding siblings ...)
2026-09-30 10:57 ` [PATCH 6/7] gpu: nova-core: publish typed SR-IOV PF APIs Zhi Wang
@ 2026-09-30 10:57 ` Zhi Wang
6 siblings, 0 replies; 9+ messages in thread
From: Zhi Wang @ 2026-09-30 10:57 UTC (permalink / raw)
To: dakr, acourbot, kvm, rust-for-linux, nova-gpu, linux-pci
Cc: alex, jgg, yishaih, skolothumtho, kevin.tian, airlied, simona,
ojeda, alex.gaynor, boqun.feng, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, jhubbard, ecourtney, cjia,
smitra, kjaju, alkumar, ankita, aniketa, kwankhede, targupta,
linux-kernel, zhiwang, Zhi Wang, Boqun Feng, Daniel Almeida,
Tamir Duberstein, Onur Özkan, Jason Gunthorpe,
Bjorn Helgaas, Krzysztof Wilczyński, dri-devel,
Peter Colberg, Simon Song
From: Danilo Krummrich <dakr@kernel.org>
NVIDIA vGPU VFs need nova-core to manage their instances when assigned
to userspace through VFIO.
Add a Rust VFIO PCI variant driver that uses the typed SR-IOV PF APIs
to create an instance on first open and close it on last close. Expose
the assigned device and subsystem IDs, enforce the assigned BAR1
aperture, and coordinate firmware reset with PCI reset.
Match NVIDIA devices through a VFIO override-only entry. Devices
without an available Nova vGPU interface use ordinary PCI passthrough.
Use the VFIO PCI adapter to keep callback data available throughout
registration and removal. Extend the GPU build rules to make nova-core
crate metadata available to the VFIO driver and export the Rust symbols
it references.
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
drivers/gpu/Makefile | 15 +-
drivers/vfio/pci/Kconfig | 2 +
drivers/vfio/pci/Makefile | 2 +
drivers/vfio/pci/nvidia-vgpu/Kconfig | 20 ++
drivers/vfio/pci/nvidia-vgpu/Makefile | 3 +
drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs | 366 ++++++++++++++++++++
6 files changed, 405 insertions(+), 3 deletions(-)
create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
create mode 100644 drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
diff --git a/drivers/gpu/Makefile b/drivers/gpu/Makefile
index e372fc02139f..1b8a907e716b 100644
--- a/drivers/gpu/Makefile
+++ b/drivers/gpu/Makefile
@@ -19,8 +19,9 @@ nova-core-y := nova-core/nova_core.o nova-core/nova_core_exports.o
obj-$(CONFIG_DRM_NOVA) += nova-drm.o
nova-drm-y := drm/nova/nova.o
-# Export Rust symbols from nova-core only if nova-drm actually references them.
-nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o)
+# Export Rust symbols from nova-core only if dependent modules reference them.
+nova-core-export-deps := $(if $(CONFIG_DRM_NOVA),$(obj)/drm/nova/nova.o) \
+ $(if $(CONFIG_NVIDIA_VGPU_VFIO_PCI),$(obj)/../vfio/pci/nvidia-vgpu/nvidia_vgpu.o)
rust_needed_exports = \
{ $(if $(strip $(2)),$(NM) -u $(2);,) echo "__DEFINED_RUST_SYMBOLS__"; \
@@ -57,10 +58,18 @@ $(obj)/nova-core/nova_core_exports.o: private cmd_gensymtypes_c = \
$(obj)/nova-core/nova_core.o
endif
-# Output nova-core's crate metadata for use by nova-drm at compile time.
+# Output nova-core's crate metadata for its Rust consumers at compile time.
RUSTFLAGS_nova-core/nova_core.o += \
--emit=metadata=$(objtree)/$(obj)/nova-core/libnova_core.rmeta
# Allow nova-drm to import nova-core's types.
$(obj)/drm/nova/nova.o: $(obj)/nova-core/nova_core.o
RUSTFLAGS_drm/nova/nova.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
+
+# Build the NVIDIA VFIO driver here so metadata and generated exports share
+# the same dependency graph as nova-drm.
+obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu-vfio-pci.o
+nvidia-vgpu-vfio-pci-y := ../vfio/pci/nvidia-vgpu/nvidia_vgpu.o
+
+$(obj)/../vfio/pci/nvidia-vgpu/nvidia_vgpu.o: $(obj)/nova-core/nova_core.o
+RUSTFLAGS_../vfio/pci/nvidia-vgpu/nvidia_vgpu.o := -L $(objtree)/$(obj)/nova-core --extern nova_core
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..b48d8d1af42a 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -74,4 +74,6 @@ source "drivers/vfio/pci/qat/Kconfig"
source "drivers/vfio/pci/xe/Kconfig"
+source "drivers/vfio/pci/nvidia-vgpu/Kconfig"
+
endmenu
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..58c88304a185 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -24,3 +24,5 @@ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
obj-$(CONFIG_QAT_VFIO_PCI) += qat/
obj-$(CONFIG_XE_VFIO_PCI) += xe/
+
+# nvidia-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
diff --git a/drivers/vfio/pci/nvidia-vgpu/Kconfig b/drivers/vfio/pci/nvidia-vgpu/Kconfig
new file mode 100644
index 000000000000..1363f84163e2
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Kconfig
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVIDIA_VGPU_VFIO_PCI
+ tristate "VFIO support for the NVIDIA vGPU"
+ depends on NOVA_CORE && PCI_IOV && RUST
+ select VFIO_PCI_CORE
+ help
+ This option enables VFIO (Virtual Function I/O) support for
+ NVIDIA virtual GPUs (vGPU). It allows the assignment of a virtual
+ GPU instance to userspace applications via VFIO, typically used
+ with hypervisors such as KVM and device emulators like QEMU.
+
+ Devices without an available Nova vGPU interface use ordinary PCI
+ passthrough. Both paths use the VFIO PCI core defaults; vfio-pci
+ module options do not apply.
+
+ The NVIDIA vGPU allows a physical GPU to be partitioned into
+ multiple virtual GPUs, each of which can be passed to a virtual
+ machine as a PCI device using the standard VFIO infrastructure.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile
new file mode 100644
index 000000000000..bce0133562f3
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+# nvidia-vgpu is built from drivers/gpu/Makefile for nova-core crate linkage.
+# nvidia_vgpu.o (rust-analyzer marker - DO NOT REMOVE).
diff --git a/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs b/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
new file mode 100644
index 000000000000..e3f1cd0abf5f
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/nvidia_vgpu.rs
@@ -0,0 +1,366 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! NVIDIA vGPU VFIO variant driver.
+//!
+//! Nova VFs own an instance between the first VFIO open and last close. Other
+//! matching devices use ordinary PCI passthrough.
+
+use kernel::{
+ bindings,
+ device::{
+ Bound,
+ Core, //
+ },
+ io::resource::Flags,
+ pci,
+ prelude::*,
+ sync::{
+ CondVar,
+ Mutex,
+ MutexGuard, //
+ },
+ types::CovariantForLt,
+ vfio::{
+ self,
+ pci::{
+ GetRegionInfo,
+ Ioctl,
+ Mapping,
+ Mmap,
+ Open,
+ Position,
+ Read,
+ Write, //
+ }, //
+ }, //
+};
+use nova_core::{
+ NovaCoreVfApi,
+ NovaCoreVfApiHandle,
+ VgpuInstance, //
+};
+
+struct NvidiaVgpuOps;
+
+struct VgpuRegistration<'a> {
+ api: NovaCoreVfApiHandle<'a>,
+ gfid: u32,
+}
+
+#[derive(Default)]
+struct InstanceState {
+ active: bool,
+ resetting: bool,
+ reset_error: Option<Error>,
+}
+
+#[pin_data]
+struct NvidiaVgpuRegData<'a> {
+ pdev: &'a pci::Device<Bound>,
+ vgpu: Option<VgpuRegistration<'a>>,
+ fb_bar: u32,
+ #[pin]
+ state: Mutex<InstanceState>,
+ #[pin]
+ reset_done: CondVar,
+}
+
+impl NvidiaVgpuRegData<'_> {
+ fn lock_instance(&self) -> MutexGuard<'_, InstanceState> {
+ let mut state = self.state.lock();
+ while state.resetting {
+ self.reset_done.wait(&mut state);
+ }
+ state
+ }
+}
+
+struct NvidiaVgpuOpenData<'a> {
+ registration: &'a NvidiaVgpuRegData<'a>,
+ instance: Option<VgpuInstance<'a>>,
+}
+
+impl<'a> NvidiaVgpuOpenData<'a> {
+ fn new(
+ dev: &vfio::pci::Device<NvidiaVgpuOps, Open>,
+ rd: &'a NvidiaVgpuRegData<'a>,
+ ) -> Result<Self> {
+ let mut state = rd.lock_instance();
+ let instance = match &rd.vgpu {
+ Some(vgpu) => {
+ // The firmware DBDF format has a 16-bit PCI segment field.
+ let segment = u16::try_from(rd.pdev.domain_nr()).map_err(|_| EOVERFLOW)?;
+ let dbdf = (u32::from(segment) << 16) | u32::from(rd.pdev.dev_id());
+ let vm_pid = kernel::current!().tgid().try_into()?;
+ let instance = vgpu.api.open(vgpu.gfid, dbdf, vm_pid)?;
+ let info = instance.type_info();
+ dev.set_device_id(info.pci_dev_id as u16);
+ state.active = true;
+ state.reset_error = None;
+ Some(instance)
+ }
+ None => None,
+ };
+ Ok(Self {
+ registration: rd,
+ instance,
+ })
+ }
+
+ fn bar1_size(&self) -> Result<Option<u64>> {
+ let Some(instance) = &self.instance else {
+ return Ok(None);
+ };
+ let physical = self
+ .registration
+ .pdev
+ .resource_len(self.registration.fb_bar)?;
+ let size = instance
+ .type_info()
+ .bar1_length
+ .checked_mul(1 << 20)
+ .ok_or(EOVERFLOW)?;
+ Ok(Some(if size == 0 {
+ physical
+ } else {
+ size.min(physical)
+ }))
+ }
+
+ fn limit_bar1(&self, buf: &mut vfio::UserBuf, position: &Position<'_>) -> Result {
+ if buf.is_empty() || position.region_index() != self.registration.fb_bar {
+ return Ok(());
+ }
+ if let Some(size) = self.bar1_size()? {
+ let offset = position.region_offset();
+ if offset >= size {
+ return Err(EINVAL);
+ }
+ if size - offset < buf.len() as u64 {
+ buf.truncate((size - offset) as usize);
+ }
+ }
+ Ok(())
+ }
+}
+
+impl Drop for NvidiaVgpuOpenData<'_> {
+ fn drop(&mut self) {
+ let mut state = self.registration.lock_instance();
+ state.active = false;
+ drop(self.instance.take());
+ }
+}
+
+impl vfio::pci::Operations for NvidiaVgpuOps {
+ const NAME: &'static CStr = c"nvidia-vgpu-vfio-pci";
+ type RegistrationData = CovariantForLt!(NvidiaVgpuRegData<'_>);
+ type OpenData<'a> = NvidiaVgpuOpenData<'a>;
+
+ fn open_device<'a>(
+ dev: &'a vfio::pci::Device<Self, Open>,
+ rd: &'a NvidiaVgpuRegData<'a>,
+ ) -> impl PinInit<Self::OpenData<'a>, Error> + 'a {
+ NvidiaVgpuOpenData::new(dev, rd)
+ }
+
+ fn ioctl<'a>(
+ dev: &vfio::pci::Device<Self, Ioctl>,
+ rd: &NvidiaVgpuRegData<'a>,
+ _open_data: Pin<&Self::OpenData<'a>>,
+ cmd: u32,
+ arg: usize,
+ ) -> Result<isize> {
+ let reset = cmd == vfio::DEVICE_RESET || cmd == vfio::pci::DEVICE_PCI_HOT_RESET;
+ if reset {
+ if let Some(error) = rd.state.lock().reset_error {
+ return Err(error);
+ }
+ }
+ let result = dev.core_ioctl(cmd, arg)?;
+ if reset {
+ if let Some(error) = rd.state.lock().reset_error {
+ return Err(error);
+ }
+ }
+ Ok(result)
+ }
+
+ fn read<'a>(
+ dev: &vfio::pci::Device<Self, Read>,
+ _rd: &NvidiaVgpuRegData<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ buf: &mut vfio::UserBuf,
+ position: &mut Position<'_>,
+ ) -> Result<isize> {
+ if let Some(instance) = &open_data.instance {
+ if position.region_index() == vfio::pci::CONFIG_REGION_INDEX {
+ let offset = position.region_offset();
+ return position.with_temporary(|next| {
+ let count = dev.core_read(buf, next)?;
+ buf.write_overlapping(
+ offset,
+ count.try_into()?,
+ u64::from(bindings::PCI_SUBSYSTEM_ID),
+ &(instance.type_info().pci_subsys_id as u16).to_le_bytes(),
+ )?;
+ Ok(count)
+ });
+ }
+ }
+ open_data.limit_bar1(buf, position)?;
+ dev.core_read(buf, position)
+ }
+
+ fn write<'a>(
+ dev: &vfio::pci::Device<Self, Write>,
+ _rd: &NvidiaVgpuRegData<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ buf: &mut vfio::UserBuf,
+ position: &mut Position<'_>,
+ ) -> Result<isize> {
+ open_data.limit_bar1(buf, position)?;
+ dev.core_write(buf, position)
+ }
+
+ fn mmap<'a>(
+ dev: &vfio::pci::Device<Self, Mmap>,
+ rd: &NvidiaVgpuRegData<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ mapping: &mut Mapping<'_>,
+ ) -> Result {
+ if mapping.region_index() == rd.fb_bar {
+ if let Some(size) = open_data.bar1_size()? {
+ if mapping.region_end()? > size {
+ return Err(EINVAL);
+ }
+ }
+ }
+ dev.core_mmap(mapping)
+ }
+
+ fn get_region_info<'a>(
+ dev: &vfio::pci::Device<Self, GetRegionInfo>,
+ rd: &NvidiaVgpuRegData<'a>,
+ open_data: Pin<&Self::OpenData<'a>>,
+ info: &mut bindings::vfio_region_info,
+ caps: &mut vfio::InfoCap<'_>,
+ ) -> Result {
+ dev.core_get_region_info(info, caps)?;
+ if info.index == rd.fb_bar && info.size != 0 {
+ if let Some(size) = open_data.bar1_size()? {
+ info.size = info.size.min(size);
+ }
+ }
+ Ok(())
+ }
+
+ fn reset_prepare(rd: &NvidiaVgpuRegData<'_>) {
+ // PCI holds the device lock, and VFIO may hold its memory lock. Open and
+ // close release this mutex before entering the VFIO core.
+ let mut state = rd.state.lock();
+ state.resetting = true;
+ if state.active {
+ if let Some(vgpu) = &rd.vgpu {
+ if let Err(error) = vgpu.api.reset(vgpu.gfid) {
+ state.reset_error.get_or_insert(error);
+ }
+ }
+ }
+ }
+
+ fn reset_done(rd: &NvidiaVgpuRegData<'_>) -> Result {
+ let mut state = rd.state.lock();
+ let error = state.reset_error;
+ state.resetting = false;
+ rd.reset_done.notify_all();
+ error.map_or(Ok(()), Err)
+ }
+}
+
+struct NvidiaVgpuDriver;
+
+kernel::pci_device_table!(
+ PCI_TABLE,
+ <NvidiaVgpuDriver as pci::Driver>::IdInfo,
+ [(
+ pci::DeviceId::from_id_vfio_override(pci::Vendor::NVIDIA, bindings::PCI_ANY_ID as u32),
+ (),
+ ),]
+);
+
+#[pin_data]
+struct NvidiaVgpuData<'bound> {
+ reg: vfio::pci::Registration<'bound, NvidiaVgpuOps>,
+}
+
+// SAFETY: The selector only borrows this device's sole registration and cannot sleep. This driver
+// uses the VFIO adapter below and does not enable SR-IOV from probe or unbind.
+unsafe impl vfio::pci::Driver for NvidiaVgpuDriver {
+ type Operations = NvidiaVgpuOps;
+
+ fn registration<'a, 'bound>(
+ data: Pin<&'a Self::Data<'bound>>,
+ ) -> &'a vfio::pci::Registration<'bound, NvidiaVgpuOps> {
+ &data.get_ref().reg
+ }
+}
+
+#[vtable]
+impl pci::Driver for NvidiaVgpuDriver {
+ type IdInfo = ();
+ type Data<'bound> = NvidiaVgpuData<'bound>;
+ const ID_TABLE: pci::IdTable<Self::IdInfo> = &PCI_TABLE;
+ const DRIVER_MANAGED_DMA: bool = true;
+
+ fn probe<'bound>(
+ pdev: &'bound pci::Device<Core<'_>>,
+ _info: Option<&'bound Self::IdInfo>,
+ ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+ try_pin_init!(NvidiaVgpuData {
+ reg: {
+ let vgpu = match NovaCoreVfApi::handle(pdev) {
+ Ok(api) => Some(VgpuRegistration {
+ api,
+ gfid: pdev.vf_id()?.checked_add(1).ok_or(EOVERFLOW)?,
+ }),
+ Err(_) => None,
+ };
+ let dev = if vgpu.is_some() {
+ vfio::pci::Device::<NvidiaVgpuOps>::new(pdev)?
+ } else {
+ vfio::pci::Device::<NvidiaVgpuOps>::new_passthrough(pdev)?
+ };
+ let fb_bar = if pdev.resource_flags(0)?.contains(Flags::IORESOURCE_MEM_64) {
+ 2
+ } else {
+ 1
+ };
+ // SAFETY: The device was allocated for this probe and has not been
+ // registered. Private data owns it, and the VFIO adapter handles its registration.
+ unsafe {
+ vfio::pci::Registration::new(
+ pdev,
+ &dev,
+ try_pin_init!(NvidiaVgpuRegData {
+ pdev,
+ vgpu,
+ fb_bar,
+ state <- kernel::new_mutex!(InstanceState::default()),
+ reset_done <- kernel::new_condvar!(),
+ }),
+ )?
+ }
+ },
+ })
+ }
+}
+
+kernel::module_driver!(<T>, vfio::pci::Adapter<T>, {
+ type: NvidiaVgpuDriver,
+ name: "nvidia-vgpu-vfio-pci",
+ authors: ["NVIDIA"],
+ description: "NVIDIA vGPU VFIO variant driver",
+ license: "GPL v2",
+});
--
2.53.0
^ permalink raw reply [flat|nested] 9+ messages in thread