[PATCH 0/2] Nova vGPU lifecycle FFI and the NVIDIA VFIO driver

[PATCH 0/2] Nova vGPU lifecycle FFI and the NVIDIA VFIO driver

[Zhi Wang]

The NVIDIA vGPU VFIO driver is written in C and needs to invoke lifecycle
operations owned by the Rust nova-core PF driver. VFIO open creates and
boots a vGPU instance; reset resets it and scrubs guest VRAM, and close
shuts it down and releases its resources.

The discussion about implementing the VFIO driver in Rust or C is still
ongoing [2]. In the meantime, this series updates the patches to build on
"Add Rust PCI SR-IOV support" [3] and addresses Alex Williamson's review
comments [1, 2].

The first patch publishes these operations through NovaCoreVfApi in
Nova's typed SR-IOV registration. Managed SR-IOV keeps the operations and
their pinned context alive until the VF drivers have been removed.

The second patch adds the NVIDIA vGPU VFIO PCI variant. It uses the Nova
interface when the VF's PF provides a compatible one and supports
ordinary PCI passthrough otherwise. Configuration-space and BAR handling
stay within the variant; this series does not modify VFIO core.

Changes in response to Alex Williamson's review [1, 2]:

  - Restrict matching to NVIDIA 10de:2bb5 in the 3D display class (0302).
    PFs share this ID with VFs, so matched PFs and VFs without a compatible
    Nova interface use vfio-pci-core passthrough instead of being
    rejected by probe.

  - Store the assigned PCI device ID in vconfig after enabling the
    device, so configuration reads use the core's virtualized value.

  - Enforce the assigned BAR1 aperture for read, write and mmap, using
    the same limit reported by GET_REGION_INFO.

  - Use reset_prepare/reset_done callbacks to cover configuration-space
    FLR as well as reset ioctls. Serialize instance open and close with
    the reset interval. Preserve firmware reset errors for reset ioctls
    and signal the VFIO error IRQ when configured. These callbacks cannot
    abort the PCI reset.

  - Use SBDF naming and document the packed VF address. Remove the
    redundant VF check from open; probe selects the appropriate ops.

Subsystem ID reads remain in the variant's configuration handler because
the core reads that field from hardware. The C driver preserves the
existing VFIO drvdata contract and adds no VFIO-specific pointer to
struct pci_dev.

The passthrough path uses vfio-pci-core defaults. This driver has its own
enable_sriov parameter and does not inherit vfio-pci's other module
options; full module-option parity remains open.

[1] https://lore.kernel.org/nova-gpu/20260908210044.6514a092@shazbot.org/
[2] https://lore.kernel.org/nova-gpu/20260914121217.70fa0d93@shazbot.org/
[3] https://lore.kernel.org/all/20260915205659.76841-1-zhiw@nvidia.com/

Zhi Wang (2):
  gpu: nova-core: vgpu: export lifecycle operations to VFIO
  vfio/nvidia-vgpu: add the NVIDIA vGPU VFIO variant driver

 MAINTAINERS                               |   1 +
 drivers/gpu/nova-core/driver.rs           |  59 ++-
 drivers/gpu/nova-core/gpu.rs              |  58 ++-
 drivers/gpu/nova-core/vgpu.rs             |   7 +
 drivers/gpu/nova-core/vgpu/commands.rs    |  11 +-
 drivers/gpu/nova-core/vgpu/fw/commands.rs |   1 +
 drivers/gpu/nova-core/vgpu/instance.rs    | 153 ++++++-
 drivers/gpu/nova-core/vgpu/vgpu_api.rs    | 235 ++++++++++
 drivers/vfio/pci/Kconfig                  |   2 +
 drivers/vfio/pci/Makefile                 |   2 +
 drivers/vfio/pci/nvidia-vgpu/Kconfig      |  21 +
 drivers/vfio/pci/nvidia-vgpu/Makefile     |   2 +
 drivers/vfio/pci/nvidia-vgpu/main.c       | 509 ++++++++++++++++++++++
 include/drm/nvidia_vgpu.h                 |  51 +++
 rust/bindings/bindings_helper.h           |   1 +
 15 files changed, 1084 insertions(+), 29 deletions(-)
 create mode 100644 drivers/gpu/nova-core/vgpu/vgpu_api.rs
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/main.c
 create mode 100644 include/drm/nvidia_vgpu.h


base-commit: 4c627824f040e6bd4a10bff4b1c353cdfaf1fc15

[PATCH 1/2] gpu: nova-core: vgpu: export lifecycle operations to VFIO

[Zhi Wang]

VFIO open queries the VF's assigned vGPU type, allocates its resources,
and boots the GSP plugin. Close destroys the instance, and reset resets
the plugin and scrubs guest VRAM.

Expose these operations through NovaCoreVfApi, embedded in the PF's typed
SR-IOV registration. Its C operations table borrows the API until VF
removal. Keep instance locking, activation and teardown in VgpuManager.

Leave the C descriptor empty when vGPU mode is disabled so VF drivers can
select their ordinary PCI passthrough path.

Co-developed-by: Alok Kumar <alkumar@nvidia.com>
Signed-off-by: Alok Kumar <alkumar@nvidia.com>
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
 MAINTAINERS                               |   1 +
 drivers/gpu/nova-core/driver.rs           |  59 ++++--
 drivers/gpu/nova-core/gpu.rs              |  58 +++++-
 drivers/gpu/nova-core/vgpu.rs             |   7 +
 drivers/gpu/nova-core/vgpu/commands.rs    |  11 +-
 drivers/gpu/nova-core/vgpu/fw/commands.rs |   1 +
 drivers/gpu/nova-core/vgpu/instance.rs    | 153 +++++++++++++-
 drivers/gpu/nova-core/vgpu/vgpu_api.rs    | 235 ++++++++++++++++++++++
 include/drm/nvidia_vgpu.h                 |  51 +++++
 rust/bindings/bindings_helper.h           |   1 +
 10 files changed, 548 insertions(+), 29 deletions(-)
 create mode 100644 drivers/gpu/nova-core/vgpu/vgpu_api.rs
 create mode 100644 include/drm/nvidia_vgpu.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 922cfddcb2dc..92d5c0424452 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8383,6 +8383,7 @@ C:	irc://irc.oftc.net/nouveau
 T:	git https://gitlab.freedesktop.org/drm/rust/kernel.git drm-rust-next
 F:	Documentation/gpu/nova/
 F:	drivers/gpu/nova-core/
+F:	include/drm/nvidia_vgpu.h
 
 DRM DRIVER FOR NVIDIA GPUS [RUST]
 M:	Danilo Krummrich <dakr@kernel.org>
diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index 1373417386a2..67ac3b348f38 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -30,6 +30,15 @@
     }, //
 };
 
+#[cfg(CONFIG_PCI_IOV)]
+use kernel::types::ForLt;
+
+#[cfg(CONFIG_PCI_IOV)]
+use crate::vgpu::vgpu_api::{
+    NovaCoreVfAbi,
+    NovaCoreVfApi, //
+};
+
 /// Counter for generating unique auxiliary device IDs.
 static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
 
@@ -38,7 +47,7 @@ pub(crate) struct NovaCore<'bound> {
     #[cfg(CONFIG_PCI_IOV)]
     #[allow(clippy::type_complexity)]
     #[pin]
-    _vf_registration: pci::VfRegistration<'bound, CovariantForLt!(())>,
+    _vf_registration: pci::VfRegistration<'bound, ForLt!(NovaCoreVfApi<'_>)>,
     #[pin]
     pub(crate) gpu: Gpu<'bound>,
     bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -111,18 +120,15 @@ fn probe<'bound>(
         pin_init::pin_init_scope(move || {
             dev_dbg!(pdev, "Probe Nova Core GPU driver.\n");
 
-            Ok(try_pin_init!(NovaCore {
-                #[cfg(CONFIG_PCI_IOV)]
-                // SAFETY:
-                // - probe has exclusive access before SR-IOV can be enabled;
-                // - the registration is pinned in driver data and is its first field;
-                // - no other registration is created for this device; and
-                // - the PCI adapter uses managed SR-IOV.
-                _vf_registration <- unsafe { pci::VfRegistration::new(pdev, Ok(())) },
-                _: {
-                    pdev.enable_device_mem()?;
-                    pdev.set_master();
-                },
+            #[cfg(CONFIG_PCI_IOV)]
+            if pdev.is_virtfn() {
+                return Err(ENODEV);
+            }
+
+            pdev.enable_device_mem()?;
+            pdev.set_master();
+
+            Ok(try_pin_init!(&this in NovaCore {
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 bar1: {
                     let bar1_idx = bar1_resource_index(pdev)?;
@@ -149,6 +155,33 @@ fn probe<'bound>(
                 // Run optional GPU selftests.
                 #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
                 _: { gpu.run_selftests(pdev) },
+                #[cfg(CONFIG_PCI_IOV)]
+                _vf_registration <- {
+                    // SAFETY: `gpu` is initialized at its pinned address and
+                    // outlives the registration and its borrowed API data.
+                    let gpu = unsafe { &(*this.as_ptr()).gpu };
+                    let api = NovaCoreVfApi::new(gpu, pdev);
+                    let publish_ffi = api.is_available();
+                    let data = Ok(api);
+                    // SAFETY: Probe has exclusive access to the registration
+                    // slot and no VFs are enabled before successful probe. The
+                    // PCI adapter uses managed SR-IOV, and the pinned driver
+                    // data drops this registration before its borrowed GPU resources.
+                    // Each branch initializes the supplied pinned slot exactly
+                    // once and delegates error cleanup to its initializer.
+                    unsafe {
+                        pin_init::pin_init_from_closure(move |slot| {
+                            if publish_ffi {
+                                pin_init::raw_try_init(
+                                    slot,
+                                    pci::VfRegistration::new_ffi::<NovaCoreVfAbi, _>(pdev, data),
+                                )
+                            } else {
+                                pin_init::raw_try_init(slot, pci::VfRegistration::new(pdev, data))
+                            }
+                        })
+                    }
+                },
                 _reg: auxiliary::Registration::new(
                     pdev.as_ref(),
                     c"nova-drm",
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 289cbd171250..f28cd4611633 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -1,6 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0
 
-use core::ops::Range;
+use core::{
+    num::NonZero,
+    ops::Range, //
+};
 
 use kernel::{
     device,
@@ -8,6 +11,7 @@
     fmt,
     gpu::buddy::GpuBuddyParams,
     io::Io,
+    new_mutex,
     num::Bounded,
     pci,
     prelude::*,
@@ -16,6 +20,7 @@
         SizeConstants,
         SZ_4K, //
     },
+    sync::Mutex,
 };
 
 use crate::{
@@ -33,6 +38,7 @@
     fsp::Fsp,
     gsp::{
         self,
+        cmdq::Cmdq,
         Gsp,
         GspBootContext, //
     },
@@ -326,7 +332,8 @@ pub(crate) struct Gpu<'gpu> {
     ///
     /// Must be kept declared *before* `gsp_resources`, so that its components are dropped while
     /// the GSP is still operational.
-    mm: GpuMm<'gpu>,
+    #[pin]
+    mm: Mutex<GpuMm<'gpu>>,
     /// BAR1 user interface for CPU access to GPU virtual memory.
     #[pin]
     bar_user: BarUser<'gpu>,
@@ -380,6 +387,33 @@ fn drop(self: Pin<&mut Self>) {
 }
 
 impl<'gpu> Gpu<'gpu> {
+    pub(crate) fn cmdq(&self) -> &Cmdq<'gpu> {
+        &self.gsp_resources.gsp.cmdq
+    }
+
+    pub(crate) fn vgpu_manager(&self) -> Option<&VgpuManager<'gpu>> {
+        self.vgpu.as_ref().map(|vgpu| vgpu.as_ref().get_ref())
+    }
+
+    pub(crate) fn vgpu_total_vfs(&self) -> Option<NonZero<u16>> {
+        match self.gsp_resources.vgpu_state {
+            VgpuState::Disabled => None,
+            VgpuState::Enabled { total_vfs } => Some(total_vfs),
+        }
+    }
+
+    pub(crate) fn mm(&self) -> &Mutex<GpuMm<'gpu>> {
+        &self.mm
+    }
+
+    pub(crate) fn bar_user(&self) -> &BarUser<'gpu> {
+        &self.bar_user
+    }
+
+    pub(crate) fn bar0(&self) -> Bar0<'gpu> {
+        self.gsp_resources.bar
+    }
+
     pub(crate) fn new<'a>(
         pdev: &'gpu pci::Device<device::Core<'a>>,
         bar: Bar0<'gpu>,
@@ -509,7 +543,7 @@ pub(crate) fn new<'a>(
             },
 
             // Create GPU memory manager owning memory management resources.
-            mm: {
+            mm <- {
                 let info = &gsp_resources.boot_result.static_info;
                 let usable_vram = info.usable_fb_regions.first().ok_or(ENODEV)?;
                 let buddy_params = GpuBuddyParams {
@@ -518,12 +552,15 @@ pub(crate) fn new<'a>(
                     chunk_size: Alignment::new::<SZ_4K>(),
                 };
 
-                GpuMm::new(
-                    bar,
-                    gsp_resources.spec.chipset,
-                    buddy_params,
-                    VramAddress::from_raw(info.total_fb_end),
-                )?
+                new_mutex!(
+                    GpuMm::new(
+                        bar,
+                        gsp_resources.spec.chipset,
+                        buddy_params,
+                        VramAddress::from_raw(info.total_fb_end),
+                    )?,
+                    "nova-core::gpu-mm",
+                )
             },
 
             // Create BAR1 user interface for CPU access to GPU virtual memory.
@@ -548,10 +585,11 @@ pub(crate) fn run_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bou
         let this = self.project();
         let dev = pdev.as_ref();
         let regions = &this.gsp_resources.boot_result.static_info.usable_fb_regions;
+        let mut mm = this.mm.lock();
 
         if let Err(err) = crate::mm::selftest::run(
             dev,
-            this.mm,
+            &mut mm,
             regions,
             this.bar_user.as_ref().get_ref(),
             this.gsp_resources.boot_result.static_info.bar1_pde_base,
diff --git a/drivers/gpu/nova-core/vgpu.rs b/drivers/gpu/nova-core/vgpu.rs
index fdcf22fa4415..7075d4e32cae 100644
--- a/drivers/gpu/nova-core/vgpu.rs
+++ b/drivers/gpu/nova-core/vgpu.rs
@@ -31,6 +31,8 @@
 mod instance;
 mod log;
 mod scrubber;
+#[cfg_attr(not(CONFIG_PCI_IOV), expect(dead_code))]
+pub(crate) mod vgpu_api;
 mod vram;
 
 /// vGPU state detected during GPU construction.
@@ -142,6 +144,11 @@ const fn total_channels(&self) -> u32 {
         self.total_channels
     }
 
+    /// Returns the live-instance registry.
+    fn instances(&self) -> &Mutex<VgpuInstances<'gpu>> {
+        &self.instances
+    }
+
     fn fifo_engine_list(&self) -> &FifoEngineList {
         &self.fifo_engine_list
     }
diff --git a/drivers/gpu/nova-core/vgpu/commands.rs b/drivers/gpu/nova-core/vgpu/commands.rs
index 63b6f0e3d862..2a9f7ec18ae6 100644
--- a/drivers/gpu/nova-core/vgpu/commands.rs
+++ b/drivers/gpu/nova-core/vgpu/commands.rs
@@ -67,7 +67,6 @@
 };
 
 /// Query the vGPU type assigned to a VF by its DBDF.
-#[expect(dead_code)]
 pub(super) fn query_assigned_vf_type(cmdq: &Cmdq<'_>, dbdf: Dbdf) -> Result<u32> {
     let request = u64::from(dbdf.into_raw()).to_le_bytes();
     let response =
@@ -210,6 +209,16 @@ pub(super) fn set_plugin_bme(
     rpc.rpc_call_nvkv(dev, bar0, gfid, RpcMessage::UpdateBmeState, &bme)
 }
 
+/// Reset an active GSP plugin.
+pub(super) fn reset_plugin(
+    dev: &device::Device<device::Bound>,
+    bar: Bar0<'_>,
+    gfid: Gfid,
+    rpc: &mut PluginRpc<'_, '_>,
+) -> Result {
+    rpc.rpc_call(dev, bar, gfid, RpcMessage::Reset, &[])
+}
+
 /// Whether a failed allocation may still have transferred CHID ownership to firmware.
 pub(super) enum CeUtilsAllocError {
     /// A matching firmware response explicitly rejected the allocation.
diff --git a/drivers/gpu/nova-core/vgpu/fw/commands.rs b/drivers/gpu/nova-core/vgpu/fw/commands.rs
index aabbf0987daa..c5e5c1eb714e 100644
--- a/drivers/gpu/nova-core/vgpu/fw/commands.rs
+++ b/drivers/gpu/nova-core/vgpu/fw/commands.rs
@@ -33,6 +33,7 @@
 pub(in crate::vgpu) enum RpcMessage {
     VersionNegotiation = bindings::MESSAGE_NV_VGPU_CPU_RPC_MSG_VERSION_NEGOTIATION,
     SetupConfigParamsAndInit = bindings::MESSAGE_NV_VGPU_CPU_RPC_MSG_SETUP_CONFIG_PARAMS_AND_INIT,
+    Reset = bindings::MESSAGE_NV_VGPU_CPU_RPC_MSG_RESET,
     UpdateBmeState = bindings::MESSAGE_NV_VGPU_CPU_RPC_MSG_UPDATE_BME_STATE,
 }
 
diff --git a/drivers/gpu/nova-core/vgpu/instance.rs b/drivers/gpu/nova-core/vgpu/instance.rs
index 95d300803ef6..e1fa128ce70a 100644
--- a/drivers/gpu/nova-core/vgpu/instance.rs
+++ b/drivers/gpu/nova-core/vgpu/instance.rs
@@ -9,7 +9,8 @@
     prelude::*,
     ptr::Alignment,
     sizes::SizeConstants,
-    str::CString, //
+    str::CString,
+    sync::Mutex, //
     time::{
         delay::fsleep,
         Delta,
@@ -56,6 +57,7 @@
     free_ceutils,
     negotiate_plugin_version,
     query_vgpu_properties,
+    reset_plugin,
     send_bootload,
     send_cleanup,
     send_plugin_config,
@@ -116,7 +118,6 @@ fn wait_plugin_ready(
 pub(super) struct Gfid(pub(super) u32);
 
 /// Resource requirements and device identity for one vGPU type.
-#[expect(dead_code)]
 pub(super) struct VgpuType {
     vgpu_type_id: u32,
     bar1_length: u64,
@@ -132,6 +133,22 @@ pub(super) const fn vgpu_type_id(&self) -> u32 {
         self.vgpu_type_id
     }
 
+    pub(super) const fn bar1_length(&self) -> u64 {
+        self.bar1_length
+    }
+
+    pub(super) const fn pci_dev_id(&self) -> u32 {
+        self.pci_dev_id
+    }
+
+    pub(super) const fn pci_subsys_id(&self) -> u32 {
+        self.pci_subsys_id
+    }
+
+    pub(super) const fn fb_length(&self) -> u64 {
+        self.fb_length
+    }
+
     fn from_properties(properties: &VgpuProperties) -> Self {
         Self {
             vgpu_type_id: properties.type_id,
@@ -158,6 +175,7 @@ pub(super) struct VgpuInstance<'gpu> {
     pub(super) plugin_rpc: PluginRpc<'gpu, 'gpu>,
     ceutils: Option<CeUtils>,
     initialized: bool,
+    active: bool,
     needs_teardown: bool,
 }
 
@@ -233,6 +251,7 @@ fn shutdown(&mut self, dev: &device::Device<device::Bound>, cmdq: &Cmdq<'_>) ->
             send_shutdown(dev, cmdq, self.gfid)?;
             dev_dbg!(dev, "shutdown: gfid={} stopped\n", self.gfid.0);
         }
+        self.active = false;
         Ok(())
     }
 }
@@ -245,7 +264,6 @@ pub(super) struct InstanceInfo {
     vm_pid: u32,
 }
 
-#[expect(dead_code)]
 impl InstanceInfo {
     pub(super) const fn new(gfid: Gfid, dbdf: Dbdf, vgpu_type: VgpuType, vm_pid: u32) -> Self {
         Self {
@@ -296,7 +314,6 @@ pub(super) struct VgpuInstances<'gpu> {
     vram_slots: Option<VgpuVramSlotAllocator>,
 }
 
-#[expect(dead_code)]
 impl<'gpu> VgpuInstances<'gpu> {
     pub(super) const fn new() -> Self {
         Self {
@@ -420,6 +437,7 @@ pub(super) fn allocate_instance(
             plugin_rpc: PluginRpc::new(comm),
             ceutils: None,
             initialized: false,
+            active: false,
             needs_teardown: false,
         };
         // Register ownership before firmware work so an uncertain result leaves
@@ -503,6 +521,35 @@ pub(super) fn activate_instance(
         Ok(())
     }
 
+    /// Reset an active instance and scrub its guest VRAM.
+    pub(super) fn reset_instance(
+        &mut self,
+        dev: &device::Device<device::Bound>,
+        cmdq: &Cmdq<'_>,
+        bar: Bar0<'_>,
+        bar_user: &BarUser<'gpu>,
+        mm: &mut GpuMm<'_>,
+        gfid: Gfid,
+    ) -> Result {
+        let instance = self
+            .instances
+            .iter_mut()
+            .find(|instance| instance.gfid == gfid)
+            .ok_or(ENOENT)?;
+        if !instance.active {
+            return Err(EBUSY);
+        }
+
+        reset_plugin(dev, bar, instance.gfid, &mut instance.plugin_rpc)?;
+        instance.ceutils.as_ref().ok_or(EINVAL)?.scrub_guest_fb(
+            dev,
+            cmdq,
+            bar_user,
+            mm,
+            &instance.vram_slot.fbmem,
+        )
+    }
+
     /// Stop the plugin and release the instance's firmware and host resources.
     pub(super) fn destroy_instance(
         &mut self,
@@ -534,8 +581,104 @@ pub(super) fn destroy_instance(
 }
 
 /// Query and decode one vGPU type using the typed NVKV schema.
-#[expect(dead_code)]
 pub(super) fn query_vgpu_type(cmdq: &Cmdq<'_>, type_id: u32) -> Result<VgpuType> {
     let properties = query_vgpu_properties(cmdq, type_id)?;
     Ok(VgpuType::from_properties(&properties))
 }
+
+/// Activate an instance already owned by the live-instance registry.
+///
+/// If activation fails, attempt full teardown before returning the original
+/// error.
+#[expect(clippy::too_many_arguments)]
+fn activate_registered_instance<'gpu>(
+    instances: &mut VgpuInstances<'gpu>,
+    dev: &device::Device<device::Bound>,
+    cmdq: &Cmdq<'_>,
+    bar: Bar0<'_>,
+    bar_user: &BarUser<'gpu>,
+    mm: &mut GpuMm<'_>,
+    gfid: Gfid,
+    vgpu: &VgpuManager<'gpu>,
+) -> Result {
+    let index = instances
+        .instances
+        .iter()
+        .position(|instance| instance.gfid == gfid)
+        .ok_or(EIO)?;
+    let activation_result = instances.activate_instance(dev, cmdq, bar, gfid, vgpu);
+
+    if let Err(original_error) = activation_result {
+        if let Err(cleanup_error) = instances.destroy_instance(dev, cmdq, bar_user, mm, gfid) {
+            dev_err!(
+                dev,
+                "vgpu_open: cleanup failed for gfid={} after activation error {:?}: {:?}\n",
+                gfid.0,
+                original_error,
+                cleanup_error,
+            );
+        }
+        return Err(original_error);
+    }
+
+    instances.instances[index].active = true;
+    Ok(())
+}
+
+impl<'gpu> VgpuManager<'gpu> {
+    /// Allocate, register, and activate a vGPU instance.
+    ///
+    /// Keep the registry locked from allocation through activation or rollback
+    /// so duplicate checks and vGPU type limits remain stable.
+    pub(super) fn create_instance(
+        &self,
+        dev: &device::Device<device::Bound>,
+        cmdq: &Cmdq<'_>,
+        bar: Bar0<'_>,
+        bar_user: &'gpu BarUser<'gpu>,
+        mm: &Mutex<GpuMm<'gpu>>,
+        info: InstanceInfo,
+    ) -> Result {
+        let mut instances = self.instances().lock();
+        // Global vGPU lock order: instances -> MM -> BAR-user VMM.
+        let mut mm = mm.lock();
+        let gfid = instances.allocate_instance(dev, cmdq, bar_user, &mut mm, self, info)?;
+
+        activate_registered_instance(
+            &mut instances,
+            dev,
+            cmdq,
+            bar,
+            bar_user,
+            &mut mm,
+            gfid,
+            self,
+        )
+    }
+    pub(super) fn close_instance(
+        &self,
+        dev: &device::Device<device::Bound>,
+        cmdq: &Cmdq<'_>,
+        bar_user: &BarUser<'gpu>,
+        mm: &Mutex<GpuMm<'gpu>>,
+        gfid: Gfid,
+    ) -> Result {
+        let mut instances = self.instances().lock();
+        let mut mm = mm.lock();
+        instances.destroy_instance(dev, cmdq, bar_user, &mut mm, gfid)
+    }
+
+    pub(super) fn reset_instance(
+        &self,
+        dev: &device::Device<device::Bound>,
+        cmdq: &Cmdq<'_>,
+        bar: Bar0<'_>,
+        bar_user: &BarUser<'gpu>,
+        mm: &Mutex<GpuMm<'gpu>>,
+        gfid: Gfid,
+    ) -> Result {
+        let mut instances = self.instances().lock();
+        let mut mm = mm.lock();
+        instances.reset_instance(dev, cmdq, bar, bar_user, &mut mm, gfid)
+    }
+}
diff --git a/drivers/gpu/nova-core/vgpu/vgpu_api.rs b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
new file mode 100644
index 000000000000..c60e31ba2623
--- /dev/null
+++ b/drivers/gpu/nova-core/vgpu/vgpu_api.rs
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Nova Core VF API and its C operations table.
+
+use core::num::NonZero;
+
+use kernel::{
+    bindings,
+    device,
+    interop::ffi::{
+        Abi,
+        Token, //
+    },
+    pci,
+    prelude::*,
+    sync::Mutex,
+    types::ForLt, //
+};
+
+use crate::{
+    driver::Bar0,
+    gpu::Gpu,
+    gsp::cmdq::Cmdq,
+    mm::{
+        bar_user::BarUser,
+        GpuMm, //
+    },
+    vgpu::instance::{
+        query_vgpu_type,
+        Gfid,
+        InstanceInfo,
+        VgpuType, //
+    },
+};
+
+use super::{
+    commands::{
+        query_assigned_vf_type,
+        Dbdf, //
+    },
+    VgpuManager, //
+};
+
+#[repr(transparent)]
+struct VgpuTypeInfo(kernel::bindings::nvidia_vgpu_type_info);
+
+impl VgpuTypeInfo {
+    fn from_vgpu_type(vgpu_type: &VgpuType) -> Self {
+        Self(kernel::bindings::nvidia_vgpu_type_info {
+            pci_dev_id: vgpu_type.pci_dev_id(),
+            pci_subsys_id: vgpu_type.pci_subsys_id(),
+            bar1_length: vgpu_type.bar1_length(),
+        })
+    }
+}
+
+/// PF-owned operations available while a VF driver is bound.
+pub(crate) struct NovaCoreVfApi<'gpu> {
+    pdev: &'gpu pci::Device<device::Bound>,
+    cmdq: &'gpu Cmdq<'gpu>,
+    bar: Bar0<'gpu>,
+    bar_user: &'gpu BarUser<'gpu>,
+    mm: &'gpu Mutex<GpuMm<'gpu>>,
+    vgpu: Option<&'gpu VgpuManager<'gpu>>,
+    total_vfs: Option<NonZero<u16>>,
+}
+
+impl<'gpu> NovaCoreVfApi<'gpu> {
+    pub(crate) fn new(gpu: &'gpu Gpu<'gpu>, pdev: &'gpu pci::Device<device::Bound>) -> Self {
+        Self {
+            pdev,
+            cmdq: gpu.cmdq(),
+            bar: gpu.bar0(),
+            bar_user: gpu.bar_user(),
+            mm: gpu.mm(),
+            vgpu: gpu.vgpu_manager(),
+            total_vfs: gpu.vgpu_total_vfs(),
+        }
+    }
+
+    pub(crate) fn is_available(&self) -> bool {
+        self.vgpu.is_some()
+    }
+
+    fn gfid(&self, gfid: u32) -> Result<Gfid> {
+        let total_vfs = self.total_vfs.ok_or(ENODEV)?;
+        if gfid == 0 || gfid > u32::from(total_vfs.get()) {
+            return Err(EINVAL);
+        }
+
+        Ok(Gfid(gfid))
+    }
+}
+
+impl NovaCoreVfApi<'_> {
+    fn open_instance(&self, gfid: u32, sbdf: u32, vm_pid: u32) -> Result<VgpuTypeInfo> {
+        let dev = self.pdev.as_ref();
+        let gfid = self.gfid(gfid)?;
+        let dbdf = Dbdf::from_raw(sbdf);
+
+        dev_dbg!(
+            dev,
+            "vgpu_open: gfid={} sbdf={:#x}\n",
+            gfid.0,
+            dbdf.into_raw()
+        );
+
+        let bar = self.bar;
+        let cmdq = self.cmdq;
+        let vgpu = self.vgpu.ok_or(ENODEV)?;
+
+        let type_id = query_assigned_vf_type(cmdq, dbdf)?;
+        dev_dbg!(
+            dev,
+            "vgpu_open: gfid={} assigned type_id={}\n",
+            gfid.0,
+            type_id
+        );
+
+        let vgpu_type = query_vgpu_type(cmdq, type_id)?;
+        dev_dbg!(
+            dev,
+            "vgpu_open: gfid={} vgpu_type={} fb_length={:#x}\n",
+            gfid.0,
+            vgpu_type.vgpu_type_id(),
+            vgpu_type.fb_length()
+        );
+
+        let type_info = VgpuTypeInfo::from_vgpu_type(&vgpu_type);
+        vgpu.create_instance(
+            dev,
+            cmdq,
+            bar,
+            self.bar_user,
+            self.mm,
+            InstanceInfo::new(gfid, dbdf, vgpu_type, vm_pid),
+        )?;
+
+        Ok(type_info)
+    }
+
+    fn close_instance(&self, gfid: u32) -> Result {
+        let dev = self.pdev.as_ref();
+        let gfid = self.gfid(gfid)?;
+
+        dev_dbg!(dev, "vgpu_close: gfid={}\n", gfid.0);
+
+        let cmdq = self.cmdq;
+        let result =
+            self.vgpu
+                .ok_or(ENODEV)?
+                .close_instance(dev, cmdq, self.bar_user, self.mm, gfid);
+        if let Err(error) = result {
+            dev_err!(dev, "vgpu_close: gfid={} failed: {:?}\n", gfid.0, error);
+        }
+        result
+    }
+
+    fn reset_instance(&self, gfid: u32) -> Result {
+        let dev = self.pdev.as_ref();
+        let gfid = self.gfid(gfid)?;
+
+        dev_dbg!(dev, "vgpu_reset: gfid={}\n", gfid.0);
+
+        let cmdq = self.cmdq;
+        self.vgpu.ok_or(ENODEV)?.reset_instance(
+            dev,
+            cmdq,
+            self.bar,
+            self.bar_user,
+            self.mm,
+            gfid,
+        )?;
+
+        dev_dbg!(dev, "vgpu_reset: gfid={} done\n", gfid.0);
+        Ok(())
+    }
+}
+
+#[kernel::macros::ffi_vtable(NVIDIA_VGPU_OPS: bindings::nvidia_vgpu_ops)]
+impl NovaCoreVfApi<'_> {
+    /// Create and activate an instance, writing its vGPU type information on success.
+    ///
+    /// # Safety
+    ///
+    /// If `type_info` is non-null, it must point to aligned, writable storage
+    /// with no concurrent access for the duration of this call.
+    unsafe fn open(
+        self: Pin<&Self>,
+        gfid: core::ffi::c_uint,
+        sbdf: core::ffi::c_uint,
+        vm_pid: core::ffi::c_uint,
+        type_info: *mut bindings::nvidia_vgpu_type_info,
+    ) -> Result {
+        if type_info.is_null() {
+            return Err(EINVAL);
+        }
+
+        let info = self.open_instance(gfid, sbdf, vm_pid)?;
+        // SAFETY: `type_info` is non-null and the caller guarantees aligned,
+        // writable storage with no concurrent access.
+        unsafe { type_info.write(info.0) };
+        Ok(())
+    }
+
+    /// Tear down an instance, retaining resources if firmware cleanup fails.
+    fn close(self: Pin<&Self>, gfid: core::ffi::c_uint) {
+        let _ = self.close_instance(gfid);
+    }
+
+    /// Reset an instance and scrub its guest VRAM.
+    fn reset(self: Pin<&Self>, gfid: core::ffi::c_uint) -> Result {
+        self.reset_instance(gfid)
+    }
+}
+
+/// C ABI published for VF lifecycle calls.
+pub(crate) struct NovaCoreVfAbi;
+
+// SAFETY: The C header defines the operations layout and ABI identity. The
+// macro initializes that complete operations table, and every trampoline
+// borrows the pinned `NovaCoreVfApi` published by `new_ffi`.
+unsafe impl Abi for NovaCoreVfAbi {
+    type Context = ForLt!(NovaCoreVfApi<'_>);
+    type RawOps = bindings::nvidia_vgpu_ops;
+
+    const OPS: &'static Self::RawOps = &NVIDIA_VGPU_OPS;
+    const TOKEN: Token = Token::new(
+        bindings::NVIDIA_VGPU_FFI_TOKEN_HIGH as u64,
+        bindings::NVIDIA_VGPU_FFI_TOKEN_LOW as u64,
+    );
+    const ABI_MAJOR: u16 = bindings::NVIDIA_VGPU_FFI_ABI_MAJOR as u16;
+    const ABI_MINOR: u16 = bindings::NVIDIA_VGPU_FFI_ABI_MINOR as u16;
+}
diff --git a/include/drm/nvidia_vgpu.h b/include/drm/nvidia_vgpu.h
new file mode 100644
index 000000000000..4a1579a9044d
--- /dev/null
+++ b/include/drm/nvidia_vgpu.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#ifndef __DRM_NVIDIA_VGPU_H__
+#define __DRM_NVIDIA_VGPU_H__
+
+#include <linux/rust_ffi.h>
+
+#define NVIDIA_VGPU_FFI_TOKEN_HIGH	0xa70c87381a844092ULL
+#define NVIDIA_VGPU_FFI_TOKEN_LOW		0xaeedbe595835927cULL
+#define NVIDIA_VGPU_FFI_ABI_MAJOR		1U
+#define NVIDIA_VGPU_FFI_ABI_MINOR		0U
+
+/**
+ * struct nvidia_vgpu_type_info - vGPU type descriptor returned by open
+ * @pci_dev_id: PCI device ID to present to the guest
+ * @pci_subsys_id: PCI subsystem ID to present to the guest
+ * @bar1_length: BAR1 aperture size in MiB
+ */
+struct nvidia_vgpu_type_info {
+	u32 pci_dev_id;
+	u32 pci_subsys_id;
+	u64 bar1_length;
+};
+
+/**
+ * struct nvidia_vgpu_ops - PF operations for NVIDIA vGPU virtual functions
+ * @open: Create and activate an instance, returning its vGPU type information on success
+ * @close: Tear down an instance, retaining resources if firmware cleanup fails
+ * @reset: Reset an instance and scrub its guest VRAM
+ *
+ * Each operation receives the context from the borrowed struct rust_ffi and
+ * a Guest Function ID (VF index + 1). The context remains valid until the VF
+ * driver is fully unbound, including its remove callback. The VF driver must
+ * drain all calls before returning from remove or a failed probe.
+ *
+ * All operations may sleep. They must not acquire the PF device lock because
+ * disabling SR-IOV removes VFs synchronously while holding that lock.
+ *
+ * @open receives the VF address encoded as (segment << 16) | (bus << 8) | devfn
+ * and the VM process's thread-group ID. Its type_info argument must point to
+ * writable storage with no concurrent access. @open and @reset return zero
+ * on success or a negative errno.
+ */
+struct nvidia_vgpu_ops {
+	int (*open)(const void *context, unsigned int gfid, unsigned int sbdf,
+		    unsigned int vm_pid, struct nvidia_vgpu_type_info *type_info);
+	void (*close)(const void *context, unsigned int gfid);
+	int (*reset)(const void *context, unsigned int gfid);
+};
+
+#endif /* __DRM_NVIDIA_VGPU_H__ */
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 2467305c4842..1cf79bec756a 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -37,6 +37,7 @@
 #include <drm/drm_gem_shmem_helper.h>
 #include <drm/drm_gpuvm.h>
 #include <drm/drm_ioctl.h>
+#include <drm/nvidia_vgpu.h>
 #include <kunit/test.h>
 #include <linux/auxiliary_bus.h>
 #include <linux/bitmap.h>

[PATCH 2/2] vfio/nvidia-vgpu: add the NVIDIA vGPU VFIO variant driver

[Zhi Wang]

NVIDIA vGPU VFs require their open, reset, and close lifecycle to be
coordinated with the PF-side nova-core driver.

Add a VFIO PCI variant with an override alias for NVIDIA 10de:2bb5
devices with class/subclass 0302. Leave the programming-interface byte
unmasked so the generated alias ends in i*, as required by libvirt
VFIO variant matching. PFs share this ID with their VFs. Use ordinary
PCI passthrough unless a VF can borrow a compatible, complete Nova
lifecycle interface through the typed SR-IOV descriptor. Both paths use
VFIO PCI core defaults and support PASID attachment; generic passthrough
also retains the core DMA-BUF export operation. Provide a driver-local
enable_sriov option for PFs bound to this driver.

For vGPU instances, expose the firmware-selected device and subsystem
IDs through configuration-space reads and constrain BAR1 region reporting,
read, write, and mmap access to the assigned vGPU type's aperture.
Coordinate firmware reset through PCI reset callbacks and serialize the
entire reset window against instance open and close without retaining a
mutex across a multi-device bus reset. Latch firmware reset failures
until successful reopen, report them through the VFIO error IRQ and
reset ioctls, and preserve core configuration-write return values.

Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvidia-vgpu/Kconfig  |  21 ++
 drivers/vfio/pci/nvidia-vgpu/Makefile |   2 +
 drivers/vfio/pci/nvidia-vgpu/main.c   | 509 ++++++++++++++++++++++++++
 5 files changed, 536 insertions(+)
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/main.c

diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..b48d8d1af42a 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -74,4 +74,6 @@ source "drivers/vfio/pci/qat/Kconfig"
 
 source "drivers/vfio/pci/xe/Kconfig"
 
+source "drivers/vfio/pci/nvidia-vgpu/Kconfig"
+
 endmenu
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..f3498e541555 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -24,3 +24,5 @@ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
 obj-$(CONFIG_QAT_VFIO_PCI) += qat/
 
 obj-$(CONFIG_XE_VFIO_PCI) += xe/
+
+obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu/
diff --git a/drivers/vfio/pci/nvidia-vgpu/Kconfig b/drivers/vfio/pci/nvidia-vgpu/Kconfig
new file mode 100644
index 000000000000..e7791fa79984
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Kconfig
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVIDIA_VGPU_VFIO_PCI
+	tristate "VFIO support for the NVIDIA vGPU"
+	depends on NOVA_CORE && PCI_IOV
+	select VFIO_PCI_CORE
+	help
+	  This option enables VFIO (Virtual Function I/O) support for
+	  NVIDIA virtual GPUs (vGPU). It allows the assignment of a virtual
+	  GPU instance to userspace applications via VFIO, typically used
+	  with hypervisors such as KVM and device emulators like QEMU.
+
+	  Devices without an available Nova vGPU interface use ordinary PCI
+	  passthrough. Both paths use the VFIO PCI core defaults; vfio-pci
+	  module options do not apply. The enable_sriov parameter of this
+	  driver controls SR-IOV configuration for PFs bound to it.
+
+	  The NVIDIA vGPU allows a physical GPU to be partitioned into
+	  multiple virtual GPUs, each of which can be passed to a virtual
+	  machine as a PCI device using the standard VFIO infrastructure.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile
new file mode 100644
index 000000000000..193cc801a081
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu-vfio-pci.o
+nvidia-vgpu-vfio-pci-y := main.o
diff --git a/drivers/vfio/pci/nvidia-vgpu/main.c b/drivers/vfio/pci/nvidia-vgpu/main.c
new file mode 100644
index 000000000000..742979fb0ed5
--- /dev/null
+++ b/drivers/vfio/pci/nvidia-vgpu/main.c
@@ -0,0 +1,509 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/compiler.h>
+#include <linux/completion.h>
+#include <linux/minmax.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/overflow.h>
+#include <linux/pci.h>
+#include <linux/pid.h>
+#include <linux/uaccess.h>
+#include <linux/unaligned.h>
+#include <linux/vfio_pci_core.h>
+#include <drm/nvidia_vgpu.h>
+
+#define NVIDIA_VGPU_DRIVER_NAME "nvidia-vgpu-vfio-pci"
+
+static bool enable_sriov;
+module_param(enable_sriov, bool, 0644);
+MODULE_PARM_DESC(enable_sriov, "Enable SR-IOV configuration for PFs bound to this driver");
+
+static int nvidia_vgpu_fb_bar_index(struct pci_dev *pdev)
+{
+	if (pci_resource_flags(pdev, 0) & IORESOURCE_MEM_64)
+		return 2;
+	return 1;
+}
+
+static const struct rust_ffi_token nvidia_vgpu_ffi_token = {
+	.high = NVIDIA_VGPU_FFI_TOKEN_HIGH,
+	.low = NVIDIA_VGPU_FFI_TOKEN_LOW,
+};
+
+struct nvidia_vgpu_pci_core_device {
+	struct vfio_pci_core_device core_device;
+	struct nvidia_vgpu_type_info type_info;
+	const struct nvidia_vgpu_ops *pf_ops;
+	const void *pf_context;
+	/* Protects lifecycle calls and the active/resetting state. */
+	struct mutex instance_lock;
+	struct completion reset_completion;
+	unsigned int gfid;
+	int reset_error;
+	bool instance_active;
+	bool resetting;
+};
+
+/* Encode the VF's PCI segment:bus:device.function as a 32-bit address. */
+static inline unsigned int nvidia_vgpu_vf_sbdf(struct pci_dev *vf)
+{
+	return ((u32)pci_domain_nr(vf->bus) << 16) | pci_dev_id(vf);
+}
+
+/* Keep open and close outside the complete prepare/PCI-reset/done window. */
+static void nvidia_vgpu_lock_instance(struct nvidia_vgpu_pci_core_device *nvdev)
+{
+	for (;;) {
+		mutex_lock(&nvdev->instance_lock);
+		if (!nvdev->resetting)
+			return;
+		mutex_unlock(&nvdev->instance_lock);
+		wait_for_completion(&nvdev->reset_completion);
+	}
+}
+
+static int nvidia_vgpu_open_device(struct vfio_device *core_vdev)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+	struct pci_dev *vf = to_pci_dev(core_vdev->dev);
+	struct nvidia_vgpu_type_info type_info;
+	int ret;
+
+	ret = vfio_pci_core_enable(&nvdev->core_device);
+	if (ret)
+		return ret;
+
+	nvidia_vgpu_lock_instance(nvdev);
+	ret = nvdev->pf_ops->open(nvdev->pf_context, nvdev->gfid,
+				 nvidia_vgpu_vf_sbdf(vf), task_tgid_nr(current),
+				 &type_info);
+	if (!ret) {
+		nvdev->instance_active = true;
+		WRITE_ONCE(nvdev->reset_error, 0);
+	}
+	mutex_unlock(&nvdev->instance_lock);
+	if (ret) {
+		vfio_pci_core_disable(&nvdev->core_device);
+		return ret;
+	}
+
+	nvdev->type_info = type_info;
+	put_unaligned_le16(type_info.pci_dev_id,
+			   nvdev->core_device.vconfig + PCI_DEVICE_ID);
+	pci_dbg(vf, "vgpu open: dev_id=0x%x subsys_id=0x%x bar1_length=0x%llx\n",
+		type_info.pci_dev_id, type_info.pci_subsys_id,
+		type_info.bar1_length);
+	vfio_pci_core_finish_enable(&nvdev->core_device);
+	return 0;
+}
+
+static void nvidia_vgpu_close_device(struct vfio_device *core_vdev)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+
+	nvidia_vgpu_lock_instance(nvdev);
+	nvdev->instance_active = false;
+	nvdev->pf_ops->close(nvdev->pf_context, nvdev->gfid);
+	mutex_unlock(&nvdev->instance_lock);
+	vfio_pci_core_close_device(core_vdev);
+}
+
+static int nvidia_vgpu_bar1_size(struct nvidia_vgpu_pci_core_device *nvdev,
+				 u64 *size)
+{
+	struct pci_dev *pdev = nvdev->core_device.pdev;
+	u64 physical_size = pci_resource_len(pdev, nvidia_vgpu_fb_bar_index(pdev));
+
+	/* The assigned vGPU type reports its VRAM aperture in MiB. */
+	if (check_shl_overflow(nvdev->type_info.bar1_length, 20, size))
+		return -EOVERFLOW;
+
+	/* An unspecified or larger vGPU type aperture uses the physical limit. */
+	if (!*size || *size > physical_size)
+		*size = physical_size;
+
+	return 0;
+}
+
+/* Enforce the same vGPU type aperture reported by GET_REGION_INFO. */
+static int nvidia_vgpu_bar1_access(struct vfio_device *core_vdev,
+				   loff_t pos, size_t *count)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(pos);
+	u64 size;
+	int ret;
+
+	if (!*count || index != nvidia_vgpu_fb_bar_index(nvdev->core_device.pdev))
+		return 0;
+
+	ret = nvidia_vgpu_bar1_size(nvdev, &size);
+	if (ret)
+		return ret;
+
+	pos &= VFIO_PCI_OFFSET_MASK;
+	if (pos >= size)
+		return -EINVAL;
+
+	*count = min_t(u64, *count, size - pos);
+	return 0;
+}
+
+static ssize_t nvidia_vgpu_read_config(struct vfio_device *core_vdev,
+				      char __user *buf, size_t count,
+				      loff_t *ppos)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	loff_t next = *ppos, copy_offset;
+	size_t copy_count, register_offset;
+	__le16 value;
+	ssize_t ret;
+
+	ret = vfio_pci_core_read(core_vdev, buf, count, &next);
+	if (ret <= 0)
+		return ret;
+
+	/* The core reads the subsystem ID from physical configuration space. */
+	if (vfio_pci_core_range_intersect_range(pos, ret, PCI_SUBSYSTEM_ID,
+					      sizeof(value), &copy_offset,
+					      &copy_count, &register_offset)) {
+		value = cpu_to_le16(nvdev->type_info.pci_subsys_id);
+		if (copy_to_user(buf + copy_offset,
+				(u8 *)&value + register_offset, copy_count))
+			return -EFAULT;
+	}
+
+	*ppos = next;
+	return ret;
+}
+
+static ssize_t nvidia_vgpu_pci_read(struct vfio_device *core_vdev,
+				    char __user *buf, size_t count,
+				    loff_t *ppos)
+{
+	int ret;
+
+	if (VFIO_PCI_OFFSET_TO_INDEX(*ppos) == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvidia_vgpu_read_config(core_vdev, buf, count, ppos);
+
+	ret = nvidia_vgpu_bar1_access(core_vdev, *ppos, &count);
+	if (ret)
+		return ret;
+
+	return vfio_pci_core_read(core_vdev, buf, count, ppos);
+}
+
+static ssize_t nvidia_vgpu_pci_write(struct vfio_device *core_vdev,
+				     const char __user *buf, size_t count,
+				     loff_t *ppos)
+{
+	int ret;
+
+	ret = nvidia_vgpu_bar1_access(core_vdev, *ppos, &count);
+	if (ret)
+		return ret;
+
+	/* Reset callbacks report a firmware FLR failure through the error IRQ. */
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+static int nvidia_vgpu_pci_mmap(struct vfio_device *core_vdev,
+				struct vm_area_struct *vma)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+	unsigned int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
+	u64 size, req_start, req_len, end;
+	int ret;
+
+	if (index != nvidia_vgpu_fb_bar_index(nvdev->core_device.pdev))
+		return vfio_pci_core_mmap(core_vdev, vma);
+
+	ret = nvidia_vgpu_bar1_size(nvdev, &size);
+	if (ret)
+		return ret;
+
+	req_start = (vma->vm_pgoff & (VFIO_PCI_OFFSET_MASK >> PAGE_SHIFT)) << PAGE_SHIFT;
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
+	    check_add_overflow(req_start, req_len, &end))
+		return -EOVERFLOW;
+	if (end > size)
+		return -EINVAL;
+
+	return vfio_pci_core_mmap(core_vdev, vma);
+}
+
+static int nvidia_vgpu_get_region_info(struct vfio_device *core_vdev,
+				       struct vfio_region_info *info,
+				       struct vfio_info_cap *caps)
+{
+	struct pci_dev *pdev = to_pci_dev(core_vdev->dev);
+	int ret;
+
+	ret = vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
+	if (ret)
+		return ret;
+
+	if (info->index == nvidia_vgpu_fb_bar_index(pdev) && info->size) {
+		struct nvidia_vgpu_pci_core_device *nvdev =
+			container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+				     core_device.vdev);
+		u64 vgpu_bar1;
+
+		ret = nvidia_vgpu_bar1_size(nvdev, &vgpu_bar1);
+		if (ret)
+			return ret;
+
+		if (vgpu_bar1 && vgpu_bar1 < info->size)
+			info->size = vgpu_bar1;
+	}
+
+	return 0;
+}
+
+static long nvidia_vgpu_pci_ioctl(struct vfio_device *core_vdev,
+				  unsigned int cmd, unsigned long arg)
+{
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvidia_vgpu_pci_core_device,
+			     core_device.vdev);
+	bool reset = cmd == VFIO_DEVICE_RESET || cmd == VFIO_DEVICE_PCI_HOT_RESET;
+	long ret;
+
+	/* A failed firmware reset requires closing and reopening the instance. */
+	if (reset && READ_ONCE(nvdev->reset_error))
+		return READ_ONCE(nvdev->reset_error);
+
+	ret = vfio_pci_core_ioctl(core_vdev, cmd, arg);
+	if (ret)
+		return ret;
+
+	if (reset)
+		return READ_ONCE(nvdev->reset_error);
+
+	return 0;
+}
+
+static void nvidia_vgpu_reset_prepare(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_device, struct nvidia_vgpu_pci_core_device,
+			     core_device);
+	int ret = 0;
+
+	/*
+	 * PCI holds the VF device lock, and VFIO may hold its memory lock.
+	 * Open and close release instance_lock before entering vfio-pci-core;
+	 * PF operations take only GPU locks, never the PF or VF device lock.
+	 */
+	mutex_lock(&nvdev->instance_lock);
+	nvdev->resetting = true;
+	reinit_completion(&nvdev->reset_completion);
+	if (nvdev->instance_active)
+		ret = nvdev->pf_ops->reset(nvdev->pf_context, nvdev->gfid);
+	if (ret && !nvdev->reset_error)
+		WRITE_ONCE(nvdev->reset_error, ret);
+	mutex_unlock(&nvdev->instance_lock);
+}
+
+static void nvidia_vgpu_reset_done(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+	struct nvidia_vgpu_pci_core_device *nvdev =
+		container_of(core_device, struct nvidia_vgpu_pci_core_device,
+			     core_device);
+	int ret;
+
+	mutex_lock(&nvdev->instance_lock);
+	ret = nvdev->reset_error;
+	/* PCI reset callbacks cannot return an error or abort the PCI reset. */
+	if (ret) {
+		pci_err(pdev, "vGPU instance has a failed firmware reset: %d\n", ret);
+		vfio_pci_core_aer_err_detected(pdev, pci_channel_io_normal);
+	}
+	nvdev->resetting = false;
+	complete_all(&nvdev->reset_completion);
+	mutex_unlock(&nvdev->instance_lock);
+}
+
+static const struct pci_error_handlers nvidia_vgpu_err_handlers = {
+	.error_detected = vfio_pci_core_aer_err_detected,
+	.reset_prepare = nvidia_vgpu_reset_prepare,
+	.reset_done = nvidia_vgpu_reset_done,
+};
+
+static const struct vfio_device_ops nvidia_vgpu_pci_ops = {
+	.name		= NVIDIA_VGPU_DRIVER_NAME,
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvidia_vgpu_open_device,
+	.close_device	= nvidia_vgpu_close_device,
+	.ioctl		= nvidia_vgpu_pci_ioctl,
+	.get_region_info_caps = nvidia_vgpu_get_region_info,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= nvidia_vgpu_pci_read,
+	.write		= nvidia_vgpu_pci_write,
+	.mmap		= nvidia_vgpu_pci_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.match_token_uuid = vfio_pci_core_match_token_uuid,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+	.pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas,
+	.pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas,
+};
+
+static int nvidia_vgpu_passthrough_open(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+	return 0;
+}
+
+/* PFs and VFs without a compatible Nova interface retain generic VFIO behavior. */
+static const struct vfio_device_ops nvidia_vgpu_passthrough_ops = {
+	.name		= NVIDIA_VGPU_DRIVER_NAME,
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvidia_vgpu_passthrough_open,
+	.close_device	= vfio_pci_core_close_device,
+	.ioctl		= vfio_pci_core_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= vfio_pci_core_read,
+	.write		= vfio_pci_core_write,
+	.mmap		= vfio_pci_core_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.match_token_uuid = vfio_pci_core_match_token_uuid,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+	.pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas,
+	.pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas,
+};
+
+static const struct vfio_pci_device_ops nvidia_vgpu_passthrough_dev_ops = {
+	.get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
+};
+
+static int nvidia_vgpu_pci_probe(struct pci_dev *pdev,
+				 const struct pci_device_id *id)
+{
+	const struct vfio_device_ops *device_ops = &nvidia_vgpu_passthrough_ops;
+	struct nvidia_vgpu_pci_core_device *nvdev;
+	const struct nvidia_vgpu_ops *pf_ops = NULL;
+	const void *pf_context = NULL;
+	int vf_id = -1;
+	int ret;
+
+	if (pdev->is_virtfn) {
+		const struct rust_ffi *ffi;
+
+		ffi = pci_iov_borrow_rust_pf_data(pdev, &nvidia_vgpu_ffi_token,
+						  NVIDIA_VGPU_FFI_ABI_MAJOR,
+						  NVIDIA_VGPU_FFI_ABI_MINOR,
+						  sizeof(*pf_ops));
+		if (!IS_ERR(ffi)) {
+			const struct nvidia_vgpu_ops *ops = ffi->ops;
+
+			if (ops->open && ops->close && ops->reset) {
+				pf_ops = ops;
+				pf_context = ffi->context;
+				device_ops = &nvidia_vgpu_pci_ops;
+			}
+		}
+	}
+
+	if (pf_ops) {
+		vf_id = pci_iov_vf_id(pdev);
+		if (vf_id < 0)
+			return vf_id;
+	}
+
+	nvdev = vfio_alloc_device(nvidia_vgpu_pci_core_device, core_device.vdev,
+				  &pdev->dev, device_ops);
+	if (IS_ERR(nvdev))
+		return PTR_ERR(nvdev);
+
+	mutex_init(&nvdev->instance_lock);
+	init_completion(&nvdev->reset_completion);
+	nvdev->pf_ops = pf_ops;
+	nvdev->pf_context = pf_context;
+	nvdev->gfid = vf_id + 1;
+	if (!pf_ops)
+		nvdev->core_device.pci_ops = &nvidia_vgpu_passthrough_dev_ops;
+	dev_set_drvdata(&pdev->dev, &nvdev->core_device);
+	ret = vfio_pci_core_register_device(&nvdev->core_device);
+	if (ret)
+		goto out_put_vdev;
+
+	return 0;
+
+out_put_vdev:
+	vfio_put_device(&nvdev->core_device.vdev);
+	return ret;
+}
+
+static void nvidia_vgpu_pci_remove(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+
+	/* Drain all VFIO callbacks before the borrowed PF context expires. */
+	vfio_pci_core_unregister_device(core_device);
+	vfio_put_device(&core_device->vdev);
+}
+
+static const struct pci_device_id nvidia_vgpu_pci_table[] = {
+	/* RTX PRO 6000 Blackwell Server Edition PFs and VFs share this ID. */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2bb5),
+	  .class = PCI_CLASS_DISPLAY_3D << 8, .class_mask = 0xffff00 },
+	{}
+};
+MODULE_DEVICE_TABLE(pci, nvidia_vgpu_pci_table);
+
+static int nvidia_vgpu_sriov_configure(struct pci_dev *pdev, int nr_virtfn)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	if (!enable_sriov)
+		return -ENOENT;
+
+	return vfio_pci_core_sriov_configure(vdev, nr_virtfn);
+}
+
+static struct pci_driver nvidia_vgpu_pci_driver = {
+	.name		= NVIDIA_VGPU_DRIVER_NAME,
+	.id_table	= nvidia_vgpu_pci_table,
+	.probe		= nvidia_vgpu_pci_probe,
+	.remove		= nvidia_vgpu_pci_remove,
+	.sriov_configure = nvidia_vgpu_sriov_configure,
+	.err_handler	= &nvidia_vgpu_err_handlers,
+	.driver_managed_dma = true,
+};
+module_pci_driver(nvidia_vgpu_pci_driver);
+
+MODULE_DESCRIPTION("NVIDIA vGPU vfio-pci driver");
+MODULE_LICENSE("GPL");