[PATCH v2 00/32] Introduce NVIDIA vGPU manager

[Zhi Wang <zhiw@nvidia.com>]

Add the nova-core support needed to create and manage NVIDIA vGPU
instances.

The overall control path is shown below. This v2 posting contains the
nova-core manager; the VFIO lifecycle interface and variant driver are
split out for separate review.

 +-----------------------+         +-----------------------+
 | Linux guest           |         | Windows guest         |
 | NVIDIA guest driver   |         | NVIDIA guest driver   |
 +-----------+-----------+         +-----------+-----------+
             |                                 |
 +-----------v-----------+         +-----------v-----------+
 | QEMU (VFIO user)      |         | QEMU (VFIO user)      |
 +-----------+-----------+         +-----------+-----------+
             +----------------+----------------+
                              |
                      +-------v-------+
                      |   VFIO core   |
                      +-------+-------+
                              |
        +---------------------v---------------------+
        | NVIDIA vGPU VFIO variant driver           |
        +---------------------+---------------------+
                              |
              +---------------+----------------------+
              | binds PCI VF                         | PF lifecycle API
              |                                      | open / close / reset
              v                                      v
     +--------+--------+               +-------------+-------------+
     |    PCI VF(s)    |               | nova-core (PF driver)     |
     |  (virtual GPU)  |               |       vGPU Manager        |
     +--------+--------+               |     VRAM / channels ...   |
              |                        +-------------+-------------+
              |                                      |
+-------------v--------------------------------------v----------------+
|                      NVIDIA physical GPU                            |
|            VF resources                 PF / GSP / VRAM             |
+---------------------------------------------------------------------+

On the nova-core side, the PF driver owns the GPU firmware-facing part
of vGPU management. It already detects vGPU mode before GSP boot and
supplies the SR-IOV topology to the GPU System Processor firmware
(GSP-RM). This series extends it to:

  - Retain the firmware-reported VMMU segment size and FIFO engine
    ordering after GSP_INIT, together with nova-core's channel capacity.
  - Add VramBlock and VramRegion ownership plus BarMapping for bounded CPU
    mappings of VRAM-backed control structures.
  - Query GSP-RM through its command queue for the vGPU type already
    assigned to a VF and its properties.
  - Build a layout-validated pool of paired guest VRAM and management-heap
    slots for the vGPU type, reserve contiguous channel ranges, and keep
    live allocations in a per-PF instance registry.
  - Use the typed r000 firmware bindings for control and response layouts,
    message IDs, and firmware-defined region sizes.
  - Handle GMC transactions over the GSP queues, match GMC replies by
    command and sequence, dispatch interleaved RM RPCs from the shared
    GSP-to-CPU message queue, and consume stale GMC messages.
  - Implement the GMCAPI boot, shutdown, and cleanup operations with
    typed channel and resource maps for each plugin instance.
  - Establish the BAR1-backed PluginRpc channel, negotiate its protocol,
    and send the VM configuration and plugin BME-state update.
  - Use per-instance CeUtils channels to scrub guest VRAM on allocation
    and shutdown. If ownership or completion is uncertain, keep the
    affected VRAM and channels out of the allocators.
  - Expose the three per-instance plugin log buffers through debugfs with
    the header needed by nvlog_decoder.
  - Keep resource locking, teardown, and rollback inside nova-core.
  - Select the firmware-defined 48-VM WPR2 heap when a device advertises
    more than 32 VFs.

This is a ground-up rework of the earlier vGPU RFC [1]. In particular,
the vGPU manager has been moved from the VFIO driver into the nova-core
implementation, and the companion VFIO side no longer carries copies of
RM firmware headers. The VFIO driver attaches to the VF and uses its
PF's nova-core state for management, as discussed in that thread.

The patches are organized as follows:

  1-8   Initialize VgpuManager and select the larger WPR2 heap for 48-VF
        devices.
  9-12  Add the memory building blocks.
 13-16  Implement GMC transactions.
 17-21  Add firmware bindings and allocate instances.
 22-29  Implement plugin boot and RPC.
 30-32  Scrub guest VRAM and expose plugin diagnostics.

Changes since v1 [2]:

  - Split the nova-core work into smaller patches and separate the VFIO
    lifecycle interface and variant driver from this series.
  - Separate boot state from VgpuManager and use borrowed GPU resources.
  - Split VramBlock, VramRegion, and BarMapping; separate firmware
    encoding, command submission, and instance lifecycle operations.
  - Include build ID/log-header support and introduce command schemas
    with their consumers.

The series is based on drm-rust-next plus the nova-core r000 GSP,
NVKV v2, memory-management, SR-IOV, Rust bitmap/id-pool, and PCI
abstraction work. The exact base commit is recorded below, and the
complete prerequisite stack is available in [3].

[1] https://lore.kernel.org/kvm/20250903221111.3866249-1-zhiw@nvidia.com/
[3] https://lore.kernel.org/nova-gpu/20260905081116.106613-1-zhiw@nvidia.com/
[2] https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-wip-nova-gsp-20260914

Alok Kumar (2):
  gpu: nova-core: gsp: decode the FIFO engine table
  gpu: nova-core: vgpu: add VRAM slot allocator

John Hubbard (1):
  gpu: nova-core: add build ID headers to debugfs log buffer dumps

Zhi Wang (29):
  gpu: nova-core: gsp: pass boot context through setup helpers
  gpu: nova-core: gsp: decouple boot context from VgpuManager
  gpu: nova-core: vgpu: detect boot state independently
  gpu: nova-core: gpu: add a channel ID pool for vGPU
  gpu: nova-core: gsp: expose the VMMU segment size
  gpu: nova-core: vgpu: initialize runtime parameters after GSP boot
  gpu: nova-core: vgpu: reserve the 48-VM WPR2 heap
  gpu: nova-core: mm: borrow BarUser for temporary BAR1 access
  gpu: nova-core: mm: add VramBlock
  gpu: nova-core: mm: add VramRegion
  gpu: nova-core: mm: add BarMapping
  gpu: nova-core: gsp: dispatch RM RPC messages during GMC receive
  gpu: nova-core: gsp: add synchronous GMC transactions
  gpu: nova-core: gsp: add status-only GMC transactions
  gpu: nova-core: gsp: wait for GMC completion events
  gpu: nova-core: vgpu: add r000 plugin bindings
  gpu: nova-core: gsp: factor out NVKV payload conversion
  gpu: nova-core: vgpu: query VF assignments and properties
  gpu: nova-core: vgpu: add instance create/destroy
  gpu: nova-core: vgpu: encode vGPU boot requests
  gpu: nova-core: vgpu: add GSP plugin communication buffers
  gpu: nova-core: vgpu: add instance boot and shutdown
  gpu: nova-core: vgpu: initialize GSP plugin RPC buffers
  gpu: nova-core: vgpu: add GSP plugin RPC transactions
  gpu: nova-core: vgpu: negotiate the GSP plugin RPC version
  gpu: nova-core: vgpu: send GSP plugin configuration parameters
  gpu: nova-core: vgpu: update the GSP plugin BME state
  gpu: nova-core: vgpu: scrub guest VRAM with CeUtils
  gpu: nova-core: vgpu: export plugin log buffers via debugfs

 drivers/gpu/nova-core/fb.rs                   |  28 +-
 drivers/gpu/nova-core/firmware.rs             |  34 ++
 drivers/gpu/nova-core/firmware/gsp.rs         |  16 +-
 drivers/gpu/nova-core/gpu.rs                  |  68 ++-
 drivers/gpu/nova-core/gpu/channel.rs          |   4 +
 drivers/gpu/nova-core/gsp.rs                  | 206 ++++++-
 drivers/gpu/nova-core/gsp/boot.rs             |   4 +-
 drivers/gpu/nova-core/gsp/cmdq.rs             | 287 ++++++++--
 drivers/gpu/nova-core/gsp/commands.rs         | 105 ++--
 drivers/gpu/nova-core/gsp/fw.rs               |  26 +-
 drivers/gpu/nova-core/gsp/fw/commands.rs      |  39 +-
 .../gpu/nova-core/gsp/fw/r000_00/bindings.rs  | 175 ++++++
 drivers/gpu/nova-core/gsp/hal/gh100.rs        |   2 +-
 drivers/gpu/nova-core/gsp/hal/tu102.rs        |   2 +-
 drivers/gpu/nova-core/gsp/nvkv.rs             |  35 ++
 drivers/gpu/nova-core/gsp/nvkv/encode.rs      |  12 +
 drivers/gpu/nova-core/mm.rs                   |   6 +-
 drivers/gpu/nova-core/mm/bar_user.rs          | 145 ++++-
 drivers/gpu/nova-core/mm/vram.rs              | 140 +++++
 drivers/gpu/nova-core/regs.rs                 |  33 +-
 drivers/gpu/nova-core/vgpu.rs                 |  85 ++-
 drivers/gpu/nova-core/vgpu/commands.rs        | 333 +++++++++++
 drivers/gpu/nova-core/vgpu/fw.rs              |  70 +++
 drivers/gpu/nova-core/vgpu/fw/commands.rs     | 457 +++++++++++++++
 drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs | 495 ++++++++++++++++
 drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs  | 169 ++++++
 drivers/gpu/nova-core/vgpu/instance.rs        | 538 ++++++++++++++++++
 drivers/gpu/nova-core/vgpu/log.rs             | 171 ++++++
 drivers/gpu/nova-core/vgpu/scrubber.rs        | 162 ++++++
 drivers/gpu/nova-core/vgpu/vram.rs            | 146 +++++
 30 files changed, 3783 insertions(+), 210 deletions(-)
 create mode 100644 drivers/gpu/nova-core/mm/vram.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/commands.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw/commands.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/instance.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/log.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/scrubber.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/vram.rs


base-commit: 9ff5fd561368ebc3cf1b82a930823ace445564cc