From: Zhi Wang <zhiw@nvidia.com>
To: <dakr@kernel.org>, <airlied@gmail.com>, <simona@ffwll.ch>
Cc: <ojeda@kernel.org>, <alex.gaynor@gmail.com>,
<boqun.feng@gmail.com>, <gary@garyguo.net>,
<bjorn3_gh@protonmail.com>, <lossin@kernel.org>,
<a.hindborg@kernel.org>, <aliceryhl@google.com>,
<tmgross@umich.edu>, <jhubbard@nvidia.com>, <acourbot@nvidia.com>,
<ecourtney@nvidia.com>, <joelagnelf@nvidia.com>,
<apopple@nvidia.com>, <cjia@nvidia.com>, <smitra@nvidia.com>,
<kjaju@nvidia.com>, <alkumar@nvidia.com>, <ankita@nvidia.com>,
<aniketa@nvidia.com>, <kwankhede@nvidia.com>,
<targupta@nvidia.com>, <nova-gpu@lists.linux.dev>,
<linux-kernel@vger.kernel.org>, <zhiwang@kernel.org>,
Zhi Wang <zhiw@nvidia.com>
Subject: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation
Date: Thu, 4 Jun 2026 14:43:35 +0300 [thread overview]
Message-ID: <20260604114339.1565660-6-zhiw@nvidia.com> (raw)
In-Reply-To: <20260604114339.1565660-1-zhiw@nvidia.com>
Add documentation for the Foundation Security Processor (FSP) interface
covering the simplified Hopper/Blackwell boot flow, the Chain of Trust
(COT) message protocol, the MCTP/NVDM message format, and the Product
Reconfiguration Control (PRC) protocol used to query device configuration
knobs such as vGPU mode.
Signed-off-by: Zhi Wang <zhiw@nvidia.com>
---
Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 143 insertions(+)
create mode 100644 Documentation/gpu/nova/core/fsp.rst
diff --git a/Documentation/gpu/nova/core/fsp.rst b/Documentation/gpu/nova/core/fsp.rst
new file mode 100644
index 000000000000..52d618d22bb8
--- /dev/null
+++ b/Documentation/gpu/nova/core/fsp.rst
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+FSP (Foundation Security Processor) and Secure Boot
+===================================================
+This document describes the role of the FSP in the GPU boot sequence on
+Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot
+flow. It also provides a brief overview of the PRC (Product Reconfiguration
+Control) protocol used to query device configuration through FSP. As with
+other documents in this directory, the information is subject to change and
+is intended to help developers understand the corresponding kernel code.
+
+What is FSP?
+============
+The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust
+(IROT). It is a dedicated security processor that boots from immutable ROM
+(Boot ROM) inside the GPU and is responsible for establishing the Chain of
+Trust before any other firmware is allowed to run.
+
+FSP runs independently of the host CPU and starts executing as soon as the
+GPU is powered on. By the time the nova-core driver is loaded, FSP has
+already completed its own secure boot and is ready to accept commands from
+the driver.
+
+Simplified boot flow (Hopper/Blackwell)
+=======================================
+Starting with Hopper, the boot flow is significantly simplified compared to
+earlier GPU generations like Ampere.
+
+On an **Ampere** GPU, the boot verification chain involves multiple Falcon
+engines and multiple ucode stages (see falcon.rst for details)::
+
+ Hardware BROM (SEC2)
+ -> HS Booter (SEC2)
+ -> LS GSP-RM (GSP)
+
+The driver must extract ucode from VBIOS, manage SEC2 and GSP, and
+orchestrate the Booter to load GSP-RM. This involves FWSEC-FRTS, devinit,
+and the Booter stages.
+
+On **Hopper/Blackwell** GPUs, FSP replaces this multi-stage process with a
+single message-driven interface::
+
+ FSP (hardware root of trust, boots from ROM)
+ -> FMC (Falcon Microcontroller, verified by FSP)
+ -> GSP-RM (verified and loaded by FMC)
+
+The driver only needs to:
+
+1. Wait for FSP to complete its own secure boot (polling a scratch register).
+2. Send a Chain of Trust (COT) message to FSP with the FMC firmware location,
+ cryptographic signatures, and GSP boot parameters.
+3. FSP authenticates the FMC firmware and boots it, FMC in turn loads GSP-RM.
+
+There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage. The
+entire secure boot is driven by a single FSP message exchange.
+
+Chain of Trust (COT) protocol
+=============================
+The Chain of Trust establishes a cryptographically enforced boot sequence,
+ensuring the GPU reaches a known, trusted state.
+
+The driver communicates with FSP using a message queue (Falcon MSGQ
+interface). Each message consists of an MCTP (Management Component Transport
+Protocol) transport header and an NVDM (NVIDIA Vendor Defined Message) header,
+followed by a protocol-specific payload.
+
+For Chain of Trust, the payload includes:
+
+- The system memory address of the FMC firmware image.
+- Cryptographic material: a SHA-384 hash, RSA-3K public key, and RSA-3K
+ signature extracted from the FMC ELF firmware.
+- FRTS (Firmware Runtime Services) region information (vidmem offset and size).
+- The system memory address of the GSP boot arguments structure.
+
+FSP verifies the signature against the provided public key and hash, and if
+verification succeeds, boots the FMC. The FMC then authenticates and launches
+GSP-RM.
+
+The message flow is::
+
+ nova-core FSP
+ | |
+ | 1. Poll scratch register |
+ | (wait for FSP boot complete) |
+ | |
+ | 2. COT message ------------> |
+ | (FMC addr, signatures, |
+ | boot params) |
+ | |
+ | |--- Verify FMC signature
+ | |--- Boot FMC
+ | |--- FMC loads GSP-RM
+ | |
+ | 3. COT response <------------ |
+ | (success/error) |
+ | |
+
+FSP message format
+==================
+All FSP messages share a common header format consisting of two 32-bit words:
+
+**MCTP header** (Management Component Transport Protocol):
+
+- Bit 31: SOM (Start of Message)
+- Bit 30: EOM (End of Message)
+- Bits 29:28: Packet sequence number
+- Bits 23:16: Source Endpoint ID
+
+**NVDM header** (NVIDIA Vendor Defined Message):
+
+- Bits 6:0: MCTP message type (0x7e = vendor-defined PCI)
+- Bits 23:8: PCI vendor ID (0x10de = NVIDIA)
+- Bits 31:24: NVDM type (0x14 = COT, 0x13 = PRC, 0x15 = FSP response)
+
+PRC (Product Reconfiguration Control) protocol
+===============================================
+PRC is an API system exposed through FSP's Management Partition that allows
+querying and modifying device configuration without firmware updates.
+
+Configuration parameters are called "knobs". Each knob has a unique object
+ID and controls a specific device behavior. Examples include vGPU mode, ECC
+enable, confidential computing mode, and NVLINK configuration.
+
+Each knob has two values:
+
+- **Active**: the currently effective value for this boot cycle.
+- **Persistent**: the value stored in InfoROM, applied on subsequent boots.
+
+The nova-core driver uses PRC to read the vGPU mode knob (object ID 0x29)
+during early boot, before firmware loading, to determine whether the GPU
+should operate in vGPU mode.
+
+The PRC message format follows the same MCTP/NVDM header structure as COT,
+with NVDM type 0x13. The payload contains:
+
+- A sub-command (e.g., 0x0c for read).
+- Flags indicating which value to read (bit 0 = persistent, bit 1 = active).
+- The knob object ID.
+
+The response includes the common FSP response header (with error status)
+followed by the knob's 16-bit state value.
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index e39cb3163581..1783513cbd05 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -30,5 +30,6 @@ vGPU manager VFIO driver and the nova-drm driver.
core/todo
core/vbios
core/devinit
+ core/fsp
core/fwsec
core/falcon
--
2.51.0
next prev parent reply other threads:[~2026-06-04 11:45 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-04 11:43 [PATCH 0/9] gpu: nova-core: boot GSP with vGPU enabled on Zhi Wang
2026-06-04 11:43 ` [PATCH 1/9] rust: pci: expose sriov_get_totalvfs() helper Zhi Wang
2026-06-05 14:08 ` Alexandre Courbot
2026-06-17 7:51 ` Zhi Wang
2026-06-04 11:43 ` [PATCH 2/9] gpu: nova-core: factor out common FSP message header Zhi Wang
2026-06-05 13:21 ` Alexandre Courbot
2026-06-04 11:43 ` [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller Zhi Wang
2026-06-05 13:25 ` Alexandre Courbot
2026-06-05 16:04 ` Zhi Wang
2026-06-09 6:07 ` Alexandre Courbot
2026-06-17 7:52 ` Zhi Wang
2026-06-04 11:43 ` [PATCH 4/9] gpu: nova-core: read vGPU mode from FSP via PRC protocol Zhi Wang
2026-06-16 8:35 ` Alexandre Courbot
2026-06-17 8:01 ` Zhi Wang
2026-06-04 11:43 ` Zhi Wang [this message]
2026-06-16 8:17 ` [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation Alexandre Courbot
2026-06-17 7:51 ` Zhi Wang
2026-06-17 13:21 ` Alexandre Courbot
2026-06-04 11:43 ` [PATCH 6/9] gpu: nova-core: consolidate GSP boot parameters into GspBootContext Zhi Wang
2026-06-16 14:13 ` Alexandre Courbot
2026-06-17 13:22 ` Alexandre Courbot
2026-06-04 11:43 ` [PATCH 7/9] gpu: nova-core: add vGPU preludes Zhi Wang
2026-06-17 3:08 ` Alexandre Courbot
2026-06-04 11:43 ` [PATCH 8/9] gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled Zhi Wang
2026-06-17 3:13 ` Alexandre Courbot
2026-06-04 11:43 ` [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when " Zhi Wang
2026-06-16 14:20 ` Alexandre Courbot
2026-06-17 3:09 ` Alexandre Courbot
2026-06-17 8:07 ` Zhi Wang
2026-06-17 12:02 ` Alexandre Courbot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260604114339.1565660-6-zhiw@nvidia.com \
--to=zhiw@nvidia.com \
--cc=a.hindborg@kernel.org \
--cc=acourbot@nvidia.com \
--cc=airlied@gmail.com \
--cc=alex.gaynor@gmail.com \
--cc=aliceryhl@google.com \
--cc=alkumar@nvidia.com \
--cc=aniketa@nvidia.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bjorn3_gh@protonmail.com \
--cc=boqun.feng@gmail.com \
--cc=cjia@nvidia.com \
--cc=dakr@kernel.org \
--cc=ecourtney@nvidia.com \
--cc=gary@garyguo.net \
--cc=jhubbard@nvidia.com \
--cc=joelagnelf@nvidia.com \
--cc=kjaju@nvidia.com \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lossin@kernel.org \
--cc=nova-gpu@lists.linux.dev \
--cc=ojeda@kernel.org \
--cc=simona@ffwll.ch \
--cc=smitra@nvidia.com \
--cc=targupta@nvidia.com \
--cc=tmgross@umich.edu \
--cc=zhiwang@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox