* [PATCH v5 01/15] rust: pci: declare IrqType and IrqTypes with impl_flags
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout() John Hubbard
` (13 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The kernel provides impl_flags! for declaring a bitmask type alongside
the enum of its individual flags, generating the bit operators and the
containment queries.
IrqTypes open-coded that pattern with a with() builder, so a caller
naming two interrupt types chained two calls onto IrqTypes::default().
Declare both types through impl_flags!, so the same set reads as
IrqType::Msi | IrqType::MsiX.
Suggested-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
rust/kernel/pci/irq.rs | 68 +++++++++++++-----------------------------
1 file changed, 21 insertions(+), 47 deletions(-)
diff --git a/rust/kernel/pci/irq.rs b/rust/kernel/pci/irq.rs
index 6741046ec1c0..f074aad7f1d8 100644
--- a/rust/kernel/pci/irq.rs
+++ b/rust/kernel/pci/irq.rs
@@ -13,27 +13,26 @@
};
use core::num::NonZero;
-/// IRQ type flags for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy)]
-pub enum IrqType {
- /// INTx interrupts.
- Intx,
- /// Message Signaled Interrupts (MSI).
- Msi,
- /// Extended Message Signaled Interrupts (MSI-X).
- MsiX,
-}
-
-impl IrqType {
- /// Convert to the corresponding kernel flags.
- const fn as_raw(self) -> u32 {
- match self {
- IrqType::Intx => bindings::PCI_IRQ_INTX,
- IrqType::Msi => bindings::PCI_IRQ_MSI,
- IrqType::MsiX => bindings::PCI_IRQ_MSIX,
- }
+crate::impl_flags!(
+ /// Set of IRQ types that can be used for PCI interrupt allocation.
+ #[derive(Debug, Clone, Copy, Default)]
+ pub struct IrqTypes(u32);
+
+ /// IRQ type flags for PCI interrupt allocation.
+ #[derive(Debug, Clone, Copy)]
+ pub enum IrqType {
+ /// INTx interrupts.
+ Intx = bindings::PCI_IRQ_INTX,
+
+ /// Message Signaled Interrupts (MSI).
+ Msi = bindings::PCI_IRQ_MSI,
+
+ /// Extended Message Signaled Interrupts (MSI-X).
+ MsiX = bindings::PCI_IRQ_MSIX,
}
+);
+impl IrqType {
/// Construct from raw value.
#[inline]
const fn from_raw(raw: u32) -> Self {
@@ -45,33 +44,10 @@ const fn from_raw(raw: u32) -> Self {
}
}
-/// Set of IRQ types that can be used for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy, Default)]
-pub struct IrqTypes(u32);
-
impl IrqTypes {
/// Create a set containing all IRQ types (MSI-X, MSI, and INTx).
pub const fn all() -> Self {
- Self(bindings::PCI_IRQ_ALL_TYPES)
- }
-
- /// Build a set of IRQ types.
- ///
- /// # Examples
- ///
- /// ```ignore
- /// // Create a set with only MSI and MSI-X (no INTx interrupts).
- /// let msi_only = IrqTypes::default()
- /// .with(IrqType::Msi)
- /// .with(IrqType::MsiX);
- /// ```
- pub const fn with(self, irq_type: IrqType) -> Self {
- Self(self.0 | irq_type.as_raw())
- }
-
- /// Get the raw flags value.
- const fn as_raw(self) -> u32 {
- self.0
+ Self(Self::all_bits())
}
}
@@ -203,9 +179,7 @@ impl Device<device::Bound> {
/// let vectors = dev.alloc_irq_vectors(1, 32, pci::IrqTypes::all())?;
///
/// // Allocate MSI or MSI-X only (no INTx interrupts).
- /// let msi_only = pci::IrqTypes::default()
- /// .with(pci::IrqType::Msi)
- /// .with(pci::IrqType::MsiX);
+ /// let msi_only = pci::IrqType::Msi | pci::IrqType::MsiX;
/// let vectors = dev.alloc_irq_vectors(4, 16, msi_only)?;
/// # Ok(())
/// # }
@@ -222,7 +196,7 @@ pub fn alloc_irq_vectors(
// - `pci_alloc_irq_vectors` internally validates all other parameters
// and returns error codes.
let ret = unsafe {
- bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
+ bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, u32::from(irq_types))
};
to_result(ret)?;
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
2026-09-30 3:41 ` [PATCH v5 01/15] rust: pci: declare IrqType and IrqTypes with impl_flags John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:59 ` sashiko-bot
2026-09-30 3:41 ` [PATCH v5 03/15] gpu: nova-core: add the GIN vector, leaf and subtree types John Hubbard
` (12 subsequent siblings)
14 siblings, 1 reply; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, Joel Fernandes, John Hubbard
From: Joel Fernandes <joelagnelf@nvidia.com>
A driver that runs an interrupt self-test during probe waits for the
handler to fire. wait_for_completion() has no timeout, so a broken
interrupt path stalls probe indefinitely. Add a timeout variant of
wait_for_completion().
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: return the remaining jiffies]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/rust/kernel/sync/completion.rs b/rust/kernel/sync/completion.rs
index 35ff049ff078..7e8b3c1c880e 100644
--- a/rust/kernel/sync/completion.rs
+++ b/rust/kernel/sync/completion.rs
@@ -6,7 +6,12 @@
//!
//! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+ bindings,
+ prelude::*,
+ time::Jiffies,
+ types::Opaque, //
+};
/// Synchronization primitive to signal when a certain task has been completed.
///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
// SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
unsafe { bindings::wait_for_completion(self.as_raw()) };
}
+
+ /// Wait for completion of a task, with a timeout.
+ ///
+ /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+ /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+ /// `timeout` elapsed first.
+ ///
+ /// See also [`Completion::complete_all`].
+ #[inline]
+ pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {
+ // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+ match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
+ 0 => None,
+ remaining => Some(remaining),
+ }
+ }
}
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()
2026-09-30 3:41 ` [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout() John Hubbard
@ 2026-09-30 3:59 ` sashiko-bot
0 siblings, 0 replies; 18+ messages in thread
From: sashiko-bot @ 2026-09-30 3:59 UTC (permalink / raw)
To: John Hubbard, Danilo Krummrich, Alexandre Courbot
Cc: Alex Gaynor, Zhi Wang, Boqun Feng, Danilo Krummrich,
David Airlie, Alistair Popple, Benno Lossin, LKML, Alice Ryhl,
Miguel Ojeda, Timur Tabi, Eliot Courtney, Trevor Gross,
Andreas Hindborg, nova-gpu, Björn Roy Baron,
Alexandre Courbot, Joel Fernandes, Simona Vetter, Gary Guo,
Bjorn Helgaas
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] rust: sync: wait_for_completion_timeout stack dump on large timeouts
--
commit f89eb4d12071b7e3c61766cd1bbbd5c2cfee1ea5
Author: Joel Fernandes <joelagnelf@nvidia.com>
rust: sync: completion: add wait_for_completion_timeout()
This commit adds wait_for_completion_timeout() to the Rust sync completion
primitives. It provides a way to wait for a completion with a specified
timeout in jiffies, preventing indefinite stalls in drivers.
> diff --git a/rust/kernel/sync/completion.rs b/rust/kernel/sync/completion.rs
> --- a/rust/kernel/sync/completion.rs
> +++ b/rust/kernel/sync/completion.rs
[ ... ]
> @@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
> // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
> unsafe { bindings::wait_for_completion(self.as_raw()) };
> }
> +
> + /// Wait for completion of a task, with a timeout.
[ ... ]
> + #[inline]
> + pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {
> + // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
> + match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
[Severity: Medium]
Does this code overflow the timeout parameter if large values are provided?
The timeout parameter is passed directly to the C API without bounds checking.
Because C implicitly casts it to a signed long downstream, values greater than
LONG_MAX can lead to an immediate unhandled timeout and a kernel stack dump.
For example, passing Jiffies::MAX causes the unsigned timeout to overflow into
a negative long inside the C wait_for_common() and schedule_timeout()
functions. This triggers a kernel stack dump (printk(KERN_ERR ...);
dump_stack();) and causes the function to return immediately without waiting,
violating the safe function's contract.
Could this be clamped to MAX_SCHEDULE_TIMEOUT to prevent unexpected behavior
from excessively large timeouts?
> + 0 => None,
> + remaining => Some(remaining),
> + }
> + }
> }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260930034148.590687-1-jhubbard@nvidia.com?part=2
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v5 03/15] gpu: nova-core: add the GIN vector, leaf and subtree types
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
2026-09-30 3:41 ` [PATCH v5 01/15] rust: pci: declare IrqType and IrqTypes with impl_flags John Hubbard
2026-09-30 3:41 ` [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout() John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 04/15] gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers John Hubbard
` (11 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt
controller. Each interrupt source has a GIN vector number, and the
controller latches a pending vector in a two-level tree: one bit of a
LEAF register, summarized two leaves at a time by one bit of the TOP
register. A vector's number fixes its position in that tree:
leaf = vector / 32
bit = vector % 32
subtree = leaf / 2
A tree implements either 8 or 16 leaves, depending on the GPU family.
The leaf count sets both the number of subtrees and the highest vector
that the tree carries.
Without distinct types, a vector, a leaf index, a set of vectors within
one leaf, a subtree and a set of subtrees are all plain integers. A
caller can pass one where another belongs, and a register field can
accept the wrong one.
Add a type for each of those, and for the leaf count. A vector converts
to its own leaf, bit and subtree. Its constructor rejects, at build
time, a number beyond the widest supported tree, and a validation
method rejects, at run time, a number beyond the leaves that the
current tree implements. A leaf count yields the set of subtrees that
it implements.
The module has no user yet.
Suggested-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/irq.rs | 12 +
drivers/gpu/nova-core/irq/interrupt_tree.rs | 238 ++++++++++++++++++++
drivers/gpu/nova-core/nova_core.rs | 2 +
3 files changed, 252 insertions(+)
create mode 100644 drivers/gpu/nova-core/irq.rs
create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
new file mode 100644
index 000000000000..f1323f633a03
--- /dev/null
+++ b/drivers/gpu/nova-core/irq.rs
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! GPU interrupt support.
+//!
+//! GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller. It latches
+//! every interrupt source in a two-level register tree and delivers the tree to the CPU as a
+//! message-signaled PCI interrupt.
+//!
+//! See `Documentation/gpu/nova/core/interrupts.rst`.
+
+mod interrupt_tree;
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
new file mode 100644
index 000000000000..206e6940ce77
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -0,0 +1,238 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Vector addressing in the GIN CPU interrupt tree.
+//!
+//! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
+//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
+//! keep the four from being confused with one another.
+//!
+//! See `Documentation/gpu/nova/core/interrupts.rst`.
+
+use kernel::{
+ num::Bounded,
+ prelude::*, //
+};
+
+use crate::num;
+
+/// Number of vectors one leaf register carries, one per bit.
+const VECTORS_PER_LEAF: u32 = u32::BITS;
+
+/// Number of leaves one subtree covers.
+const LEAVES_PER_SUBTREE: u32 = 2;
+
+/// Number of subtrees the widest supported tree implements.
+const MAX_NUM_SUBTREES: u32 = 8;
+
+/// Number of leaves the widest supported tree implements.
+const MAX_NUM_LEAVES: u32 = MAX_NUM_SUBTREES * LEAVES_PER_SUBTREE;
+
+/// Number of bits needed to address every vector in the widest supported tree.
+const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
+
+/// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
+/// lower half of the range.
+pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
+
+/// Number of leaves a tree implements.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+#[repr(u32)]
+pub(super) enum LeafCount {
+ /// Turing through Ada.
+ Eight = 8,
+
+ /// Hopper and later.
+ Sixteen = 16,
+}
+
+impl LeafCount {
+ pub(super) const fn into_u32(self) -> u32 {
+ // CAST: `LeafCount` is `repr(u32)`, so the cast is lossless.
+ self as u32
+ }
+
+ pub(super) const fn into_raw(self) -> usize {
+ num::u32_as_usize(self.into_u32())
+ }
+
+ /// Returns the number of subtrees a tree of this size implements.
+ pub(super) const fn subtree_count(self) -> u32 {
+ self.into_u32() / LEAVES_PER_SUBTREE
+ }
+
+ /// Returns the set of every subtree a tree of this size implements.
+ pub(super) const fn subtree_set(self) -> SubtreeSet {
+ SubtreeSet((1u32 << self.subtree_count()) - 1)
+ }
+
+ /// Returns the number of vectors a tree of this size carries.
+ pub(super) const fn vector_count(self) -> u32 {
+ self.into_u32() * VECTORS_PER_LEAF
+ }
+}
+
+// `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
+// agreement about the widest supported tree.
+static_assert!(1 << VECTOR_BITS == LeafCount::Sixteen.vector_count());
+
+/// Set of vectors within one leaf, one bit per vector.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct LeafMask(u32);
+
+impl LeafMask {
+ /// Returns the mask with every vector set.
+ pub(super) const fn all() -> Self {
+ Self(u32::MAX)
+ }
+
+ pub(super) const fn from_raw(raw: u32) -> Self {
+ Self(raw)
+ }
+
+ pub(super) const fn into_raw(self) -> u32 {
+ self.0
+ }
+
+ pub(super) const fn is_empty(self) -> bool {
+ self.0 == 0
+ }
+
+ /// Returns whether every vector in `other` is also in this mask.
+ pub(super) const fn contains(self, other: Self) -> bool {
+ self.0 & other.0 == other.0
+ }
+}
+
+impl From<Bounded<u32, 32>> for LeafMask {
+ fn from(vectors: Bounded<u32, 32>) -> Self {
+ Self(vectors.get())
+ }
+}
+
+impl From<LeafMask> for Bounded<u32, 32> {
+ fn from(vectors: LeafMask) -> Self {
+ vectors.0.into()
+ }
+}
+
+/// One subtree, held as the `TOP` bit that covers it.
+///
+/// # Invariants
+///
+/// Exactly one bit is set.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct Subtree(u32);
+
+impl Subtree {
+ /// Returns the subtree at index `idx`.
+ const fn new(idx: u32) -> Self {
+ // INVARIANT: shifting `1` left leaves exactly one bit set.
+ Self(1 << idx)
+ }
+
+ /// Returns this subtree's index within the tree.
+ pub(super) const fn index(self) -> u32 {
+ self.0.trailing_zeros()
+ }
+
+ pub(super) const fn into_raw(self) -> u32 {
+ self.0
+ }
+}
+
+/// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct SubtreeSet(u32);
+
+impl SubtreeSet {
+ pub(super) const fn contains(self, subtree: Subtree) -> bool {
+ self.0 & subtree.into_raw() != 0
+ }
+
+ pub(super) const fn is_empty(self) -> bool {
+ self.0 == 0
+ }
+
+ pub(super) const fn intersection(self, other: Self) -> Self {
+ Self(self.0 & other.0)
+ }
+
+ /// Returns one more than the highest index in this set, or `0` for an empty set. An MSI-X
+ /// allocation that covers the set needs this many entries.
+ pub(super) const fn span(self) -> u32 {
+ u32::BITS - self.0.leading_zeros()
+ }
+
+ /// Returns the subtrees of this set, lowest index first.
+ #[expect(dead_code)]
+ pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
+ (0..u32::BITS)
+ .map(Subtree::new)
+ .filter(move |subtree| self.contains(*subtree))
+ }
+}
+
+impl From<Subtree> for SubtreeSet {
+ fn from(subtree: Subtree) -> Self {
+ Self(subtree.into_raw())
+ }
+}
+
+impl From<Bounded<u32, 32>> for SubtreeSet {
+ fn from(subtrees: Bounded<u32, 32>) -> Self {
+ Self(subtrees.get())
+ }
+}
+
+impl From<SubtreeSet> for Bounded<u32, 32> {
+ fn from(subtrees: SubtreeSet) -> Self {
+ subtrees.0.into()
+ }
+}
+
+/// A GIN interrupt vector, bounded to the widest tree that any supported chipset implements.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct GinVector(Bounded<u32, VECTOR_BITS>);
+
+impl GinVector {
+ /// Returns vector number `VECTOR`.
+ ///
+ /// Fails to compile if `VECTOR` is beyond the widest supported tree.
+ pub(super) const fn new<const VECTOR: u32>() -> Self {
+ Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
+ }
+
+ pub(super) const fn into_raw(self) -> u32 {
+ self.0.get()
+ }
+
+ /// Returns this vector's leaf.
+ pub(super) fn leaf_index(self) -> LeafIndex {
+ // CALC: `self.0 / VECTORS_PER_LEAF`.
+ self.0.shr::<{ VECTORS_PER_LEAF.ilog2() }, _>().cast()
+ }
+
+ /// Returns this vector's bit within its leaf.
+ pub(super) const fn leaf_mask(self) -> LeafMask {
+ LeafMask(1 << (self.0.get() % VECTORS_PER_LEAF))
+ }
+
+ /// Returns this vector's subtree.
+ pub(super) const fn subtree(self) -> Subtree {
+ Subtree::new(self.0.get() / (VECTORS_PER_LEAF * LEAVES_PER_SUBTREE))
+ }
+
+ /// Checks that a tree with `leaves` leaves implements this vector.
+ ///
+ /// # Errors
+ ///
+ /// `EINVAL` if it does not.
+ pub(super) const fn validate(self, leaves: LeafCount) -> Result {
+ if self.0.get() >= leaves.vector_count() {
+ Err(EINVAL)
+ } else {
+ Ok(())
+ }
+ }
+}
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 4f8ce4e4c187..8202c4982efa 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -18,6 +18,8 @@
mod fsp;
mod gpu;
mod gsp;
+#[expect(dead_code)]
+mod irq;
mod mctp;
mod mm;
#[macro_use]
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 04/15] gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (2 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 03/15] gpu: nova-core: add the GIN vector, leaf and subtree types John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 05/15] gpu: nova-core: add the per-architecture GIN CPU interrupt HAL John Hubbard
` (10 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard, Will Pierce
GIN is the GPU's interrupt controller. It latches each interrupt source
in a two-level tree of LEAF registers summarized by TOP, and raises the
PCI interrupt when an enabled vector in an enabled subtree becomes
pending. A message-signaled interrupt is delivered once per edge, and
pre-Hopper MSI rearms delivery by writing the end-of-interrupt register
in the BAR0 mirror of PCI configuration space.
Add the CPU tree registers that receiving GSP interrupts and running the
software-triggered self-test need: the leaf pending and enable arrays,
the TOP enables, and the leaf trigger. Add the end-of-interrupt
register, NV_XVE_CYA_2, alongside them.
Use the NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_* names on every chipset. The
pre-Hopper headers call the same tree NV_CTRL, but each function reaches
its own tree through this aperture on both families. Declare the leaf
arrays at 16 entries, the widest tree that any supported chipset
implements.
Leave the read-only TOP summary undeclared. A vector that latched while
disabled does not appear in TOP, so nova-core never descends from it and
reads every implemented leaf instead.
Declare each leaf field as a set of vectors within one leaf and each TOP
field as a set of subtrees, so that a set of subtrees cannot be written
to a leaf register, nor the reverse. The trigger register's vector field
is 12 bits wide, wider than any GIN vector, so a vector converts into it
infallibly.
Assisted-by: LLM
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/irq.rs | 1 +
drivers/gpu/nova-core/irq/interrupt_tree.rs | 14 ++++
drivers/gpu/nova-core/irq/regs.rs | 88 +++++++++++++++++++++
3 files changed, 103 insertions(+)
create mode 100644 drivers/gpu/nova-core/irq/regs.rs
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index f1323f633a03..1ec0bb055d3b 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -10,3 +10,4 @@
//! See `Documentation/gpu/nova/core/interrupts.rst`.
mod interrupt_tree;
+mod regs;
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 206e6940ce77..279a43514b1f 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -16,6 +16,8 @@
use crate::num;
+use super::regs::*;
+
/// Number of vectors one leaf register carries, one per bit.
const VECTORS_PER_LEAF: u32 = u32::BITS;
@@ -31,6 +33,12 @@
/// Number of bits needed to address every vector in the widest supported tree.
const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
+/// Width of the vector field in the leaf trigger register.
+const TRIGGER_VECTOR_BITS: u32 = {
+ let range = NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::VECTOR_RANGE;
+ num::u8_as_u32(*range.end() - *range.start() + 1)
+};
+
/// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
/// lower half of the range.
pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
@@ -236,3 +244,9 @@ pub(super) const fn validate(self, leaves: LeafCount) -> Result {
}
}
}
+
+impl From<GinVector> for Bounded<u32, TRIGGER_VECTOR_BITS> {
+ fn from(vector: GinVector) -> Self {
+ vector.0.extend()
+ }
+}
diff --git a/drivers/gpu/nova-core/irq/regs.rs b/drivers/gpu/nova-core/irq/regs.rs
new file mode 100644
index 000000000000..4ddd01bcbb23
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/regs.rs
@@ -0,0 +1,88 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use kernel::io::register;
+
+use crate::driver::NovaRegisters;
+
+use super::interrupt_tree::{
+ LeafMask,
+ SubtreeSet, //
+};
+
+// The GIN CPU interrupt tree, reached through the `NV_VIRTUAL_FUNCTION_PRIV` aperture. See
+// "Register naming" in `Documentation/gpu/nova/core/interrupts.rst`. The leaf arrays are declared
+// with 16 entries, the widest tree that any supported chipset implements.
+
+register! {
+ base: NovaRegisters;
+
+ /// Pending bits of one leaf, one per vector.
+ ///
+ /// Vector `v` is bit `v % 32` of leaf `v / 32`. The bit is set when the vector's source
+ /// drives it, whether or not the vector is enabled. Writing a `1` clears the bit, and a `0`
+ /// leaves it as it was.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF(u32)[16] @ 0x00b81000 {
+ /// The vectors pending in this leaf.
+ 31:0 vectors => LeafMask;
+ }
+
+ /// Enables vectors of one leaf.
+ ///
+ /// A `1` enables the matching vector, and a `0` leaves it as it was.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET(u32)[16] @ 0x00b81200 {
+ /// Vectors to enable.
+ 31:0 vectors => LeafMask;
+ }
+
+ /// Disables vectors of one leaf.
+ ///
+ /// A `1` disables the matching vector, and a `0` leaves it as it was. A disabled vector still
+ /// latches in `LEAF`, and `TOP` does not show it.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR(u32)[16] @ 0x00b81400 {
+ /// Vectors to disable.
+ 31:0 vectors => LeafMask;
+ }
+
+ /// Enables subtrees.
+ ///
+ /// Bit `N` covers subtree `N`, which is leaves `2N` and `2N + 1`. A `1` enables the matching
+ /// subtree, and a `0` leaves it as it was.
+ ///
+ /// The hardware headers declare a one-element array, so nova-core declares a scalar.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET(u32) @ 0x00b81608 {
+ /// Subtrees to enable.
+ 31:0 subtrees => SubtreeSet;
+ }
+
+ /// Disables subtrees, with the bit layout of `TOP_EN_SET`.
+ ///
+ /// A `1` disables the matching subtree, and a `0` leaves it as it was. A disabled subtree
+ /// delivers nothing, and `TOP` still reports it.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR(u32) @ 0x00b81610 {
+ /// Subtrees to disable.
+ 31:0 subtrees => SubtreeSet;
+ }
+
+ /// Latches a vector from software. Write-only.
+ ///
+ /// The write sets the pending bit of the vector in its `LEAF` register, as if the vector's
+ /// source had raised it, and the vector reaches the CPU under the same enables. Implemented on
+ /// every supported chipset.
+ pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER(u32) @ 0x00b81640 {
+ /// Vector to latch.
+ 11:0 vector;
+ }
+}
+
+// PCI configuration-space mirror in BAR0.
+
+register! {
+ base: NovaRegisters;
+
+ /// MSI end-of-interrupt register. Writing any value rearms MSI delivery.
+ ///
+ /// Only pre-Hopper MSI rearms through this register. See "Rearming PCI interrupt delivery"
+ /// in `Documentation/gpu/nova/core/interrupts.rst`.
+ pub(super) NV_XVE_CYA_2(u32) @ 0x00088704 {}
+}
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 05/15] gpu: nova-core: add the per-architecture GIN CPU interrupt HAL
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (3 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 04/15] gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 06/15] gpu: nova-core: add the GIN interrupt tree and allocate its vectors John Hubbard
` (9 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard, Will Pierce
The GIN CPU interrupt tree differs by GPU family in two ways:
* The size of the tree. Turing, Ampere and Ada implement 8 leaves.
Hopper and later families implement 16.
* The write that rearms delivery. A message-signaled interrupt is
delivered once per edge, and the PCI side delivers nothing more until
the CPU rearms it. Before Hopper, MSI rearms by writing the
end-of-interrupt register in the BAR0 mirror of PCI configuration
space. On Hopper and later, MSI rearms by clearing and then setting
the TOP enables of every serviced subtree, which produces a new edge.
MSI-X rearms the same way on every family, but for the handler's own
subtree only, since each subtree has its own table entry.
Add an interrupt HAL that provides the leaf count and performs the rearm
for each family. Name the interrupt type with two variants, MSI and
MSI-X, since nova-core never allocates the level-triggered INTx that the
kernel's PCI interrupt type also names.
The HAL has no user yet.
Assisted-by: LLM
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/irq.rs | 13 +++++
drivers/gpu/nova-core/irq/hal.rs | 67 ++++++++++++++++++++++++++
drivers/gpu/nova-core/irq/hal/gh100.rs | 40 +++++++++++++++
drivers/gpu/nova-core/irq/hal/tu102.rs | 43 +++++++++++++++++
4 files changed, 163 insertions(+)
create mode 100644 drivers/gpu/nova-core/irq/hal.rs
create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 1ec0bb055d3b..62c242b71dc8 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -9,5 +9,18 @@
//!
//! See `Documentation/gpu/nova/core/interrupts.rst`.
+mod hal;
mod interrupt_tree;
mod regs;
+
+/// The message-signaled interrupt type that Linux granted.
+///
+/// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum MsiType {
+ /// A single message, which every subtree raises.
+ Msi,
+
+ /// One table entry per subtree.
+ MsiX,
+}
diff --git a/drivers/gpu/nova-core/irq/hal.rs b/drivers/gpu/nova-core/irq/hal.rs
new file mode 100644
index 000000000000..387695cca37b
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal.rs
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Per-architecture differences in the GIN CPU interrupt tree.
+//!
+//! See "Per-architecture differences" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+mod gh100;
+mod tu102;
+
+use kernel::{
+ io::Io,
+ prelude::*, //
+};
+
+use crate::{
+ driver::Bar0,
+ gpu::{
+ Architecture,
+ Chipset, //
+ }, //
+};
+
+use super::{
+ interrupt_tree::{
+ LeafCount,
+ Subtree,
+ SubtreeSet, //
+ },
+ regs::*,
+ MsiType, //
+};
+
+/// Clears and then sets the `TOP` enables of `subtrees`, which produces a new delivery edge.
+fn cycle_top_enables(bar: Bar0<'_>, subtrees: SubtreeSet) {
+ bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(subtrees));
+ bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(subtrees));
+}
+
+/// The leaf count and the rearm of the GIN CPU tree, which differ by GPU family.
+pub(super) trait CpuInterruptHal: Send + Sync {
+ /// Returns the number of leaves the tree implements.
+ fn leaf_count(&self) -> LeafCount;
+
+ /// Rearms PCI interrupt delivery after a handler serviced `subtree`.
+ ///
+ /// `serviced` is every subtree that nova-core services, for the Hopper-plus MSI form, which
+ /// cycles the `TOP` enables of them all. See "Rearming PCI interrupt delivery" in
+ /// `Documentation/gpu/nova/core/interrupts.rst`.
+ fn rearm_pci_irq(
+ &self,
+ bar: Bar0<'_>,
+ msi_type: MsiType,
+ serviced: SubtreeSet,
+ subtree: Subtree,
+ );
+}
+
+/// Returns the [`CpuInterruptHal`] for `chipset`'s architecture.
+pub(super) fn cpu_interrupt_hal(chipset: Chipset) -> &'static dyn CpuInterruptHal {
+ match chipset.arch() {
+ Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
+ Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
+ gh100::GH100_HAL
+ }
+ }
+}
diff --git a/drivers/gpu/nova-core/irq/hal/gh100.rs b/drivers/gpu/nova-core/irq/hal/gh100.rs
new file mode 100644
index 000000000000..6e307c8d120b
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal/gh100.rs
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use crate::driver::Bar0;
+
+use super::{
+ cycle_top_enables,
+ CpuInterruptHal,
+ LeafCount,
+ MsiType,
+ Subtree,
+ SubtreeSet, //
+};
+
+/// The CPU interrupt tree properties of Hopper and Blackwell.
+struct Gh100;
+
+impl CpuInterruptHal for Gh100 {
+ fn leaf_count(&self) -> LeafCount {
+ LeafCount::Sixteen
+ }
+
+ fn rearm_pci_irq(
+ &self,
+ bar: Bar0<'_>,
+ msi_type: MsiType,
+ serviced: SubtreeSet,
+ subtree: Subtree,
+ ) {
+ let subtrees = match msi_type {
+ MsiType::Msi => serviced,
+ MsiType::MsiX => subtree.into(),
+ };
+
+ cycle_top_enables(bar, subtrees);
+ }
+}
+
+const GH100: Gh100 = Gh100;
+pub(super) const GH100_HAL: &dyn CpuInterruptHal = &GH100;
diff --git a/drivers/gpu/nova-core/irq/hal/tu102.rs b/drivers/gpu/nova-core/irq/hal/tu102.rs
new file mode 100644
index 000000000000..7e3aea946d05
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal/tu102.rs
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use kernel::io::Io;
+
+use crate::{
+ driver::Bar0,
+ irq::regs::NV_XVE_CYA_2, //
+};
+
+use super::{
+ cycle_top_enables,
+ CpuInterruptHal,
+ LeafCount,
+ MsiType,
+ Subtree,
+ SubtreeSet, //
+};
+
+/// The CPU interrupt tree properties of Turing, Ampere, and Ada.
+struct Tu102;
+
+impl CpuInterruptHal for Tu102 {
+ fn leaf_count(&self) -> LeafCount {
+ LeafCount::Eight
+ }
+
+ fn rearm_pci_irq(
+ &self,
+ bar: Bar0<'_>,
+ msi_type: MsiType,
+ _serviced: SubtreeSet,
+ subtree: Subtree,
+ ) {
+ match msi_type {
+ MsiType::Msi => bar.write(NV_XVE_CYA_2, 0u32.into()),
+ MsiType::MsiX => cycle_top_enables(bar, subtree.into()),
+ }
+ }
+}
+
+const TU102: Tu102 = Tu102;
+pub(super) const TU102_HAL: &dyn CpuInterruptHal = &TU102;
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 06/15] gpu: nova-core: add the GIN interrupt tree and allocate its vectors
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (4 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 05/15] gpu: nova-core: add the per-architecture GIN CPU interrupt HAL John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor John Hubbard
` (8 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, Joel Fernandes, John Hubbard, Will Pierce
From: Joel Fernandes <joelagnelf@nvidia.com>
Servicing a GIN leaf has a required order: read its pending bits, then
clear them. Clearing a leaf first discards every vector latched in it,
and the hardware keeps no record of what was discarded. The interrupt
never arrives, and no register shows that it was ever pending.
Every subtree enabled at TOP also needs an allocated PCI vector with a
handler registered on it. Under MSI-X each subtree has its own table
entry. Linux masks every entry until a driver requests its IRQ, and a
masked entry sends no message. An enabled subtree whose entry was never
requested raises interrupts that never reach a handler, while the leaf
and TOP registers show them pending and enabled. Under MSI the whole
tree raises a single message, so one entry serves every subtree.
Add the CPU interrupt tree of one PCIe function. The tree allocates the
PCI vectors for the subtrees that it services and hands out the request
for each, so that the chipset is given once and the vectors live as
long as the tree. Reading a leaf yields the handle that clears it, so
that the wrong order does not compile. Building a tree fails if it
names a subtree that the GPU does not implement. A reset disables every
vector, clears every pending bit, rearms delivery, and leaves the
serviced subtrees disabled at TOP, so that a handler registered
afterwards receives no interrupt left over from boot.
Request MSI-X entries 0 through the highest serviced subtree, since an
MSI-X allocation cannot be sparse, and fall back to a single MSI
message, never to INTx.
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: reworked on top of the GIN vector types: a leaf read yields
the handle that clears it, enables are guarded, the leaf count and the
rearm come from the interrupt HAL, the drain reads every implemented
leaf rather than descending from TOP, the tree owns its PCI vectors,
and a reset returns the tree to a known state before a handler is
registered]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/irq.rs | 2 +-
drivers/gpu/nova-core/irq/interrupt_tree.rs | 298 +++++++++++++++++++-
2 files changed, 293 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 62c242b71dc8..b2cfe73af114 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -10,7 +10,7 @@
//! See `Documentation/gpu/nova/core/interrupts.rst`.
mod hal;
-mod interrupt_tree;
+pub(crate) mod interrupt_tree;
mod regs;
/// The message-signaled interrupt type that Linux granted.
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 279a43514b1f..6e97828bbd42 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -1,22 +1,47 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-//! Vector addressing in the GIN CPU interrupt tree.
+//! The GIN CPU interrupt tree for one PCIe function.
//!
//! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
//! keep the four from being confused with one another.
//!
+//! Servicing a leaf requires reading its pending bits before clearing them. Only
+//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] clears a leaf,
+//! so that the wrong order does not compile. This module does not serialize access to the tree.
+//!
//! See `Documentation/gpu/nova/core/interrupts.rst`.
use kernel::{
+ device::Bound,
+ io::{
+ register::Array,
+ Io, //
+ },
+ irq,
num::Bounded,
+ pci::{
+ self,
+ IrqType, //
+ },
prelude::*, //
};
-use crate::num;
+use crate::{
+ driver::Bar0,
+ gpu::Chipset,
+ num, //
+};
-use super::regs::*;
+use super::{
+ hal::{
+ cpu_interrupt_hal,
+ CpuInterruptHal, //
+ },
+ regs::*,
+ MsiType, //
+};
/// Number of vectors one leaf register carries, one per bit.
const VECTORS_PER_LEAF: u32 = u32::BITS;
@@ -78,6 +103,11 @@ pub(super) const fn subtree_set(self) -> SubtreeSet {
pub(super) const fn vector_count(self) -> u32 {
self.into_u32() * VECTORS_PER_LEAF
}
+
+ /// Returns every leaf a tree of this size implements.
+ pub(super) fn iter(self) -> impl Iterator<Item = LeafIndex> {
+ (0..self.into_raw()).filter_map(LeafIndex::try_new)
+ }
}
// `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
@@ -130,7 +160,7 @@ fn from(vectors: LeafMask) -> Self {
///
/// Exactly one bit is set.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct Subtree(u32);
+pub(crate) struct Subtree(u32);
impl Subtree {
/// Returns the subtree at index `idx`.
@@ -151,7 +181,7 @@ pub(super) const fn into_raw(self) -> u32 {
/// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct SubtreeSet(u32);
+pub(crate) struct SubtreeSet(u32);
impl SubtreeSet {
pub(super) const fn contains(self, subtree: Subtree) -> bool {
@@ -173,7 +203,6 @@ pub(super) const fn span(self) -> u32 {
}
/// Returns the subtrees of this set, lowest index first.
- #[expect(dead_code)]
pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
(0..u32::BITS)
.map(Subtree::new)
@@ -250,3 +279,260 @@ fn from(vector: GinVector) -> Self {
vector.0.extend()
}
}
+
+/// Disables `vectors` in `leaf`.
+fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+ bar.write(
+ Array::at(*leaf),
+ NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
+ );
+}
+
+/// Disables the subtrees in `serviced` at `TOP`.
+fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
+ bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
+}
+
+/// The CPU tree of one PCIe function, the subtrees that nova-core services, and the PCI vectors
+/// that deliver them.
+///
+/// A subtree may be enabled at `TOP` only once a handler is registered on the PCI vector that
+/// delivers it. See "The serviced-subtree invariant" in
+/// `Documentation/gpu/nova/core/interrupts.rst`.
+pub(crate) struct Tree<'a> {
+ bar: Bar0<'a>,
+ hal: &'static dyn CpuInterruptHal,
+ serviced: SubtreeSet,
+ /// The interrupt type that Linux granted for `vectors`.
+ pub(super) msi_type: MsiType,
+ vectors: pci::IrqVectorRegistration<'a>,
+}
+
+impl<'a> Tree<'a> {
+ /// Creates the tree of `chipset` and allocates the PCI vectors for the subtrees in `serviced`.
+ ///
+ /// Requests MSI-X entries `0` through the highest subtree in `serviced`, since an allocation
+ /// cannot be sparse, and falls back to a single MSI message for the whole tree.
+ ///
+ /// # Errors
+ ///
+ /// `EINVAL` if `serviced` is empty or names a subtree that `chipset` does not implement.
+ /// Otherwise, when neither interrupt type could be allocated, the error from the MSI request.
+ pub(crate) fn new(
+ pdev: &'a pci::Device<Bound>,
+ bar: Bar0<'a>,
+ chipset: Chipset,
+ serviced: SubtreeSet,
+ ) -> Result<Self> {
+ let hal = cpu_interrupt_hal(chipset);
+ let num_leaves = hal.leaf_count();
+
+ if serviced.is_empty() || serviced.intersection(num_leaves.subtree_set()) != serviced {
+ return Err(EINVAL);
+ }
+
+ let entries = serviced.span();
+ let (vectors, msi_type) = pdev
+ .alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
+ .map(|vectors| (vectors, MsiType::MsiX))
+ // Every subtree raises the one MSI message, so one MSI vector serves the whole tree.
+ // See "Delivery over PCI" in `Documentation/gpu/nova/core/interrupts.rst`.
+ .or_else(|_| {
+ pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
+ .map(|vectors| (vectors, MsiType::Msi))
+ })?;
+
+ Ok(Self {
+ bar,
+ hal,
+ serviced,
+ msi_type,
+ vectors,
+ })
+ }
+
+ /// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
+ ///
+ /// # Errors
+ ///
+ /// `EINVAL` if `subtree` is not one of the serviced subtrees.
+ pub(super) fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
+ if !self.serviced.contains(subtree) {
+ return Err(EINVAL);
+ }
+
+ let entry = match self.msi_type {
+ MsiType::MsiX => num::u32_as_usize(subtree.index()),
+ MsiType::Msi => 0,
+ };
+
+ self.vectors.index(entry).map(Into::into)
+ }
+
+ /// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree that
+ /// the calling handler serves.
+ ///
+ /// A handler must call this before returning, or it receives no further interrupts.
+ pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
+ self.hal
+ .rearm_pci_irq(self.bar, self.msi_type, self.serviced, subtree);
+ }
+
+ /// Enables the serviced subtrees at `TOP`.
+ ///
+ /// Each of them must have a handler registered on its PCI vector.
+ pub(super) fn enable_top(&self) {
+ self.bar.write_reg(
+ NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
+ );
+ }
+
+ /// Disables the serviced subtrees at `TOP`.
+ pub(super) fn disable_top(&self) {
+ clear_top_enables(self.bar, self.serviced);
+ }
+
+ /// Enables the serviced subtrees at `TOP` until the returned guard drops.
+ pub(crate) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
+ self.enable_top();
+
+ TopEnableGuard {
+ bar: self.bar,
+ serviced: self.serviced,
+ }
+ }
+
+ /// Enables `vectors` in `leaf`.
+ pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+ self.bar.write(
+ Array::at(*leaf),
+ NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
+ );
+ }
+
+ /// Disables `vectors` in `leaf`.
+ pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+ clear_leaf_enables(self.bar, leaf, vectors);
+ }
+
+ /// Enables `vectors` in `leaf` until the returned guard drops.
+ pub(super) fn enable_leaf_guarded(
+ &self,
+ leaf: LeafIndex,
+ vectors: LeafMask,
+ ) -> LeafEnableGuard<'a> {
+ self.enable_leaf(leaf, vectors);
+
+ LeafEnableGuard {
+ bar: self.bar,
+ leaf,
+ vectors,
+ }
+ }
+
+ /// Reads the pending bits of `leaf`, and returns the handle that clears them.
+ pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
+ let pending = self
+ .bar
+ .read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
+ .vectors();
+
+ LeafPending {
+ bar: self.bar,
+ leaf,
+ pending,
+ }
+ }
+
+ /// Disables every vector in every implemented leaf, including the subtrees that nova-core does
+ /// not service. Call this only during probe.
+ pub(super) fn disable_all_leaves(&self) {
+ for leaf in self.hal.leaf_count().iter() {
+ self.disable_leaf(leaf, LeafMask::all());
+ }
+ }
+
+ /// Clears every pending bit in every implemented leaf, including the subtrees that nova-core
+ /// does not service.
+ ///
+ /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with no
+ /// interrupt handler registered.
+ pub(super) fn drain(&self) {
+ self.disable_top();
+
+ // A vector that latched while disabled does not show in `TOP`, so read every leaf rather
+ // than descending from it.
+ for leaf in self.hal.leaf_count().iter() {
+ self.read_pending(leaf).clear();
+ }
+ }
+
+ /// Disables every vector, clears every pending bit, and rearms PCI interrupt delivery.
+ ///
+ /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with
+ /// no interrupt handler registered.
+ pub(super) fn reset(&self) {
+ self.disable_all_leaves();
+ self.drain();
+ for subtree in self.serviced.iter() {
+ self.rearm_pci_irq(subtree);
+ }
+
+ // A `TOP`-enable rearm leaves the subtrees that it cycles enabled, and a subtree may be
+ // enabled only once a handler is registered on its vector.
+ self.disable_top();
+ }
+}
+
+/// The pending bits of one leaf as they were read, and the handle that clears them.
+pub(super) struct LeafPending<'a> {
+ bar: Bar0<'a>,
+ leaf: LeafIndex,
+ pending: LeafMask,
+}
+
+impl LeafPending<'_> {
+ pub(super) fn vectors(&self) -> LeafMask {
+ self.pending
+ }
+
+ /// Clears the vectors that were pending at the read. A vector that latched since stays pending.
+ pub(super) fn clear(&self) {
+ self.clear_vectors(self.pending);
+ }
+
+ /// Clears `vectors` and no other bit.
+ pub(super) fn clear_vectors(&self, vectors: LeafMask) {
+ if !vectors.is_empty() {
+ self.bar.write(
+ Array::at(*self.leaf),
+ NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
+ );
+ }
+ }
+}
+
+/// Disables a set of vectors in one leaf when dropped.
+pub(super) struct LeafEnableGuard<'a> {
+ bar: Bar0<'a>,
+ leaf: LeafIndex,
+ vectors: LeafMask,
+}
+
+impl Drop for LeafEnableGuard<'_> {
+ fn drop(&mut self) {
+ clear_leaf_enables(self.bar, self.leaf, self.vectors);
+ }
+}
+
+/// Disables the serviced subtrees at `TOP` when dropped.
+pub(crate) struct TopEnableGuard<'a> {
+ bar: Bar0<'a>,
+ serviced: SubtreeSet,
+}
+
+impl Drop for TopEnableGuard<'_> {
+ fn drop(&mut self) {
+ clear_top_enables(self.bar, self.serviced);
+ }
+}
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (5 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 06/15] gpu: nova-core: add the GIN interrupt tree and allocate its vectors John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 12:51 ` Danilo Krummrich
2026-09-30 3:41 ` [PATCH v5 08/15] gpu: nova-core: add an interrupt delivery self-test John Hubbard
` (7 subsequent siblings)
14 siblings, 1 reply; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The GPU boots its own firmware, GFW, out of reset, and the driver must
not program the GPU until GFW reports completion.
nova-core waited for GFW inside the Gpu constructor, which also boots
the GSP. Code that has to run after GFW and before GSP boot, such as a
probe-time hardware self-test, had nowhere to go.
Move the wait into probe, ahead of the Gpu constructor, and read the
chipset there from a Spec that probe builds itself. Leave the DMA mask
in the constructor, since it programs the host rather than the GPU.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/driver.rs | 14 +++++++++++++-
drivers/gpu/nova-core/gpu.rs | 26 +++++++++++++++++++-------
2 files changed, 32 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index 291a047e4d86..fc321c6a10b0 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -24,7 +24,11 @@
use crate::{
api::NovaCoreApi,
- gpu::Gpu, //
+ gpu::{
+ self,
+ Gpu,
+ Spec, //
+ }, //
};
/// Counter for generating unique auxiliary device IDs.
@@ -113,6 +117,14 @@ fn probe<'bound>(
pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
},
+ _: {
+ let spec = Spec::new(pdev.as_ref(), bar)?;
+
+ // We must wait for GFW_BOOT completion before doing any significant setup on
+ // the GPU.
+ gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+ },
+
// TODO: Use self-referential pin-init syntax once available.
gpu <- Gpu::new(
pdev,
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index fb6f8a86a503..65715f906030 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -251,7 +251,7 @@ pub struct Spec {
}
impl Spec {
- fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
+ pub(crate) fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
// Some brief notes about boot0 and boot42, in chronological order:
//
// NV04 through NV50:
@@ -397,10 +397,8 @@ pub(crate) fn new<'a>(
dev_info!(dev,"NVIDIA ({})\n", spec);
})?,
- // We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
_: {
- let hal = hal::gpu_hal(spec.chipset);
- let dma_mask = hal.dma_mask();
+ let dma_mask = hal::gpu_hal(spec.chipset).dma_mask();
// SAFETY: `Gpu` owns all DMA allocations for this device, and we are
// still constructing it, so no concurrent DMA allocations can exist.
@@ -412,9 +410,6 @@ pub(crate) fn new<'a>(
// SAFETY: `Gpu` owns all DMA allocations for this device, and we are
// still constructing it, so no concurrent DMA allocations can exist.
unsafe { pdev.dma_set_max_seg_size(u32::MAX) };
-
- hal.wait_gfw_boot_completion(bar)
- .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))?;
},
// Initialize this early because `gsp_resources` depends on it.
@@ -533,6 +528,23 @@ pub(crate) fn run_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bou
}
}
+/// Waits for GFW, the GPU's boot firmware, to report completion.
+///
+/// The driver must not program the GPU before then.
+///
+/// # Errors
+///
+/// `ETIMEDOUT` if GFW does not report completion in time.
+pub(crate) fn wait_gfw_boot_completion(
+ dev: &device::Device<device::Bound>,
+ bar: Bar0<'_>,
+ chipset: Chipset,
+) -> Result {
+ hal::gpu_hal(chipset)
+ .wait_gfw_boot_completion(bar)
+ .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))
+}
+
/// Reads the boot0 register and returns its raw value.
pub(crate) fn boot_0_raw(bar: Bar0<'_>) -> u32 {
bar.read(regs::NV_PMC_BOOT_0).into_raw()
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
2026-09-30 3:41 ` [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor John Hubbard
@ 2026-09-30 12:51 ` Danilo Krummrich
0 siblings, 0 replies; 18+ messages in thread
From: Danilo Krummrich @ 2026-09-30 12:51 UTC (permalink / raw)
To: John Hubbard
Cc: Alexandre Courbot, Timur Tabi, Alistair Popple, Eliot Courtney,
Zhi Wang, David Airlie, Simona Vetter, Bjorn Helgaas,
Miguel Ojeda, Alex Gaynor, Boqun Feng, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, nova-gpu, LKML
On Wed Sep 30, 2026 at 5:41 AM CEST, John Hubbard wrote:
> @@ -113,6 +117,14 @@ fn probe<'bound>(
> pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
> },
>
> + _: {
> + let spec = Spec::new(pdev.as_ref(), bar)?;
> +
> + // We must wait for GFW_BOOT completion before doing any significant setup on
> + // the GPU.
> + gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
> + },
So, apparently this is done in order to be able to add this
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
in a subsequent patch.
I prefer to make wait_gfw_boot_completion() an associated function of Gpu, keep
it called from Gpu::new() and guard the whole block with
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)], such that eventually it becomes:
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
_: {
let spec = Spec::new(pdev.as_ref(), bar)?;
Gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
},
With this Gpu::new() remains self-contained and we don't build up corss
initialization dependencies.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v5 08/15] gpu: nova-core: add an interrupt delivery self-test
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (6 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 09/15] gpu: nova-core: log GSP events instead of discarding them John Hubbard
` (6 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard, Joel Fernandes
A GPU interrupt can be lost in the MSI or MSI-X allocation, in the GIN
tree's enables, or in the rearm, and every one of those failures looks
the same: no interrupt arrives, and no register or log line says which
one broke.
Add a probe-time self-test, built under NOVA_CORE_SELFTESTS, that
latches the CPU doorbell vector through the GIN software trigger and
waits for a registered handler to service it. One delivery would pass
with a broken rearm, because the first message-signaled interrupt
arrives whether or not the driver rearms, so the test triggers twice and
waits for the first handler to finish before the second trigger. It runs
after GFW boot and before GSP boot, on a quiesced tree, and fails probe
unless both deliveries arrive, each finds only the doorbell pending, and
the leaf ends clear.
The doorbell has the same vector on every supported GPU, so the test
names it without asking GSP-RM. It allocates the PCI vectors for the
doorbell's subtree and releases them before returning, so under MSI-X
the delivery also exercises that subtree's table entry.
Assisted-by: LLM
Co-developed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/Kconfig | 5 +
drivers/gpu/nova-core/driver.rs | 5 +
drivers/gpu/nova-core/irq.rs | 2 +
drivers/gpu/nova-core/irq/doorbell_test.rs | 257 ++++++++++++++++++++
drivers/gpu/nova-core/irq/interrupt_tree.rs | 15 ++
drivers/gpu/nova-core/nova_core.rs | 2 +-
6 files changed, 285 insertions(+), 1 deletion(-)
create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs
diff --git a/drivers/gpu/nova-core/Kconfig b/drivers/gpu/nova-core/Kconfig
index 1934f17baa8b..2e11e46c99c7 100644
--- a/drivers/gpu/nova-core/Kconfig
+++ b/drivers/gpu/nova-core/Kconfig
@@ -24,4 +24,9 @@ config NOVA_CORE_SELFTESTS
help
Build the driver self-tests and run them when the GPU is probed.
+ If the interrupt delivery test fails, the probe fails and the driver
+ does not bind to the GPU. A broken interrupt path would otherwise
+ show up later as a hang, far from its cause. Every other self-test
+ logs its failure and lets the probe continue.
+
If unsure, say N.
diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index fc321c6a10b0..6d45fec6d7cc 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -123,6 +123,11 @@ fn probe<'bound>(
// We must wait for GFW_BOOT completion before doing any significant setup on
// the GPU.
gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+
+ // The self-test disables and drains the whole tree, so it has to run before
+ // `Gpu::new` boots the GSP.
+ #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+ crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
},
// TODO: Use self-referential pin-init syntax once available.
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index b2cfe73af114..ff8b00a82442 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -9,6 +9,8 @@
//!
//! See `Documentation/gpu/nova/core/interrupts.rst`.
+#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+pub(crate) mod doorbell_test;
mod hal;
pub(crate) mod interrupt_tree;
mod regs;
diff --git a/drivers/gpu/nova-core/irq/doorbell_test.rs b/drivers/gpu/nova-core/irq/doorbell_test.rs
new file mode 100644
index 000000000000..aef9f4c7f2e8
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/doorbell_test.rs
@@ -0,0 +1,257 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Interrupt delivery self-test.
+//!
+//! The test triggers the CPU doorbell vector from software, twice, and checks that each trigger
+//! reaches a registered handler. It runs during probe under `CONFIG_NOVA_CORE_SELFTESTS`.
+//!
+//! See "Self-test" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+use core::pin::Pin;
+
+use kernel::{
+ device::Bound,
+ irq,
+ pci,
+ prelude::*,
+ sync::{
+ atomic::{
+ Atomic,
+ Relaxed, //
+ },
+ Completion, //
+ },
+ time, //
+};
+
+use super::interrupt_tree::{
+ GinVector,
+ LeafEnableGuard,
+ LeafMask,
+ Subtree,
+ TopEnableGuard,
+ Tree, //
+};
+
+use crate::{
+ driver::Bar0,
+ gpu::Chipset,
+ selftest_assert,
+ selftest_assert_eq, //
+};
+
+/// The CPU doorbell vector. Every supported GPU uses this number, so the test needs nothing from
+/// GSP-RM, which is not running yet.
+const DOORBELL_VECTOR: GinVector = GinVector::new::<129>();
+
+/// The only subtree that this test services.
+const DOORBELL_SUBTREE: Subtree = DOORBELL_VECTOR.subtree();
+
+/// Time allowed for each delivery to arrive.
+const DELIVERY_TIMEOUT_MS: time::Msecs = 1000;
+
+/// The self-test's interrupt handler.
+///
+/// It clears only the doorbell's bit, rearms delivery, and never walks the tree. A missing rearm
+/// shows up as a timeout on the second delivery.
+#[pin_data]
+struct DoorbellTestHandler<'a> {
+ tree: &'a Tree<'a>,
+ /// Completed by the first delivery.
+ #[pin]
+ first: Completion,
+ /// Completed by the second delivery.
+ #[pin]
+ second: Completion,
+ /// Deliveries that found the doorbell bit set.
+ irq_count: Atomic<u32>,
+ /// The doorbell leaf's pending bits, as read by the first delivery.
+ first_pending: Atomic<u32>,
+ /// The doorbell leaf's pending bits, as read by the second delivery.
+ second_pending: Atomic<u32>,
+}
+
+impl irq::Handler for DoorbellTestHandler<'_> {
+ fn handle(&self) -> irq::IrqReturn {
+ let leaf = self.tree.read_pending(DOORBELL_VECTOR.leaf_index());
+ let pending = leaf.vectors();
+ if !pending.contains(DOORBELL_VECTOR.leaf_mask()) {
+ self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
+ return irq::IrqReturn::None;
+ }
+ leaf.clear_vectors(DOORBELL_VECTOR.leaf_mask());
+ // Rearm before completing, since the waiting thread triggers the next doorbell as soon as
+ // it wakes.
+ self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
+
+ match self.irq_count.fetch_add(1, Relaxed) {
+ 0 => {
+ self.first_pending.store(pending.into_raw(), Relaxed);
+ self.first.complete_all();
+ }
+ 1 => {
+ self.second_pending.store(pending.into_raw(), Relaxed);
+ self.second.complete_all();
+ }
+ _ => (),
+ }
+
+ irq::IrqReturn::Handled
+ }
+}
+
+/// The self-test's handler registration and the enables that deliver to it.
+///
+/// Drops in the order that "Enabling the GSP event" in
+/// `Documentation/gpu/nova/core/interrupts.rst` requires: the vector is disabled, then the
+/// handler is freed, then the subtree is disabled.
+struct SelftestResources<'a, 'r> {
+ _leaf_guard: LeafEnableGuard<'a>,
+ reg: Pin<KBox<irq::Registration<'r, DoorbellTestHandler<'a>>>>,
+ _top_guard: TopEnableGuard<'a>,
+}
+
+impl<'a> SelftestResources<'a, '_> {
+ fn handler(&self) -> &DoorbellTestHandler<'a> {
+ self.reg.handler()
+ }
+
+ /// Disables the doorbell vector and waits for a handler in flight on another CPU to finish.
+ ///
+ /// The handler's counters and the leaf's pending bits are final on return.
+ fn quiesce_source(&self) {
+ self.handler()
+ .tree
+ .disable_leaf(DOORBELL_VECTOR.leaf_index(), DOORBELL_VECTOR.leaf_mask());
+ self.reg.synchronize();
+ }
+}
+
+/// Runs the interrupt delivery self-test.
+///
+/// Call this only during probe, before GSP boot: it disables every vector in the tree and clears
+/// every pending bit. On return, the doorbell's subtree is disabled at `TOP`, and the test's PCI
+/// vectors and handler are released.
+///
+/// # Errors
+///
+/// `EINVAL` if `chipset` does not implement the doorbell's subtree. `ETIMEDOUT` if a delivery
+/// does not arrive within [`DELIVERY_TIMEOUT_MS`]. `EIO` if a self-test assertion fails.
+/// Otherwise the error from allocating the PCI vectors or registering the handler.
+pub(crate) fn run_selftest(pdev: &pci::Device<Bound>, bar: Bar0<'_>, chipset: Chipset) -> Result {
+ let tree = Tree::new(pdev, bar, chipset, DOORBELL_SUBTREE.into())?;
+ let tree_ref = &tree;
+ let request = tree.request_for(DOORBELL_SUBTREE)?;
+ let doorbell = DOORBELL_VECTOR.leaf_index();
+ let doorbell_mask = DOORBELL_VECTOR.leaf_mask();
+
+ dev_info!(
+ pdev,
+ "interrupt self-test: starting on vector {}, subtree {}, with {:?}\n",
+ DOORBELL_VECTOR.into_raw(),
+ DOORBELL_SUBTREE.index(),
+ tree.msi_type,
+ );
+
+ // GFW boot can leave vectors enabled and pending. Registering a handler unmasks the PCI
+ // interrupt, and they would be delivered to a handler that services only the doorbell.
+ tree.reset();
+
+ // A delivery proves nothing unless the doorbell bit starts out clear.
+ let pre_pending = tree.read_pending(doorbell).vectors();
+ selftest_assert!(
+ pdev,
+ !pre_pending.contains(doorbell_mask),
+ "vector {} already pending, leaf[{}] is {:#x}",
+ DOORBELL_VECTOR.into_raw(),
+ doorbell.get(),
+ pre_pending.into_raw()
+ );
+
+ let handler_init = try_pin_init!(DoorbellTestHandler {
+ tree: tree_ref,
+ first <- Completion::new(),
+ second <- Completion::new(),
+ irq_count: Atomic::new(0),
+ first_pending: Atomic::new(0),
+ second_pending: Atomic::new(0),
+ }? Error);
+
+ // Registration must precede any enable, or a delivery reaches no handler.
+ let reg = KBox::pin_init(
+ // SAFETY: this registration is dropped before the enclosing function returns, so its
+ // `Drop`, which calls `free_irq()`, always runs.
+ unsafe {
+ irq::Registration::new(
+ request,
+ irq::Flags::TRIGGER_NONE,
+ c"nova-core-selftest",
+ handler_init,
+ )
+ },
+ GFP_KERNEL,
+ )?;
+
+ let resources = SelftestResources {
+ _leaf_guard: tree.enable_leaf_guarded(doorbell, doorbell_mask),
+ _top_guard: tree.enable_top_guarded(),
+ reg,
+ };
+ let handler = resources.handler();
+
+ tree.trigger(DOORBELL_VECTOR)?;
+ let mut completed = handler
+ .first
+ .wait_for_completion_timeout(time::msecs_to_jiffies(DELIVERY_TIMEOUT_MS))
+ .is_some();
+
+ // The second trigger waits for the first delivery, or the two could coalesce.
+ if completed {
+ tree.trigger(DOORBELL_VECTOR)?;
+ completed = handler
+ .second
+ .wait_for_completion_timeout(time::msecs_to_jiffies(DELIVERY_TIMEOUT_MS))
+ .is_some();
+ }
+
+ resources.quiesce_source();
+
+ let count = handler.irq_count.load(Relaxed);
+ let first_pending = LeafMask::from_raw(handler.first_pending.load(Relaxed));
+ let second_pending = LeafMask::from_raw(handler.second_pending.load(Relaxed));
+ let residual = tree.read_pending(doorbell).vectors();
+
+ if !completed {
+ dev_err!(
+ pdev,
+ "interrupt self-test: only {} of 2 deliveries arrived within {} ms\n",
+ count,
+ DELIVERY_TIMEOUT_MS,
+ );
+ return Err(ETIMEDOUT);
+ }
+
+ selftest_assert_eq!(pdev, count, 2, "delivery count");
+
+ // Every other vector in the leaf is disabled and was drained, so require the exact mask.
+ selftest_assert_eq!(pdev, first_pending, doorbell_mask, "first delivery");
+ selftest_assert_eq!(pdev, second_pending, doorbell_mask, "second delivery");
+ selftest_assert!(
+ pdev,
+ !residual.contains(doorbell_mask),
+ "vector {} still pending, leaf[{}] is {:#x}",
+ DOORBELL_VECTOR.into_raw(),
+ doorbell.get(),
+ residual.into_raw()
+ );
+
+ dev_info!(
+ pdev,
+ "interrupt self-test: passed, subtree {}, {} deliveries\n",
+ DOORBELL_SUBTREE.index(),
+ count,
+ );
+
+ Ok(())
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 6e97828bbd42..4bb27cc8b6cd 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -444,6 +444,21 @@ pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
}
}
+ /// Sets the pending bit of `vector` in its leaf, as if the vector's source had raised it.
+ ///
+ /// # Errors
+ ///
+ /// `EINVAL` if this tree does not implement `vector`.
+ #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
+ pub(super) fn trigger(&self, vector: GinVector) -> Result {
+ vector.validate(self.hal.leaf_count())?;
+ self.bar.write_reg(
+ NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
+ );
+
+ Ok(())
+ }
+
/// Disables every vector in every implemented leaf, including the subtrees that nova-core does
/// not service. Call this only during probe.
pub(super) fn disable_all_leaves(&self) {
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 8202c4982efa..8c0761dedda4 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -18,7 +18,7 @@
mod fsp;
mod gpu;
mod gsp;
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
mod irq;
mod mctp;
mod mm;
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 09/15] gpu: nova-core: log GSP events instead of discarding them
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (7 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 08/15] gpu: nova-core: add an interrupt delivery self-test John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 10/15] gpu: nova-core: return ENOMSG for an unmatched GSP message John Hubbard
` (5 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The GSP posts unsolicited messages on the same queue that carries
command replies: log records, OS error and robust-channel records, and
lifecycle notices.
nova-core discarded every message that was not the reply that a caller
was waiting for, and an unrecognized function code aborted the in-flight
command. The GSP's error reports never reached the kernel log.
Log every non-reply message according to its function code, on the
receive path that already reads it, and leave the in-flight command
waiting for its reply. Event payloads, such as XID numbers and log
contents, are not decoded.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/gsp/cmdq.rs | 52 ++++++++++++++++++++++++-------
1 file changed, 41 insertions(+), 11 deletions(-)
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index a5595da23407..4591f6c61ce6 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -559,8 +559,7 @@ fn notify_gsp(bar: Bar0<'_>) {
/// Sends `command` to the GSP and waits for the reply.
///
- /// Messages with non-matching function codes are silently consumed until the expected reply
- /// arrives.
+ /// Events that arrive before the reply are logged and consumed.
///
/// The queue is locked for the entire send+receive cycle to ensure that no other command can
/// be interleaved.
@@ -819,8 +818,8 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
/// Receive a message from the GSP.
///
- /// The expected message type is specified using the `M` generic parameter. If the pending
- /// message has a different function code, `ERANGE` is returned and the message is consumed.
+ /// A message whose function code is `M::FUNCTION` is decoded and returned. Any other message
+ /// is logged as an event.
///
/// The read pointer is always advanced past the message, regardless of whether it matched.
///
@@ -829,8 +828,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
/// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
/// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
/// message queue.
- /// - `EINVAL` if the function code of the message was not recognized.
- /// - `ERANGE` if the message had a recognized but non-matching function code.
+ /// - `ERANGE` if the message was not the awaited reply.
///
/// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -839,11 +837,11 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
Error: From<M::InitError>,
{
let message = self.wait_for_msg(timeout)?;
- let function = message.header.function().map_err(|_| EINVAL)?;
+ let function = message.header.function();
+ let seq = message.header.sequence();
- // Extract the message. Store the result as we want to advance the read pointer even in
- // case of failure.
- let result = if function == M::FUNCTION {
+ // An early return here would leave the read pointer on this message.
+ let result = if matches!(function, Ok(f) if f == M::FUNCTION) {
let (cmd, contents_1) = M::Message::from_bytes_prefix(message.contents.0).ok_or(EIO)?;
let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
@@ -854,11 +852,13 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
dev_warn!(
&self.dev,
"GSP message {:?} has unprocessed data\n",
- function
+ M::FUNCTION
);
}
})
} else {
+ self.log_event(function, seq);
+
Err(ERANGE)
};
@@ -869,4 +869,34 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
result
}
+
+ /// Logs an event, meaning a message that no caller was waiting for.
+ ///
+ /// An OS error or robust-channel record is logged at error level and an unknown function code
+ /// at warning level. Every other event is recorded only by the receive trace in
+ /// [`Self::wait_for_msg`].
+ fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
+ match function {
+ Ok(MsgFunction::OsErrorLog) => {
+ dev_err!(&self.dev, "GSP reported an OS error (seq {})\n", seq);
+ }
+ Ok(MsgFunction::RcTriggered) => {
+ dev_err!(
+ &self.dev,
+ "GSP triggered robust-channel recovery (seq {})\n",
+ seq
+ );
+ }
+ // Nothing to do for the remaining known function codes.
+ Ok(_) => {}
+ Err(raw) => {
+ dev_warn!(
+ &self.dev,
+ "unknown GSP message function {:#x} (seq {})\n",
+ raw,
+ seq
+ );
+ }
+ }
+ }
}
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 10/15] gpu: nova-core: return ENOMSG for an unmatched GSP message
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (8 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 09/15] gpu: nova-core: log GSP events instead of discarding them John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 11/15] gpu: nova-core: bound a GSP wait by a single deadline John Hubbard
` (4 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The GSP posts unsolicited events on the same queue as command replies,
so a receive that asks for one message type has to report that the
message at the queue head was a different one.
That case returned ERANGE, which means a value outside a valid range and
says nothing about a message.
Return ENOMSG, no message of the desired type, instead.
Suggested-by: Gary Guo <gary@garyguo.net>
Suggested-by: Alexandre Courbot <acourbot@nvidia.com>
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/gsp/cmdq.rs | 6 +++---
drivers/gpu/nova-core/gsp/commands.rs | 2 +-
drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 4591f6c61ce6..ad2a14d08cb0 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -585,7 +585,7 @@ pub(crate) fn send_command<M>(&self, command: M) -> Result<M::Reply>
loop {
match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
Ok(reply) => break Ok(reply),
- Err(ERANGE) => continue,
+ Err(ENOMSG) => continue,
Err(e) => break Err(e),
}
}
@@ -828,7 +828,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
/// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
/// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
/// message queue.
- /// - `ERANGE` if the message was not the awaited reply.
+ /// - `ENOMSG` if the message was not the awaited reply.
///
/// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -859,7 +859,7 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
} else {
self.log_event(function, seq);
- Err(ERANGE)
+ Err(ENOMSG)
};
// Advance the read pointer past this message.
diff --git a/drivers/gpu/nova-core/gsp/commands.rs b/drivers/gpu/nova-core/gsp/commands.rs
index 59d7d7fb15e8..0c787fd689bf 100644
--- a/drivers/gpu/nova-core/gsp/commands.rs
+++ b/drivers/gpu/nova-core/gsp/commands.rs
@@ -191,7 +191,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
loop {
match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
Ok(_) => break Ok(()),
- Err(ERANGE) => continue,
+ Err(ENOMSG) => continue,
Err(e) => break Err(e),
}
}
diff --git a/drivers/gpu/nova-core/gsp/sequencer.rs b/drivers/gpu/nova-core/gsp/sequencer.rs
index dae34c11eb05..1782ed7d7ca6 100644
--- a/drivers/gpu/nova-core/gsp/sequencer.rs
+++ b/drivers/gpu/nova-core/gsp/sequencer.rs
@@ -346,7 +346,7 @@ pub(crate) fn run(
let seq_info = loop {
match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
Ok(seq_info) => break seq_info,
- Err(ERANGE) => continue,
+ Err(ENOMSG) => continue,
Err(e) => return Err(e),
}
};
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 11/15] gpu: nova-core: bound a GSP wait by a single deadline
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (9 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 10/15] gpu: nova-core: return ENOMSG for an unmatched GSP message John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 12/15] gpu: nova-core: add the falcon interrupt registers and HAL methods John Hubbard
` (3 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The GSP posts unsolicited events on the same queue as command replies,
so a caller waiting for one message consumes whatever arrives first and
reads again.
Every read started a fresh five-second timeout, so a steady stream of
events extended the wait without bound. The two boot-time waits for an
unsolicited event also released the queue mutex between reads, so a
command sent from another thread could consume the event and leave the
waiter to time out.
Compute one deadline when the wait begins and pass the time remaining to
each read, and hold the queue mutex across the whole wait. Put the loop
in one helper that the command reply wait and both boot-time event waits
share.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/gsp/cmdq.rs | 72 ++++++++++++++++++++------
drivers/gpu/nova-core/gsp/commands.rs | 8 +--
drivers/gpu/nova-core/gsp/sequencer.rs | 8 +--
3 files changed, 59 insertions(+), 29 deletions(-)
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index ad2a14d08cb0..ae3808de44e2 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -28,7 +28,11 @@
},
Mutex, //
},
- time::Delta,
+ time::{
+ Delta,
+ Instant,
+ Monotonic, //
+ },
transmute::{
AsBytes,
FromBytes, //
@@ -130,7 +134,9 @@ fn size(&self) -> usize {
/// Trait representing messages received from the GSP.
///
-/// This trait tells [`Cmdq::receive_msg`] how it can receive a given type of message.
+/// A reply that [`Cmdq::send_command`] waits for, or an event that [`Cmdq::await_msg`] waits for.
+/// The receiver matches a message's function code against [`Self::FUNCTION`] and decodes the
+/// message with [`Self::read`].
pub(crate) trait MessageFromGsp: Sized {
/// Function identifying this message from the GSP.
const FUNCTION: MsgFunction;
@@ -566,8 +572,9 @@ fn notify_gsp(bar: Bar0<'_>) {
///
/// # Errors
///
- /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
- /// not received within the timeout.
+ /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply does
+ /// not arrive within [`Self::RECEIVE_TIMEOUT`] of the send, however many events arrive
+ /// while waiting.
/// - `EIO` if the variable payload requested by the command has not been entirely
/// written to by its [`CommandToGsp::init_variable_payload`] method.
///
@@ -582,13 +589,7 @@ pub(crate) fn send_command<M>(&self, command: M) -> Result<M::Reply>
let mut inner = self.inner.lock();
inner.send_command(command)?;
- loop {
- match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
- Ok(reply) => break Ok(reply),
- Err(ENOMSG) => continue,
- Err(e) => break Err(e),
- }
- }
+ inner.await_msg()
}
/// Sends `command` to the GSP without waiting for a reply.
@@ -608,15 +609,25 @@ pub(crate) fn send_command_no_wait<M>(&self, command: M) -> Result
self.inner.lock().send_command(command)
}
- /// Receive a message from the GSP.
+ /// Waits for an unsolicited GSP event of type `M`. Events that arrive before it are logged and
+ /// consumed.
+ ///
+ /// The queue mutex is held for the whole wait, up to [`Self::RECEIVE_TIMEOUT`], so no other
+ /// caller can send a command or consume an event meanwhile.
///
- /// See [`CmdqInner::receive_msg`] for details.
- pub(crate) fn receive_msg<M: MessageFromGsp>(&self, timeout: Delta) -> Result<M>
+ /// # Errors
+ ///
+ /// - `ETIMEDOUT` if the event does not arrive within [`Self::RECEIVE_TIMEOUT`] of the call,
+ /// however many other events arrive while waiting.
+ /// - `EIO` if a message fails framing or checksum validation.
+ ///
+ /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+ pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
where
// This allows all error types, including `Infallible`, to be used for `M::InitError`.
Error: From<M::InitError>,
{
- self.inner.lock().receive_msg(timeout)
+ self.inner.lock().await_msg()
}
}
@@ -870,6 +881,37 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
result
}
+ /// Receives a message of type `M`, waiting up to [`Cmdq::RECEIVE_TIMEOUT`] from the call.
+ ///
+ /// Any other message that arrives first is logged as an event and does not extend the
+ /// deadline.
+ ///
+ /// # Errors
+ ///
+ /// - `ETIMEDOUT` if no message of type `M` arrives before the deadline, however many other
+ /// messages arrive while waiting.
+ /// - `EIO` if a message fails framing or checksum validation (see [`Self::wait_for_msg`]).
+ ///
+ /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+ fn await_msg<M: MessageFromGsp>(&mut self) -> Result<M>
+ where
+ // This allows all error types, including `Infallible`, to be used for `M::InitError`.
+ Error: From<M::InitError>,
+ {
+ let deadline = Instant::<Monotonic>::now() + Cmdq::RECEIVE_TIMEOUT;
+ loop {
+ let remaining = deadline - Instant::<Monotonic>::now();
+ if remaining.is_negative() {
+ break Err(ETIMEDOUT);
+ }
+ match self.receive_msg::<M>(remaining) {
+ Ok(msg) => break Ok(msg),
+ Err(ENOMSG) => continue,
+ Err(e) => break Err(e),
+ }
+ }
+ }
+
/// Logs an event, meaning a message that no caller was waiting for.
///
/// An OS error or robust-channel record is logged at error level and an unknown function code
diff --git a/drivers/gpu/nova-core/gsp/commands.rs b/drivers/gpu/nova-core/gsp/commands.rs
index 0c787fd689bf..cdbb13674f08 100644
--- a/drivers/gpu/nova-core/gsp/commands.rs
+++ b/drivers/gpu/nova-core/gsp/commands.rs
@@ -188,13 +188,7 @@ fn read(
/// Waits for GSP initialization to complete.
pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
- loop {
- match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
- Ok(_) => break Ok(()),
- Err(ENOMSG) => continue,
- Err(e) => break Err(e),
- }
- }
+ cmdq.await_msg::<GspInitDone>().map(|_| ())
}
/// The `GetGspStaticInfo` command.
diff --git a/drivers/gpu/nova-core/gsp/sequencer.rs b/drivers/gpu/nova-core/gsp/sequencer.rs
index 1782ed7d7ca6..250adc9fe74f 100644
--- a/drivers/gpu/nova-core/gsp/sequencer.rs
+++ b/drivers/gpu/nova-core/gsp/sequencer.rs
@@ -343,13 +343,7 @@ pub(crate) fn run(
libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
bootloader_app_version: u32,
) -> Result {
- let seq_info = loop {
- match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
- Ok(seq_info) => break seq_info,
- Err(ENOMSG) => continue,
- Err(e) => return Err(e),
- }
- };
+ let seq_info = cmdq.await_msg::<GspSequence>()?;
let sequencer = GspSequencer {
bar: ctx.bar,
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 12/15] gpu: nova-core: add the falcon interrupt registers and HAL methods
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (10 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 11/15] gpu: nova-core: bound a GSP wait by a single deadline John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 13/15] gpu: nova-core: service GSP events from the SWGEN0 interrupt John Hubbard
` (2 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
A falcon latches each interrupt cause that is raised in its IRQSTAT
register. On a RISC-V falcon, each cause is routed either to the host,
meaning the CPU, or to the falcon's own core, and two routing registers
hold that routing. The falcon signals the interrupt tree only when its
set of host-routed causes goes from empty to non-empty, so a handler
that clears the tree leaf while a cause is still latched has to write
INTR_RETRIGGER, which makes the falcon re-emit its causes. Turing
falcons have no INTR_RETRIGGER, and the routing registers have
different offsets from GA102 on.
Add IRQSTAT, INTR_RETRIGGER, and the two routing registers, the last at
both sets of offsets, and add two falcon HAL methods: one intersects an
IRQSTAT value with the routing registers, and one retriggers the
falcon, or does nothing on Turing. GA100 keeps the Turing HAL, as it
does for boot, and the retrigger register is a property of that HAL's
instance.
The two methods have no user yet.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/falcon/hal.rs | 21 ++++++-
drivers/gpu/nova-core/falcon/hal/ga102.rs | 16 ++++++
drivers/gpu/nova-core/falcon/hal/tu102.rs | 51 ++++++++++++++++-
drivers/gpu/nova-core/regs.rs | 69 +++++++++++++++++++++++
4 files changed, 154 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 7e532889a1f4..2f02da52334e 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -12,6 +12,7 @@
Architecture,
Chipset, //
},
+ regs,
};
mod ga102;
@@ -70,6 +71,24 @@ fn signature_reg_fuse_version(
/// these. For anything above, the PIO registers appear to be masked to the CPU, so DMA is the
/// only usable method.
fn load_method(&self) -> LoadMethod;
+
+ /// Returns the causes in `latched` that are routed to the host, meaning the CPU, rather than
+ /// to the falcon's own RISC-V core.
+ ///
+ /// The causes routed to the core belong to the firmware running on it, and the host does not
+ /// service them.
+ #[expect(dead_code)]
+ fn host_routed_causes(
+ &self,
+ falcon: &Falcon<'_, E>,
+ latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+ ) -> regs::NV_PFALCON_FALCON_IRQSTAT;
+
+ /// Retriggers the falcon, which then re-emits its host-routed causes into the interrupt tree.
+ ///
+ /// Turing falcons have no retrigger register, so on Turing this does nothing.
+ #[expect(dead_code)]
+ fn retrigger(&self, falcon: &Falcon<'_, E>);
}
/// Returns a boxed falcon HAL adequate for `chipset`.
@@ -86,7 +105,7 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
}
// GA100 boots like Turing so use Turing HAL
Architecture::Ampere if chipset == Chipset::GA100 => {
- KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
+ KBox::new(tu102::Tu102::<E>::ga100(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
}
Architecture::Ampere
| Architecture::Ada
diff --git a/drivers/gpu/nova-core/falcon/hal/ga102.rs b/drivers/gpu/nova-core/falcon/hal/ga102.rs
index f9a8444cf840..b0eb1901d213 100644
--- a/drivers/gpu/nova-core/falcon/hal/ga102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/ga102.rs
@@ -169,4 +169,20 @@ fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
fn load_method(&self) -> LoadMethod {
LoadMethod::Dma
}
+
+ fn host_routed_causes(
+ &self,
+ falcon: &Falcon<'_, E>,
+ latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+ ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+ let pfalcon2 = falcon.pfalcon2;
+ let mask = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQMASK).value();
+ let dest = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQDEST).value();
+
+ regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
+ }
+
+ fn retrigger(&self, falcon: &Falcon<'_, E>) {
+ super::tu102::retrigger_ga100(falcon);
+ }
}
diff --git a/drivers/gpu/nova-core/falcon/hal/tu102.rs b/drivers/gpu/nova-core/falcon/hal/tu102.rs
index 7fc6e83c2566..bd66b7759ea6 100644
--- a/drivers/gpu/nova-core/falcon/hal/tu102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/tu102.rs
@@ -5,6 +5,7 @@
use kernel::{
io::{
poll::read_poll_timeout,
+ register::Array,
Io, //
},
prelude::*,
@@ -23,14 +24,42 @@
use super::FalconHal;
-pub(super) struct Tu102<E: FalconEngine>(PhantomData<E>);
+/// The falcon HAL for Turing and GA100. GA100 boots like a Turing, but unlike Turing, it has a
+/// retrigger register.
+pub(super) struct Tu102<E: FalconEngine> {
+ /// If `true`, the falcons have `NV_PFALCON_FALCON_INTR_RETRIGGER`.
+ #[expect(dead_code)]
+ has_intr_retrigger: bool,
+ _engine: PhantomData<E>,
+}
impl<E: FalconEngine> Tu102<E> {
+ /// Returns the HAL of Turing falcons.
pub(super) fn new() -> Self {
- Self(PhantomData)
+ Self {
+ has_intr_retrigger: false,
+ _engine: PhantomData,
+ }
+ }
+
+ /// Returns the HAL of GA100 falcons: the Turing HAL, with the retrigger register.
+ pub(super) fn ga100() -> Self {
+ Self {
+ has_intr_retrigger: true,
+ _engine: PhantomData,
+ }
}
}
+/// Writes `NV_PFALCON_FALCON_INTR_RETRIGGER`.
+#[expect(dead_code)]
+pub(super) fn retrigger_ga100<E: FalconEngine>(falcon: &Falcon<'_, E>) {
+ falcon.pfalcon.write(
+ Array::at(0),
+ regs::NV_PFALCON_FALCON_INTR_RETRIGGER::zeroed().with_trigger(true),
+ );
+}
+
impl<E: FalconEngine> FalconHal<E> for Tu102<E> {
fn select_core(&self, _falcon: &Falcon<'_, E>) -> Result {
Ok(())
@@ -79,4 +108,22 @@ fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
fn load_method(&self) -> LoadMethod {
LoadMethod::Pio
}
+
+ fn host_routed_causes(
+ &self,
+ falcon: &Falcon<'_, E>,
+ latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+ ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+ let pfalcon2 = falcon.pfalcon2;
+ let mask = pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQMASK).value();
+ let dest = pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQDEST).value();
+
+ regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
+ }
+
+ fn retrigger(&self, falcon: &Falcon<'_, E>) {
+ if self.has_intr_retrigger {
+ retrigger_ga100(falcon);
+ }
+ }
}
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 9978fb2803b0..88662da634dc 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -124,11 +124,25 @@ pub(crate) fn usable_fb_size(self) -> u64 {
register! {
base: PFalconRegisters;
+ /// Clears the latch of every cause whose bit is written as `1`. Write-only.
+ ///
+ /// The write ends the latch and not the source, so a cause driven from outside the falcon
+ /// stays set. "Retriggering a falcon" in `Documentation/gpu/nova/core/interrupts.rst` names
+ /// those causes.
pub(crate) NV_PFALCON_FALCON_IRQSCLR(u32) @ 0x00000004 {
6:6 swgen0 => bool;
4:4 halt => bool;
}
+ /// Interrupt causes latched in the falcon, one bit per cause, whichever target each is routed
+ /// to.
+ ///
+ /// The causes routed to the host are the ones also set in `NV_PRISCV_RISCV_IRQMASK` and
+ /// `NV_PRISCV_RISCV_IRQDEST`.
+ pub(crate) NV_PFALCON_FALCON_IRQSTAT(u32) @ 0x00000008 {
+ 6:6 swgen0 => bool;
+ }
+
pub(crate) NV_PFALCON_FALCON_MAILBOX0(u32) @ 0x00000040 {
31:0 value => u32;
}
@@ -256,6 +270,16 @@ pub(crate) fn usable_fb_size(self) -> u64 {
0:0 reset => bool;
}
+ /// Makes the falcon re-emit its host-routed causes into the interrupt tree. Write-only.
+ ///
+ /// Present from GA100 on. See "Retriggering a falcon" in
+ /// `Documentation/gpu/nova/core/interrupts.rst`.
+ ///
+ /// The hardware headers declare two elements, and OpenRM writes only the first.
+ pub(crate) NV_PFALCON_FALCON_INTR_RETRIGGER(u32)[2] @ 0x000003e8 {
+ 0:0 trigger => bool;
+ }
+
pub(crate) NV_PFALCON_FBIF_TRANSCFG(u32)[8] @ 0x00000600 {
2:2 mem_type => FalconFbifMemType;
1:0 target ?=> FalconFbifTarget;
@@ -414,6 +438,29 @@ pub(crate) mod gm107 {
}
}
+pub(crate) mod tu102 {
+ use kernel::io::register;
+
+ use crate::falcon::PFalcon2Registers;
+
+ // The RISC-V interrupt routing registers, at the offsets that Turing and GA100 use.
+
+ register! {
+ base: PFalcon2Registers;
+
+ /// Enabled causes, one bit per cause. Read-only to the host.
+ pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x000002b4 {
+ 31:0 value => u32;
+ }
+
+ /// Causes routed to the host, one bit per cause. A clear bit routes the cause to the
+ /// RISC-V core.
+ pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x000002b8 {
+ 31:0 value => u32;
+ }
+ }
+}
+
pub(crate) mod ga100 {
use kernel::io::register;
@@ -430,6 +477,28 @@ pub(crate) mod ga100 {
}
}
+pub(crate) mod ga102 {
+ use kernel::io::register;
+
+ use crate::falcon::PFalcon2Registers;
+
+ // The RISC-V interrupt routing registers, at the offsets that GA102 and later use.
+
+ register! {
+ base: PFalcon2Registers;
+
+ /// Same as [`super::tu102::NV_PRISCV_RISCV_IRQMASK`], at the GA102 offset.
+ pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x00000528 {
+ 31:0 value => u32;
+ }
+
+ /// Same as [`super::tu102::NV_PRISCV_RISCV_IRQDEST`], at the GA102 offset.
+ pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x0000052c {
+ 31:0 value => u32;
+ }
+ }
+}
+
pub(crate) const NV_THERM_I2CS_SCRATCH_FSP_BOOT_COMPLETE_STATUS_SUCCESS: u32 = 0xff;
pub(crate) mod gh100 {
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 13/15] gpu: nova-core: service GSP events from the SWGEN0 interrupt
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (11 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 12/15] gpu: nova-core: add the falcon interrupt registers and HAL methods John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 14/15] gpu: nova-core: add KUnit tests for the interrupt tree John Hubbard
2026-09-30 3:41 ` [PATCH v5 15/15] gpu: nova-core: document the GIN interrupt controller and GSP events John Hubbard
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The GSP posts each message for the CPU to the GSP-to-CPU queue and then
raises SWGEN0, a software-generated interrupt cause of its falcon. GIN
delivers SWGEN0 to the CPU on vector 155.
A thread waiting for a command reply was the only reader of the queue,
so an event posted between commands stayed unread until the next
command was sent.
Register a threaded handler on the GSP vector. Draining the queue takes
the command-queue mutex, which can sleep, so the top half reads and
writes only registers and wakes the IRQ thread to drain the queue.
Every existing read of the queue asks for one message type and waits on
a deadline, so add a drain that logs and consumes whatever the GSP has
already posted and returns at once.
When a falcon cause other than SWGEN0 stays set after its latch is
cleared, as the Blackwell fault-containment and ECC causes do, disable
the GSP vector rather than retrigger the falcon. A retrigger would
re-emit the cause at once, and the CPU would take the same interrupt
again and again.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/falcon/gsp.rs | 47 ++++-
drivers/gpu/nova-core/falcon/hal.rs | 2 -
drivers/gpu/nova-core/falcon/hal/tu102.rs | 2 -
drivers/gpu/nova-core/gpu.rs | 96 +++++++++-
drivers/gpu/nova-core/gsp/cmdq.rs | 40 ++++
drivers/gpu/nova-core/irq.rs | 1 +
drivers/gpu/nova-core/irq/gsp.rs | 196 ++++++++++++++++++++
drivers/gpu/nova-core/irq/interrupt_tree.rs | 3 +
drivers/gpu/nova-core/nova_core.rs | 1 -
9 files changed, 375 insertions(+), 13 deletions(-)
create mode 100644 drivers/gpu/nova-core/irq/gsp.rs
diff --git a/drivers/gpu/nova-core/falcon/gsp.rs b/drivers/gpu/nova-core/falcon/gsp.rs
index 4c96ae325fda..1faefaede9fe 100644
--- a/drivers/gpu/nova-core/falcon/gsp.rs
+++ b/drivers/gpu/nova-core/falcon/gsp.rs
@@ -47,13 +47,56 @@ fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
}
impl<'a> Falcon<'a, Gsp> {
- /// Clears the SWGEN0 bit in the Falcon's IRQ status clear register to
- /// allow GSP to signal CPU for processing new messages in message queue.
+ /// Clears the SWGEN0 latch in the GSP falcon.
+ ///
+ /// While the latch is set, no later message signals the tree, so a caller that consumed a
+ /// notification by polling must clear it.
pub(crate) fn clear_swgen0_intr(&self) {
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true));
}
+ /// Reads the GSP falcon causes that are routed to the host, without clearing any latch.
+ ///
+ /// Every one of them other than SWGEN0 reports a GSP fault.
+ pub(crate) fn read_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+ let latched = self.pfalcon.read(regs::NV_PFALCON_FALCON_IRQSTAT);
+
+ self.hal.host_routed_causes(self, latched)
+ }
+
+ /// Reads the host-routed causes and clears the SWGEN0 latch if it was set.
+ ///
+ /// Returns the causes as read, before the clear. No other latch changes.
+ pub(crate) fn take_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+ let status = self.read_host_intr();
+
+ if status.swgen0() {
+ self.clear_swgen0_intr();
+ }
+
+ status
+ }
+
+ /// Clears the latch of every interrupt cause set in `status`.
+ ///
+ /// A cause driven from outside the falcon is still set on return, and
+ /// [`Self::read_host_intr`] reports the causes that remain.
+ pub(crate) fn clear_intr(&self, status: regs::NV_PFALCON_FALCON_IRQSTAT) {
+ self.pfalcon
+ .write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::from(status.into_raw()));
+ }
+
+ /// Retriggers the GSP falcon, which then re-emits its host-routed causes into the tree.
+ ///
+ /// Call this only once every host cause is clear. A cause still set is re-emitted at once, and
+ /// its vector arrives again as soon as delivery is rearmed.
+ ///
+ /// Does nothing on Turing, whose falcons have no retrigger register.
+ pub(crate) fn retrigger_intr(&self) {
+ self.hal.retrigger(self);
+ }
+
/// Checks if GSP reload/resume has completed during the boot process.
pub(crate) fn check_reload_completed(&self, timeout: Delta) -> Result<bool> {
read_poll_timeout(
diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 2f02da52334e..2643a677caad 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -77,7 +77,6 @@ fn signature_reg_fuse_version(
///
/// The causes routed to the core belong to the firmware running on it, and the host does not
/// service them.
- #[expect(dead_code)]
fn host_routed_causes(
&self,
falcon: &Falcon<'_, E>,
@@ -87,7 +86,6 @@ fn host_routed_causes(
/// Retriggers the falcon, which then re-emits its host-routed causes into the interrupt tree.
///
/// Turing falcons have no retrigger register, so on Turing this does nothing.
- #[expect(dead_code)]
fn retrigger(&self, falcon: &Falcon<'_, E>);
}
diff --git a/drivers/gpu/nova-core/falcon/hal/tu102.rs b/drivers/gpu/nova-core/falcon/hal/tu102.rs
index bd66b7759ea6..b25ac645a648 100644
--- a/drivers/gpu/nova-core/falcon/hal/tu102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/tu102.rs
@@ -28,7 +28,6 @@
/// retrigger register.
pub(super) struct Tu102<E: FalconEngine> {
/// If `true`, the falcons have `NV_PFALCON_FALCON_INTR_RETRIGGER`.
- #[expect(dead_code)]
has_intr_retrigger: bool,
_engine: PhantomData<E>,
}
@@ -52,7 +51,6 @@ pub(super) fn ga100() -> Self {
}
/// Writes `NV_PFALCON_FALCON_INTR_RETRIGGER`.
-#[expect(dead_code)]
pub(super) fn retrigger_ga100<E: FalconEngine>(falcon: &Falcon<'_, E>) {
falcon.pfalcon.write(
Array::at(0),
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 65715f906030..b042d68bec1e 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -34,10 +34,19 @@
fsp::Fsp,
gsp::{
self,
+ cmdq::Cmdq,
commands::GetGspStaticInfoReply,
Gsp,
GspBootContext, //
},
+ irq::{
+ self,
+ gsp::GspIrq,
+ interrupt_tree::{
+ TopEnableGuard,
+ Tree, //
+ }, //
+ },
mm::{
bar_user::BarUser,
pagetable::MmuVersion,
@@ -331,10 +340,53 @@ struct GspResources<'gpu> {
unload_bundle: Option<gsp::UnloadBundle<'gpu>>,
}
+/// The GSP event handler's registration and the enable of its subtree at `TOP`.
+///
+/// The two drop as a unit, the handler first, on the drop of [`Gpu`] and on the error path of
+/// its constructor alike, so that the subtree is disabled only after the handler is freed.
+#[pin_data]
+struct GspSubtree<'a> {
+ #[pin]
+ irq: GspIrq<'a>,
+ /// Must be kept declared *after* `irq`. A handler still in flight enables the subtree again
+ /// through its rearm.
+ _top: TopEnableGuard<'a>,
+}
+
+impl<'a> GspSubtree<'a> {
+ /// Returns an initializer that registers the GSP event handler and then enables its subtree at
+ /// `TOP`.
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the initialized `GspSubtree` or otherwise prevent its
+ /// [`Drop`] implementation, which runs `free_irq`, from running.
+ unsafe fn new(
+ pdev: &'a pci::Device<device::Bound>,
+ tree: &'a Tree<'a>,
+ falcon: &'a Falcon<'a, GspFalcon>,
+ cmdq: &'a Cmdq<'a>,
+ ) -> impl PinInit<Self, Error> + 'a {
+ try_pin_init!(Self {
+ // SAFETY: this function's caller must not leak the `GspSubtree` that owns this
+ // registration, so the registration's `Drop` runs.
+ irq <- unsafe { GspIrq::new(pdev, tree, falcon, cmdq) },
+ _top: tree.enable_top_guarded(),
+ })
+ }
+}
+
/// Structure holding the resources required to operate the GPU.
#[pin_data]
pub(crate) struct Gpu<'gpu> {
pub(crate) spec: Spec,
+ /// GSP event interrupt registration, and the enable of its subtree.
+ ///
+ /// Must be kept declared *before* `gsp_resources`, so that the handler is unregistered, and
+ /// any in-flight run of it has finished, before the command queue that it drains and the
+ /// falcon that it reads are freed, and before the GSP is unloaded.
+ #[pin]
+ _gsp_subtree: GspSubtree<'gpu>,
/// Static GPU information as provided by the GSP.
pub(crate) gsp_static_info: GetGspStaticInfoReply,
/// GPU memory manager owning memory management resources.
@@ -354,6 +406,14 @@ pub(crate) struct Gpu<'gpu> {
/// Must be kept declared *after* `gsp_resources`, as the latter's `PinnedDrop` implementation
/// requires the sysmem flush page to be in place.
sysmem_flush: SysmemFlush<'gpu>,
+ /// Borrow of `tree` that `_gsp_subtree` holds. A field that borrows a sibling field is
+ /// self-referential, which `pin_init` cannot express, so the borrow is taken by hand.
+ tree_ref: &'gpu Tree<'gpu>,
+ /// The GIN CPU interrupt tree and the PCI vectors that deliver it.
+ ///
+ /// Must be kept declared *after* `_gsp_subtree`, which holds a borrow of it.
+ #[pin]
+ tree: Tree<'gpu>,
}
#[pinned_drop]
@@ -397,6 +457,12 @@ pub(crate) fn new<'a>(
dev_info!(dev,"NVIDIA ({})\n", spec);
})?,
+ tree: Tree::new(pdev, bar, spec.chipset, irq::gsp::GSP_SUBTREE.into())?,
+
+ // SAFETY: `tree` is initialized above, is pinned at a stable address, and is dropped
+ // after every field that uses `tree_ref` (struct field drop order).
+ tree_ref: unsafe { &*core::ptr::from_ref(tree.as_ref().get_ref()) },
+
_: {
let dma_mask = hal::gpu_hal(spec.chipset).dma_mask();
@@ -422,12 +488,7 @@ pub(crate) fn new<'a>(
bar,
- gsp_falcon: Falcon::new(
- dev,
- spec.chipset,
- bar
- )
- .inspect(|falcon| falcon.clear_swgen0_intr())?,
+ gsp_falcon: Falcon::new(dev, spec.chipset, bar)?,
sec2_falcon: Falcon::new(dev, spec.chipset, bar)?,
@@ -451,6 +512,29 @@ pub(crate) fn new<'a>(
})?,
}),
+ _: {
+ irq::gsp::quiesce(tree_ref, &gsp_resources.gsp_falcon);
+ },
+
+ // SAFETY: the GSP falcon and the command queue are fields of `gsp_resources`, which
+ // is initialized above and pinned, so both references outlive the registration. The
+ // registration is a field of `Gpu` and is never leaked, so its `Drop` runs, and field
+ // drop order runs it before either is freed.
+ _gsp_subtree <- unsafe {
+ GspSubtree::new(
+ pdev,
+ tree_ref,
+ &*core::ptr::from_ref(&gsp_resources.gsp_falcon),
+ &*core::ptr::from_ref(&gsp_resources.gsp.cmdq),
+ )
+ },
+
+ // No interrupt announces the messages that the GSP posted during boot, before the
+ // SWGEN0 latch was cleared.
+ _: {
+ gsp_resources.gsp.cmdq.drain()?;
+ },
+
gsp_static_info: {
// Obtain and display basic GPU information.
let info = gsp_resources.gsp.get_static_info()?;
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index ae3808de44e2..1b2f346a237f 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -629,6 +629,20 @@ pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
{
self.inner.lock().await_msg()
}
+
+ /// Logs and consumes every message the GSP has already posted, and returns without waiting for
+ /// more.
+ ///
+ /// No caller is waiting for a reply while this holds the queue mutex, so every message is
+ /// logged as an event. See "Draining the GSP-to-CPU queue" in
+ /// `Documentation/gpu/nova/core/interrupts.rst`.
+ ///
+ /// # Errors
+ ///
+ /// `EIO` if a message fails framing or checksum validation.
+ pub(crate) fn drain(&self) -> Result {
+ self.inner.lock().drain()
+ }
}
/// Inner mutex protected state of [`Cmdq`].
@@ -941,4 +955,30 @@ fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
}
}
}
+
+ /// Logs and consumes every message the queue holds.
+ ///
+ /// # Errors
+ ///
+ /// `EIO` if a message fails framing or checksum validation, or if a message's page count
+ /// overflows a `u32`.
+ fn drain(&mut self) -> Result {
+ while !self.gsp_mem.driver_read_area().0.is_empty() {
+ // A message is available, so this returns without waiting.
+ let msg = self.wait_for_msg(Delta::ZERO)?;
+
+ let pages =
+ u32::try_from(msg.header.length().div_ceil(GSP_PAGE_SIZE)).map_err(|_| {
+ dev_err!(&self.dev, "GSP drain: message length overflow\n");
+ EIO
+ })?;
+ let function = msg.header.function();
+ let seq = msg.header.sequence();
+
+ self.gsp_mem.advance_cpu_read_ptr(pages);
+ self.log_event(function, seq);
+ }
+
+ Ok(())
+ }
}
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index ff8b00a82442..02f9bd84dd19 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -11,6 +11,7 @@
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(crate) mod doorbell_test;
+pub(crate) mod gsp;
mod hal;
pub(crate) mod interrupt_tree;
mod regs;
diff --git a/drivers/gpu/nova-core/irq/gsp.rs b/drivers/gpu/nova-core/irq/gsp.rs
new file mode 100644
index 000000000000..bb504e104bb8
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/gsp.rs
@@ -0,0 +1,196 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! The GSP event interrupt.
+//!
+//! The GSP posts messages to the GSP-to-CPU queue and raises SWGEN0, a software-generated cause
+//! of its falcon. A threaded handler services it: the top half clears the tree and falcon state,
+//! and the IRQ thread drains the queue.
+//!
+//! See "The GSP event" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+use kernel::{
+ device,
+ irq,
+ pci,
+ prelude::*, //
+};
+
+use super::interrupt_tree::{
+ GinVector,
+ LeafEnableGuard,
+ Subtree,
+ Tree, //
+};
+use crate::{
+ falcon::{
+ gsp::Gsp as GspFalcon,
+ Falcon, //
+ },
+ gsp::cmdq::Cmdq,
+ regs, //
+};
+
+/// The GSP event vector, which has the same number on every supported GPU.
+const GSP_INTR_0_VECTOR: GinVector = GinVector::new::<155>();
+
+/// The GSP event's subtree, the only one that nova-core services.
+pub(crate) const GSP_SUBTREE: Subtree = GSP_INTR_0_VECTOR.subtree();
+
+/// Clears the tree and falcon interrupt state that GSP boot leaves behind, and rearms PCI
+/// interrupt delivery.
+///
+/// On return, no vector is enabled at its leaf, the serviced subtrees are disabled at `TOP`, and
+/// the SWGEN0 latch is clear, so the next message that the GSP posts signals the tree.
+pub(crate) fn quiesce(tree: &Tree<'_>, falcon: &Falcon<'_, GspFalcon>) {
+ tree.reset();
+ // The latch is cleared after the tree reset. The other order can leave the latch set with its
+ // leaf bit cleared. See "Enabling the GSP event" in interrupts.rst.
+ falcon.clear_swgen0_intr();
+}
+
+/// Threaded IRQ handler for the GSP event.
+pub(crate) struct GspInterrupt<'a> {
+ falcon: &'a Falcon<'a, GspFalcon>,
+ cmdq: &'a Cmdq<'a>,
+ tree: &'a Tree<'a>,
+ /// For logging. The command queue's device reference is behind its mutex, which the top half
+ /// cannot take.
+ dev: &'a device::Device,
+}
+
+impl GspInterrupt<'_> {
+ /// Clears the latch of every host-routed cause in `status` other than SWGEN0, and logs them.
+ ///
+ /// Returns the causes still set after the clear. A cause driven from outside the falcon stays
+ /// set, and only a device reset ends it.
+ fn clear_faults(
+ &self,
+ status: regs::NV_PFALCON_FALCON_IRQSTAT,
+ ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+ let faults = status.with_swgen0(false);
+ if faults.into_raw() == 0 {
+ return faults;
+ }
+
+ dev_err!(
+ &self.dev,
+ "unserviceable GSP falcon interrupt, IRQSTAT {:#x}\n",
+ status.into_raw()
+ );
+ self.falcon.clear_intr(faults);
+
+ self.falcon.read_host_intr().with_swgen0(false)
+ }
+}
+
+impl irq::ThreadedHandler for GspInterrupt<'_> {
+ /// Top half, in hard interrupt context. Services the GSP vector only, so another vector
+ /// pending in the same leaf stays pending.
+ fn handle(&self) -> irq::ThreadedIrqReturn {
+ let leaf = self.tree.read_pending(GSP_INTR_0_VECTOR.leaf_index());
+ if !leaf.vectors().contains(GSP_INTR_0_VECTOR.leaf_mask()) {
+ self.tree.rearm_pci_irq(GSP_SUBTREE);
+ return irq::ThreadedIrqReturn::None;
+ }
+ leaf.clear_vectors(GSP_INTR_0_VECTOR.leaf_mask());
+
+ let status = self.falcon.take_host_intr();
+
+ let remaining_faults = self.clear_faults(status);
+ if remaining_faults.into_raw() == 0 {
+ self.falcon.retrigger_intr();
+ } else {
+ // Disabling the vector loses no notification: the falcon signals nothing further
+ // while a cause stays set. See "Retriggering a falcon" in interrupts.rst.
+ self.tree.disable_leaf(
+ GSP_INTR_0_VECTOR.leaf_index(),
+ GSP_INTR_0_VECTOR.leaf_mask(),
+ );
+ dev_err!(
+ &self.dev,
+ "GSP falcon cause {:#x} needs a device reset, GSP events are no longer serviced\n",
+ remaining_faults.into_raw()
+ );
+ }
+
+ self.tree.rearm_pci_irq(GSP_SUBTREE);
+
+ if status.swgen0() {
+ irq::ThreadedIrqReturn::WakeThread
+ } else {
+ irq::ThreadedIrqReturn::Handled
+ }
+ }
+
+ /// IRQ thread. Drains the GSP-to-CPU queue, which may sleep.
+ fn handle_threaded(&self) -> irq::IrqReturn {
+ if let Err(e) = self.cmdq.drain() {
+ // The failed message stays at the queue head, so every later drain fails the same way.
+ self.tree.disable_leaf(
+ GSP_INTR_0_VECTOR.leaf_index(),
+ GSP_INTR_0_VECTOR.leaf_mask(),
+ );
+ dev_err!(
+ &self.dev,
+ "GSP event drain failed ({:?}), the message queue is no longer serviced\n",
+ e
+ );
+ }
+ irq::IrqReturn::Handled
+ }
+}
+
+/// The registered GSP event handler and the enable of its vector.
+///
+/// The declaration order is the drop order, and it is required: the vector is disabled before
+/// `free_irq` runs. See "Enabling the GSP event" in `Documentation/gpu/nova/core/interrupts.rst`.
+#[pin_data]
+pub(crate) struct GspIrq<'a> {
+ _leaf_guard: LeafEnableGuard<'a>,
+ #[pin]
+ reg: irq::ThreadedRegistration<'a, GspInterrupt<'a>>,
+}
+
+impl<'a> GspIrq<'a> {
+ /// Returns an initializer that registers the threaded handler and then enables the GSP
+ /// vector at its leaf.
+ ///
+ /// An event that latched while the vector was disabled is delivered as soon as the GSP
+ /// subtree is enabled at `TOP`.
+ ///
+ /// # Errors
+ ///
+ /// `EINVAL` if `tree` does not service the GSP subtree. Otherwise the error from
+ /// `request_threaded_irq`.
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the initialized `GspIrq` or otherwise prevent its [`Drop`]
+ /// implementation, which runs `free_irq`, from running.
+ pub(crate) unsafe fn new(
+ pdev: &'a pci::Device<device::Bound>,
+ tree: &'a Tree<'a>,
+ falcon: &'a Falcon<'a, GspFalcon>,
+ cmdq: &'a Cmdq<'a>,
+ ) -> impl PinInit<Self, Error> + 'a {
+ let dev = pdev.as_ref();
+
+ try_pin_init!(Self {
+ // SAFETY: this function's caller must not leak the `GspIrq` that owns this
+ // registration, so the registration's `Drop` runs.
+ reg <- unsafe {
+ irq::ThreadedRegistration::new(
+ tree.request_for(GSP_SUBTREE)?,
+ irq::Flags::TRIGGER_NONE,
+ c"nova-core",
+ Ok(GspInterrupt { falcon, cmdq, tree, dev }),
+ )
+ },
+ _leaf_guard: tree.enable_leaf_guarded(
+ GSP_INTR_0_VECTOR.leaf_index(),
+ GSP_INTR_0_VECTOR.leaf_mask(),
+ ),
+ })
+ }
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 4bb27cc8b6cd..90d4d11e8ba3 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -124,10 +124,12 @@ pub(super) const fn all() -> Self {
Self(u32::MAX)
}
+ #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) const fn from_raw(raw: u32) -> Self {
Self(raw)
}
+ #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) const fn into_raw(self) -> u32 {
self.0
}
@@ -240,6 +242,7 @@ pub(super) const fn new<const VECTOR: u32>() -> Self {
Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
}
+ #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) const fn into_raw(self) -> u32 {
self.0.get()
}
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 8c0761dedda4..a8f6dc22057b 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -18,7 +18,6 @@
mod fsp;
mod gpu;
mod gsp;
-#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
mod irq;
mod mctp;
mod mm;
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 14/15] gpu: nova-core: add KUnit tests for the interrupt tree
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (12 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 13/15] gpu: nova-core: service GSP events from the SWGEN0 interrupt John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
2026-09-30 3:41 ` [PATCH v5 15/15] gpu: nova-core: document the GIN interrupt controller and GSP events John Hubbard
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The vector arithmetic and the leaf and subtree sets touch no hardware,
so KUnit can cover them without a GPU. A wrong leaf, bit or subtree for
a vector would otherwise show up only as a lost interrupt on hardware.
Add the nova_core_gin_tree suite. It checks that a leaf index stops at
the widest supported tree, that a leaf count implies the right number of
subtrees and vectors, that a leaf count enumerates every leaf in order,
that a vector maps to the right leaf, bit and subtree, and that a vector
beyond an 8-leaf tree is rejected there and accepted in a 16-leaf tree.
It also exercises the subtree set operations and checks that every
supported chipset implements the subtree that carries the GSP event.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
drivers/gpu/nova-core/irq/interrupt_tree.rs | 128 +++++++++++++++++++-
1 file changed, 127 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 90d4d11e8ba3..590f921894b6 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -129,7 +129,10 @@ pub(super) const fn from_raw(raw: u32) -> Self {
Self(raw)
}
- #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
+ #[cfg_attr(
+ not(any(CONFIG_NOVA_CORE_SELFTESTS, CONFIG_KUNIT = "y")),
+ expect(dead_code)
+ )]
pub(super) const fn into_raw(self) -> u32 {
self.0
}
@@ -554,3 +557,126 @@ fn drop(&mut self) {
clear_top_enables(self.bar, self.serviced);
}
}
+
+#[kunit_tests(nova_core_gin_tree)]
+mod tests {
+ use super::*;
+
+ /// A leaf index cannot name a leaf beyond the widest supported tree.
+ #[test]
+ fn leaf_index_bounds() {
+ assert!(LeafIndex::try_new(0).is_some());
+ assert!(LeafIndex::try_new(15).is_some());
+ assert!(LeafIndex::try_new(16).is_none());
+ }
+
+ /// The subtree count, the implemented-subtree set, and the vector count follow the leaf count.
+ #[test]
+ fn leaf_count_derives_subtrees_and_vectors() {
+ assert_eq!(LeafCount::Eight.subtree_count(), 4);
+ assert_eq!(
+ Bounded::<u32, 32>::from(LeafCount::Eight.subtree_set()).get(),
+ 0x0f
+ );
+ assert_eq!(LeafCount::Eight.vector_count(), 256);
+
+ assert_eq!(LeafCount::Sixteen.subtree_count(), 8);
+ assert_eq!(
+ Bounded::<u32, 32>::from(LeafCount::Sixteen.subtree_set()).get(),
+ 0xff
+ );
+ assert_eq!(LeafCount::Sixteen.vector_count(), 512);
+ }
+
+ /// A tree enumerates every leaf that it implements, in order, and no more.
+ #[test]
+ fn leaf_count_iter_covers_the_tree() {
+ for (count, expected) in [(LeafCount::Eight, 8usize), (LeafCount::Sixteen, 16)] {
+ let mut seen = 0;
+
+ for (index, leaf) in count.iter().enumerate() {
+ assert_eq!(leaf.get(), index);
+ seen += 1;
+ }
+
+ assert_eq!(seen, expected);
+ }
+ }
+
+ /// A vector maps to its leaf, its bit within that leaf, and its subtree. The doorbell (129)
+ /// and the GSP event (155) share a subtree.
+ #[test]
+ fn vector_maps_to_leaf_bit_and_subtree() {
+ let doorbell = GinVector::new::<129>();
+ let gsp = GinVector::new::<155>();
+
+ assert_eq!(doorbell.leaf_index().get(), 4);
+ assert_eq!(doorbell.leaf_mask().into_raw(), 1 << 1);
+ assert_eq!(doorbell.subtree().index(), 2);
+
+ assert_eq!(gsp.leaf_index().get(), 4);
+ assert_eq!(gsp.leaf_mask().into_raw(), 1 << 27);
+ assert_eq!(gsp.subtree().index(), 2);
+
+ assert_eq!(doorbell.subtree(), gsp.subtree());
+ }
+
+ /// Both fixed vectors are within the 8-leaf tree, so every supported chipset implements them.
+ #[test]
+ fn fixed_vectors_fit_the_narrowest_tree() {
+ assert!(GinVector::new::<129>().validate(LeafCount::Eight).is_ok());
+ assert!(GinVector::new::<155>().validate(LeafCount::Eight).is_ok());
+
+ // The first vector beyond an 8-leaf tree.
+ assert!(GinVector::new::<256>().validate(LeafCount::Eight).is_err());
+ assert!(GinVector::new::<256>().validate(LeafCount::Sixteen).is_ok());
+ }
+
+ /// A subtree set reports membership, intersection, and its span from subtree 0.
+ #[test]
+ fn subtree_set_operations() {
+ let gsp = GinVector::new::<155>().subtree();
+
+ assert!(LeafCount::Eight.subtree_set().contains(gsp));
+ assert!(!LeafCount::Eight.subtree_set().is_empty());
+
+ // The GSP needs no subtree above 2, so an MSI-X request covers entries 0 through 2.
+ assert_eq!(SubtreeSet::from(gsp).span(), 3);
+
+ // A 16-leaf tree implements every subtree that an 8-leaf tree does.
+ assert_eq!(
+ LeafCount::Sixteen
+ .subtree_set()
+ .intersection(LeafCount::Eight.subtree_set()),
+ LeafCount::Eight.subtree_set()
+ );
+ }
+
+ /// Iterating a subtree set yields each subtree once, lowest index first, and nothing for an
+ /// empty set.
+ #[test]
+ fn subtree_set_iterates_its_members() {
+ assert!(LeafCount::Eight
+ .subtree_set()
+ .iter()
+ .map(Subtree::index)
+ .eq([0u32, 1, 2, 3]));
+
+ let gsp = SubtreeSet::from(GinVector::new::<155>().subtree());
+ assert!(gsp.iter().map(Subtree::index).eq([2u32]));
+
+ let empty = SubtreeSet::from(Bounded::<u32, 32>::new::<0>());
+ assert_eq!(empty.iter().count(), 0);
+ }
+
+ /// Every supported chipset implements the subtree that carries the GSP event.
+ #[test]
+ fn gsp_subtree_is_implemented_everywhere() {
+ for &chipset in Chipset::ALL {
+ assert!(cpu_interrupt_hal(chipset)
+ .leaf_count()
+ .subtree_set()
+ .contains(crate::irq::gsp::GSP_SUBTREE));
+ }
+ }
+}
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 15/15] gpu: nova-core: document the GIN interrupt controller and GSP events
2026-09-30 3:41 [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery John Hubbard
` (13 preceding siblings ...)
2026-09-30 3:41 ` [PATCH v5 14/15] gpu: nova-core: add KUnit tests for the interrupt tree John Hubbard
@ 2026-09-30 3:41 ` John Hubbard
14 siblings, 0 replies; 18+ messages in thread
From: John Hubbard @ 2026-09-30 3:41 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: Timur Tabi, Alistair Popple, Eliot Courtney, Zhi Wang,
David Airlie, Simona Vetter, Bjorn Helgaas, Miguel Ojeda,
Alex Gaynor, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross,
nova-gpu, LKML, John Hubbard
The interrupt code rests on hardware behavior that the code cannot show
on its own: how GIN, the GPU's interrupt controller, records and
delivers interrupts, what edge-triggered delivery requires of a handler,
and how the GSP signals the CPU. Some of those requirements come from
OpenRM rather than from the hardware manuals.
Add a design document that records that behavior, the rules that
nova-core follows because of it, and the terms that the code uses for
it. The code comments cite the document by section rather than repeating
it.
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
Documentation/gpu/nova/core/interrupts.rst | 679 +++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 680 insertions(+)
create mode 100644 Documentation/gpu/nova/core/interrupts.rst
diff --git a/Documentation/gpu/nova/core/interrupts.rst b/Documentation/gpu/nova/core/interrupts.rst
new file mode 100644
index 000000000000..bd6b35aa8dcd
--- /dev/null
+++ b/Documentation/gpu/nova/core/interrupts.rst
@@ -0,0 +1,679 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+=============================================
+GPU interrupt handling: GIN and the GSP event
+=============================================
+
+This document describes how nova-core receives interrupts from the GPU on Turing
+and later chipsets. It covers the GPU Interrupt and Notification unit (GIN),
+which is the GPU's interrupt controller, and the GSP event, the interrupt that
+nova-core services in normal operation.
+
+Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU
+also has on-chip processors that run their own firmware and receive their own
+interrupts. The GSP (GPU System Processor) is one of them.
+
+Register names are the names from the GPU hardware reference headers. The
+pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers
+call it ``NV_GIN``. This document calls it GIN throughout, because the tree that
+nova-core services is the same on every supported chipset. "Register naming"
+at the end says how the names map onto the headers. OpenRM, NVIDIA's
+open-source GPU kernel driver, is cited wherever nova-core follows it.
+
+Terminology
+===========
+
+The three levels of the controller, innermost first:
+
+leaf
+ One ``LEAF`` register. Each of its 32 bits is the pending bit of one
+ interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or
+ Blackwell tree has 16.
+
+subtree
+ Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the
+ unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery:
+ every interrupt from one subtree arrives on one MSI-X entry.
+
+tree
+ One ``TOP`` register and the leaves under it. Every PCIe function has its
+ own tree, and nova-core services the CPU tree of one function.
+
+The hardware headers, OpenRM, and the Linux PCI API all use the word "vector",
+each for a different number. This document gives each one its own name, and a
+bare "vector" always means a GIN vector.
+
+GIN vector
+ The GPU-internal interrupt source number. It addresses one bit of one leaf.
+ A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0
+ through 255. The CPU doorbell is vector 129 and the GSP event is vector
+ 155.
+
+MSI-X entry
+ An index into the device's MSI-X table. One entry serves one subtree.
+
+PCI vector
+ One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X
+ entry, or the single MSI message.
+
+Linux IRQ number
+ What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI
+ vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as
+ well.
+
+The remaining terms, each named for the register or the specification that
+defines it:
+
+enable, disable a vector
+ Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``.
+
+enable, disable a subtree
+ Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``.
+
+serviced subtree
+ A subtree that nova-core enables and has a handler for.
+
+rearm
+ Restoring PCI interrupt delivery after servicing an interrupt. See
+ "Rearming PCI interrupt delivery".
+
+mask
+ Reserved for the two places where hardware and the PCI specification use
+ the word: the MSI-X per-entry Vector Control mask bit, which Linux
+ controls, and the falcon interrupt masks. It never names a GIN enable.
+
+latched, pending
+ Two names for one state, a set ``LEAF`` bit. The vector's source sets the
+ bit whether or not the vector is enabled.
+
+clear a vector
+ Write a 1 to the vector's bit in ``LEAF``. OpenRM calls the same operation
+ ``intrClearLeafVector_HAL``.
+
+pending bits
+ The plain 32-bit value read from a ``LEAF`` register.
+
+notification
+ An interrupt whose only content is that something happened, such as a
+ posted message. Servicing a notification means reading what it announces.
+ The unit that raised it needs no attention. The GSP event is one.
+
+unit
+ Any block that raises an interrupt. "Engine" is reserved for the blocks
+ that do user work: GR, CE, NVDEC, and the like.
+
+falcon
+ One of the GPU's microcontrollers (see
+ Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core
+ inside its falcon. A falcon latches each of its interrupt causes and routes
+ it either to the host, meaning the CPU, or to its own core.
+
+The GIN controller
+==================
+
+A GPU has many interrupt sources: the GSP, the copy engines, the graphics
+engine, video decode and encode, the MMU fault path, timers, and others. GIN
+records which of them are pending and raises the PCI interrupt to the CPU.
+
+Trees
+-----
+
+GIN keeps one tree for each destination it can deliver an interrupt to. The CPU
+has one tree per PCIe function, so the physical function and each virtual
+function have their own. The GSP has a tree, and so do the other on-chip
+processors that receive interrupts. Every tree has the same two-level layout,
+and a function reaches its own tree through the per-function register aperture.
+
+nova-core services the CPU tree of one function. A virtual function's tree
+belongs to that function's driver, and a processor's tree belongs to the
+firmware running on that processor.
+
+The two-level tree
+------------------
+
+A tree is a set of ``LEAF`` registers and one ``TOP`` register.
+
+* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors
+ ``32i`` through ``32i + 31``. A set bit is a pending vector.
+* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``,
+ which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled
+ vector is pending in either leaf.
+
+A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1.
+The other TOP bits read 0. The leaves and subtrees that a chipset has are its
+implemented leaves and subtrees, and "Per-architecture differences" gives the
+counts. For an 8-leaf tree::
+
+ TOP bit 0 -> subtree 0 -> LEAF(0), LEAF(1) vectors 0..63
+ TOP bit 1 -> subtree 1 -> LEAF(2), LEAF(3) vectors 64..127
+ TOP bit 2 -> subtree 2 -> LEAF(4), LEAF(5) vectors 128..191
+ TOP bit 3 -> subtree 3 -> LEAF(6), LEAF(7) vectors 192..255
+
+ LEAF(4), one bit per vector, holds vectors 128..159:
+
+ bit 1 = vector 129 (CPU doorbell)
+ bit 27 = vector 155 (GSP event)
+
+A vector's number fixes its place in the tree::
+
+ leaf = vector / 32
+ bit = vector % 32
+ subtree = leaf / 2
+
+Registers
+---------
+
+nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The
+leaf registers are arrays indexed by leaf number.
+
+* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit
+ clears that vector, and a 0 leaves the bit as it was.
+* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors
+ of leaf ``i``, one bit per vector.
+* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per
+ subtree.
+* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as
+ the vector's own source would. It is write-only. The self-test uses it.
+
+nova-core does not read ``TOP``. "Servicing the tree" says why.
+
+Every set and clear register acts per bit: a 1 performs the action for that
+bit, and a 0 leaves the bit alone. No register needs a read-modify-write.
+
+GIN delivers a vector to the CPU only when its leaf enable bit and its
+subtree's TOP enable bit are both set. The enables do not affect the latch. The
+source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show
+that bit, so reading the leaf is the only way to see it.
+
+How a unit interrupt reaches the CPU
+------------------------------------
+
+A unit does not write a ``LEAF`` register. Each unit has an interrupt control
+register that GSP firmware programs. The control register holds the unit's
+vector, the GFID that identifies the PCIe function whose tree receives the
+interrupt, and one enable bit per destination: the CPU, the GSP, and the other
+on-chip processors. When the unit has an event::
+
+ 1. The unit sends GIN an interrupt message carrying the vector, the GFID,
+ and the destination enables from its control register.
+ 2. In the tree of each destination that the message selects, GIN sets bit
+ (vector % 32) of LEAF(vector / 32).
+ 3. If the vector and its subtree are enabled in the CPU tree, GIN raises
+ the PCI interrupt.
+
+Because firmware assigns the vectors, nova-core does not hardcode which vector
+belongs to which unit, with two exceptions. The hardware headers of every
+supported chipset define the GSP event as vector 155, and GSP firmware's own
+interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper
+hardware fixes that number, and GSP firmware keeps it on Hopper and later.
+nova-core names both by number. A driver can fetch the full unit-to-vector
+table from the GSP by RPC, and nouveau does. nova-core does not, because a
+fixed vector needs no lookup.
+
+Edge-triggered delivery
+-----------------------
+
+A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set
+until the CPU clears it. A source that stays high does not set the bit again.
+
+GIN raises the PCI interrupt for a subtree when the subtree's enabled pending
+state goes from low to high::
+
+ Per vector, in leaf i at bit b:
+ LEAF(i)[b] AND LEAF_EN(i)[b]
+
+ Per subtree N, across leaves 2N and 2N + 1:
+ OR of every enabled pending bit -> TOP[N]
+
+ Delivery for subtree N:
+ TOP[N] AND TOP_EN[N] -> rising edge -> PCI interrupt
+
+``TOP_EN`` applies after the summary, so disabling a subtree stops delivery
+without changing what ``TOP`` reports.
+
+Three consequences:
+
+* Code that must find every pending vector reads the leaves. A vector that
+ latched while disabled is not in ``TOP``.
+* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a
+ new edge. GIN delivers an interrupt for a pending bit left uncleared as soon
+ as its subtree is enabled again.
+* A source that holds its signal high produces no new edge after the CPU
+ clears the leaf bit. Such a source has to re-emit its interrupt. The falcons
+ do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon").
+
+Delivery over PCI
+=================
+
+GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux
+grants. nova-core requests MSI-X first and falls back to MSI, and never uses
+INTx.
+
+MSI has a single message, and every subtree raises that one message, so one
+Linux IRQ serves the whole tree.
+
+MSI-X gives each subtree its own table entry, at the index equal to the subtree
+number. Linux masks every entry until a driver requests its Linux IRQ number,
+and a masked entry sends no message: the GPU records the interrupt in the MSI-X
+pending bit array, where it stays until Linux unmasks the entry. An entry that
+the driver never requests is never unmasked. A driver that enables a subtree
+without requesting that subtree's entry loses every interrupt from that
+subtree, with nothing reported: the leaf and TOP registers show the vector
+pending and enabled while no handler runs.
+
+The serviced-subtree invariant
+------------------------------
+
+Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered
+handler.
+
+MSI satisfies this with its single message. MSI-X needs one allocated entry per
+serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests
+entries 0 through the highest serviced subtree::
+
+ MSI-X, with subtree 2 serviced:
+
+ subtree 0 -> entry 0 allocated, no handler, stays masked
+ subtree 1 -> entry 1 allocated, no handler, stays masked
+ subtree 2 -> entry 2 handler here, and its rearm covers subtree 2
+
+ MSI, with any serviced set:
+
+ every serviced subtree -> the one allocated PCI vector, whose
+ handler's rearm covers the whole serviced set
+
+An allocated entry whose subtree nova-core does not service costs nothing. The
+entry stays masked, and a disabled subtree raises no interrupt.
+
+nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which
+is in subtree 2. OpenRM's headers place its UVM_SHARED interrupt category in
+subtree 2 on every chipset that nova-core supports. The self-test doorbell,
+vector 129, is in the same leaf, and the test allocates its own vectors for it
+(see "Self-test").
+
+Rearming PCI interrupt delivery
+-------------------------------
+
+A message-signaled interrupt is delivered once per edge, and the PCI side
+delivers no further interrupt until the CPU rearms it. The rearm operation
+depends on the GPU family and on the interrupt type that Linux granted:
+
+================== ===== ===========================================
+Architecture Type Rearm operation
+================== ===== ===========================================
+Turing through Ada MSI write the configuration-mirror EOI register
+Hopper and later MSI clear then set the serviced TOP enables
+Any MSI-X clear then set the handler's own TOP enable
+================== ===== ===========================================
+
+The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI
+configuration space, and the value written does not matter. The ``TOP_EN``
+cycle produces a new delivery edge. The MSI forms cover every serviced subtree,
+because one message serves all of them. The MSI-X form covers one subtree,
+because each serviced subtree has its own entry and its own handler.
+
+A handler rearms once per delivered interrupt, on every path, including the
+path where it finds its vector not pending. A handler that skips the rearm
+receives no further interrupts.
+
+OpenRM makes the same split. It writes the configuration-space EOI for MSI on
+pre-Hopper chipsets, and cycles the TOP enables of the subtrees that it
+services for Hopper-plus MSI and for MSI-X.
+
+Servicing the tree
+==================
+
+Servicing a leaf has a required order: read its pending bits, then clear them.
+Clearing a leaf before reading it discards every vector latched in it, and no
+register reports the loss. In nova-core, reading a leaf produces the handle that
+clears it, so that the wrong order does not compile. The handle clears exactly
+the bits that it read, so a vector that latched after the read stays pending.
+
+A handler clears its bit before it services the vector. Clearing afterwards
+would discard an interrupt that the source raised while the handler ran.
+
+nova-core services the tree in two ways.
+
+The notification path services one vector. It reads the vector's leaf, clears
+only the vector's bit, and rearms. The subtree stays enabled, and a vector
+pending beside it in the same leaf keeps its bit set for the code that services
+that vector. The GSP event handler takes this path, and so does the self-test
+handler.
+
+The startup drain walks the whole tree, because it must clear whatever is
+pending across every subtree rather than one known vector. It disables the
+serviced subtrees at ``TOP``, reads and clears every implemented leaf, and
+leaves the subtrees disabled for its caller to enable once the caller is ready
+for deliveries. The drain reads every leaf rather than descending from ``TOP``,
+because sources latch vectors during boot while those vectors are disabled, and
+``TOP`` does not show them. OpenRM's stall-interrupt path reads every leaf for
+the same reason.
+
+The two paths as register operations::
+
+ Startup drain, run once during probe:
+ write TOP_EN_CLEAR = serviced stop new deliveries
+ for each implemented leaf i:
+ pending = read LEAF(i)
+ write LEAF(i) = pending clear what was read
+ (returns with TOP_EN still clear)
+
+ Notification, the subtree stays enabled:
+ pending = read LEAF(leaf) is the handler's bit set?
+ write LEAF(leaf) = bit clear that one bit
+ rearm PCI interrupt delivery
+
+The drain clears every pending bit, including bits that nova-core never
+services. An uncleared bit holds its subtree in the pending state, and enabling
+that subtree again would deliver an interrupt for a vector that no handler
+services.
+
+The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through
+the configuration mirror, which the drain never writes. An interrupt delivered
+before probe had no handler to rearm it, so the tree reset rearms explicitly
+after the drain. A TOP-enable rearm leaves the serviced subtrees enabled, so
+the reset then disables them again.
+
+nova-core does not serialize access to the tree. The GSP event handler
+touches only its own leaf, and the drain runs during probe, before that handler
+is registered.
+
+Per-architecture differences
+============================
+
+The tree is the same on every supported GPU except for its size, which changes
+at Hopper:
+
+=================== ====== ======== ====================
+GPUs Leaves Subtrees Implemented subtrees
+=================== ====== ======== ====================
+Turing, Ampere, Ada 8 4 ``0x0f``
+Hopper, Blackwell 16 8 ``0xff``
+=================== ====== ======== ====================
+
+The interrupt HAL provides the leaf count, and the subtree count and the
+implemented-subtree set derive from it. A subtree that the chipset does not
+implement has no TOP bit, so building a tree that services one fails with
+``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported
+chipset has them.
+
+OpenRM's headers assign every interrupt category of a 16-leaf tree to leaves 0
+through 11. The drain reads all 16, because a vector can be latched in any
+implemented leaf.
+
+The HAL performs the rearm as well (see "Rearming PCI interrupt delivery"). Two
+falcon properties also differ by family, and the falcon HAL carries them:
+Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing registers
+moved at GA102 (see "Retriggering a falcon").
+
+The GSP event
+=============
+
+When the GSP has output for the CPU, it writes messages into the GSP-to-CPU
+queue in shared memory and raises SWGEN0, one of the software-generated
+interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector
+155, leaf 4 bit 27, in subtree 2.
+
+The queue carries notifications (log records, error records, lifecycle events)
+and command replies. A thread waiting for a reply reads the queue itself, so
+the interrupt is only the trigger to drain the queue (see "Draining the
+GSP-to-CPU queue").
+
+The falcon latches every cause that it raises, SWGEN0 among them, in its
+``IRQSTAT`` register. The handler services the host-routed causes and clears
+their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon re-emits
+any cause that latched in the meantime. "Retriggering a falcon" has the details.
+
+Draining the queue takes the command-queue mutex and walks shared memory, so it
+cannot run in hard interrupt context. nova-core registers a threaded handler,
+under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard
+interrupt context and reads and writes only registers, and it wakes the IRQ
+thread to drain the queue::
+
+ GSP writes messages into the GSP-to-CPU queue
+ GSP raises SWGEN0
+ GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending
+ PCI interrupt -> Linux IRQ -> top half, in hard interrupt context:
+ read LEAF(4), and if bit 27 is clear, rearm and return
+ clear bit 27 (the subtree stays enabled)
+ read the falcon causes routed to the host, clearing SWGEN0 if set
+ for every other host cause: log it, clear its latch, and read the
+ host causes back
+ if the clear ended all of them: retrigger the falcon
+ otherwise: disable vector 155 at its leaf and skip the retrigger
+ rearm PCI interrupt delivery
+ wake the IRQ thread if SWGEN0 was set
+ IRQ thread, which may sleep:
+ take the command-queue mutex and drain the GSP-to-CPU queue
+
+A halt and a posted message can be pending together, so the top half services
+every cause that the status reports.
+
+A drain fails when a message's framing or checksum is bad (see "Draining the
+GSP-to-CPU queue"). The message stays at the queue head, so every later drain
+would fail the same way, and the IRQ thread disables vector 155 and logs the
+failure, which leaves the queue unserviced until the device is reset.
+
+The handler, the self-test, and the rest of the driver read BAR0 through one
+shared mapping. nova-core unregisters an interrupt handler when the device
+unbinds, so a handler runs only while the mapping exists.
+
+Retriggering a falcon
+---------------------
+
+A falcon signals the tree when its set of host-routed causes goes from empty to
+non-empty. A cause left latched keeps the set non-empty, so no later cause
+signals the tree, and the vector is lost. For a cause that stays latched, the
+handler can clear the tree leaf first or the falcon latch first, and the loss
+is the same.
+
+``IRQSTAT`` latches every cause in the falcon, including the causes routed to
+the falcon's own RISC-V core and owned by the firmware running on it. A host
+handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and
+``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both
+before it reads or clears a cause. OpenRM computes the same intersection in
+``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing
+registers and GA102 moves them, so the offsets change at GA102 rather than at
+the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is
+read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on
+a RISC-V falcon.
+
+``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the
+tree, which supplies the transition that clearing the leaf lost. The handler
+writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it
+read, because a re-emitted cause that stays set arrives again at once and
+on every pass after that.
+
+``IRQSCLR`` ends a latch and does not end the source behind it, so a cause
+driven from outside the falcon stays set after the write. On Blackwell the
+fault-containment and ECC causes are driven that way: they appear in
+``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and
+``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler
+clears the latch of every host cause other than SWGEN0, reads the host causes
+back, and retriggers only when the read-back is empty. When a cause is still
+set, the handler disables vector 155 instead and reports that the device needs
+a reset. Disabling loses no notification: the cause that is still set holds the
+host-routed set non-empty, so the falcon would signal nothing further either
+way. OpenRM makes the same choice, and ``kgspService_TU102`` skips
+``kflcnIntrRetrigger`` once it has recorded a fatal error.
+
+The handler cannot distinguish a fault cause that arrives after the clear from
+one that the clear failed to end, so it disables the vector in that case too.
+Both mean that the GSP has faulted.
+
+Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create
+a transition that it has lost. It must leave no host cause latched: it reads the
+host-routed status once and takes every cause that the status reports, rather
+than stopping at the first one that it recognizes. One window stays open. A
+cause that arrives after the handler has read the status is not in the value
+that the handler clears, so it stays latched after the leaf has been cleared,
+and no later cause from that falcon signals the tree. OpenRM has the same window
+on Turing, where ``kflcnIntrRetrigger`` does nothing.
+
+Enabling the GSP event
+----------------------
+
+SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays
+set. nova-core's GSP boot code consumes the GSP's notifications by polling the
+queue, which leaves the latch set and leaves pending bits in the tree. The
+handoff from polling to interrupts has a required order::
+
+ disable every implemented vector drop enables left by boot, or by a
+ driver that ran before this one
+ drain the tree clear stale pending bits
+ rearm PCI interrupt delivery required under pre-Hopper MSI, which
+ no later step rearms
+ disable subtree 2 at TOP a TOP-enable rearm enabled it again
+ clear the SWGEN0 latch so the next message makes an edge
+ register the threaded handler no delivery can reach it yet
+ enable vector 155 at LEAF(4) the subtree is still disabled
+ enable subtree 2 at TOP deliveries become possible here
+ drain the GSP-to-CPU queue messages posted before the clear
+
+The first four steps are the tree reset. nova-core runs them, and clears the
+latch, before it registers the handler. Registering unmasks the PCI interrupt,
+and GIN would then deliver a vector that boot left enabled to a handler that
+services one vector and has no way to service any other. OpenRM clears every
+leaf enable at the same point for the same reason.
+
+nova-core clears the latch after the drain. If nova-core cleared the latch
+first, a message posted before the drain could set it again, along with bit 27
+of LEAF(4). The drain would then clear the leaf bit while the latch stays set,
+and no later message would signal the tree. Clearing after the drain can
+instead leave the leaf bit pending with the latch already clear. Enabling the
+subtree then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half
+clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue
+drain that follows reads the message.
+
+Clearing the latch makes the first interrupt possible. A message that the GSP
+posted before that clear produces no interrupt, so the sequence ends by
+draining the queue.
+
+The subtree is enabled at ``TOP`` only once the handler is registered. A
+TOP-enable rearm sets the enables that it cycles, so the reset disables the
+serviced subtrees again after its rearm, and the tree reaches the registration
+with subtree 2 disabled under every rearm method. On teardown nova-core disables
+the vector at its leaf, so that no vector in the subtree can be delivered, then
+calls ``free_irq()``, and disables the subtree last. Disabling the subtree
+earlier would let a handler still in flight enable it again through the
+``TOP_EN`` cycle of its rearm, which would leave the subtree enabled with no
+handler registered. The driver's ``Gpu`` object, which owns the tree, holds the
+subtree enable, and a handler's registration holds only the enable of its own
+vector, so that tearing down one handler does not disable a subtree that another
+handler shares. nova-core tears down the registration before it frees the queue
+that the handler drains and the falcon that the handler reads, and before it
+unloads the GSP.
+
+Draining the GSP-to-CPU queue
+-----------------------------
+
+The queue carries command replies and unsolicited events, and a message's
+function code says which it is.
+
+* A function code that matches the awaited reply: the message is decoded and
+ returned to the caller that sent the command.
+* Anything else is an event. An OS error record and a robust-channel record
+ are logged at error level, and an unrecognized function code at warning
+ level. The other known events (GSP logs, libos prints, assertion records,
+ lifecycle notices) need no action and get no line of their own, because the
+ receive trace at debug level already records every message's arrival with
+ its sequence number, function code, and length.
+
+The sequence number takes no part in the match, because the GSP does not echo
+the command's sequence number on every reply. On r570 the reply to
+``UnloadingGuestDriver`` carries sequence 0.
+
+The read pointer advances past every message that passes framing and checksum
+validation, whether it matched or was an event. A matched message that is too
+short to decode is the exception, and it stays at the queue head.
+
+A message's length is inside the region that the checksum covers, so once the
+framing or the checksum fails there is no trustworthy length with which to skip
+the message. Such a message also stays at the queue head, and every later
+receive fails on it with ``EIO`` until the device is reset.
+
+The polling path and the IRQ thread both read the queue under the command-queue
+mutex. Replies and events share one queue and one read pointer, so one lock is
+held across the whole drain. A thread waiting for a reply logs each event that
+arrives before the reply and keeps waiting. One deadline of 5 seconds applies to
+the whole wait, rather than a fresh timeout after each message, and the thread
+holds the mutex for the whole wait, so that no other caller consumes the message
+that it waits for.
+
+With one lock, a drain waits for an in-flight command's receive to finish or
+time out. For log and error records that delay does not matter.
+
+Self-test
+=========
+
+The self-test confirms that an interrupt injected at the GPU is delivered to a
+registered handler. It runs during probe, after the GPU's boot firmware, GFW,
+has completed, and before nova-core boots the GSP, because the test disables
+and drains the whole tree, including the GSP's leaf. The test is built only
+under ``CONFIG_NOVA_CORE_SELFTESTS``, like the other probe-time hardware tests.
+Those other tests log a failure and let probe continue. A failed delivery test
+fails probe, because an interrupt path that does not work leaves the driver
+unable to make progress later, in a place that says nothing about the cause.
+
+The test writes ``LEAF_TRIGGER`` with vector 129, the CPU doorbell, at leaf 4
+bit 1. The vector then takes the ordinary path to the CPU under the ordinary
+enables. The doorbell keeps the same number on every supported chipset, so the
+test names it without asking the GSP, which is not running yet.
+
+The test allocates PCI vectors for subtree 2 and resets the tree, which
+disables every vector, drains the tree, and rearms delivery. It checks that the
+doorbell bit starts out clear. It registers a non-threaded handler with a
+completion, under the name ``nova-core-selftest`` in ``/proc/interrupts``, and
+enables the vector and the subtree. It triggers the doorbell, waits up to 1000
+ms for the first delivery, triggers it again, and waits up to 1000 ms for the
+second. The handler takes the notification path: it clears only its own bit and
+rearms, and never walks the tree.
+
+The test triggers twice, and the second trigger waits for the first handler to
+finish. One delivery would prove nothing about the rearm, because the first
+message-signaled interrupt arrives whether the driver rearms or not, and two
+triggers in a row could coalesce into one delivery. A handler that walked the
+tree would prove nothing either: on every configuration except pre-Hopper MSI
+the rearm is a ``TOP_EN`` cycle, so a walk that enabled ``TOP`` again would
+rearm delivery whether the handler asked for it or not.
+
+The test passes only if both deliveries arrive, each finds the doorbell bit and
+nothing else pending in leaf 4, and the bit is clear once the vector is
+disabled. Anything else fails probe. Requiring the exact pending bits on the
+second delivery shows that the first handler's clear reached the hardware. No
+other vector in leaf 4 can be pending, because the test disabled every vector
+and runs before GSP boot.
+
+The test releases its PCI vectors and its handler before returning, so the
+driver's own allocation covers the GSP subtree and nothing else. Sharing the
+driver's allocation would have let the test pass only because the doorbell and
+the GSP event happen to be in the same subtree.
+
+The test exercises the path from the GPU to the handler without GSP firmware,
+which helps when bringing up PCI, MSI, MSI-X, and passthrough setups. Under
+MSI-X a pass also shows that the delivery arrived on the entry belonging to the
+serviced subtree.
+
+The vector arithmetic and the leaf and subtree sets depend on no hardware, and
+KUnit tests cover them instead.
+
+Register naming
+===============
+
+nova-core uses the ``NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_*`` names for the CPU
+tree on every supported chipset. That is the per-function aperture: each PCIe
+function reaches its own tree through it, at the same offsets. The controller
+also has a central aperture that exposes every function's tree. The pre-Hopper
+headers name it ``NV_CTRL_CPU_INTR_*`` and the Hopper-plus headers
+``NV_GIN_CPU_INTR_*``. nova-core does not use it. OpenRM's kernel-side code
+for the controller is the ``Intr`` object.
+
+Scope
+=====
+
+nova-core services the CPU tree of one function and nothing else. It implements
+no virtual-function tree management and no GFID routing, which belong to the
+physical function's driver or to firmware in a virtualized setup, and no MIG
+(multi-instance GPU) support. It services none of the subtrees that the
+hardware headers reserve for the stall interrupts of the host-driven engines.
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 59b206238498..224caef9ea42 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -35,3 +35,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/falcon
core/tlv
core/pramin
+ core/interrupts
--
2.55.0
^ permalink raw reply [flat|nested] 18+ messages in thread