[PATCH v4 00/17] nova-core: GPU interrupt support and GSP event delivery

[PATCH v4 00/17] nova-core: GPU interrupt support and GSP event delivery

[John Hubbard]

This series adds support for GIN, the GPU Interrupt and Notification
unit, which is the GPU's interrupt controller, so that GSP events reach
the driver as interrupts rather than only when the driver polls for
them.

The handler is threaded. The top half touches only GPU registers, and
the IRQ thread drains the GSP-to-CPU message queue.

This is based on drm-rust-next at d669686f8170 ("gpu: nova-core: mm: Add
BAR1 memory management self-tests"), which now includes the PRAMIN and
BAR1 memory management series from Joel and Eliot. The PRAMIN series
brought the NOVA_CORE_SELFTESTS option and the assertion macros for
probe-time hardware tests, and the interrupt self-test uses both.

Patches 7, 11 and 13 are new, so v4 has 17 patches where v3 had 14.

Changes in v4, at a high level:

* Rebased onto the current drm-rust-next.

* The comments, doc comments and commit messages are rewritten across
  the series, and the design document is reworked.

* A GSP fault no longer hangs the CPU. The top half reads the falcon
  causes back after clearing them, and disables the GSP vector instead
  of retriggering the falcon when one is still set.

* The self-test allocates its own vectors and runs from probe, after
  the GFW boot wait, which a new patch moves out of the Gpu
  constructor. It shares the NOVA_CORE_SELFTESTS option and the
  assertion macros with the memory management tests.

* A GSP message of the wrong type returns ENOMSG instead of ERANGE, in
  a new patch.

* Three pieces moved out of the SWGEN0 patch: the GSP queue drain to a
  new patch, a SubtreeSet method to patch 3, and the falcon interrupt
  HAL to patch 14. The SWGEN0 patch now carries only the handler and
  its registration.

There is a git branch with the patches as applied to drm-rust-next:

    https://github.com/johnhubbard/linux/tree/nova-core-gin-interrupt-tree-v4/

The branch also carries two small bindgen warning fixes of mine below
the series. They are not part of it.

Changes in v4, in more detail:

* I rewrote the comments, doc comments and commit messages across
  patches 3 through 17, and reworked patch 17's design document.

* Rebased onto the current drm-rust-next. Coherent DMA allocations are
  tied to the device's bound lifetime there, so Cmdq and Gsp carry a
  lifetime parameter and the GSP event handler borrows a Cmdq<'_>.

* Patch 6 has Tree::new() take a &SubtreeVectors, rather than the MSI
  type and the serviced subtrees that its callers were unpacking from
  one. (Alex)

* Patch 7 is new. It moves the wait for the GPU's own firmware from the
  Gpu constructor out to the PCI probe, so that patch 8's self-test can
  run from probe as the memory management self-tests do. The interrupt
  self-test runs after that wait and before the GSP boot that the
  constructor does, while the memory tests run after it. (Alex)

* Patch 8 uses NOVA_CORE_SELFTESTS and the selftest_assert macros
  instead of its own Kconfig option. The memory management tests and
  the interrupt test share the option but not the failure behavior: a
  failed interrupt delivery test fails the PCI probe, while a PRAMIN or
  BAR1 failure only logs. The option's help text says so. (Alex)

* Patch 8 also allocates the vectors for the doorbell's own subtree and
  releases them, instead of borrowing the driver's allocation for the
  GSP subtree, which worked only because both vectors are in subtree 2.
  It registers as "nova-core-selftest". Every supported chip has the
  doorbell at the same vector, so the test names it without asking
  GSP-RM. (Alex)

* Patch 9 renames classify_event() to log_event(). The function only
  logs, and never returns the classification that the old name
  promised.

* Patch 10 warns when a GSP message is too short to decode as the
  expected type. (Alex, Gary)

* Patch 11 is new: a message of the wrong type now returns ENOMSG
  instead of ERANGE, which says nothing about a message. Two retry
  loops still matched ERANGE, so the patch converts them to keep GSP
  boot working. Patch 12 removes both loops. (Gary, Alex)

* Patch 12 refactors the deadline loop into one place, and drops a
  reference to a private method from the public documentation. (Alex)

* Patch 13 is new, and carries the GSP message queue drain that v3
  added inside the SWGEN0 patch. (Alex)

* Patch 14 carries the falcon interrupt HAL that v3 put in the SWGEN0
  patch, as a HAL now rather than two functions matching on chipset.
  It is separate from the falcon boot HAL, which the top half cannot
  reach without allocating. (Alex)

* Patch 15 fixes the interrupt storm. IRQSCLR does not end a cause
  driven from outside the falcon, so v3's retrigger re-emitted it at
  once. The top half now reads the causes back after the clear, and
  disables the GSP vector instead of retriggering when one is still
  set. (Sashiko reported it. I have not reproduced it.)

* Patch 15 also returns the host-routed causes as a typed IRQSTAT value,
  and replaces the module-wide expect(dead_code) with a per-item
  cfg_attr on the items that the off-by-default self-test leaves dead.

* Patch 16 tests both falcon interrupt properties through the new HAL.

Will, I kept your Reviewed-by on patches 4, 5 and 6. I dropped it from
the rest, because the comments changed everywhere and patches 8, 14,
15, 16 and 17 changed in substance. Patch 15 is the one I would ask you
to look at first: its top half can now disable the GSP vector instead
of retriggering the falcon.

TESTING: to fill in after the test run. v3 was tested on Turing
(TU117), Ampere (GA104) and Blackwell (GB202), covering probe and
chipset identification, the interrupt self-test, a GSP name query over
the command queue, driver unload/reload, and the KUnit suites. I have
not produced a GSP fault on any of them, so the new fault path is
untested.

Joel Fernandes (2):
  rust: sync: completion: add wait_for_completion_timeout()
  gpu: nova-core: add the GIN interrupt tree and allocate its vectors

John Hubbard (15):
  rust: pci: declare IrqType and IrqTypes with impl_flags
  gpu: nova-core: add the GIN vector, leaf and subtree types
  gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers
  gpu: nova-core: add the per-architecture GIN CPU interrupt HAL
  gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
  gpu: nova-core: add an interrupt delivery self-test
  gpu: nova-core: log GSP events instead of discarding them
  gpu: nova-core: stop re-parsing a bad GSP message
  gpu: nova-core: return ENOMSG for an unmatched GSP message
  gpu: nova-core: bound a GSP wait by a single deadline
  gpu: nova-core: add a GSP message queue drain
  gpu: nova-core: add the falcon interrupt registers and their HAL
  gpu: nova-core: service GSP events from the SWGEN0 interrupt
  gpu: nova-core: add KUnit tests for the interrupt tree and HALs
  gpu: nova-core: document the GIN interrupt controller and GSP events

 Documentation/gpu/nova/core/interrupts.rst  | 674 ++++++++++++++++++++
 Documentation/gpu/nova/index.rst            |   1 +
 drivers/gpu/nova-core/Kconfig               |   5 +
 drivers/gpu/nova-core/driver.rs             |  18 +-
 drivers/gpu/nova-core/falcon/gsp.rs         |  73 ++-
 drivers/gpu/nova-core/falcon/hal.rs         | 124 +++-
 drivers/gpu/nova-core/falcon/hal/ga102.rs   |  21 +-
 drivers/gpu/nova-core/falcon/hal/tu102.rs   |  36 +-
 drivers/gpu/nova-core/gpu.rs                |  85 ++-
 drivers/gpu/nova-core/gsp.rs                |   2 +-
 drivers/gpu/nova-core/gsp/cmdq.rs           | 260 ++++++--
 drivers/gpu/nova-core/gsp/commands.rs       |   8 +-
 drivers/gpu/nova-core/gsp/sequencer.rs      |   8 +-
 drivers/gpu/nova-core/irq.rs                | 149 +++++
 drivers/gpu/nova-core/irq/doorbell_test.rs  | 266 ++++++++
 drivers/gpu/nova-core/irq/gsp.rs            | 236 +++++++
 drivers/gpu/nova-core/irq/hal.rs            | 154 +++++
 drivers/gpu/nova-core/irq/hal/gh100.rs      |  28 +
 drivers/gpu/nova-core/irq/hal/tu102.rs      |  28 +
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 625 ++++++++++++++++++
 drivers/gpu/nova-core/irq/regs.rs           |  87 +++
 drivers/gpu/nova-core/nova_core.rs          |   1 +
 drivers/gpu/nova-core/regs.rs               |  69 ++
 rust/kernel/pci/irq.rs                      |  68 +-
 rust/kernel/sync/completion.rs              |  23 +-
 25 files changed, 2905 insertions(+), 144 deletions(-)
 create mode 100644 Documentation/gpu/nova/core/interrupts.rst
 create mode 100644 drivers/gpu/nova-core/irq.rs
 create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs
 create mode 100644 drivers/gpu/nova-core/irq/gsp.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs
 create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs
 create mode 100644 drivers/gpu/nova-core/irq/regs.rs


base-commit: d669686f8170c234edf12212fac9180ea18b1448
prerequisite-patch-id: 2e07fc4124e1e822f2a4eedf3e814f7e438a6afd
prerequisite-patch-id: 826e07a6bce50fa20188b01e9817fac0c7b6797c
-- 
2.55.0

[PATCH v4 01/17] rust: pci: declare IrqType and IrqTypes with impl_flags

[John Hubbard]

The kernel provides impl_flags! for declaring a bitmask type alongside
the enum of its individual flags, generating the bit operators and the
containment queries.

IrqTypes open-coded that pattern with a with() builder, so a caller
naming two interrupt types chained two calls onto IrqTypes::default().

Declare both types through impl_flags!, so the same set reads as
IrqType::Msi | IrqType::MsiX.

Suggested-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 rust/kernel/pci/irq.rs | 68 +++++++++++++-----------------------------
 1 file changed, 21 insertions(+), 47 deletions(-)

diff --git a/rust/kernel/pci/irq.rs b/rust/kernel/pci/irq.rs
index 6741046ec1c0..f074aad7f1d8 100644
--- a/rust/kernel/pci/irq.rs
+++ b/rust/kernel/pci/irq.rs
@@ -13,27 +13,26 @@
 };
 use core::num::NonZero;
 
-/// IRQ type flags for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy)]
-pub enum IrqType {
-    /// INTx interrupts.
-    Intx,
-    /// Message Signaled Interrupts (MSI).
-    Msi,
-    /// Extended Message Signaled Interrupts (MSI-X).
-    MsiX,
-}
-
-impl IrqType {
-    /// Convert to the corresponding kernel flags.
-    const fn as_raw(self) -> u32 {
-        match self {
-            IrqType::Intx => bindings::PCI_IRQ_INTX,
-            IrqType::Msi => bindings::PCI_IRQ_MSI,
-            IrqType::MsiX => bindings::PCI_IRQ_MSIX,
-        }
+crate::impl_flags!(
+    /// Set of IRQ types that can be used for PCI interrupt allocation.
+    #[derive(Debug, Clone, Copy, Default)]
+    pub struct IrqTypes(u32);
+
+    /// IRQ type flags for PCI interrupt allocation.
+    #[derive(Debug, Clone, Copy)]
+    pub enum IrqType {
+        /// INTx interrupts.
+        Intx = bindings::PCI_IRQ_INTX,
+
+        /// Message Signaled Interrupts (MSI).
+        Msi = bindings::PCI_IRQ_MSI,
+
+        /// Extended Message Signaled Interrupts (MSI-X).
+        MsiX = bindings::PCI_IRQ_MSIX,
     }
+);
 
+impl IrqType {
     /// Construct from raw value.
     #[inline]
     const fn from_raw(raw: u32) -> Self {
@@ -45,33 +44,10 @@ const fn from_raw(raw: u32) -> Self {
     }
 }
 
-/// Set of IRQ types that can be used for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy, Default)]
-pub struct IrqTypes(u32);
-
 impl IrqTypes {
     /// Create a set containing all IRQ types (MSI-X, MSI, and INTx).
     pub const fn all() -> Self {
-        Self(bindings::PCI_IRQ_ALL_TYPES)
-    }
-
-    /// Build a set of IRQ types.
-    ///
-    /// # Examples
-    ///
-    /// ```ignore
-    /// // Create a set with only MSI and MSI-X (no INTx interrupts).
-    /// let msi_only = IrqTypes::default()
-    ///     .with(IrqType::Msi)
-    ///     .with(IrqType::MsiX);
-    /// ```
-    pub const fn with(self, irq_type: IrqType) -> Self {
-        Self(self.0 | irq_type.as_raw())
-    }
-
-    /// Get the raw flags value.
-    const fn as_raw(self) -> u32 {
-        self.0
+        Self(Self::all_bits())
     }
 }
 
@@ -203,9 +179,7 @@ impl Device<device::Bound> {
     /// let vectors = dev.alloc_irq_vectors(1, 32, pci::IrqTypes::all())?;
     ///
     /// // Allocate MSI or MSI-X only (no INTx interrupts).
-    /// let msi_only = pci::IrqTypes::default()
-    ///     .with(pci::IrqType::Msi)
-    ///     .with(pci::IrqType::MsiX);
+    /// let msi_only = pci::IrqType::Msi | pci::IrqType::MsiX;
     /// let vectors = dev.alloc_irq_vectors(4, 16, msi_only)?;
     /// # Ok(())
     /// # }
@@ -222,7 +196,7 @@ pub fn alloc_irq_vectors(
         // - `pci_alloc_irq_vectors` internally validates all other parameters
         //   and returns error codes.
         let ret = unsafe {
-            bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
+            bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, u32::from(irq_types))
         };
         to_result(ret)?;
 
-- 
2.55.0

[PATCH v4 02/17] rust: sync: completion: add wait_for_completion_timeout()

[John Hubbard]

From: Joel Fernandes <joelagnelf@nvidia.com>

A driver that runs an interrupt self-test during probe waits for the
handler to fire. wait_for_completion() has no timeout, so a broken
interrupt path stalls probe indefinitely. Add a timeout variant of
wait_for_completion().

Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: return the remaining jiffies]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/rust/kernel/sync/completion.rs b/rust/kernel/sync/completion.rs
index 35ff049ff078..7e8b3c1c880e 100644
--- a/rust/kernel/sync/completion.rs
+++ b/rust/kernel/sync/completion.rs
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {
+        // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+        match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
+            0 => None,
+            remaining => Some(remaining),
+        }
+    }
 }
-- 
2.55.0

[PATCH v4 03/17] gpu: nova-core: add the GIN vector, leaf and subtree types

[John Hubbard]

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt
controller. Each interrupt source has a GIN vector number, and the
controller latches a pending vector in a two-level tree: one bit of a
LEAF register, summarized two leaves at a time by one bit of the TOP
register. A vector's number fixes its position in that tree:

    leaf    = vector / 32
    bit     = vector % 32
    subtree = leaf / 2

A tree implements either 8 or 16 leaves, depending on the GPU family.
The leaf count sets both the number of subtrees and the highest vector
the tree carries.

Without distinct types, a vector, a leaf index, a set of vectors within
one leaf, a subtree and a set of subtrees are all plain integers.
Nothing stops a caller from passing one where another belongs, or a
register field from accepting the wrong one.

Add a type for each of those, and for the leaf count. A vector converts
to its own leaf, bit and subtree. Its constructor rejects, at build
time, a number beyond the widest supported tree, and a validation
method rejects, at run time, a number beyond the leaves that the
current tree implements. A leaf count yields the set of subtrees it
implements.

Nothing uses the module yet. The following patches declare the tree
registers and the tree itself in terms of these types.

Suggested-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |  12 +
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 238 ++++++++++++++++++++
 drivers/gpu/nova-core/nova_core.rs          |   2 +
 3 files changed, 252 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq.rs
 create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
new file mode 100644
index 000000000000..f1323f633a03
--- /dev/null
+++ b/drivers/gpu/nova-core/irq.rs
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! GPU interrupt support.
+//!
+//! GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller. It latches
+//! every interrupt source in a two-level register tree and delivers the tree to the CPU as a
+//! message-signaled PCI interrupt.
+//!
+//! See `Documentation/gpu/nova/core/interrupts.rst`.
+
+mod interrupt_tree;
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
new file mode 100644
index 000000000000..24976a3146be
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -0,0 +1,238 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Vector addressing in the GIN CPU interrupt tree.
+//!
+//! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
+//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
+//! keep the four from being confused with one another.
+//!
+//! See `Documentation/gpu/nova/core/interrupts.rst`.
+
+use kernel::{
+    num::Bounded,
+    prelude::*, //
+};
+
+use crate::num;
+
+/// Number of vectors one leaf register carries, one per bit.
+const VECTORS_PER_LEAF: u32 = u32::BITS;
+
+/// Number of leaves one subtree covers.
+const LEAVES_PER_SUBTREE: u32 = 2;
+
+/// Number of subtrees the widest supported tree implements.
+const MAX_NUM_SUBTREES: u32 = 8;
+
+/// Number of leaves the widest supported tree implements.
+const MAX_NUM_LEAVES: u32 = MAX_NUM_SUBTREES * LEAVES_PER_SUBTREE;
+
+/// Number of bits needed to address every vector in the widest supported tree.
+const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
+
+/// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
+/// lower half of the range.
+pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
+
+/// Number of leaves a tree implements.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+#[repr(usize)]
+pub(super) enum LeafCount {
+    /// Turing through Ada.
+    Eight = 8,
+
+    /// Hopper and later.
+    Sixteen = 16,
+}
+
+impl LeafCount {
+    pub(super) const fn into_u32(self) -> u32 {
+        // CAST: both discriminants are 16 or below.
+        self as u32
+    }
+
+    pub(super) const fn into_raw(self) -> usize {
+        num::u32_as_usize(self.into_u32())
+    }
+
+    /// Returns the number of subtrees a tree of this size implements.
+    pub(super) const fn subtree_count(self) -> u32 {
+        self.into_u32() / LEAVES_PER_SUBTREE
+    }
+
+    /// Returns the set of every subtree a tree of this size implements.
+    pub(super) const fn subtree_set(self) -> SubtreeSet {
+        SubtreeSet((1u32 << self.subtree_count()) - 1)
+    }
+
+    /// Returns the number of vectors a tree of this size carries.
+    pub(super) const fn vector_count(self) -> u32 {
+        self.into_u32() * VECTORS_PER_LEAF
+    }
+}
+
+// `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
+// agreement about the widest supported tree.
+static_assert!(1 << VECTOR_BITS == LeafCount::Sixteen.vector_count());
+
+/// Set of vectors within one leaf, one bit per vector.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct LeafMask(u32);
+
+impl LeafMask {
+    /// Returns the mask with every vector set.
+    pub(super) const fn all() -> Self {
+        Self(u32::MAX)
+    }
+
+    pub(super) const fn from_raw(raw: u32) -> Self {
+        Self(raw)
+    }
+
+    pub(super) const fn into_raw(self) -> u32 {
+        self.0
+    }
+
+    pub(super) const fn is_empty(self) -> bool {
+        self.0 == 0
+    }
+
+    /// Returns whether every vector in `other` is also in this mask.
+    pub(super) const fn contains(self, other: Self) -> bool {
+        self.0 & other.0 == other.0
+    }
+}
+
+impl From<Bounded<u32, 32>> for LeafMask {
+    fn from(vectors: Bounded<u32, 32>) -> Self {
+        Self(vectors.get())
+    }
+}
+
+impl From<LeafMask> for Bounded<u32, 32> {
+    fn from(vectors: LeafMask) -> Self {
+        vectors.0.into()
+    }
+}
+
+/// One subtree, held as the `TOP` bit that covers it.
+///
+/// # Invariants
+///
+/// Exactly one bit is set.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct Subtree(u32);
+
+impl Subtree {
+    /// Returns the subtree at index `idx`.
+    const fn new(idx: u32) -> Self {
+        // INVARIANT: shifting `1` left leaves exactly one bit set.
+        Self(1 << idx)
+    }
+
+    /// Returns this subtree's index within the tree.
+    pub(super) const fn index(self) -> u32 {
+        self.0.trailing_zeros()
+    }
+
+    pub(super) const fn into_raw(self) -> u32 {
+        self.0
+    }
+}
+
+/// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct SubtreeSet(u32);
+
+impl SubtreeSet {
+    pub(super) const fn contains(self, subtree: Subtree) -> bool {
+        self.0 & subtree.into_raw() != 0
+    }
+
+    pub(super) const fn is_empty(self) -> bool {
+        self.0 == 0
+    }
+
+    pub(super) const fn intersection(self, other: Self) -> Self {
+        Self(self.0 & other.0)
+    }
+
+    /// Returns one more than the highest index in this set, or `0` for an empty set. An MSI-X
+    /// allocation that covers the set needs this many entries.
+    pub(super) const fn span(self) -> u32 {
+        u32::BITS - self.0.leading_zeros()
+    }
+
+    /// Returns the subtrees of this set, lowest index first.
+    #[expect(dead_code)]
+    pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
+        (0..u32::BITS)
+            .map(Subtree::new)
+            .filter(move |subtree| self.contains(*subtree))
+    }
+}
+
+impl From<Subtree> for SubtreeSet {
+    fn from(subtree: Subtree) -> Self {
+        Self(subtree.into_raw())
+    }
+}
+
+impl From<Bounded<u32, 32>> for SubtreeSet {
+    fn from(subtrees: Bounded<u32, 32>) -> Self {
+        Self(subtrees.get())
+    }
+}
+
+impl From<SubtreeSet> for Bounded<u32, 32> {
+    fn from(subtrees: SubtreeSet) -> Self {
+        subtrees.0.into()
+    }
+}
+
+/// A GIN interrupt vector, bounded to the widest tree any supported part implements.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) struct GinVector(Bounded<u32, VECTOR_BITS>);
+
+impl GinVector {
+    /// Returns vector number `VECTOR`.
+    ///
+    /// Fails to compile if `VECTOR` is beyond the widest supported tree.
+    pub(super) const fn new<const VECTOR: u32>() -> Self {
+        Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
+    }
+
+    pub(super) const fn into_raw(self) -> u32 {
+        self.0.get()
+    }
+
+    /// Returns this vector's leaf.
+    pub(super) fn leaf_index(self) -> LeafIndex {
+        // CALC: `self.0 / VECTORS_PER_LEAF`.
+        self.0.shr::<{ VECTORS_PER_LEAF.ilog2() }, _>().cast()
+    }
+
+    /// Returns this vector's bit within its leaf.
+    pub(super) const fn leaf_mask(self) -> LeafMask {
+        LeafMask(1 << (self.0.get() % VECTORS_PER_LEAF))
+    }
+
+    /// Returns this vector's subtree.
+    pub(super) const fn subtree(self) -> Subtree {
+        Subtree::new(self.0.get() / (VECTORS_PER_LEAF * LEAVES_PER_SUBTREE))
+    }
+
+    /// Checks that a tree with `leaves` leaves implements this vector.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if it does not.
+    pub(super) const fn validate(self, leaves: LeafCount) -> Result {
+        if self.0.get() >= leaves.vector_count() {
+            return Err(EINVAL);
+        }
+
+        Ok(())
+    }
+}
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 1133c6ce5c55..5176a5fe2da2 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -17,6 +17,8 @@
 mod fsp;
 mod gpu;
 mod gsp;
+#[expect(dead_code)]
+mod irq;
 mod mctp;
 mod mm;
 #[macro_use]
-- 
2.55.0

[PATCH v4 04/17] gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers

[John Hubbard]

GIN is the GPU's interrupt controller. It latches each interrupt source
in a two-level tree of LEAF registers summarized by TOP, and raises the
PCI interrupt when an enabled vector in an enabled subtree becomes
pending. A message-signaled interrupt is delivered once per edge, and
pre-Hopper MSI rearms delivery by writing the end-of-interrupt register
in the BAR0 mirror of PCI configuration space.

Add the CPU tree registers that receiving GSP interrupts and running the
software-triggered self-test need: the leaf pending and enable arrays,
the TOP enables, and the leaf trigger. Add the end-of-interrupt
register, NV_XVE_CYA_2, alongside them.

Use the NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_* names on every part. The
pre-Hopper headers call the same tree NV_CTRL, but each function reaches
its own tree through this aperture on both families. Declare the leaf
arrays at 16 entries, the widest tree any supported part implements.
Later patches bound every access by the part's leaf count.

Leave the read-only TOP summary undeclared. A vector that latched while
disabled does not appear in TOP, so nova-core never descends from it and
reads every implemented leaf instead.

Declare each leaf field as a set of vectors within one leaf and each TOP
field as a set of subtrees, so a set of subtrees cannot be written to a
leaf register, nor the reverse. The trigger register's vector field is
12 bits wide, wider than any GIN vector, so a vector converts into it
infallibly.

Assisted-by: LLM
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |  1 +
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 14 ++++
 drivers/gpu/nova-core/irq/regs.rs           | 87 +++++++++++++++++++++
 3 files changed, 102 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq/regs.rs

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index f1323f633a03..1ec0bb055d3b 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -10,3 +10,4 @@
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 mod interrupt_tree;
+mod regs;
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 24976a3146be..5c7829ea3bc5 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -16,6 +16,8 @@
 
 use crate::num;
 
+use super::regs::*;
+
 /// Number of vectors one leaf register carries, one per bit.
 const VECTORS_PER_LEAF: u32 = u32::BITS;
 
@@ -31,6 +33,12 @@
 /// Number of bits needed to address every vector in the widest supported tree.
 const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
 
+/// Width of the vector field in the leaf trigger register.
+const TRIGGER_VECTOR_BITS: u32 = {
+    let range = NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::VECTOR_RANGE;
+    num::u8_as_u32(*range.end() - *range.start() + 1)
+};
+
 /// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
 /// lower half of the range.
 pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
@@ -236,3 +244,9 @@ pub(super) const fn validate(self, leaves: LeafCount) -> Result {
         Ok(())
     }
 }
+
+impl From<GinVector> for Bounded<u32, TRIGGER_VECTOR_BITS> {
+    fn from(vector: GinVector) -> Self {
+        vector.0.extend()
+    }
+}
diff --git a/drivers/gpu/nova-core/irq/regs.rs b/drivers/gpu/nova-core/irq/regs.rs
new file mode 100644
index 000000000000..eef28425a74a
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/regs.rs
@@ -0,0 +1,87 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use kernel::io::register;
+
+use crate::driver::NovaRegisters;
+
+use super::interrupt_tree::{
+    LeafMask,
+    SubtreeSet, //
+};
+
+// The GIN CPU interrupt tree, reached through the `NV_VIRTUAL_FUNCTION_PRIV` aperture. See
+// "Register naming" in `Documentation/gpu/nova/core/interrupts.rst`. The leaf arrays are declared
+// with 16 entries, the widest tree any supported part implements.
+
+register! {
+    base: NovaRegisters;
+
+    /// Pending bits of one leaf, one per vector.
+    ///
+    /// Vector `v` is bit `v % 32` of leaf `v / 32`. The bit is set when the vector's source
+    /// drives it, whether or not the vector is enabled. Writing a `1` clears the bit, and a `0`
+    /// leaves it as it was.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF(u32)[16] @ 0x00b81000 {
+        /// The vectors pending in this leaf.
+        31:0    vectors => LeafMask;
+    }
+
+    /// Enables vectors of one leaf.
+    ///
+    /// A `1` enables the matching vector, and a `0` leaves it as it was.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET(u32)[16] @ 0x00b81200 {
+        /// Vectors to enable.
+        31:0    vectors => LeafMask;
+    }
+
+    /// Disables vectors of one leaf.
+    ///
+    /// A `1` disables the matching vector, and a `0` leaves it as it was. A disabled vector still
+    /// latches in `LEAF`, and `TOP` does not show it.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR(u32)[16] @ 0x00b81400 {
+        /// Vectors to disable.
+        31:0    vectors => LeafMask;
+    }
+
+    /// Enables subtrees.
+    ///
+    /// Bit `N` covers subtree `N`, which is leaves `2N` and `2N + 1`. A `1` enables the matching
+    /// subtree, and a `0` leaves it as it was.
+    ///
+    /// The hardware headers declare a one-element array, so nova-core declares a scalar.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET(u32) @ 0x00b81608 {
+        /// Subtrees to enable.
+        31:0    subtrees => SubtreeSet;
+    }
+
+    /// Disables subtrees, with the bit layout of `TOP_EN_SET`.
+    ///
+    /// A `1` disables the matching subtree, and a `0` leaves it as it was. A disabled subtree
+    /// delivers nothing, and `TOP` still reports it.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR(u32) @ 0x00b81610 {
+        /// Subtrees to disable.
+        31:0    subtrees => SubtreeSet;
+    }
+
+    /// Latches a vector from software. Write-only.
+    ///
+    /// The written vector latches in its `LEAF` register as its own source would, and reaches the
+    /// CPU under the same enables. Implemented on every supported part.
+    pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER(u32) @ 0x00b81640 {
+        /// Vector to latch.
+        11:0    vector;
+    }
+}
+
+// PCI configuration-space mirror in BAR0.
+
+register! {
+    base: NovaRegisters;
+
+    /// MSI end-of-interrupt register. Writing any value rearms MSI delivery.
+    ///
+    /// Only pre-Hopper MSI rearms through this register. See "Rearming PCI interrupt delivery"
+    /// in `Documentation/gpu/nova/core/interrupts.rst`.
+    pub(super) NV_XVE_CYA_2(u32) @ 0x00088704 {}
+}
-- 
2.55.0

[PATCH v4 05/17] gpu: nova-core: add the per-architecture GIN CPU interrupt HAL

[John Hubbard]

The GIN CPU interrupt tree differs by GPU family in two ways:

* The size of the tree. Turing through Ada implement 8 leaves, and
  Hopper and later implement 16.

* The write that rearms delivery. A message-signaled interrupt is
  delivered once per edge, and the PCI side delivers nothing more until
  the CPU rearms it. Before Hopper, MSI rearms by writing the
  end-of-interrupt register in the BAR0 mirror of PCI configuration
  space. On Hopper and later, MSI rearms by clearing and then setting
  the TOP enables of every serviced subtree, which produces a new edge.
  MSI-X rearms the same way on every family, but for the handler's own
  subtree only, since each subtree has its own table entry.

Add an interrupt HAL that provides the leaf count and the rearm method
for each family. Name the interrupt type with two variants, MSI and
MSI-X, since nova-core never allocates the level-triggered INTx that
the PCI core's type also names.

Assisted-by: LLM
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs           | 13 ++++
 drivers/gpu/nova-core/irq/hal.rs       | 90 ++++++++++++++++++++++++++
 drivers/gpu/nova-core/irq/hal/gh100.rs | 28 ++++++++
 drivers/gpu/nova-core/irq/hal/tu102.rs | 28 ++++++++
 4 files changed, 159 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq/hal.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 1ec0bb055d3b..62c242b71dc8 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -9,5 +9,18 @@
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
+mod hal;
 mod interrupt_tree;
 mod regs;
+
+/// The message-signaled interrupt type that Linux granted.
+///
+/// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum MsiType {
+    /// A single message, which every subtree raises.
+    Msi,
+
+    /// One table entry per subtree.
+    MsiX,
+}
diff --git a/drivers/gpu/nova-core/irq/hal.rs b/drivers/gpu/nova-core/irq/hal.rs
new file mode 100644
index 000000000000..ede9a10ccda6
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal.rs
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Per-architecture properties of the GIN CPU interrupt tree.
+//!
+//! See "Per-architecture differences" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+mod gh100;
+mod tu102;
+
+use kernel::{
+    io::Io,
+    prelude::*, //
+};
+
+use crate::{
+    driver::Bar0,
+    gpu::{
+        Architecture,
+        Chipset, //
+    }, //
+};
+
+use super::{
+    interrupt_tree::{
+        LeafCount,
+        Subtree,
+        SubtreeSet, //
+    },
+    regs::*,
+    MsiType, //
+};
+
+/// The register write that rearms PCI interrupt delivery after an interrupt.
+///
+/// The GPU family and the interrupt type that Linux granted select the write. See "Rearming PCI
+/// interrupt delivery" in `Documentation/gpu/nova/core/interrupts.rst`.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(super) enum PciIrqRearmMethod {
+    /// Writes the MSI end-of-interrupt register, `NV_XVE_CYA_2`. Pre-Hopper MSI.
+    ConfigMirrorEoi,
+
+    /// Clears and then sets the `TOP` enables of every serviced subtree. Hopper-plus MSI.
+    TopEnableCycleServiced,
+
+    /// Clears and then sets the `TOP` enable of the handler's own subtree. MSI-X.
+    TopEnableCycleSubtree,
+}
+
+impl PciIrqRearmMethod {
+    /// Rearms PCI interrupt delivery after a handler serviced `subtree`.
+    ///
+    /// `serviced` is every subtree that nova-core services, for the method that cycles them all.
+    pub(super) fn rearm(self, bar: Bar0<'_>, serviced: SubtreeSet, subtree: Subtree) {
+        let subtrees = match self {
+            Self::ConfigMirrorEoi => {
+                bar.write(NV_XVE_CYA_2, 0u32.into());
+                return;
+            }
+            Self::TopEnableCycleServiced => serviced,
+            Self::TopEnableCycleSubtree => SubtreeSet::from(subtree),
+        };
+
+        bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(subtrees),
+        );
+        bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(subtrees),
+        );
+    }
+}
+
+/// The properties of the GIN CPU tree that differ by GPU family.
+pub(super) trait CpuInterruptHal {
+    /// Returns the number of leaves the tree implements.
+    fn leaf_count(&self) -> LeafCount;
+
+    /// Returns the rearm method for `msi_type`.
+    fn pci_irq_rearm_method(&self, msi_type: MsiType) -> PciIrqRearmMethod;
+}
+
+/// Returns the [`CpuInterruptHal`] for `chipset`'s architecture.
+pub(super) fn cpu_interrupt_hal(chipset: Chipset) -> &'static dyn CpuInterruptHal {
+    match chipset.arch() {
+        Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
+        Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
+            gh100::GH100_HAL
+        }
+    }
+}
diff --git a/drivers/gpu/nova-core/irq/hal/gh100.rs b/drivers/gpu/nova-core/irq/hal/gh100.rs
new file mode 100644
index 000000000000..dd3d0d12d779
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal/gh100.rs
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use super::{
+    CpuInterruptHal,
+    LeafCount,
+    MsiType,
+    PciIrqRearmMethod, //
+};
+
+/// The CPU interrupt tree properties of Hopper and Blackwell.
+struct Gh100;
+
+impl CpuInterruptHal for Gh100 {
+    fn leaf_count(&self) -> LeafCount {
+        LeafCount::Sixteen
+    }
+
+    fn pci_irq_rearm_method(&self, msi_type: MsiType) -> PciIrqRearmMethod {
+        match msi_type {
+            MsiType::Msi => PciIrqRearmMethod::TopEnableCycleServiced,
+            MsiType::MsiX => PciIrqRearmMethod::TopEnableCycleSubtree,
+        }
+    }
+}
+
+const GH100: Gh100 = Gh100;
+pub(super) const GH100_HAL: &dyn CpuInterruptHal = &GH100;
diff --git a/drivers/gpu/nova-core/irq/hal/tu102.rs b/drivers/gpu/nova-core/irq/hal/tu102.rs
new file mode 100644
index 000000000000..121f5779accf
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/hal/tu102.rs
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use super::{
+    CpuInterruptHal,
+    LeafCount,
+    MsiType,
+    PciIrqRearmMethod, //
+};
+
+/// The CPU interrupt tree properties of Turing, Ampere, and Ada.
+struct Tu102;
+
+impl CpuInterruptHal for Tu102 {
+    fn leaf_count(&self) -> LeafCount {
+        LeafCount::Eight
+    }
+
+    fn pci_irq_rearm_method(&self, msi_type: MsiType) -> PciIrqRearmMethod {
+        match msi_type {
+            MsiType::Msi => PciIrqRearmMethod::ConfigMirrorEoi,
+            MsiType::MsiX => PciIrqRearmMethod::TopEnableCycleSubtree,
+        }
+    }
+}
+
+const TU102: Tu102 = Tu102;
+pub(super) const TU102_HAL: &dyn CpuInterruptHal = &TU102;
-- 
2.55.0

[PATCH v4 06/17] gpu: nova-core: add the GIN interrupt tree and allocate its vectors

[John Hubbard]

From: Joel Fernandes <joelagnelf@nvidia.com>

Servicing a GIN leaf has a required order: read its pending bits, then
clear them. Clearing a leaf first discards every vector latched in it,
and the hardware keeps no record of what was discarded. The interrupt
never arrives, and no register shows that it was ever pending.

Every subtree enabled at TOP also needs an allocated PCI vector with a
handler registered on it. Under MSI-X each subtree has its own table
entry. Linux masks every entry until a driver requests its IRQ, and a
masked entry sends no message. An enabled subtree whose entry was never
requested raises interrupts that never reach a handler, while the leaf
and TOP registers show them pending and enabled. Under MSI the whole
tree raises a single message, so one entry serves every subtree.

Add the CPU interrupt tree of one PCIe function. Reading a leaf yields
the handle that clears it, so the wrong order does not compile. Building
a tree fails if it names a subtree that the GPU does not implement.

Allocate the PCI vectors from the set of serviced subtrees. Request
MSI-X entries 0 through the highest serviced subtree, since an MSI-X
allocation cannot be sparse, and fall back to a single MSI message,
never to INTx.

Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: reworked on top of the GIN vector types: a leaf read yields
 the handle that clears it, enables are guarded, the leaf count and
 rearm method come from the interrupt HAL, and the drain reads every
 implemented leaf rather than descending from TOP]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |  82 +++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 252 +++++++++++++++++++-
 2 files changed, 327 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 62c242b71dc8..28f147641024 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -13,6 +13,23 @@
 mod interrupt_tree;
 mod regs;
 
+use kernel::{
+    device::Bound,
+    irq,
+    pci::{
+        self,
+        IrqType, //
+    },
+    prelude::*, //
+};
+
+use crate::num;
+
+use interrupt_tree::{
+    Subtree,
+    SubtreeSet, //
+};
+
 /// The message-signaled interrupt type that Linux granted.
 ///
 /// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
@@ -24,3 +41,68 @@ enum MsiType {
     /// One table entry per subtree.
     MsiX,
 }
+
+/// The PCI interrupt vectors allocated for the subtrees that nova-core services.
+///
+/// A subtree may be enabled at `TOP` only once a vector is allocated for it and a handler is
+/// registered on that vector. See "The serviced-subtree invariant" in
+/// `Documentation/gpu/nova/core/interrupts.rst`.
+pub(crate) struct SubtreeVectors<'a> {
+    vectors: pci::IrqVectorRegistration<'a>,
+    serviced: SubtreeSet,
+    msi_type: MsiType,
+}
+
+impl SubtreeVectors<'_> {
+    /// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `subtree` is not one of the serviced subtrees.
+    fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
+        if !self.serviced.contains(subtree) {
+            return Err(EINVAL);
+        }
+
+        let entry = match self.msi_type {
+            MsiType::MsiX => num::u32_as_usize(subtree.index()),
+            MsiType::Msi => 0,
+        };
+
+        self.vectors.index(entry).map(Into::into)
+    }
+}
+
+/// Allocates the PCI interrupt vectors for the subtrees in `serviced`.
+///
+/// Requests MSI-X entries `0` through the highest subtree in `serviced`, since an allocation
+/// cannot be sparse, and falls back to a single MSI message for the whole tree.
+///
+/// # Errors
+///
+/// `EINVAL` if `serviced` is empty. Otherwise, when neither type could be allocated, the error
+/// from the MSI request.
+pub(crate) fn alloc_vectors(
+    pdev: &pci::Device<Bound>,
+    serviced: SubtreeSet,
+) -> Result<SubtreeVectors<'_>> {
+    if serviced.is_empty() {
+        return Err(EINVAL);
+    }
+
+    let entries = serviced.span();
+
+    let (vectors, msi_type) = pdev
+        .alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
+        .map(|vectors| (vectors, MsiType::MsiX))
+        .or_else(|_| {
+            pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
+                .map(|vectors| (vectors, MsiType::Msi))
+        })?;
+
+    Ok(SubtreeVectors {
+        vectors,
+        serviced,
+        msi_type,
+    })
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 5c7829ea3bc5..2583b006019e 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -1,22 +1,40 @@
 // SPDX-License-Identifier: GPL-2.0
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
-//! Vector addressing in the GIN CPU interrupt tree.
+//! The GIN CPU interrupt tree for one PCIe function.
 //!
 //! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
-//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
-//! keep the four from being confused with one another.
+//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit.
+//!
+//! Servicing a leaf requires reading its pending bits before clearing them. Only
+//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] clears a leaf,
+//! so the wrong order does not compile. Nothing in this module serializes access to the tree.
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 use kernel::{
+    io::{
+        register::Array,
+        Io, //
+    },
     num::Bounded,
     prelude::*, //
 };
 
-use crate::num;
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    num, //
+};
 
-use super::regs::*;
+use super::{
+    hal::{
+        cpu_interrupt_hal,
+        PciIrqRearmMethod, //
+    },
+    regs::*,
+    SubtreeVectors, //
+};
 
 /// Number of vectors one leaf register carries, one per bit.
 const VECTORS_PER_LEAF: u32 = u32::BITS;
@@ -78,6 +96,11 @@ pub(super) const fn subtree_set(self) -> SubtreeSet {
     pub(super) const fn vector_count(self) -> u32 {
         self.into_u32() * VECTORS_PER_LEAF
     }
+
+    /// Returns every leaf a tree of this size implements.
+    pub(super) fn iter(self) -> impl Iterator<Item = LeafIndex> {
+        (0..self.into_raw()).filter_map(LeafIndex::try_new)
+    }
 }
 
 // `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
@@ -130,7 +153,7 @@ fn from(vectors: LeafMask) -> Self {
 ///
 /// Exactly one bit is set.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct Subtree(u32);
+pub(crate) struct Subtree(u32);
 
 impl Subtree {
     /// Returns the subtree at index `idx`.
@@ -151,7 +174,7 @@ pub(super) const fn into_raw(self) -> u32 {
 
 /// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct SubtreeSet(u32);
+pub(crate) struct SubtreeSet(u32);
 
 impl SubtreeSet {
     pub(super) const fn contains(self, subtree: Subtree) -> bool {
@@ -250,3 +273,218 @@ fn from(vector: GinVector) -> Self {
         vector.0.extend()
     }
 }
+
+/// Disables `vectors` in `leaf`.
+fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+    bar.write(
+        Array::at(*leaf),
+        NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
+    );
+}
+
+/// Disables the subtrees in `serviced` at `TOP`.
+fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
+    bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
+}
+
+/// The CPU tree of one PCIe function, and the subtrees that nova-core services.
+pub(super) struct Tree<'a> {
+    bar: Bar0<'a>,
+    leaves: LeafCount,
+    serviced: SubtreeSet,
+    rearm: PciIrqRearmMethod,
+}
+
+impl<'a> Tree<'a> {
+    /// Creates the tree of `chipset`, covering the subtrees that `vectors` services.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `chipset` does not implement every subtree that `vectors` services.
+    pub(super) fn new(
+        bar: Bar0<'a>,
+        chipset: Chipset,
+        vectors: &SubtreeVectors<'_>,
+    ) -> Result<Self> {
+        let hal = cpu_interrupt_hal(chipset);
+        let leaves = hal.leaf_count();
+        let serviced = vectors.serviced;
+
+        if serviced.intersection(leaves.subtree_set()) != serviced {
+            return Err(EINVAL);
+        }
+
+        Ok(Self {
+            bar,
+            leaves,
+            serviced,
+            rearm: hal.pci_irq_rearm_method(vectors.msi_type),
+        })
+    }
+
+    /// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree that
+    /// the calling handler serves.
+    ///
+    /// A handler must call this before returning, or it receives no further interrupts.
+    pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
+        self.rearm.rearm(self.bar, self.serviced, subtree);
+    }
+
+    /// Enables the serviced subtrees at `TOP`.
+    ///
+    /// Each of them must have a handler registered on its PCI vector.
+    pub(super) fn enable_top(&self) {
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
+        );
+    }
+
+    /// Disables the serviced subtrees at `TOP`.
+    pub(super) fn disable_top(&self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+
+    /// Enables the serviced subtrees at `TOP` until the returned guard drops.
+    pub(super) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
+        self.enable_top();
+
+        TopEnableGuard {
+            bar: self.bar,
+            serviced: self.serviced,
+        }
+    }
+
+    /// Enables `vectors` in `leaf`.
+    pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        self.bar.write(
+            Array::at(*leaf),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
+        );
+    }
+
+    /// Disables `vectors` in `leaf`.
+    pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        clear_leaf_enables(self.bar, leaf, vectors);
+    }
+
+    /// Enables `vectors` in `leaf` until the returned guard drops.
+    pub(super) fn enable_leaf_guarded(
+        &self,
+        leaf: LeafIndex,
+        vectors: LeafMask,
+    ) -> LeafEnableGuard<'a> {
+        self.enable_leaf(leaf, vectors);
+
+        LeafEnableGuard {
+            bar: self.bar,
+            leaf,
+            vectors,
+        }
+    }
+
+    /// Reads the pending bits of `leaf`, and returns the handle that clears them.
+    pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
+        let pending = self
+            .bar
+            .read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
+            .vectors();
+
+        LeafPending {
+            bar: self.bar,
+            leaf,
+            pending,
+        }
+    }
+
+    /// Latches `vector` as its own source would.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if this tree does not implement `vector`.
+    // The interrupt self-test is the only caller.
+    #[expect(dead_code)]
+    pub(super) fn trigger(&self, vector: GinVector) -> Result {
+        vector.validate(self.leaves)?;
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
+        );
+
+        Ok(())
+    }
+
+    /// Disables every vector in every implemented leaf, including the subtrees that nova-core does
+    /// not service. Call this only during probe.
+    pub(super) fn disable_all_leaves(&self) {
+        for leaf in self.leaves.iter() {
+            self.disable_leaf(leaf, LeafMask::all());
+        }
+    }
+
+    /// Clears every pending bit in every implemented leaf, including the subtrees that nova-core
+    /// does not service.
+    ///
+    /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with no
+    /// interrupt handler registered.
+    pub(super) fn drain(&self) {
+        self.disable_top();
+
+        // A vector that latched while disabled does not show in `TOP`, so read every leaf rather
+        // than descending from it.
+        for leaf in self.leaves.iter() {
+            self.read_pending(leaf).clear();
+        }
+    }
+}
+
+/// The pending bits of one leaf as they were read, and the handle that clears them.
+pub(super) struct LeafPending<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    pending: LeafMask,
+}
+
+impl LeafPending<'_> {
+    pub(super) fn vectors(&self) -> LeafMask {
+        self.pending
+    }
+
+    /// Clears the vectors that were pending at the read. A vector that latched since stays pending.
+    pub(super) fn clear(&self) {
+        self.clear_vectors(self.pending);
+    }
+
+    /// Clears `vectors` and no other bit.
+    pub(super) fn clear_vectors(&self, vectors: LeafMask) {
+        if !vectors.is_empty() {
+            self.bar.write(
+                Array::at(*self.leaf),
+                NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
+            );
+        }
+    }
+}
+
+/// Disables a set of vectors in one leaf when dropped.
+pub(super) struct LeafEnableGuard<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    vectors: LeafMask,
+}
+
+impl Drop for LeafEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_leaf_enables(self.bar, self.leaf, self.vectors);
+    }
+}
+
+/// Disables the serviced subtrees at `TOP` when dropped.
+pub(super) struct TopEnableGuard<'a> {
+    bar: Bar0<'a>,
+    serviced: SubtreeSet,
+}
+
+impl Drop for TopEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+}
-- 
2.55.0

[PATCH v4 07/17] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor

[John Hubbard]

The GPU boots its own firmware, GFW, out of reset, and nothing may
program the GPU until GFW reports completion.

nova-core waited for GFW inside the Gpu constructor, which also boots
the GSP. Code that has to run after GFW and before GSP boot, such as a
probe-time hardware self-test, had nowhere to go.

Move the wait into probe, ahead of the Gpu constructor, and read the
chipset there from a Spec that probe builds itself. Leave the DMA mask
in the constructor, since it programs the host rather than the GPU.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/driver.rs | 13 ++++++++++++-
 drivers/gpu/nova-core/gpu.rs    | 28 ++++++++++++++++++++--------
 2 files changed, 32 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index 0672a0707a71..15a44f9a6441 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -22,7 +22,13 @@
     types::CovariantForLt,
 };
 
-use crate::gpu::Gpu;
+use crate::{
+    gpu,
+    gpu::{
+        Gpu,
+        Spec, //
+    }, //
+};
 
 /// Counter for generating unique auxiliary device IDs.
 static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
@@ -109,6 +115,11 @@ fn probe<'bound>(
                     let bar1_idx = bar1_resource_index(pdev)?;
                     pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
                 },
+                _: {
+                    let spec = Spec::new(pdev.as_ref(), bar)?;
+
+                    gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+                },
                 // TODO: Use self-referential pin-init syntax once available.
                 gpu <- Gpu::new(
                     pdev,
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index d763bc8d3827..3d796d6c7013 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -212,12 +212,12 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
 /// Structure holding a basic description of the GPU: `Chipset` and `Revision`.
 #[derive(Clone, Copy)]
 pub(crate) struct Spec {
-    chipset: Chipset,
+    pub(crate) chipset: Chipset,
     revision: Revision,
 }
 
 impl Spec {
-    fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
+    pub(crate) fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
         // Some brief notes about boot0 and boot42, in chronological order:
         //
         // NV04 through NV50:
@@ -362,17 +362,12 @@ pub(crate) fn new<'a>(
                 dev_info!(dev,"NVIDIA ({})\n", spec);
             })?,
 
-            // We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
             _: {
-                let hal = hal::gpu_hal(spec.chipset);
-                let dma_mask = hal.dma_mask();
+                let dma_mask = hal::gpu_hal(spec.chipset).dma_mask();
 
                 // SAFETY: `Gpu` owns all DMA allocations for this device, and we are
                 // still constructing it, so no concurrent DMA allocations can exist.
                 unsafe { pdev.dma_set_mask_and_coherent(dma_mask)? };
-
-                hal.wait_gfw_boot_completion(bar)
-                    .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))?;
             },
 
             // Initialize this early because `gsp_resources` depends on it.
@@ -495,6 +490,23 @@ pub(crate) fn run_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bou
     }
 }
 
+/// Waits for GFW, the GPU's boot firmware, to report completion.
+///
+/// Nothing may program the GPU before then.
+///
+/// # Errors
+///
+/// `ETIMEDOUT` if GFW does not report completion in time.
+pub(crate) fn wait_gfw_boot_completion(
+    dev: &device::Device<device::Bound>,
+    bar: Bar0<'_>,
+    chipset: Chipset,
+) -> Result {
+    hal::gpu_hal(chipset)
+        .wait_gfw_boot_completion(bar)
+        .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))
+}
+
 /// Reads the boot0 register and returns its raw value.
 pub(crate) fn boot_0_raw(bar: Bar0<'_>) -> u32 {
     bar.read(regs::NV_PMC_BOOT_0).into_raw()
-- 
2.55.0

[PATCH v4 08/17] gpu: nova-core: add an interrupt delivery self-test

[John Hubbard]

A GPU interrupt can be lost in the MSI or MSI-X allocation, in the GIN
tree's enables, or in the rearm, and every one of those failures looks
the same: no interrupt arrives, and nothing says which one broke.

Add a probe-time self-test, built under NOVA_CORE_SELFTESTS, that
latches the CPU doorbell vector through the GIN software trigger and
waits for a registered handler to service it. One delivery would pass
with a broken rearm, because the first message-signaled interrupt
arrives whether or not the driver rearms, so the test triggers twice and
waits for the first handler to finish before the second trigger. It runs
after GFW boot and before GSP boot, on a quiesced tree, and fails probe
unless both deliveries arrive, each finds only the doorbell pending, and
the leaf ends clear.

The doorbell has the same vector on every supported GPU, so the test
names it without asking GSP-RM. It allocates the PCI vectors for the
doorbell's subtree and releases them before returning, so under MSI-X
the delivery also exercises that subtree's table entry.

Assisted-by: LLM
Co-developed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/Kconfig               |   5 +
 drivers/gpu/nova-core/driver.rs             |   5 +
 drivers/gpu/nova-core/irq.rs                |   2 +
 drivers/gpu/nova-core/irq/doorbell_test.rs  | 266 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs |   2 +-
 drivers/gpu/nova-core/nova_core.rs          |   2 +-
 6 files changed, 280 insertions(+), 2 deletions(-)
 create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs

diff --git a/drivers/gpu/nova-core/Kconfig b/drivers/gpu/nova-core/Kconfig
index 1934f17baa8b..2e11e46c99c7 100644
--- a/drivers/gpu/nova-core/Kconfig
+++ b/drivers/gpu/nova-core/Kconfig
@@ -24,4 +24,9 @@ config NOVA_CORE_SELFTESTS
 	help
 	  Build the driver self-tests and run them when the GPU is probed.
 
+	  If the interrupt delivery test fails, the probe fails and the driver
+	  does not bind to the GPU. A broken interrupt path would otherwise
+	  show up later as a hang, far from its cause. Every other self-test
+	  logs its failure and lets the probe continue.
+
 	  If unsure, say N.
diff --git a/drivers/gpu/nova-core/driver.rs b/drivers/gpu/nova-core/driver.rs
index 15a44f9a6441..4400cae8c8ce 100644
--- a/drivers/gpu/nova-core/driver.rs
+++ b/drivers/gpu/nova-core/driver.rs
@@ -119,6 +119,11 @@ fn probe<'bound>(
                     let spec = Spec::new(pdev.as_ref(), bar)?;
 
                     gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+
+                    // The self-test disables and drains the whole tree, so it has to run before
+                    // `Gpu::new` boots the GSP.
+                    #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+                    crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
                 },
                 // TODO: Use self-referential pin-init syntax once available.
                 gpu <- Gpu::new(
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 28f147641024..7fb7d9f2e237 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -9,6 +9,8 @@
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
+#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+pub(crate) mod doorbell_test;
 mod hal;
 mod interrupt_tree;
 mod regs;
diff --git a/drivers/gpu/nova-core/irq/doorbell_test.rs b/drivers/gpu/nova-core/irq/doorbell_test.rs
new file mode 100644
index 000000000000..a1f8b3cc377b
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/doorbell_test.rs
@@ -0,0 +1,266 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Interrupt delivery self-test.
+//!
+//! The test triggers the CPU doorbell vector from software, twice, and checks that each trigger
+//! reaches a registered handler. It runs during probe under `CONFIG_NOVA_CORE_SELFTESTS`.
+//!
+//! See "Self-test" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+use core::pin::Pin;
+
+use kernel::{
+    device::Bound,
+    irq,
+    pci,
+    prelude::*,
+    sync::{
+        atomic::{
+            Atomic,
+            Relaxed, //
+        },
+        Completion, //
+    },
+    time, //
+};
+
+use super::interrupt_tree::{
+    GinVector,
+    LeafEnableGuard,
+    LeafMask,
+    Subtree,
+    TopEnableGuard,
+    Tree, //
+};
+
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    selftest_assert,
+    selftest_assert_eq, //
+};
+
+/// The CPU doorbell vector. Every supported GPU uses this number, so the test needs nothing from
+/// GSP-RM, which is not running yet.
+const DOORBELL_VECTOR: GinVector = GinVector::new::<129>();
+
+/// The only subtree that this test services.
+const DOORBELL_SUBTREE: Subtree = DOORBELL_VECTOR.subtree();
+
+/// Time allowed for each delivery to arrive.
+const DELIVERY_TIMEOUT_MS: time::Msecs = 1000;
+
+/// The self-test's interrupt handler.
+///
+/// It clears only the doorbell's bit, rearms delivery, and never walks the tree. A missing rearm
+/// shows up as a timeout on the second delivery.
+#[pin_data]
+struct DoorbellTestHandler<'a> {
+    tree: Tree<'a>,
+    /// Completed by the first delivery.
+    #[pin]
+    first: Completion,
+    /// Completed by the second delivery.
+    #[pin]
+    second: Completion,
+    /// Deliveries that found the doorbell bit set.
+    irq_count: Atomic<u32>,
+    /// The doorbell leaf's pending bits, as read by the first delivery.
+    first_pending: Atomic<u32>,
+    /// The doorbell leaf's pending bits, as read by the second delivery.
+    second_pending: Atomic<u32>,
+}
+
+impl irq::Handler for DoorbellTestHandler<'_> {
+    fn handle(&self) -> irq::IrqReturn {
+        let leaf = self.tree.read_pending(DOORBELL_VECTOR.leaf_index());
+        let pending = leaf.vectors();
+        if !pending.contains(DOORBELL_VECTOR.leaf_mask()) {
+            self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
+            return irq::IrqReturn::None;
+        }
+        leaf.clear_vectors(DOORBELL_VECTOR.leaf_mask());
+
+        let count = self.irq_count.fetch_add(1, Relaxed);
+
+        // Rearm before completing, since the waiting thread triggers the next doorbell as soon as
+        // it wakes.
+        self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
+
+        match count {
+            0 => {
+                self.first_pending.store(pending.into_raw(), Relaxed);
+                self.first.complete_all();
+            }
+            1 => {
+                self.second_pending.store(pending.into_raw(), Relaxed);
+                self.second.complete_all();
+            }
+            _ => (),
+        }
+
+        irq::IrqReturn::Handled
+    }
+}
+
+/// The self-test's handler registration and the enables that deliver to it.
+///
+/// Drops in the order that "Enabling the GSP event" in
+/// `Documentation/gpu/nova/core/interrupts.rst` requires: the vector is disabled, then the
+/// handler is freed, then the subtree is disabled.
+struct SelftestResources<'a, 'r> {
+    _leaf_guard: LeafEnableGuard<'a>,
+    reg: Pin<KBox<irq::Registration<'r, DoorbellTestHandler<'a>>>>,
+    _top_guard: TopEnableGuard<'a>,
+}
+
+impl<'a> SelftestResources<'a, '_> {
+    fn handler(&self) -> &DoorbellTestHandler<'a> {
+        self.reg.handler()
+    }
+
+    /// Disables the doorbell vector and waits for a handler in flight on another CPU to finish.
+    ///
+    /// The handler's counters and the leaf's pending bits are final on return.
+    fn quiesce_source(&self) {
+        self.handler()
+            .tree
+            .disable_leaf(DOORBELL_VECTOR.leaf_index(), DOORBELL_VECTOR.leaf_mask());
+        self.reg.synchronize();
+    }
+}
+
+/// Runs the interrupt delivery self-test.
+///
+/// Call this only during probe, before GSP boot: it disables every vector in the tree and clears
+/// every pending bit. On return, the doorbell's subtree is disabled at `TOP`, and the test's PCI
+/// vectors and handler are released.
+///
+/// # Errors
+///
+/// `EINVAL` if `chipset` does not implement the doorbell's subtree. `ETIMEDOUT` if a delivery
+/// does not arrive within [`DELIVERY_TIMEOUT_MS`]. `EIO` if a self-test assertion fails.
+/// Otherwise the error from allocating the PCI vectors or registering the handler.
+pub(crate) fn run_selftest(pdev: &pci::Device<Bound>, bar: Bar0<'_>, chipset: Chipset) -> Result {
+    let dev = pdev.as_ref();
+
+    let vectors = super::alloc_vectors(pdev, DOORBELL_SUBTREE.into())?;
+    let request = vectors.request_for(DOORBELL_SUBTREE)?;
+    let tree = Tree::new(bar, chipset, &vectors)?;
+    let doorbell = DOORBELL_VECTOR.leaf_index();
+    let doorbell_mask = DOORBELL_VECTOR.leaf_mask();
+
+    dev_info!(
+        dev,
+        "interrupt self-test: starting on vector {}, subtree {}, with {:?}\n",
+        DOORBELL_VECTOR.into_raw(),
+        DOORBELL_SUBTREE.index(),
+        vectors.msi_type,
+    );
+
+    // GFW boot can leave vectors enabled and pending. Registering a handler unmasks the PCI
+    // interrupt, and they would be delivered to a handler that services only the doorbell.
+    tree.disable_all_leaves();
+    tree.drain();
+
+    // A delivery proves nothing unless the doorbell bit starts out clear.
+    let pre_pending = tree.read_pending(doorbell).vectors();
+    selftest_assert!(
+        dev,
+        !pre_pending.contains(doorbell_mask),
+        "vector {} already pending, leaf[{}] is {:#x}",
+        DOORBELL_VECTOR.into_raw(),
+        doorbell.get(),
+        pre_pending.into_raw()
+    );
+
+    let handler_init = try_pin_init!(DoorbellTestHandler {
+        tree,
+        first <- Completion::new(),
+        second <- Completion::new(),
+        irq_count: Atomic::new(0),
+        first_pending: Atomic::new(0),
+        second_pending: Atomic::new(0),
+    }? Error);
+
+    // Registration must precede any enable, or a delivery reaches no handler.
+    let reg = KBox::pin_init(
+        // SAFETY: this registration is dropped before the enclosing function returns, so its
+        // `Drop`, which calls `free_irq()`, always runs.
+        unsafe {
+            irq::Registration::new(
+                request,
+                irq::Flags::TRIGGER_NONE,
+                c"nova-core-selftest",
+                handler_init,
+            )
+        },
+        GFP_KERNEL,
+    )?;
+
+    let resources = SelftestResources {
+        _leaf_guard: reg
+            .handler()
+            .tree
+            .enable_leaf_guarded(doorbell, doorbell_mask),
+        _top_guard: reg.handler().tree.enable_top_guarded(),
+        reg,
+    };
+    let handler = resources.handler();
+
+    handler.tree.trigger(DOORBELL_VECTOR)?;
+    let mut completed = handler
+        .first
+        .wait_for_completion_timeout(time::msecs_to_jiffies(DELIVERY_TIMEOUT_MS))
+        .is_some();
+
+    // The second trigger waits for the first delivery, or the two could coalesce.
+    if completed {
+        handler.tree.trigger(DOORBELL_VECTOR)?;
+        completed = handler
+            .second
+            .wait_for_completion_timeout(time::msecs_to_jiffies(DELIVERY_TIMEOUT_MS))
+            .is_some();
+    }
+
+    resources.quiesce_source();
+
+    let count = handler.irq_count.load(Relaxed);
+    let first_pending = LeafMask::from_raw(handler.first_pending.load(Relaxed));
+    let second_pending = LeafMask::from_raw(handler.second_pending.load(Relaxed));
+    let residual = handler.tree.read_pending(doorbell).vectors();
+
+    if !completed {
+        dev_err!(
+            dev,
+            "interrupt self-test: only {} of 2 deliveries arrived within {} ms\n",
+            count,
+            DELIVERY_TIMEOUT_MS,
+        );
+        return Err(ETIMEDOUT);
+    }
+
+    selftest_assert_eq!(dev, count, 2, "delivery count");
+
+    // Every other vector in the leaf is disabled and was drained, so require the exact mask.
+    selftest_assert_eq!(dev, first_pending, doorbell_mask, "first delivery");
+    selftest_assert_eq!(dev, second_pending, doorbell_mask, "second delivery");
+    selftest_assert!(
+        dev,
+        !residual.contains(doorbell_mask),
+        "vector {} still pending, leaf[{}] is {:#x}",
+        DOORBELL_VECTOR.into_raw(),
+        doorbell.get(),
+        residual.into_raw()
+    );
+
+    dev_info!(
+        dev,
+        "interrupt self-test: passed, subtree {}, {} deliveries\n",
+        DOORBELL_SUBTREE.index(),
+        count,
+    );
+
+    Ok(())
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 2583b006019e..f77920a3b30a 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -402,7 +402,7 @@ pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
     ///
     /// `EINVAL` if this tree does not implement `vector`.
     // The interrupt self-test is the only caller.
-    #[expect(dead_code)]
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) fn trigger(&self, vector: GinVector) -> Result {
         vector.validate(self.leaves)?;
         self.bar.write_reg(
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index 5176a5fe2da2..abafe4f2968d 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -17,7 +17,7 @@
 mod fsp;
 mod gpu;
 mod gsp;
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
 mod irq;
 mod mctp;
 mod mm;
-- 
2.55.0

[PATCH v4 09/17] gpu: nova-core: log GSP events instead of discarding them

[John Hubbard]

The GSP posts unsolicited messages on the same queue that carries
command replies: log records, OS error and robust-channel records, and
lifecycle notices.

nova-core discarded every message that was not the reply a caller was
waiting for, and an unrecognized function code aborted the in-flight
command. The GSP's error reports never reached the kernel log.

Log every non-reply message according to its function code, on the
receive path that already reads it, and leave the in-flight command
waiting for its reply. Event payloads, such as XID numbers and log
contents, are not decoded.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 52 ++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 9f99e6bbb4fa..340760e384d0 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -557,8 +557,7 @@ fn notify_gsp(bar: Bar0<'_>) {
 
     /// Sends `command` to the GSP and waits for the reply.
     ///
-    /// Messages with non-matching function codes are silently consumed until the expected reply
-    /// arrives.
+    /// Events that arrive before the reply are logged and consumed.
     ///
     /// The queue is locked for the entire send+receive cycle to ensure that no other command can
     /// be interleaved.
@@ -815,8 +814,8 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
     /// Receive a message from the GSP.
     ///
-    /// The expected message type is specified using the `M` generic parameter. If the pending
-    /// message has a different function code, `ERANGE` is returned and the message is consumed.
+    /// A message whose function code is `M::FUNCTION` is decoded and returned. Any other message
+    /// is logged as an event.
     ///
     /// The read pointer is always advanced past the message, regardless of whether it matched.
     ///
@@ -825,8 +824,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `EINVAL` if the function code of the message was not recognized.
-    /// - `ERANGE` if the message had a recognized but non-matching function code.
+    /// - `ERANGE` if the message was not the awaited reply.
     ///
     /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -835,11 +833,11 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         Error: From<M::InitError>,
     {
         let message = self.wait_for_msg(timeout)?;
-        let function = message.header.function().map_err(|_| EINVAL)?;
+        let function = message.header.function();
+        let seq = message.header.sequence();
 
-        // Extract the message. Store the result as we want to advance the read pointer even in
-        // case of failure.
-        let result = if function == M::FUNCTION {
+        // An early return here would leave the read pointer on this message.
+        let result = if matches!(function, Ok(f) if f == M::FUNCTION) {
             let (cmd, contents_1) = M::Message::from_bytes_prefix(message.contents.0).ok_or(EIO)?;
             let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
 
@@ -850,11 +848,13 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
                         dev_warn!(
                             &self.dev,
                             "GSP message {:?} has unprocessed data\n",
-                            function
+                            M::FUNCTION
                         );
                     }
                 })
         } else {
+            self.log_event(function, seq);
+
             Err(ERANGE)
         };
 
@@ -865,4 +865,34 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
 
         result
     }
+
+    /// Logs an event, meaning a message that no caller was waiting for.
+    ///
+    /// An OS error or robust-channel record is logged at error level and an unknown function code
+    /// at warning level. Every other event is recorded only by the receive trace in
+    /// [`Self::wait_for_msg`].
+    fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
+        match function {
+            Ok(MsgFunction::OsErrorLog) => {
+                dev_err!(&self.dev, "GSP reported an OS error (seq {})\n", seq);
+            }
+            Ok(MsgFunction::RcTriggered) => {
+                dev_err!(
+                    &self.dev,
+                    "GSP triggered robust-channel recovery (seq {})\n",
+                    seq
+                );
+            }
+            // Nothing to do for the remaining known function codes.
+            Ok(_) => {}
+            Err(raw) => {
+                dev_warn!(
+                    &self.dev,
+                    "unknown GSP message function {:#x} (seq {})\n",
+                    raw,
+                    seq
+                );
+            }
+        }
+    }
 }
-- 
2.55.0

[PATCH v4 10/17] gpu: nova-core: stop re-parsing a bad GSP message

[John Hubbard]

A GSP message carries its length inside the checksummed region. Once the
framing or the checksum fails, there is no trustworthy length with which
to skip the message.

Two failures left a bad message at the queue head. A framing or checksum
failure returned without advancing the read pointer, so every later
receive parsed the same message again. A validly framed message whose
typed payload failed to decode returned early and did the same.

Poison the queue on a framing or checksum failure: log what was
inconsistent and fail every later receive, so the bad message is parsed
once and recovery takes a device reset. Advance the read pointer past a
validly framed message whether or not its payload decodes, and warn when
the payload is shorter than the type it decodes into.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 93 +++++++++++++++++++++----------
 1 file changed, 64 insertions(+), 29 deletions(-)

diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 340760e384d0..855c5a708525 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -2,7 +2,10 @@
 
 mod continuation;
 
-use core::mem;
+use core::{
+    cell::Cell,
+    mem, //
+};
 
 use kernel::{
     device,
@@ -11,6 +14,7 @@
         CoherentBox,
         DmaAddress, //
     },
+    fmt,
     io::{
         io_project,
         poll::read_poll_timeout,
@@ -532,6 +536,7 @@ pub(crate) fn new(
                     dev,
                     gsp_mem,
                     seq: 0,
+                    poisoned: Cell::new(false),
                 }),
             }))
         })
@@ -624,6 +629,12 @@ struct CmdqInner<'a> {
     dev: &'a device::Device,
     /// Current command sequence number.
     seq: u32,
+    /// Set once a message fails framing or checksum validation. Every later receive fails, since
+    /// the bad message cannot be skipped. See "Draining the GSP-to-CPU queue" in
+    /// `Documentation/gpu/nova/core/interrupts.rst`.
+    ///
+    /// A [`Cell`] because [`Self::wait_for_msg`] sets it through `&self`.
+    poisoned: Cell<bool>,
     /// Memory area shared with the GSP for communicating commands and messages.
     gsp_mem: DmaGspMem<'a>,
 }
@@ -732,6 +743,14 @@ fn send_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
         }
     }
 
+    /// Logs `reason`, poisons the queue, and returns `EIO` for the caller to propagate.
+    fn poison(&self, reason: fmt::Arguments<'_>) -> Error {
+        dev_err!(&self.dev, "GSP RPC: receive: queue poisoned: {}\n", reason);
+        self.poisoned.set(true);
+
+        EIO
+    }
+
     /// Wait for a message to become available on the message queue.
     ///
     /// This works purely at the transport layer and does not interpret or validate the message
@@ -746,11 +765,13 @@ fn send_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
     /// # Errors
     ///
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
-    /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
-    ///   message queue.
-    ///
-    /// Error codes returned by the message constructor are propagated as-is.
+    /// - `EIO` if the queue is already poisoned, or if the framing or the checksum is invalid,
+    ///   which poisons it (see [`Self::poisoned`]).
     fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
+        if self.poisoned.get() {
+            return Err(EIO);
+        }
+
         // Wait for a message to arrive from the GSP.
         let (slice_1, slice_2) = read_poll_timeout(
             || Ok(self.gsp_mem.driver_read_area()),
@@ -761,7 +782,12 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
         .map(|(slice_1, slice_2)| (slice_1.as_flattened(), slice_2.as_flattened()))?;
 
         // Extract the `GspMsgElement`.
-        let (header, slice_1) = GspMsgElement::from_bytes_prefix(slice_1).ok_or(EIO)?;
+        let Some((header, slice_1)) = GspMsgElement::from_bytes_prefix(slice_1) else {
+            return Err(self.poison(fmt!(
+                "read area of {} bytes is shorter than a message header",
+                slice_1.len()
+            )));
+        };
 
         dev_dbg!(
             &self.dev,
@@ -775,7 +801,11 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
         // Check that the driver read area is large enough for the message.
         if slice_1.len() + slice_2.len() < payload_length {
-            return Err(EIO);
+            return Err(self.poison(fmt!(
+                "message advertises {} payload bytes but only {} are readable",
+                payload_length,
+                slice_1.len() + slice_2.len()
+            )));
         }
 
         // Cut the message slices down to the actual length of the message.
@@ -798,12 +828,10 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
             slice_2,
         ])) != 0
         {
-            dev_err!(
-                &self.dev,
-                "GSP RPC: receive: Call {} - bad checksum\n",
+            return Err(self.poison(fmt!(
+                "message with sequence {} has a bad checksum",
                 header.sequence()
-            );
-            return Err(EIO);
+            )));
         }
 
         Ok(GspMessage {
@@ -817,13 +845,13 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
     /// A message whose function code is `M::FUNCTION` is decoded and returned. Any other message
     /// is logged as an event.
     ///
-    /// The read pointer is always advanced past the message, regardless of whether it matched.
+    /// The read pointer advances past the message in every case, including a decode failure.
     ///
     /// # Errors
     ///
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
-    /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
-    ///   message queue.
+    /// - `EIO` if the queue is poisoned or the message fails framing or checksum validation (see
+    ///   [`Self::wait_for_msg`]), or if the matched message is too short for `M::Message`.
     /// - `ERANGE` if the message was not the awaited reply.
     ///
     /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
@@ -838,20 +866,27 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
 
         // An early return here would leave the read pointer on this message.
         let result = if matches!(function, Ok(f) if f == M::FUNCTION) {
-            let (cmd, contents_1) = M::Message::from_bytes_prefix(message.contents.0).ok_or(EIO)?;
-            let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
-
-            M::read(cmd, &mut sbuffer)
-                .map_err(|e| e.into())
-                .inspect(|_| {
-                    if !sbuffer.is_empty() {
-                        dev_warn!(
-                            &self.dev,
-                            "GSP message {:?} has unprocessed data\n",
-                            M::FUNCTION
-                        );
-                    }
-                })
+            match M::Message::from_bytes_prefix(message.contents.0) {
+                Some((cmd, contents_1)) => {
+                    let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
+
+                    M::read(cmd, &mut sbuffer)
+                        .map_err(|e| e.into())
+                        .inspect(|_| {
+                            if !sbuffer.is_empty() {
+                                dev_warn!(
+                                    &self.dev,
+                                    "GSP message {:?} has unprocessed data\n",
+                                    M::FUNCTION
+                                );
+                            }
+                        })
+                }
+                None => {
+                    dev_warn!(&self.dev, "GSP message {:?} too short\n", M::FUNCTION);
+                    Err(EIO)
+                }
+            }
         } else {
             self.log_event(function, seq);
 
-- 
2.55.0

[PATCH v4 11/17] gpu: nova-core: return ENOMSG for an unmatched GSP message

[John Hubbard]

The GSP posts unsolicited events on the same queue as command replies,
so a receive that asks for one message type has to report that the
message at the queue head was a different one.

That case returned ERANGE, which means a value outside a valid range and
says nothing about a message.

Return ENOMSG, no message of the desired type, instead.

Suggested-by: Gary Guo <gary@garyguo.net>
Suggested-by: Alexandre Courbot <acourbot@nvidia.com>
Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 6 +++---
 drivers/gpu/nova-core/gsp/commands.rs  | 2 +-
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 855c5a708525..45c3c3aea8f9 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -588,7 +588,7 @@ pub(crate) fn send_command<M>(&self, bar: Bar0<'_>, command: M) -> Result<M::Rep
         loop {
             match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
                 Ok(reply) => break Ok(reply),
-                Err(ERANGE) => continue,
+                Err(ENOMSG) => continue,
                 Err(e) => break Err(e),
             }
         }
@@ -852,7 +852,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if the queue is poisoned or the message fails framing or checksum validation (see
     ///   [`Self::wait_for_msg`]), or if the matched message is too short for `M::Message`.
-    /// - `ERANGE` if the message was not the awaited reply.
+    /// - `ENOMSG` if the message was not the awaited reply.
     ///
     /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -890,7 +890,7 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         } else {
             self.log_event(function, seq);
 
-            Err(ERANGE)
+            Err(ENOMSG)
         };
 
         // Advance the read pointer past this message.
diff --git a/drivers/gpu/nova-core/gsp/commands.rs b/drivers/gpu/nova-core/gsp/commands.rs
index e087c9e8c35c..d1c80cf3c452 100644
--- a/drivers/gpu/nova-core/gsp/commands.rs
+++ b/drivers/gpu/nova-core/gsp/commands.rs
@@ -191,7 +191,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
     loop {
         match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
-            Err(ERANGE) => continue,
+            Err(ENOMSG) => continue,
             Err(e) => break Err(e),
         }
     }
diff --git a/drivers/gpu/nova-core/gsp/sequencer.rs b/drivers/gpu/nova-core/gsp/sequencer.rs
index dae34c11eb05..1782ed7d7ca6 100644
--- a/drivers/gpu/nova-core/gsp/sequencer.rs
+++ b/drivers/gpu/nova-core/gsp/sequencer.rs
@@ -346,7 +346,7 @@ pub(crate) fn run(
         let seq_info = loop {
             match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
-                Err(ERANGE) => continue,
+                Err(ENOMSG) => continue,
                 Err(e) => return Err(e),
             }
         };
-- 
2.55.0

[PATCH v4 12/17] gpu: nova-core: bound a GSP wait by a single deadline

[John Hubbard]

The GSP posts unsolicited events on the same queue as command replies,
so a caller waiting for one message consumes whatever arrives first and
reads again.

Every read started a fresh five-second timeout, so a steady stream of
events extended the wait without bound. The two boot-time waits for an
unsolicited event also released the queue mutex between reads, so a
command sent from another thread could consume the event and leave the
waiter to time out.

Compute one deadline when the wait begins and pass the time remaining to
each read, and hold the queue mutex across the whole wait. Put the loop
in one helper that the command reply wait and both boot-time event waits
share.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 73 ++++++++++++++++++++------
 drivers/gpu/nova-core/gsp/commands.rs  |  8 +--
 drivers/gpu/nova-core/gsp/sequencer.rs |  8 +--
 3 files changed, 60 insertions(+), 29 deletions(-)

diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 45c3c3aea8f9..4595aa2176e5 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -32,7 +32,11 @@
         },
         Mutex, //
     },
-    time::Delta,
+    time::{
+        Delta,
+        Instant,
+        Monotonic, //
+    },
     transmute::{
         AsBytes,
         FromBytes, //
@@ -134,7 +138,9 @@ fn size(&self) -> usize {
 
 /// Trait representing messages received from the GSP.
 ///
-/// This trait tells [`Cmdq::receive_msg`] how it can receive a given type of message.
+/// A reply that [`Cmdq::send_command`] waits for, or an event that [`Cmdq::await_msg`] waits for.
+/// The receiver matches a message's function code against [`Self::FUNCTION`] and decodes the
+/// message with [`Self::read`].
 pub(crate) trait MessageFromGsp: Sized {
     /// Function identifying this message from the GSP.
     const FUNCTION: MsgFunction;
@@ -569,8 +575,9 @@ fn notify_gsp(bar: Bar0<'_>) {
     ///
     /// # Errors
     ///
-    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
-    ///   not received within the timeout.
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply does
+    ///   not arrive within [`Self::RECEIVE_TIMEOUT`] of the send, however many events arrive
+    ///   while waiting.
     /// - `EIO` if the variable payload requested by the command has not been entirely
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
     ///
@@ -585,13 +592,7 @@ pub(crate) fn send_command<M>(&self, bar: Bar0<'_>, command: M) -> Result<M::Rep
         let mut inner = self.inner.lock();
         inner.send_command(bar, command)?;
 
-        loop {
-            match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
-                Ok(reply) => break Ok(reply),
-                Err(ENOMSG) => continue,
-                Err(e) => break Err(e),
-            }
-        }
+        inner.await_msg()
     }
 
     /// Sends `command` to the GSP without waiting for a reply.
@@ -611,15 +612,25 @@ pub(crate) fn send_command_no_wait<M>(&self, bar: Bar0<'_>, command: M) -> Resul
         self.inner.lock().send_command(bar, command)
     }
 
-    /// Receive a message from the GSP.
+    /// Waits for an unsolicited GSP event of type `M`. Events that arrive before it are logged and
+    /// consumed.
+    ///
+    /// The queue mutex is held for the whole wait, up to [`Self::RECEIVE_TIMEOUT`], so no other
+    /// caller can send a command or consume an event meanwhile.
     ///
-    /// See [`CmdqInner::receive_msg`] for details.
-    pub(crate) fn receive_msg<M: MessageFromGsp>(&self, timeout: Delta) -> Result<M>
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if the event does not arrive within [`Self::RECEIVE_TIMEOUT`] of the call,
+    ///   however many other events arrive while waiting.
+    /// - `EIO` if the queue is poisoned, or if a message fails framing or checksum validation.
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+    pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
     where
         // This allows all error types, including `Infallible`, to be used for `M::InitError`.
         Error: From<M::InitError>,
     {
-        self.inner.lock().receive_msg(timeout)
+        self.inner.lock().await_msg()
     }
 }
 
@@ -901,6 +912,38 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         result
     }
 
+    /// Receives a message of type `M`, waiting up to [`Cmdq::RECEIVE_TIMEOUT`] from the call.
+    ///
+    /// Any other message that arrives first is logged as an event and does not extend the
+    /// deadline.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if no message of type `M` arrives before the deadline, however many other
+    ///   messages arrive while waiting.
+    /// - `EIO` if the queue is poisoned or a message fails framing or checksum validation (see
+    ///   [`Self::wait_for_msg`]).
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+    fn await_msg<M: MessageFromGsp>(&mut self) -> Result<M>
+    where
+        // This allows all error types, including `Infallible`, to be used for `M::InitError`.
+        Error: From<M::InitError>,
+    {
+        let deadline = Instant::<Monotonic>::now() + Cmdq::RECEIVE_TIMEOUT;
+        loop {
+            let remaining = deadline - Instant::<Monotonic>::now();
+            if remaining.is_negative() {
+                break Err(ETIMEDOUT);
+            }
+            match self.receive_msg::<M>(remaining) {
+                Ok(msg) => break Ok(msg),
+                Err(ENOMSG) => continue,
+                Err(e) => break Err(e),
+            }
+        }
+    }
+
     /// Logs an event, meaning a message that no caller was waiting for.
     ///
     /// An OS error or robust-channel record is logged at error level and an unknown function code
diff --git a/drivers/gpu/nova-core/gsp/commands.rs b/drivers/gpu/nova-core/gsp/commands.rs
index d1c80cf3c452..a01ab14299c6 100644
--- a/drivers/gpu/nova-core/gsp/commands.rs
+++ b/drivers/gpu/nova-core/gsp/commands.rs
@@ -188,13 +188,7 @@ fn read(
 
 /// Waits for GSP initialization to complete.
 pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
-    loop {
-        match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(_) => break Ok(()),
-            Err(ENOMSG) => continue,
-            Err(e) => break Err(e),
-        }
-    }
+    cmdq.await_msg::<GspInitDone>().map(|_| ())
 }
 
 /// The `GetGspStaticInfo` command.
diff --git a/drivers/gpu/nova-core/gsp/sequencer.rs b/drivers/gpu/nova-core/gsp/sequencer.rs
index 1782ed7d7ca6..250adc9fe74f 100644
--- a/drivers/gpu/nova-core/gsp/sequencer.rs
+++ b/drivers/gpu/nova-core/gsp/sequencer.rs
@@ -343,13 +343,7 @@ pub(crate) fn run(
         libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
         bootloader_app_version: u32,
     ) -> Result {
-        let seq_info = loop {
-            match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
-                Ok(seq_info) => break seq_info,
-                Err(ENOMSG) => continue,
-                Err(e) => return Err(e),
-            }
-        };
+        let seq_info = cmdq.await_msg::<GspSequence>()?;
 
         let sequencer = GspSequencer {
             bar: ctx.bar,
-- 
2.55.0

[PATCH v4 13/17] gpu: nova-core: add a GSP message queue drain

[John Hubbard]

The GSP posts unsolicited events on the GSP-to-CPU queue whenever it has
something to report, whether or not a caller is waiting for a reply.

Every existing way to read the queue asks for one message type and waits
on a deadline. A caller that only wants to empty the queue had nothing
to call.

Add a drain that logs and consumes whatever the GSP has already posted
and returns as soon as the queue is empty, without waiting. It holds the
queue mutex, so no reply is among the messages it reads, and it treats
every one of them as an event.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 41 +++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 4595aa2176e5..acee444e898d 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -632,6 +632,21 @@ pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
     {
         self.inner.lock().await_msg()
     }
+
+    /// Logs and consumes every message the GSP has already posted, and returns without waiting for
+    /// more.
+    ///
+    /// No caller is waiting for a reply while this holds the queue mutex, so every message is
+    /// logged as an event. See "Draining the GSP-to-CPU queue" in
+    /// `Documentation/gpu/nova/core/interrupts.rst`.
+    ///
+    /// # Errors
+    ///
+    /// `EIO` if the queue is poisoned, or if a message fails framing or checksum validation.
+    #[expect(dead_code)]
+    pub(crate) fn drain(&self) -> Result {
+        self.inner.lock().drain()
+    }
 }
 
 /// Inner mutex protected state of [`Cmdq`].
@@ -973,4 +988,30 @@ fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
             }
         }
     }
+
+    /// Logs and consumes every message the queue holds.
+    ///
+    /// # Errors
+    ///
+    /// `EIO` if the queue is poisoned, a message fails framing or checksum validation, or a
+    /// message's page count overflows a `u32`.
+    fn drain(&mut self) -> Result {
+        while !self.gsp_mem.driver_read_area().0.is_empty() {
+            // A message is available, so this returns without waiting.
+            let msg = self.wait_for_msg(Delta::ZERO)?;
+
+            let pages =
+                u32::try_from(msg.header.length().div_ceil(GSP_PAGE_SIZE)).map_err(|_| {
+                    dev_err!(&self.dev, "GSP drain: message length overflow\n");
+                    EIO
+                })?;
+            let function = msg.header.function();
+            let seq = msg.header.sequence();
+
+            self.gsp_mem.advance_cpu_read_ptr(pages);
+            self.log_event(function, seq);
+        }
+
+        Ok(())
+    }
 }
-- 
2.55.0

[PATCH v4 14/17] gpu: nova-core: add the falcon interrupt registers and their HAL

[John Hubbard]

A falcon has a set of interrupt causes, and it latches each one that is
raised in its IRQSTAT register. On a RISC-V falcon, each cause is routed
either to the host, meaning the CPU, or to the falcon's own RISC-V core,
and IRQSTAT holds the causes of both. Two more registers say which is
which: PRISCV_RISCV_IRQMASK holds the enabled causes, and
PRISCV_RISCV_IRQDEST holds the causes routed to the host. Open RM
intersects the three to get the causes that the host has to service,
and nova-core does the same.

A falcon signals the interrupt tree only when its set of host-routed
causes goes from empty to non-empty. A handler that clears the tree
leaf while a cause is still latched in the falcon leaves that set
non-empty, so no later cause produces a transition, and the falcon's
interrupts stop arriving. INTR_RETRIGGER makes the falcon re-emit its
host-routed causes into the tree, which supplies the missing
transition. Turing falcons do not implement it.

IRQSCLR clears a cause's latch, but it cannot end the source behind the
cause. A cause driven from outside the falcon, such as a fault
containment or ECC error on Blackwell, stays set through the write.

Add the four registers: IRQSTAT, INTR_RETRIGGER, and the two routing
registers. Record the IRQSCLR limit on its existing definition. The
routing registers go in per-chip modules, because their offsets move at
GA102 rather than at the Turing-to-Ampere boundary.

Add a HAL for the two properties that follow, the retrigger register
and the routing offsets, which split the chipsets three ways:

* Turing falcons have no retrigger register.

* GA100 has the retrigger register, and keeps the Turing routing
  offsets.

* GA102 and later have the retrigger register, and their routing
  registers moved.

Keep this HAL apart from the falcon boot HAL. The boot HAL is generic
over the falcon's engine type, so obtaining one is a heap allocation,
and the interrupt handler that needs these two properties runs in hard
interrupt context, where it cannot allocate.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/falcon/hal.rs       | 80 ++++++++++++++++++++++-
 drivers/gpu/nova-core/falcon/hal/ga102.rs | 21 +++++-
 drivers/gpu/nova-core/falcon/hal/tu102.rs | 36 +++++++++-
 drivers/gpu/nova-core/regs.rs             | 69 +++++++++++++++++++
 4 files changed, 202 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 7e532889a1f4..052610c4a4da 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -1,17 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0
 
-use kernel::prelude::*;
+use kernel::{
+    io::{
+        Io,
+        Mmio, //
+    },
+    prelude::*, //
+};
 
 use crate::{
     falcon::{
         Falcon,
         FalconBromParams,
-        FalconEngine, //
+        FalconEngine,
+        PFalcon2Registers, //
     },
     gpu::{
         Architecture,
         Chipset, //
     },
+    regs,
 };
 
 mod ga102;
@@ -72,6 +80,74 @@ fn signature_reg_fuse_version(
     fn load_method(&self) -> LoadMethod;
 }
 
+/// Offsets of a falcon's RISC-V interrupt routing registers.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+#[expect(dead_code)]
+pub(crate) enum RiscvRouting {
+    /// The Turing offsets. GA100 uses them too.
+    Tu102,
+
+    /// The offsets from GA102 on.
+    Ga102,
+}
+
+impl RiscvRouting {
+    /// Returns the causes in `latched` that are routed to the host, meaning the CPU, rather than
+    /// to the falcon's own RISC-V core.
+    ///
+    /// The causes routed to the core belong to the firmware running on it, and the host does not
+    /// service them.
+    #[expect(dead_code)]
+    pub(crate) fn host_routed_causes(
+        self,
+        pfalcon2: Mmio<'_, PFalcon2Registers>,
+        latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let (mask, dest) = match self {
+            Self::Tu102 => (
+                pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQMASK).value(),
+                pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQDEST).value(),
+            ),
+            Self::Ga102 => (
+                pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQMASK).value(),
+                pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQDEST).value(),
+            ),
+        };
+
+        regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
+    }
+}
+
+/// Interrupt properties of a falcon that differ by GPU family.
+///
+/// Separate from [`FalconHal`] because the GSP event handler calls these from hard interrupt
+/// context, where it cannot make the heap allocation that a `FalconHal` takes.
+#[expect(dead_code)]
+pub(crate) trait FalconIntrHal {
+    /// Returns whether these falcons implement `NV_PFALCON_FALCON_INTR_RETRIGGER`.
+    fn has_intr_retrigger(&self) -> bool;
+
+    /// Returns the offsets of `PRISCV_RISCV_IRQMASK` and `PRISCV_RISCV_IRQDEST`.
+    fn riscv_routing(&self) -> RiscvRouting;
+}
+
+/// Returns the [`FalconIntrHal`] for `chipset`.
+///
+/// GA100 has its own arm: it has the retrigger register, which Turing lacks, and the Turing
+/// routing offsets, which GA102 moved.
+#[expect(dead_code)]
+pub(crate) fn falcon_intr_hal(chipset: Chipset) -> &'static dyn FalconIntrHal {
+    match chipset.arch() {
+        Architecture::Turing => tu102::TU102_INTR_HAL,
+        Architecture::Ampere if chipset == Chipset::GA100 => tu102::GA100_INTR_HAL,
+        Architecture::Ampere
+        | Architecture::Ada
+        | Architecture::Hopper
+        | Architecture::BlackwellGB10x
+        | Architecture::BlackwellGB20x => ga102::GA102_INTR_HAL,
+    }
+}
+
 /// Returns a boxed falcon HAL adequate for `chipset`.
 ///
 /// We use a heap-allocated trait object instead of a statically defined one because the
diff --git a/drivers/gpu/nova-core/falcon/hal/ga102.rs b/drivers/gpu/nova-core/falcon/hal/ga102.rs
index f9a8444cf840..ff97983f22fe 100644
--- a/drivers/gpu/nova-core/falcon/hal/ga102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/ga102.rs
@@ -28,7 +28,11 @@
     regs,
 };
 
-use super::FalconHal;
+use super::{
+    FalconHal,
+    FalconIntrHal,
+    RiscvRouting, //
+};
 
 fn select_core_ga102(pfalcon2: Mmio<'_, PFalcon2Registers>) -> Result {
     let bcr_ctrl = pfalcon2.read(regs::NV_PRISCV_RISCV_BCR_CTRL);
@@ -170,3 +174,18 @@ fn load_method(&self) -> LoadMethod {
         LoadMethod::Dma
     }
 }
+
+/// The falcon interrupt properties of GA102 and later.
+struct Ga102Intr;
+
+impl FalconIntrHal for Ga102Intr {
+    fn has_intr_retrigger(&self) -> bool {
+        true
+    }
+
+    fn riscv_routing(&self) -> RiscvRouting {
+        RiscvRouting::Ga102
+    }
+}
+
+pub(super) const GA102_INTR_HAL: &dyn FalconIntrHal = &Ga102Intr;
diff --git a/drivers/gpu/nova-core/falcon/hal/tu102.rs b/drivers/gpu/nova-core/falcon/hal/tu102.rs
index 7fc6e83c2566..f79aa85e6a62 100644
--- a/drivers/gpu/nova-core/falcon/hal/tu102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/tu102.rs
@@ -21,7 +21,11 @@
     regs, //
 };
 
-use super::FalconHal;
+use super::{
+    FalconHal,
+    FalconIntrHal,
+    RiscvRouting, //
+};
 
 pub(super) struct Tu102<E: FalconEngine>(PhantomData<E>);
 
@@ -80,3 +84,33 @@ fn load_method(&self) -> LoadMethod {
         LoadMethod::Pio
     }
 }
+
+/// The falcon interrupt properties of Turing.
+struct Tu102Intr;
+
+impl FalconIntrHal for Tu102Intr {
+    fn has_intr_retrigger(&self) -> bool {
+        false
+    }
+
+    fn riscv_routing(&self) -> RiscvRouting {
+        RiscvRouting::Tu102
+    }
+}
+
+pub(super) const TU102_INTR_HAL: &dyn FalconIntrHal = &Tu102Intr;
+
+/// GA100's falcon interrupt properties: the Turing routing offsets and the retrigger register.
+struct Ga100Intr;
+
+impl FalconIntrHal for Ga100Intr {
+    fn has_intr_retrigger(&self) -> bool {
+        true
+    }
+
+    fn riscv_routing(&self) -> RiscvRouting {
+        RiscvRouting::Tu102
+    }
+}
+
+pub(super) const GA100_INTR_HAL: &dyn FalconIntrHal = &Ga100Intr;
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 9978fb2803b0..c6ba226dcfe3 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -124,11 +124,25 @@ pub(crate) fn usable_fb_size(self) -> u64 {
 register! {
     base: PFalconRegisters;
 
+    /// Clears the latch of every cause whose bit is written as `1`. Write-only.
+    ///
+    /// The write ends the latch and not the source, so a cause driven from outside the falcon
+    /// stays set. "Retriggering a falcon" in `Documentation/gpu/nova/core/interrupts.rst` names
+    /// those causes.
     pub(crate) NV_PFALCON_FALCON_IRQSCLR(u32) @ 0x00000004 {
         6:6     swgen0 => bool;
         4:4     halt => bool;
     }
 
+    /// Interrupt causes latched in the falcon, one bit per cause, whichever target each is routed
+    /// to.
+    ///
+    /// The causes routed to the host are the ones also set in `NV_PRISCV_RISCV_IRQMASK` and
+    /// `NV_PRISCV_RISCV_IRQDEST`.
+    pub(crate) NV_PFALCON_FALCON_IRQSTAT(u32) @ 0x00000008 {
+        6:6     swgen0 => bool;
+    }
+
     pub(crate) NV_PFALCON_FALCON_MAILBOX0(u32) @ 0x00000040 {
         31:0    value => u32;
     }
@@ -256,6 +270,16 @@ pub(crate) fn usable_fb_size(self) -> u64 {
         0:0     reset => bool;
     }
 
+    /// Makes the falcon re-emit its host-routed causes into the interrupt tree. Write-only.
+    ///
+    /// Present from GA100 on. See "Retriggering a falcon" in
+    /// `Documentation/gpu/nova/core/interrupts.rst`.
+    ///
+    /// The hardware headers declare two elements, and Open RM writes only the first.
+    pub(crate) NV_PFALCON_FALCON_INTR_RETRIGGER(u32)[2] @ 0x000003e8 {
+        0:0     trigger => bool;
+    }
+
     pub(crate) NV_PFALCON_FBIF_TRANSCFG(u32)[8] @ 0x00000600 {
         2:2     mem_type => FalconFbifMemType;
         1:0     target ?=> FalconFbifTarget;
@@ -414,6 +438,29 @@ pub(crate) mod gm107 {
     }
 }
 
+pub(crate) mod tu102 {
+    use kernel::io::register;
+
+    use crate::falcon::PFalcon2Registers;
+
+    // The RISC-V interrupt routing registers, at the offsets that Turing and GA100 use.
+
+    register! {
+        base: PFalcon2Registers;
+
+        /// Enabled causes, one bit per cause. Read-only to the host.
+        pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x000002b4 {
+            31:0    value => u32;
+        }
+
+        /// Causes routed to the host, one bit per cause. A clear bit routes the cause to the
+        /// RISC-V core.
+        pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x000002b8 {
+            31:0    value => u32;
+        }
+    }
+}
+
 pub(crate) mod ga100 {
     use kernel::io::register;
 
@@ -430,6 +477,28 @@ pub(crate) mod ga100 {
     }
 }
 
+pub(crate) mod ga102 {
+    use kernel::io::register;
+
+    use crate::falcon::PFalcon2Registers;
+
+    // The RISC-V interrupt routing registers, at the offsets that GA102 and later use.
+
+    register! {
+        base: PFalcon2Registers;
+
+        /// Same as [`super::tu102::NV_PRISCV_RISCV_IRQMASK`], at the GA102 offset.
+        pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x00000528 {
+            31:0    value => u32;
+        }
+
+        /// Same as [`super::tu102::NV_PRISCV_RISCV_IRQDEST`], at the GA102 offset.
+        pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x0000052c {
+            31:0    value => u32;
+        }
+    }
+}
+
 pub(crate) const NV_THERM_I2CS_SCRATCH_FSP_BOOT_COMPLETE_STATUS_SUCCESS: u32 = 0xff;
 
 pub(crate) mod gh100 {
-- 
2.55.0

[PATCH v4 15/17] gpu: nova-core: service GSP events from the SWGEN0 interrupt

[John Hubbard]

When the GSP has something for the CPU, it posts a message to the
GSP-to-CPU queue and raises SWGEN0, a software-generated interrupt cause
of its falcon. SWGEN0 is routed to the host and reaches the CPU on GIN
vector 155. It is a latch: while it stays set, the GSP cannot signal the
tree again. GSP boot consumes the GSP's notifications by polling the
queue, so it leaves the latch set and pending bits behind in the tree.

nova-core read the queue only while a caller was waiting for a command
reply, so an event posted between commands sat unread until the next
command was sent.

Register a threaded handler on the GSP vector. The top half runs in
hard interrupt context and touches only registers:

* Read the GSP vector's leaf and clear its bit, leaving any other
  vector in the leaf pending.

* Read the falcon causes routed to the host, and clear the SWGEN0 latch
  if it is set.

* Clear the latch of every other host cause, then retrigger the falcon
  or disable the vector, as described below.

* Rearm PCI interrupt delivery.

Draining the queue takes the command-queue mutex, which can sleep, and
walks shared memory, so the top half leaves it to the IRQ thread and
wakes the thread when SWGEN0 was set.

A host cause other than SWGEN0 reports a GSP fault. IRQSCLR clears a
cause's latch but cannot end the source behind it, and on Blackwell the
fault-containment and ECC causes are driven from outside the falcon, so
they stay set through the write. A retrigger would then re-emit them at
once, and the CPU would take the same interrupt again and again. So
after clearing the fault latches, read the host causes back. If the
clear ended every one of them, retrigger the falcon, so that a cause
latched in the meantime still signals the tree. If a cause is still
set, disable the GSP vector at its leaf instead and log that the device
needs a reset. Disabling loses nothing: while a cause stays set, the
falcon signals nothing further either way.

Make the handler registration one of the GPU's resources, rather than
something probe registers, so that it drops before the command queue
the handler drains is freed and before the GSP is unloaded. Before
registering, quiesce the tree and clear the SWGEN0 latch, so nothing
left over from boot reaches a handler that services one vector and
cannot service any other. The quiesce leaves the GSP subtree disabled at
TOP, and the pre-Hopper MSI rearm is a configuration-space write that
does not enable it again, so enable the subtree explicitly. Keep it
enabled for as long as the handler is registered, and disable it only
after the handler is freed, since a handler still in flight would enable
it again through its rearm. Once the handler is in place, drain the
queue once: a message posted before the latch was cleared produced no
interrupt.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/falcon/gsp.rs         |  73 +++++-
 drivers/gpu/nova-core/falcon/hal.rs         |   4 -
 drivers/gpu/nova-core/gpu.rs                |  57 ++++-
 drivers/gpu/nova-core/gsp.rs                |   2 +-
 drivers/gpu/nova-core/gsp/cmdq.rs           |   1 -
 drivers/gpu/nova-core/irq.rs                |  43 +++-
 drivers/gpu/nova-core/irq/gsp.rs            | 236 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs |   4 +-
 drivers/gpu/nova-core/nova_core.rs          |   1 -
 9 files changed, 399 insertions(+), 22 deletions(-)
 create mode 100644 drivers/gpu/nova-core/irq/gsp.rs

diff --git a/drivers/gpu/nova-core/falcon/gsp.rs b/drivers/gpu/nova-core/falcon/gsp.rs
index 4c96ae325fda..dfa08bc6867c 100644
--- a/drivers/gpu/nova-core/falcon/gsp.rs
+++ b/drivers/gpu/nova-core/falcon/gsp.rs
@@ -5,6 +5,7 @@
         io_project,
         poll::read_poll_timeout,
         register,
+        register::Array,
         Io,
         Mmio, //
     },
@@ -18,9 +19,11 @@
         NovaRegisters, //
     },
     falcon::{
+        hal,
         Falcon,
         FalconEngine, //
     },
+    gpu::Chipset,
     regs,
 };
 
@@ -46,14 +49,72 @@ fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
     }
 }
 
-impl<'a> Falcon<'a, Gsp> {
-    /// Clears the SWGEN0 bit in the Falcon's IRQ status clear register to
-    /// allow GSP to signal CPU for processing new messages in message queue.
-    pub(crate) fn clear_swgen0_intr(&self) {
-        self.pfalcon
-            .write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true));
+impl Gsp {
+    /// Clears the SWGEN0 latch in the GSP falcon.
+    ///
+    /// While the latch is set, no later message signals the tree, so a caller that consumed a
+    /// notification by polling must clear it.
+    pub(crate) fn clear_swgen0_intr(bar: Bar0<'_>) {
+        Self::pfalcon(bar).write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true));
+    }
+
+    /// Reads the GSP falcon causes that are routed to the host, without clearing any latch.
+    ///
+    /// Every one of them other than SWGEN0 reports a GSP fault.
+    pub(crate) fn read_host_intr(
+        bar: Bar0<'_>,
+        chipset: Chipset,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let latched = Self::pfalcon(bar).read(regs::NV_PFALCON_FALCON_IRQSTAT);
+
+        hal::falcon_intr_hal(chipset)
+            .riscv_routing()
+            .host_routed_causes(Self::pfalcon2(bar), latched)
+    }
+
+    /// Reads the host-routed causes and clears the SWGEN0 latch if it was set.
+    ///
+    /// Returns the causes as read, before the clear. No other latch changes.
+    pub(crate) fn take_host_intr(
+        bar: Bar0<'_>,
+        chipset: Chipset,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let status = Self::read_host_intr(bar, chipset);
+
+        if status.swgen0() {
+            Self::clear_swgen0_intr(bar);
+        }
+
+        status
+    }
+
+    /// Clears the latch of every interrupt cause set in `status`.
+    ///
+    /// A cause driven from outside the falcon is still set on return, and
+    /// [`Self::read_host_intr`] reports the causes that remain.
+    pub(crate) fn clear_intr(bar: Bar0<'_>, status: regs::NV_PFALCON_FALCON_IRQSTAT) {
+        Self::pfalcon(bar).write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::from(status.into_raw()));
+    }
+
+    /// Retriggers the GSP falcon, which then re-emits its host-routed causes into the tree.
+    ///
+    /// Call this only once every host cause is clear. A cause still set is re-emitted at once, and
+    /// its vector arrives again as soon as delivery is rearmed.
+    ///
+    /// Does nothing on Turing, whose falcons have no retrigger register.
+    pub(crate) fn retrigger_intr(bar: Bar0<'_>, chipset: Chipset) {
+        if !hal::falcon_intr_hal(chipset).has_intr_retrigger() {
+            return;
+        }
+
+        Self::pfalcon(bar).write(
+            Array::at(0),
+            regs::NV_PFALCON_FALCON_INTR_RETRIGGER::zeroed().with_trigger(true),
+        );
     }
+}
 
+impl<'a> Falcon<'a, Gsp> {
     /// Checks if GSP reload/resume has completed during the boot process.
     pub(crate) fn check_reload_completed(&self, timeout: Delta) -> Result<bool> {
         read_poll_timeout(
diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 052610c4a4da..3f1f509eccbd 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -82,7 +82,6 @@ fn signature_reg_fuse_version(
 
 /// Offsets of a falcon's RISC-V interrupt routing registers.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-#[expect(dead_code)]
 pub(crate) enum RiscvRouting {
     /// The Turing offsets. GA100 uses them too.
     Tu102,
@@ -97,7 +96,6 @@ impl RiscvRouting {
     ///
     /// The causes routed to the core belong to the firmware running on it, and the host does not
     /// service them.
-    #[expect(dead_code)]
     pub(crate) fn host_routed_causes(
         self,
         pfalcon2: Mmio<'_, PFalcon2Registers>,
@@ -122,7 +120,6 @@ pub(crate) fn host_routed_causes(
 ///
 /// Separate from [`FalconHal`] because the GSP event handler calls these from hard interrupt
 /// context, where it cannot make the heap allocation that a `FalconHal` takes.
-#[expect(dead_code)]
 pub(crate) trait FalconIntrHal {
     /// Returns whether these falcons implement `NV_PFALCON_FALCON_INTR_RETRIGGER`.
     fn has_intr_retrigger(&self) -> bool;
@@ -135,7 +132,6 @@ pub(crate) trait FalconIntrHal {
 ///
 /// GA100 has its own arm: it has the retrigger register, which Turing lacks, and the Turing
 /// routing offsets, which GA102 moved.
-#[expect(dead_code)]
 pub(crate) fn falcon_intr_hal(chipset: Chipset) -> &'static dyn FalconIntrHal {
     match chipset.arch() {
         Architecture::Turing => tu102::TU102_INTR_HAL,
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 3d796d6c7013..d1e0da7b8682 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -38,6 +38,11 @@
         Gsp,
         GspBootContext, //
     },
+    irq::{
+        self,
+        gsp::GspIrq,
+        SubtreeVectors, //
+    },
     mm::{
         bar_user::BarUser,
         pagetable::MmuVersion,
@@ -301,6 +306,13 @@ struct GspResources<'gpu> {
 #[pin_data]
 pub(crate) struct Gpu<'gpu> {
     spec: Spec,
+    /// GSP event interrupt registration.
+    ///
+    /// Must be kept declared *before* `gsp_resources`, so that the handler is unregistered, and
+    /// any in-flight run of it has finished, before the command queue it drains is freed and
+    /// before the GSP is unloaded.
+    #[pin]
+    _gsp_irq: GspIrq<'gpu>,
     /// Static GPU information as provided by the GSP.
     gsp_static_info: GetGspStaticInfoReply,
     /// GPU memory manager owning memory management resources.
@@ -319,6 +331,14 @@ pub(crate) struct Gpu<'gpu> {
     /// Must be kept declared *after* `gsp_resources`, as the latter's `PinnedDrop` implementation
     /// requires the sysmem flush page to be in place.
     sysmem_flush: SysmemFlush<'gpu>,
+    /// Borrow of `vectors` that `_gsp_irq` holds. A field that borrows a sibling field is
+    /// self-referential, which `pin_init` cannot express, so the borrow is taken by hand.
+    vectors_ref: &'gpu SubtreeVectors<'gpu>,
+    /// PCI interrupt vector allocation.
+    ///
+    /// Must be kept declared *after* `_gsp_irq`, which holds a borrow of it.
+    #[pin]
+    vectors: SubtreeVectors<'gpu>,
 }
 
 #[pinned_drop]
@@ -358,6 +378,12 @@ pub(crate) fn new<'a>(
         let dev = pdev.as_ref();
 
         try_pin_init!(Self {
+            vectors: irq::alloc_vectors(pdev, irq::gsp::GSP_SUBTREE.into())?,
+
+            // SAFETY: `vectors` is initialized above, is pinned at a stable address, and is
+            // dropped after every field that uses `vectors_ref` (struct field drop order).
+            vectors_ref: unsafe { &*core::ptr::from_ref(vectors.as_ref().get_ref()) },
+
             spec: Spec::new(dev, bar).inspect(|spec| {
                 dev_info!(dev,"NVIDIA ({})\n", spec);
             })?,
@@ -380,12 +406,7 @@ pub(crate) fn new<'a>(
 
                 bar,
 
-                gsp_falcon: Falcon::new(
-                    dev,
-                    spec.chipset,
-                    bar
-                )
-                .inspect(|falcon| falcon.clear_swgen0_intr())?,
+                gsp_falcon: Falcon::new(dev, spec.chipset, bar)?,
 
                 sec2_falcon: Falcon::new(dev, spec.chipset, bar)?,
 
@@ -409,6 +430,30 @@ pub(crate) fn new<'a>(
                 })?,
             }),
 
+            _: {
+                irq::gsp::quiesce(bar, gsp_resources.spec.chipset, vectors_ref)?;
+            },
+
+            // SAFETY: the command queue is a field of `gsp_resources`, which is initialized
+            // above and pinned, so the reference outlives the registration. The registration is
+            // a field of `Gpu` and is never leaked, so its `Drop` runs, and field drop order
+            // runs it before the queue is freed.
+            _gsp_irq <- unsafe {
+                GspIrq::new(
+                    pdev,
+                    vectors_ref,
+                    bar,
+                    &*core::ptr::from_ref(&gsp_resources.gsp.cmdq),
+                    gsp_resources.spec.chipset,
+                )
+            },
+
+            // No interrupt announces the messages that the GSP posted during boot, before the
+            // SWGEN0 latch was cleared.
+            _: {
+                gsp_resources.gsp.cmdq.drain()?;
+            },
+
             gsp_static_info: {
                 // Obtain and display basic GPU information.
                 let info = gsp_resources.gsp.get_static_info(bar)?;
diff --git a/drivers/gpu/nova-core/gsp.rs b/drivers/gpu/nova-core/gsp.rs
index 25ea43f1cbe9..fcfb4210d435 100644
--- a/drivers/gpu/nova-core/gsp.rs
+++ b/drivers/gpu/nova-core/gsp.rs
@@ -152,7 +152,7 @@ pub(crate) struct Gsp<'gsp> {
     /// Log buffers, optionally exposed via debugfs.
     #[pin]
     logs: debugfs::Scope<LogBuffers<'gsp>>,
-    /// Command queue.
+    /// Command queue, borrowed by the GSP event interrupt handler.
     #[pin]
     pub(crate) cmdq: Cmdq<'gsp>,
     /// RM arguments.
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index acee444e898d..f1231569aa33 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -643,7 +643,6 @@ pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
     /// # Errors
     ///
     /// `EIO` if the queue is poisoned, or if a message fails framing or checksum validation.
-    #[expect(dead_code)]
     pub(crate) fn drain(&self) -> Result {
         self.inner.lock().drain()
     }
diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 7fb7d9f2e237..cafcc613770a 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -11,6 +11,7 @@
 
 #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
 pub(crate) mod doorbell_test;
+pub(crate) mod gsp;
 mod hal;
 mod interrupt_tree;
 mod regs;
@@ -25,11 +26,16 @@
     prelude::*, //
 };
 
-use crate::num;
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    num, //
+};
 
 use interrupt_tree::{
     Subtree,
-    SubtreeSet, //
+    SubtreeSet,
+    Tree, //
 };
 
 /// The message-signaled interrupt type that Linux granted.
@@ -56,6 +62,39 @@ pub(crate) struct SubtreeVectors<'a> {
 }
 
 impl SubtreeVectors<'_> {
+    /// Returns the tree of `chipset`, covering the serviced subtrees.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `chipset` does not implement every serviced subtree.
+    fn tree<'b>(&self, bar: Bar0<'b>, chipset: Chipset) -> Result<Tree<'b>> {
+        Tree::new(bar, chipset, self)
+    }
+
+    /// Disables every vector in the tree, clears every pending bit, and rearms PCI interrupt
+    /// delivery.
+    ///
+    /// On return, the serviced subtrees are enabled at `TOP` under a `TOP` rearm method and
+    /// disabled under the configuration-space one. A caller that needs delivery enables them
+    /// itself.
+    ///
+    /// Call this only during probe, with no interrupt handler registered.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `chipset` does not implement every serviced subtree.
+    pub(crate) fn reset_tree(&self, bar: Bar0<'_>, chipset: Chipset) -> Result {
+        let tree = self.tree(bar, chipset)?;
+
+        tree.disable_all_leaves();
+        tree.drain();
+        for subtree in self.serviced.iter() {
+            tree.rearm_pci_irq(subtree);
+        }
+
+        Ok(())
+    }
+
     /// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
     ///
     /// # Errors
diff --git a/drivers/gpu/nova-core/irq/gsp.rs b/drivers/gpu/nova-core/irq/gsp.rs
new file mode 100644
index 000000000000..8996f4215e40
--- /dev/null
+++ b/drivers/gpu/nova-core/irq/gsp.rs
@@ -0,0 +1,236 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! The GSP event interrupt.
+//!
+//! The GSP posts messages to the GSP-to-CPU queue and raises SWGEN0, a software-generated cause
+//! of its falcon. A threaded handler services it: the top half clears the tree and falcon state,
+//! and the IRQ thread drains the queue.
+//!
+//! See "The GSP event" in `Documentation/gpu/nova/core/interrupts.rst`.
+
+use kernel::{
+    device,
+    irq,
+    pci,
+    prelude::*, //
+};
+
+use super::{
+    interrupt_tree::{
+        GinVector,
+        LeafEnableGuard,
+        Subtree,
+        TopEnableGuard,
+        Tree, //
+    },
+    SubtreeVectors, //
+};
+use crate::{
+    driver::Bar0,
+    falcon::gsp::Gsp as GspFalcon,
+    gpu::Chipset,
+    gsp::cmdq::Cmdq,
+    regs, //
+};
+
+/// The GSP event vector, which has the same number on every supported GPU.
+const GSP_INTR_0_VECTOR: GinVector = GinVector::new::<155>();
+
+/// The GSP event's subtree, the only one that nova-core services.
+pub(crate) const GSP_SUBTREE: Subtree = GSP_INTR_0_VECTOR.subtree();
+
+/// Clears the tree and falcon interrupt state that GSP boot leaves behind, and rearms PCI
+/// interrupt delivery.
+///
+/// On return, no vector is enabled at its leaf and the SWGEN0 latch is clear, so the next message
+/// that the GSP posts signals the tree.
+///
+/// # Errors
+///
+/// `EINVAL` if `chipset` does not implement every subtree that `vectors` services.
+pub(crate) fn quiesce(bar: Bar0<'_>, chipset: Chipset, vectors: &SubtreeVectors<'_>) -> Result {
+    vectors.reset_tree(bar, chipset)?;
+    // The latch is cleared after the tree reset. The other order can leave the latch set with its
+    // leaf bit cleared. See "Enabling the GSP event" in interrupts.rst.
+    GspFalcon::clear_swgen0_intr(bar);
+
+    Ok(())
+}
+
+/// Threaded IRQ handler for the GSP event.
+pub(crate) struct GspInterrupt<'a> {
+    /// For the GSP falcon's registers. The tree holds its own copy.
+    bar: Bar0<'a>,
+    cmdq: &'a Cmdq<'a>,
+    tree: Tree<'a>,
+    /// Selects the falcon's retrigger and routing registers, which differ by family.
+    chipset: Chipset,
+    /// For logging. The command queue's device reference is behind its mutex, which the top half
+    /// cannot take.
+    dev: &'a device::Device,
+}
+
+impl<'a> GspInterrupt<'a> {
+    fn new(
+        bar: Bar0<'a>,
+        cmdq: &'a Cmdq<'a>,
+        tree: Tree<'a>,
+        chipset: Chipset,
+        dev: &'a device::Device,
+    ) -> Self {
+        Self {
+            bar,
+            cmdq,
+            tree,
+            chipset,
+            dev,
+        }
+    }
+
+    /// Clears the latch of every host-routed cause in `status` other than SWGEN0, and logs them.
+    ///
+    /// Returns the causes still set after the clear. A cause driven from outside the falcon stays
+    /// set, and only a device reset ends it.
+    fn clear_faults(
+        &self,
+        status: regs::NV_PFALCON_FALCON_IRQSTAT,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let faults = status.with_swgen0(false);
+        if faults.into_raw() == 0 {
+            return faults;
+        }
+
+        dev_err!(
+            &self.dev,
+            "unserviceable GSP falcon interrupt, IRQSTAT {:#x}\n",
+            status.into_raw()
+        );
+        GspFalcon::clear_intr(self.bar, faults);
+
+        GspFalcon::read_host_intr(self.bar, self.chipset).with_swgen0(false)
+    }
+}
+
+impl irq::ThreadedHandler for GspInterrupt<'_> {
+    /// Top half, in hard interrupt context. Services the GSP vector only, so another vector
+    /// pending in the same leaf stays pending.
+    fn handle(&self) -> irq::ThreadedIrqReturn {
+        let bar = self.bar;
+
+        let leaf = self.tree.read_pending(GSP_INTR_0_VECTOR.leaf_index());
+        if !leaf.vectors().contains(GSP_INTR_0_VECTOR.leaf_mask()) {
+            self.tree.rearm_pci_irq(GSP_SUBTREE);
+            return irq::ThreadedIrqReturn::None;
+        }
+        leaf.clear_vectors(GSP_INTR_0_VECTOR.leaf_mask());
+
+        let status = GspFalcon::take_host_intr(bar, self.chipset);
+
+        let remaining_faults = self.clear_faults(status);
+        if remaining_faults.into_raw() == 0 {
+            GspFalcon::retrigger_intr(bar, self.chipset);
+        } else {
+            // Disabling the vector loses no notification: the falcon signals nothing further
+            // while a cause stays set. See "Retriggering a falcon" in interrupts.rst.
+            self.tree.disable_leaf(
+                GSP_INTR_0_VECTOR.leaf_index(),
+                GSP_INTR_0_VECTOR.leaf_mask(),
+            );
+            dev_err!(
+                &self.dev,
+                "GSP falcon cause {:#x} needs a device reset, GSP events are no longer serviced\n",
+                remaining_faults.into_raw()
+            );
+        }
+
+        self.tree.rearm_pci_irq(GSP_SUBTREE);
+
+        if status.swgen0() {
+            irq::ThreadedIrqReturn::WakeThread
+        } else {
+            irq::ThreadedIrqReturn::Handled
+        }
+    }
+
+    /// IRQ thread. Drains the GSP-to-CPU queue, which may sleep.
+    fn handle_threaded(&self) -> irq::IrqReturn {
+        if let Err(e) = self.cmdq.drain() {
+            // A poisoned queue fails every later drain the same way.
+            self.tree.disable_leaf(
+                GSP_INTR_0_VECTOR.leaf_index(),
+                GSP_INTR_0_VECTOR.leaf_mask(),
+            );
+            dev_err!(
+                &self.dev,
+                "GSP event drain failed ({:?}), the message queue is no longer serviced\n",
+                e
+            );
+        }
+        irq::IrqReturn::Handled
+    }
+}
+
+/// The registered GSP event handler and the enables that deliver to it.
+///
+/// The declaration order is the drop order, and it is required: the vector is disabled first,
+/// `free_irq` runs second, and the subtree is disabled last. See "Enabling the GSP event" in
+/// `Documentation/gpu/nova/core/interrupts.rst`.
+#[pin_data]
+pub(crate) struct GspIrq<'a> {
+    _leaf_guard: LeafEnableGuard<'a>,
+    #[pin]
+    reg: irq::ThreadedRegistration<'a, GspInterrupt<'a>>,
+    _top_guard: TopEnableGuard<'a>,
+}
+
+impl<'a> GspIrq<'a> {
+    /// Returns an initializer that registers the threaded handler and then enables the GSP
+    /// subtree at `TOP` and the GSP vector at its leaf.
+    ///
+    /// An event that latched while the vector was disabled is delivered as soon as the vector is
+    /// enabled.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `vectors` does not service the GSP subtree, or if `chipset` does not implement
+    /// every subtree that `vectors` services. Otherwise the error from `request_threaded_irq`.
+    ///
+    /// # Safety
+    ///
+    /// Callers must not `mem::forget()` the initialized `GspIrq` or otherwise prevent its [`Drop`]
+    /// implementation, which runs `free_irq`, from running.
+    pub(crate) unsafe fn new(
+        pdev: &'a pci::Device<device::Bound>,
+        vectors: &'a SubtreeVectors<'a>,
+        bar: Bar0<'a>,
+        cmdq: &'a Cmdq<'a>,
+        chipset: Chipset,
+    ) -> impl PinInit<Self, Error> + 'a {
+        let dev = pdev.as_ref();
+
+        try_pin_init!(Self {
+            // SAFETY: this function's caller must not leak the `GspIrq` that owns this
+            // registration, so the registration's `Drop` runs.
+            reg <- unsafe {
+                irq::ThreadedRegistration::new(
+                    vectors.request_for(GSP_SUBTREE)?,
+                    irq::Flags::TRIGGER_NONE,
+                    c"nova-core",
+                    Ok(GspInterrupt::new(
+                        bar,
+                        cmdq,
+                        vectors.tree(bar, chipset)?,
+                        chipset,
+                        dev,
+                    )),
+                )
+            },
+            _top_guard: reg.handler().tree.enable_top_guarded(),
+            _leaf_guard: reg.handler().tree.enable_leaf_guarded(
+                GSP_INTR_0_VECTOR.leaf_index(),
+                GSP_INTR_0_VECTOR.leaf_mask(),
+            ),
+        })
+    }
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index f77920a3b30a..eae1a1d4b933 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -117,10 +117,12 @@ pub(super) const fn all() -> Self {
         Self(u32::MAX)
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn from_raw(raw: u32) -> Self {
         Self(raw)
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn into_raw(self) -> u32 {
         self.0
     }
@@ -196,7 +198,6 @@ pub(super) const fn span(self) -> u32 {
     }
 
     /// Returns the subtrees of this set, lowest index first.
-    #[expect(dead_code)]
     pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
         (0..u32::BITS)
             .map(Subtree::new)
@@ -234,6 +235,7 @@ pub(super) const fn new<const VECTOR: u32>() -> Self {
         Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn into_raw(self) -> u32 {
         self.0.get()
     }
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index abafe4f2968d..cbaef6d3d9f1 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -17,7 +17,6 @@
 mod fsp;
 mod gpu;
 mod gsp;
-#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
 mod irq;
 mod mctp;
 mod mm;
-- 
2.55.0

[PATCH v4 16/17] gpu: nova-core: add KUnit tests for the interrupt tree and HALs

[John Hubbard]

The vector arithmetic and the per-architecture interrupt properties
touch no hardware, so KUnit can cover both without a GPU. A wrong leaf
count or rearm method for one family would otherwise show up only on
that family's hardware.

Add three suites:

* nova_core_gin_tree covers the vector types. It checks that a leaf
  index stops at the widest supported tree, that a leaf count implies
  the right number of subtrees and vectors and enumerates every leaf in
  order, that a vector maps to the right leaf, bit and subtree, and
  that a vector beyond an 8-leaf tree is rejected there and accepted in
  a 16-leaf tree. It also exercises the subtree set operations and
  checks that every supported chipset implements the subtree that
  carries the GSP event.

* nova_core_gin_hal covers the CPU interrupt HAL: the 8-leaf tree on
  Turing through Ada and the 16-leaf tree on Hopper and later, the
  configuration-space rearm for pre-Hopper MSI, the TOP-enable rearm
  for Hopper-plus MSI, and the single-subtree rearm for MSI-X on every
  family.

* nova_core_falcon_hal covers the falcon interrupt HAL: which chipsets
  have the retrigger register, and which routing offsets each uses.
  GA100 appears on the Turing side of one split and the Ampere side of
  the other, because it has the retrigger register but keeps the Turing
  routing offsets.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/falcon/hal.rs         |  48 +++++++
 drivers/gpu/nova-core/irq/hal.rs            |  64 ++++++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 135 +++++++++++++++++++-
 3 files changed, 246 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 3f1f509eccbd..70bddcaf4266 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -171,3 +171,51 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
 
     Ok(hal)
 }
+
+#[kunit_tests(nova_core_falcon_hal)]
+mod tests {
+    use super::*;
+
+    /// Turing falcons have no retrigger register. GA100 and every later chipset have it.
+    #[test]
+    fn intr_retrigger_gate_per_arch() {
+        for chipset in [Chipset::TU102, Chipset::TU116] {
+            assert!(!falcon_intr_hal(chipset).has_intr_retrigger());
+        }
+
+        for chipset in [
+            Chipset::GA100,
+            Chipset::GA102,
+            Chipset::AD102,
+            Chipset::GH100,
+            Chipset::GB100,
+            Chipset::GB202,
+        ] {
+            assert!(falcon_intr_hal(chipset).has_intr_retrigger());
+        }
+    }
+
+    /// The RISC-V routing offsets change at GA102, so GA100 still uses the Turing ones.
+    #[test]
+    fn riscv_routing_offsets_split_at_ga102() {
+        for chipset in [Chipset::TU102, Chipset::TU116, Chipset::GA100] {
+            assert_eq!(
+                falcon_intr_hal(chipset).riscv_routing(),
+                RiscvRouting::Tu102
+            );
+        }
+
+        for chipset in [
+            Chipset::GA102,
+            Chipset::AD102,
+            Chipset::GH100,
+            Chipset::GB100,
+            Chipset::GB202,
+        ] {
+            assert_eq!(
+                falcon_intr_hal(chipset).riscv_routing(),
+                RiscvRouting::Ga102
+            );
+        }
+    }
+}
diff --git a/drivers/gpu/nova-core/irq/hal.rs b/drivers/gpu/nova-core/irq/hal.rs
index ede9a10ccda6..03852918013d 100644
--- a/drivers/gpu/nova-core/irq/hal.rs
+++ b/drivers/gpu/nova-core/irq/hal.rs
@@ -88,3 +88,67 @@ pub(super) fn cpu_interrupt_hal(chipset: Chipset) -> &'static dyn CpuInterruptHa
         }
     }
 }
+
+#[kunit_tests(nova_core_gin_hal)]
+mod tests {
+    use super::*;
+
+    use crate::gpu::Chipset;
+
+    /// Turing through Ada implement an 8-leaf tree.
+    #[test]
+    fn pre_hopper_tree_size() {
+        for chipset in [Chipset::TU102, Chipset::GA102, Chipset::AD102] {
+            assert_eq!(cpu_interrupt_hal(chipset).leaf_count(), LeafCount::Eight);
+        }
+    }
+
+    /// Hopper and later implement a 16-leaf tree.
+    #[test]
+    fn hopper_plus_tree_size() {
+        for chipset in [Chipset::GH100, Chipset::GB100, Chipset::GB202] {
+            assert_eq!(cpu_interrupt_hal(chipset).leaf_count(), LeafCount::Sixteen);
+        }
+    }
+
+    /// MSI rearms through the configuration-space mirror only before Hopper. Hopper and later
+    /// cycle the `TOP` enables of every serviced subtree.
+    #[test]
+    fn msi_rearm_method_per_arch() {
+        for chipset in [Chipset::TU102, Chipset::GA102, Chipset::AD102] {
+            let hal = cpu_interrupt_hal(chipset);
+            assert_eq!(
+                hal.pci_irq_rearm_method(MsiType::Msi),
+                PciIrqRearmMethod::ConfigMirrorEoi
+            );
+        }
+
+        for chipset in [Chipset::GH100, Chipset::GB100, Chipset::GB202] {
+            let hal = cpu_interrupt_hal(chipset);
+            assert_eq!(
+                hal.pci_irq_rearm_method(MsiType::Msi),
+                PciIrqRearmMethod::TopEnableCycleServiced
+            );
+        }
+    }
+
+    /// MSI-X rearms one subtree on every architecture, since each subtree has its own table
+    /// entry.
+    #[test]
+    fn msix_rearms_one_subtree_on_every_arch() {
+        for chipset in [
+            Chipset::TU102,
+            Chipset::GA102,
+            Chipset::AD102,
+            Chipset::GH100,
+            Chipset::GB100,
+            Chipset::GB202,
+        ] {
+            let hal = cpu_interrupt_hal(chipset);
+            assert_eq!(
+                hal.pci_irq_rearm_method(MsiType::MsiX),
+                PciIrqRearmMethod::TopEnableCycleSubtree
+            );
+        }
+    }
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index eae1a1d4b933..66b7d2b16454 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -122,7 +122,10 @@ pub(super) const fn from_raw(raw: u32) -> Self {
         Self(raw)
     }
 
-    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
+    #[cfg_attr(
+        not(any(CONFIG_NOVA_CORE_SELFTESTS, CONFIG_KUNIT = "y")),
+        expect(dead_code)
+    )]
     pub(super) const fn into_raw(self) -> u32 {
         self.0
     }
@@ -490,3 +493,133 @@ fn drop(&mut self) {
         clear_top_enables(self.bar, self.serviced);
     }
 }
+
+#[kunit_tests(nova_core_gin_tree)]
+mod tests {
+    use super::*;
+
+    /// A leaf index cannot name a leaf beyond the widest supported tree.
+    #[test]
+    fn leaf_index_bounds() {
+        assert!(LeafIndex::try_new(0).is_some());
+        assert!(LeafIndex::try_new(15).is_some());
+        assert!(LeafIndex::try_new(16).is_none());
+    }
+
+    /// The subtree count, the implemented-subtree set, and the vector count follow the leaf count.
+    #[test]
+    fn leaf_count_derives_subtrees_and_vectors() {
+        assert_eq!(LeafCount::Eight.subtree_count(), 4);
+        assert_eq!(
+            Bounded::<u32, 32>::from(LeafCount::Eight.subtree_set()).get(),
+            0x0f
+        );
+        assert_eq!(LeafCount::Eight.vector_count(), 256);
+
+        assert_eq!(LeafCount::Sixteen.subtree_count(), 8);
+        assert_eq!(
+            Bounded::<u32, 32>::from(LeafCount::Sixteen.subtree_set()).get(),
+            0xff
+        );
+        assert_eq!(LeafCount::Sixteen.vector_count(), 512);
+    }
+
+    /// A tree enumerates every leaf that it implements, in order, and no more.
+    #[test]
+    fn leaf_count_iter_covers_the_tree() {
+        for (count, expected) in [(LeafCount::Eight, 8usize), (LeafCount::Sixteen, 16)] {
+            let mut seen = 0;
+
+            for (index, leaf) in count.iter().enumerate() {
+                assert_eq!(leaf.get(), index);
+                seen += 1;
+            }
+
+            assert_eq!(seen, expected);
+        }
+    }
+
+    /// A vector maps to its leaf, its bit within that leaf, and its subtree. The doorbell (129)
+    /// and the GSP event (155) share a subtree.
+    #[test]
+    fn vector_maps_to_leaf_bit_and_subtree() {
+        let doorbell = GinVector::new::<129>();
+        let gsp = GinVector::new::<155>();
+
+        assert_eq!(doorbell.leaf_index().get(), 4);
+        assert_eq!(doorbell.leaf_mask().into_raw(), 1 << 1);
+        assert_eq!(doorbell.subtree().index(), 2);
+
+        assert_eq!(gsp.leaf_index().get(), 4);
+        assert_eq!(gsp.leaf_mask().into_raw(), 1 << 27);
+        assert_eq!(gsp.subtree().index(), 2);
+
+        assert_eq!(doorbell.subtree(), gsp.subtree());
+    }
+
+    /// Both fixed vectors are within the 8-leaf tree, so every supported part implements them.
+    #[test]
+    fn fixed_vectors_fit_the_narrowest_tree() {
+        assert!(GinVector::new::<129>().validate(LeafCount::Eight).is_ok());
+        assert!(GinVector::new::<155>().validate(LeafCount::Eight).is_ok());
+
+        // The first vector beyond an 8-leaf tree.
+        assert!(GinVector::new::<256>().validate(LeafCount::Eight).is_err());
+        assert!(GinVector::new::<256>().validate(LeafCount::Sixteen).is_ok());
+    }
+
+    /// A subtree set reports membership, intersection, and its span from subtree 0.
+    #[test]
+    fn subtree_set_operations() {
+        let gsp = GinVector::new::<155>().subtree();
+
+        assert!(LeafCount::Eight.subtree_set().contains(gsp));
+        assert!(!LeafCount::Eight.subtree_set().is_empty());
+
+        // The GSP needs no subtree above 2, so an MSI-X request covers entries 0 through 2.
+        assert_eq!(SubtreeSet::from(gsp).span(), 3);
+
+        // A 16-leaf tree implements every subtree that an 8-leaf tree does.
+        assert_eq!(
+            LeafCount::Sixteen
+                .subtree_set()
+                .intersection(LeafCount::Eight.subtree_set()),
+            LeafCount::Eight.subtree_set()
+        );
+    }
+
+    /// Iterating a subtree set yields each subtree once, lowest index first, and nothing for an
+    /// empty set.
+    #[test]
+    fn subtree_set_iterates_its_members() {
+        assert!(LeafCount::Eight
+            .subtree_set()
+            .iter()
+            .map(Subtree::index)
+            .eq([0u32, 1, 2, 3]));
+
+        let gsp = SubtreeSet::from(GinVector::new::<155>().subtree());
+        assert!(gsp.iter().map(Subtree::index).eq([2u32]));
+
+        let empty = SubtreeSet::from(Bounded::<u32, 32>::new::<0>());
+        assert_eq!(empty.iter().count(), 0);
+    }
+
+    /// Every supported chipset implements the subtree that carries the GSP event.
+    #[test]
+    fn gsp_subtree_is_implemented_everywhere() {
+        for chipset in [
+            Chipset::TU102,
+            Chipset::GA102,
+            Chipset::AD102,
+            Chipset::GH100,
+            Chipset::GB100,
+            Chipset::GB202,
+        ] {
+            assert!(cpu_interrupt_hal(chipset)
+                .leaf_count()
+                .subtree_set()
+                .contains(crate::irq::gsp::GSP_SUBTREE));
+        }
+    }
+}
-- 
2.55.0

[PATCH v4 17/17] gpu: nova-core: document the GIN interrupt controller and GSP events

[John Hubbard]

The interrupt code rests on hardware behavior that the code cannot show
on its own: how GIN, the GPU's interrupt controller, records and
delivers interrupts, what edge-triggered delivery requires of a handler,
and how the GSP signals the CPU. Some of those requirements come from
Open RM rather than from the hardware manuals.

Add a design document that records that behavior, the rules nova-core
follows because of it, and the terms the code uses for it. The code
comments cite the document by section rather than repeating it.

Assisted-by: LLM
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 Documentation/gpu/nova/core/interrupts.rst | 674 +++++++++++++++++++++
 Documentation/gpu/nova/index.rst           |   1 +
 2 files changed, 675 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/interrupts.rst

diff --git a/Documentation/gpu/nova/core/interrupts.rst b/Documentation/gpu/nova/core/interrupts.rst
new file mode 100644
index 000000000000..280dcf97688a
--- /dev/null
+++ b/Documentation/gpu/nova/core/interrupts.rst
@@ -0,0 +1,674 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+=============================================
+GPU interrupt handling: GIN and the GSP event
+=============================================
+
+This document describes how nova-core receives interrupts from the GPU on Turing
+and later parts. It covers the GPU Interrupt and Notification unit (GIN), which
+is the GPU's interrupt controller, and the GSP event, the interrupt that
+nova-core services in normal operation.
+
+Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU
+also has on-chip processors that run their own firmware and receive their own
+interrupts. The GSP (GPU System Processor) is one of them.
+
+Register names are the names from the GPU hardware reference headers. The
+pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers
+call it ``NV_GIN``. This document calls it GIN throughout, because the tree that
+nova-core services is the same on every supported part. "Register naming" at
+the end says how the names map onto the headers. Open RM, NVIDIA's open-source
+GPU kernel driver, is cited wherever nova-core follows it.
+
+Terminology
+===========
+
+The three levels of the controller, innermost first:
+
+leaf
+    One ``LEAF`` register. Each of its 32 bits is the pending bit of one
+    interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or
+    Blackwell tree has 16.
+
+subtree
+    Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the
+    unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery:
+    every interrupt from one subtree arrives on one MSI-X entry.
+
+tree
+    One ``TOP`` register and the leaves under it. Every PCIe function has its
+    own tree, and nova-core services the CPU tree of one function.
+
+The hardware headers, Open RM, and the Linux PCI API all use the word "vector",
+each for a different number. This document gives each one its own name, and a
+bare "vector" always means a GIN vector.
+
+GIN vector
+    The GPU-internal interrupt source number. It addresses one bit of one leaf.
+    A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0
+    through 255. The CPU doorbell is vector 129 and the GSP event is vector
+    155.
+
+MSI-X entry
+    An index into the device's MSI-X table. One entry serves one subtree.
+
+PCI vector
+    One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X
+    entry, or the single MSI message.
+
+Linux IRQ number
+    What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI
+    vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as
+    well.
+
+The remaining terms, each named for the register or the specification that
+defines it:
+
+enable, disable a vector
+    Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``.
+
+enable, disable a subtree
+    Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``.
+
+serviced subtree
+    A subtree that nova-core enables and has a handler for.
+
+rearm
+    Restoring PCI interrupt delivery after servicing an interrupt. See
+    "Rearming PCI interrupt delivery".
+
+mask
+    Reserved for the two places where hardware and the PCI specification use
+    the word: the MSI-X per-entry Vector Control mask bit, which Linux
+    controls, and the falcon interrupt masks. It never names a GIN enable.
+
+latched, pending
+    Two names for one state, a set ``LEAF`` bit. The vector's source sets the
+    bit whether or not the vector is enabled.
+
+clear a vector
+    Write a 1 to the vector's bit in ``LEAF``. Open RM calls the same operation
+    ``intrClearLeafVector_HAL``.
+
+pending bits
+    The plain 32-bit value read from a ``LEAF`` register.
+
+notification
+    An interrupt whose only content is that something happened, such as a
+    posted message. Servicing a notification means reading what it announces.
+    The unit that raised it needs no attention. The GSP event is one.
+
+unit
+    Any block that raises an interrupt. "Engine" is reserved for the blocks
+    that do user work: GR, CE, NVDEC, and the like.
+
+falcon
+    One of the GPU's microcontrollers (see
+    Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core
+    inside its falcon. A falcon latches each of its interrupt causes and routes
+    it either to the host, meaning the CPU, or to its own core.
+
+The GIN controller
+==================
+
+A GPU has many interrupt sources: the GSP, the copy engines, the graphics
+engine, video decode and encode, the MMU fault path, timers, and others. GIN
+records which of them are pending and raises the PCI interrupt to the CPU.
+
+Trees
+-----
+
+GIN keeps one tree for each destination it can deliver an interrupt to. The CPU
+has one tree per PCIe function, so the physical function and each virtual
+function have their own. The GSP has a tree, and so do the other on-chip
+processors that receive interrupts. Every tree has the same two-level layout,
+and a function reaches its own tree through the per-function register aperture.
+
+nova-core services the CPU tree of one function. A virtual function's tree
+belongs to that function's driver, and a processor's tree belongs to the
+firmware running on that processor.
+
+The two-level tree
+------------------
+
+A tree is a set of ``LEAF`` registers and one ``TOP`` register.
+
+* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors
+  ``32i`` through ``32i + 31``. A set bit is a pending vector.
+* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``,
+  which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled
+  vector is pending in either leaf.
+
+A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1.
+The other TOP bits read 0. The leaves and subtrees that a part has are its
+implemented leaves and subtrees, and "Per-architecture differences" gives the
+counts. For an 8-leaf tree::
+
+    TOP bit 0  ->  subtree 0  ->  LEAF(0), LEAF(1)   vectors   0..63
+    TOP bit 1  ->  subtree 1  ->  LEAF(2), LEAF(3)   vectors  64..127
+    TOP bit 2  ->  subtree 2  ->  LEAF(4), LEAF(5)   vectors 128..191
+    TOP bit 3  ->  subtree 3  ->  LEAF(6), LEAF(7)   vectors 192..255
+
+    LEAF(4), one bit per vector, holds vectors 128..159:
+
+      bit 1  = vector 129  (CPU doorbell)
+      bit 27 = vector 155  (GSP event)
+
+A vector's number fixes its place in the tree::
+
+    leaf    = vector / 32
+    bit     = vector % 32
+    subtree = leaf / 2
+
+Registers
+---------
+
+nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The
+leaf registers are arrays indexed by leaf number.
+
+* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit
+  clears that vector, and a 0 leaves the bit as it was.
+* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors
+  of leaf ``i``, one bit per vector.
+* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per
+  subtree.
+* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as
+  the vector's own source would. It is write-only. The self-test uses it.
+
+nova-core does not read ``TOP``. "Servicing the tree" says why.
+
+Every set and clear register acts per bit: a 1 performs the action for that
+bit, and a 0 leaves the bit alone. No register needs a read-modify-write.
+
+GIN delivers a vector to the CPU only when its leaf enable bit and its
+subtree's TOP enable bit are both set. The enables do not affect the latch. The
+source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show
+that bit, so reading the leaf is the only way to see it.
+
+How a unit interrupt reaches the CPU
+------------------------------------
+
+A unit does not write a ``LEAF`` register. Each unit has an interrupt control
+register that GSP firmware programs. The control register holds the unit's
+vector, the GFID that identifies the PCIe function whose tree receives the
+interrupt, and one enable bit per destination: the CPU, the GSP, and the other
+on-chip processors. When the unit has an event::
+
+    1. The unit sends GIN an interrupt message carrying the vector, the GFID,
+       and the destination enables from its control register.
+    2. In the tree of each destination that the message selects, GIN sets bit
+       (vector % 32) of LEAF(vector / 32).
+    3. If the vector and its subtree are enabled in the CPU tree, GIN raises
+       the PCI interrupt.
+
+Because firmware assigns the vectors, nova-core does not hardcode which vector
+belongs to which unit, with two exceptions. The hardware headers of every
+supported part define the GSP event as vector 155, and GSP firmware's own
+interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper
+hardware fixes that number, and GSP firmware keeps it on Hopper and later.
+nova-core names both by number. A driver can fetch the full unit-to-vector
+table from the GSP by RPC, and nouveau does. nova-core does not, because a
+fixed vector needs no lookup.
+
+Edge-triggered delivery
+-----------------------
+
+A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set
+until the CPU clears it. A source that stays high does not set the bit again.
+
+GIN raises the PCI interrupt for a subtree when the subtree's enabled pending
+state goes from low to high::
+
+    Per vector, in leaf i at bit b:
+        LEAF(i)[b] AND LEAF_EN(i)[b]
+
+    Per subtree N, across leaves 2N and 2N + 1:
+        OR of every enabled pending bit  ->  TOP[N]
+
+    Delivery for subtree N:
+        TOP[N] AND TOP_EN[N]  ->  rising edge  ->  PCI interrupt
+
+``TOP_EN`` applies after the summary, so disabling a subtree stops delivery
+without changing what ``TOP`` reports.
+
+Three consequences:
+
+* Code that must find every pending vector reads the leaves. A vector that
+  latched while disabled is not in ``TOP``.
+* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a
+  new edge. GIN delivers an interrupt for a pending bit left uncleared as soon
+  as its subtree is enabled again.
+* A source that holds its signal high produces no new edge after the CPU
+  clears the leaf bit. Such a source has to re-emit its interrupt. The falcons
+  do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon").
+
+Delivery over PCI
+=================
+
+GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux
+grants. nova-core requests MSI-X first and falls back to MSI, and never uses
+INTx.
+
+MSI has a single message, and every subtree raises that one message, so one
+Linux IRQ serves the whole tree.
+
+MSI-X gives each subtree its own table entry, at the index equal to the subtree
+number. Linux masks every entry until a driver requests its Linux IRQ number,
+and a masked entry sends no message: the GPU records the interrupt in the MSI-X
+pending bit array, where it stays until Linux unmasks the entry. An entry that
+the driver never requests is never unmasked. A driver that enables a subtree
+without requesting that subtree's entry loses every interrupt from that
+subtree, with nothing reported: the leaf and TOP registers show the vector
+pending and enabled while no handler runs.
+
+The serviced-subtree invariant
+------------------------------
+
+Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered
+handler.
+
+MSI satisfies this with its single message. MSI-X needs one allocated entry per
+serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests
+entries 0 through the highest serviced subtree::
+
+    MSI-X, with subtree 2 serviced:
+
+      subtree 0  ->  entry 0   allocated, no handler, stays masked
+      subtree 1  ->  entry 1   allocated, no handler, stays masked
+      subtree 2  ->  entry 2   handler here, and its rearm covers subtree 2
+
+    MSI, with any serviced set:
+
+      every serviced subtree  ->  the one allocated PCI vector, whose
+                                  handler's rearm covers the whole serviced set
+
+An allocated entry whose subtree nova-core does not service costs nothing. The
+entry stays masked, and a disabled subtree raises no interrupt.
+
+nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which
+is in subtree 2. Open RM's headers place its UVM_SHARED interrupt category in
+subtree 2 on every part nova-core supports. The self-test doorbell, vector 129,
+is in the same leaf, and the test allocates its own vectors for it (see
+"Self-test").
+
+Rearming PCI interrupt delivery
+-------------------------------
+
+A message-signaled interrupt is delivered once per edge, and the PCI side
+delivers no further interrupt until the CPU rearms it. The rearm operation
+depends on the GPU family and on the interrupt type Linux granted:
+
+==================  =====  ===========================================
+Architecture        Type   Rearm operation
+==================  =====  ===========================================
+Turing through Ada  MSI    write the configuration-mirror EOI register
+Hopper and later    MSI    clear then set the serviced TOP enables
+Any                 MSI-X  clear then set the handler's own TOP enable
+==================  =====  ===========================================
+
+The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI
+configuration space, and the value written does not matter. The ``TOP_EN``
+cycle produces a new delivery edge. The MSI forms cover every serviced subtree,
+because one message serves all of them. The MSI-X form covers one subtree,
+because each serviced subtree has its own entry and its own handler.
+
+A handler rearms once per delivered interrupt, on every path, including the
+path where it finds its vector not pending. A handler that skips the rearm
+receives no further interrupts.
+
+Open RM makes the same split. It writes the configuration-space EOI for MSI on
+pre-Hopper parts, and cycles the TOP enables of the subtrees it services for
+Hopper-plus MSI and for MSI-X.
+
+Servicing the tree
+==================
+
+Servicing a leaf has a required order: read its pending bits, then clear them.
+Clearing a leaf before reading it discards every vector latched in it, and
+nothing reports the loss. In nova-core, reading a leaf produces the handle that
+clears it, so the wrong order does not compile. The handle clears exactly the
+bits it read, so a vector that latched after the read stays pending.
+
+A handler clears its bit before it services the vector. Clearing afterwards
+would discard an interrupt that the source raised while the handler ran.
+
+nova-core services the tree in two ways.
+
+The notification path services one vector. It reads the vector's leaf, clears
+only the vector's bit, and rearms. The subtree stays enabled, and a vector
+pending beside it in the same leaf keeps its bit set for the code that services
+that vector. The GSP event handler takes this path, and so does the self-test
+handler.
+
+The startup drain walks the whole tree, because it must clear whatever is
+pending across every subtree rather than one known vector. It disables the
+serviced subtrees at ``TOP``, reads and clears every implemented leaf, and
+leaves the subtrees disabled for its caller to enable once the caller is ready
+for deliveries. The drain reads every leaf rather than descending from ``TOP``,
+because sources latch vectors during boot while those vectors are disabled, and
+``TOP`` does not show them. Open RM's stall-interrupt path reads every leaf for
+the same reason.
+
+The two paths as register operations::
+
+    Startup drain, run once during probe:
+        write TOP_EN_CLEAR = serviced        stop new deliveries
+        for each implemented leaf i:
+            pending = read LEAF(i)
+            write LEAF(i) = pending          clear what was read
+        (returns with TOP_EN still clear)
+
+    Notification, the subtree stays enabled:
+        pending = read LEAF(leaf)            is the handler's bit set?
+        write LEAF(leaf) = bit               clear that one bit
+        rearm PCI interrupt delivery
+
+The drain clears every pending bit, including bits that nova-core never
+services. An uncleared bit holds its subtree in the pending state, and enabling
+that subtree again would deliver an interrupt for a vector that no handler
+services.
+
+The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through
+the configuration mirror, which the drain never writes. An interrupt delivered
+before probe had no handler to rearm it, so the startup sequence rearms
+explicitly after the drain.
+
+Nothing in nova-core serializes access to the tree. The GSP event handler
+touches only its own leaf, and the drain runs during probe, before that handler
+is registered.
+
+Per-architecture differences
+============================
+
+The tree is the same on every supported GPU except for its size, which changes
+at Hopper:
+
+===================  ======  ========  ====================
+GPUs                 Leaves  Subtrees  Implemented subtrees
+===================  ======  ========  ====================
+Turing, Ampere, Ada  8       4         ``0x0f``
+Hopper, Blackwell    16      8         ``0xff``
+===================  ======  ========  ====================
+
+The interrupt HAL provides the leaf count, and the subtree count and the
+implemented-subtree set derive from it. A subtree that the part does not
+implement has no TOP bit, so building a tree that services one fails with
+``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported part
+has them.
+
+Open RM's headers assign every interrupt category of a 16-leaf tree to leaves 0
+through 11. The drain reads all 16, because a vector can be latched in any
+implemented leaf.
+
+The HAL's other value is the rearm method (see "Rearming PCI interrupt
+delivery"). Two falcon properties also differ by family and have a HAL of their
+own. Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing
+registers moved at GA102 (see "Retriggering a falcon").
+
+The GSP event
+=============
+
+When the GSP has output for the CPU, it writes messages into the GSP-to-CPU
+queue in shared memory and raises SWGEN0, one of the software-generated
+interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector
+155, leaf 4 bit 27, in subtree 2.
+
+The queue carries notifications (log records, error records, lifecycle events)
+and command replies. A thread waiting for a reply reads the queue itself, so
+the interrupt is only the trigger to drain the queue (see "Draining the
+GSP-to-CPU queue").
+
+The falcon latches every cause it raises, SWGEN0 among them, in its
+``IRQSTAT`` register. The handler services the host-routed causes and clears
+their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon
+re-emits any cause that latched in the meantime. "Retriggering a falcon" has
+the details.
+
+Draining the queue takes the command-queue mutex and walks shared memory, so it
+cannot run in hard interrupt context. nova-core registers a threaded handler,
+under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard
+interrupt context and reads and writes only registers, and it wakes the IRQ
+thread to drain the queue::
+
+    GSP writes messages into the GSP-to-CPU queue
+    GSP raises SWGEN0
+    GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending
+    PCI interrupt -> Linux IRQ -> top half, in hard interrupt context:
+        read LEAF(4), and if bit 27 is clear, rearm and return
+        clear bit 27 (the subtree stays enabled)
+        read the falcon causes routed to the host, clearing SWGEN0 if set
+        for every other host cause: log it, clear its latch, and read the
+            host causes back
+        if the clear ended all of them: retrigger the falcon
+        otherwise: disable vector 155 at its leaf and skip the retrigger
+        rearm PCI interrupt delivery
+        wake the IRQ thread if SWGEN0 was set
+    IRQ thread, which may sleep:
+        take the command-queue mutex and drain the GSP-to-CPU queue
+
+A halt and a posted message can be pending together, so the top half services
+every cause that the status reports.
+
+A drain fails when a message's framing or checksum is bad, which poisons the
+queue (see "Draining the GSP-to-CPU queue"). Every later event would fail the
+same way, so the IRQ thread disables vector 155 and logs the failure, which
+leaves the queue unserviced until the device is reset.
+
+The handler, the self-test, and the rest of the driver read BAR0 through one
+shared mapping. nova-core unregisters an interrupt handler when the device
+unbinds, so a handler runs only while the mapping exists.
+
+Retriggering a falcon
+---------------------
+
+A falcon signals the tree when its set of host-routed causes goes from empty to
+non-empty. A cause left latched keeps the set non-empty, so no later cause
+signals the tree, and the vector is lost. For a cause that stays latched, the
+handler can clear the tree leaf first or the falcon latch first, and the loss
+is the same.
+
+``IRQSTAT`` latches every cause in the falcon, including the causes routed to
+the falcon's own RISC-V core and owned by the firmware running on it. A host
+handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and
+``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both
+before it reads or clears a cause. Open RM computes the same intersection in
+``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing
+registers and GA102 moves them, so the offsets change at GA102 rather than at
+the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is
+read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on
+a RISC-V falcon.
+
+``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the
+tree, which supplies the transition that clearing the leaf lost. The handler
+writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it
+read, because a re-emitted cause that nothing clears arrives again at once and
+on every pass after that.
+
+``IRQSCLR`` ends a latch and does not end the source behind it, so a cause
+driven from outside the falcon stays set after the write. On Blackwell the
+fault-containment and ECC causes are driven that way: they appear in
+``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and
+``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler
+clears the latch of every host cause other than SWGEN0, reads the host causes
+back, and retriggers only when the read-back is empty. When a cause is still
+set, the handler disables vector 155 instead and reports that the device needs
+a reset. Disabling loses no notification: the cause that is still set holds the
+host-routed set non-empty, so the falcon would signal nothing further either
+way. Open RM makes the same choice, and ``kgspService_TU102`` skips
+``kflcnIntrRetrigger`` once it has recorded a fatal error.
+
+A fault cause that arrives after the clear cannot be told apart from one that
+the clear failed to end, so the handler disables the vector in that case too.
+Both mean the GSP has faulted.
+
+Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create
+a transition it has lost. It must leave no host cause latched: it reads the
+host-routed status once and takes every cause that the status reports, rather
+than stopping at the first one it recognizes. One window stays open. A cause
+that arrives after the handler has read the status is not in the value the
+handler clears, so it stays latched after the leaf has been cleared, and no
+later cause from that falcon signals the tree. Open RM has the same window on
+Turing, where ``kflcnIntrRetrigger`` does nothing.
+
+Enabling the GSP event
+----------------------
+
+SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays
+set. nova-core's GSP boot code consumes the GSP's notifications by polling the
+queue, which leaves the latch set and leaves pending bits in the tree. The
+handoff from polling to interrupts has a required order::
+
+    disable every implemented vector    drop enables left by boot, or by a
+                                        driver that ran before this one
+    drain the tree                      clear stale pending bits
+    rearm PCI interrupt delivery        required under pre-Hopper MSI, where
+                                        nothing else does it
+    clear the SWGEN0 latch              so the next message makes an edge
+    register the threaded handler       nothing can reach it yet
+    enable subtree 2 at TOP             the drain left it disabled
+    enable vector 155 at LEAF(4)        deliveries become possible here
+    drain the GSP-to-CPU queue          messages posted before the clear
+
+nova-core quiesces the tree before it registers the handler. Registering
+unmasks the PCI interrupt, and GIN would then deliver a vector that boot left
+enabled to a handler that services one vector and has no way to service any
+other. Open RM clears every leaf enable at the same point for the same reason.
+
+nova-core clears the latch after the drain. If nova-core cleared the latch
+first, a message posted before the drain could set it again, along with bit 27
+of LEAF(4). The drain would then clear the leaf bit while the latch stays set,
+and no later message would signal the tree. Clearing after the drain can
+instead leave the leaf bit pending with the latch already clear. Enabling the
+vector then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half
+clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue
+drain that follows reads the message.
+
+Clearing the latch makes the first interrupt possible. A message that the GSP
+posted before that clear produces no interrupt, so the sequence ends by
+draining the queue.
+
+The subtree is enabled at ``TOP`` once the handler is registered. The drain
+left it disabled, and under pre-Hopper MSI the rearm is a configuration-space
+write that does not enable it again, so the enable is explicit. On teardown
+nova-core disables the vector at its leaf, so that nothing in the subtree can
+be delivered, then calls ``free_irq()``, and disables the subtree last.
+Disabling the subtree earlier would let a handler still in flight enable it
+again through the ``TOP_EN`` cycle of its rearm, which would leave the subtree
+enabled with no handler registered. nova-core tears down the registration
+before it frees the queue that the handler drains and before it unloads the
+GSP.
+
+Draining the GSP-to-CPU queue
+-----------------------------
+
+The queue carries command replies and unsolicited events, and a message's
+function code says which it is.
+
+* A function code that matches the awaited reply: the message is decoded and
+  returned to the caller that sent the command.
+* Anything else is an event. An OS error record and a robust-channel record
+  are logged at error level, and an unrecognized function code at warning
+  level. The other known events (GSP logs, libos prints, assertion records,
+  lifecycle notices) need no action and get no line of their own, because the
+  receive trace at debug level already records every message's arrival with
+  its sequence number, function code, and length.
+
+The sequence number takes no part in the match, because the GSP does not echo
+the command's sequence number on every reply. On r570 the reply to
+``UnloadingGuestDriver`` carries sequence 0.
+
+The read pointer advances past every message, whether it matched, was an event,
+or matched but failed to decode, so a message is never left at the queue head
+for the next receive to parse again.
+
+Corrupt framing is the exception. A message's length is inside the region that
+the checksum covers, so once the framing or the checksum fails there is no
+trustworthy length with which to skip the message. Such a failure poisons the
+queue: nova-core logs it once, and every later receive fails with ``EIO`` until
+the device is reset.
+
+The polling path and the IRQ thread both read the queue under the command-queue
+mutex. Replies and events share one queue and one read pointer, so one lock is
+held across the whole drain. A thread waiting for a reply logs each event that
+arrives before the reply and keeps waiting. One deadline of 5 seconds applies
+to the whole wait, rather than a fresh timeout after each message, and the
+thread holds the mutex for the whole wait, so no other caller consumes the
+message it waits for.
+
+With one lock, a drain waits for an in-flight command's receive to finish or
+time out. For log and error records that delay does not matter.
+
+Self-test
+=========
+
+The self-test confirms that an interrupt injected at the GPU is delivered to a
+registered handler. It runs during probe, after the GPU's boot firmware, GFW,
+has completed, and before nova-core boots the GSP, because the test disables
+and drains the whole tree, including the GSP's leaf. The test is built only
+under ``CONFIG_NOVA_CORE_SELFTESTS``, like the other probe-time hardware tests.
+Those other tests log a failure and let probe continue. A failed delivery test
+fails probe, because an interrupt path that does not work leaves the driver
+unable to make progress later, in a place that says nothing about the cause.
+
+The test writes ``LEAF_TRIGGER`` with vector 129, the CPU doorbell, at leaf 4
+bit 1. The vector then takes the ordinary path to the CPU under the ordinary
+enables. The doorbell keeps the same number on every supported part, so the
+test names it without asking the GSP, which is not running yet.
+
+The test allocates PCI vectors for subtree 2, disables every vector, drains the
+tree, and checks that the doorbell bit starts out clear. It registers a
+non-threaded handler with a completion, under the name ``nova-core-selftest``
+in ``/proc/interrupts``, and enables the vector and the subtree. It triggers
+the doorbell, waits up to 1000 ms for the first delivery, triggers it again,
+and waits up to 1000 ms for the second. The handler takes the notification
+path: it clears only its own bit and rearms, and never walks the tree.
+
+The test triggers twice, and the second trigger waits for the first handler to
+finish. One delivery would prove nothing about the rearm, because the first
+message-signaled interrupt arrives whether the driver rearms or not, and two
+triggers in a row could coalesce into one delivery. A handler that walked the
+tree would prove nothing either: on every configuration except pre-Hopper MSI
+the rearm is a ``TOP_EN`` cycle, so a walk that enabled ``TOP`` again would
+rearm delivery whether the handler asked for it or not.
+
+The test passes only if both deliveries arrive, each finds the doorbell bit and
+nothing else pending in leaf 4, and the bit is clear once the vector is
+disabled. Anything else fails probe. Requiring the exact pending bits on the
+second delivery shows that the first handler's clear reached the hardware. No
+other vector in leaf 4 can be pending, because the test disabled every vector
+and runs before GSP boot.
+
+The test releases its PCI vectors and its handler before returning, so the
+driver's own allocation covers the GSP subtree and nothing else. Sharing the
+driver's allocation would have let the test pass only because the doorbell and
+the GSP event happen to be in the same subtree.
+
+The test exercises the path from the GPU to the handler without GSP firmware,
+which helps when bringing up PCI, MSI, MSI-X, and passthrough setups. Under
+MSI-X a pass also shows that the delivery arrived on the entry belonging to the
+serviced subtree.
+
+The parts with no hardware dependency have KUnit tests instead: the vector
+arithmetic, the leaf and subtree sets, the per-architecture leaf count and
+rearm method, and the falcon retrigger and routing HAL.
+
+Register naming
+===============
+
+nova-core uses the ``NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_*`` names for the CPU
+tree on every supported part. That is the per-function aperture: each PCIe
+function reaches its own tree through it, at the same offsets. The controller
+also has a central aperture that exposes every function's tree. The pre-Hopper
+headers name it ``NV_CTRL_CPU_INTR_*`` and the Hopper-plus headers
+``NV_GIN_CPU_INTR_*``. nova-core does not use it. Open RM's kernel-side code
+for the controller is the ``Intr`` object.
+
+Scope
+=====
+
+nova-core services the CPU tree of one function and nothing else. It implements
+no virtual-function tree management and no GFID routing, which belong to the
+physical function's driver or to firmware in a virtualized setup, and no MIG
+(multi-instance GPU) support. It services none of the subtrees that the
+hardware headers reserve for the stall interrupts of the host-driven engines.
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 59b206238498..224caef9ea42 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -35,3 +35,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/falcon
    core/tlv
    core/pramin
+   core/interrupts
-- 
2.55.0