[PATCH v5 06/15] gpu: nova-core: add the GIN interrupt tree and allocate its vectors

[John Hubbard <jhubbard@nvidia.com>]

From: Joel Fernandes <joelagnelf@nvidia.com>

Servicing a GIN leaf has a required order: read its pending bits, then
clear them. Clearing a leaf first discards every vector latched in it,
and the hardware keeps no record of what was discarded. The interrupt
never arrives, and no register shows that it was ever pending.

Every subtree enabled at TOP also needs an allocated PCI vector with a
handler registered on it. Under MSI-X each subtree has its own table
entry. Linux masks every entry until a driver requests its IRQ, and a
masked entry sends no message. An enabled subtree whose entry was never
requested raises interrupts that never reach a handler, while the leaf
and TOP registers show them pending and enabled. Under MSI the whole
tree raises a single message, so one entry serves every subtree.

Add the CPU interrupt tree of one PCIe function. The tree allocates the
PCI vectors for the subtrees that it services and hands out the request
for each, so that the chipset is given once and the vectors live as
long as the tree. Reading a leaf yields the handle that clears it, so
that the wrong order does not compile. Building a tree fails if it
names a subtree that the GPU does not implement. A reset disables every
vector, clears every pending bit, rearms delivery, and leaves the
serviced subtrees disabled at TOP, so that a handler registered
afterwards receives no interrupt left over from boot.

Request MSI-X entries 0 through the highest serviced subtree, since an
MSI-X allocation cannot be sparse, and fall back to a single MSI
message, never to INTx.

Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: reworked on top of the GIN vector types: a leaf read yields
 the handle that clears it, enables are guarded, the leaf count and the
 rearm come from the interrupt HAL, the drain reads every implemented
 leaf rather than descending from TOP, the tree owns its PCI vectors,
 and a reset returns the tree to a known state before a handler is
 registered]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |   2 +-
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 298 +++++++++++++++++++-
 2 files changed, 293 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 62c242b71dc8..b2cfe73af114 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -10,7 +10,7 @@
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 mod hal;
-mod interrupt_tree;
+pub(crate) mod interrupt_tree;
 mod regs;
 
 /// The message-signaled interrupt type that Linux granted.
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 279a43514b1f..6e97828bbd42 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -1,22 +1,47 @@
 // SPDX-License-Identifier: GPL-2.0
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
-//! Vector addressing in the GIN CPU interrupt tree.
+//! The GIN CPU interrupt tree for one PCIe function.
 //!
 //! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
 //! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
 //! keep the four from being confused with one another.
 //!
+//! Servicing a leaf requires reading its pending bits before clearing them. Only
+//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] clears a leaf,
+//! so that the wrong order does not compile. This module does not serialize access to the tree.
+//!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 use kernel::{
+    device::Bound,
+    io::{
+        register::Array,
+        Io, //
+    },
+    irq,
     num::Bounded,
+    pci::{
+        self,
+        IrqType, //
+    },
     prelude::*, //
 };
 
-use crate::num;
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    num, //
+};
 
-use super::regs::*;
+use super::{
+    hal::{
+        cpu_interrupt_hal,
+        CpuInterruptHal, //
+    },
+    regs::*,
+    MsiType, //
+};
 
 /// Number of vectors one leaf register carries, one per bit.
 const VECTORS_PER_LEAF: u32 = u32::BITS;
@@ -78,6 +103,11 @@ pub(super) const fn subtree_set(self) -> SubtreeSet {
     pub(super) const fn vector_count(self) -> u32 {
         self.into_u32() * VECTORS_PER_LEAF
     }
+
+    /// Returns every leaf a tree of this size implements.
+    pub(super) fn iter(self) -> impl Iterator<Item = LeafIndex> {
+        (0..self.into_raw()).filter_map(LeafIndex::try_new)
+    }
 }
 
 // `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
@@ -130,7 +160,7 @@ fn from(vectors: LeafMask) -> Self {
 ///
 /// Exactly one bit is set.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct Subtree(u32);
+pub(crate) struct Subtree(u32);
 
 impl Subtree {
     /// Returns the subtree at index `idx`.
@@ -151,7 +181,7 @@ pub(super) const fn into_raw(self) -> u32 {
 
 /// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct SubtreeSet(u32);
+pub(crate) struct SubtreeSet(u32);
 
 impl SubtreeSet {
     pub(super) const fn contains(self, subtree: Subtree) -> bool {
@@ -173,7 +203,6 @@ pub(super) const fn span(self) -> u32 {
     }
 
     /// Returns the subtrees of this set, lowest index first.
-    #[expect(dead_code)]
     pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
         (0..u32::BITS)
             .map(Subtree::new)
@@ -250,3 +279,260 @@ fn from(vector: GinVector) -> Self {
         vector.0.extend()
     }
 }
+
+/// Disables `vectors` in `leaf`.
+fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+    bar.write(
+        Array::at(*leaf),
+        NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
+    );
+}
+
+/// Disables the subtrees in `serviced` at `TOP`.
+fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
+    bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
+}
+
+/// The CPU tree of one PCIe function, the subtrees that nova-core services, and the PCI vectors
+/// that deliver them.
+///
+/// A subtree may be enabled at `TOP` only once a handler is registered on the PCI vector that
+/// delivers it. See "The serviced-subtree invariant" in
+/// `Documentation/gpu/nova/core/interrupts.rst`.
+pub(crate) struct Tree<'a> {
+    bar: Bar0<'a>,
+    hal: &'static dyn CpuInterruptHal,
+    serviced: SubtreeSet,
+    /// The interrupt type that Linux granted for `vectors`.
+    pub(super) msi_type: MsiType,
+    vectors: pci::IrqVectorRegistration<'a>,
+}
+
+impl<'a> Tree<'a> {
+    /// Creates the tree of `chipset` and allocates the PCI vectors for the subtrees in `serviced`.
+    ///
+    /// Requests MSI-X entries `0` through the highest subtree in `serviced`, since an allocation
+    /// cannot be sparse, and falls back to a single MSI message for the whole tree.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `serviced` is empty or names a subtree that `chipset` does not implement.
+    /// Otherwise, when neither interrupt type could be allocated, the error from the MSI request.
+    pub(crate) fn new(
+        pdev: &'a pci::Device<Bound>,
+        bar: Bar0<'a>,
+        chipset: Chipset,
+        serviced: SubtreeSet,
+    ) -> Result<Self> {
+        let hal = cpu_interrupt_hal(chipset);
+        let num_leaves = hal.leaf_count();
+
+        if serviced.is_empty() || serviced.intersection(num_leaves.subtree_set()) != serviced {
+            return Err(EINVAL);
+        }
+
+        let entries = serviced.span();
+        let (vectors, msi_type) = pdev
+            .alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
+            .map(|vectors| (vectors, MsiType::MsiX))
+            // Every subtree raises the one MSI message, so one MSI vector serves the whole tree.
+            // See "Delivery over PCI" in `Documentation/gpu/nova/core/interrupts.rst`.
+            .or_else(|_| {
+                pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
+                    .map(|vectors| (vectors, MsiType::Msi))
+            })?;
+
+        Ok(Self {
+            bar,
+            hal,
+            serviced,
+            msi_type,
+            vectors,
+        })
+    }
+
+    /// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `subtree` is not one of the serviced subtrees.
+    pub(super) fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
+        if !self.serviced.contains(subtree) {
+            return Err(EINVAL);
+        }
+
+        let entry = match self.msi_type {
+            MsiType::MsiX => num::u32_as_usize(subtree.index()),
+            MsiType::Msi => 0,
+        };
+
+        self.vectors.index(entry).map(Into::into)
+    }
+
+    /// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree that
+    /// the calling handler serves.
+    ///
+    /// A handler must call this before returning, or it receives no further interrupts.
+    pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
+        self.hal
+            .rearm_pci_irq(self.bar, self.msi_type, self.serviced, subtree);
+    }
+
+    /// Enables the serviced subtrees at `TOP`.
+    ///
+    /// Each of them must have a handler registered on its PCI vector.
+    pub(super) fn enable_top(&self) {
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
+        );
+    }
+
+    /// Disables the serviced subtrees at `TOP`.
+    pub(super) fn disable_top(&self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+
+    /// Enables the serviced subtrees at `TOP` until the returned guard drops.
+    pub(crate) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
+        self.enable_top();
+
+        TopEnableGuard {
+            bar: self.bar,
+            serviced: self.serviced,
+        }
+    }
+
+    /// Enables `vectors` in `leaf`.
+    pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        self.bar.write(
+            Array::at(*leaf),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
+        );
+    }
+
+    /// Disables `vectors` in `leaf`.
+    pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        clear_leaf_enables(self.bar, leaf, vectors);
+    }
+
+    /// Enables `vectors` in `leaf` until the returned guard drops.
+    pub(super) fn enable_leaf_guarded(
+        &self,
+        leaf: LeafIndex,
+        vectors: LeafMask,
+    ) -> LeafEnableGuard<'a> {
+        self.enable_leaf(leaf, vectors);
+
+        LeafEnableGuard {
+            bar: self.bar,
+            leaf,
+            vectors,
+        }
+    }
+
+    /// Reads the pending bits of `leaf`, and returns the handle that clears them.
+    pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
+        let pending = self
+            .bar
+            .read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
+            .vectors();
+
+        LeafPending {
+            bar: self.bar,
+            leaf,
+            pending,
+        }
+    }
+
+    /// Disables every vector in every implemented leaf, including the subtrees that nova-core does
+    /// not service. Call this only during probe.
+    pub(super) fn disable_all_leaves(&self) {
+        for leaf in self.hal.leaf_count().iter() {
+            self.disable_leaf(leaf, LeafMask::all());
+        }
+    }
+
+    /// Clears every pending bit in every implemented leaf, including the subtrees that nova-core
+    /// does not service.
+    ///
+    /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with no
+    /// interrupt handler registered.
+    pub(super) fn drain(&self) {
+        self.disable_top();
+
+        // A vector that latched while disabled does not show in `TOP`, so read every leaf rather
+        // than descending from it.
+        for leaf in self.hal.leaf_count().iter() {
+            self.read_pending(leaf).clear();
+        }
+    }
+
+    /// Disables every vector, clears every pending bit, and rearms PCI interrupt delivery.
+    ///
+    /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with
+    /// no interrupt handler registered.
+    pub(super) fn reset(&self) {
+        self.disable_all_leaves();
+        self.drain();
+        for subtree in self.serviced.iter() {
+            self.rearm_pci_irq(subtree);
+        }
+
+        // A `TOP`-enable rearm leaves the subtrees that it cycles enabled, and a subtree may be
+        // enabled only once a handler is registered on its vector.
+        self.disable_top();
+    }
+}
+
+/// The pending bits of one leaf as they were read, and the handle that clears them.
+pub(super) struct LeafPending<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    pending: LeafMask,
+}
+
+impl LeafPending<'_> {
+    pub(super) fn vectors(&self) -> LeafMask {
+        self.pending
+    }
+
+    /// Clears the vectors that were pending at the read. A vector that latched since stays pending.
+    pub(super) fn clear(&self) {
+        self.clear_vectors(self.pending);
+    }
+
+    /// Clears `vectors` and no other bit.
+    pub(super) fn clear_vectors(&self, vectors: LeafMask) {
+        if !vectors.is_empty() {
+            self.bar.write(
+                Array::at(*self.leaf),
+                NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
+            );
+        }
+    }
+}
+
+/// Disables a set of vectors in one leaf when dropped.
+pub(super) struct LeafEnableGuard<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    vectors: LeafMask,
+}
+
+impl Drop for LeafEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_leaf_enables(self.bar, self.leaf, self.vectors);
+    }
+}
+
+/// Disables the serviced subtrees at `TOP` when dropped.
+pub(crate) struct TopEnableGuard<'a> {
+    bar: Bar0<'a>,
+    serviced: SubtreeSet,
+}
+
+impl Drop for TopEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+}
-- 
2.55.0