[PATCH v4 06/17] gpu: nova-core: add the GIN interrupt tree and allocate its vectors

[John Hubbard <jhubbard@nvidia.com>]

From: Joel Fernandes <joelagnelf@nvidia.com>

Servicing a GIN leaf has a required order: read its pending bits, then
clear them. Clearing a leaf first discards every vector latched in it,
and the hardware keeps no record of what was discarded. The interrupt
never arrives, and no register shows that it was ever pending.

Every subtree enabled at TOP also needs an allocated PCI vector with a
handler registered on it. Under MSI-X each subtree has its own table
entry. Linux masks every entry until a driver requests its IRQ, and a
masked entry sends no message. An enabled subtree whose entry was never
requested raises interrupts that never reach a handler, while the leaf
and TOP registers show them pending and enabled. Under MSI the whole
tree raises a single message, so one entry serves every subtree.

Add the CPU interrupt tree of one PCIe function. Reading a leaf yields
the handle that clears it, so the wrong order does not compile. Building
a tree fails if it names a subtree that the GPU does not implement.

Allocate the PCI vectors from the set of serviced subtrees. Request
MSI-X entries 0 through the highest serviced subtree, since an MSI-X
allocation cannot be sparse, and fall back to a single MSI message,
never to INTx.

Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: reworked on top of the GIN vector types: a leaf read yields
 the handle that clears it, enables are guarded, the leaf count and
 rearm method come from the interrupt HAL, and the drain reads every
 implemented leaf rather than descending from TOP]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |  82 +++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 252 +++++++++++++++++++-
 2 files changed, 327 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index 62c242b71dc8..28f147641024 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -13,6 +13,23 @@
 mod interrupt_tree;
 mod regs;
 
+use kernel::{
+    device::Bound,
+    irq,
+    pci::{
+        self,
+        IrqType, //
+    },
+    prelude::*, //
+};
+
+use crate::num;
+
+use interrupt_tree::{
+    Subtree,
+    SubtreeSet, //
+};
+
 /// The message-signaled interrupt type that Linux granted.
 ///
 /// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
@@ -24,3 +41,68 @@ enum MsiType {
     /// One table entry per subtree.
     MsiX,
 }
+
+/// The PCI interrupt vectors allocated for the subtrees that nova-core services.
+///
+/// A subtree may be enabled at `TOP` only once a vector is allocated for it and a handler is
+/// registered on that vector. See "The serviced-subtree invariant" in
+/// `Documentation/gpu/nova/core/interrupts.rst`.
+pub(crate) struct SubtreeVectors<'a> {
+    vectors: pci::IrqVectorRegistration<'a>,
+    serviced: SubtreeSet,
+    msi_type: MsiType,
+}
+
+impl SubtreeVectors<'_> {
+    /// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `subtree` is not one of the serviced subtrees.
+    fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
+        if !self.serviced.contains(subtree) {
+            return Err(EINVAL);
+        }
+
+        let entry = match self.msi_type {
+            MsiType::MsiX => num::u32_as_usize(subtree.index()),
+            MsiType::Msi => 0,
+        };
+
+        self.vectors.index(entry).map(Into::into)
+    }
+}
+
+/// Allocates the PCI interrupt vectors for the subtrees in `serviced`.
+///
+/// Requests MSI-X entries `0` through the highest subtree in `serviced`, since an allocation
+/// cannot be sparse, and falls back to a single MSI message for the whole tree.
+///
+/// # Errors
+///
+/// `EINVAL` if `serviced` is empty. Otherwise, when neither type could be allocated, the error
+/// from the MSI request.
+pub(crate) fn alloc_vectors(
+    pdev: &pci::Device<Bound>,
+    serviced: SubtreeSet,
+) -> Result<SubtreeVectors<'_>> {
+    if serviced.is_empty() {
+        return Err(EINVAL);
+    }
+
+    let entries = serviced.span();
+
+    let (vectors, msi_type) = pdev
+        .alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
+        .map(|vectors| (vectors, MsiType::MsiX))
+        .or_else(|_| {
+            pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
+                .map(|vectors| (vectors, MsiType::Msi))
+        })?;
+
+    Ok(SubtreeVectors {
+        vectors,
+        serviced,
+        msi_type,
+    })
+}
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 5c7829ea3bc5..2583b006019e 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -1,22 +1,40 @@
 // SPDX-License-Identifier: GPL-2.0
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
-//! Vector addressing in the GIN CPU interrupt tree.
+//! The GIN CPU interrupt tree for one PCIe function.
 //!
 //! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
-//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
-//! keep the four from being confused with one another.
+//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit.
+//!
+//! Servicing a leaf requires reading its pending bits before clearing them. Only
+//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] clears a leaf,
+//! so the wrong order does not compile. Nothing in this module serializes access to the tree.
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 use kernel::{
+    io::{
+        register::Array,
+        Io, //
+    },
     num::Bounded,
     prelude::*, //
 };
 
-use crate::num;
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    num, //
+};
 
-use super::regs::*;
+use super::{
+    hal::{
+        cpu_interrupt_hal,
+        PciIrqRearmMethod, //
+    },
+    regs::*,
+    SubtreeVectors, //
+};
 
 /// Number of vectors one leaf register carries, one per bit.
 const VECTORS_PER_LEAF: u32 = u32::BITS;
@@ -78,6 +96,11 @@ pub(super) const fn subtree_set(self) -> SubtreeSet {
     pub(super) const fn vector_count(self) -> u32 {
         self.into_u32() * VECTORS_PER_LEAF
     }
+
+    /// Returns every leaf a tree of this size implements.
+    pub(super) fn iter(self) -> impl Iterator<Item = LeafIndex> {
+        (0..self.into_raw()).filter_map(LeafIndex::try_new)
+    }
 }
 
 // `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
@@ -130,7 +153,7 @@ fn from(vectors: LeafMask) -> Self {
 ///
 /// Exactly one bit is set.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct Subtree(u32);
+pub(crate) struct Subtree(u32);
 
 impl Subtree {
     /// Returns the subtree at index `idx`.
@@ -151,7 +174,7 @@ pub(super) const fn into_raw(self) -> u32 {
 
 /// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct SubtreeSet(u32);
+pub(crate) struct SubtreeSet(u32);
 
 impl SubtreeSet {
     pub(super) const fn contains(self, subtree: Subtree) -> bool {
@@ -250,3 +273,218 @@ fn from(vector: GinVector) -> Self {
         vector.0.extend()
     }
 }
+
+/// Disables `vectors` in `leaf`.
+fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+    bar.write(
+        Array::at(*leaf),
+        NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
+    );
+}
+
+/// Disables the subtrees in `serviced` at `TOP`.
+fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
+    bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
+}
+
+/// The CPU tree of one PCIe function, and the subtrees that nova-core services.
+pub(super) struct Tree<'a> {
+    bar: Bar0<'a>,
+    leaves: LeafCount,
+    serviced: SubtreeSet,
+    rearm: PciIrqRearmMethod,
+}
+
+impl<'a> Tree<'a> {
+    /// Creates the tree of `chipset`, covering the subtrees that `vectors` services.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `chipset` does not implement every subtree that `vectors` services.
+    pub(super) fn new(
+        bar: Bar0<'a>,
+        chipset: Chipset,
+        vectors: &SubtreeVectors<'_>,
+    ) -> Result<Self> {
+        let hal = cpu_interrupt_hal(chipset);
+        let leaves = hal.leaf_count();
+        let serviced = vectors.serviced;
+
+        if serviced.intersection(leaves.subtree_set()) != serviced {
+            return Err(EINVAL);
+        }
+
+        Ok(Self {
+            bar,
+            leaves,
+            serviced,
+            rearm: hal.pci_irq_rearm_method(vectors.msi_type),
+        })
+    }
+
+    /// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree that
+    /// the calling handler serves.
+    ///
+    /// A handler must call this before returning, or it receives no further interrupts.
+    pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
+        self.rearm.rearm(self.bar, self.serviced, subtree);
+    }
+
+    /// Enables the serviced subtrees at `TOP`.
+    ///
+    /// Each of them must have a handler registered on its PCI vector.
+    pub(super) fn enable_top(&self) {
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
+        );
+    }
+
+    /// Disables the serviced subtrees at `TOP`.
+    pub(super) fn disable_top(&self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+
+    /// Enables the serviced subtrees at `TOP` until the returned guard drops.
+    pub(super) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
+        self.enable_top();
+
+        TopEnableGuard {
+            bar: self.bar,
+            serviced: self.serviced,
+        }
+    }
+
+    /// Enables `vectors` in `leaf`.
+    pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        self.bar.write(
+            Array::at(*leaf),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
+        );
+    }
+
+    /// Disables `vectors` in `leaf`.
+    pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        clear_leaf_enables(self.bar, leaf, vectors);
+    }
+
+    /// Enables `vectors` in `leaf` until the returned guard drops.
+    pub(super) fn enable_leaf_guarded(
+        &self,
+        leaf: LeafIndex,
+        vectors: LeafMask,
+    ) -> LeafEnableGuard<'a> {
+        self.enable_leaf(leaf, vectors);
+
+        LeafEnableGuard {
+            bar: self.bar,
+            leaf,
+            vectors,
+        }
+    }
+
+    /// Reads the pending bits of `leaf`, and returns the handle that clears them.
+    pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
+        let pending = self
+            .bar
+            .read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
+            .vectors();
+
+        LeafPending {
+            bar: self.bar,
+            leaf,
+            pending,
+        }
+    }
+
+    /// Latches `vector` as its own source would.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if this tree does not implement `vector`.
+    // The interrupt self-test is the only caller.
+    #[expect(dead_code)]
+    pub(super) fn trigger(&self, vector: GinVector) -> Result {
+        vector.validate(self.leaves)?;
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
+        );
+
+        Ok(())
+    }
+
+    /// Disables every vector in every implemented leaf, including the subtrees that nova-core does
+    /// not service. Call this only during probe.
+    pub(super) fn disable_all_leaves(&self) {
+        for leaf in self.leaves.iter() {
+            self.disable_leaf(leaf, LeafMask::all());
+        }
+    }
+
+    /// Clears every pending bit in every implemented leaf, including the subtrees that nova-core
+    /// does not service.
+    ///
+    /// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with no
+    /// interrupt handler registered.
+    pub(super) fn drain(&self) {
+        self.disable_top();
+
+        // A vector that latched while disabled does not show in `TOP`, so read every leaf rather
+        // than descending from it.
+        for leaf in self.leaves.iter() {
+            self.read_pending(leaf).clear();
+        }
+    }
+}
+
+/// The pending bits of one leaf as they were read, and the handle that clears them.
+pub(super) struct LeafPending<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    pending: LeafMask,
+}
+
+impl LeafPending<'_> {
+    pub(super) fn vectors(&self) -> LeafMask {
+        self.pending
+    }
+
+    /// Clears the vectors that were pending at the read. A vector that latched since stays pending.
+    pub(super) fn clear(&self) {
+        self.clear_vectors(self.pending);
+    }
+
+    /// Clears `vectors` and no other bit.
+    pub(super) fn clear_vectors(&self, vectors: LeafMask) {
+        if !vectors.is_empty() {
+            self.bar.write(
+                Array::at(*self.leaf),
+                NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
+            );
+        }
+    }
+}
+
+/// Disables a set of vectors in one leaf when dropped.
+pub(super) struct LeafEnableGuard<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    vectors: LeafMask,
+}
+
+impl Drop for LeafEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_leaf_enables(self.bar, self.leaf, self.vectors);
+    }
+}
+
+/// Disables the serviced subtrees at `TOP` when dropped.
+pub(super) struct TopEnableGuard<'a> {
+    bar: Bar0<'a>,
+    serviced: SubtreeSet,
+}
+
+impl Drop for TopEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+}
-- 
2.55.0