Re: [PATCH v3 08/33] gpu: nova-core: gsp: compute the queue regions from a count and a slot

["Alexandre Courbot" <acourbot@nvidia.com>]

On Fri Sep 18, 2026 at 10:06 AM JST, John Hubbard wrote:
> The r000 firmware uses msgq v2, the queue layout that keeps the queue
> pointers in BAR0 registers as counts that do not wrap at the ring size.
> The r570 firmware's layout keeps the pointers in shared memory as
> indices into the ring. The two layouts differ in where a pointer is
> read and in how the size of the region that the driver may write, and
> of the region that it may read, follows from a queue's write pointer
> and read pointer. Splitting a region across the end of the ring is the
> same in both, and whether a region wraps follows from the order of the
> two pointers.
>
> The functions for the writable region and for the readable region each
> branched on the order of the write pointer and the read pointer. Each
> branch chose where the two slices ended, and the function then built
> the slices with open-coded pointer arithmetic. The SAFETY comments
> argued the slice bounds branch by branch, so the switch to msgq v2
> would have had to rewrite the branches and the argument along with the
> pointer rules.
>
> Compute the number of slots in a region and its start slot once, and
> split the ring at the start slot. The first slice ends at the end of
> the region or at the end of the ring, whichever comes first, and the
> second slice carries the rest.
>
> No functional changes.
>
> Assisted-by: LLM
> Signed-off-by: John Hubbard <jhubbard@nvidia.com>
> ---
>  drivers/gpu/nova-core/gsp/cmdq.rs | 120 ++++++++++--------------------
>  1 file changed, 41 insertions(+), 79 deletions(-)

This looks like an improvement regardless of the r000 switch!

>
> diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
> index 80e6e79c5f3c..a1c9b7cce255 100644
> --- a/drivers/gpu/nova-core/gsp/cmdq.rs
> +++ b/drivers/gpu/nova-core/gsp/cmdq.rs
> @@ -262,107 +262,69 @@ fn new(dev: &'a device::Device<device::Bound>, bar: Bar0<'a>) -> Result<Self> {
>          Ok(Self { mem: gsp_mem, bar })
>      }
>  
> -    /// Returns the region of the CPU message queue that the driver is currently allowed to write
> -    /// to.
> +    /// Returns the region of the CPU message queue that the driver may write to.
>      ///
> -    /// As the message queue is a circular buffer, the region may be discontiguous in memory. In
> -    /// that case the second slice will have a non-zero length.
> +    /// The ring wraps, so the region comes as two slices, and the second is empty unless the
> +    /// region crosses the end of the ring.

There is a recurring pattern in this series to drive-by rewrite comments
when there is no real need to do so. The new comment is not even
marginally better as we lose the temporal nature ("currently") of the
borrow. This creates churn restating the same thing using different
words and disrupts the diff, so can we avoid doing that unless the patch
actually changes what the comment describes?

>      fn driver_write_area(&mut self) -> (&mut [[u8; GSP_PAGE_SIZE]], &mut [[u8; GSP_PAGE_SIZE]]) {
> -        let tx = self.cpu_write_ptr();
> -        let rx = self.gsp_read_ptr();
> +        let avail = num::u32_as_usize(self.free_slots());
> +        let w_slot = num::u32_as_usize(self.cpu_write_ptr());
>  
>          // Pointer to the first entry of the CPU message queue.
>          let data = ptr::project!(mut self.mem.as_mut_ptr(), .cpuq.msgq.data[build: 0]);
>  
> -        let (tail_end, wrap_end) = if rx == 0 {
> -            // The write area is non-wrapping, and stops at the second-to-last entry of the command
> -            // queue (to leave the last one empty).
> -            (MSGQ_NUM_PAGES - 1, 0)
> -        } else if rx <= tx {
> -            // The write area wraps and continues until `rx - 1`.
> -            (MSGQ_NUM_PAGES, rx - 1)
> -        } else {
> -            // The write area doesn't wrap and stops at `rx - 1`.
> -            (rx - 1, 0)
> -        };
> -
>          // SAFETY:
> -        // - `data` was created from a valid pointer, and `rx` and `tx` are in the
> -        //   `0..MSGQ_NUM_PAGES` range per the invariants of `cpu_write_ptr` and `gsp_read_ptr`,
> -        //   thus the created slices are valid.
> -        // - The area starting at `tx` and ending at `rx - 2` modulo `MSGQ_NUM_PAGES`,
> -        //   inclusive, belongs to the driver for writing and is not accessed concurrently by
> -        //   the GSP.
> -        // - The caller holds a reference to `self` for as long as the returned slices are live,
> -        //   meaning the CPU write pointer cannot be advanced and thus that the returned area
> -        //   remains exclusive to the CPU for the duration of the slices.
> -        // - The created slices point to non-overlapping sub-ranges of `data` in all
> -        //   branches (in the `rx <= tx` case, the second slice ends at `rx - 1` which is strictly
> -        //   less than `tx` where the first slice starts; in the other cases the second slice is
> -        //   empty), so creating two `&mut` references from them does not violate aliasing rules.
> -        unsafe {
> -            (
> -                core::slice::from_raw_parts_mut(
> -                    data.add(num::u32_as_usize(tx)),
> -                    num::u32_as_usize(tail_end - tx),
> -                ),
> -                core::slice::from_raw_parts_mut(data, num::u32_as_usize(wrap_end)),
> -            )
> -        }
> +        // - `data` points to the `MSGQ_NUM_PAGES` initialized entries of the CPU message queue.
> +        // - The returned slices cover the `avail` free slots from the write pointer on, which the
> +        //   GSP does not read until `advance_cpu_write_ptr` publishes them.
> +        // - `split_at_mut` gives two non-overlapping halves, and the `&mut self` borrow lasts as
> +        //   long as the returned slices, so that no other call hands out the same region while
> +        //   they live.
> +        let data =
> +            unsafe { core::slice::from_raw_parts_mut(data, num::u32_as_usize(MSGQ_NUM_PAGES)) };
> +        let (before_w, after_w) = data.split_at_mut(w_slot);

This creates a reference over the whole ring, including the parts owned
by the GSP, which breaks the `Coherent` safety contract that the device
must not be able to read or write to a live slice. So we'll need to call
`from_raw_parts_mut` twice, with the correct sizes, instead of
splitting.

(also `split_at_mut` is panicking and should have a `PANIC:` comment
justifying why it cannot, but once the point above is addressed that
call will go away).

I wanted to try it locally and ended up with something that seems to
work, so let me share it to save some time:

    fn driver_write_area(&mut self) -> (&mut [[u8; GSP_PAGE_SIZE]], &mut [[u8; GSP_PAGE_SIZE]]) {
        let avail = self.free_slots();
        let w_slot = self.cpu_write_ptr();

        // Pointer to the first entry of the CPU message queue.
        let data = ptr::project!(mut self.mem.as_mut_ptr(), .cpuq.msgq.data[build: 0]);

        let in_after = avail.min(MSGQ_NUM_PAGES - w_slot);
        let in_before = avail - in_after;

        // SAFETY:
        // - `data` was created from a valid pointer of `MSGQ_NUM_PAGES` entries.
        // - The `in_after` entries after `w_slot` belong to the `avail` entries that the driver is
        //   currently allowed to write.
        // - The `in_before` first entries belong to the `avail` entries that the driver is
        //   currently allowed to write.
        // - The slices do not overlap.
        unsafe {
            (
                core::slice::from_raw_parts_mut(
                    data.add(num::u32_as_usize(w_slot)),
                    num::u32_as_usize(in_after),
                ),
                core::slice::from_raw_parts_mut(data, num::u32_as_usize(in_before)),
            )
        }
    }

It has turned out quite short, which I like! I also opted to work with
the original `u32` until the very end, as it results in less conversions
overall.

> +
> +        let in_after = avail.min(after_w.len());
> +        let in_before = avail - in_after;
> +        (&mut after_w[..in_after], &mut before_w[..in_before])
>      }
>  
> -    /// Returns the size of the region of the CPU message queue that the driver is currently allowed
> -    /// to write to, in bytes.
> -    fn driver_write_area_size(&self) -> usize {
> +    /// Returns the number of command queue slots that the driver may still write.
> +    fn free_slots(&self) -> u32 {

The method name should specify which queue we are dealing with here,
"free_slots" is too generic. And for symmetry the read side should
also get the same helper, even if `driver_read_area` is the only user,
as it makes the code easier to parse.

>          let tx = self.cpu_write_ptr();
>          let rx = self.gsp_read_ptr();
>  
> -        // `rx` and `tx` are both in `0..MSGQ_NUM_PAGES` per the invariants of `gsp_read_ptr` and
> -        // `cpu_write_ptr`. The minimum value case is where `rx == 0` and `tx == MSGQ_NUM_PAGES -
> -        // 1`, which gives `0 + MSGQ_NUM_PAGES - (MSGQ_NUM_PAGES - 1) - 1 == 0`.
> -        let slots = (rx + MSGQ_NUM_PAGES - tx - 1) % MSGQ_NUM_PAGES;
> -        num::u32_as_usize(slots) * GSP_PAGE_SIZE
> +        // One slot always stays empty, so that a full ring and an empty ring differ in their
> +        // pointers. `tx` is below `MSGQ_NUM_PAGES`, so the subtraction does not underflow.
> +        (rx + MSGQ_NUM_PAGES - tx - 1) % MSGQ_NUM_PAGES
>      }
>  
> -    /// Returns the region of the GSP message queue that the driver is currently allowed to read
> -    /// from.
> -    ///
> -    /// As the message queue is a circular buffer, the region may be discontiguous in memory. In
> -    /// that case the second slice will have a non-zero length.
> +    /// Returns the number of bytes that the driver can still write to the command queue.
> +    fn driver_write_area_size(&self) -> usize {
> +        num::u32_as_usize(self.free_slots()) * GSP_PAGE_SIZE
> +    }
> +
> +    /// Returns the region of the GSP message queue that the driver may read, as two slices
> +    /// because the ring wraps.
>      fn driver_read_area(&self) -> (&[[u8; GSP_PAGE_SIZE]], &[[u8; GSP_PAGE_SIZE]]) {
>          let tx = self.gsp_write_ptr();
>          let rx = self.cpu_read_ptr();
> +        let avail = num::u32_as_usize((tx + MSGQ_NUM_PAGES - rx) % MSGQ_NUM_PAGES);
> +        let r_slot = num::u32_as_usize(rx);

Indeed, with a helper like `free_slots` for the read side we could get
rid of `tx` and `rx` and spell what the code does more clearly, so I
think that'd be a win.

The helper makes us read the CPU read ptr one extra time, but that's a
negligible tradeoff for better code readability.

>  
>          // Pointer to the first entry of the GSP message queue.
>          let data = ptr::project!(self.mem.as_ptr(), .gspq.msgq.data[build: 0]);
>  
> -        let (tail_end, wrap_end) = if rx <= tx {
> -            // Read area is non-wrapping and stops right before `tx`.
> -            (tx, 0)
> -        } else {
> -            // Read area is wrapping and stops right before `tx`.
> -            (MSGQ_NUM_PAGES, tx)
> -        };
> -
>          // SAFETY:
> -        // - `data` was created from a valid pointer, and `rx` and `tx` are in the
> -        //   `0..MSGQ_NUM_PAGES` range per the invariants of `gsp_write_ptr` and `cpu_read_ptr`,
> -        //   thus the created slices are valid.
> -        // - The area starting at `rx` and ending at `tx - 1` modulo `MSGQ_NUM_PAGES`,
> -        //   inclusive, belongs to the driver for reading and is not accessed concurrently by
> -        //   the GSP.
> -        // - The caller holds a reference to `self` for as long as the returned slices are live,
> -        //   meaning the CPU read pointer cannot be advanced and thus that the returned area
> -        //   remains exclusive to the CPU for the duration of the slices.
> -        unsafe {
> -            (
> -                core::slice::from_raw_parts(
> -                    data.add(num::u32_as_usize(rx)),
> -                    num::u32_as_usize(tail_end - rx),
> -                ),
> -                core::slice::from_raw_parts(data, num::u32_as_usize(wrap_end)),
> -            )
> -        }
> +        // - `data` points to the `MSGQ_NUM_PAGES` initialized entries of the GSP message queue.
> +        // - The returned slices cover the `avail` slots that the GSP has already written. The GSP
> +        //   does not write them again until `advance_cpu_read_ptr` releases them.
> +        let data = unsafe { core::slice::from_raw_parts(data, num::u32_as_usize(MSGQ_NUM_PAGES)) };
> +        let (before_r, after_r) = data.split_at(r_slot);

Same problem of ring ownership - we need to call `from_raw_parts` twice.
The pattern used in `driver_write_area` should apply smoothly here as well.