[PATCH 0/8] rust: Add drm::JobQueue

[PATCH 0/8] rust: Add drm::JobQueue

[Philipp Stanner]

Changes since RFCv7:
  - Write a bunch of documentation, mostly about the kill() backend_ops.
  - Add creations of own work_queues (requires some work with the life
    times still; last relevant TODO AFAICS).

Depends on the workqueue series:
https://lore.kernel.org/rust-for-linux/20261020-rust-workqueue-v3-0-ea7c74d02068@google.com/

Besides based on drm-rust-next

Daniel tested an earlier RFC (v6 I think) on Tyr and reports that it ran
well. In v2 I want to have our own workqueues ready, perhaps after the
workqueue has gone upstream.

But I thought it's time to present this as v1 since we settled on the
design, it's mostly working already and we can now begin diving out the
remaining problems.

So please review :)

Thx again to everyone who helped with moving this forward.

Last RFC:
https://lore.kernel.org/dri-devel/20261103153138.1087330-2-phasta@kernel.org/

Repo:
https://gitlab.freedesktop.org/pstanner/linux-drm-work/-/tree/jobqueue-jobqueue-v1-2?ref_type=heads


PS:
Note that this version now obtains a fence->error under lock,
because I strongly suspect that this pattern

if (dma_fence_is_signaled(f))
  pr_info("err: %i\n", f->error);

is UB since there is no memory ordering between the reads (which would
proof that fences' locks are not "just for the cb_list", as was sometimes
suggested). My discussions regarding fixing that in C can be found here:

https://lore.kernel.org/dri-devel/20261020100034.759183-2-phasta@kernel.org/
https://lore.kernel.org/dri-devel/2db0f1b4-6628-465e-928e-d779b89f59dc@amd.com/


Philipp Stanner (8):
  rust: DmaFence: Implement Deref for FenceContext
  rust: DmaFence: Add ID accessor for FenceContext
  rust: DmaFence: Don't drop on signal
  rust: DmaFence: Replace call_rcu() with synchronize_rcu()
  rust: DmaFence: Replace is_signaled() with status()
  rust: DmaFence: Pass result in callbacks
  rust: drm: Add JobQueue
  MAINTAINERS: Add drm::JobQueue to drm_sched's entry

 MAINTAINERS                      |   1 +
 rust/helpers/dma_fence.c         |   4 +-
 rust/kernel/dma_buf/dma_fence.rs | 234 ++++-----
 rust/kernel/dma_buf/mod.rs       |   3 +-
 rust/kernel/drm/job_queue.rs     | 873 +++++++++++++++++++++++++++++++
 rust/kernel/drm/mod.rs           |   4 +
 6 files changed, 996 insertions(+), 123 deletions(-)
 create mode 100644 rust/kernel/drm/job_queue.rs


base-commit: 7a6bc132f1b51437063c9203e7221067b40a791e
prerequisite-patch-id: f5e24f7b3717f2ab0445b5395c7f12b554861d78
prerequisite-patch-id: e4ebcb108207c2d06130382c59c12ffe09a7a6db
prerequisite-patch-id: 8dc24064e858240a6bb5411a261a987a7eae3fad
prerequisite-patch-id: eefbd4a72ed6da0083e20811882738cc3b05604f
prerequisite-patch-id: ea89aac6f42582c91174068c1db4d171bd7a8920
-- 
2.56.0

[PATCH 1/8] rust: DmaFence: Implement Deref for FenceContext

[Philipp Stanner]

A FenceContext carries data, which the user should be able to access.

Implement Deref for FenceContext.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/kernel/dma_buf/dma_fence.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index a1d851656562..5300448882e7 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -86,6 +86,15 @@ pub struct FenceContext<T: FenceContextOps + Send + Sync> {
     data: T,
 }
 
+impl<T: Send + Sync + FenceContextOps> Deref for FenceContext<T> {
+    type Target = T;
+
+    #[inline]
+    fn deref(&self) -> &Self::Target {
+        &self.data
+    }
+}
+
 impl<'a, T: Send + Sync + FenceContextOps> FenceContext<T> {
     // This can later be extended as a vtable in case other parties need support
     // for the more "exotic" callbacks.
-- 
2.56.0

[PATCH 2/8] rust: DmaFence: Add ID accessor for FenceContext

[Philipp Stanner]

A FenceContext has a fence context number, which can serve as an ID
field. This number can be useful for owners of multiple contexts to
identify them, store them in an XArray etc.

Add an accessor for fctx.nr.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/kernel/dma_buf/dma_fence.rs | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index 5300448882e7..897308f60dd3 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -135,6 +135,12 @@ pub fn new<E>(
         })
     }
 
+    /// Obtain the FenceContext's ID number.
+    #[inline]
+    pub fn id(&self) -> u64 {
+        self.nr
+    }
+
     fn next_seqno(&self) -> u64 {
         self.seqno.fetch_add(1, Relaxed)
     }
-- 
2.56.0

[PATCH 3/8] rust: DmaFence: Don't drop on signal

[Philipp Stanner]

The original design for DmaFence intended to comply as strictly as
possible with the "dma_fence contract", which, according to some, says
that a fence must only be signaled once. Thus, the implementation was
made so that a DriverFence drops on signal.

While extending JobQueue's capabilities, it was discovered that this
characteristic causes - unnecessary - trouble: fence-data must be
dropped in an RCU-deferred manner because the C backend demands that.
However, fences are often signaled in atomic context (e.g., interrupt
handler) where you don't want to drop, but signal as quickly as
possible.

Actually, there is no real hard reason to enforce that a fence can only
get signaled once, because the C backend is robust against multiple
signal attempts.

What is decisive is that a) all fences signal and b) they only drop no
earlier than 1 RCU grace period after signaling.

Don't drop a DriverFence on signal().

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/kernel/dma_buf/dma_fence.rs | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index 897308f60dd3..c04b68fe67fe 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -820,7 +820,7 @@ pub fn as_fence(&self) -> &Fence {
     }
 
     /// Signal the fence. This will invoke all registered callbacks.
-    pub fn signal(self, res: Result) {
+    pub fn signal(&self, res: Result) {
         let fence = self.as_fence().lock();
 
         // SAFETY: `fence` is valid because `self` is valid. The lock must be
@@ -831,14 +831,15 @@ pub fn signal(self, res: Result) {
                 // must not have been signaled yet, which we check directly above.
                 unsafe { bindings::dma_fence_set_error(fence.as_raw(), err.to_errno()) };
             }
+
+            // SAFETY: `self.data` is valid because `self` is valid.
+            let fctx = unsafe { self.data.as_ref().fctx };
+            let _ = fctx.nr_of_unsignaled_fences.fetch_sub(1, Relaxed);
+
             // SAFETY: `fence` is valid because `self` is valid. The lock must
             // be held, which we acquired above.
             unsafe { bindings::dma_fence_signal_locked(fence.as_raw()) };
         }
-
-        // SAFETY: `self.data` is valid because `self` is valid.
-        let fctx = unsafe { self.data.as_ref().fctx };
-        let _ = fctx.nr_of_unsignaled_fences.fetch_sub(1, Relaxed);
     }
 }
 
-- 
2.56.0

[PATCH 4/8] rust: DmaFence: Replace call_rcu() with synchronize_rcu()

[Philipp Stanner]

The C dma_fence contract demands that a fence's data disappears no
earlier than 1 RCU grace period after the DriverFence was signaled. So
far, we achieved this with dropping the DriverFence's data in a deferred
manner with a manual, raw binding call to call_rcu().

However, this call_rcu() does not solve one problem: It is conceivable
that some driver data must not drop in atomic context; but call_rcu()'s
payload *can* be executed in atomic context.

Moreover, the current developments in drm::JobQueue, where a DriverFence
is always coupled 1:1 with a Job, allows for dropping job and fence
through a work item.

The undesirable blocking behavior of synchronize_rcu() would, thus, not
be annoying any thread anymore. The direct users of DmaFence could
achieve the same by dropping their DriverFences through work items.

The only conceivable other solution, queue_rcu_work(), would, for many
users, schedule a work_item from another work_item, which seems
undesirable.

Additionally, using the existing synchronize_rcu() abstraction has the
advantage of us getting rid of a raw bindings call.

Replace DriverFence::drop()'s call_rcu() with synchronize_rcu().

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/kernel/dma_buf/dma_fence.rs | 105 +++++++++----------------------
 1 file changed, 28 insertions(+), 77 deletions(-)

diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index c04b68fe67fe..f74119b4d7f8 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -42,7 +42,7 @@
             Atomic,
             Relaxed, //
         },
-        rcu::rcu_barrier, //
+        rcu::synchronize_rcu, //
     }, //
 };
 
@@ -58,6 +58,8 @@ pub trait FenceContextOps {
 /// A dma-fence context. A fence context takes care of associating related fences
 /// with each other, providing each with raising sequence numbers and a common
 /// identifier.
+///
+/// A FenceContext must drop in non-atomic context.
 #[pin_data(PinnedDrop)]
 pub struct FenceContext<T: FenceContextOps + Send + Sync> {
     /// The fence context number.
@@ -65,7 +67,7 @@ pub struct FenceContext<T: FenceContextOps + Send + Sync> {
     /// The sequence number for the next fence created.
     seqno: Atomic<u64>,
     // The name parameters can be accessed by the dma_fence backend_ops. UAF
-    // errors are prevented by the `call_rcu()` in `drop_driver_fence_data()`.
+    // errors are prevented by `synchronize_rcu()`.
     /// The name of the driver this FenceContext's fences belong to.
     driver_name: CString,
     /// The name of the timeline this FenceContext's fences belong to.
@@ -149,15 +151,11 @@ fn next_seqno(&self) -> u64 {
     ///
     /// This is needed because many times, creation of a [`DriverFence`] must not
     /// fail, and allocating might deadlock in some situations.
-    ///
-    /// The `data` you pass here must not perform any operations that are illegal
-    /// in atomic context in its [`Drop`] implementation.
     pub fn new_fence_allocation(
         &self,
         data: T::FenceDataType,
     ) -> Result<DriverFenceAllocation<'_, T>> {
         let fence_data = DriverFenceData {
-            rcu_head: Default::default(),
             // `inner` remains uninitialized until a `DriverFence` takes over.
             inner: Fence {
                 inner: Opaque::uninit(),
@@ -243,15 +241,10 @@ fn drop(self: Pin<&mut Self>) {
             panic!("Forgotten fences in FenceContext.");
         }
 
-        // Ensure that the driver cannot unload while there are still dma_fence
-        // callbacks running. At the same time, the RCU barrier addresses the
-        // problem inherited by the C backend, in which backend ops callbacks
-        // might be accessing the fence while it is being signaled (or shortly
-        // after). This could cause UAF access on the fence context's
-        // `fctx.driver_name` and `fctx.timeline_name`.
-        //
-        // Wait for the RCU callbacks in `DriverFence::drop`.
-        rcu_barrier();
+        // If a DriverFence signals without doing a synchronize_rcu(), and a
+        // fctx drops immediately afterwards, then backend_ops still accessing
+        // the fctx might run into UAF if we don't wait for them here.
+        synchronize_rcu();
     }
 }
 
@@ -615,12 +608,9 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
     // necessary so that the C backend can free the allocation (coming from our
     // Rust code) with kfree_rcu().
     inner: Fence,
-    /// Callback head for dropping this in a deferred manner through RCU.
-    rcu_head: bindings::callback_head,
     /// Reference to access the FenceContext.
     fctx: &'a FenceContext<T>,
-    /// The API user's data. It is essential that the data only performs
-    /// operations legal in atomic context in its [`Drop`] implementation.
+    /// The API user's data.
     #[pin]
     data: T::FenceDataType,
 }
@@ -659,6 +649,8 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 ///
 /// In the unlikely case of such violations, a panic is thrown.
 ///
+/// A DriverFence must drop in non-atomic context.
+///
 /// # Examples
 ///
 /// ```
@@ -955,15 +947,15 @@ fn drop(&mut self) {
         // dma_fence_is_signaled_locked() because the C backend wants to get rid
         // of the latter.
 
-        // SAFETY: `guard` is valid until the `call_rcu()` below.
+        // SAFETY: `guard` is valid.
         let signaled: bool = unsafe { bindings::dma_fence_test_signaled_flag(guard.as_raw()) };
         if !signaled {
             pr_err!("DriverFence drops unsignaled. Danger of memory corruption!\n");
-            // SAFETY: `guard` is valid until the `call_rcu()` below. The fence
-            // must not have been signaled yet, which we check directly above.
+            // SAFETY: `guard` is valid. The fence must not have been signaled
+            // yet, which we check directly above.
             unsafe { bindings::dma_fence_set_error(guard.as_raw(), ECANCELED.to_errno()) };
-            // SAFETY: `guard` is valid until the `call_rcu()` below. The lock
-            // must be held, which we acquired above.
+            // SAFETY: `guard` is valid. The lock must be held, which we
+            // acquired above.
             unsafe { bindings::dma_fence_signal_locked(guard.as_raw()) };
 
             // SAFETY: `self.data` is valid because `self` is valid.
@@ -988,59 +980,18 @@ fn drop(&mut self) {
             return;
         }
 
-        // SAFETY: Valid because `self` is valid.
-        let rcu_head_ptr = unsafe { &raw mut (*self.data.as_ptr()).rcu_head };
+        // Make sure none of the fence backend_ops can access fence-data anymore.
+        //
+        // TODO:
+        // This would not be necessary if the C dma_fence backend were using a
+        // spinlock to properly synchronize its signaled state. Fix it in C and
+        // then remove synchronize_rcu().
+        synchronize_rcu();
 
-        // SAFETY: `call_rcu()` is always safe to be called. `rcu_head_ptr` was
-        // created validly above. The module must perform a `synchronize_rcu()`
-        // or `rcu_barrier()` call to guard against module unload.
-        unsafe { bindings::call_rcu(rcu_head_ptr, Some(drop_driver_fence_data::<T>)) };
+        // SAFETY: Valid because `self` is valid.
+        unsafe { drop_in_place(&raw mut self.data) };
+
+        // SAFETY: The `synchronize_rcu()` above ensures all accessors are gone.
+        unsafe { bindings::dma_fence_put(self.as_raw()) };
     }
 }
-
-// TODO:
-// The entire call_rcu() mechanism in the drop above and the code below would be
-// unnecessary if C's dma_fence_signal() could be reworked in a way that after it
-// ran, the caller knows that no fence_ops callbacks can be running anymore.
-// In other words, if the dma_fence backend would use its spinlock for full
-// synchronization.
-//
-// Then we could move the drop_in_place() and dma_fence_put() upwards into the
-// drop() implementation and call it a day.
-
-/// Finally really drop this `DriverFence<T>`
-///
-/// # Safety
-///
-/// `head` references the `rcu_head` field of an `DriverFenceData<T>`. All
-/// accessors to that `DriverFenceData<T>` must be gone by now. This must be
-/// ensured by signalling the associated `DriverFence<T>` and then waiting
-/// for a grace period until calling this function here.
-unsafe extern "C" fn drop_driver_fence_data<T: Send + Sync + FenceContextOps>(
-    head: *mut bindings::callback_head,
-) {
-    // SAFETY: Caller provides a pointer to the `rcu_head` field of a `DriverFenceData<C>`.
-    let fence_data = unsafe { container_of!(head, DriverFenceData<'_, T>, rcu_head) };
-
-    // SAFETY: `fence_data` was created validly above. All the fence's data will
-    // only drop below, but the raw pointer to the raw C `dma_fence` remains
-    // valid because the reference count is only decremented at the end of the
-    // function.
-    let fence = unsafe { (*fence_data).inner.inner.get() };
-
-    // SAFETY: `fence_data` was created validly above. The user has already
-    // dropped the only conventional accessor to the user data, the `DriverFence`,
-    // one grace period ago. All accessors are gone now.
-    unsafe { drop_in_place(&raw mut (*fence_data).data) };
-
-    // The inner `Fence` explicitly does not get dropped because there may be
-    // many more users / consumers, each holding their own reference.
-
-    // SAFETY: Once a `DriverFence` is initialized, the inner `fence` is valid
-    // and initialized. It is valid until the refcount drops to 0, which can
-    // earliest happen once we drop the `DriverFence`'s reference here.
-    unsafe { bindings::dma_fence_put(fence) };
-
-    // The actual memory the data associated with a `DriverFence` lives in
-    // gets freed by the C dma_fence backend once the fence's refcount reaches 0.
-}
-- 
2.56.0

[PATCH 5/8] rust: DmaFence: Replace is_signaled() with status()

[Philipp Stanner]

As it turns out, our dma_fence abstractions lack an essential feature; a
problem which is partially inherited from the C backend:

Despite it being a common pattern in the kernel, you actually cannot
just check whether a fence was signaled and call it a day - it is
crucial that you also check whether a fence was signaled *with an
error*.

To make things more complicated, an error can get modified at any point
in time before a fence gets signaled. Thus, the only robust way of
obtaining the error code is by simultaneously checking the
signaled-state under lock-protection.

Since the fence status is essential information that must not be
ignored, replace Fence::is_signaled() with Fence::status(), making it
impossible to ignore the error code.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/helpers/dma_fence.c         |  4 +-
 rust/kernel/dma_buf/dma_fence.rs | 78 +++++++++++++++++++-------------
 rust/kernel/dma_buf/mod.rs       |  3 +-
 3 files changed, 51 insertions(+), 34 deletions(-)

diff --git a/rust/helpers/dma_fence.c b/rust/helpers/dma_fence.c
index 549f6b6a7171..9d8eb7db19bf 100644
--- a/rust/helpers/dma_fence.c
+++ b/rust/helpers/dma_fence.c
@@ -22,9 +22,9 @@ __rust_helper void rust_helper_dma_fence_end_signalling(bool cookie)
 	dma_fence_end_signalling(cookie);
 }
 
-__rust_helper bool rust_helper_dma_fence_is_signaled(struct dma_fence *f)
+__rust_helper bool rust_helper_dma_fence_is_signaled_locked(struct dma_fence *f)
 {
-	return dma_fence_is_signaled(f);
+	return dma_fence_is_signaled_locked(f);
 }
 
 __rust_helper bool rust_helper_dma_fence_test_signaled_flag(struct dma_fence *f)
diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index f74119b4d7f8..b0c0e32462ae 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -504,40 +504,49 @@ unsafe impl Send for Fence {}
 // SAFETY: Fences are literally designed to be shared between threads.
 unsafe impl Sync for Fence {}
 
+/// Fence-Status queried through [`Fence::status()`].
+#[derive(Debug)]
+pub enum FenceStatus {
+    /// The fence is not yet signaled.
+    Unsignaled,
+    /// The fence is signaled. Might have been signaled with an error.
+    Signaled(Result),
+}
+
 impl Fence {
-    /// Check whether the fence was signaled at the moment of the function call.
+    /// Query a fence's error code and turn it into a [`Result`].
     ///
-    /// Note that this can return `true` for a [`Fence`] whose [`DriverFence`]
-    /// has not yet been dropped. The reason is that the fence ops callbacks can
-    /// cause the fence to get signaled by the C backend.
-    #[inline]
-    pub fn is_signaled(&self) -> bool {
-        // We should not use `dma_fence_is_signaled_locked()` here, because
-        // according to the C backend's recommendations, that function is
-        // problematic and we should avoid calling that function with a lock
-        // held.
+    /// # Safety
+    ///
+    /// The caller must only call this on fences that are already signaled.
+    unsafe fn __status(&self) -> Result {
+        // SAFETY: Safe by the function's safety requirements.
+        let res = unsafe { (*self.as_raw()).error };
 
-        // SAFETY: Inner `fence` is valid because `self` is valid.
-        let ret = unsafe { bindings::dma_fence_is_signaled(self.as_raw()) };
+        to_result(res)
+    }
 
-        // To be as robust as possible for the future we guarantee that an API
-        // caller can 100% rely on the signaling being completed (i.e., all
-        // fence callbacks ran), so we have to take the lock.
-        //
-        // The reason is that the C dma_fence backend currently does not
-        // carefully synchronize the `dma_fence_is_signaled()` function with the
-        // proper spinlock. This can lead to the function returning `true` while
-        // fence callbacks are still being executed. This can be mitigated by
-        // guarding the entire function with the spinlock.
-        //
-        // The fundamental reason is that the C backend currently does guard
-        // setting of the fence's signaled-bit with the fence's spinlock, but
-        // reading is done locklessly.
-        //
-        // See commit c8a5d5ea3ba6a.
-        let _ = self.lock();
+    /// Query a fence's status at the moment of the function call.
+    ///
+    /// If it is signaled, return the fence's result, i.e., determine if it was
+    /// signaled with an error.
+    pub fn status(&self) -> FenceStatus {
+        // The C backend provides `dma_fence_is_signaled()`. However, this is
+        // currently racy and could lead to misordering with our reading of the
+        // fence error code. Take the lock to prevent that.
+        let fence = self.lock();
+        // SAFETY: `fence` is valid because `self` is valid. The lock was
+        // taken above.
+        let signaled = unsafe { bindings::dma_fence_is_signaled_locked(fence.as_raw()) };
 
-        ret
+        if signaled {
+            // SAFETY: The fence is signaled.
+            let status = unsafe { fence.__status() };
+
+            return FenceStatus::Signaled(status);
+        }
+
+        FenceStatus::Unsignaled
     }
 
     /// Lock the fence. A helper only to be used internally in this module.
@@ -661,6 +670,7 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 ///         FenceContextOps,
 ///         FenceCallback,
 ///         FenceCallbackRegistration,
+///         FenceStatus, //
 ///     },
 ///     str::CString,
 ///     sync::aref::ARef, //
@@ -712,9 +722,15 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 /// let cb_reg = KBox::pin_init(cb_reg, GFP_KERNEL)?;
 ///
 /// // TODO signalling guards
-/// assert_eq!(waiting_fence.is_signaled(), false);
+///
+/// // TODO: add support for `assert_matches!()`.
+/// if let FenceStatus::Signaled(_) = waiting_fence.status() {
+///      return Err(ENOMSG);
+/// }
 /// fence.signal(Ok(()));
-/// assert_eq!(waiting_fence.is_signaled(), true);
+/// if let FenceStatus::Unsignaled = waiting_fence.status() {
+///     return Err(ENOENT);
+/// }
 ///
 /// Ok::<(), Error>(())
 /// ```
diff --git a/rust/kernel/dma_buf/mod.rs b/rust/kernel/dma_buf/mod.rs
index 4764a828642e..b5254144f69e 100644
--- a/rust/kernel/dma_buf/mod.rs
+++ b/rust/kernel/dma_buf/mod.rs
@@ -10,5 +10,6 @@
     FenceCallback,
     FenceCallbackRegistration,
     FenceContext,
-    FenceContextOps, //
+    FenceContextOps,
+    FenceStatus, //
 };
-- 
2.56.0

[PATCH 6/8] rust: DmaFence: Pass result in callbacks

[Philipp Stanner]

A FenceCallbackRegistration invokes the on_signal() trait functions
specified by the user. These callbacks always get executed, even if the
fence was signaled with an error. The user must then be able to react
accordingly.

Since the fence->error must only be set before a fence signals, and must
not change after it signaled, we better acquire it with the lock.

Change the on_signal() trait function so that the implementor always
receives the fence's actual state.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 rust/kernel/dma_buf/dma_fence.rs | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/rust/kernel/dma_buf/dma_fence.rs b/rust/kernel/dma_buf/dma_fence.rs
index b0c0e32462ae..f7f9887b8502 100644
--- a/rust/kernel/dma_buf/dma_fence.rs
+++ b/rust/kernel/dma_buf/dma_fence.rs
@@ -293,9 +293,12 @@ fn from(e: AllocError) -> Self {
 /// }
 ///
 /// impl FenceCallback for MyCallback {
-///     fn on_signal(&mut self) {
-///         pr_info!("Fence signaled!\n");
-///         // Handle fence completion
+///     fn on_signal(&mut self, status: Result) {
+///         if status.is_ok() {
+///             pr_info!("Fence signaled!\n");
+///         } else {
+///             pr_err!("Fence signaled with an error!\n");
+///         }
 ///     }
 /// }
 /// ```
@@ -310,7 +313,7 @@ pub trait FenceCallback: Send {
     /// An implementation likely wants to inform itself (e.g., through a work item)
     /// within this callback that the associated [`FenceCallbackRegistration`]
     /// can now be dropped.
-    fn on_signal(&mut self);
+    fn on_signal(&mut self, status: Result);
 }
 
 /// A callback registration on a fence.
@@ -397,11 +400,15 @@ pub unsafe fn new<'a>(
         // embedded into a `FenceCallbackRegistration`.
         let reg: *mut Self = unsafe { container_of!(ptr, Self, callback_foreign) };
 
+        // SAFETY: `reg` was created validly above. The fence is guaranteed to be
+        // signaled since this here is a fence callback.
+        let fence_result = unsafe { (*reg).fence.__status() };
+
         // SAFETY: `reg` is a valid `Self` pointer.
         //
         // The backend ensures synchronisation so whoever holds the registration object cannot drop
         // it while this code is running. See `FenceCallbackRegistration::drop`.
-        unsafe { (*reg).callback.on_signal() };
+        unsafe { (*reg).callback.on_signal(fence_result) };
     }
 
     /// Returns a reference to the fence this callback is registered on.
@@ -680,8 +687,12 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 /// struct CallbackData { }
 ///
 /// impl FenceCallback for CallbackData {
-///     fn on_signal(&mut self) {
-///         pr_info!("DmaFence callback executed.\n");
+///     fn on_signal(&mut self, status: Result) {
+///         if status.is_ok() {
+///             pr_info!("DmaFence callback executed.\n");
+///         } else {
+///             pr_err!("DmaFence callback executed, but fence has an error.\n");
+///         }
 ///     }
 /// }
 ///
-- 
2.56.0

[PATCH 7/8] rust: drm: Add JobQueue

[Philipp Stanner]

JobQueue is a pure Rust component that handles job submissions for GPUs
with firmware scheduling, i.e., GPUs that support one exclusive ring per
execution context.

JobQueue's primary task is dependency handling, where the user can set
dependencies, each in the form of a `dma_buf::Fence`, on a job and then
submit it to the queue.

Jobs are being executed in order, following the FIFO principle. Only
jobs whose dependencies are all fulfilled get executed, otherwise the
JobQueue pauses.

A core design principle, just like with DmaFence, is to avoid all
allocations in job submission paths, since this runs danger of
deadlocking.

Another design principle, new when compared to the older drafts, is the
ownership relationship with fences:

  1. The JobQueue owns the FenceContext exclusively.
  2. Jobs are created on the JobQueue, ensuring correct fence sequence
     number ordering.
  3. Jobs own the respective DriverFence.
  4. Jobs are owned by the JobQueue.

Consequently, a driver can only interact with fences and FenceContext
indirectly through the JobQueue, and can also only complete jobs (and,
hence, signal fences) through the JobQueue.

Unfortunately, since jobs need a reference to the JobQueue for their
fence dependency objects (DependencyWaker), this code can currently not
be used in this form because some work on self-referential PinInit is
required. See the respective comments in the Example.

When a job shall be run, both its JobQueue::data and Job::data get
borrowed back to it, together with the sequence number of this job, so
that the driver can perform the appropriate interaction with the
hardware.

The submit_worker runs all ready jobs based on a credit count system, so
that run_job() can be infallible. The same worker drops jobs (and their
fences) that are already completed. Should this proof to be a
performance issue, a second worker could trivially be added for this.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
Co-developed-by: Gary Guo <gary@garyguo.net>

(^ waiting for Gary to sign-off)
---
 rust/kernel/drm/job_queue.rs | 873 +++++++++++++++++++++++++++++++++++
 rust/kernel/drm/mod.rs       |   4 +
 2 files changed, 877 insertions(+)
 create mode 100644 rust/kernel/drm/job_queue.rs

diff --git a/rust/kernel/drm/job_queue.rs b/rust/kernel/drm/job_queue.rs
new file mode 100644
index 000000000000..9ddcdda6e4d1
--- /dev/null
+++ b/rust/kernel/drm/job_queue.rs
@@ -0,0 +1,873 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026 Red Hat Inc.
+ * Author: Philipp Stanner <pstanner@redhat.com>
+ */
+
+//! DRM's job dependency manager and load balancer for GPUs with firmware
+//! scheduling, i.e., GPUs that can spawn one ring exclusive per user context.
+//!
+//! JobQueue abstracts the entire handling of [`dma_buf::FenceContext`] and its
+//! associated [`dma_buf::DriverFence`]s for the user.
+//!
+//! In this design, a JobQueue is always the owner of a [`dma_buf::FenceContext`]
+//! which represents the timeline on the associated GPU ring. Jobs created on
+//! this JobQueue always contain a [`dma_buf::DriverFence`] from this JobQueue,
+//! where the fence sequence number corresponds with the job sequence number,
+//! and the fence's generic payload data is also the job's payload data. This
+//! allows for the dma_fence backend_ops callbacks interacting with the driver
+//! through JobQueue. The trait [`JobQueueOps`] is the interface between driver
+//! and fence backend_ops.
+//!
+//! [`Job`]s created on the [`JobQueue`] take over ownership of the user's data
+//! and prepare all memory allocations. A later job submission through
+//! [`Job::submit()`] sets up the fence and performs no fallible operations.
+//!
+//! Before jobs are submitted to a queue, dependencies in the form of
+//! [`dma_buf::Fence`] can be set on them. Jobs are run by the JobQueue in
+//! process context through a work item in FIFO order. Only jobs whose
+//! dependency fences have all been signaled are run. If the front job's
+//! dependency is not yet fulfilled, the JobQueue pauses until that's the case.
+//!
+//! If a dependency fence gets signaled with an error, what happens next depends
+//! on the timing:
+//!
+//! a) If it signals with an error, or was already signaled with one, when one
+//!    tries to set a dependency through [`Job::add_dependency()`], the error is
+//!    returned to that function's caller, who can then decide how to proceed.
+//! b) If it signals afterwards, JobQueue will invoke [`JobQueueOps::kill()`],
+//!    requesting from the driver that it tears down the JobQueue.
+//!
+//! JobQueue implements a credit count system, where the queue itself gets
+//! assigned a capacity by the user, and jobs get assigned individual cost. Jobs
+//! are only run if there is enough capacity; if not, running stops until that
+//! is the case. These credits must accurately represent the capacity on the GPU
+//! ring the queue was created for.
+//!
+//! The driver tells the JobQueue which jobs are finished through
+//! [`JobQueue::complete_jobs_up_to_seqno()`]. This function is typically
+//! invoked through the driver's GPU interrupt handler.
+//!
+//! Since jobs might contain data that must not drop in atomic context, and
+//! since jobs' [`dma_buf::DriverFence`]s perform a call to
+//! [`rcu::synchronize_rcu()`], they are dropped through a work item if [`Drop`]
+//! is implemented for the job's payload data.
+//!
+//! From JobQueue's perspective, it is not necessary to call any of its API
+//! functions with a driver lock held; this is actually discouraged to reduce
+//! the probability of deadlocks. All JobQueue API functions are properly
+//! ordered using the internal lock.
+//!
+//! JobQueue tells drivers to run jobs through [`JobQueueOps::run_job`], which
+//! is called without the JobQueue's spinlock being held, i.e., the driver is
+//! permitted to perform sleeping operations in the callback. JobQueue handles
+//! races between `run_job` and [`JobQueue::complete_jobs_up_to_seqno()`].
+//!
+//! This is pure Rust code that does not directly abstract C.
+
+#![allow(clippy::multiple_bound_locations)] // Clippy false positive, to be addressed by pin-init.
+
+use core::{
+    convert::Infallible,
+    ops::Deref, //
+};
+
+use kernel::{
+    dma_buf::dma_fence::{
+        DriverFence,
+        DriverFenceAllocation,
+        Fence,
+        FenceCallback,
+        FenceCallbackRegistration,
+        FenceContext,
+        FenceContextOps,
+        FenceStatus,
+    },
+    list::{
+        impl_list_arc_safe,
+        impl_list_item,
+        List,
+        ListItem,
+        ListLinks, //
+    },
+    new_srcu,
+    prelude::*,
+    str::CString,
+    sync::{
+        aref::ARef,
+        atomic::{
+            Atomic,
+            Relaxed, //
+        }, //
+        new_spinlock_irq,
+        SpinLockIrq,
+    }, //
+    sync::{
+        atomic::AtomicFlag,
+        Srcu, //
+    },
+    sync::{Arc, UniqueArc},
+    workqueue::{
+        self,
+        new_scoped_work,
+        ScopedWork,
+        ScopedWorkItem,
+        ScopedWorkRef,
+        ScopedQueue, //
+    }, //
+};
+
+/// A linked list where all elements are owned.
+///
+/// This is a thin wrapper of `List` with all elements are `UniqueArc`.
+struct UniqueList<T: ListItem>(List<T>);
+
+impl<T: ListItem> Default for UniqueList<T> {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl<T: ListItem> UniqueList<T> {
+    fn new() -> Self {
+        Self(List::new())
+    }
+
+    fn push_front(&mut self, item: Pin<UniqueArc<T>>) {
+        self.0.push_front(item.into())
+    }
+
+    fn push_back(&mut self, item: Pin<UniqueArc<T>>) {
+        self.0.push_back(item.into())
+    }
+
+    fn pop_front(&mut self) -> Option<Pin<UniqueArc<T>>> {
+        Some(
+            Arc::into_unique_or_drop(self.0.pop_front()?.into_arc())
+                .expect("UniqueList should only contain UniqueArc"),
+        )
+    }
+}
+
+struct DependencyWaker<'b, 'nr_of_deps, T: JobQueueOps + Send + Sync> {
+    nr_of_deps: &'nr_of_deps Atomic<u32>,
+    jq: &'b JobQueue<T>,
+}
+
+impl<'b, T: JobQueueOps + Send + Sync> FenceCallback for DependencyWaker<'b, '_, T> {
+    fn on_signal(&mut self, res: Result) {
+        if let Err(err) = res {
+            T::kill(self.jq.fctx.deref(), self.jq.id(), err);
+            return;
+        }
+
+        // Last dependency kicks the worker.
+        if self.nr_of_deps.fetch_sub(1, Relaxed) == 1 {
+            // The drop-order of JobQueue ensures that all DependencyWakers are
+            // cancelled before the JobQueue itself is dropped.
+            self.jq.check_start_submit_worker();
+        }
+    }
+}
+
+/// A JobQueue job before it has been submitted.
+///
+/// This is only useful for registering dependencies on and for submitting it.
+pub struct Job<'a, T: JobQueueOps + Send + Sync> {
+    inner: Pin<UniqueArc<JobInternal<'a, T>>>,
+    fence_allocation: DriverFenceAllocation<'a, DriverDataWrapper<T>>,
+}
+
+impl<'a, T: JobQueueOps + Send + Sync> Job<'a, T> {
+    /// Add a dependency for the job.
+    ///
+    /// A JobQueue will only run a job once `fence` has been signaled with `Ok(())`.
+    ///
+    /// This action can race with the fence being signaled. The returned Result
+    /// indicates success, obviously. Should an error be returned, this means
+    /// that the dependency fence signaled with an error, meaning the associated
+    /// action will not be completed. It is then up to the caller to decide
+    /// whether it can continue operating on this [`JobQueue`].
+    pub fn add_dependency(&mut self, fence: &Fence) -> Result {
+        self.inner.as_mut().with_project(|job| {
+            // If the registration below gets registered successfully, a signaling
+            // fence will decrement the atomic. That could race. Thus, increment in
+            // advance. Ordering is ensured thanks to the dma_fence backend.
+            job.nr_of_deps.fetch_add(1, Relaxed);
+
+            let dependency = try_pin_init!(Dependency {
+                links <- ListLinks::new(),
+                // SAFETY: The `registration` is not forgotten. It drops with JobQueue.
+                registration <- unsafe {
+                    FenceCallbackRegistration::new(
+                        fence,
+                        DependencyWaker {
+                            nr_of_deps: job.nr_of_deps,
+                            jq: job.jq,
+                        },
+                    )
+                }
+            });
+            match UniqueArc::pin_init(dependency, GFP_KERNEL) {
+                Err(e) => {
+                    let _ = job.nr_of_deps.fetch_sub(1, Relaxed);
+                    if e == ENOENT {
+                        // Was already signaled. Return the fence's state.
+                        match fence.status() {
+                            FenceStatus::Signaled(res) => return res,
+                            FenceStatus::Unsignaled => return Err(ENOENT), // Can never happen.
+                        }
+                    }
+                    return Err(e);
+                }
+                Ok(dep) => {
+                    job.dependencies.push_back(dep);
+                }
+            }
+
+            Ok(())
+        })
+    }
+
+    /// Submit the job.
+    ///
+    /// Returns a [`dma_buf::Fence`] on which the user can register callbacks.
+    /// The fence will be signaled once the job is completed through
+    /// [`JobQueue::complete_jobs_up_to_seqno`], which can happen immediately
+    /// after the job has been submitted.
+    ///
+    /// This function does all fence initialization under the JobQueue lock in
+    /// one go, including setting of the fence sequence number. Thus, it is
+    /// typically unnecessary (and advaised against) to call this function with
+    /// a driver lock held.
+    pub fn submit(self) -> ARef<Fence> {
+        let jq = self.inner.jq;
+        jq.with_inner(|inner| {
+            // SAFETY: this is a lifetime transmute from `Job<'queue, T>` to
+            // `Job<'fctx, T>`.
+            //
+            // `job` internally holds a reference to `JobQueue`, while we are
+            // now moving `job` into `inner`, which may borrow from `fctx` but
+            // not the full `JobQueue`. So on surface this would be incorrect.
+            // However, as we drain `inner` inside `JobQueue::drop`, the
+            // reference is gone before any field is destructed.
+            let mut job = unsafe { core::mem::transmute::<Job<'_, T>, Job<'_, T>>(self) };
+
+            // The lock orders setting of the sequence number, so no driver lock is needed.
+            let mut inner = inner.lock();
+            let fence = job.fence_allocation.new_fence();
+            let ret_fence = ARef::from(fence.as_fence());
+
+            *job.inner.as_mut().project().fence = Some(fence);
+
+            inner.waiting_jobs.push_back(job.inner);
+            drop(inner);
+            jq.check_start_submit_worker();
+
+            ret_fence
+        })
+    }
+}
+
+#[pin_data]
+struct Dependency<'b, 'nr_of_deps, T: JobQueueOps + Send + Sync> {
+    #[pin]
+    links: ListLinks,
+    #[pin]
+    registration: FenceCallbackRegistration<DependencyWaker<'b, 'nr_of_deps, T>>,
+}
+
+impl_list_arc_safe! {
+    impl{'b, T: JobQueueOps + Send + Sync} ListArcSafe<0> for Dependency<'b, '_, T> { untracked; }
+}
+impl_list_item! {
+    impl{'b, T: JobQueueOps + Send + Sync} ListItem<0> for Dependency<'b, '_, T> {
+        using ListLinks { self.links }; }
+}
+
+#[pin_data]
+#[repr(C)]
+struct JobInternal<'b, T: JobQueueOps + Send + Sync> {
+    #[pin]
+    links: ListLinks,
+    cost: u32,
+    fence: Option<DriverFence<'b, DriverDataWrapper<T>>>,
+    dependencies: UniqueList<Dependency<'b, 'nr_of_deps, T>>,
+    nr_of_deps: Atomic<u32>,
+    jq: &'b JobQueue<T>,
+}
+
+impl_list_arc_safe! {
+    impl{'b, T: JobQueueOps + Send + Sync} ListArcSafe<0> for JobInternal<'b, T> { untracked; }
+}
+impl_list_item! {
+    impl{'b, T: JobQueueOps + Send + Sync} ListItem<0> for JobInternal<'b, T> {
+        using ListLinks { self.links }; }
+}
+
+struct SubmitWorker<'a, 'fctx, T: JobQueueOps + Send + Sync> {
+    srcu: &'a Srcu,
+    revoked: &'a AtomicFlag,
+    inner: &'a SpinLockIrq<JobQueueInner<'fctx, T>>,
+    fctx: &'fctx FenceContext<DriverDataWrapper<T>>,
+}
+
+impl<T: JobQueueOps + Send + Sync> ScopedWorkItem for SubmitWorker<'_, '_, T> {
+    fn run(work: &ScopedWorkRef<Self>) {
+        // Synchronize against revocation in `JobQueue::drop`.
+        let _lock_guard = work.srcu.read_lock();
+        if work.revoked.load(Relaxed) {
+            return;
+        }
+
+        let mut jq = work.inner.lock();
+
+        while let Some(job) = jq.waiting_jobs.pop_front() {
+            // While we didn't hold the lock below for `run_job()`, it could be
+            // that a new job entered the waiting list which has an unfulfilled
+            // dependency.
+            if job.nr_of_deps.load(Relaxed) > 0 {
+                jq.waiting_jobs.push_front(job);
+                break;
+            }
+            let new_capacity = jq.capacity as i64 - i64::from(job.cost);
+            if new_capacity < 0 {
+                jq.waiting_jobs.push_front(job);
+                break;
+            }
+            *jq.as_mut().project().capacity = new_capacity as u64;
+
+            let fence = job.fence.as_ref().expect("fence not present");
+            let seqno = fence.as_fence().seqno();
+            let runnable_job = JobRunnable {
+                seqno,
+                data: fence.deref(),
+            };
+
+            drop(jq); // Release lock.
+            T::run_job(work.fctx, runnable_job);
+            jq = work.inner.lock(); // Retake lock.
+
+            // run_job() above needs to be called without lock protection. Thus,
+            // it is possible that the driver instructed completion right before
+            // we could re-take the lock.
+            if jq.last_completed >= seqno {
+                *jq.as_mut().project().capacity += u64::from(job.cost);
+                fence.signal(Ok(()));
+
+                // Drop here directly if possible. Otherwise, let the DropWorker
+                // take care of it.
+                if core::mem::needs_drop::<T::JobDataType>() {
+                    jq.drop_list.push_back(job);
+                }
+            } else {
+                jq.running_jobs.push_back(job);
+            }
+        }
+    }
+}
+
+struct DropWorker<'a, 'fctx, T: JobQueueOps + Send + Sync> {
+    srcu: &'a Srcu,
+    revoked: &'a AtomicFlag,
+    inner: &'a SpinLockIrq<JobQueueInner<'fctx, T>>,
+}
+
+impl<T: JobQueueOps + Send + Sync> ScopedWorkItem for DropWorker<'_, '_, T> {
+    fn run(work: &ScopedWorkRef<Self>) {
+        // Synchronize against revocation in `JobQueue::drop`.
+        let _lock_guard = work.srcu.read_lock();
+        if work.revoked.load(Relaxed) {
+            return;
+        }
+
+        let mut jq_locked = work.inner.lock();
+
+        while let Some(job) = jq_locked.drop_list.pop_front() {
+            drop(jq_locked);
+            drop(job);
+            jq_locked = work.inner.lock();
+        }
+    }
+}
+
+#[pin_data]
+struct JobQueueInner<'a, T: JobQueueOps + Send + Sync> {
+    #[pin]
+    /// List of jobs waiting to be run on the hardware.
+    waiting_jobs: UniqueList<JobInternal<'a, T>>,
+    #[pin]
+    /// List of jobs that are currently executing on the hardware.
+    running_jobs: UniqueList<JobInternal<'a, T>>,
+    /// The queue's current capacity.
+    capacity: u64,
+    /// The sequence number of the last completed job.
+    last_completed: u64,
+    #[pin]
+    /// Jobs that have finished executing and whose payload data `T` implements
+    /// `Drop`. These jobs are dropped in process context, which also allows for
+    /// their [`dma_buf::DriverFence`]s to implement the necessary RCU grace
+    /// period delay.
+    ///
+    /// If `Drop` is not implemented, this list is unused and the elements drop
+    /// directly in [`JobQueue::complete_jobs_up_to_seqno`].
+    ///
+    /// Should the JobQueue drop and the [`DropWorker`] not yet have dropped all
+    /// jobs, that is no problem because the JobQueue itself drops in process
+    /// context.
+    drop_list: UniqueList<JobInternal<'a, T>>,
+}
+
+/// A JobQueue.
+///
+/// When you drop your JobQueue, it might still attempt for one RCU grace period
+/// to submit more jobs via the [`JobQueueOps::run_job`] callback into your driver.
+///
+/// Dropping a JobQueue results in all remaining waiting jobs'
+/// fences being signaled with ECANCELED. All remaining running jobs' fences get
+/// signaled with the error you provided in [`JobQueue::set_error()`], or
+/// ECANCELED otherwise.
+///
+/// A JobQueue must only ever drop in non-atomic / process context.
+///
+/// # Examples
+///
+/// ```
+/// use core::ops::Deref;
+///
+/// use kernel::{
+///     str::CString,
+///     dma_buf::*,
+///     drm::{
+///         JobQueue,
+///         JobQueueOps,
+///         JobRunnable,
+///     },
+/// };
+///
+/// let driver_name = CString::try_from_fmt(fmt!("dummy_driver"))?;
+/// let timeline_name = CString::try_from_fmt(fmt!("dummy_timeline"))?;
+///
+/// #[pin_data]
+/// struct FenceContextData {}
+///
+/// impl FenceContextData {
+///     fn new() -> impl PinInit<Self> {
+///         pin_init!(Self {})
+///     }
+/// }
+///
+/// impl FenceContextOps for FenceContextData {
+///     type FenceDataType = ();
+/// }
+///
+/// let fctx_data = FenceContextData::new();
+///
+/// let fctx = KBox::pin_init(
+///     FenceContext::new(0, c"dependency_driver", c"dependency_timeline", fctx_data),
+///     GFP_KERNEL
+/// )?;
+///
+/// let dependency1 = fctx.new_fence_allocation(())?.new_fence();
+/// let dependency2 = fctx.new_fence_allocation(())?.new_fence();
+///
+/// #[pin_data]
+/// struct JqData {}
+///
+/// impl JqData {
+///     fn new() -> impl PinInit<Self> {
+///         pin_init!(Self {})
+///     }
+/// }
+///
+/// struct Payload { }
+///
+/// impl JobQueueOps for JqData {
+///     type JobDataType = Payload;
+///
+///     fn run_job<'a>(&self, job: JobRunnable<'a, Payload>) {
+///         pr_info!("Job Nr. {} gets run now.\n", job.seqno);
+///     }
+///
+///     fn kill(&self, jq_nr: u64, err: Error) {
+///         pr_err!("The JQ Nr. {} became blocked forever because deps are blocked with err: {:?}\n",
+///                 jq_nr, err);
+///
+///         // Here we would now schedule a driver work-item which invokes:
+///         // `job_queue.set_error(err)` and then `drop(job_queue)`.
+///         //
+///         // We can't drop here right away because `kill()` is called with a
+///         // fence lock held.
+///     }
+/// }
+///
+/// let jq_data = JqData::new();
+/// let jq = KBox::pin_init(
+///     JobQueue::new(driver_name, timeline_name, 1000, jq_data),
+///     GFP_KERNEL
+/// )?;
+///
+///
+/// let mut job1 = jq.new_job(1, Payload { })?;
+/// let mut job2 = jq.new_job(1, Payload { })?;
+///
+/// job1.add_dependency(dependency1.as_fence())?;
+/// job1.add_dependency(dependency2.as_fence())?;
+///
+/// job2.add_dependency(dependency2.as_fence())?;
+///
+/// struct Callback {
+///     seqno: u64,
+/// }
+///
+/// impl FenceCallback for Callback {
+///     fn on_signal(&mut self, res: Result) {
+///         pr_info!("Job Nr. {} got completed.\n", self.seqno);
+///     }
+/// }
+///
+/// let done_fence1 = job1.submit();
+/// let done_fence2 = job2.submit();
+///
+/// let cb = Callback { seqno: done_fence2.seqno() };
+/// // SAFETY: The registration is not leaked.
+/// let reg = unsafe { FenceCallbackRegistration::new(&done_fence2, cb) };
+/// let reg = KBox::new(reg, GFP_KERNEL)?;
+///
+/// dependency1.signal(Ok(()));
+/// dependency2.signal(Ok(()));
+///
+/// Ok::<(), Error>(())
+/// ```
+#[pin_data(PinnedDrop)]
+pub struct JobQueue<T: JobQueueOps + Send + Sync> {
+    /// The queue's maximum available capacity.
+    max_capacity: u64,
+    #[pin]
+    #[uses('_: invariant)]
+    /// The work queue on which the JobQueue drops the user's payload data.
+    drop_queue: ScopedQueue<'inner>,
+    #[pin]
+    #[uses('_: invariant, 'revoked: invariant, 'srcu: invariant)]
+    /// A worker for dropping the queue's jobs.
+    drop_worker: ScopedWork<DropWorker<'inner, 'fctx, T>>,
+    #[pin]
+    #[uses('_: invariant)]
+    /// The work queue on which the JobQueue runs jobs ([`JobQueueOps::run_job`]).
+    submit_queue: ScopedQueue<'submit_worker>,
+    #[pin]
+    #[uses('_: invariant)]
+    /// The worker submitting the queue's ready jobs.
+    submit_worker: ScopedWork<SubmitWorker<'inner, 'fctx, T>>,
+    #[pin]
+    #[uses('fctx: invariant)]
+    /// The inner queue.
+    inner: SpinLockIrq<JobQueueInner<'fctx, T>>,
+    #[pin]
+    /// The [`dma_buf::FenceContext`] on which the queue creates
+    /// [`dma_buf::DriverFence`]s for the jobs.
+    fctx: FenceContext<DriverDataWrapper<T>>,
+    /// A mechanism for preventing rescheduling of work items when the queue
+    /// drops.
+    revoked: AtomicFlag,
+    #[pin]
+    /// Needed for the `revoked` mechanism, since [`JobQueueOps::run_job`] might
+    /// sleep.
+    srcu: Srcu,
+    /// The error with which the JobQueue will signal still running jobs when it
+    /// drops.
+    err: Error,
+}
+
+// SAFETY: The JobQueue is locked and pinned.
+unsafe impl<T: JobQueueOps + Send + Sync> Sync for JobQueue<T> {}
+// SAFETY: The JobQueue is locked and pinned.
+unsafe impl<T: JobQueueOps + Send + Sync> Send for JobQueue<T> {}
+
+/// Object passed to the driver to run a job.
+pub struct JobRunnable<'a, T> {
+    /// The job's sequence number.
+    pub seqno: u64,
+    /// The associated job data.
+    pub data: &'a T,
+}
+
+/// Ops for the JobQueue.
+pub trait JobQueueOps {
+    /// The type of the data passed over in [`JobQueue::new_job`].
+    type JobDataType: Send + Sync;
+
+    /// Instructs the driver to run this job.
+    ///
+    /// JobQueue calls this callback from process context without any locks
+    /// held, thus, the driver is permitted to sleep in its implementation.
+    ///
+    /// The driver must be aware that JobQueue trusts on its credit count system
+    /// for this operation to be infallible.
+    ///
+    /// This callback might be called while the driver is in the process of
+    /// dropping the JobQueue, but will never be called afterwards.
+    fn run_job<'a>(&self, job: JobRunnable<'a, Self::JobDataType>);
+
+    /// Tells the driver to tear this JobQueue down.
+    ///
+    /// The JobQueue backend invokes this in case a job's dependency got
+    /// signaled with an error. This means that the dependency's associated
+    /// action on a GPU ring has failed and will not be recovered.
+    ///
+    /// Thus, it is necessary to tear this JobQueue down, too. Upon receiving
+    /// this ops, the driver should schedule a work_item in which it passes
+    /// through `err` with [`JobQueue::set_error()`] and then drops the
+    /// JobQueue. The JobQueue is robust against the driver continuing to call
+    /// [`JobQueue::submit()`], but these jobs will never run and will
+    /// ultimately be canceled with ECANCELED once tho JobQueue drops.
+    ///
+    /// The JobQueue might continue calling [`JobQueueOps::run_job()`] for jobs
+    /// with fulfilled dependencies, even after `kill()` was called.
+    ///
+    /// This ops is invoked in atomic context. Hence, you should schedule a
+    /// JobQueue teardown through an asynchronous path.
+    ///
+    /// The backend might call this function multiple times until the driver has
+    /// dropped the Jobqueue.
+    fn kill(&self, jq_id: u64, err: Error);
+}
+
+#[pin_data]
+struct DriverDataWrapper<T: JobQueueOps + Send + Sync> {
+    #[pin]
+    data: T,
+}
+
+impl<T: JobQueueOps + Send + Sync> Deref for DriverDataWrapper<T> {
+    type Target = T;
+
+    fn deref(&self) -> &Self::Target {
+        &self.data
+    }
+}
+
+impl<T: JobQueueOps + Send + Sync> DriverDataWrapper<T> {
+    fn new<E>(data: impl PinInit<T, E>) -> impl PinInit<Self, E>
+    where
+        Error: From<E>,
+        E: From<Infallible>,
+    {
+        try_pin_init!(Self {
+            data <- data,
+        }? E)
+    }
+}
+
+impl<T: JobQueueOps + Send + Sync> FenceContextOps for DriverDataWrapper<T> {
+    type FenceDataType = T::JobDataType;
+}
+
+impl<T: JobQueueOps + Send + Sync> Deref for JobQueue<T> {
+    type Target = T;
+
+    fn deref(&self) -> &Self::Target {
+        &self.fctx.deref()
+    }
+}
+impl<T: JobQueueOps + Send + Sync> JobQueue<T> {
+    /// Create a new JobQueue.
+    pub fn new<E>(
+        // TODO: pass these as &CStr. Figure out how to get the lifetimes for PinInit right.
+        driver_name: CString,
+        timeline_name: CString,
+        capacity_param: u64,
+        data: impl PinInit<T, E>,
+    ) -> impl PinInit<Self, Error>
+    where
+        Error: From<E>,
+        E: From<Infallible>,
+    {
+        try_pin_init!(Self {
+            srcu <- new_srcu!(),
+            revoked: AtomicFlag::new(false),
+            max_capacity: capacity_param,
+            err: ECANCELED,
+            // The initial sequence number always starts at 1, since hardware is
+            // initialized to 0. Also makes internal logic easier.
+            fctx <- FenceContext::new(1, &driver_name, &timeline_name,
+                                      DriverDataWrapper::new(data)),
+            inner <- new_spinlock_irq!(try_pin_init!(JobQueueInner {
+                capacity: capacity_param,
+                last_completed: 0,
+                waiting_jobs: UniqueList::new(),
+                running_jobs: UniqueList::new(),
+                drop_list: UniqueList::new(),
+            })),
+            // TODO: We need to set the WQ_MEM_RECLAIM bit!
+            // SAFETY: The queue is not leaked; its lifetime is tied to JobQueue.
+            submit_queue <- unsafe { ScopedQueue::new(c"JobQueueSubmitQueue") },
+            submit_worker <- new_scoped_work!("JobQueueSubmitWorker", Ok(SubmitWorker {
+                srcu: Pin::get_ref(srcu),
+                revoked,
+                inner: Pin::get_ref(inner),
+                fctx: Pin::get_ref(fctx)
+            })),
+            // SAFETY: The queue is not leaked; its lifetime is tied to JobQueue.
+            drop_queue <- unsafe { ScopedQueue::new(c"JobQueueDropQueue") },
+            drop_worker <- new_scoped_work!("JobQueueDropWorker", Ok(DropWorker {
+                srcu: Pin::get_ref(srcu),
+                revoked,
+                inner: Pin::get_ref(inner)
+            })),
+        })
+    }
+
+    /// Give the unique identifier for this JobQueue.
+    ///
+    /// This corresponds to the JobQueue's underlying [`dma_buf::FenceContext`].
+    /// It is unique for all JobQueues in the entire system.
+    pub fn id(&self) -> u64 {
+        self.fctx.id()
+    }
+
+    /// Create a new job.
+    ///
+    /// The `cost` specified here is what JobQueue subtracts from its capacity
+    /// to determine whether a job is runnable, i.e., has space on the GPU ring.
+    ///
+    /// The `data` contained by a job will be borrowed back to the driver once
+    /// the jobs shall be run through [`JobQueueOps::run_job`].
+    pub fn new_job<'a>(&'a self, cost: u32, data: T::JobDataType) -> Result<Job<'a, T>> {
+        if u64::from(cost) > self.max_capacity {
+            return Err(EINVAL);
+        }
+
+        let fence_allocation = self.fctx.new_fence_allocation(data)?;
+        let job = try_pin_init!(JobInternal {
+            links <- ListLinks::new(),
+            cost,
+            fence: None,
+            nr_of_deps: Atomic::new(0),
+            dependencies: UniqueList::new(),
+            jq: &self,
+        });
+
+        let inner = UniqueArc::pin_init(job, GFP_KERNEL)?;
+
+        Ok(Job {
+            inner,
+            fence_allocation,
+        })
+    }
+
+    fn check_start_submit_worker(&self) {
+        self.with_submit_worker(|submit_worker| {
+            // TODO this shouldn't be the system wq
+            // SAFETY: The worker is carried by the JobQueue and, thus, cannot be
+            // forgotten.
+            let _ = unsafe { workqueue::system_dfl().enqueue_scoped(submit_worker) };
+        })
+    }
+
+    fn check_start_drop_worker(&self) {
+        self.with_drop_worker(|drop_worker| {
+            // TODO this shouldn't be the system wq
+            // Use the long-running wq for now, since here all the RCU delays manifest.
+            // SAFETY: The worker is carried by the JobQueue and, thus, cannot be
+            // forgotten.
+            let _ = unsafe { workqueue::system_long().enqueue_scoped(drop_worker) };
+        })
+    }
+
+    /// Complete all the jobs in the queue's running_list, up to `seqno`.
+    ///
+    /// This signals the jobs' fences and schedules dropping of both the [`Job`]
+    /// and its [`DriverFence`].
+    ///
+    /// This signals the associated fences with the `Ok(())` status.
+    ///
+    /// This function tells the implementation that there is now more capacity
+    /// for more jobs to be run.
+    pub fn complete_jobs_up_to_seqno(&self, seqno: u64) {
+        self.with_inner(|inner| {
+            let needs_drop = core::mem::needs_drop::<T::JobDataType>();
+            let mut jq_locked = inner.lock();
+            let mut jq = jq_locked.as_mut().project();
+
+            if seqno < *jq.last_completed {
+                return;
+            }
+
+            while let Some(mut job) = jq.running_jobs.pop_front() {
+                if job
+                    .as_mut()
+                    .project()
+                    .fence
+                    .as_mut()
+                    .expect("fence not present")
+                    .as_fence()
+                    .seqno()
+                    > seqno
+                {
+                    jq.running_jobs.push_front(job);
+                    break;
+                }
+
+                *jq.capacity += u64::from(job.cost);
+
+                let fence = job.fence.as_ref().expect("fence not present");
+                fence.signal(Ok(()));
+
+                if needs_drop {
+                    jq.drop_list.push_back(job);
+                }
+                // Otherwise, job is dropped here directly. Its fence won't perform
+                // an RCU grace period wait.
+            }
+
+            // This enables the SubmitWorker to complete a job that is not yet
+            // in 'running_jobs', in case of a race against the lock.
+            *jq.last_completed = seqno;
+            drop(jq_locked);
+
+            self.check_start_submit_worker();
+            if needs_drop {
+                self.check_start_drop_worker();
+            }
+        })
+    }
+
+    /// Set the error with which this JobQueue will signal all running jobs'
+    /// fences. (The jobs on the waiting_list will all complete with ECANCELED
+    /// as their error)
+    ///
+    /// You should call this in the worker you kick in [`JobQueueOps::kill()`].
+    ///
+    /// You can also call it when you yourself decide you want to drop this
+    /// JobQueue in another error path.
+    pub fn set_error(&mut self, error: Error) {
+        self.err = error;
+    }
+}
+
+#[pinned_drop]
+impl<T: JobQueueOps + Send + Sync> PinnedDrop for JobQueue<T> {
+    fn drop(self: Pin<&mut Self>) {
+        // Revoke to prevent still existing DependencyWakers from kicking the
+        // SubmitWorker again before we could deregister them.
+        self.revoked.store(true, Relaxed);
+        self.srcu.synchronize();
+        self.with_inner(|inner| {
+            let mut inner = inner.lock();
+            let drop_list = core::mem::take(&mut inner.drop_list);
+            let waiting_jobs = core::mem::take(&mut inner.waiting_jobs);
+            let mut running_jobs = core::mem::take(&mut inner.running_jobs);
+
+            while let Some(job) = running_jobs.pop_front() {
+                let fence = job.fence.as_ref().expect("fence not present");
+                fence.signal(Err(self.err));
+            }
+
+            drop(inner);
+            drop(drop_list);
+            drop(waiting_jobs);
+            drop(running_jobs);
+        })
+    }
+}
diff --git a/rust/kernel/drm/mod.rs b/rust/kernel/drm/mod.rs
index fd6ed35bc35a..f2b7fbbce5bb 100644
--- a/rust/kernel/drm/mod.rs
+++ b/rust/kernel/drm/mod.rs
@@ -8,6 +8,7 @@
 pub mod gem;
 pub mod gpuvm;
 pub mod ioctl;
+pub mod job_queue;
 
 pub use self::device::Device;
 pub use self::device::DeviceContext;
@@ -20,6 +21,9 @@
 pub use self::driver::DriverInfo;
 pub use self::driver::Registration;
 pub use self::file::File;
+pub use self::job_queue::JobQueue;
+pub use self::job_queue::JobQueueOps;
+pub use self::job_queue::JobRunnable;
 
 pub(crate) mod private {
     pub trait Sealed {}
-- 
2.56.0

[PATCH 8/8] MAINTAINERS: Add drm::JobQueue to drm_sched's entry

[Philipp Stanner]

The DRM subsystem now provides drm::JobQueue, a Rust dependency manager
and load balancer for GPUs with firmware scheduling. Since it resides in
the same problem space as drm_sched, it is reasonable to maintain it
through a common entry.

Signed-off-by: Philipp Stanner <phasta@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 19818be10616..5cb896c4e0eb 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9267,6 +9267,7 @@ T:	git https://gitlab.freedesktop.org/drm/misc/kernel.git
 F:	drivers/gpu/drm/scheduler/
 F:	include/drm/gpu_scheduler.h
 F:	include/drm/spsc_queue.h
+F:	rust/kernel/drm/job_queue.rs
 
 DRM GPUVM
 M:	Danilo Krummrich <dakr@kernel.org>
-- 
2.56.0