From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com [136.143.188.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 159F51CAA79; Tue, 17 Mar 2026 02:47:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=136.143.188.112 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773715683; cv=pass; b=C0P8i4miRU6gwcUigWH3m3NCx9sfAyX0P30Ef7W+HXiwpQhxjTJBQX0zcWV2Lo86ov1LGHgPPm//12mOZWxyt3RZOHtZYFGRjgtYo/VDuXVRjKIHXIcoGw8hMXtpZtEdexif3KMd65wxY4V0bG3PAvDsAvoiVQon8/6LrNFxAQ8= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773715683; c=relaxed/simple; bh=QTYtaNAIoMct4H0Cik2hxdZkoe750KsM24ZyPghqH9s=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc: Message-Id:References:To; b=NC3zzH+7xh0G9mOFe/Iq8eKzZtcL+qDk3cWgzqAY2gU0dKdV+2hnq3e+p7JwGLYWss38U95OlGFHmsNdoOYDalS4tNBH8ASo/g9h5ZrgqxgPHI2hONgBpdqrm/PHvYmQDq95kZ7fDZ7dznwGr/c1++aIQ7oxJd9sy7Js/JVrGWk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (1024-bit key) header.d=collabora.com header.i=daniel.almeida@collabora.com header.b=VwNEK5+K; arc=pass smtp.client-ip=136.143.188.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=collabora.com header.i=daniel.almeida@collabora.com header.b="VwNEK5+K" ARC-Seal: i=1; a=rsa-sha256; t=1773715645; cv=none; d=zohomail.com; s=zohoarc; b=S869bDSCk/IGzdB+MuDVCZciWzw6+PL/jN2JgnBTY7+CI/U75l6nh+RMs6kpqESVzZ+VMicMQMFwO2+EncR+20jNXm4P1z4b/iLTTfpbJFYL3XN1JrJ07IhlC8NrZ0koWG2sRALGhjAL88NyPZd09gYgd2b0lKyAF9X4+GlVeUw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1773715645; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=H+dijfyzrxgIH4a91MJfU51qDr7GqJblxLLFn1v3uUI=; b=ZEhIVawqYPk8/5Wrsbi1rHCtYsZ+mZLiYjTh3tE/Mw49hbzbhD7+o8pelPdPJ1caXChjQSqIXHKqTd+tI3d9i2RTa3VUDKJRnfs6sjRP3lExYRXvXq53WNAOqtay3MBkq0caF3UmhO7kQcSmRZ8pLj0JZx5jqvQVsr3F/rtrB0Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=daniel.almeida@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1773715645; s=zohomail; d=collabora.com; i=daniel.almeida@collabora.com; h=Content-Type:Mime-Version:Subject:Subject:From:From:In-Reply-To:Date:Date:Cc:Cc:Content-Transfer-Encoding:Message-Id:Message-Id:References:To:To:Reply-To; bh=H+dijfyzrxgIH4a91MJfU51qDr7GqJblxLLFn1v3uUI=; b=VwNEK5+Kd8LEpfs54/YPD9/mKtF76tfFKx4/fAsSG5o7QWFKonr1ixboz6Sn/4Xo byldXLCmZQabRJdNOieaVtVpEYT6a09IgSR/d45iHjZ8BX4Irr6DMiyfNFeDS6Y13ri GKZPn3Z6Asjnb6bRTOUJLhdOBlL8TSZrq87GsHiI= Received: by mx.zohomail.com with SMTPS id 177371564229741.90346342361158; Mon, 16 Mar 2026 19:47:22 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer From: Daniel Almeida In-Reply-To: <20260316043255.226352-3-matthew.brost@intel.com> Date: Mon, 16 Mar 2026 23:47:01 -0300 Cc: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, Boris Brezillon , Tvrtko Ursulin , Rodrigo Vivi , =?utf-8?Q?Thomas_Hellstr=C3=B6m?= , =?utf-8?Q?Christian_K=C3=B6nig?= , Danilo Krummrich , David Airlie , Maarten Lankhorst , Maxime Ripard , Philipp Stanner , Simona Vetter , Sumit Semwal , Thomas Zimmermann , linux-kernel@vger.kernel.org, Sami Tolvanen , Jeffrey Vander Stoep , Alice Ryhl , Daniel Stone , Alexandre Courbot , John Hubbard , shashanks@nvidia.com, jajones@nvidia.com, Eliot Courtney , Joel Fernandes , rust-for-linux Content-Transfer-Encoding: quoted-printable Message-Id: <7A8108C7-7CF0-4EA4-95ED-8003502DC35A@collabora.com> References: <20260316043255.226352-1-matthew.brost@intel.com> <20260316043255.226352-3-matthew.brost@intel.com> To: Matthew Brost X-Mailer: Apple Mail (2.3826.700.81) X-ZohoMailClient: External (+cc a few other people + Rust-for-Linux ML) Hi Matthew, I agree with what Danilo said below, i.e.: IMHO, with the direction = that DRM is going, it is much more ergonomic to add a Rust component with a nice = C interface than doing it the other way around. > On 16 Mar 2026, at 01:32, Matthew Brost = wrote: >=20 > Diverging requirements between GPU drivers using firmware scheduling > and those using hardware scheduling have shown that drm_gpu_scheduler = is > no longer sufficient for firmware-scheduled GPU drivers. The technical > debt, lack of memory-safety guarantees, absence of clear = object-lifetime > rules, and numerous driver-specific hacks have rendered > drm_gpu_scheduler unmaintainable. It is time for a fresh design for > firmware-scheduled GPU drivers=E2=80=94one that addresses all of the > aforementioned shortcomings. >=20 > Add drm_dep, a lightweight GPU submission queue intended as a > replacement for drm_gpu_scheduler for firmware-managed GPU schedulers > (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlike > drm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) > from the queue (drm_sched_entity) into two objects requiring external > coordination, drm_dep merges both roles into a single struct > drm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping > that is unnecessary for firmware schedulers which manage their own > run-lists internally. >=20 > Unlike drm_gpu_scheduler, which relies on external locking and = lifetime > management by the driver, drm_dep uses reference counting (kref) on = both > queues and jobs to guarantee object lifetime safety. A job holds a = queue In a domain that has been plagued by lifetime issues, we really should = be enforcing RAII for resource management instead of manual calls. > reference from init until its last put, and the queue holds a job = reference > from dispatch until the put_job worker runs. This makes use-after-free > impossible even when completion arrives from IRQ context or concurrent > teardown is in flight. It makes use-after-free impossible _if_ you=E2=80=99re careful. It is = not a property of the type system, and incorrect code will compile just fine. >=20 > The core objects are: >=20 > struct drm_dep_queue - a per-context submission queue owning an > ordered submit workqueue, a TDR timeout workqueue, an SPSC job > queue, and a pending-job list. Reference counted; drivers can embed > it and provide a .release vfunc for RCU-safe teardown. >=20 > struct drm_dep_job - a single unit of GPU work. Drivers embed this > and provide a .release vfunc. Jobs carry an xarray of input > dma_fence dependencies and produce a drm_dep_fence as their > finished fence. >=20 > struct drm_dep_fence - a dma_fence subclass wrapping an optional > parent hardware fence. The finished fence is armed (sequence > number assigned) before submission and signals when the hardware > fence signals (or immediately on synchronous completion). >=20 > Job lifecycle: > 1. drm_dep_job_init() - allocate and initialise; job acquires a > queue reference. > 2. drm_dep_job_add_dependency() and friends - register input fences; > duplicates from the same context are deduplicated. > 3. drm_dep_job_arm() - assign sequence number, obtain finished fence. > 4. drm_dep_job_push() - submit to queue. You cannot enforce this sequence easily in C code. Once again, we are = trusting drivers that it is followed, but in Rust, you can simply reject code = that does not follow this order at compile time. >=20 > Submission paths under queue lock: > - Bypass path: if DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the > SPSC queue is empty, no dependencies are pending, and credits are > available, the job is dispatched inline on the calling thread. > - Queued path: job is pushed onto the SPSC queue and the run_job > worker is kicked. The worker resolves remaining dependencies > (installing wakeup callbacks for unresolved fences) before calling > ops->run_job(). >=20 > Credit-based throttling prevents hardware overflow: each job declares > a credit cost at init time; dispatch is deferred until sufficient > credits are available. Why can=E2=80=99t we design an API where the driver can refuse jobs in ops->run_job() if there are no resources to run it? This would do away = with the credit system that has been in place for quite a while. Has this = approach been tried in the past? >=20 > Timeout Detection and Recovery (TDR): a per-queue delayed work item > fires when the head pending job exceeds q->job.timeout jiffies, = calling > ops->timedout_job(). drm_dep_queue_trigger_timeout() forces immediate > expiry for device teardown. >=20 > IRQ-safe completion: queues flagged = DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE > allow drm_dep_job_done() to be called from hardirq context (e.g. a > dma_fence callback). Dependency cleanup is deferred to process context > after ops->run_job() returns to avoid calling xa_destroy() from IRQ. >=20 > Zombie-state guard: workers use kref_get_unless_zero() on entry and > bail immediately if the queue refcount has already reached zero and > async teardown is in flight, preventing use-after-free. In rust, when you queue work, you have to pass a reference-counted = pointer (Arc). We simply never have this problem in a Rust design. If there = is work queued, the queue is alive. By the way, why can=E2=80=99t we simply require synchronous teardowns? >=20 > Teardown is always deferred to a module-private workqueue = (dep_free_wq) > so that destroy_workqueue() is never called from within one of the > queue's own workers. Each queue holds a drm_dev_get() reference on its > owning struct drm_device, released as the final step of teardown via > drm_dev_put(). This prevents the driver module from being unloaded > while any queue is still alive without requiring a separate drain API. >=20 > Cc: Boris Brezillon > Cc: Tvrtko Ursulin > Cc: Rodrigo Vivi > Cc: Thomas Hellstr=C3=B6m > Cc: Christian K=C3=B6nig > Cc: Danilo Krummrich > Cc: David Airlie > Cc: dri-devel@lists.freedesktop.org > Cc: Maarten Lankhorst > Cc: Maxime Ripard > Cc: Philipp Stanner > Cc: Simona Vetter > Cc: Sumit Semwal > Cc: Thomas Zimmermann > Cc: linux-kernel@vger.kernel.org > Signed-off-by: Matthew Brost > Assisted-by: GitHub Copilot:claude-sonnet-4.6 > --- > drivers/gpu/drm/Kconfig | 4 + > drivers/gpu/drm/Makefile | 1 + > drivers/gpu/drm/dep/Makefile | 5 + > drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++++ > drivers/gpu/drm/dep/drm_dep_fence.h | 25 + > drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++++++ > drivers/gpu/drm/dep/drm_dep_job.h | 13 + > drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++ > drivers/gpu/drm/dep/drm_dep_queue.h | 31 + > include/drm/drm_dep.h | 597 ++++++++++ > 10 files changed, 3404 insertions(+) > create mode 100644 drivers/gpu/drm/dep/Makefile > create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c > create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h > create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c > create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h > create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c > create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h > create mode 100644 include/drm/drm_dep.h >=20 > diff --git a/drivers/gpu/drm/Kconfig b/drivers/gpu/drm/Kconfig > index 5386248e75b6..834f6e210551 100644 > --- a/drivers/gpu/drm/Kconfig > +++ b/drivers/gpu/drm/Kconfig > @@ -276,6 +276,10 @@ config DRM_SCHED > tristate > depends on DRM >=20 > +config DRM_DEP > + tristate > + depends on DRM > + > # Separate option as not all DRM drivers use it > config DRM_PANEL_BACKLIGHT_QUIRKS > tristate > diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile > index e97faabcd783..1ad87cc0e545 100644 > --- a/drivers/gpu/drm/Makefile > +++ b/drivers/gpu/drm/Makefile > @@ -173,6 +173,7 @@ obj-y +=3D clients/ > obj-y +=3D display/ > obj-$(CONFIG_DRM_TTM) +=3D ttm/ > obj-$(CONFIG_DRM_SCHED) +=3D scheduler/ > +obj-$(CONFIG_DRM_DEP) +=3D dep/ > obj-$(CONFIG_DRM_RADEON)+=3D radeon/ > obj-$(CONFIG_DRM_AMDGPU)+=3D amd/amdgpu/ > obj-$(CONFIG_DRM_AMDGPU)+=3D amd/amdxcp/ > diff --git a/drivers/gpu/drm/dep/Makefile = b/drivers/gpu/drm/dep/Makefile > new file mode 100644 > index 000000000000..335f1af46a7b > --- /dev/null > +++ b/drivers/gpu/drm/dep/Makefile > @@ -0,0 +1,5 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +drm_dep-y :=3D drm_dep_queue.o drm_dep_job.o drm_dep_fence.o > + > +obj-$(CONFIG_DRM_DEP) +=3D drm_dep.o > diff --git a/drivers/gpu/drm/dep/drm_dep_fence.c = b/drivers/gpu/drm/dep/drm_dep_fence.c > new file mode 100644 > index 000000000000..ae05b9077772 > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_fence.c > @@ -0,0 +1,406 @@ > +// SPDX-License-Identifier: MIT > +/* > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +/** > + * DOC: DRM dependency fence > + * > + * Each struct drm_dep_job has an associated struct drm_dep_fence = that > + * provides a single dma_fence (@finished) signalled when the = hardware > + * completes the job. > + * > + * The hardware fence returned by &drm_dep_queue_ops.run_job is = stored as > + * @parent. @finished is chained to @parent via drm_dep_job_done_cb() = and > + * is signalled once @parent signals (or immediately if run_job() = returns > + * NULL or an error). I thought this fence proxy mechanism was going away due to recent work = being carried out by Christian? > + * > + * Drivers should expose @finished as the out-fence for GPU work = since it is > + * valid from the moment drm_dep_job_arm() returns, whereas the = hardware fence > + * could be a compound fence, which is disallowed when installed into > + * drm_syncobjs or dma-resv. > + * > + * The fence uses the kernel's inline spinlock (NULL passed to = dma_fence_init()) > + * so no separate lock allocation is required. > + * > + * Deadline propagation is supported: if a consumer sets a deadline = via > + * dma_fence_set_deadline(), it is forwarded to @parent when @parent = is set. > + * If @parent has not been set yet the deadline is stored in = @deadline and > + * forwarded at that point. > + * > + * Memory management: drm_dep_fence objects are allocated with = kzalloc() and > + * freed via kfree_rcu() once the fence is released, ensuring safety = with > + * RCU-protected fence accesses. > + */ > + > +#include > +#include > +#include "drm_dep_fence.h" > + > +/** > + * DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has = been set > + * > + * Set by the deadline callback on the finished fence to indicate a = deadline > + * has been set which may need to be propagated to the parent = hardware fence. > + */ > +#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT (DMA_FENCE_FLAG_USER_BITS = + 1) > + > +/** > + * struct drm_dep_fence - fence tracking the completion of a dep job > + * > + * Contains a single dma_fence (@finished) that is signalled when the > + * hardware completes the job. The fence uses the kernel's = inline_lock > + * (no external spinlock required). > + * > + * This struct is private to the drm_dep module; external code = interacts > + * through the accessor functions declared in drm_dep_fence.h. > + */ > +struct drm_dep_fence { > + /** > + * @finished: signalled when the job completes on hardware. > + * > + * Drivers should use this fence as the out-fence for a job since it > + * is available immediately upon drm_dep_job_arm(). > + */ > + struct dma_fence finished; > + > + /** > + * @deadline: deadline set on @finished which potentially needs to be > + * propagated to @parent. > + */ > + ktime_t deadline; > + > + /** > + * @parent: The hardware fence returned by = &drm_dep_queue_ops.run_job. > + * > + * @finished is signaled once @parent is signaled. The initial store = is > + * performed via smp_store_release to synchronize with deadline = handling. > + * > + * All readers must access this under the fence lock and take a = reference to > + * it, as @parent is set to NULL under the fence lock when the = drm_dep_fence > + * signals, and this drop also releases its internal reference. > + */ > + struct dma_fence *parent; > + > + /** > + * @q: the queue this fence belongs to. > + */ > + struct drm_dep_queue *q; > +}; > + > +static const struct dma_fence_ops drm_dep_fence_ops; > + > +/** > + * to_drm_dep_fence() - cast a dma_fence to its enclosing = drm_dep_fence > + * @f: dma_fence to cast > + * > + * Context: No context requirements (inline helper). > + * Return: pointer to the enclosing &drm_dep_fence. > + */ > +static struct drm_dep_fence *to_drm_dep_fence(struct dma_fence *f) > +{ > + return container_of(f, struct drm_dep_fence, finished); > +} > + > +/** > + * drm_dep_fence_set_parent() - store the hardware fence and = propagate > + * any deadline > + * @dfence: dep fence > + * @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or = NULL/error > + * > + * Stores @parent on @dfence under smp_store_release() so that a = concurrent > + * drm_dep_fence_set_deadline() call sees the parent before checking = the > + * deadline bit. If a deadline has already been set on = @dfence->finished it is > + * forwarded to @parent immediately. Does nothing if @parent is NULL = or an > + * error pointer. > + * > + * Context: Any context. > + */ > +void drm_dep_fence_set_parent(struct drm_dep_fence *dfence, > + struct dma_fence *parent) > +{ > + if (IS_ERR_OR_NULL(parent)) > + return; > + > + /* > + * smp_store_release() to ensure a thread racing us in > + * drm_dep_fence_set_deadline() sees the parent set before > + * it calls test_bit(HAS_DEADLINE_BIT). > + */ > + smp_store_release(&dfence->parent, dma_fence_get(parent)); > + if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, > + &dfence->finished.flags)) > + dma_fence_set_deadline(parent, dfence->deadline); > +} > + > +/** > + * drm_dep_fence_finished() - signal the finished fence with a result > + * @dfence: dep fence to signal > + * @result: error code to set, or 0 for success > + * > + * Sets the fence error to @result if non-zero, then signals > + * @dfence->finished. Also removes parent visibility under the fence = lock > + * and drops the parent reference. Dropping the parent here allows = the > + * DRM dep fence to be completely decoupled from the DRM dep module. > + * > + * Context: Any context. > + */ > +static void drm_dep_fence_finished(struct drm_dep_fence *dfence, int = result) > +{ > + struct dma_fence *parent; > + unsigned long flags; > + > + dma_fence_lock_irqsave(&dfence->finished, flags); > + if (result) > + dma_fence_set_error(&dfence->finished, result); > + dma_fence_signal_locked(&dfence->finished); > + parent =3D dfence->parent; > + dfence->parent =3D NULL; > + dma_fence_unlock_irqrestore(&dfence->finished, flags); > + > + dma_fence_put(parent); > +} We should really try to move away from manual locks and unlocks. > + > +static const char *drm_dep_fence_get_driver_name(struct dma_fence = *fence) > +{ > + return "drm_dep"; > +} > + > +static const char *drm_dep_fence_get_timeline_name(struct dma_fence = *f) > +{ > + struct drm_dep_fence *dfence =3D to_drm_dep_fence(f); > + > + return dfence->q->name; > +} > + > +/** > + * drm_dep_fence_get_parent() - get a reference to the parent = hardware fence > + * @dfence: dep fence to query > + * > + * Returns a new reference to @dfence->parent, or NULL if the parent = has > + * already been cleared (i.e. @dfence->finished has signalled and the = parent > + * reference was dropped under the fence lock). > + * > + * Uses smp_load_acquire() to pair with the smp_store_release() in > + * drm_dep_fence_set_parent(), ensuring that if we race a concurrent > + * drm_dep_fence_set_parent() call we observe the parent pointer only = after > + * the store is fully visible =E2=80=94 before set_parent() tests > + * %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT. > + * > + * Caller must hold the fence lock on @dfence->finished. > + * > + * Context: Any context, fence lock on @dfence->finished must be = held. > + * Return: a new reference to the parent fence, or NULL. > + */ > +static struct dma_fence *drm_dep_fence_get_parent(struct = drm_dep_fence *dfence) > +{ > + dma_fence_assert_held(&dfence->finished); > + > + return dma_fence_get(smp_load_acquire(&dfence->parent)); > +} > + > +/** > + * drm_dep_fence_set_deadline() - dma_fence_ops deadline callback > + * @f: fence on which the deadline is being set > + * @deadline: the deadline hint to apply > + * > + * Stores the earliest deadline under the fence lock, then propagates > + * it to the parent hardware fence via smp_load_acquire() to race > + * safely with drm_dep_fence_set_parent(). > + * > + * Context: Any context. > + */ > +static void drm_dep_fence_set_deadline(struct dma_fence *f, ktime_t = deadline) > +{ > + struct drm_dep_fence *dfence =3D to_drm_dep_fence(f); > + struct dma_fence *parent; > + unsigned long flags; > + > + dma_fence_lock_irqsave(f, flags); > + > + /* If we already have an earlier deadline, keep it: */ > + if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags) && > + ktime_before(dfence->deadline, deadline)) { > + dma_fence_unlock_irqrestore(f, flags); > + return; > + } > + > + dfence->deadline =3D deadline; > + set_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags); > + > + parent =3D drm_dep_fence_get_parent(dfence); > + dma_fence_unlock_irqrestore(f, flags); > + > + if (parent) > + dma_fence_set_deadline(parent, deadline); > + > + dma_fence_put(parent); > +} > + > +static const struct dma_fence_ops drm_dep_fence_ops =3D { > + .get_driver_name =3D drm_dep_fence_get_driver_name, > + .get_timeline_name =3D drm_dep_fence_get_timeline_name, > + .set_deadline =3D drm_dep_fence_set_deadline, > +}; > + > +/** > + * drm_dep_fence_alloc() - allocate a dep fence > + * > + * Allocates a &drm_dep_fence with kzalloc() without initialising the > + * dma_fence. Call drm_dep_fence_init() to fully initialise it. > + * > + * Context: Process context. > + * Return: new &drm_dep_fence on success, NULL on allocation failure. > + */ > +struct drm_dep_fence *drm_dep_fence_alloc(void) > +{ > + return kzalloc_obj(struct drm_dep_fence); > +} > + > +/** > + * drm_dep_fence_init() - initialise the dma_fence inside a dep fence > + * @dfence: dep fence to initialise > + * @q: queue the owning job belongs to > + * > + * Initialises @dfence->finished using the context and sequence = number from @q. > + * Passes NULL as the lock so the fence uses its inline spinlock. > + * > + * Context: Any context. > + */ > +void drm_dep_fence_init(struct drm_dep_fence *dfence, struct = drm_dep_queue *q) > +{ > + u32 seq =3D ++q->fence.seqno; > + > + /* > + * XXX: Inline fence hazard: currently all expected users of DRM dep > + * hardware fences have a unique lockdep class. If that ever changes, > + * we will need to assign a unique lockdep class here so lockdep = knows > + * this fence is allowed to nest with driver hardware fences. > + */ > + > + dfence->q =3D q; > + dma_fence_init(&dfence->finished, &drm_dep_fence_ops, > + NULL, q->fence.context, seq); > +} > + > +/** > + * drm_dep_fence_cleanup() - release a dep fence at job teardown > + * @dfence: dep fence to clean up > + * > + * Called from drm_dep_job_fini(). If the dep fence was armed = (refcount > 0) > + * it is released via dma_fence_put() and will be freed by the RCU = release > + * callback once all waiters have dropped their references. If it was = never > + * armed it is freed directly with kfree(). > + * > + * Context: Any context. > + */ > +void drm_dep_fence_cleanup(struct drm_dep_fence *dfence) > +{ > + if (drm_dep_fence_is_armed(dfence)) > + dma_fence_put(&dfence->finished); > + else > + kfree(dfence); > +} > + > +/** > + * drm_dep_fence_is_armed() - check whether the fence has been armed > + * @dfence: dep fence to check > + * > + * Returns true if drm_dep_job_arm() has been called, i.e. = @dfence->finished > + * has been initialised and its reference count is non-zero. Used by > + * assertions to enforce correct job lifecycle ordering (arm before = push, > + * add_dependency before arm). > + * > + * Context: Any context. > + * Return: true if the fence is armed, false otherwise. > + */ > +bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence) > +{ > + return !!kref_read(&dfence->finished.refcount); > +} > + > +/** > + * drm_dep_fence_is_finished() - test whether the finished fence has = signalled > + * @dfence: dep fence to check > + * > + * Uses dma_fence_test_signaled_flag() to read = %DMA_FENCE_FLAG_SIGNALED_BIT > + * directly without invoking the fence's ->signaled() callback or = triggering > + * any signalling side-effects. > + * > + * Context: Any context. > + * Return: true if @dfence->finished has been signalled, false = otherwise. > + */ > +bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence) > +{ > + return dma_fence_test_signaled_flag(&dfence->finished); > +} > + > +/** > + * drm_dep_fence_is_complete() - test whether the job has completed > + * @dfence: dep fence to check > + * > + * Takes the fence lock on @dfence->finished and calls > + * drm_dep_fence_get_parent() to safely obtain a reference to the = parent > + * hardware fence =E2=80=94 or NULL if the parent has already been = cleared after > + * signalling. Calls dma_fence_is_signaled() on @parent outside the = lock, > + * which may invoke the fence's ->signaled() callback and trigger = signalling > + * side-effects if the fence has completed but the signalled flag has = not yet > + * been set. The finished fence is tested via = dma_fence_test_signaled_flag(), > + * without side-effects. > + * > + * May only be called on a stopped queue (see = drm_dep_queue_is_stopped()). > + * > + * Context: Process context. The queue must be stopped before calling = this. > + * Return: true if the job is complete, false otherwise. > + */ > +bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence) > +{ > + struct dma_fence *parent; > + unsigned long flags; > + bool complete; > + > + dma_fence_lock_irqsave(&dfence->finished, flags); > + parent =3D drm_dep_fence_get_parent(dfence); > + dma_fence_unlock_irqrestore(&dfence->finished, flags); > + > + complete =3D (parent && dma_fence_is_signaled(parent)) || > + dma_fence_test_signaled_flag(&dfence->finished); > + > + dma_fence_put(parent); > + > + return complete; > +} > + > +/** > + * drm_dep_fence_to_dma() - return the finished dma_fence for a dep = fence > + * @dfence: dep fence to query > + * > + * No reference is taken; the caller must hold its own reference to = the owning > + * &drm_dep_job for the duration of the access. > + * > + * Context: Any context. > + * Return: the finished &dma_fence. > + */ > +struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence) > +{ > + return &dfence->finished; > +} > + > +/** > + * drm_dep_fence_done() - signal the finished fence on job completion > + * @dfence: dep fence to signal > + * @result: job error code, or 0 on success > + * > + * Gets a temporary reference to @dfence->finished to guard against a = racing > + * last-put, signals the fence with @result, then drops the temporary > + * reference. Called from drm_dep_job_done() in the queue core when a > + * hardware completion callback fires or when run_job() returns = immediately. > + * > + * Context: Any context. > + */ > +void drm_dep_fence_done(struct drm_dep_fence *dfence, int result) > +{ > + dma_fence_get(&dfence->finished); > + drm_dep_fence_finished(dfence, result); > + dma_fence_put(&dfence->finished); > +} Proper refcounting is automated (and enforced) in Rust. > diff --git a/drivers/gpu/drm/dep/drm_dep_fence.h = b/drivers/gpu/drm/dep/drm_dep_fence.h > new file mode 100644 > index 000000000000..65a1582f858b > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_fence.h > @@ -0,0 +1,25 @@ > +/* SPDX-License-Identifier: MIT */ > +/* > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +#ifndef _DRM_DEP_FENCE_H_ > +#define _DRM_DEP_FENCE_H_ > + > +#include > + > +struct drm_dep_fence; > +struct drm_dep_queue; > + > +struct drm_dep_fence *drm_dep_fence_alloc(void); > +void drm_dep_fence_init(struct drm_dep_fence *dfence, struct = drm_dep_queue *q); > +void drm_dep_fence_cleanup(struct drm_dep_fence *dfence); > +void drm_dep_fence_set_parent(struct drm_dep_fence *dfence, > + struct dma_fence *parent); > +void drm_dep_fence_done(struct drm_dep_fence *dfence, int result); > +bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence); > +bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence); > +bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence); > +struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence); > + > +#endif /* _DRM_DEP_FENCE_H_ */ > diff --git a/drivers/gpu/drm/dep/drm_dep_job.c = b/drivers/gpu/drm/dep/drm_dep_job.c > new file mode 100644 > index 000000000000..2d012b29a5fc > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_job.c > @@ -0,0 +1,675 @@ > +// SPDX-License-Identifier: MIT > +/* > + * Copyright 2015 Advanced Micro Devices, Inc. > + * > + * Permission is hereby granted, free of charge, to any person = obtaining a > + * copy of this software and associated documentation files (the = "Software"), > + * to deal in the Software without restriction, including without = limitation > + * the rights to use, copy, modify, merge, publish, distribute, = sublicense, > + * and/or sell copies of the Software, and to permit persons to whom = the > + * Software is furnished to do so, subject to the following = conditions: > + * > + * The above copyright notice and this permission notice shall be = included in > + * all copies or substantial portions of the Software. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, = EXPRESS OR > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF = MERCHANTABILITY, > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT = SHALL > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, = DAMAGES OR > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR = OTHERWISE, > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE = OR > + * OTHER DEALINGS IN THE SOFTWARE. > + * > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +/** > + * DOC: DRM dependency job > + * > + * A struct drm_dep_job represents a single unit of GPU work = associated with > + * a struct drm_dep_queue. The lifecycle of a job is: > + * > + * 1. **Allocation**: the driver allocates memory for the job = (typically by > + * embedding struct drm_dep_job in a larger structure) and calls > + * drm_dep_job_init() to initialise it. On success the job holds = one > + * kref reference and a reference to its queue. > + * > + * 2. **Dependency collection**: the driver calls = drm_dep_job_add_dependency(), > + * drm_dep_job_add_syncobj_dependency(), = drm_dep_job_add_resv_dependencies(), > + * or drm_dep_job_add_implicit_dependencies() to register = dma_fence objects > + * that must be signalled before the job can run. Duplicate fences = from the > + * same fence context are deduplicated automatically. > + * > + * 3. **Arming**: drm_dep_job_arm() initialises the job's finished = fence, > + * consuming a sequence number from the queue. After arming, > + * drm_dep_job_finished_fence() returns a valid fence that may be = passed to > + * userspace or used as a dependency by other jobs. > + * > + * 4. **Submission**: drm_dep_job_push() submits the job to the = queue. The > + * queue takes a reference that it holds until the job's finished = fence > + * signals and the job is freed by the put_job worker. > + * > + * 5. **Completion**: when the job's hardware work finishes its = finished fence > + * is signalled and drm_dep_job_put() is called by the queue. The = driver > + * must release any driver-private resources in = &drm_dep_job_ops.release. > + * > + * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The > + * internal drm_dep_job_fini() tears down the dependency xarray and = fence > + * objects before the driver's release callback is invoked. > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include "drm_dep_fence.h" > +#include "drm_dep_job.h" > +#include "drm_dep_queue.h" > + > +/** > + * drm_dep_job_init() - initialise a dep job > + * @job: dep job to initialise > + * @args: initialisation arguments > + * > + * Initialises @job with the queue, ops and credit count from @args. = Acquires > + * a reference to @args->q via drm_dep_queue_get(); this reference is = held for > + * the lifetime of the job and released by drm_dep_job_release() when = the last > + * job reference is dropped. > + * > + * Resources are released automatically when the last reference is = dropped > + * via drm_dep_job_put(), which must be called to release the job; = drivers > + * must not free the job directly. Again, can=E2=80=99t enforce that in C. > + * > + * Context: Process context. Allocates memory with GFP_KERNEL. > + * Return: 0 on success, -%EINVAL if credits is 0, > + * -%ENOMEM on fence allocation failure. > + */ > +int drm_dep_job_init(struct drm_dep_job *job, > + const struct drm_dep_job_init_args *args) > +{ > + if (unlikely(!args->credits)) { > + pr_err("drm_dep: %s: credits cannot be 0\n", __func__); > + return -EINVAL; > + } > + > + memset(job, 0, sizeof(*job)); > + > + job->dfence =3D drm_dep_fence_alloc(); > + if (!job->dfence) > + return -ENOMEM; > + > + job->ops =3D args->ops; > + job->q =3D drm_dep_queue_get(args->q); > + job->credits =3D args->credits; > + > + kref_init(&job->refcount); > + xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC); > + INIT_LIST_HEAD(&job->pending_link); > + > + return 0; > +} > +EXPORT_SYMBOL(drm_dep_job_init); > + > +/** > + * drm_dep_job_drop_dependencies() - release all input dependency = fences > + * @job: dep job whose dependency xarray to drain > + * > + * Walks @job->dependencies, puts each fence, and destroys the = xarray. > + * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel =E2=80= =94 > + * i.e. slots that were pre-allocated but never replaced =E2=80=94 = are silently > + * skipped; the sentinel carries no reference. Called from > + * drm_dep_queue_run_job() in process context immediately after > + * @ops->run_job() returns, before the final drm_dep_job_put(). = Releasing > + * dependencies here =E2=80=94 while still in process context =E2=80=94= avoids calling > + * xa_destroy() from IRQ context if the job's last reference is later > + * dropped from a dma_fence callback. > + * > + * Context: Process context. > + */ > +void drm_dep_job_drop_dependencies(struct drm_dep_job *job) > +{ > + struct dma_fence *fence; > + unsigned long index; > + > + xa_for_each(&job->dependencies, index, fence) { > + if (unlikely(fence =3D=3D DRM_DEP_JOB_FENCE_PREALLOC)) > + continue; > + dma_fence_put(fence); > + } > + xa_destroy(&job->dependencies); > +} This is automated in Rust. You also can=E2=80=99t =E2=80=9Cforget=E2=80=9D= to call this. > + > +/** > + * drm_dep_job_fini() - clean up a dep job > + * @job: dep job to clean up > + * > + * Cleans up the dep fence and drops the queue reference held by = @job. > + * > + * If the job was never armed (e.g. init failed before = drm_dep_job_arm()), > + * the dependency xarray is also released here. For armed jobs the = xarray > + * has already been drained by drm_dep_job_drop_dependencies() in = process > + * context immediately after run_job(), so it is left untouched to = avoid > + * calling xa_destroy() from IRQ context. > + * > + * Warns if @job is still linked on the queue's pending list, which = would > + * indicate a bug in the teardown ordering. > + * > + * Context: Any context. > + */ > +static void drm_dep_job_fini(struct drm_dep_job *job) > +{ > + bool armed =3D drm_dep_fence_is_armed(job->dfence); > + > + WARN_ON(!list_empty(&job->pending_link)); > + > + drm_dep_fence_cleanup(job->dfence); > + job->dfence =3D NULL; > + > + /* > + * Armed jobs have their dependencies drained by > + * drm_dep_job_drop_dependencies() in process context after = run_job(). > + * Skip here to avoid calling xa_destroy() from IRQ context. > + */ > + if (!armed) > + drm_dep_job_drop_dependencies(job); > +} Same here. > + > +/** > + * drm_dep_job_get() - acquire a reference to a dep job > + * @job: dep job to acquire a reference on, or NULL > + * > + * Context: Any context. > + * Return: @job with an additional reference held, or NULL if @job is = NULL. > + */ > +struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job) > +{ > + if (job) > + kref_get(&job->refcount); > + return job; > +} > +EXPORT_SYMBOL(drm_dep_job_get); > + Same here. > +/** > + * drm_dep_job_release() - kref release callback for a dep job > + * @kref: kref embedded in the dep job > + * > + * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if = set, > + * otherwise frees @job with kfree(). Finally, releases the queue = reference > + * that was acquired by drm_dep_job_init() via drm_dep_queue_put(). = The > + * queue put is performed last to ensure no queue state is accessed = after > + * the job memory is freed. > + * > + * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is = set on the > + * job's queue; otherwise process context only, as the release = callback may > + * sleep. > + */ > +static void drm_dep_job_release(struct kref *kref) > +{ > + struct drm_dep_job *job =3D > + container_of(kref, struct drm_dep_job, refcount); > + struct drm_dep_queue *q =3D job->q; > + > + drm_dep_job_fini(job); > + > + if (job->ops && job->ops->release) > + job->ops->release(job); > + else > + kfree(job); > + > + drm_dep_queue_put(q); > +} Same here. > + > +/** > + * drm_dep_job_put() - release a reference to a dep job > + * @job: dep job to release a reference on, or NULL > + * > + * When the last reference is dropped, calls &drm_dep_job_ops.release = if set, > + * otherwise frees @job with kfree(). Does nothing if @job is NULL. > + * > + * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is = set on the > + * job's queue; otherwise process context only, as the release = callback may > + * sleep. > + */ > +void drm_dep_job_put(struct drm_dep_job *job) > +{ > + if (job) > + kref_put(&job->refcount, drm_dep_job_release); > +} > +EXPORT_SYMBOL(drm_dep_job_put); > + Same here. > +/** > + * drm_dep_job_arm() - arm a dep job for submission > + * @job: dep job to arm > + * > + * Initialises the finished fence on @job->dfence, assigning > + * it a sequence number from the job's queue. Must be called after > + * drm_dep_job_init() and before drm_dep_job_push(). Once armed, > + * drm_dep_job_finished_fence() returns a valid fence that may be = passed to > + * userspace or used as a dependency by other jobs. > + * > + * Begins the DMA fence signalling path via = dma_fence_begin_signalling(). > + * After this point, memory allocations that could trigger reclaim = are > + * forbidden; lockdep enforces this. arm() must always be paired with > + * drm_dep_job_push(); lockdep also enforces this pairing. > + * > + * Warns if the job has already been armed. > + * > + * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED = is set > + * (takes @q->sched.lock, a mutex); any context otherwise. DMA = fence signaling > + * path. > + */ > +void drm_dep_job_arm(struct drm_dep_job *job) > +{ > + drm_dep_queue_push_job_begin(job->q); > + WARN_ON(drm_dep_fence_is_armed(job->dfence)); > + drm_dep_fence_init(job->dfence, job->q); > + job->signalling_cookie =3D dma_fence_begin_signalling(); > +} > +EXPORT_SYMBOL(drm_dep_job_arm); > + > +/** > + * drm_dep_job_push() - submit a job to its queue for execution > + * @job: dep job to push > + * > + * Submits @job to the queue it was initialised with. Must be called = after > + * drm_dep_job_arm(). Acquires a reference on @job on behalf of the = queue, > + * held until the queue is fully done with it. The reference is = released > + * directly in the finished-fence dma_fence callback for queues with > + * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() = may run > + * from hardirq context), or via the put_job work item on the submit > + * workqueue otherwise. > + * > + * Ends the DMA fence signalling path begun by drm_dep_job_arm() via > + * dma_fence_end_signalling(). This must be paired with arm(); = lockdep > + * enforces the pairing. > + * > + * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called = for > + * @job exactly once, even if the queue is killed or torn down before = the > + * job reaches the head of the queue. Drivers can use this guarantee = to > + * perform bookkeeping cleanup; the actual backend operation should = be > + * skipped when drm_dep_queue_is_killed() returns true. > + * > + * If the queue does not support the bypass path, the job is pushed = directly > + * onto the SPSC submission queue via drm_dep_queue_push_job() = without holding > + * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is = either > + * run immediately via drm_dep_queue_run_job() if it qualifies for = bypass, or > + * enqueued via drm_dep_queue_push_job() for dispatch by the run_job = work item. > + * > + * Warns if the job has not been armed. > + * > + * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED = is set > + * (takes @q->sched.lock, a mutex); any context otherwise. DMA = fence signaling > + * path. > + */ > +void drm_dep_job_push(struct drm_dep_job *job) > +{ > + struct drm_dep_queue *q =3D job->q; > + > + WARN_ON(!drm_dep_fence_is_armed(job->dfence)); > + > + drm_dep_job_get(job); > + > + if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) { > + drm_dep_queue_push_job(q, job); > + dma_fence_end_signalling(job->signalling_cookie); Signaling is enforced in a more thorough way in Rust. I=E2=80=99ll = expand on this later in this patch. > + drm_dep_queue_push_job_end(job->q); > + return; > + } > + > + scoped_guard(mutex, &q->sched.lock) { > + if (drm_dep_queue_can_job_bypass(q, job)) > + drm_dep_queue_run_job(q, job); > + else > + drm_dep_queue_push_job(q, job); > + } > + > + dma_fence_end_signalling(job->signalling_cookie); > + drm_dep_queue_push_job_end(job->q); > +} > +EXPORT_SYMBOL(drm_dep_job_push); > + > +/** > + * drm_dep_job_add_dependency() - adds the fence as a job dependency > + * @job: dep job to add the dependencies to > + * @fence: the dma_fence to add to the list of dependencies, or > + * %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later. > + * > + * Note that @fence is consumed in both the success and error cases = (except > + * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no = reference). > + * > + * Signalled fences and fences belonging to the same queue as @job = (i.e. where > + * fence->context matches the queue's finished fence context) are = silently > + * dropped; the job need not wait on its own queue's output. > + * > + * Warns if the job has already been armed (dependencies must be = added before > + * drm_dep_job_arm()). > + * > + * **Pre-allocation pattern** > + * > + * When multiple jobs across different queues must be prepared and = submitted > + * together in a single atomic commit =E2=80=94 for example, where = job A's finished > + * fence is an input dependency of job B =E2=80=94 all jobs must be = armed and pushed > + * within a single dma_fence_begin_signalling() / = dma_fence_end_signalling() > + * region. Once that region has started no memory allocation is = permitted. > + * > + * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the = preparation > + * phase (before arming any job, while GFP_KERNEL allocation is still = allowed) > + * to pre-allocate a slot in @job->dependencies. The slot index = assigned by > + * the underlying xarray must be tracked by the caller separately = (e.g. it is > + * always index 0 when the dependency array is empty, as Xe relies = on). > + * After all jobs have been armed and the finished fences are = available, call > + * drm_dep_job_replace_dependency() with that index and the real = fence. > + * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and = may be > + * called from atomic or signalling context. > + * > + * The sentinel slot is never skipped by the signalled-fence = fast-path, > + * ensuring a slot is always allocated even when the real fence is = not yet > + * known. > + * > + * **Example: bind job feeding TLB invalidation jobs** > + * > + * Consider a GPU with separate queues for page-table bind operations = and for > + * TLB invalidation. A single atomic commit must: > + * > + * 1. Run a bind job that modifies page tables. > + * 2. Run one TLB-invalidation job per MMU that depends on the bind > + * completing, so stale translations are flushed before the = engines > + * continue. > + * > + * Because all jobs must be armed and pushed inside a signalling = region (where > + * GFP_KERNEL is forbidden), pre-allocate slots before entering the = region:: > + * > + * // Phase 1 =E2=80=94 process context, GFP_KERNEL allowed > + * drm_dep_job_init(bind_job, bind_queue, ops); > + * for_each_mmu(mmu) { > + * drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops); > + * // Pre-allocate slot at index 0; real fence not available = yet > + * drm_dep_job_add_dependency(tlb_job[mmu], = DRM_DEP_JOB_FENCE_PREALLOC); > + * } > + * > + * // Phase 2 =E2=80=94 inside signalling region, no GFP_KERNEL > + * dma_fence_begin_signalling(); > + * drm_dep_job_arm(bind_job); > + * for_each_mmu(mmu) { > + * // Swap sentinel for bind job's finished fence > + * drm_dep_job_replace_dependency(tlb_job[mmu], 0, > + * = dma_fence_get(bind_job->finished)); > + * drm_dep_job_arm(tlb_job[mmu]); > + * } > + * drm_dep_job_push(bind_job); > + * for_each_mmu(mmu) > + * drm_dep_job_push(tlb_job[mmu]); > + * dma_fence_end_signalling(); > + * > + * Context: Process context. May allocate memory with GFP_KERNEL. > + * Return: If fence =3D=3D DRM_DEP_JOB_FENCE_PREALLOC index of = allocation on > + * success, else 0 on success, or a negative error code. > + */ > +int drm_dep_job_add_dependency(struct drm_dep_job *job, struct = dma_fence *fence) > +{ > + struct drm_dep_queue *q =3D job->q; > + struct dma_fence *entry; > + unsigned long index; > + u32 id =3D 0; > + int ret; > + > + WARN_ON(drm_dep_fence_is_armed(job->dfence)); > + might_alloc(GFP_KERNEL); > + > + if (!fence) > + return 0; > + > + if (fence =3D=3D DRM_DEP_JOB_FENCE_PREALLOC) > + goto add_fence; > + > + /* > + * Ignore signalled fences or fences from our own queue =E2=80=94 = finished > + * fences use q->fence.context. > + */ > + if (dma_fence_test_signaled_flag(fence) || > + fence->context =3D=3D q->fence.context) { > + dma_fence_put(fence); > + return 0; > + } > + > + /* Deduplicate if we already depend on a fence from the same = context. > + * This lets the size of the array of deps scale with the number of > + * engines involved, rather than the number of BOs. > + */ > + xa_for_each(&job->dependencies, index, entry) { > + if (entry =3D=3D DRM_DEP_JOB_FENCE_PREALLOC || > + entry->context !=3D fence->context) > + continue; > + > + if (dma_fence_is_later(fence, entry)) { > + dma_fence_put(entry); > + xa_store(&job->dependencies, index, fence, GFP_KERNEL); > + } else { > + dma_fence_put(fence); > + } > + return 0; > + } > + > +add_fence: > + ret =3D xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, > + GFP_KERNEL); > + if (ret !=3D 0) { > + if (fence !=3D DRM_DEP_JOB_FENCE_PREALLOC) > + dma_fence_put(fence); > + return ret; > + } > + > + return (fence =3D=3D DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0; > +} > +EXPORT_SYMBOL(drm_dep_job_add_dependency); > + > +/** > + * drm_dep_job_replace_dependency() - replace a pre-allocated = dependency slot > + * @job: dep job to update > + * @index: xarray index of the slot to replace, as returned when the = sentinel > + * was originally inserted via drm_dep_job_add_dependency() > + * @fence: the real dma_fence to store; its reference is always = consumed > + * > + * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in > + * @job->dependencies with @fence. The slot must have been = pre-allocated by > + * passing %DRM_DEP_JOB_FENCE_PREALLOC to = drm_dep_job_add_dependency(); the > + * existing entry is asserted to be the sentinel. > + * > + * This is the second half of the pre-allocation pattern described in > + * drm_dep_job_add_dependency(). It is intended to be called inside = a > + * dma_fence_begin_signalling() / dma_fence_end_signalling() region = where > + * memory allocation with GFP_KERNEL is forbidden. It uses = GFP_NOWAIT > + * internally so it is safe to call from atomic or signalling = context, but > + * since the slot has been pre-allocated no actual memory allocation = occurs. > + * > + * If @fence is already signalled the slot is erased rather than = storing a > + * redundant dependency. The successful store is asserted =E2=80=94 = if the store > + * fails it indicates a programming error (slot index out of range or > + * concurrent modification). > + * > + * Must be called before drm_dep_job_arm(). @fence is consumed in all = cases. Can=E2=80=99t enforce this in C. Also, how is the fence =E2=80=9Cconsumed=E2= =80=9D ? You can=E2=80=99t enforce that the user can=E2=80=99t access the fence anymore after this function = returns, like we can do at compile time in Rust. > + * > + * Context: Any context. DMA fence signaling path. > + */ > +void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 = index, > + struct dma_fence *fence) > +{ > + WARN_ON(xa_load(&job->dependencies, index) !=3D > + DRM_DEP_JOB_FENCE_PREALLOC); > + > + if (dma_fence_test_signaled_flag(fence)) { > + xa_erase(&job->dependencies, index); > + dma_fence_put(fence); > + return; > + } > + > + if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence, > + GFP_NOWAIT)))) { > + dma_fence_put(fence); > + return; > + } > +} > +EXPORT_SYMBOL(drm_dep_job_replace_dependency); > + > +/** > + * drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a > + * job dependency > + * @job: dep job to add the dependencies to > + * @file: drm file private pointer > + * @handle: syncobj handle to lookup > + * @point: timeline point > + * > + * This adds the fence matching the given syncobj to @job. > + * > + * Context: Process context. > + * Return: 0 on success, or a negative error code. > + */ > +int drm_dep_job_add_syncobj_dependency(struct drm_dep_job *job, > + struct drm_file *file, u32 handle, > + u32 point) > +{ > + struct dma_fence *fence; > + int ret; > + > + ret =3D drm_syncobj_find_fence(file, handle, point, 0, &fence); > + if (ret) > + return ret; > + > + return drm_dep_job_add_dependency(job, fence); > +} > +EXPORT_SYMBOL(drm_dep_job_add_syncobj_dependency); > + > +/** > + * drm_dep_job_add_resv_dependencies() - add all fences from the resv = to the job > + * @job: dep job to add the dependencies to > + * @resv: the dma_resv object to get the fences from > + * @usage: the dma_resv_usage to use to filter the fences > + * > + * This adds all fences matching the given usage from @resv to @job. > + * Must be called with the @resv lock held. > + * > + * Context: Process context. > + * Return: 0 on success, or a negative error code. > + */ > +int drm_dep_job_add_resv_dependencies(struct drm_dep_job *job, > + struct dma_resv *resv, > + enum dma_resv_usage usage) > +{ > + struct dma_resv_iter cursor; > + struct dma_fence *fence; > + int ret; > + > + dma_resv_assert_held(resv); > + > + dma_resv_for_each_fence(&cursor, resv, usage, fence) { > + /* > + * As drm_dep_job_add_dependency always consumes the fence > + * reference (even when it fails), and dma_resv_for_each_fence > + * is not obtaining one, we need to grab one before calling. > + */ > + ret =3D drm_dep_job_add_dependency(job, dma_fence_get(fence)); > + if (ret) > + return ret; > + } > + return 0; > +} > +EXPORT_SYMBOL(drm_dep_job_add_resv_dependencies); > + > +/** > + * drm_dep_job_add_implicit_dependencies() - adds implicit = dependencies > + * as job dependencies > + * @job: dep job to add the dependencies to > + * @obj: the gem object to add new dependencies from. > + * @write: whether the job might write the object (so we need to = depend on > + * shared fences in the reservation object). > + * > + * This should be called after drm_gem_lock_reservations() on your = array of > + * GEM objects used in the job but before updating the reservations = with your > + * own fences. > + * > + * Context: Process context. > + * Return: 0 on success, or a negative error code. > + */ > +int drm_dep_job_add_implicit_dependencies(struct drm_dep_job *job, > + struct drm_gem_object *obj, > + bool write) > +{ > + return drm_dep_job_add_resv_dependencies(job, obj->resv, > + dma_resv_usage_rw(write)); > +} > +EXPORT_SYMBOL(drm_dep_job_add_implicit_dependencies); > + > +/** > + * drm_dep_job_is_signaled() - check whether a dep job has completed > + * @job: dep job to check > + * > + * Determines whether @job has signalled. The queue should be stopped = before > + * calling this to obtain a stable snapshot of state. Both the parent = hardware > + * fence and the finished software fence are checked. > + * > + * Context: Process context. The queue must be stopped before calling = this. > + * Return: true if the job is signalled, false otherwise. > + */ > +bool drm_dep_job_is_signaled(struct drm_dep_job *job) > +{ > + WARN_ON(!drm_dep_queue_is_stopped(job->q)); > + return drm_dep_fence_is_complete(job->dfence); > +} > +EXPORT_SYMBOL(drm_dep_job_is_signaled); > + > +/** > + * drm_dep_job_is_finished() - test whether a dep job's finished = fence has signalled > + * @job: dep job to check > + * > + * Tests whether the job's software finished fence has been = signalled, using > + * dma_fence_test_signaled_flag() to avoid any signalling = side-effects. Unlike > + * drm_dep_job_is_signaled(), this does not require the queue to be = stopped and > + * does not check the parent hardware fence =E2=80=94 it is a = lightweight test of the > + * finished fence only. > + * > + * Context: Any context. > + * Return: true if the job's finished fence has been signalled, false = otherwise. > + */ > +bool drm_dep_job_is_finished(struct drm_dep_job *job) > +{ > + return drm_dep_fence_is_finished(job->dfence); > +} > +EXPORT_SYMBOL(drm_dep_job_is_finished); > + > +/** > + * drm_dep_job_invalidate_job() - increment the invalidation count = for a job > + * @job: dep job to invalidate > + * @threshold: threshold above which the job is considered = invalidated > + * > + * Increments @job->invalidate_count and returns true if it exceeds = @threshold, > + * indicating the job should be considered hung and discarded. The = queue must > + * be stopped before calling this function. > + * > + * Context: Process context. The queue must be stopped before calling = this. > + * Return: true if @job->invalidate_count exceeds @threshold, false = otherwise. > + */ > +bool drm_dep_job_invalidate_job(struct drm_dep_job *job, int = threshold) > +{ > + WARN_ON(!drm_dep_queue_is_stopped(job->q)); > + return ++job->invalidate_count > threshold; > +} > +EXPORT_SYMBOL(drm_dep_job_invalidate_job); > + > +/** > + * drm_dep_job_finished_fence() - return the finished fence for a job > + * @job: dep job to query > + * > + * No reference is taken on the returned fence; the caller must hold = its own > + * reference to @job for the duration of any access. Can=E2=80=99t enforce this in C. > + * > + * Context: Any context. > + * Return: the finished &dma_fence for @job. > + */ > +struct dma_fence *drm_dep_job_finished_fence(struct drm_dep_job *job) > +{ > + return drm_dep_fence_to_dma(job->dfence); > +} > +EXPORT_SYMBOL(drm_dep_job_finished_fence); > diff --git a/drivers/gpu/drm/dep/drm_dep_job.h = b/drivers/gpu/drm/dep/drm_dep_job.h > new file mode 100644 > index 000000000000..35c61d258fa1 > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_job.h > @@ -0,0 +1,13 @@ > +/* SPDX-License-Identifier: MIT */ > +/* > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +#ifndef _DRM_DEP_JOB_H_ > +#define _DRM_DEP_JOB_H_ > + > +struct drm_dep_queue; > + > +void drm_dep_job_drop_dependencies(struct drm_dep_job *job); > + > +#endif /* _DRM_DEP_JOB_H_ */ > diff --git a/drivers/gpu/drm/dep/drm_dep_queue.c = b/drivers/gpu/drm/dep/drm_dep_queue.c > new file mode 100644 > index 000000000000..dac02d0d22c4 > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_queue.c > @@ -0,0 +1,1647 @@ > +// SPDX-License-Identifier: MIT > +/* > + * Copyright 2015 Advanced Micro Devices, Inc. > + * > + * Permission is hereby granted, free of charge, to any person = obtaining a > + * copy of this software and associated documentation files (the = "Software"), > + * to deal in the Software without restriction, including without = limitation > + * the rights to use, copy, modify, merge, publish, distribute, = sublicense, > + * and/or sell copies of the Software, and to permit persons to whom = the > + * Software is furnished to do so, subject to the following = conditions: > + * > + * The above copyright notice and this permission notice shall be = included in > + * all copies or substantial portions of the Software. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, = EXPRESS OR > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF = MERCHANTABILITY, > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT = SHALL > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, = DAMAGES OR > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR = OTHERWISE, > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE = OR > + * OTHER DEALINGS IN THE SOFTWARE. > + * > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +/** > + * DOC: DRM dependency queue > + * > + * The drm_dep subsystem provides a lightweight GPU submission queue = that > + * combines the roles of drm_gpu_scheduler and drm_sched_entity into = a > + * single object (struct drm_dep_queue). Each queue owns its own = ordered > + * submit workqueue, timeout workqueue, and TDR delayed-work. > + * > + * **Job lifecycle** > + * > + * 1. Allocate and initialise a job with drm_dep_job_init(). > + * 2. Add dependency fences with drm_dep_job_add_dependency() and = friends. > + * 3. Arm the job with drm_dep_job_arm() to obtain its out-fences. > + * 4. Submit with drm_dep_job_push(). > + * > + * **Submission paths** > + * > + * drm_dep_job_push() decides between two paths under @q->sched.lock: > + * > + * - **Bypass path** (drm_dep_queue_can_job_bypass()): if > + * %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not = stopped, > + * the SPSC queue is empty, the job has no dependency fences, and = credits > + * are available, the job is submitted inline on the calling thread = without > + * touching the submit workqueue. > + * > + * - **Queued path** (drm_dep_queue_push_job()): the job is pushed = onto an > + * SPSC queue and the run_job worker is kicked. The run_job worker = pops the > + * job, resolves any remaining dependency fences (installing wakeup > + * callbacks for unresolved ones), and calls = drm_dep_queue_run_job(). > + * > + * **Running a job** > + * > + * drm_dep_queue_run_job() accounts credits, appends the job to the = pending > + * list (starting the TDR timer only when the list was previously = empty), > + * calls @ops->run_job(), stores the returned hardware fence as the = parent > + * of the job's dep fence, then installs a callback on it. When the = hardware > + * fence fires (or the job completes synchronously), = drm_dep_job_done() > + * signals the finished fence, returns credits, and kicks the put_job = worker > + * to free the job. > + * > + * **Timeout detection and recovery (TDR)** > + * > + * A delayed work item fires when a job on the pending list takes = longer than > + * @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on = the > + * returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or > + * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB). > + * drm_dep_queue_trigger_timeout() forces the timer to fire = immediately (without > + * changing the stored timeout), for example during device teardown. > + * > + * **Reference counting** > + * > + * Jobs and queues are both reference counted. > + * > + * A job holds a reference to its queue from drm_dep_job_init() until > + * drm_dep_job_put() drops the job's last reference and its release = callback > + * runs. This ensures the queue remains valid for the entire lifetime = of any > + * job that was submitted to it. > + * > + * The queue holds its own reference to a job for as long as the job = is > + * internally tracked: from the moment the job is added to the = pending list > + * in drm_dep_queue_run_job() until drm_dep_job_done() kicks the = put_job > + * worker, which calls drm_dep_job_put() to release that reference. Why not simply keep track that the job was completed, instead of = relinquishing the reference? We can then release the reference once the job is cleaned = up (by the queue, using a worker) in process context. > + * > + * **Hazard: use-after-free from within a worker** > + * > + * Because a job holds a queue reference, drm_dep_job_put() dropping = the last > + * job reference will also drop a queue reference via the job's = release path. > + * If that happens to be the last queue reference, = drm_dep_queue_fini() can be > + * called, which queues @q->free_work on dep_free_wq and returns = immediately. > + * free_work calls disable_work_sync() / disable_delayed_work_sync() = on the > + * queue's own workers before destroying its workqueues, so in = practice a > + * running worker always completes before the queue memory is freed. > + * > + * However, there is a secondary hazard: a worker can be queued while = the > + * queue is in a "zombie" state =E2=80=94 refcount has already = reached zero and async > + * teardown is in flight, but the work item has not yet been disabled = by > + * free_work. To guard against this every worker uses > + * drm_dep_queue_get_unless_zero() at entry; if the refcount is = already zero > + * the worker bails immediately without touching the queue state. Again, this problem is gone in Rust. > + * > + * Because all actual teardown (disable_*_sync, destroy_workqueue) = runs on > + * dep_free_wq =E2=80=94 which is independent of the queue's own = submit/timeout > + * workqueues =E2=80=94 there is no deadlock risk. Each queue holds = a drm_dev_get() > + * reference on its owning &drm_device, which is released as the last = step of > + * teardown. This ensures the driver module cannot be unloaded while = any queue > + * is still alive. > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include "drm_dep_fence.h" > +#include "drm_dep_job.h" > +#include "drm_dep_queue.h" > + > +/* > + * Dedicated workqueue for deferred drm_dep_queue teardown. Using a > + * module-private WQ instead of system_percpu_wq keeps teardown = isolated > + * from unrelated kernel subsystems. > + */ > +static struct workqueue_struct *dep_free_wq; > + > +/** > + * drm_dep_queue_flags_set() - set a flag on the queue under = sched.lock > + * @q: dep queue > + * @flag: flag to set (one of &enum drm_dep_queue_flags) > + * > + * Sets @flag in @q->sched.flags. Must be called with @q->sched.lock > + * held; the lockdep assertion enforces this. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + */ > +static void drm_dep_queue_flags_set(struct drm_dep_queue *q, > + enum drm_dep_queue_flags flag) > +{ > + lockdep_assert_held(&q->sched.lock); We can enforce this in Rust at compile-time. The code does not compile = if the lock is not taken. Same here and everywhere else where the sched lock = has to be taken. > + q->sched.flags |=3D flag; > +} > + > +/** > + * drm_dep_queue_flags_clear() - clear a flag on the queue under = sched.lock > + * @q: dep queue > + * @flag: flag to clear (one of &enum drm_dep_queue_flags) > + * > + * Clears @flag in @q->sched.flags. Must be called with = @q->sched.lock > + * held; the lockdep assertion enforces this. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + */ > +static void drm_dep_queue_flags_clear(struct drm_dep_queue *q, > + enum drm_dep_queue_flags flag) > +{ > + lockdep_assert_held(&q->sched.lock); > + q->sched.flags &=3D ~flag; > +} > + > +/** > + * drm_dep_queue_has_credits() - check whether the queue has enough = credits > + * @q: dep queue > + * @job: job requesting credits > + * > + * Checks whether the queue has enough available credits to dispatch > + * @job. If @job->credits exceeds the queue's credit limit, it is > + * clamped with a WARN. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + * Return: true if available credits >=3D @job->credits, false = otherwise. > + */ > +static bool drm_dep_queue_has_credits(struct drm_dep_queue *q, > + struct drm_dep_job *job) > +{ > + u32 available; > + > + lockdep_assert_held(&q->sched.lock); > + > + if (job->credits > q->credit.limit) { > + drm_warn(q->drm, > + "Jobs may not exceed the credit limit, truncate.\n"); > + job->credits =3D q->credit.limit; > + } > + > + WARN_ON(check_sub_overflow(q->credit.limit, > + atomic_read(&q->credit.count), > + &available)); > + > + return available >=3D job->credits; > +} > + > +/** > + * drm_dep_queue_run_job_queue() - kick the run-job worker > + * @q: dep queue > + * > + * Queues @q->sched.run_job on @q->sched.submit_wq unless the queue = is stopped > + * or the job queue is empty. The empty-queue check avoids queueing = a work item > + * that would immediately return with nothing to do. > + * > + * Context: Any context. > + */ > +static void drm_dep_queue_run_job_queue(struct drm_dep_queue *q) > +{ > + if (!drm_dep_queue_is_stopped(q) && spsc_queue_count(&q->job.queue)) > + queue_work(q->sched.submit_wq, &q->sched.run_job); > +} > + > +/** > + * drm_dep_queue_put_job_queue() - kick the put-job worker > + * @q: dep queue > + * > + * Queues @q->sched.put_job on @q->sched.submit_wq unless the queue > + * is stopped. > + * > + * Context: Any context. > + */ > +static void drm_dep_queue_put_job_queue(struct drm_dep_queue *q) > +{ > + if (!drm_dep_queue_is_stopped(q)) > + queue_work(q->sched.submit_wq, &q->sched.put_job); > +} > + > +/** > + * drm_queue_start_timeout() - arm or re-arm the TDR delayed work > + * @q: dep queue > + * > + * Arms the TDR delayed work with @q->job.timeout. No-op if > + * @q->ops->timedout_job is NULL, the timeout is = MAX_SCHEDULE_TIMEOUT, > + * or the pending list is empty. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + */ > +static void drm_queue_start_timeout(struct drm_dep_queue *q) > +{ > + lockdep_assert_held(&q->job.lock); > + > + if (!q->ops->timedout_job || > + q->job.timeout =3D=3D MAX_SCHEDULE_TIMEOUT || > + list_empty(&q->job.pending)) > + return; > + > + mod_delayed_work(q->sched.timeout_wq, &q->sched.tdr, = q->job.timeout); > +} > + > +/** > + * drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock > + * @q: dep queue > + * > + * Acquires @q->job.lock with irq and calls > + * drm_queue_start_timeout(). > + * > + * Context: Process context (workqueue). > + */ > +static void drm_queue_start_timeout_unlocked(struct drm_dep_queue *q) > +{ > + guard(spinlock_irq)(&q->job.lock); > + drm_queue_start_timeout(q); > +} > + > +/** > + * drm_dep_queue_remove_dependency() - clear the active dependency = and wake > + * the run-job worker > + * @q: dep queue > + * @f: the dependency fence being removed > + * > + * Stores @f into @q->dep.removed_fence via smp_store_release() so = that the > + * run-job worker can drop the reference to it in = drm_dep_queue_is_ready(), > + * paired with smp_load_acquire(). Clears @q->dep.fence and kicks = the > + * run-job worker. > + * > + * The fence reference is not dropped here; it is deferred to the = run-job > + * worker via @q->dep.removed_fence to keep this path suitable = dma_fence > + * callback removal in drm_dep_queue_kill(). This is a comment in C, but in Rust this is encoded directly in the type = system. > + * > + * Context: Any context. > + */ > +static void drm_dep_queue_remove_dependency(struct drm_dep_queue *q, > + struct dma_fence *f) > +{ > + /* removed_fence must be visible to the reader before &q->dep.fence = */ > + smp_store_release(&q->dep.removed_fence, f); > + > + WRITE_ONCE(q->dep.fence, NULL); > + drm_dep_queue_run_job_queue(q); > +} > + > +/** > + * drm_dep_queue_wakeup() - dma_fence callback to wake the run-job = worker > + * @f: the signalled dependency fence > + * @cb: callback embedded in the dep queue > + * > + * Called from dma_fence_signal() when the active dependency fence = signals. > + * Delegates to drm_dep_queue_remove_dependency() to clear = @q->dep.fence and > + * kick the run-job worker. The fence reference is not dropped here; = it is > + * deferred to the run-job worker via @q->dep.removed_fence. Same here. > + * > + * Context: Any context. > + */ > +static void drm_dep_queue_wakeup(struct dma_fence *f, struct = dma_fence_cb *cb) > +{ > + struct drm_dep_queue *q =3D > + container_of(cb, struct drm_dep_queue, dep.cb); > + > + drm_dep_queue_remove_dependency(q, f); > +} > + > +/** > + * drm_dep_queue_is_ready() - check whether the queue has a = dispatchable job > + * @q: dep queue > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. Can=E2=80=99t call this in Rust if the lock is not taken. > + * Return: true if SPSC queue non-empty and no dep fence pending, > + * false otherwise. > + */ > +static bool drm_dep_queue_is_ready(struct drm_dep_queue *q) > +{ > + lockdep_assert_held(&q->sched.lock); > + > + if (!spsc_queue_count(&q->job.queue)) > + return false; > + > + if (READ_ONCE(q->dep.fence)) > + return false; > + > + /* Paired with smp_store_release in = drm_dep_queue_remove_dependency() */ > + dma_fence_put(smp_load_acquire(&q->dep.removed_fence)); > + > + q->dep.removed_fence =3D NULL; > + > + return true; > +} > + > +/** > + * drm_dep_queue_is_killed() - check whether a dep queue has been = killed > + * @q: dep queue to check > + * > + * Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false = otherwise. > + * > + * Context: Any context. > + */ > +bool drm_dep_queue_is_killed(struct drm_dep_queue *q) > +{ > + return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_KILLED); > +} > +EXPORT_SYMBOL(drm_dep_queue_is_killed); > + > +/** > + * drm_dep_queue_is_initialized() - check whether a dep queue has = been initialized > + * @q: dep queue to check > + * > + * A queue is considered initialized once its ops pointer has been = set by a > + * successful call to drm_dep_queue_init(). Drivers that embed a > + * &drm_dep_queue inside a larger structure may call this before = attempting any > + * other queue operation to confirm that initialization has taken = place. > + * drm_dep_queue_put() must be called if this function returns true = to drop the > + * initialization reference from drm_dep_queue_init(). > + * > + * Return: true if @q has been initialized, false otherwise. > + * > + * Context: Any context. > + */ > +bool drm_dep_queue_is_initialized(struct drm_dep_queue *q) > +{ > + return !!q->ops; > +} > +EXPORT_SYMBOL(drm_dep_queue_is_initialized); > + > +/** > + * drm_dep_queue_set_stopped() - pre-mark a queue as stopped before = first use > + * @q: dep queue to mark > + * > + * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going = through the > + * normal drm_dep_queue_stop() path. This is only valid during the = driver-side > + * queue initialisation sequence =E2=80=94 i.e. after = drm_dep_queue_init() returns but > + * before the queue is made visible to other threads (e.g. before it = is added > + * to any lookup structures). Using this after the queue is live is = a driver > + * bug; use drm_dep_queue_stop() instead. > + * > + * Context: Process context, queue not yet visible to other threads. > + */ > +void drm_dep_queue_set_stopped(struct drm_dep_queue *q) > +{ > + q->sched.flags |=3D DRM_DEP_QUEUE_FLAGS_STOPPED; > +} > +EXPORT_SYMBOL(drm_dep_queue_set_stopped); > + > +/** > + * drm_dep_queue_refcount() - read the current reference count of a = queue > + * @q: dep queue to query > + * > + * Returns the instantaneous kref value. The count may change = immediately > + * after this call; callers must not make safety decisions based = solely on > + * the returned value. Intended for diagnostic snapshots and debugfs = output. > + * > + * Context: Any context. > + * Return: current reference count. > + */ > +unsigned int drm_dep_queue_refcount(const struct drm_dep_queue *q) > +{ > + return kref_read(&q->refcount); > +} > +EXPORT_SYMBOL(drm_dep_queue_refcount); > + > +/** > + * drm_dep_queue_timeout() - read the per-job TDR timeout for a queue > + * @q: dep queue to query > + * > + * Returns the per-job timeout in jiffies as set at init time. > + * %MAX_SCHEDULE_TIMEOUT means no timeout is configured. > + * > + * Context: Any context. > + * Return: timeout in jiffies. > + */ > +long drm_dep_queue_timeout(const struct drm_dep_queue *q) > +{ > + return q->job.timeout; > +} > +EXPORT_SYMBOL(drm_dep_queue_timeout); > + > +/** > + * drm_dep_queue_is_job_put_irq_safe() - test whether job-put from = IRQ is allowed > + * @q: dep queue > + * > + * Context: Any context. > + * Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set, > + * false otherwise. > + */ > +static bool drm_dep_queue_is_job_put_irq_safe(const struct = drm_dep_queue *q) > +{ > + return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE); > +} > + > +/** > + * drm_dep_queue_job_dependency() - get next unresolved dep fence > + * @q: dep queue > + * @job: job whose dependencies to advance > + * > + * Returns NULL immediately if the queue has been killed via > + * drm_dep_queue_kill(), bypassing all dependency waits so that jobs > + * drain through run_job as quickly as possible. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + * Return: next unresolved &dma_fence with a new reference, or NULL > + * when all dependencies have been consumed (or the queue is = killed). > + */ > +static struct dma_fence * > +drm_dep_queue_job_dependency(struct drm_dep_queue *q, > + struct drm_dep_job *job) > +{ > + struct dma_fence *f; > + > + lockdep_assert_held(&q->sched.lock); > + > + if (drm_dep_queue_is_killed(q)) > + return NULL; > + > + f =3D xa_load(&job->dependencies, job->last_dependency); > + if (f) { > + job->last_dependency++; > + if (WARN_ON(DRM_DEP_JOB_FENCE_PREALLOC =3D=3D f)) > + return dma_fence_get_stub(); > + return dma_fence_get(f); > + } > + > + return NULL; > +} > + > +/** > + * drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence > + * @q: dep queue > + * @job: job whose dependency fence is stored in @q->dep.fence > + * > + * Installs a wakeup callback on @q->dep.fence. Returns true if the > + * callback was installed (the queue must wait), false if the fence = is > + * already signalled or is a self-fence from the same queue context. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + * Return: true if callback installed, false if fence already done. > + */ In Rust, we can encode the signaling paths with a =E2=80=9Ctoken = type=E2=80=9D. So any sections that are part of the signaling path can simply take this token = as an argument. This type also enforces that end_signaling() is called = automatically when it goes out of scope. By the way, we can easily offer an irq handler type where we enforce = this: fn handle_threaded_irq(&self, device: &Device) -> IrqReturn {=20 let _annotation =3D DmaFenceSignallingAnnotation::new(); // Calls = begin_signaling() self.driver.handle_threaded_irq(device)=20 // end_signaling() is called here automatically. } Same for workqueues: fn work_fn(&self, device: &Device) { let _annotation =3D DmaFenceSignallingAnnotation::new(); // Calls = begin_signaling() self.driver.work_fn(device)=20 // end_signaling() is called here automatically. } This is not Rust-specific, of course, but it is more ergonomic to write = in Rust. > +static bool drm_dep_queue_add_dep_cb(struct drm_dep_queue *q, > + struct drm_dep_job *job) > +{ > + struct dma_fence *fence =3D q->dep.fence; > + > + lockdep_assert_held(&q->sched.lock); > + > + if (WARN_ON(fence->context =3D=3D q->fence.context)) { > + dma_fence_put(q->dep.fence); > + q->dep.fence =3D NULL; > + return false; > + } > + > + if (!dma_fence_add_callback(q->dep.fence, &q->dep.cb, > + drm_dep_queue_wakeup)) > + return true; > + > + dma_fence_put(q->dep.fence); > + q->dep.fence =3D NULL; > + > + return false; > +} In rust we can enforce that all callbacks take a reference to the fence automatically. If the callback is =E2=80=9Cforgotten=E2=80=9D in a buggy = path, it is automatically removed, and the fence is automatically signaled with = -ECANCELED. > + > +/** > + * drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC = queue > + * @q: dep queue > + * > + * Peeks at the head of the SPSC queue and drains all resolved > + * dependencies. If a dependency is still pending, installs a wakeup > + * callback and returns NULL. On success pops the job and returns it. > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + * Return: next dispatchable job, or NULL if a dep is still pending. > + */ > +static struct drm_dep_job *drm_dep_queue_pop_job(struct drm_dep_queue = *q) > +{ > + struct spsc_node *node; > + struct drm_dep_job *job; > + > + lockdep_assert_held(&q->sched.lock); > + > + node =3D spsc_queue_peek(&q->job.queue); > + if (!node) > + return NULL; > + > + job =3D container_of(node, struct drm_dep_job, queue_node); > + > + while ((q->dep.fence =3D drm_dep_queue_job_dependency(q, job))) { > + if (drm_dep_queue_add_dep_cb(q, job)) > + return NULL; > + } > + > + spsc_queue_pop(&q->job.queue); > + > + return job; > +} > + > +/* > + * drm_dep_queue_get_unless_zero() - try to acquire a queue reference > + * > + * Workers use this instead of drm_dep_queue_get() to guard against = the zombie > + * state: the queue's refcount has already reached zero (async = teardown is in > + * flight) but a work item was queued before free_work had a chance = to cancel > + * it. If kref_get_unless_zero() fails the caller must bail = immediately. > + * > + * Context: Any context. > + * Returns true if the reference was acquired, false if the queue is = zombie. > + */ Again, this function is totally gone in Rust. > +bool drm_dep_queue_get_unless_zero(struct drm_dep_queue *q) > +{ > + return kref_get_unless_zero(&q->refcount); > +} > +EXPORT_SYMBOL(drm_dep_queue_get_unless_zero); > + > +/** > + * drm_dep_queue_run_job_work() - run-job worker > + * @work: work item embedded in the dep queue > + * > + * Acquires @q->sched.lock, checks stopped state, queue readiness and > + * available credits, pops the next job via drm_dep_queue_pop_job(), > + * dispatches it via drm_dep_queue_run_job(), then re-kicks itself. > + * > + * Uses drm_dep_queue_get_unless_zero() at entry and bails = immediately if the > + * queue is in zombie state (refcount already zero, async teardown in = flight). > + * > + * Context: Process context (workqueue). DMA fence signaling path. > + */ > +static void drm_dep_queue_run_job_work(struct work_struct *work) > +{ > + struct drm_dep_queue *q =3D > + container_of(work, struct drm_dep_queue, sched.run_job); > + struct spsc_node *node; > + struct drm_dep_job *job; > + bool cookie =3D dma_fence_begin_signalling(); > + > + /* Bail if queue is zombie (refcount already zero, teardown in = flight). */ > + if (!drm_dep_queue_get_unless_zero(q)) { > + dma_fence_end_signalling(cookie); > + return; > + } > + > + mutex_lock(&q->sched.lock); > + > + if (drm_dep_queue_is_stopped(q)) > + goto put_queue; > + > + if (!drm_dep_queue_is_ready(q)) > + goto put_queue; > + > + /* Peek to check credits before committing to pop and dep resolution = */ > + node =3D spsc_queue_peek(&q->job.queue); > + if (!node) > + goto put_queue; > + > + job =3D container_of(node, struct drm_dep_job, queue_node); > + if (!drm_dep_queue_has_credits(q, job)) > + goto put_queue; > + > + job =3D drm_dep_queue_pop_job(q); > + if (!job) > + goto put_queue; > + > + drm_dep_queue_run_job(q, job); > + drm_dep_queue_run_job_queue(q); > + > +put_queue: > + mutex_unlock(&q->sched.lock); > + drm_dep_queue_put(q); > + dma_fence_end_signalling(cookie); > +} > + > +/* > + * drm_dep_queue_remove_job() - unlink a job from the pending list = and reset TDR > + * @q: dep queue owning @job > + * @job: job to remove > + * > + * Splices @job out of @q->job.pending, cancels any pending TDR = delayed work, > + * and arms the timeout for the new list head (if any). > + * > + * Context: Process context. Must hold @q->sched.lock. DMA fence = signaling path. > + */ > +static void drm_dep_queue_remove_job(struct drm_dep_queue *q, > + struct drm_dep_job *job) > +{ > + lockdep_assert_held(&q->job.lock); > + > + list_del_init(&job->pending_link); > + cancel_delayed_work(&q->sched.tdr); > + drm_queue_start_timeout(q); > +} > + > +/** > + * drm_dep_queue_get_finished_job() - dequeue a finished job > + * @q: dep queue > + * > + * Under @q->job.lock checks the head of the pending list for a > + * finished dep fence. If found, removes the job from the list, > + * cancels the TDR, and re-arms it for the new head. > + * > + * Context: Process context (workqueue). DMA fence signaling path. > + * Return: the finished &drm_dep_job, or NULL if none is ready. > + */ > +static struct drm_dep_job * > +drm_dep_queue_get_finished_job(struct drm_dep_queue *q) > +{ > + struct drm_dep_job *job; > + > + guard(spinlock_irq)(&q->job.lock); > + > + job =3D list_first_entry_or_null(&q->job.pending, struct = drm_dep_job, > + pending_link); > + if (job && drm_dep_fence_is_finished(job->dfence)) > + drm_dep_queue_remove_job(q, job); > + else > + job =3D NULL; > + > + return job; > +} > + > +/** > + * drm_dep_queue_put_job_work() - put-job worker > + * @work: work item embedded in the dep queue > + * > + * Drains all finished jobs by calling drm_dep_job_put() in a loop, > + * then kicks the run-job worker. > + * > + * Uses drm_dep_queue_get_unless_zero() at entry and bails = immediately if the > + * queue is in zombie state (refcount already zero, async teardown in = flight). > + * > + * Wraps execution in dma_fence_begin_signalling() / = dma_fence_end_signalling() > + * because workqueue is shared with other items in the fence = signaling path. > + * > + * Context: Process context (workqueue). DMA fence signaling path. > + */ > +static void drm_dep_queue_put_job_work(struct work_struct *work) > +{ > + struct drm_dep_queue *q =3D > + container_of(work, struct drm_dep_queue, sched.put_job); > + struct drm_dep_job *job; > + bool cookie =3D dma_fence_begin_signalling(); > + > + /* Bail if queue is zombie (refcount already zero, teardown in = flight). */ > + if (!drm_dep_queue_get_unless_zero(q)) { > + dma_fence_end_signalling(cookie); > + return; > + } > + > + while ((job =3D drm_dep_queue_get_finished_job(q))) > + drm_dep_job_put(job); > + > + drm_dep_queue_run_job_queue(q); > + > + drm_dep_queue_put(q); > + dma_fence_end_signalling(cookie); > +} > + > +/** > + * drm_dep_queue_tdr_work() - TDR worker > + * @work: work item embedded in the delayed TDR work > + * > + * Removes the head job from the pending list under @q->job.lock, > + * asserts @q->ops->timedout_job is non-NULL, calls it outside the = lock, > + * requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the > + * queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and = always > + * restarts the TDR timer after handling the job (unless @q is = stopping). > + * Any other return value triggers a WARN. > + * > + * The TDR is never armed when @q->ops->timedout_job is NULL, so = firing > + * this worker without a timedout_job callback is a driver bug. > + * > + * Uses drm_dep_queue_get_unless_zero() at entry and bails = immediately if the > + * queue is in zombie state (refcount already zero, async teardown in = flight). > + * > + * Wraps execution in dma_fence_begin_signalling() / = dma_fence_end_signalling() > + * because timedout_job() is expected to signal the guilty job's = fence as part > + * of reset. > + * > + * Context: Process context (workqueue). DMA fence signaling path. > + */ > +static void drm_dep_queue_tdr_work(struct work_struct *work) > +{ > + struct drm_dep_queue *q =3D > + container_of(work, struct drm_dep_queue, sched.tdr.work); > + struct drm_dep_job *job; > + bool cookie =3D dma_fence_begin_signalling(); > + > + /* Bail if queue is zombie (refcount already zero, teardown in = flight). */ > + if (!drm_dep_queue_get_unless_zero(q)) { > + dma_fence_end_signalling(cookie); > + return; > + } > + > + scoped_guard(spinlock_irq, &q->job.lock) { > + job =3D list_first_entry_or_null(&q->job.pending, > + struct drm_dep_job, > + pending_link); > + if (job) > + /* > + * Remove from pending so it cannot be freed > + * concurrently by drm_dep_queue_get_finished_job() or > + * .drm_dep_job_done(). > + */ > + list_del_init(&job->pending_link); > + } > + > + if (job) { > + enum drm_dep_timedout_stat status; > + > + if (WARN_ON(!q->ops->timedout_job)) { > + drm_dep_job_put(job); > + goto out; > + } > + > + status =3D q->ops->timedout_job(job); > + > + switch (status) { > + case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: > + scoped_guard(spinlock_irq, &q->job.lock) > + list_add(&job->pending_link, &q->job.pending); > + drm_dep_queue_put_job_queue(q); > + break; > + case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: > + drm_dep_job_put(job); > + break; > + default: > + WARN_ON("invalid drm_dep_timedout_stat"); > + break; > + } > + } > + > +out: > + drm_queue_start_timeout_unlocked(q); > + drm_dep_queue_put(q); > + dma_fence_end_signalling(cookie); > +} > + > +/** > + * drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue > + * @name: name for the workqueue > + * @flags: DRM_DEP_QUEUE_FLAGS_* flags > + * > + * Allocates an ordered workqueue for job submission with = %WQ_MEM_RECLAIM and > + * %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use = from > + * memory reclaim context and properly annotated for lockdep taint = tracking. > + * Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When > + * CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for = annotation. > + * > + * Context: Process context. > + * Return: the new &workqueue_struct, or NULL on failure. > + */ > +static struct workqueue_struct * > +drm_dep_alloc_submit_wq(const char *name, enum drm_dep_queue_flags = flags) > +{ > + unsigned int wq_flags =3D WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM; > + > + if (flags & DRM_DEP_QUEUE_FLAGS_HIGHPRI) > + wq_flags |=3D WQ_HIGHPRI; > + > +#if IS_ENABLED(CONFIG_LOCKDEP) > + static struct lockdep_map map =3D { > + .name =3D "drm_dep_submit_lockdep_map" > + }; > + return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map); > +#else > + return alloc_ordered_workqueue(name, wq_flags); > +#endif > +} > + > +/** > + * drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue > + * @name: name for the workqueue > + * > + * Allocates an ordered workqueue for timeout detection and recovery = with > + * %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring = consistent taint > + * annotation with the submit workqueue. When CONFIG_LOCKDEP is = enabled, uses > + * a dedicated lockdep map for annotation. > + * > + * Context: Process context. > + * Return: the new &workqueue_struct, or NULL on failure. > + */ > +static struct workqueue_struct *drm_dep_alloc_timeout_wq(const char = *name) > +{ > + unsigned int wq_flags =3D WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM; > + > +#if IS_ENABLED(CONFIG_LOCKDEP) > + static struct lockdep_map map =3D { > + .name =3D "drm_dep_timeout_lockdep_map" > + }; > + return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map); > +#else > + return alloc_ordered_workqueue(name, wq_flags); > +#endif > +} > + > +/** > + * drm_dep_queue_init() - initialize a dep queue > + * @q: dep queue to initialize > + * @args: initialization arguments > + * > + * Initializes all fields of @q from @args. If @args->submit_wq is = NULL an > + * ordered workqueue is allocated and owned by the queue > + * (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL = an > + * ordered workqueue is allocated and owned by the queue > + * (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds = one kref > + * reference and drm_dep_queue_put() must be called to drop this = reference > + * (i.e., drivers cannot directly free the queue). > + * > + * When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against = the > + * fs_reclaim pseudo-lock so that lockdep can detect any lock = ordering > + * inversion between @sched.lock and memory reclaim. > + * > + * Return: 0 on success, %-EINVAL when @args->credit_limit is zero, = @args->ops > + * is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when > + * @args->submit_wq or @args->timeout_wq is non-NULL but was not = allocated with > + * %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails. > + * > + * Context: Process context. May allocate memory and create = workqueues. > + */ > +int drm_dep_queue_init(struct drm_dep_queue *q, > + const struct drm_dep_queue_init_args *args) > +{ > + if (!args->credit_limit || !args->drm || !args->ops || > + !args->ops->run_job) > + return -EINVAL; > + > + if (args->submit_wq && = !workqueue_is_reclaim_annotated(args->submit_wq)) > + return -EINVAL; > + > + if (args->timeout_wq && > + !workqueue_is_reclaim_annotated(args->timeout_wq)) > + return -EINVAL; > + > + memset(q, 0, sizeof(*q)); > + > + q->name =3D args->name; > + q->drm =3D args->drm; > + q->credit.limit =3D args->credit_limit; > + q->job.timeout =3D args->timeout ? args->timeout : = MAX_SCHEDULE_TIMEOUT; > + > + init_rcu_head(&q->rcu); > + INIT_LIST_HEAD(&q->job.pending); > + spin_lock_init(&q->job.lock); > + spsc_queue_init(&q->job.queue); > + > + mutex_init(&q->sched.lock); > + if (IS_ENABLED(CONFIG_LOCKDEP)) { > + fs_reclaim_acquire(GFP_KERNEL); > + might_lock(&q->sched.lock); > + fs_reclaim_release(GFP_KERNEL); > + } > + > + if (args->submit_wq) { > + q->sched.submit_wq =3D args->submit_wq; > + } else { > + q->sched.submit_wq =3D drm_dep_alloc_submit_wq(args->name ?: = "drm_dep", > + args->flags); > + if (!q->sched.submit_wq) > + return -ENOMEM; > + > + q->sched.flags |=3D DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ; > + } > + > + if (args->timeout_wq) { > + q->sched.timeout_wq =3D args->timeout_wq; > + } else { > + q->sched.timeout_wq =3D drm_dep_alloc_timeout_wq(args->name ?: = "drm_dep"); > + if (!q->sched.timeout_wq) > + goto err_submit_wq; > + > + q->sched.flags |=3D DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ; > + } > + > + q->sched.flags |=3D args->flags & > + ~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ | > + DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ); > + > + INIT_DELAYED_WORK(&q->sched.tdr, drm_dep_queue_tdr_work); > + INIT_WORK(&q->sched.run_job, drm_dep_queue_run_job_work); > + INIT_WORK(&q->sched.put_job, drm_dep_queue_put_job_work); > + > + q->fence.context =3D dma_fence_context_alloc(1); > + > + kref_init(&q->refcount); > + q->ops =3D args->ops; > + drm_dev_get(q->drm); > + > + return 0; > + > +err_submit_wq: > + if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ) > + destroy_workqueue(q->sched.submit_wq); > + mutex_destroy(&q->sched.lock); > + > + return -ENOMEM; > +} > +EXPORT_SYMBOL(drm_dep_queue_init); > + > +#if IS_ENABLED(CONFIG_PROVE_LOCKING) > +/** > + * drm_dep_queue_push_job_begin() - mark the start of an arm/push = critical section > + * @q: dep queue the job belongs to > + * > + * Called at the start of drm_dep_job_arm() and warns if the push = context is > + * already owned by another task, which would indicate concurrent = arm/push on > + * the same queue. > + * > + * No-op when CONFIG_PROVE_LOCKING is disabled. > + * > + * Context: Process context. DMA fence signaling path. > + */ > +void drm_dep_queue_push_job_begin(struct drm_dep_queue *q) > +{ > + WARN_ON(q->job.push.owner); > + q->job.push.owner =3D current; > +} > + > +/** > + * drm_dep_queue_push_job_end() - mark the end of an arm/push = critical section > + * @q: dep queue the job belongs to > + * > + * Called at the end of drm_dep_job_push() and warns if the push = context is not > + * owned by the current task, which would indicate a mismatched = begin/end pair > + * or a push from the wrong thread. > + * > + * No-op when CONFIG_PROVE_LOCKING is disabled. > + * > + * Context: Process context. DMA fence signaling path. > + */ > +void drm_dep_queue_push_job_end(struct drm_dep_queue *q) > +{ > + WARN_ON(q->job.push.owner !=3D current); > + q->job.push.owner =3D NULL; > +} > +#endif > + > +/** > + * drm_dep_queue_assert_teardown_invariants() - assert teardown = invariants > + * @q: dep queue being torn down > + * > + * Warns if the pending-job list, the SPSC submission queue, or the = credit > + * counter is non-zero when called, or if the queue still has a = non-zero > + * reference count. > + * > + * Context: Any context. > + */ > +static void drm_dep_queue_assert_teardown_invariants(struct = drm_dep_queue *q) > +{ > + WARN_ON(!list_empty(&q->job.pending)); > + WARN_ON(spsc_queue_count(&q->job.queue)); > + WARN_ON(atomic_read(&q->credit.count)); > + WARN_ON(drm_dep_queue_refcount(q)); > +} > + > +/** > + * drm_dep_queue_release() - final internal cleanup of a dep queue > + * @q: dep queue to clean up > + * > + * Asserts teardown invariants and destroys internal resources = allocated by > + * drm_dep_queue_init() that cannot be torn down earlier in the = teardown > + * sequence. Currently this destroys @q->sched.lock. > + * > + * Drivers that implement &drm_dep_queue_ops.release **must** call = this > + * function after removing @q from any internal bookkeeping (e.g. = lookup > + * tables or lists) but before freeing the memory that contains @q. = When > + * &drm_dep_queue_ops.release is NULL, drm_dep follows the default = teardown > + * path and calls this function automatically. > + * > + * Context: Any context. > + */ > +void drm_dep_queue_release(struct drm_dep_queue *q) > +{ > + drm_dep_queue_assert_teardown_invariants(q); > + mutex_destroy(&q->sched.lock); > +} > +EXPORT_SYMBOL(drm_dep_queue_release); > + > +/** > + * drm_dep_queue_free() - final cleanup of a dep queue > + * @q: dep queue to free > + * > + * Invokes &drm_dep_queue_ops.release if set, in which case the = driver is > + * responsible for calling drm_dep_queue_release() and freeing @q = itself. > + * If &drm_dep_queue_ops.release is NULL, calls = drm_dep_queue_release() > + * and then frees @q with kfree_rcu(). > + * > + * In either case, releases the drm_dev_get() reference taken at init = time > + * via drm_dev_put(), allowing the owning &drm_device to be unloaded = once > + * all queues have been freed. > + * > + * Context: Process context (workqueue), reclaim safe. > + */ > +static void drm_dep_queue_free(struct drm_dep_queue *q) > +{ > + struct drm_device *drm =3D q->drm; > + > + if (q->ops->release) { > + q->ops->release(q); > + } else { > + drm_dep_queue_release(q); > + kfree_rcu(q, rcu); > + } > + drm_dev_put(drm); > +} > + > +/** > + * drm_dep_queue_free_work() - deferred queue teardown worker > + * @work: free_work item embedded in the dep queue > + * > + * Runs on dep_free_wq. Disables all work items synchronously > + * (preventing re-queue and waiting for in-flight instances), > + * destroys any owned workqueues, then calls drm_dep_queue_free(). > + * Running on dep_free_wq ensures destroy_workqueue() is never > + * called from within one of the queue's own workers (deadlock) > + * and disable_*_sync() cannot deadlock either. > + * > + * Context: Process context (workqueue), reclaim safe. > + */ > +static void drm_dep_queue_free_work(struct work_struct *work) > +{ > + struct drm_dep_queue *q =3D > + container_of(work, struct drm_dep_queue, free_work); > + > + drm_dep_queue_assert_teardown_invariants(q); > + > + disable_delayed_work_sync(&q->sched.tdr); > + disable_work_sync(&q->sched.run_job); > + disable_work_sync(&q->sched.put_job); > + > + if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ) > + destroy_workqueue(q->sched.timeout_wq); > + > + if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ) > + destroy_workqueue(q->sched.submit_wq); > + > + drm_dep_queue_free(q); > +} > + > +/** > + * drm_dep_queue_fini() - tear down a dep queue > + * @q: dep queue to tear down > + * > + * Asserts teardown invariants and nitiates teardown of @q by = queuing the > + * deferred free work onto tht module-private dep_free_wq workqueue. = The work > + * item disables any pending TDR and run/put-job work synchronously, = destroys > + * any workqueues that were allocated by drm_dep_queue_init(), and = then releases > + * the queue memory. > + * > + * Running teardown from dep_free_wq ensures that destroy_workqueue() = is never > + * called from within one of the queue's own workers (e.g. via > + * drm_dep_queue_put()), which would deadlock. > + * > + * Drivers can wait for all outstanding deferred work to complete by = waiting > + * for the last drm_dev_put() reference on their &drm_device, which = is > + * released as the final step of each queue's teardown. > + * > + * Drivers that implement &drm_dep_queue_ops.fini **must** call this > + * function after removing @q from any device bookkeeping but before = freeing the > + * memory that contains @q. When &drm_dep_queue_ops.fini is NULL, = drm_dep > + * follows the default teardown path and calls this function = automatically. > + * > + * Context: Any context. > + */ > +void drm_dep_queue_fini(struct drm_dep_queue *q) > +{ > + drm_dep_queue_assert_teardown_invariants(q); > + > + INIT_WORK(&q->free_work, drm_dep_queue_free_work); > + queue_work(dep_free_wq, &q->free_work); > +} > +EXPORT_SYMBOL(drm_dep_queue_fini); > + > +/** > + * drm_dep_queue_get() - acquire a reference to a dep queue > + * @q: dep queue to acquire a reference on, or NULL > + * > + * Return: @q with an additional reference held, or NULL if @q is = NULL. > + * > + * Context: Any context. > + */ > +struct drm_dep_queue *drm_dep_queue_get(struct drm_dep_queue *q) > +{ > + if (q) > + kref_get(&q->refcount); > + return q; > +} > +EXPORT_SYMBOL(drm_dep_queue_get); > + > +/** > + * __drm_dep_queue_release() - kref release callback for a dep queue > + * @kref: kref embedded in the dep queue > + * > + * Calls &drm_dep_queue_ops.fini if set, otherwise calls > + * drm_dep_queue_fini() to initiate deferred teardown. > + * > + * Context: Any context. > + */ > +static void __drm_dep_queue_release(struct kref *kref) > +{ > + struct drm_dep_queue *q =3D > + container_of(kref, struct drm_dep_queue, refcount); > + > + if (q->ops->fini) > + q->ops->fini(q); > + else > + drm_dep_queue_fini(q); > +} > + > +/** > + * drm_dep_queue_put() - release a reference to a dep queue > + * @q: dep queue to release a reference on, or NULL > + * > + * When the last reference is dropped, calls &drm_dep_queue_ops.fini = if set, > + * otherwise calls drm_dep_queue_fini(). Final memory release is = handled by > + * &drm_dep_queue_ops.release (which must call = drm_dep_queue_release()) if set, > + * or drm_dep_queue_release() followed by kfree_rcu() otherwise. > + * Does nothing if @q is NULL. > + * > + * Context: Any context. > + */ > +void drm_dep_queue_put(struct drm_dep_queue *q) > +{ > + if (q) > + kref_put(&q->refcount, __drm_dep_queue_release); > +} > +EXPORT_SYMBOL(drm_dep_queue_put); > + > +/** > + * drm_dep_queue_stop() - stop a dep queue from processing new jobs > + * @q: dep queue to stop > + * > + * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock = (mutex) > + * and @q->job.lock (spinlock_irq), making the flag safe to test from = finished > + * fenced signaling context. Then cancels any in-flight run_job and = put_job work > + * items. Once stopped, the bypass path and the submit workqueue will = not > + * dispatch further jobs nor will any jobs be removed from the = pending list. > + * Call drm_dep_queue_start() to resume processing. > + * > + * Context: Process context. Waits for in-flight workers to complete. > + */ > +void drm_dep_queue_stop(struct drm_dep_queue *q) > +{ > + scoped_guard(mutex, &q->sched.lock) { > + scoped_guard(spinlock_irq, &q->job.lock) > + drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_STOPPED); > + } > + cancel_work_sync(&q->sched.run_job); > + cancel_work_sync(&q->sched.put_job); > +} > +EXPORT_SYMBOL(drm_dep_queue_stop); > + > +/** > + * drm_dep_queue_start() - resume a stopped dep queue > + * @q: dep queue to start > + * > + * Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both = @q->sched.lock (mutex) > + * and @q->job.lock (spinlock_irq), making the flag safe to test from = IRQ > + * context. Then re-queues the run_job and put_job work items so that = any jobs > + * pending since the queue was stopped are processed. Must only be = called after > + * drm_dep_queue_stop(). > + * > + * Context: Process context. > + */ > +void drm_dep_queue_start(struct drm_dep_queue *q) > +{ > + scoped_guard(mutex, &q->sched.lock) { > + scoped_guard(spinlock_irq, &q->job.lock) > + drm_dep_queue_flags_clear(q, DRM_DEP_QUEUE_FLAGS_STOPPED); > + } > + drm_dep_queue_run_job_queue(q); > + drm_dep_queue_put_job_queue(q); > +} > +EXPORT_SYMBOL(drm_dep_queue_start); > + > +/** > + * drm_dep_queue_trigger_timeout() - trigger the TDR immediately for > + * all pending jobs > + * @q: dep queue to trigger timeout on > + * > + * Sets @q->job.timeout to 1 and arms the TDR delayed work with a = one-jiffy > + * delay, causing it to fire almost immediately without hot-spinning = at zero > + * delay. This is used to force-expire any pendind jobs on the queue, = for > + * example when the device is being torn down or has encountered an > + * unrecoverable error. > + * > + * It is suggested that when this function is used, the first = timedout_job call > + * causes the driver to kick the queue off the hardware and signal = all pending > + * job fences. Subsequent calls continue to signal all pending job = fences. > + * > + * Has no effect if the pending list is empty. > + * > + * Context: Any context. > + */ > +void drm_dep_queue_trigger_timeout(struct drm_dep_queue *q) > +{ > + guard(spinlock_irqsave)(&q->job.lock); > + q->job.timeout =3D 1; > + drm_queue_start_timeout(q); > +} > +EXPORT_SYMBOL(drm_dep_queue_trigger_timeout); > + > +/** > + * drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait > + * for it to finish > + * @q: dep queue whose TDR to cancel > + * > + * Cancels the TDR delayed work item if it has not yet started, and = waits for > + * it to complete if it is already running. After this call returns, = the TDR > + * worker is guaranteed not to be executing and will not fire again = until > + * explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by = a new > + * job being submitted). > + * > + * Useful during error recovery or queue teardown when the caller = needs to > + * know that no timeout handling races with its own reset logic. > + * > + * Context: Process context. May sleep waiting for the TDR worker to = finish. > + */ > +void drm_dep_queue_cancel_tdr_sync(struct drm_dep_queue *q) > +{ > + cancel_delayed_work_sync(&q->sched.tdr); > +} > +EXPORT_SYMBOL(drm_dep_queue_cancel_tdr_sync); > + > +/** > + * drm_dep_queue_resume_timeout() - restart the TDR timer with the > + * configured timeout > + * @q: dep queue to resume the timeout for > + * > + * Restarts the TDR delayed work using @q->job.timeout. Called after = device > + * recovery to give pending jobs a fresh full timeout window. Has no = effect > + * if the pending list is empty. > + * > + * Context: Any context. > + */ > +void drm_dep_queue_resume_timeout(struct drm_dep_queue *q) > +{ > + drm_queue_start_timeout_unlocked(q); > +} > +EXPORT_SYMBOL(drm_dep_queue_resume_timeout); > + > +/** > + * drm_dep_queue_is_stopped() - check whether a dep queue is stopped > + * @q: dep queue to check > + * > + * Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false = otherwise. > + * > + * Context: Any context. > + */ > +bool drm_dep_queue_is_stopped(struct drm_dep_queue *q) > +{ > + return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_STOPPED); > +} > +EXPORT_SYMBOL(drm_dep_queue_is_stopped); > + > +/** > + * drm_dep_queue_kill() - kill a dep queue and flush all pending jobs > + * @q: dep queue to kill > + * > + * Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock. If a > + * dependency fence is currently being waited on, its callback is = removed and > + * the run-job worker is kicked immediately so that the blocked job = drains > + * without waiting. > + * > + * Once killed, drm_dep_queue_job_dependency() returns NULL for all = jobs, > + * bypassing dependency waits so that every queued job drains through > + * &drm_dep_queue_ops.run_job without blocking. > + * > + * The &drm_dep_queue_ops.run_job callback is guaranteed to be called = for every > + * job that was pushed before or after drm_dep_queue_kill(), even = during queue > + * teardown. Drivers should use this guarantee to perform any = necessary > + * bookkeeping cleanup without executing the actual backend operation = when the > + * queue is killed. > + * > + * Unlike drm_dep_queue_stop(), killing is one-way: there is no = corresponding > + * start function. > + * > + * **Driver safety requirement** > + * > + * drm_dep_queue_kill() must only be called once the driver can = guarantee that > + * no job in the queue will touch memory associated with any of its = fences > + * (i.e., the queue has been removed from the device and will never = be put back > + * on). > + * > + * Context: Process context. > + */ > +void drm_dep_queue_kill(struct drm_dep_queue *q) > +{ > + scoped_guard(mutex, &q->sched.lock) { > + struct dma_fence *fence; > + > + drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_KILLED); > + > + /* > + * Holding &q->sched.lock guarantees that the run-job work item > + * cannot drop its reference to q->dep.fence concurrently, so > + * reading q->dep.fence here is safe. > + */ > + fence =3D READ_ONCE(q->dep.fence); > + if (fence && dma_fence_remove_callback(fence, &q->dep.cb)) > + drm_dep_queue_remove_dependency(q, fence); > + } > +} > +EXPORT_SYMBOL(drm_dep_queue_kill); > + > +/** > + * drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep = queue > + * @q: dep queue whose workqueue to retrieve > + * > + * Drivers may use this to queue their own work items alongside the = queue's > + * internal run-job and put-job workers =E2=80=94 for example to = process incoming > + * messages in the same serialisation domain. > + * > + * Prefer drm_dep_queue_work_enqueue() when the only need is to = enqueue a > + * work item, as it additionally checks the stopped state. Use this = accessor > + * when the workqueue itself is required (e.g. for = alloc_ordered_workqueue > + * replacement or drain_workqueue calls). > + * > + * Context: Any context. > + * Return: the &workqueue_struct used by @q for job submission. > + */ > +struct workqueue_struct *drm_dep_queue_submit_wq(struct drm_dep_queue = *q) > +{ > + return q->sched.submit_wq; > +} > +EXPORT_SYMBOL(drm_dep_queue_submit_wq); > + > +/** > + * drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a = dep queue > + * @q: dep queue whose workqueue to retrieve > + * > + * Returns the workqueue used by @q to run TDR (timeout detection and = recovery) > + * work. Drivers may use this to queue their own timeout-domain work = items, or > + * to call drain_workqueue() when tearing down and needing to ensure = all pending > + * timeout callbacks have completed before proceeding. > + * > + * Context: Any context. > + * Return: the &workqueue_struct used by @q for TDR work. > + */ > +struct workqueue_struct *drm_dep_queue_timeout_wq(struct = drm_dep_queue *q) > +{ > + return q->sched.timeout_wq; > +} > +EXPORT_SYMBOL(drm_dep_queue_timeout_wq); > + > +/** > + * drm_dep_queue_work_enqueue() - queue work on the dep queue's = submit workqueue > + * @q: dep queue to enqueue work on > + * @work: work item to enqueue > + * > + * Queues @work on @q->sched.submit_wq if the queue is not stopped. = This > + * allows drivers to schedule custom work items that run serialised = with the > + * queue's own run-job and put-job workers. > + * > + * Return: true if the work was queued, false if the queue is stopped = or the > + * work item was already pending. > + * > + * Context: Any context. > + */ > +bool drm_dep_queue_work_enqueue(struct drm_dep_queue *q, > + struct work_struct *work) > +{ > + if (drm_dep_queue_is_stopped(q)) > + return false; > + > + return queue_work(q->sched.submit_wq, work); > +} > +EXPORT_SYMBOL(drm_dep_queue_work_enqueue); > + > +/** > + * drm_dep_queue_can_job_bypass() - test whether a job can skip the = SPSC queue > + * @q: dep queue > + * @job: job to test > + * > + * A job may bypass the submit workqueue and run inline on the = calling thread > + * if all of the following hold: > + * > + * - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue > + * - the queue is not stopped > + * - the SPSC submission queue is empty (no other jobs waiting) > + * - the queue has enough credits for @job > + * - @job has no unresolved dependency fences > + * > + * Must be called under @q->sched.lock. > + * > + * Context: Process context. Must hold @q->sched.lock (a mutex). > + * Return: true if the job may be run inline, false otherwise. > + */ > +bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q, > + struct drm_dep_job *job) > +{ > + lockdep_assert_held(&q->sched.lock); > + > + return q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED && > + !drm_dep_queue_is_stopped(q) && > + !spsc_queue_count(&q->job.queue) && > + drm_dep_queue_has_credits(q, job) && > + xa_empty(&job->dependencies); > +} > + > +/** > + * drm_dep_job_done() - mark a job as complete > + * @job: the job that finished > + * @result: error code to propagate, or 0 for success > + * > + * Subtracts @job->credits from the queue credit counter, then = signals the > + * job's dep fence with @result. > + * > + * When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), = a > + * temporary extra reference is taken on @job before signalling the = fence. > + * This prevents a concurrent put-job worker =E2=80=94 which may be = woken by timeouts or > + * queue starting =E2=80=94 from freeing the job while this function = still holds a > + * pointer to it. The extra reference is released at the end of the = function. > + * > + * After signalling, the IRQ-safe path removes the job from the = pending list > + * under @q->job.lock, provided the queue is not stopped. Removal is = skipped > + * when the queue is stopped so that = drm_dep_queue_for_each_pending_job() can > + * iterate the list without racing with the completion path. On = successful > + * removal, kicks the run-job worker so the next queued job can be = dispatched > + * immediately, then drops the job reference. If the job was already = removed > + * by TDR, or removal was skipped because the queue is stopped, kicks = the > + * put-job worker instead to allow the deferred put to complete. > + * > + * Context: Any context. > + */ > +static void drm_dep_job_done(struct drm_dep_job *job, int result) > +{ > + struct drm_dep_queue *q =3D job->q; > + bool irq_safe =3D drm_dep_queue_is_job_put_irq_safe(q), removed =3D = false; > + > + /* > + * Local ref to ensure the put worker=E2=80=94which may be woken by = external > + * forces (TDR, driver-side queue starting)=E2=80=94doesn't free the = job behind > + * this function's back after drm_dep_fence_done() while it is still = on > + * the pending list. > + */ > + if (irq_safe) > + drm_dep_job_get(job); > + > + atomic_sub(job->credits, &q->credit.count); > + drm_dep_fence_done(job->dfence, result); > + > + /* Only safe to touch job after fence signal if we have a local ref. = */ > + > + if (irq_safe) { > + scoped_guard(spinlock_irqsave, &q->job.lock) { > + removed =3D !list_empty(&job->pending_link) && > + !drm_dep_queue_is_stopped(q); > + > + /* Guard against TDR operating on job */ > + if (removed) > + drm_dep_queue_remove_job(q, job); > + } > + } > + > + if (removed) { > + drm_dep_queue_run_job_queue(q); > + drm_dep_job_put(job); > + } else { > + drm_dep_queue_put_job_queue(q); > + } > + > + if (irq_safe) > + drm_dep_job_put(job); > +} > + > +/** > + * drm_dep_job_done_cb() - dma_fence callback to complete a job > + * @f: the hardware fence that signalled > + * @cb: fence callback embedded in the dep job > + * > + * Extracts the job from @cb and calls drm_dep_job_done() with > + * @f->error as the result. > + * > + * Context: Any context, but with IRQ disabled. May not sleep. > + */ > +static void drm_dep_job_done_cb(struct dma_fence *f, struct = dma_fence_cb *cb) > +{ > + struct drm_dep_job *job =3D container_of(cb, struct drm_dep_job, = cb); > + > + drm_dep_job_done(job, f->error); > +} > + > +/** > + * drm_dep_queue_run_job() - submit a job to hardware and set up > + * completion tracking > + * @q: dep queue > + * @job: job to run > + * > + * Accounts @job->credits against the queue, appends the job to the = pending > + * list, then calls @q->ops->run_job(). The TDR timer is started only = when > + * @job is the first entry on the pending list; subsequent jobs added = while > + * a TDR is already in flight do not reset the timer (which would = otherwise > + * extend the deadline for the already-running head job). Stores the = returned > + * hardware fence as the parent of the job's dep fence, then installs > + * drm_dep_job_done_cb() on it. If the hardware fence is already = signalled > + * (%-ENOENT from dma_fence_add_callback()) or run_job() returns = NULL/error, > + * the job is completed immediately. Must be called under = @q->sched.lock. > + * > + * Context: Process context. Must hold @q->sched.lock (a mutex). DMA = fence > + * signaling path. > + */ > +void drm_dep_queue_run_job(struct drm_dep_queue *q, struct = drm_dep_job *job) > +{ > + struct dma_fence *fence; > + int r; > + > + lockdep_assert_held(&q->sched.lock); > + > + drm_dep_job_get(job); > + atomic_add(job->credits, &q->credit.count); > + > + scoped_guard(spinlock_irq, &q->job.lock) { > + bool first =3D list_empty(&q->job.pending); > + > + list_add_tail(&job->pending_link, &q->job.pending); > + if (first) > + drm_queue_start_timeout(q); > + } > + > + fence =3D q->ops->run_job(job); > + drm_dep_fence_set_parent(job->dfence, fence); > + > + if (!IS_ERR_OR_NULL(fence)) { > + r =3D dma_fence_add_callback(fence, &job->cb, > + drm_dep_job_done_cb); > + if (r =3D=3D -ENOENT) > + drm_dep_job_done(job, fence->error); > + else if (r) > + drm_err(q->drm, "fence add callback failed (%d)\n", r); > + dma_fence_put(fence); > + } else { > + drm_dep_job_done(job, IS_ERR(fence) ? PTR_ERR(fence) : 0); > + } > + > + /* > + * Drop all input dependency fences now, in process context, before = the > + * final job put. Once the job is on the pending list its last = reference > + * may be dropped from a dma_fence callback (IRQ context), where = calling > + * xa_destroy() would be unsafe. > + */ I assume that =E2=80=9Cpending=E2=80=9D is the list of jobs that have = been handed to the driver via ops->run_job()? Can=E2=80=99t this problem be solved by not doing anything inside a = dma_fence callback other than scheduling the queue worker? > + drm_dep_job_drop_dependencies(job); > + drm_dep_job_put(job); > +} > + > +/** > + * drm_dep_queue_push_job() - enqueue a job on the SPSC submission = queue > + * @q: dep queue > + * @job: job to push > + * > + * Pushes @job onto the SPSC queue. If the queue was previously empty > + * (i.e. this is the first pending job), kicks the run_job worker so = it > + * processes the job promptly without waiting for the next wakeup. > + * May be called with or without @q->sched.lock held. > + * > + * Context: Any context. DMA fence signaling path. > + */ > +void drm_dep_queue_push_job(struct drm_dep_queue *q, struct = drm_dep_job *job) > +{ > + /* > + * spsc_queue_push() returns true if the queue was previously empty, > + * i.e. this is the first pending job. Kick the run_job worker so it > + * picks it up without waiting for the next wakeup. > + */ > + if (spsc_queue_push(&q->job.queue, &job->queue_node)) > + drm_dep_queue_run_job_queue(q); > +} > + > +/** > + * drm_dep_init() - module initialiser > + * > + * Allocates the module-private dep_free_wq unbound workqueue used = for > + * deferred queue teardown. > + * > + * Return: 0 on success, %-ENOMEM if workqueue allocation fails. > + */ > +static int __init drm_dep_init(void) > +{ > + dep_free_wq =3D alloc_workqueue("drm_dep_free", WQ_UNBOUND, 0); > + if (!dep_free_wq) > + return -ENOMEM; > + > + return 0; > +} > + > +/** > + * drm_dep_exit() - module exit > + * > + * Destroys the module-private dep_free_wq workqueue. > + */ > +static void __exit drm_dep_exit(void) > +{ > + destroy_workqueue(dep_free_wq); > + dep_free_wq =3D NULL; > +} > + > +module_init(drm_dep_init); > +module_exit(drm_dep_exit); > + > +MODULE_DESCRIPTION("DRM dependency queue"); > +MODULE_LICENSE("Dual MIT/GPL"); > diff --git a/drivers/gpu/drm/dep/drm_dep_queue.h = b/drivers/gpu/drm/dep/drm_dep_queue.h > new file mode 100644 > index 000000000000..e5c217a3fab5 > --- /dev/null > +++ b/drivers/gpu/drm/dep/drm_dep_queue.h > @@ -0,0 +1,31 @@ > +/* SPDX-License-Identifier: MIT */ > +/* > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +#ifndef _DRM_DEP_QUEUE_H_ > +#define _DRM_DEP_QUEUE_H_ > + > +#include > + > +struct drm_dep_job; > +struct drm_dep_queue; > + > +bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q, > + struct drm_dep_job *job); > +void drm_dep_queue_run_job(struct drm_dep_queue *q, struct = drm_dep_job *job); > +void drm_dep_queue_push_job(struct drm_dep_queue *q, struct = drm_dep_job *job); > + > +#if IS_ENABLED(CONFIG_PROVE_LOCKING) > +void drm_dep_queue_push_job_begin(struct drm_dep_queue *q); > +void drm_dep_queue_push_job_end(struct drm_dep_queue *q); > +#else > +static inline void drm_dep_queue_push_job_begin(struct drm_dep_queue = *q) > +{ > +} > +static inline void drm_dep_queue_push_job_end(struct drm_dep_queue = *q) > +{ > +} > +#endif > + > +#endif /* _DRM_DEP_QUEUE_H_ */ > diff --git a/include/drm/drm_dep.h b/include/drm/drm_dep.h > new file mode 100644 > index 000000000000..615926584506 > --- /dev/null > +++ b/include/drm/drm_dep.h > @@ -0,0 +1,597 @@ > +/* SPDX-License-Identifier: MIT */ > +/* > + * Copyright 2015 Advanced Micro Devices, Inc. > + * > + * Permission is hereby granted, free of charge, to any person = obtaining a > + * copy of this software and associated documentation files (the = "Software"), > + * to deal in the Software without restriction, including without = limitation > + * the rights to use, copy, modify, merge, publish, distribute, = sublicense, > + * and/or sell copies of the Software, and to permit persons to whom = the > + * Software is furnished to do so, subject to the following = conditions: > + * > + * The above copyright notice and this permission notice shall be = included in > + * all copies or substantial portions of the Software. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, = EXPRESS OR > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF = MERCHANTABILITY, > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT = SHALL > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, = DAMAGES OR > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR = OTHERWISE, > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE = OR > + * OTHER DEALINGS IN THE SOFTWARE. > + * > + * Copyright =C2=A9 2026 Intel Corporation > + */ > + > +#ifndef _DRM_DEP_H_ > +#define _DRM_DEP_H_ > + > +#include > +#include > +#include > +#include > + > +enum dma_resv_usage; > +struct dma_resv; > +struct drm_dep_fence; > +struct drm_dep_job; > +struct drm_dep_queue; > +struct drm_file; > +struct drm_gem_object; > + > +/** > + * enum drm_dep_timedout_stat - return value of = &drm_dep_queue_ops.timedout_job > + * @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's = finished > + * fence during reset; drm_dep may safely drop its reference to the = job. > + * @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; = reinsert the > + * job at the head of the pending list so it can complete normally. > + */ > +enum drm_dep_timedout_stat { > + DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, > + DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, > +}; > + > +/** > + * struct drm_dep_queue_ops - driver callbacks for a dep queue > + */ > +struct drm_dep_queue_ops { > + /** > + * @run_job: submit the job to hardware. Returns the hardware = completion > + * fence (with a reference held for the scheduler), or NULL/ERR_PTR = on > + * synchronous completion or error. > + */ > + struct dma_fence *(*run_job)(struct drm_dep_job *job); > + > + /** > + * @timedout_job: called when the TDR fires for the head job. Must = stop > + * the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if = the > + * job's fence was signalled during reset, or > + * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or > + * signalling was otherwise delayed, and the job should be = re-inserted > + * at the head of the pending list. Any other value triggers a WARN. > + */ > + enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job); > + > + /** > + * @release: called when the last kref on the queue is dropped and > + * drm_dep_queue_fini() has completed. The driver is responsible for > + * removing @q from any internal bookkeeping, calling > + * drm_dep_queue_release(), and then freeing the memory containing @q > + * (e.g. via kfree_rcu() using @q->rcu). If NULL, drm_dep calls > + * drm_dep_queue_release() and frees @q automatically via = kfree_rcu(). > + * Use this when the queue is embedded in a larger structure. > + */ > + void (*release)(struct drm_dep_queue *q); > + > + /** > + * @fini: if set, called instead of drm_dep_queue_fini() when the = last > + * kref is dropped. The driver is responsible for calling > + * drm_dep_queue_fini() itself after it is done with the queue. Use = this > + * when additional teardown logic must run before fini (e.g., cleanup > + * firmware resources associated with the queue). > + */ > + void (*fini)(struct drm_dep_queue *q); > +}; > + > +/** > + * enum drm_dep_queue_flags - flags for &drm_dep_queue and > + * &drm_dep_queue_init_args > + * > + * Flags are divided into three categories: > + * > + * - **Private static**: set internally at init time and never = changed. > + * Drivers must not read or write these. > + * %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ, > + * %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ. > + * > + * - **Public dynamic**: toggled at runtime by drivers via accessors. > + * Any modification must be performed under = &drm_dep_queue.sched.lock. Can=E2=80=99t enforce that in C. > + * Accessor functions provide unstable reads. > + * %DRM_DEP_QUEUE_FLAGS_STOPPED, > + * %DRM_DEP_QUEUE_FLAGS_KILLED. > + * > + * - **Public static**: supplied by the driver in > + * &drm_dep_queue_init_args.flags at queue creation time and not = modified > + * thereafter. Same here. > + * %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED, > + * %DRM_DEP_QUEUE_FLAGS_HIGHPRI, > + * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE. > + * > + * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit = workqueue was > + * allocated by drm_dep_queue_init() and will be destroyed by > + * drm_dep_queue_fini(). > + * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout = workqueue > + * was allocated by drm_dep_queue_init() and will be destroyed by > + * drm_dep_queue_fini(). > + * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is = stopped and > + * will not dispatch new jobs or remove jobs from the pending list, = dropping > + * the drm_dep-owned reference. Set by drm_dep_queue_stop(), = cleared by > + * drm_dep_queue_start(). > + * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been = killed > + * via drm_dep_queue_kill(). Any active dependency wait is = cancelled > + * immediately. Jobs continue to flow through run_job for = bookkeeping > + * cleanup, but dependency waiting is skipped so that queued work = drains > + * as quickly as possible. > + * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue = supports > + * the bypass path where eligible jobs skip the SPSC queue and run = inline. > + * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit = workqueue owned > + * by the queue is created with %WQ_HIGHPRI, causing run-job and = put-job > + * workers to execute at elevated priority. Only privileged clients = (e.g. > + * drivers managing time-critical or real-time GPU contexts) should = request > + * this flag; granting it to unprivileged userspace would allow = priority > + * inversion attacks. > + * @drm_dep_queue_init_args.submit_wq is provided. > + * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set, > + * drm_dep_job_done() may be called from hardirq context (e.g. from = a > + * hardware-signalled dma_fence callback). drm_dep_job_done() will = directly > + * dequeue the job and call drm_dep_job_put() without deferring to = a > + * workqueue. The driver's &drm_dep_job_ops.release callback must = therefore > + * be safe to invoke from IRQ context. > + */ > +enum drm_dep_queue_flags { > + DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ =3D BIT(0), > + DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ =3D BIT(1), > + DRM_DEP_QUEUE_FLAGS_STOPPED =3D BIT(2), > + DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED =3D BIT(3), > + DRM_DEP_QUEUE_FLAGS_HIGHPRI =3D BIT(4), > + DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE =3D BIT(5), > + DRM_DEP_QUEUE_FLAGS_KILLED =3D BIT(6), > +}; > + > +/** > + * struct drm_dep_queue - a dependency-tracked GPU submission queue > + * > + * Combines the role of &drm_gpu_scheduler and &drm_sched_entity into = a single > + * object. Each queue owns a submit workqueue (or borrows one), a = timeout > + * workqueue, an SPSC submission queue, and a pending-job list used = for TDR. > + * > + * Initialise with drm_dep_queue_init(), tear down with = drm_dep_queue_fini(). > + * Reference counted via drm_dep_queue_get() / drm_dep_queue_put(). > + * > + * All fields are **opaque to drivers**. Do not read or write any = field Can=E2=80=99t enforce this in C. > + * directly; use the provided helper functions instead. The sole = exception > + * is @rcu, which drivers may pass to kfree_rcu() when the queue is = embedded > + * inside a larger driver-managed structure and the = &drm_dep_queue_ops.release > + * vfunc performs an RCU-deferred free. > + */ > +struct drm_dep_queue { > + /** @ops: driver callbacks, set at init time. */ > + const struct drm_dep_queue_ops *ops; > + /** @name: human-readable name used for workqueue and fence naming. = */ > + const char *name; > + /** @drm: owning DRM device; a drm_dev_get() reference is held for = the > + * lifetime of the queue to prevent module unload while queues are = live. > + */ > + struct drm_device *drm; > + /** @refcount: reference count; use drm_dep_queue_get/put(). */ > + struct kref refcount; > + /** > + * @free_work: deferred teardown work queued unconditionally by > + * drm_dep_queue_fini() onto the module-private dep_free_wq. The = work > + * item disables pending workers synchronously and destroys any owned > + * workqueues before releasing the queue memory and dropping the > + * drm_dev_get() reference. Running on dep_free_wq ensures > + * destroy_workqueue() is never called from within one of the queue's > + * own workers. > + */ > + struct work_struct free_work; > + /** > + * @rcu: RCU head for deferred freeing. > + * > + * This is the **only** field drivers may access directly. When the We can enforce this in Rust at compile time. > + * queue is embedded in a larger structure, implement > + * &drm_dep_queue_ops.release, call drm_dep_queue_release() to = destroy > + * internal resources, then pass this field to kfree_rcu() so that = any > + * in-flight RCU readers referencing the queue's dma_fence timeline = name > + * complete before the memory is returned. All other fields must be > + * accessed through the provided helpers. > + */ > + struct rcu_head rcu; > + > + /** @sched: scheduling and workqueue state. */ > + struct { > + /** @sched.submit_wq: ordered workqueue for run/put-job work. */ > + struct workqueue_struct *submit_wq; > + /** @sched.timeout_wq: workqueue for the TDR delayed work. */ > + struct workqueue_struct *timeout_wq; > + /** > + * @sched.run_job: work item that dispatches the next queued > + * job. > + */ > + struct work_struct run_job; > + /** @sched.put_job: work item that frees finished jobs. */ > + struct work_struct put_job; > + /** @sched.tdr: delayed work item for timeout/reset (TDR). */ > + struct delayed_work tdr; > + /** > + * @sched.lock: mutex serialising job dispatch, bypass > + * decisions, stop/start, and flag updates. > + */ > + struct mutex lock; > + /** > + * @sched.flags: bitmask of &enum drm_dep_queue_flags. > + * Any modification after drm_dep_queue_init() must be > + * performed under @sched.lock. > + */ > + enum drm_dep_queue_flags flags; > + } sched; > + > + /** @job: pending-job tracking state. */ > + struct { > + /** > + * @job.pending: list of jobs that have been dispatched to > + * hardware and not yet freed. Protected by @job.lock. > + */ > + struct list_head pending; > + /** > + * @job.queue: SPSC queue of jobs waiting to be dispatched. > + * Producers push via drm_dep_queue_push_job(); the run_job > + * work item pops from the consumer side. > + */ > + struct spsc_queue queue; > + /** > + * @job.lock: spinlock protecting @job.pending, TDR start, and > + * the %DRM_DEP_QUEUE_FLAGS_STOPPED flag. Always acquired with > + * irqsave (spin_lock_irqsave / spin_unlock_irqrestore) to > + * support %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE queues where > + * drm_dep_job_done() may run from hardirq context. > + */ > + spinlock_t lock; > + /** > + * @job.timeout: per-job TDR timeout in jiffies. > + * %MAX_SCHEDULE_TIMEOUT means no timeout. > + */ > + long timeout; > +#if IS_ENABLED(CONFIG_PROVE_LOCKING) > + /** > + * @job.push: lockdep annotation tracking the arm-to-push > + * critical section. > + */ > + struct { > + /* > + * @job.push.owner: task that currently holds the push > + * context, used to assert single-owner invariants. > + * NULL when idle. > + */ > + struct task_struct *owner; > + } push; > +#endif > + } job; > + > + /** @credit: hardware credit accounting. */ > + struct { > + /** @credit.limit: maximum credits the queue can hold. */ > + u32 limit; > + /** @credit.count: credits currently in flight (atomic). */ > + atomic_t count; > + } credit; > + > + /** @dep: current blocking dependency for the head SPSC job. */ > + struct { > + /** > + * @dep.fence: fence being waited on before the head job can > + * run. NULL when no dependency is pending. > + */ > + struct dma_fence *fence; > + /** > + * @dep.removed_fence: dependency fence whose callback has been > + * removed. The run-job worker must drop its reference to this > + * fence before proceeding to call run_job. We can enforce this in Rust automatically. > + */ > + struct dma_fence *removed_fence; > + /** @dep.cb: callback installed on @dep.fence. */ > + struct dma_fence_cb cb; > + } dep; > + > + /** @fence: fence context and sequence number state. */ > + struct { > + /** > + * @fence.seqno: next sequence number to assign, incremented > + * each time a job is armed. > + */ > + u32 seqno; > + /** > + * @fence.context: base DMA fence context allocated at init > + * time. Finished fences use this context. > + */ > + u64 context; > + } fence; > +}; > + > +/** > + * struct drm_dep_queue_init_args - arguments for = drm_dep_queue_init() > + */ > +struct drm_dep_queue_init_args { > + /** @ops: driver callbacks; must not be NULL. */ > + const struct drm_dep_queue_ops *ops; > + /** @name: human-readable name for workqueues and fence timelines. = */ > + const char *name; > + /** @drm: owning DRM device. A drm_dev_get() reference is taken at > + * queue init and released when the queue is freed, preventing = module > + * unload while any queue is still alive. > + */ > + struct drm_device *drm; > + /** > + * @submit_wq: workqueue for job dispatch. If NULL, an ordered > + * workqueue is allocated and owned by the queue. If non-NULL, the > + * workqueue must have been allocated with %WQ_MEM_RECLAIM_TAINT; > + * drm_dep_queue_init() returns %-EINVAL otherwise. > + */ > + struct workqueue_struct *submit_wq; > + /** > + * @timeout_wq: workqueue for TDR. If NULL, an ordered workqueue > + * is allocated and owned by the queue. If non-NULL, the workqueue > + * must have been allocated with %WQ_MEM_RECLAIM_TAINT; > + * drm_dep_queue_init() returns %-EINVAL otherwise. > + */ > + struct workqueue_struct *timeout_wq; > + /** @credit_limit: maximum hardware credits; must be non-zero. */ > + u32 credit_limit; > + /** > + * @timeout: per-job TDR timeout in jiffies. Zero means no timeout > + * (%MAX_SCHEDULE_TIMEOUT is used internally). > + */ > + long timeout; > + /** > + * @flags: initial queue flags. %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ > + * and %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ are managed internally > + * and will be ignored if set here. Setting > + * %DRM_DEP_QUEUE_FLAGS_HIGHPRI requests a high-priority submit > + * workqueue; drivers must only set this for privileged clients. > + */ > + enum drm_dep_queue_flags flags; > +}; > + > +/** > + * struct drm_dep_job_ops - driver callbacks for a dep job > + */ > +struct drm_dep_job_ops { > + /** > + * @release: called when the last reference to the job is dropped. > + * > + * If set, the driver is responsible for freeing the job. If NULL, And if they don=E2=80=99t? By the way, we can also enforce this in Rust. > + * drm_dep_job_put() will call kfree() on the job directly. > + */ > + void (*release)(struct drm_dep_job *job); > +}; > + > +/** > + * struct drm_dep_job - a unit of work submitted to a dep queue > + * > + * All fields are **opaque to drivers**. Do not read or write any = field > + * directly; use the provided helper functions instead. > + */ > +struct drm_dep_job { > + /** @ops: driver callbacks for this job. */ > + const struct drm_dep_job_ops *ops; > + /** @refcount: reference count, managed by drm_dep_job_get/put(). */ > + struct kref refcount; > + /** > + * @dependencies: xarray of &dma_fence dependencies before the job = can > + * run. > + */ > + struct xarray dependencies; > + /** @q: the queue this job is submitted to. */ > + struct drm_dep_queue *q; > + /** @queue_node: SPSC queue linkage for pending submission. */ > + struct spsc_node queue_node; > + /** > + * @pending_link: list entry in the queue's pending job list. = Protected > + * by @job.q->job.lock. > + */ > + struct list_head pending_link; > + /** @dfence: finished fence for this job. */ > + struct drm_dep_fence *dfence; > + /** @cb: fence callback used to watch for dependency completion. */ > + struct dma_fence_cb cb; > + /** @credits: number of credits this job consumes from the queue. */ > + u32 credits; > + /** > + * @last_dependency: index into @dependencies of the next fence to > + * check. Advanced by drm_dep_queue_job_dependency() as each > + * dependency is consumed. > + */ > + u32 last_dependency; > + /** > + * @invalidate_count: number of times this job has been invalidated. > + * Incremented by drm_dep_job_invalidate_job(). > + */ > + u32 invalidate_count; > + /** > + * @signalling_cookie: return value of dma_fence_begin_signalling() > + * captured in drm_dep_job_arm() and consumed by drm_dep_job_push(). > + * Not valid outside the arm=E2=86=92push window. > + */ > + bool signalling_cookie; > +}; > + > +/** > + * struct drm_dep_job_init_args - arguments for drm_dep_job_init() > + */ > +struct drm_dep_job_init_args { > + /** > + * @ops: driver callbacks for the job, or NULL for default behaviour. > + */ > + const struct drm_dep_job_ops *ops; > + /** @q: the queue to associate the job with. A reference is taken. = */ > + struct drm_dep_queue *q; > + /** @credits: number of credits this job consumes; must be non-zero. = */ > + u32 credits; > +}; > + > +/* Queue API */ > + > +/** > + * drm_dep_queue_sched_guard() - acquire the queue scheduler lock as = a guard > + * @__q: dep queue whose scheduler lock to acquire > + * > + * Acquires @__q->sched.lock as a scoped mutex guard (released = automatically > + * when the enclosing scope exits). This lock serialises all = scheduler state > + * transitions =E2=80=94 stop/start/kill flag changes, bypass-path = decisions, and the > + * run-job worker =E2=80=94 so it must be held when the driver needs = to atomically > + * inspect or modify queue state in relation to job submission. > + * > + * **When to use** > + * > + * Drivers that set %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED and wish to > + * serialise their own submit work against the bypass path must = acquire this > + * guard. Without it, a concurrent caller of drm_dep_job_push() = could take > + * the bypass path and call ops->run_job() inline between the = driver's > + * eligibility check and its corresponding action, producing a race. So if you=E2=80=99re not careful, you have just introduced a race :/ > + * > + * **Constraint: only from submit_wq worker context** > + * > + * This guard must only be acquired from a work item running on the = queue's > + * submit workqueue (@q->sched.submit_wq) by drivers. > + * > + * Context: Process context only; must be called from submit_wq work = by > + * drivers. > + */ > +#define drm_dep_queue_sched_guard(__q) \ > + guard(mutex)(&(__q)->sched.lock) > + > +int drm_dep_queue_init(struct drm_dep_queue *q, > + const struct drm_dep_queue_init_args *args); > +void drm_dep_queue_fini(struct drm_dep_queue *q); > +void drm_dep_queue_release(struct drm_dep_queue *q); > +struct drm_dep_queue *drm_dep_queue_get(struct drm_dep_queue *q); > +bool drm_dep_queue_get_unless_zero(struct drm_dep_queue *q); > +void drm_dep_queue_put(struct drm_dep_queue *q); > +void drm_dep_queue_stop(struct drm_dep_queue *q); > +void drm_dep_queue_start(struct drm_dep_queue *q); > +void drm_dep_queue_kill(struct drm_dep_queue *q); > +void drm_dep_queue_trigger_timeout(struct drm_dep_queue *q); > +void drm_dep_queue_cancel_tdr_sync(struct drm_dep_queue *q); > +void drm_dep_queue_resume_timeout(struct drm_dep_queue *q); > +bool drm_dep_queue_work_enqueue(struct drm_dep_queue *q, > + struct work_struct *work); > +bool drm_dep_queue_is_stopped(struct drm_dep_queue *q); > +bool drm_dep_queue_is_killed(struct drm_dep_queue *q); > +bool drm_dep_queue_is_initialized(struct drm_dep_queue *q); > +void drm_dep_queue_set_stopped(struct drm_dep_queue *q); > +unsigned int drm_dep_queue_refcount(const struct drm_dep_queue *q); > +long drm_dep_queue_timeout(const struct drm_dep_queue *q); > +struct workqueue_struct *drm_dep_queue_submit_wq(struct drm_dep_queue = *q); > +struct workqueue_struct *drm_dep_queue_timeout_wq(struct = drm_dep_queue *q); > + > +/* Job API */ > + > +/** > + * DRM_DEP_JOB_FENCE_PREALLOC - sentinel value for pre-allocating a = dependency slot > + * > + * Pass this to drm_dep_job_add_dependency() instead of a real fence = to > + * pre-allocate a slot in the job's dependency xarray during the = preparation > + * phase (where GFP_KERNEL is available). The returned xarray index = identifies > + * the slot. Call drm_dep_job_replace_dependency() later =E2=80=94 = inside a > + * dma_fence_begin_signalling() region if needed =E2=80=94 to swap in = the real fence > + * without further allocation. > + * > + * This sentinel is never treated as a dma_fence; it carries no = reference count > + * and must not be passed to dma_fence_put(). It is only valid as an = argument > + * to drm_dep_job_add_dependency() and as the expected stored value = checked by > + * drm_dep_job_replace_dependency(). > + */ > +#define DRM_DEP_JOB_FENCE_PREALLOC ((struct dma_fence *)-1) > + > +int drm_dep_job_init(struct drm_dep_job *job, > + const struct drm_dep_job_init_args *args); > +struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job); > +void drm_dep_job_put(struct drm_dep_job *job); > +void drm_dep_job_arm(struct drm_dep_job *job); > +void drm_dep_job_push(struct drm_dep_job *job); > +int drm_dep_job_add_dependency(struct drm_dep_job *job, > + struct dma_fence *fence); > +void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 = index, > + struct dma_fence *fence); > +int drm_dep_job_add_syncobj_dependency(struct drm_dep_job *job, > + struct drm_file *file, u32 handle, > + u32 point); > +int drm_dep_job_add_resv_dependencies(struct drm_dep_job *job, > + struct dma_resv *resv, > + enum dma_resv_usage usage); > +int drm_dep_job_add_implicit_dependencies(struct drm_dep_job *job, > + struct drm_gem_object *obj, > + bool write); > +bool drm_dep_job_is_signaled(struct drm_dep_job *job); > +bool drm_dep_job_is_finished(struct drm_dep_job *job); > +bool drm_dep_job_invalidate_job(struct drm_dep_job *job, int = threshold); > +struct dma_fence *drm_dep_job_finished_fence(struct drm_dep_job = *job); > + > +/** > + * struct drm_dep_queue_pending_job_iter - iterator state for > + * drm_dep_queue_for_each_pending_job() > + * @q: queue being iterated > + */ > +struct drm_dep_queue_pending_job_iter { > + struct drm_dep_queue *q; > +}; > + > +/* Drivers should never call this directly */ Not enforceable in C. > +static inline struct drm_dep_queue_pending_job_iter > +__drm_dep_queue_pending_job_iter_begin(struct drm_dep_queue *q) > +{ > + struct drm_dep_queue_pending_job_iter iter =3D { > + .q =3D q, > + }; > + > + WARN_ON(!drm_dep_queue_is_stopped(q)); > + return iter; > +} > + > +/* Drivers should never call this directly */ > +static inline void > +__drm_dep_queue_pending_job_iter_end(struct = drm_dep_queue_pending_job_iter iter) > +{ > + WARN_ON(!drm_dep_queue_is_stopped(iter.q)); > +} > + > +/* clang-format off */ > +DEFINE_CLASS(drm_dep_queue_pending_job_iter, > + struct drm_dep_queue_pending_job_iter, > + __drm_dep_queue_pending_job_iter_end(_T), > + __drm_dep_queue_pending_job_iter_begin(__q), > + struct drm_dep_queue *__q); > +/* clang-format on */ > +static inline void * > +class_drm_dep_queue_pending_job_iter_lock_ptr( > + class_drm_dep_queue_pending_job_iter_t *_T) > +{ return _T; } > +#define class_drm_dep_queue_pending_job_iter_is_conditional false > + > +/** > + * drm_dep_queue_for_each_pending_job() - iterate over all pending = jobs > + * in a queue > + * @__job: loop cursor, a &struct drm_dep_job pointer > + * @__q: &struct drm_dep_queue to iterate > + * > + * Iterates over every job currently on @__q->job.pending. The queue = must be > + * stopped (drm_dep_queue_stop() called) before using this iterator; = a WARN_ON > + * fires at the start and end of the scope if it is not. > + * > + * Context: Any context. > + */ > +#define drm_dep_queue_for_each_pending_job(__job, __q) \ > + scoped_guard(drm_dep_queue_pending_job_iter, (__q)) \ > + list_for_each_entry((__job), &(__q)->job.pending, pending_link) > + > +#endif > --=20 > 2.34.1 >=20 By the way: I invite you to have a look at this implementation [0]. It currently = works in real hardware i.e.: our downstream "Tyr" driver for Arm Mali is using that at = the moment. It is a mere prototype that we=E2=80=99ve put together to test = different approaches, so it=E2=80=99s not meant to be a =E2=80=9Csolution=E2=80=9D = at all. It=E2=80=99s a mere data point for further discussion. Philip Stanner is working on this =E2=80=9CJob Queue=E2=80=9D concept = too, but from an upstream perspective. [0]: https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61=