mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Philipp Stanner <phasta@mailbox.org>
To: "Christian König" <christian.koenig@amd.com>,
	"Philipp Stanner" <phasta@kernel.org>,
	"Matthew Brost" <matthew.brost@intel.com>,
	"Danilo Krummrich" <dakr@kernel.org>,
	"David Airlie" <airlied@gmail.com>,
	"Simona Vetter" <simona@ffwll.ch>,
	dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] drm/sched: Discourage usage of separate workqueues
Date: Thu, 05 Jun 2025 11:59:28 +0200	[thread overview]
Message-ID: <0e117b173891294369f158872fdec132fe6bcbc1.camel@mailbox.org> (raw)
In-Reply-To: <aEBhIzccXBPyt_58@phenom.ffwll.local>

On Wed, 2025-06-04 at 17:07 +0200, Simona Vetter wrote:
> On Wed, Jun 04, 2025 at 11:41:25AM +0200, Christian König wrote:
> > On 6/4/25 10:16, Philipp Stanner wrote:
> > > struct drm_sched_init_args provides the possibility of letting
> > > the
> > > scheduler use user-controlled workqueues, instead of the
> > > scheduler
> > > creating its own workqueues. It's currently not documented who
> > > would
> > > want to use that.
> > > 
> > > Not sharing the submit_wq between driver and scheduler has the
> > > advantage
> > > of no negative intereference between them being able to occur
> > > (e.g.,
> > > MMU notifier callbacks waiting for fences to get signaled). A
> > > separate
> > > timeout_wq should rarely be necessary, since using the system_wq
> > > could,
> > > in the worst case, delay a timeout.
> > > 
> > > Discourage the usage of own workqueues in the documentation.
> > > 
> > > Suggested-by: Danilo Krummrich <dakr@kernel.org>
> > > Signed-off-by: Philipp Stanner <phasta@kernel.org>
> > > ---
> > >  include/drm/gpu_scheduler.h | 7 +++++--
> > >  1 file changed, 5 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/include/drm/gpu_scheduler.h
> > > b/include/drm/gpu_scheduler.h
> > > index 81dcbfc8c223..11740d745223 100644
> > > --- a/include/drm/gpu_scheduler.h
> > > +++ b/include/drm/gpu_scheduler.h
> > > @@ -590,14 +590,17 @@ struct drm_gpu_scheduler {
> > >   *
> > >   * @ops: backend operations provided by the driver
> > >   * @submit_wq: workqueue to use for submission. If NULL, an
> > > ordered wq is
> > > - *        allocated and used.
> > > + *        allocated and used. It is recommended to pass NULL
> > > unless there
> > > + *        is a good reason not to.
> > 
> > Yeah, that's probably a good idea. I'm not sure why xe and nouveau
> > actually wanted that.
> 
> The idea of this trick is that you have a fw scheduler which only has
> one
> queue, and a bunch of other things in your driver that also need to
> be
> stuffed into this fw queue (or handled by talking with the fw through
> these ringbuffers).
> 
> If you use one single-threaded wq for everything then you don't need
> additional locking anymore, and a lot of things become easier.
> 
> We should definitely document this trick better though, I didn't find
> any
> place where that was documented.
> 
> Maybe a new overview section about "how to concurrency with
> drm/sched"?
> That's also a good place to better highlight the existing
> documentation
> for the 2nd part here.


Let us first create consensus about when providing workqueues by the
driver makes sense. Correct wrong statements:

   1. Using a single threaded workqueue for timeout_wq is useful for
      guaranteeing that timeouts handlers are performed in the correct
      order. (Christian)
   2. Independently from the point above, using the very same single
      threaded wq for both timeout_wq and submit_wq allows for dropping
      locks from the driver's callbacks.
   3. Driver and scheduler sharing workqueues risks deadlocking, as
      described by Danilo.


I'm wondering about the following now:
   1. Why does the scheduler by default use the system_wq for timeouts?
      Could it make sense for us to always use a single threaded wq?
      After all, each scheduler only ever has 1 work item for timeouts
      in the air.
   2. Isn't point 2 from above purely a trick to avoid locking in the
      callbacks? If it is, I'm not convinced yet that we really want to
      actively recommend that. The callback implementations I've seen
      usually only need their fence context lock, and many drivers
      don't use their own workqueues to begin with.
      But I guess having a sentence somewhere teasering at the
      possibility wouldn't be hurtful.


My current feeling right now is that we should document why the
mechanism exists and discourage using it. It's optional, not really
necessary, risks deadlocking and making that more waterproof would
require more substantial work, like maybe lockdep assertions suggested
by Matthew.

I'm for the more leightweight way.


> 
> > >   * @num_rqs: Number of run-queues. This may be at most
> > > DRM_SCHED_PRIORITY_COUNT,
> > >   *      as there's usually one run-queue per priority, but may
> > > be less.
> > >   * @credit_limit: the number of credits this scheduler can hold
> > > from all jobs
> > >   * @hang_limit: number of times to allow a job to hang before
> > > dropping it.
> > >   * This mechanism is DEPRECATED. Set it to 0.
> > >   * @timeout: timeout value in jiffies for submitted jobs.
> > > - * @timeout_wq: workqueue to use for timeout work. If NULL, the
> > > system_wq is used.
> > > + * @timeout_wq: workqueue to use for timeout work. If NULL, the
> > > system_wq is
> > > + * used. It is recommended to pass NULL unless there is a good
> > > + * reason not to.
> > 
> > Well, that's a rather bad idea.
> > 
> > Using a the same single threaded work queue for the timeout of
> > multiple
> > schedulers instances has the major advantage of being able to
> > handle
> > their occurrence sequentially.
> > 
> > In other words multiple schedulers post their timeout work items on
> > the
> > same queue, the first one to run resets the specific HW block in
> > question and cancels all timeouts and work items from other
> > schedulers
> > which use the same HW block.
> > 
> > It was Sima, I and a few other people who came up with this
> > approach
> > because both amdgpu and IIRC panthor implemented that in their own
> > specific way, and as usual got it wrong.
> > 
> > If I'm not completely mistaken this approach is now used by amdgpu,
> > panthor, xe and imagination and has proven to be rather flexible
> > and
> > reliable. It just looks like we never documented that you should do
> > it
> > this way.
> 
> It is documented, just not here. See the note in
> drm_sched_backend_ops.timedout_job at the very bottom.
> 
> We should definitely have a lot more cross-links between the various
> pieces of this puzzle though, that's for sure :-)

Docu is in far better shape by now than it was last year, but there's
still much to do.

However, we probably also don't want to scather over too many places,
as it becomes harder to keep the docu consistent.


P.

> 
> Cheers, Sima
> 
> > 
> > Regards,
> > Christian.
> > 
> > >   * @score: score atomic shared with other schedulers. May be
> > > NULL.
> > >   * @name: name (typically the driver's name). Used for debugging
> > >   * @dev: associated device. Used for debugging
> > 
> 


      parent reply	other threads:[~2025-06-05  9:59 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-06-04  8:16 Philipp Stanner
2025-06-04  9:41 ` Christian König
2025-06-04 15:07   ` Simona Vetter
2025-06-04 16:45     ` Matthew Brost
2025-06-04 16:53       ` Danilo Krummrich
2025-06-04 17:10         ` Matthew Brost
2025-06-06 14:08           ` Simona Vetter
2025-06-05  9:59     ` Philipp Stanner [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0e117b173891294369f158872fdec132fe6bcbc1.camel@mailbox.org \
    --to=phasta@mailbox.org \
    --cc=airlied@gmail.com \
    --cc=christian.koenig@amd.com \
    --cc=dakr@kernel.org \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=matthew.brost@intel.com \
    --cc=phasta@kernel.org \
    --cc=simona@ffwll.ch \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®