From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0838C3DCD81 for ; Fri, 27 Mar 2026 09:19:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774603190; cv=none; b=MbgrLovw+7VpGEgDYqRfeNjFhhXRfuoLQYAeaQeSNcKbmlkVxo6QRYwcl2fCtEqHAxk6xTnUKka4BTaq+XFEzc9p6L4Cr78gF20A97qonJqew2t/EAPLO1yK9d020gdHdbuR2ITBhpP7Svz4vUBt0d/k8tYs5HvpP4Znf28sgjw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774603190; c=relaxed/simple; bh=WvkP8bqEnd6iLBZ7bWbbXQtplkboPbUzRS5g4N08QpA=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=DiaS+gx8cQD7We2JlicVEtVJ2zEz6mxtkD2uae9Ldu3bsZljJxHn3pPWQ+lNaJcvV6fNkCN6CUR+FITOhfNgycqqErHODpGEaNXLIxjKUwrIiEI2XWp1GYA3kCHD/sLy0y76unCrjJAS1G5Iunyk1HkYEJ1qJy9dF/6C8MrfPhI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ursulin.net; spf=pass smtp.mailfrom=ursulin.net; dkim=pass (2048-bit key) header.d=ursulin.net header.i=@ursulin.net header.b=Gh8tJfIf; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ursulin.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ursulin.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ursulin.net header.i=@ursulin.net header.b="Gh8tJfIf" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-43b983bb07eso747735f8f.0 for ; Fri, 27 Mar 2026 02:19:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ursulin.net; s=google; t=1774603181; x=1775207981; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=c2vYWUohPsts5c14vEaE9Au9ObEgy7+UMmOkW/ZgtuQ=; b=Gh8tJfIfxXKKyI6G6mgXoRr+BTjobGXaliPTrvZw0JbWF5o2UFf8vg9BrElP3jIOYJ GhTdwnokgoBadvKw8EvbO8Dm9crIzIsR4FzgncQQARxfGRtjY4m/10JDkNQ7HYCYUNrY H5SUBPYx2Nx3cim1gvsY0570AhFWn0dQsQ8MPpzepMMg9ajTmJ2ieX1+0cWmg69Dj+a+ Z5uKLMGVbmCO1L8ZPKCohLdOnXSWTN1Qk7ptsgCFf46xszwgFIXyzPTGaeFPLy9E5buT 4p6tXQjRxaxiCr5KT1gFSt4bY1bMPfXcPfdV5kt8nEzlLDHgSX30FaoloK2iAzaf8lzi Woqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774603181; x=1775207981; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=c2vYWUohPsts5c14vEaE9Au9ObEgy7+UMmOkW/ZgtuQ=; b=NNoUcBmc9B6YzU+mCbjN0hxnsx89ayfKYoCw/rna7d5tZQvVmeHCTPGYzMNRrxb+06 xBgnwVaNP2T+WrFF2I+7Mqcjx8B0tcfAU0hXKRCgcIAXhkP+60cv9CmkiX73GjvFUKsC ETaTjZIUjACMfHaOvJf++YHhNltdPu2zekP0psOlTmpG6EgMKCYiyQezMKEsS0dqiRss IgCbromhZjBcvLLsJH6u1Tqjnz1rWC19qvMCGV8MymmdPElSLx5xUZWhASnEvbU/yUI2 CD03jvnfsjhNSpH2aSpvHK/NTkkDYg1jC53iu9rlu91GIj8w/xcrQHt5WtdSDfS5ZN2f hXYA== X-Forwarded-Encrypted: i=1; AJvYcCWBVZWraEdZSeimuVQmNBt1tVK4o0/yxNCtg/8kSmgGGcxM/U5OREHmaBKibYiU9XP9NfYr6UcSK/+/MOQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxCmIDKAiiiI/1XpaRG84IJeH0KDlHy8fMt/bYOfRFd9EYTZ2W1 x/LTmnlHIA0RIsvIA7BbZMQPCCG6TP2Kzusp0BzuHu26umeOIf/KZiOj8mcTqgMB7uw= X-Gm-Gg: ATEYQzyrdj+2cZxcu95LChu1mzZRXfsIhT0EGlkLXR3AFi+pp1hEVQviXvyvzjcIW8R iqHWRB/rQTFNV+yTm9tiqt2hKX/YKlNOknIGeEjx/RavaMiTdf1Nw99jMuRLexZqTSIIz3dciZ3 boTWaTitRF10I2/WfYxkfRYpT85tjJfYDs4zhW9qbcO7xPjp3HcwjAZ/HSTdKMph9VcJ7W2tGKa bDKKd2FJ7UOIOdOuqaXGo9pQk3+3bhGzqOIw0kdxc/FnBv1FOT3Eqzlf+f7Nuy38jts1JMcyiog xNaqK0KKkkyLM3zjcFo/ASAR08Y7+SOcKX0X4twT9/x6VzAqxKSxy+GstPxa1s28QJ75oheQAP9 CqMHQsI4ceIXLiDY+3GIVxYEfjYNoqII9QRwGuFGXaUTGH6aZc9hG14IrBonbQ9XvJMecVRlqu4 k1UFdrcsvr2YTvApLFMSxYM5blUvTJZEunpN8+0ETcEAqO9puywlHCfNE= X-Received: by 2002:a05:6000:26c2:b0:43b:45a0:9df8 with SMTP id ffacd0b85a97d-43b9ea46af1mr2429877f8f.35.1774603180996; Fri, 27 Mar 2026 02:19:40 -0700 (PDT) Received: from [192.168.0.101] ([90.240.106.137]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43b91942e52sm13533531f8f.9.2026.03.27.02.19.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Mar 2026 02:19:40 -0700 (PDT) Message-ID: Date: Fri, 27 Mar 2026 09:19:39 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: drm_sched run_job and scheduling latency To: Boris Brezillon Cc: Chia-I Wu , ML dri-devel , intel-xe@lists.freedesktop.org, Steven Price , Liviu Dudau , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matthew Brost , Danilo Krummrich , Philipp Stanner , =?UTF-8?Q?Christian_K=C3=B6nig?= , =?UTF-8?Q?Thomas_Hellstr=C3=B6m?= , Rodrigo Vivi , open list References: <20260305104037.281991a8@fedora> Content-Language: en-GB From: Tvrtko Ursulin In-Reply-To: <20260305104037.281991a8@fedora> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 05/03/2026 09:40, Boris Brezillon wrote: > Hi Tvrtko, > > On Thu, 5 Mar 2026 08:35:33 +0000 > Tvrtko Ursulin wrote: > >> On 04/03/2026 22:51, Chia-I Wu wrote: >>> Hi, >>> >>> Our system compositor (surfaceflinger on android) submits gpu jobs >>> from a SCHED_FIFO thread to an RT gpu queue. However, because >>> workqueue threads are SCHED_NORMAL, the scheduling latency from submit >>> to run_job can sometimes cause frame misses. We are seeing this on >>> panthor and xe, but the issue should be common to all drm_sched users. >>> >>> Using a WQ_HIGHPRI workqueue helps, but it is still not RT (and won't >>> meet future android requirements). It seems either workqueue needs to >>> gain RT support, or drm_sched needs to support kthread_worker. >>> >>> I know drm_sched switched from kthread_worker to workqueue for better >> >> From a plain kthread actually. > > Oops, sorry, I hadn't seen your reply before posting mine. I basically > said the same. > >> Anyway, I suggested trying the >> kthread_worker approach a few times in the past but never got round >> implementing it. Not dual paths but simply replacing the workqueues with >> kthread_workers. >> >> What is your thinking regarding how would the priority be configured? In >> terms of the default and mechanism to select a higher priority >> scheduling class. > > If we follow the same model that exists today, where the > workqueue can be passed at drm_sched_init() time, it becomes the > driver's responsibility to create a worker of his own with the right > prio set (using sched_setscheduler()). There's still the case where the > worker is NULL, in which case the drm_sched code can probably create > his own worker and leave it with the default prio, just like existed > before the transition to workqueues. > > It's a whole different story if you want to deal with worker pools and > do some load balancing though... I prototyped this in xe in the mean time and it is looking plausible that latency can be significantly reduced. First to say that I did not go as far as worker pools because at the moment I don't see an use case for it. At least not for xe. When 1:1 entity to scheduler drivers appeared kthreads were undesirable just because they were ending up with effectively unbound number of kernel threads. There was no benefit to that but only downsides. Workqueues were good since they manage the thread pool under the hood, but it is just a handy coincidence, the design still misses to express the optimal number of CPU threads required to feed a GPU engine. For example with xe, if there was a 4096 CPU machine with 4096 user contexts feeding to the same GPU engine, the optimal number of CPU threads to feed it is really more like one rather than how much wq management decided to run in parallel. They all end up hammering on the same lock to let the firmware know there is something to schedule. For this reason in my prototype I create kthread_worker per hardware execution engine. (For xe even that could potentially be too much, maybe I should even try one kthread_worker per GuC CT.) This creates a requirement for 1:1 drivers to not use the "worker" auto-create mode of the DRM scheduler so TBD if that is okay. Anyway, onto the numbers. Well actually first onto a benchmark I hacked up.. I took xe_blt from IGT and modified it heavily to be more reasonable. What it essentially does it emits a constant stream of synchronous blit operations and measures the variance of the time each took to complete, as observed by the submitting process. In parallel it spawns a number of CPU hog threads to oversubscribe the system. And it can run the submitting thread at either normal priority, re-niced to -1, or at SCHED_FIFO. This is to simulate a typical compositor use case. Now onto the numbers. normal nice FIFO wq 100% 76% 1% kthread_worker 100% 73% 1.2% └─relative to wq: 50.5% 48.5% 58.9% Median "jitter" (variance in observed job submissions) is normalised and shows how changing the CPU priority changes the jitter observed by the submission thread. First two rows are the current wq implementation and the kthread_worker conversion. They show scaling as roughly similar. Third row are the kworker_thread results normalised against wq. And that shows roughly twice as low jitter. So a meaningful improvement. Then I went a step further to even better address the analysis of a problem done by Chia-I, solving the priority inversion problem. That is to loosely track CPU priorities of the currently active entities submitting to each scheduler (and in turn kthread_worker). This in turn further improved the latency numbers for the SCHED_FIFO case, albeit there is a strange anomaly with re-nice which I will come to later. It looks like this: normal nice FIFO kworker_follow_prio 100% 277% 0.66% └─relative to wq: 60% 222% 37.8% This effectively means that with a SCHED_FIFO compositor the submission round-trip latency could be around a third of what the current scheduler can do. Now the re-nice anomaly.. This is something I am yet to investigate. Issue may be related to what I said the kthread_workers loosely track the submission thread priority. Loosely meaning if they detect negative nice they do not follow the exact nice level but go minium nice, while my test program was using the least minimum nice level (-19 vs -1). Perhaps that causes some strange effect in the CPU scheduler. I do not know yet but it is very interesting that it appears repeatable. It is also important to view my numbers as with some margin of error. I have tried to remove the effect of intel_pstate, CPU turbo, and thermal management to a large extent, but I do not think I fully succeeded yet. There may be some +/- of 5% or so in the results is my gut feeling. Also important to say is that the prototype depends on my other DRM scheduler series (the fair scheduler one), since I needed the nicer sched_rq abstraction with better tracking of active entities to implement priority inheritance, so I am unlikely to post it all as RFC since Philipp would possible get a heart attack if I did. :) To close, I think this is interesting to check out further and could look at converting panthor next and then we could run more experiments. Regards, Tvrtko