Re: drm/sched: run queues freed before the TDR that drm_sched_fini() waits for

[Donggeun Yoo <donggeunyoo.kernel@gmail.com>]

On 9/10/26 10:51, Philipp Stanner wrote:
> Which KUnit test case exactly?

A local one. It is not in the tree - I wrote it for this, which is why you
could not find it. I should have said that explicitly.

It is a mock scheduler whose run_job() returns a hardware fence that is
never signaled, so the timeout always fires. timedout_job() then sleeps
long enough for the test thread to get into drm_sched_fini(), and calls
drm_sched_increase_karma() on the way out.

> I kindly asked you to provide more details about how and where the bug
> occurs. Can you post a longer stacktrace and also run
> scrips/decode_stacktrace.sh on it?

drm-misc-next 0878e6053d01, x86_64, KUNIT + KASAN + lockdep, run through
decode_stacktrace.sh (dropping the "? " speculative frames and shortening
the source paths, otherwise as emitted):

BUG: KASAN: slab-use-after-free in _raw_spin_lock (kernel/locking/spinlock.c:173)
Read of size 1 at addr ffff88800198b420 by task kworker/0:2/27

CPU: 0 UID: 0 PID: 27 Comm: kworker/0:2 Tainted: G                 N  7.3.0-rc2-00228-g483f69ec8ca2-dirty #7 PREEMPT(lazy)
Workqueue: events drm_sched_job_timedout
Call Trace:
 <TASK>
 dump_stack_lvl (lib/dump_stack.c:94 lib/dump_stack.c:120)
 print_report (mm/kasan/report.c:378 mm/kasan/report.c:482)
 kasan_report (mm/kasan/report.c:595)
 __kasan_check_byte (mm/kasan/common.c:574)
 lock_acquire (kernel/locking/lockdep.c:5916 kernel/locking/lockdep.c:5899)
 _raw_spin_lock (kernel/locking/spinlock.c:173)
 drm_sched_increase_karma (drivers/gpu/drm/scheduler/sched_main.c:1263)
 fini_uaf_timedout_job (drivers/gpu/drm/scheduler/tests/tests_fini_uaf.c:99)
 drm_sched_job_timedout (drivers/gpu/drm/scheduler/sched_main.c:355)
 process_one_work (kernel/workqueue.c:3396)
 worker_thread (kernel/workqueue.c:3479 kernel/workqueue.c:3560)
 kthread (kernel/kthread.c:436)
 ret_from_fork (arch/x86/kernel/process.c:158)
 ret_from_fork_asm (arch/x86/entry/entry_64.S:245)
 </TASK>

Allocated by task 28:
 __kmalloc_cache_noprof (mm/slub.c:5563)
 drm_sched_init (drivers/gpu/drm/scheduler/sched_main.c:1148)
 drm_sched_fini_frees_rq_before_tdr (drivers/gpu/drm/scheduler/tests/tests_fini_uaf.c:140)
 kunit_try_run_case (lib/kunit/test.c:454 lib/kunit/test.c:499)

Freed by task 28:
 kfree (mm/slub.c:6792)
 drm_sched_fini (drivers/gpu/drm/scheduler/sched_main.c:1214)
 drm_sched_fini_frees_rq_before_tdr (drivers/gpu/drm/scheduler/tests/tests_fini_uaf.c:167)
 kunit_try_run_case (lib/kunit/test.c:454 lib/kunit/test.c:499)

The buggy address belongs to the object at ffff88800198b400
which belongs to the cache kmalloc-128 of size 128
The buggy address is located 32 bytes inside of
freed 128-byte region [ffff88800198b400, ffff88800198b480)

sched_main.c:1148 is sched->sched_rq[i] = kzalloc_obj(*sched->sched_rq[i])
in drm_sched_init(), :1214 is kfree(sched->sched_rq[i]) in
drm_sched_fini(), and :1263 is spin_lock(&rq->lock) in
drm_sched_increase_karma(). Task 28 is the thread in drm_sched_fini();
the reader is the timeout worker on PID 27. The tree is -dirty because the
test case is added to it. I hope this is what you asked for - say the word
if you want the untrimmed log.

> If the bug only exists because someone does not signal all hardware-
> fences (that's what we call the ones returned from run_job()), then I
> tend to think that this is not a scheduler bug.

Agreed.

> Though for robustness reasons we _could_ nevertheless stop the timeout
> work item before releasing other resources.

Right, that's what my patch does: drm_sched_wqueue_stop(), then
cancel_delayed_work_sync(&sched->work_tdr), then the frees. The report is
gone and nothing else in the suite fails.

As you say, that is closer to a cleanup - or to making the teardown order
state its intent - than to a fix. Do you still want the reordering patch?

Regards,
Donggeun