Re: [PATCH] drm/sched: Extend and update documentation

["Christian König" <christian.koenig@amd.com>]

On 11.08.25 11:50, Philipp Stanner wrote:
>>>>
>>>>> But the original draft from Christian hinted at that. So, @Christian,
>>>>> this would be an opportunity to discuss this matter.
>>>>>
>>>>> Otherwise I'd drop this docu section in v2. What users don't know, they
>>>>> cannot misuse.
>>>>
>>>> I would rather like to keep that to avoid misusing the job as the object for tracking the submission lifetime.
>>>
>>> Why would a driver ever want to access struct drm_sched_fence? The
>>> driver knows when it signaled the hardware fence, and it knows when its
>>> callbacks run_job() and free_job() were invoked.
>>>
>>> I'm open to learn what amdgpu does there and why.
>>
>> The simplest use case is performance optimization. You sometimes have submissions which ideally run with others at the same time.
>>
>> So we have AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES which basically tries to cast a fence to a scheduler fence and then only waits for the dependency to be pushed to the HW instead of waiting for it to finish (see amdgpu_cs.c).
> 
> But the driver recognizes that a certain fence got / gets pushed right
> now through backend_ops.run_job(), doesn't it?

Yeah, but how does that helps?

>>
>> Another example are gang submissions (where I still have the TODO to actually fix the code to not crash in an OOM situation).
>>
>> Here we have a gang leader and gang members which are guaranteed to run together on the HW at the same time.
>>
>> This works by adding scheduled dependencies to the gang leader so that the scheduler pushes it to the HW only after all gang members have been pushed.
>>
>> The first gang member pushed now triggers a dependency handling which makes sure that no other gang can be pushed until gang leader is pushed as well.
> 
> You mean amdgpu registers callbacks to drm_sched_fence?

No, we give it as dependency to drm_sched_job_add_dependency().

>>> That's rather vague. Regarding this TODO, "racing between timing out
>>> and signaling the fence" can now be corrected by the driver. Are there
>>> more issues? If so, we want to add a new FIXME for them.
>>
>> Yeah good point. We basically worked around all those issues now.
>>
>> It's just that I still see that we are missing a general concept. E.g. we applied workaround on top of workaround until it didn't crashed any more instead of saying ok that is the design does that work? Is it valid? etc...
> 
> Yes, that seems to have been our destiny for a while now :) :(
> 
> What I'm afraid of right now is that with the callbacks vs.
> drm_sched_fence we now potentially have several distinct mechanisms for
> doing things. The hardware fence is clearly the relevant
> synchronization object for telling when a job is completed; yet, we
> also have s_fence->finished.

Not quite, the s_fence->finished is what is relevant to the outside. The HW fence is only relevant to the inside of the scheduler.

> Using it (for what?) is even encouraged by the docu:
> 
>         /**
>          * @finished: this fence is what will be signaled by the scheduler
>          * when the job is completed.
>          *
>          * When setting up an out fence for the job, you should use
>          * this, since it's available immediately upon
>          * drm_sched_job_init(), and the fence returned by the driver
>          * from run_job() won't be created until the dependencies have
>          * resolved.
>          */

That comment sounds correct to me. Drivers should mostly use s_fence->finished and not the HW fence to determine if something is done.

> Anyways.
> I think this is a big topic very suitable for our work shop at XDC. I
> also have some ideas about paths forward that I want to present.

Sounds good.

Halve a year ago I was more or less ready to suggest to rip out the scheduler and start from scratch again, but now it more and more looks like we have light at the end of the tunnel.

Christian.

> 
> 
> P.
> 
>>
>>> That said, such an RFC would obviously be great. We can discuss the
>>> paragraph above there, if you want.
>>
>> I will try to hack something together. Not necessarily complete but it should show the direction.
>>
>> Christian.
>>
>>>
>>>
>>> Regards
>>> P.
>