Re: sched/deadline: Use revised wakeup rule for dl_server

[Andreas Ziegler <br025@umbiko.net>]

On 2026-05-25 07:25, Christian Loehle wrote:
> On 5/11/26 10:47, Christian Loehle wrote:
>> On 5/9/26 12:42, Andreas Ziegler wrote:
>>> Hi Christian, Everyone,
>>> 
>>> On 2026-05-08 14:13, Christian Loehle wrote:
>>>> On 5/8/26 13:06, Andreas Ziegler wrote:
>>>>> Hi Christian,
>>>>> 
>>>>> On 2026-05-08 09:20, Christian Loehle wrote:
>>>>>> On 5/8/26 09:09, Andreas Ziegler wrote:
>>>>>>> Linux kernel version: 6.12
>>>>>>>   CONFIG_PREEMPT_RT (w/ PREEMPT_RT patch applied)
>>>>>>> Architecture: aarch64
>>>>>>> Platform: Raspberry Pi 4
>>>>>>> 
>>>>>>> Hi everyone,
>>>>>>> 
>>>>>>> Commit d66792919d4f (sched/deadline: Use revised wakeup rule for 
>>>>>>> dl_server) [1] introduced a marked degradation in scheduling 
>>>>>>> latency for real-time tasks in the presence of heavy I/O load.
>>>>>>> 
>>>>>>> --- a/kernel/sched/deadline.c
>>>>>>> +++ b/kernel/sched/deadline.c
>>>>>>> @@ -1079,7 +1079,7 @@ static void update_dl_entity(struct 
>>>>>>> sched_dl_entity *dl_se)
>>>>>>>      if (dl_time_before(dl_se->deadline, rq_clock(rq)) ||
>>>>>>>          dl_entity_overflow(dl_se, rq_clock(rq))) {
>>>>>>> 
>>>>>>> -        if (unlikely(!dl_is_implicit(dl_se) &&
>>>>>>> +        if (unlikely((!dl_is_implicit(dl_se) || dl_se->dl_defer) 
>>>>>>> &&
>>>>>>>                   !dl_time_before(dl_se->deadline, rq_clock(rq)) 
>>>>>>> &&
>>>>>>>                   !is_dl_boosted(dl_se))) {
>>>>>>>              update_dl_revised_wakeup(dl_se, rq);
>>>>>>> 
>>>>>>> This was observed using a modified version of Con Kolivas' 
>>>>>>> interactivity benchmark [2]; kernel bisection eventually pointed 
>>>>>>> to the above mentioned commit.
>>>>>>> 
>>>>>>> Benchmark results before d66792919d4f:
>>>>>>> 
>>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence 
>>>>>>> of simulated ---
>>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  
>>>>>>> Deadlines met [%]
>>>>>>> None      76.6 +/- 8.3654    76  166
>>>>>>> Video      78.5 +/- 3.9433    78  107
>>>>>>> X      76.4 +/- 8.123     75  157
>>>>>>> Burn      72.0 +/- 6.4733    71  127
>>>>>>> Write     255.3 +/- 26.627   252  331
>>>>>>> Read     226.6 +/- 12.38    227  262
>>>>>>> Ring      84.2 +/- 6.6207    83  125
>>>>>>> Compile     225.3 +/- 23.949   222  328
>>>>>>> 
>>>>>>>      136.8 +/- 78.462        331
>>>>>>> 
>>>>>>> Benchmark results after d66792919d4f:
>>>>>>> 
>>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence 
>>>>>>> of simulated ---
>>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  
>>>>>>> Deadlines met [%]
>>>>>>> None      68.4 +/- 9.7864    67  169
>>>>>>> Video      74.4 +/- 3.724     74   97
>>>>>>> X      72.0 +/- 6.5681    71  129
>>>>>>> Burn      66.9 +/- 5.9059    66  117
>>>>>>> Write    9576.9 +/- 67639    250500418        98.1         98.1
>>>>>>> Read     209.3 +/- 11.018   209  267
>>>>>>> Ring      80.5 +/- 8.0993    78  125
>>>>>>> Compile     239.0 +/- 29.447   234  372
>>>>>>> 
>>>>>>>     1298.4 +/- 24118       500418
>>>>>>> 
>>>>>>> Reverting this commit obviously solves the issue for me. I have 
>>>>>>> no idea why this issue appears exclusively with heavy write loads 
>>>>>>> in the background.
>>>>>>> 
>>>>>>> Is this a scheduler issue, or rather something in the background?
>>>>>>> 
>>>>>> 
>>>>>> Hi Andreas,
>>>>>> You're using cpufreq schedutil for your tests I'm assuming?
>>>>>> Is there a difference in cpufreq behavior (avg cpufreq or OPP 
>>>>>> residencies?)
>>>>>> Does the regression also happen on powersave/performance governor?
>>>>> 
>>>>> Actually this is a very stripped-down system. The 'performance' 
>>>>> cpufreq governor is the only one compiled in, the processor cores 
>>>>> run on a fixed frequency. CONFIG_PM_OPP is not set.
>>>> 
>>>> That certainly makes the analysis easier.
>>>> I couldn't reproduce the issue so far on my system but it does seem 
>>>> like the dl server
>>>> would get potentially unbounded running time with very frequent
>>>> starting and stopping of the dlserver (which presumably happens 
>>>> because of
>>>> the writeback) reset the runtime, which then leads to your 25s 
>>>> observed latency.
>>>> Peter, how is the revised wakeup rule supposed to behave here?
>>>> 
>>>>> [snip]
>>> 
>>> This seems to be a case of runtime starvation. If I change 
>>> sched_rt_runtime_us to a smaller value, the benchmark returns 
>>> reasonable latency values.
>>> 
>>> # echo "980000" > /proc/sys/kernel/sched_rt_runtime_us
>>> 
>>> I could live with this workaround, since it seems not to impact 
>>> overall latency values in a noticeable way.
>>> 
>> 
>> Not a very stable workaround unfortunately :/
>> While I try to reproduce this, what you're observing should imply that 
>> the
>> background SCHED_NORMAL work is enough to fully utilize the system, 
>> right?
>> interbench Write does 4k (buffered) writes of a 1GB file and then 
>> close+open
>> and repeat, nothing fancy really. Does this actually produce 
>> significant CPU
>> utilization for you? Can you just run the background work and see what 
>> that
>> looks like?
>> (What you're seeing looks like a bug in any case, just so I'm not 
>> going down
>> a wrong path when trying to reproduce here).
> 
> I'd be interested if you can still reproduce with this fix:
> https://lore.kernel.org/lkml/20260522125833.264145-1-gmonaco@redhat.com/

The current 6.12.91 kernel with this patch applied shows normal 
latencies without 50ms outliers.

There is a backport in planning for three commits that preceded 
d66792919d4f, that also fixes the issue: 
https://marc.info/?l=linux-rt-users&m=177948576595996&w=2 -> 
https://lore.kernel.org/stable/20260522213120.1205100-1-lbckmnn@mailbox.org

Applying the patch above on top of the proposed backport also tested 
without issues.