Re: scheduler performance regression since v6.11

[Chris Mason <clm@meta.com>]

On 5/12/25 3:39 PM, Chris Mason wrote:
> 
> 
> On 5/12/25 2:08 PM, Peter Zijlstra wrote:
>> On Fri, May 09, 2025 at 09:49:55PM +0200, Peter Zijlstra wrote:
>>> 152 729288bc6856 kernel/sched: Fix util_est accounting for DELAY_DEQUEUE
>>> 	average rps: 1936829.47
>>> 	average rps: 1950715.10
>>>
>>> 153 84d265281d6c sched/pelt: Use rq_clock_task() for hw_pressure
>>> 	average rps: 2176857.32
>>> 	average rps: 2223004.23
>>
>> So, a little more data on this.
>>
>> The result appears stable, but reverting 729288bc6856 on master does not
>> seem to cure things.
> 
> 729288bc6856 is in that section of commits where the regression was ~1/2
> as bad.  (bad: 1.8M, good: 2M, middle ground: 1.9M) I get the same RPS
> with and without it.
> 
> I called it "good" in my bisect run, but I'll take a second pass through
> those potential suspects and see if the bad really starts there.

This commit drops us from 2M RPS down to 1.95M.  I did a bunch of runs
on the surrounding commits and the drop was very reliable:

commit 152e11f6df293e816a6a37c69757033cdc72667d
Author: Peter Zijlstra <peterz@infradead.org>
Date:   Thu May 23 12:25:32 2024 +0200

    sched/fair: Implement delayed dequeue


e1459a50ba31, which was the next earliest commit is consistently fast.


If I take 54a58a787791 or 54a58a787791 and turn off the
DELAY_DEQUEUE/ZERO features at run time, I don't get the performance
back.  But, if I patch them such that DELAY_DEQUEUE/ZERO default off

diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 7fdeb5576188c..94409e9831e97 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -37,8 +37,8 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  *
  * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
-SCHED_FEAT(DELAY_DEQUEUE, true)
-SCHED_FEAT(DELAY_ZERO, true)
+SCHED_FEAT(DELAY_DEQUEUE, false)
+SCHED_FEAT(DELAY_ZERO, false)

It runs at 2M QPS again.  If I enable DELAY_QUEUE/ZERO, I go back to 1.95M

This is roughly 1/2 of the total regression, v6.15 runs at 1.88M maybe
1.85M on a bad run.

Peter sent me a patch to try improving my ttwu_queue rate, I'll give
that a shot in the morning.

-chris