sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Tom Gebhardt]

Hi Peter,

I would like to report a measurable pipe IPC throughput regression introduced by
commit 152e11f ("sched/fair: Implement delayed dequeue"), first
present in v6.12-rc1.

This has been independently confirmed on the official Raspberry Pi
Linux issue tracker
(raspberrypi/linux #7308), where the RPi kernel team directed the
issue upstream.


Hardware / Software
-------------------
- Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
- Raspberry Pi OS Bookworm (arm64)
- Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
6.12.81-v8-16k+ (custom)
- Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief


Observed regression
-------------------
Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):

  Kernel        pipe bogo ops/s   vs. 6.6
  6.6.78            2 487 746      100%
  6.12.75           1 651 427      -34%
  6.18.21           2 049 701      -18%

This regression pattern is consistent across two separate Raspberry Pi
5 units and has
been independently reproduced by the RPi kernel team with 20-run averages:
6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).


Runtime isolation via CONFIG_SCHED_DEBUG
-----------------------------------------
To isolate the root cause, I compiled a custom kernel (rpi-6.12.y,
6.12.81-v8-16k+)
with CONFIG_SCHED_DEBUG=y and toggled scheduler features at runtime:

  DELAY_DEQUEUE   PREEMPT_SHORT   pipe bogo ops/s   vs. baseline
  on (default)    on (default)        1 506 572          --
  OFF             on                  2 125 473         +41%   <==
  on              OFF                 1 419 026          -6%
  OFF             OFF                 2 078 182         +38%

Disabling DELAY_DEQUEUE alone recovers +41% throughput, almost closing
the gap to 6.6.
Disabling PREEMPT_SHORT alone has no positive effect on this workload.

The remaining gap to 6.6 (~15%) is likely CONFIG_SCHED_DEBUG=y overhead.


Root cause analysis
-------------------
The pipe producer-consumer loop is affected by DELAY_DEQUEUE as follows:

Before DELAY_DEQUEUE:
  consumer reads empty pipe -> blocks -> dequeue_task() removes it from runqueue
  producer writes -> wake_up_interruptible() -> consumer re-enqueued
cleanly -> runs

With DELAY_DEQUEUE (v6.12+):
  consumer reads empty pipe -> blocks -> stays on runqueue (sched_delayed = 1)
  producer writes -> wakeup path handles already-queued task ->
additional bookkeeping
  per iteration

For a tight 4-worker pipe benchmark at millions of iterations, this
per-iteration
overhead compounds directly into measured throughput.

PREEMPT_SHORT (commit 85e511d) does not contribute to this regression.
Its stated
trade-off ("massive_intr workload gets more context switches") does
not appear to be
the bottleneck here.


Mitigations tested and ruled out
---------------------------------
- Spectre mitigations: mitigations=off yields only +0.5-2.5%
improvement (confirmed by
  RPi kernel team). Not the cause.
- CPU governor: tested with both ondemand and performance. No
significant difference.


References
----------
- Commit 152e11f (DELAY_DEQUEUE):
  https://github.com/torvalds/linux/commit/152e11f6df293e816a6a37c69757033cdc72667d
- Commit 85e511d (PREEMPT_SHORT):
  https://github.com/torvalds/linux/commit/85e511df3cec46021024176672a748008ed135bf
- RPi issue tracker: https://github.com/raspberrypi/linux/issues/7308

Please let me know if additional data (perf traces, full benchmark
logs, kernel config)
would be helpful. I am happy to run further tests on the hardware.

Thank you for your work on the scheduler.

Best regards,
Thomas Gebhardt (@Kletternaut on GitHub)

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Christian Loehle]

On 4/16/26 13:22, Tom Gebhardt wrote:
> Hi Peter,
> 
> I would like to report a measurable pipe IPC throughput regression introduced by
> commit 152e11f ("sched/fair: Implement delayed dequeue"), first
> present in v6.12-rc1.
> 
> This has been independently confirmed on the official Raspberry Pi
> Linux issue tracker
> (raspberrypi/linux #7308), where the RPi kernel team directed the
> issue upstream.
> 
> 
> Hardware / Software
> -------------------
> - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
> - Raspberry Pi OS Bookworm (arm64)
> - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
> 6.12.81-v8-16k+ (custom)
> - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief
> 
> 
> Observed regression
> -------------------
> Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):
> 
>   Kernel        pipe bogo ops/s   vs. 6.6
>   6.6.78            2 487 746      100%
>   6.12.75           1 651 427      -34%
>   6.18.21           2 049 701      -18%
> 
> This regression pattern is consistent across two separate Raspberry Pi
> 5 units and has
> been independently reproduced by the RPi kernel team with 20-run averages:
> 6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).
> 

I guess running anything closer to mainline isn't an option for you?
There have been a bunch of fixes recently on eevdf, maybe some were missed,
so I did a quick skimming through.
Full list of kernel/sched/fair.c changes not on 6.18.y:
059258b0d424 ("sched/fair: Prevent negative lag increase during delayed dequeue")
089d84203ad4 ("sched/fair: Fold the sched_avg update")
0ab25ea2a3b3 ("sched/fair: Simplify task_numa_find_cpu()")
04e49d926f43 ("sched: Enable context analysis for core.c and fair.c")
15257cc2f905 ("sched/fair: Revert force wakeup preemption")
1ae5f5dfe5ad ("sched: Cleanup sched_delayed handling for class switches")
1e900f415c60 ("sched: Detect per-class runqueue changes")
2e4b28c48f88 ("treewide: Update email address")
45e09225085f ("sched/fair: Avoid rq->lock bouncing in sched_balance_newidle()")
4823725d9d1d ("sched/fair: Increase weight bits for avg_vruntime")
50653216e4ff ("sched: Add support to pick functions to take rf")
5324953c06bd ("sched/core: Fix wakeup_preempt's next_class tracking")
553255cc857c ("sched/fair: Fix math notation errors in avg_vruntime comment")
556146ce5e94 ("sched/fair: Avoid overflow in enqueue_entity()")
558c18d3fbb6 ("sched/eevdf: Fix HRTICK duration")
55b39b0cf183 ("sched/fair: Use cpumask_weight_and() in sched_balance_find_dst_group()")
5d86d542f68f ("sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead")
5d88e424ec1b ("sched/fair: Make hrtick resched hard")
6455ad5346c9 ("sched: Move sched_class::prio_changed() into the change pattern")
69050f8d6d07 ("treewide: Replace kmalloc with kmalloc_obj for non-scalar types")
6ab7973f2540 ("sched/fair: Fix sched_avg fold")
6b67c8a72e56 ("sched/fair: Move checking for nohz cpus after time check")
71fedc41c23b ("sched/fair: Switch to rcu_dereference_all()")
76504bce4ee6 ("sched/fair: Get this cpu once in find_new_ilb()")
78cde54ea5f0 ("sched/eevdf: Clear buddies for preempt_short")
82d6e01a0699 ("sched/fair: Only update stats for allowed CPUs when looking for dst group")
8d16e3c6f844 ("sched/fair: Fix comma operator misuse in NUMA fault accounting")
926475806606 ("sched/fair: Update overutilized detection")
94e70734b4d0 ("sched/fair: Change likelyhood of nohz.nr_cpus")
95a0155224a6 ("sched/fair: Limit hrtick work")
97015376642f ("sched/fair: Simplify hrtick_update()")
9fe89f022c05 ("sched/fair: More complex proportional newidle balance")
bf4afc53b77a ("Convert 'alloc_obj' family to use the new default GFP_KERNEL argument")
d3d663faa1d4 ("sched/fair: Filter false overloaded_group case for EAS")
db4551e2ba34 ("sched/fair: Use full weight to __calc_delta()")
dec9554dc036 ("sched: Move attach_one_task and attach_task helpers to sched.h")
e636ffb9e31b ("sched/deadline: Fix dl_server time accounting")
e837456fdca8 ("sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals")
f1320a8dd8ba ("sched/fair: Simplify the entry condition for update_idle_cpu_scan()")
f24165bfa7ef ("sched/headers: Rename rcu_dereference_check_sched_domain() => rcu_dereference_sched_domain()")
fa6874dfeee0 ("sched/fair: Remove superfluous rcu_read_lock() in the wakeup path")
fd54d81c2c0e ("sched/fair: Skip SCHED_IDLE rq for SCHED_IDLE task")
fe7171d0d5df ("sched/fair: Simplify SIS_UTIL handling in select_idle_cpu()")
ff1de90dd7a6 ("sched/fair: Drop useless cpumask_empty() in find_energy_efficient_cpu()")
101f3498b4bd ("sched/fair: Revert 6d71a9c61604 (sched/fair: Fix EEVDF entity placement bug causing scheduling lag)")

We will probably want to backport at least:
059258b0d424 ("sched/fair: Prevent negative lag increase during delayed dequeue")
556146ce5e94 ("sched/fair: Avoid overflow in enqueue_entity()")
101f3498b4bd ("sched/fair: Revert 6d71a9c61604 (sched/fair: Fix EEVDF entity placement bug causing scheduling lag)")
e837456fdca8 ("sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals")
78cde54ea5f0 ("sched/eevdf: Clear buddies for preempt_short")
558c18d3fbb6 ("sched/eevdf: Fix HRTICK duration")

Could you try the first three of those for you specifically?

> 
> Runtime isolation via CONFIG_SCHED_DEBUG
> -----------------------------------------
> To isolate the root cause, I compiled a custom kernel (rpi-6.12.y,
> 6.12.81-v8-16k+)
> with CONFIG_SCHED_DEBUG=y and toggled scheduler features at runtime:
> 
>   DELAY_DEQUEUE   PREEMPT_SHORT   pipe bogo ops/s   vs. baseline
>   on (default)    on (default)        1 506 572          --
>   OFF             on                  2 125 473         +41%   <==
>   on              OFF                 1 419 026          -6%
>   OFF             OFF                 2 078 182         +38%
> 
> Disabling DELAY_DEQUEUE alone recovers +41% throughput, almost closing
> the gap to 6.6.
> Disabling PREEMPT_SHORT alone has no positive effect on this workload.
> 
> The remaining gap to 6.6 (~15%) is likely CONFIG_SCHED_DEBUG=y overhead.

That seems very high, I don't think there's a measurable CONFIG_SCHED_DEBUG=y
overhead. You can also just change the default to test features with
CONFIG_SCHED_DEBUG=n

> 
> 
> Root cause analysis
> -------------------
> The pipe producer-consumer loop is affected by DELAY_DEQUEUE as follows:
> 
> Before DELAY_DEQUEUE:
>   consumer reads empty pipe -> blocks -> dequeue_task() removes it from runqueue
>   producer writes -> wake_up_interruptible() -> consumer re-enqueued
> cleanly -> runs
> 
> With DELAY_DEQUEUE (v6.12+):
>   consumer reads empty pipe -> blocks -> stays on runqueue (sched_delayed = 1)
>   producer writes -> wakeup path handles already-queued task ->
> additional bookkeeping
>   per iteration
> 
> For a tight 4-worker pipe benchmark at millions of iterations, this
> per-iteration
> overhead compounds directly into measured throughput.
> 
> PREEMPT_SHORT (commit 85e511d) does not contribute to this regression.
> Its stated
> trade-off ("massive_intr workload gets more context switches") does
> not appear to be
> the bottleneck here.
> 
> 
> Mitigations tested and ruled out
> ---------------------------------
> - Spectre mitigations: mitigations=off yields only +0.5-2.5%
> improvement (confirmed by
>   RPi kernel team). Not the cause.
> - CPU governor: tested with both ondemand and performance. No
> significant difference.
> 
> 
> References
> ----------
> - Commit 152e11f (DELAY_DEQUEUE):
>   https://github.com/torvalds/linux/commit/152e11f6df293e816a6a37c69757033cdc72667d
> - Commit 85e511d (PREEMPT_SHORT):
>   https://github.com/torvalds/linux/commit/85e511df3cec46021024176672a748008ed135bf

Generally no need to provide links if you have the short hash, but we use the 12
character ones:
152e11f6df29 ("sched/fair: Implement delayed dequeue")
781773e3b680 ("sched/fair: Implement ENQUEUE_DELAYED")


> - RPi issue tracker: https://github.com/raspberrypi/linux/issues/7308
> 
> Please let me know if additional data (perf traces, full benchmark
> logs, kernel config)
> would be helpful. I am happy to run further tests on the hardware.
> 
> Thank you for your work on the scheduler.
> 
> Best regards,
> Thomas Gebhardt (@Kletternaut on GitHub)
> 

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Vincent Guittot]

On Thu, 16 Apr 2026 at 14:23, Tom Gebhardt <tomge68@gmail.com> wrote:
>
> Hi Peter,
>
> I would like to report a measurable pipe IPC throughput regression introduced by
> commit 152e11f ("sched/fair: Implement delayed dequeue"), first
> present in v6.12-rc1.
>
> This has been independently confirmed on the official Raspberry Pi
> Linux issue tracker
> (raspberrypi/linux #7308), where the RPi kernel team directed the
> issue upstream.
>
>
> Hardware / Software
> -------------------
> - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
> - Raspberry Pi OS Bookworm (arm64)
> - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
> 6.12.81-v8-16k+ (custom)
> - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief
>
>
> Observed regression
> -------------------
> Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):
>
>   Kernel        pipe bogo ops/s   vs. 6.6
>   6.6.78            2 487 746      100%
>   6.12.75           1 651 427      -34%
>   6.18.21           2 049 701      -18%
>
> This regression pattern is consistent across two separate Raspberry Pi
> 5 units and has
> been independently reproduced by the RPi kernel team with 20-run averages:
> 6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).
>
>
> Runtime isolation via CONFIG_SCHED_DEBUG
> -----------------------------------------
> To isolate the root cause, I compiled a custom kernel (rpi-6.12.y,
> 6.12.81-v8-16k+)
> with CONFIG_SCHED_DEBUG=y and toggled scheduler features at runtime:
>
>   DELAY_DEQUEUE   PREEMPT_SHORT   pipe bogo ops/s   vs. baseline
>   on (default)    on (default)        1 506 572          --
>   OFF             on                  2 125 473         +41%   <==
>   on              OFF                 1 419 026          -6%
>   OFF             OFF                 2 078 182         +38%
>
> Disabling DELAY_DEQUEUE alone recovers +41% throughput, almost closing
> the gap to 6.6.
> Disabling PREEMPT_SHORT alone has no positive effect on this workload.
>
> The remaining gap to 6.6 (~15%) is likely CONFIG_SCHED_DEBUG=y overhead.
>
>
> Root cause analysis
> -------------------
> The pipe producer-consumer loop is affected by DELAY_DEQUEUE as follows:
>
> Before DELAY_DEQUEUE:
>   consumer reads empty pipe -> blocks -> dequeue_task() removes it from runqueue
>   producer writes -> wake_up_interruptible() -> consumer re-enqueued
> cleanly -> runs
>
> With DELAY_DEQUEUE (v6.12+):
>   consumer reads empty pipe -> blocks -> stays on runqueue (sched_delayed = 1)
>   producer writes -> wakeup path handles already-queued task ->
> additional bookkeeping

because the task is still enqueued, the enqueue of delayed entity
should be faster most of the time

>   per iteration
>
> For a tight 4-worker pipe benchmark at millions of iterations, this
> per-iteration
> overhead compounds directly into measured throughput.
>
> PREEMPT_SHORT (commit 85e511d) does not contribute to this regression.
> Its stated

Unless you set a custom slice you should not see any difference with this patch

> trade-off ("massive_intr workload gets more context switches") does
> not appear to be
> the bottleneck here.
>
>
> Mitigations tested and ruled out
> ---------------------------------
> - Spectre mitigations: mitigations=off yields only +0.5-2.5%
> improvement (confirmed by
>   RPi kernel team). Not the cause.
> - CPU governor: tested with both ondemand and performance. No
> significant difference.
>
>
> References
> ----------
> - Commit 152e11f (DELAY_DEQUEUE):
>   https://github.com/torvalds/linux/commit/152e11f6df293e816a6a37c69757033cdc72667d
> - Commit 85e511d (PREEMPT_SHORT):
>   https://github.com/torvalds/linux/commit/85e511df3cec46021024176672a748008ed135bf
> - RPi issue tracker: https://github.com/raspberrypi/linux/issues/7308
>
> Please let me know if additional data (perf traces, full benchmark
> logs, kernel config)
> would be helpful. I am happy to run further tests on the hardware.

Could you trace and check that:
- consumer and producer of one pipe are on the same cpu
- if there is a diff in the number of migration

I will try to reproduce locally once I get access to my hardware

>
> Thank you for your work on the scheduler.
>
> Best regards,
> Thomas Gebhardt (@Kletternaut on GitHub)
>

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Tom Gebhardt]

Hi Vincent,

thank you for the questions — here are the results.

Hardware: Raspberry Pi 5 (BCM2712 C1), 8 GB, arm64, overclocked (arm_freq=2800).
Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s
Measured with: perf stat -e sched:sched_migrate_task,sched:sched_switch

Results
-------

  Kernel        pipe bogo ops/s   sched_migrate_task   sched_switch
  6.6.78            2 339 827              98           21 151 454
  6.12.81           1 163 540               4           39 911 525

Key observations:

1. sched_migrate_task is much lower in 6.12 (4 vs 98). Producer and
consumer are not migrating — they stay on their assigned CPUs.
   CPU affinity sampling confirms that workers are distributed across
all 4 cores in both kernels with no significant difference in
co-location.

2. sched_switch is nearly double in 6.12 (+89%, 39.9M vs 21.2M). The
pipe worker pairs switch context far more often without doing more
useful work.

3. usr time drops significantly in 6.12 (11.7s vs 20.3s user time
across 4 workers), while sys time increases (67.4s vs 57.7s). The
workers spend less time in
   userspace and more time in the kernel per iteration.

This points to DELAY_DEQUEUE causing the sleeping consumer to remain
on the runqueue longer, generating additional scheduling overhead per
pipe iteration without improving throughput — consistent with the ~50%
regression in bogo ops/s.

Happy to run additional traces (e.g. perf sched or ftrace) if that would help.

Best regards,
Tom

Am Fr., 17. Apr. 2026 um 17:42 Uhr schrieb Vincent Guittot
<vincent.guittot@linaro.org>:
>
> On Thu, 16 Apr 2026 at 14:23, Tom Gebhardt <tomge68@gmail.com> wrote:
> >
> > Hi Peter,
> >
> > I would like to report a measurable pipe IPC throughput regression introduced by
> > commit 152e11f ("sched/fair: Implement delayed dequeue"), first
> > present in v6.12-rc1.
> >
> > This has been independently confirmed on the official Raspberry Pi
> > Linux issue tracker
> > (raspberrypi/linux #7308), where the RPi kernel team directed the
> > issue upstream.
> >
> >
> > Hardware / Software
> > -------------------
> > - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
> > - Raspberry Pi OS Bookworm (arm64)
> > - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
> > 6.12.81-v8-16k+ (custom)
> > - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief
> >
> >
> > Observed regression
> > -------------------
> > Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):
> >
> >   Kernel        pipe bogo ops/s   vs. 6.6
> >   6.6.78            2 487 746      100%
> >   6.12.75           1 651 427      -34%
> >   6.18.21           2 049 701      -18%
> >
> > This regression pattern is consistent across two separate Raspberry Pi
> > 5 units and has
> > been independently reproduced by the RPi kernel team with 20-run averages:
> > 6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).
> >
> >
> > Runtime isolation via CONFIG_SCHED_DEBUG
> > -----------------------------------------
> > To isolate the root cause, I compiled a custom kernel (rpi-6.12.y,
> > 6.12.81-v8-16k+)
> > with CONFIG_SCHED_DEBUG=y and toggled scheduler features at runtime:
> >
> >   DELAY_DEQUEUE   PREEMPT_SHORT   pipe bogo ops/s   vs. baseline
> >   on (default)    on (default)        1 506 572          --
> >   OFF             on                  2 125 473         +41%   <==
> >   on              OFF                 1 419 026          -6%
> >   OFF             OFF                 2 078 182         +38%
> >
> > Disabling DELAY_DEQUEUE alone recovers +41% throughput, almost closing
> > the gap to 6.6.
> > Disabling PREEMPT_SHORT alone has no positive effect on this workload.
> >
> > The remaining gap to 6.6 (~15%) is likely CONFIG_SCHED_DEBUG=y overhead.
> >
> >
> > Root cause analysis
> > -------------------
> > The pipe producer-consumer loop is affected by DELAY_DEQUEUE as follows:
> >
> > Before DELAY_DEQUEUE:
> >   consumer reads empty pipe -> blocks -> dequeue_task() removes it from runqueue
> >   producer writes -> wake_up_interruptible() -> consumer re-enqueued
> > cleanly -> runs
> >
> > With DELAY_DEQUEUE (v6.12+):
> >   consumer reads empty pipe -> blocks -> stays on runqueue (sched_delayed = 1)
> >   producer writes -> wakeup path handles already-queued task ->
> > additional bookkeeping
>
> because the task is still enqueued, the enqueue of delayed entity
> should be faster most of the time
>
> >   per iteration
> >
> > For a tight 4-worker pipe benchmark at millions of iterations, this
> > per-iteration
> > overhead compounds directly into measured throughput.
> >
> > PREEMPT_SHORT (commit 85e511d) does not contribute to this regression.
> > Its stated
>
> Unless you set a custom slice you should not see any difference with this patch
>
> > trade-off ("massive_intr workload gets more context switches") does
> > not appear to be
> > the bottleneck here.
> >
> >
> > Mitigations tested and ruled out
> > ---------------------------------
> > - Spectre mitigations: mitigations=off yields only +0.5-2.5%
> > improvement (confirmed by
> >   RPi kernel team). Not the cause.
> > - CPU governor: tested with both ondemand and performance. No
> > significant difference.
> >
> >
> > References
> > ----------
> > - Commit 152e11f (DELAY_DEQUEUE):
> >   https://github.com/torvalds/linux/commit/152e11f6df293e816a6a37c69757033cdc72667d
> > - Commit 85e511d (PREEMPT_SHORT):
> >   https://github.com/torvalds/linux/commit/85e511df3cec46021024176672a748008ed135bf
> > - RPi issue tracker: https://github.com/raspberrypi/linux/issues/7308
> >
> > Please let me know if additional data (perf traces, full benchmark
> > logs, kernel config)
> > would be helpful. I am happy to run further tests on the hardware.
>
> Could you trace and check that:
> - consumer and producer of one pipe are on the same cpu
> - if there is a diff in the number of migration
>
> I will try to reproduce locally once I get access to my hardware
>
> >
> > Thank you for your work on the scheduler.
> >
> > Best regards,
> > Thomas Gebhardt (@Kletternaut on GitHub)
> >

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Vincent Guittot]

Hi Tom,

On Sat, 18 Apr 2026 at 18:13, Tom Gebhardt <tomge68@gmail.com> wrote:
>
> Hi Vincent,
>
> thank you for the questions — here are the results.
>
> Hardware: Raspberry Pi 5 (BCM2712 C1), 8 GB, arm64, overclocked (arm_freq=2800).
> Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s
> Measured with: perf stat -e sched:sched_migrate_task,sched:sched_switch
>
> Results
> -------
>
>   Kernel        pipe bogo ops/s   sched_migrate_task   sched_switch
>   6.6.78            2 339 827              98           21 151 454
>   6.12.81           1 163 540               4           39 911 525
>
> Key observations:
>
> 1. sched_migrate_task is much lower in 6.12 (4 vs 98). Producer and
> consumer are not migrating — they stay on their assigned CPUs.
>    CPU affinity sampling confirms that workers are distributed across
> all 4 cores in both kernels with no significant difference in
> co-location.

This is somewhat normal because delayed dequeue tasks are requeued on
prev rq without calling select_task_rq

>
> 2. sched_switch is nearly double in 6.12 (+89%, 39.9M vs 21.2M). The
> pipe worker pairs switch context far more often without doing more
> useful work.

This is most probably our main problem. I will check why we have so
many context switches with delayed dequeues.

I think that I have been able to reproduce the regression with a
regression of 19% on my board for 6.12.82 vs 6.6.185 and 3 times more
context switch.
The drop is only 7% with tip/sched/core

Regards,
Vincent

>
> 3. usr time drops significantly in 6.12 (11.7s vs 20.3s user time
> across 4 workers), while sys time increases (67.4s vs 57.7s). The
> workers spend less time in
>    userspace and more time in the kernel per iteration.
>
> This points to DELAY_DEQUEUE causing the sleeping consumer to remain
> on the runqueue longer, generating additional scheduling overhead per
> pipe iteration without improving throughput — consistent with the ~50%
> regression in bogo ops/s.
>
> Happy to run additional traces (e.g. perf sched or ftrace) if that would help.
>
> Best regards,
> Tom
>
> Am Fr., 17. Apr. 2026 um 17:42 Uhr schrieb Vincent Guittot
> <vincent.guittot@linaro.org>:
> >
> > On Thu, 16 Apr 2026 at 14:23, Tom Gebhardt <tomge68@gmail.com> wrote:
> > >
> > > Hi Peter,
> > >
> > > I would like to report a measurable pipe IPC throughput regression introduced by
> > > commit 152e11f ("sched/fair: Implement delayed dequeue"), first
> > > present in v6.12-rc1.
> > >
> > > This has been independently confirmed on the official Raspberry Pi
> > > Linux issue tracker
> > > (raspberrypi/linux #7308), where the RPi kernel team directed the
> > > issue upstream.
> > >
> > >
> > > Hardware / Software
> > > -------------------
> > > - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
> > > - Raspberry Pi OS Bookworm (arm64)
> > > - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
> > > 6.12.81-v8-16k+ (custom)
> > > - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief
> > >
> > >
> > > Observed regression
> > > -------------------
> > > Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):
> > >
> > >   Kernel        pipe bogo ops/s   vs. 6.6
> > >   6.6.78            2 487 746      100%
> > >   6.12.75           1 651 427      -34%
> > >   6.18.21           2 049 701      -18%
> > >
> > > This regression pattern is consistent across two separate Raspberry Pi
> > > 5 units and has
> > > been independently reproduced by the RPi kernel team with 20-run averages:
> > > 6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).
> > >
> > >
> > > Runtime isolation via CONFIG_SCHED_DEBUG
> > > -----------------------------------------
> > > To isolate the root cause, I compiled a custom kernel (rpi-6.12.y,
> > > 6.12.81-v8-16k+)
> > > with CONFIG_SCHED_DEBUG=y and toggled scheduler features at runtime:
> > >
> > >   DELAY_DEQUEUE   PREEMPT_SHORT   pipe bogo ops/s   vs. baseline
> > >   on (default)    on (default)        1 506 572          --
> > >   OFF             on                  2 125 473         +41%   <==
> > >   on              OFF                 1 419 026          -6%
> > >   OFF             OFF                 2 078 182         +38%
> > >
> > > Disabling DELAY_DEQUEUE alone recovers +41% throughput, almost closing
> > > the gap to 6.6.
> > > Disabling PREEMPT_SHORT alone has no positive effect on this workload.
> > >
> > > The remaining gap to 6.6 (~15%) is likely CONFIG_SCHED_DEBUG=y overhead.
> > >
> > >
> > > Root cause analysis
> > > -------------------
> > > The pipe producer-consumer loop is affected by DELAY_DEQUEUE as follows:
> > >
> > > Before DELAY_DEQUEUE:
> > >   consumer reads empty pipe -> blocks -> dequeue_task() removes it from runqueue
> > >   producer writes -> wake_up_interruptible() -> consumer re-enqueued
> > > cleanly -> runs
> > >
> > > With DELAY_DEQUEUE (v6.12+):
> > >   consumer reads empty pipe -> blocks -> stays on runqueue (sched_delayed = 1)
> > >   producer writes -> wakeup path handles already-queued task ->
> > > additional bookkeeping
> >
> > because the task is still enqueued, the enqueue of delayed entity
> > should be faster most of the time
> >
> > >   per iteration
> > >
> > > For a tight 4-worker pipe benchmark at millions of iterations, this
> > > per-iteration
> > > overhead compounds directly into measured throughput.
> > >
> > > PREEMPT_SHORT (commit 85e511d) does not contribute to this regression.
> > > Its stated
> >
> > Unless you set a custom slice you should not see any difference with this patch
> >
> > > trade-off ("massive_intr workload gets more context switches") does
> > > not appear to be
> > > the bottleneck here.
> > >
> > >
> > > Mitigations tested and ruled out
> > > ---------------------------------
> > > - Spectre mitigations: mitigations=off yields only +0.5-2.5%
> > > improvement (confirmed by
> > >   RPi kernel team). Not the cause.
> > > - CPU governor: tested with both ondemand and performance. No
> > > significant difference.
> > >
> > >
> > > References
> > > ----------
> > > - Commit 152e11f (DELAY_DEQUEUE):
> > >   https://github.com/torvalds/linux/commit/152e11f6df293e816a6a37c69757033cdc72667d
> > > - Commit 85e511d (PREEMPT_SHORT):
> > >   https://github.com/torvalds/linux/commit/85e511df3cec46021024176672a748008ed135bf
> > > - RPi issue tracker: https://github.com/raspberrypi/linux/issues/7308
> > >
> > > Please let me know if additional data (perf traces, full benchmark
> > > logs, kernel config)
> > > would be helpful. I am happy to run further tests on the hardware.
> >
> > Could you trace and check that:
> > - consumer and producer of one pipe are on the same cpu
> > - if there is a diff in the number of migration
> >
> > I will try to reproduce locally once I get access to my hardware
> >
> > >
> > > Thank you for your work on the scheduler.
> > >
> > > Best regards,
> > > Thomas Gebhardt (@Kletternaut on GitHub)
> > >

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Mike Galbraith]

Greetings,

On Thu, 2026-04-16 at 14:22 +0200, Tom Gebhardt wrote:
> Hi Peter,
> 
> I would like to report a measurable pipe IPC throughput regression introduced by
> commit 152e11f ("sched/fair: Implement delayed dequeue"), first
> present in v6.12-rc1.
> 
> This has been independently confirmed on the official Raspberry Pi
> Linux issue tracker
> (raspberrypi/linux #7308), where the RPi kernel team directed the
> issue upstream.
> 
> 
> Hardware / Software
> -------------------
> - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM
> - Raspberry Pi OS Bookworm (arm64)
> - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712,
> 6.12.81-v8-16k+ (custom)
> - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief
> 
> 
> Observed regression
> -------------------
> Comparing pipe IPC throughput across kernels (overclocked, arm_freq=2800):
> 
>   Kernel        pipe bogo ops/s   vs. 6.6
>   6.6.78            2 487 746      100%
>   6.12.75           1 651 427      -34%
>   6.18.21           2 049 701      -18%
> 
> This regression pattern is consistent across two separate Raspberry Pi
> 5 units and has
> been independently reproduced by the RPi kernel team with 20-run averages:
> 6.6=2065 Kops/s, 6.12=1662, 6.18=1805, 7.0=1570 (lowest).

Does reproduction require more than the posted stress-ng args?

Raw data from my pi5+bookworm+6.12.75+rpt-rpi-2712 (no overclock)

rpi5:..debug/sched # stress-ng --version
stress-ng, version 0.21.00 (gcc 12.2.0, aarch64 Linux 6.12.75+rpt-rpi-2712) 💻🔥
rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2965] setting to a 20 secs run per stressor
stress-ng: info:  [2965] dispatching hogs: 4 pipe
stress-ng: metrc: [2965] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2965]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2965] pipe           57731375     20.00      6.69     73.24   2886616.84      722246.62
stress-ng: info:  [2965] skipped: 0
stress-ng: info:  [2965] passed: 4: pipe (4)
stress-ng: info:  [2965] failed: 0
stress-ng: info:  [2965] metrics untrustworthy: 0
stress-ng: info:  [2965] successful run completed in 20.00 secs
rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features
rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [3048] setting to a 20 secs run per stressor
stress-ng: info:  [3048] dispatching hogs: 4 pipe
stress-ng: metrc: [3048] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [3048]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [3048] pipe           58814508     20.00      6.78     73.17   2940713.90      735679.81
stress-ng: info:  [3048] skipped: 0
stress-ng: info:  [3048] passed: 4: pipe (4)
stress-ng: info:  [3048] failed: 0
stress-ng: info:  [3048] metrics untrustworthy: 0
stress-ng: info:  [3048] successful run completed in 20.00 secs
rpi5:..debug/sched # echo DELAY_DEQUEUE > features
rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [3349] setting to a 20 secs run per stressor
stress-ng: info:  [3349] dispatching hogs: 4 pipe
stress-ng: metrc: [3349] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [3349]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [3349] pipe           58058262     20.00      6.62     73.32   2902974.63      726327.83
stress-ng: info:  [3349] skipped: 0
stress-ng: info:  [3349] passed: 4: pipe (4)
stress-ng: info:  [3349] failed: 0
stress-ng: info:  [3349] metrics untrustworthy: 0
stress-ng: info:  [3349] successful run completed in 20.00 secs

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Tom Gebhardt]

Hi Mike,

thank you for testing — but I notice that your data only shows 6.12.75
with DELAY_DEQUEUE on/off. There is no 6.6 baseline in your results.
That is the comparison that matters: 6.6 vs 6.12 on the same hardware
with the same tool.

There is also a version difference worth noting: you are running
stress-ng 0.21.00, while our measurements used stress-ng 0.15.06 —
which was the version available on Raspberry Pi OS Bookworm at the
time of the original report. The pipe stressor calculation changed
significantly between those versions.

This creates an uncomfortable coincidence: a measurable scheduler
regression was introduced in 6.12 (DELAY_DEQUEUE), and around the same
time the standard measurement tool changed how it calculates the pipe
benchmark score. The result is that the regression becomes invisible
when comparing with the newer tool version — not because it was fixed,
but because the metric changed.

Our measurements were internally consistent: 6.6.78, 6.12.75, 6.12.81
and 6.18.21 were all compared using the same stress-ng 0.15.06 on the
same hardware. The −34% regression on 6.12 relative to 6.6 is real
within that methodology.

Additionally, our perf stat results are completely independent of
stress-ng version:

Kernel pipe bogo ops/s sched_switch
6.6.78 2 339 827 21 151 454
6.12.81 1 163 540 39 911 525

89% more context switches on 6.12 with DELAY_DEQUEUE active — that is
a kernel difference, not a measurement artifact.

Would you be willing to run the 6.6 comparison with stress-ng 0.15.06
on your Pi 5?

Best regards,
Thomas Gebhardt

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Mike Galbraith]

On Mon, 2026-04-20 at 12:59 +0200, Tom Gebhardt wrote:
> Hi Mike,
> 
> thank you for testing — but I notice that your data only shows 6.12.75
> with DELAY_DEQUEUE on/off. There is no 6.6 baseline in your results.
> That is the comparison that matters: 6.6 vs 6.12 on the same hardware
> with the same tool.
> 
> There is also a version difference worth noting: you are running
> stress-ng 0.21.00, while our measurements used stress-ng 0.15.06 —
> which was the version available on Raspberry Pi OS Bookworm at the
> time of the original report. The pipe stressor calculation changed
> significantly between those versions.
> 
> This creates an uncomfortable coincidence: a measurable scheduler
> regression was introduced in 6.12 (DELAY_DEQUEUE), and around the same
> time the standard measurement tool changed how it calculates the pipe
> benchmark score. The result is that the regression becomes invisible
> when comparing with the newer tool version — not because it was fixed,
> but because the metric changed.

Aha.

> Would you be willing to run the 6.6 comparison with stress-ng 0.15.06
> on your Pi 5?

No need, your reported DELAY_DEQUEUE impact for 6.12.75+rpt-rpi-2712
appeared with stress-ng 0.15.06.

However, my local 6.12.y branch kernel, which has all eevdf fixes as
well as Peter's still pending sched/ttwu series, still does NOT repro.

git@homer:..git/raspberrypi-kernel> quilt applied|grep delay
patches/eevdf/WIP/0043-sched-fair-Removed-unsued-cfs_rq.h_nr_delayed.patch
patches/eevdf/WIP/0047-sched-fair-Do-not-try-to-migrate-delayed-dequeue-task.patch           <== ?
patches/eevdf/WIP/0065-sched-Change-ttwu_runnable-vs-sched_delayed.patch
patches/eevdf/WIP/0066-sched-Add-ttwu_queue-support-for-delayed-tasks.patch
patches/eevdf/WIP/0089-sched-fair-Prevent-negative-lag-increase-during-delayed-dequeue.patch <== ?

	-Mike

aside: benchmark preemption model sensitivity magnitude poked me in the
eye while testing, numbers below in case that's of interest to anyone.

preempt=voluntary
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2566] setting to a 20 second run per stressor
stress-ng: info:  [2566] dispatching hogs: 4 pipe
stress-ng: metrc: [2566] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2566]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2566] pipe           45294217     20.00     22.71     57.23   2264640.83      566587.74
stress-ng: metrc: [2566] miscellaneous metrics:
stress-ng: metrc: [2566] pipe                 276.48 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2566] successful run completed in 20.01s
rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2650] setting to a 20 second run per stressor
stress-ng: info:  [2650] dispatching hogs: 4 pipe
stress-ng: metrc: [2650] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2650]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2650] pipe           45379684     20.00     22.69     57.24   2268917.79      567687.53
stress-ng: metrc: [2650] miscellaneous metrics:
stress-ng: metrc: [2650] pipe                 276.99 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2650] successful run completed in 20.00s
rpi5:..debug/sched # echo DELAY_DEQUEUE > features
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2721] setting to a 20 second run per stressor
stress-ng: info:  [2721] dispatching hogs: 4 pipe
stress-ng: metrc: [2721] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2721]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2721] pipe           45217909     20.00     21.81     58.13   2260821.48      565698.97
stress-ng: metrc: [2721] miscellaneous metrics:
stress-ng: metrc: [2721] pipe                 276.00 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2721] successful run completed in 20.00s
rpi5:..debug/sched #

preempt=lazy
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2070] setting to a 20 second run per stressor
stress-ng: info:  [2070] dispatching hogs: 4 pipe
stress-ng: metrc: [2070] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2070]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2070] pipe           32706095     20.00     17.88     62.02   1635263.85      409343.56
stress-ng: metrc: [2070] miscellaneous metrics:
stress-ng: metrc: [2070] pipe                 199.63 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2070] successful run completed in 20.00s
rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2149] setting to a 20 second run per stressor
stress-ng: info:  [2149] dispatching hogs: 4 pipe
stress-ng: metrc: [2149] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2149]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2149] pipe           32226171     20.00     17.93     61.98   1611264.75      403306.31
stress-ng: metrc: [2149] miscellaneous metrics:
stress-ng: metrc: [2149] pipe                 196.70 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2149] successful run completed in 20.01s
rpi5:..debug/sched # echo DELAY_DEQUEUE > features
rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
stress-ng: info:  [2210] setting to a 20 second run per stressor
stress-ng: info:  [2210] dispatching hogs: 4 pipe
stress-ng: metrc: [2210] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2210]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2210] pipe           32762333     20.00     18.08     61.86   1638067.42      409835.28
stress-ng: metrc: [2210] miscellaneous metrics:
stress-ng: metrc: [2210] pipe                 199.99 MB per sec pipe write rate (geometic mean of 4 instances)
stress-ng: info:  [2210] successful run completed in 20.01s

 

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Tom Gebhardt]

Hi Mike,

thank you for confirming that the DELAY_DEQUEUE impact is reproducible
with stress-ng 0.15.06 on the stock 6.12.75+rpt-rpi-2712 kernel — that
is exactly the validation we needed.

The fact that your local 6.12.y branch with patches 0047 and 0089
(Christian's "Prevent negative lag increase during delayed dequeue")
does not reproduce the issue is very encouraging. It suggests that the
downstream RPi kernel just hasn't picked up those fixes yet, which
explains why we still see the regression on stock rpi-6.12.y.

Regarding the preemption model data: both our 6.6.78 and 6.12.81
kernels are built with CONFIG_PREEMPT=y (full preemption), not
voluntary or lazy. Your observation that preempt=lazy shows ~28% lower
throughput even on the patched kernel is interesting in itself — we
were not aware of that sensitivity on arm64. On our hardware, 6.6.78
with full preempt gives ~2.34M ops/s, which is actually slightly above
your voluntary preempt score of ~2.26M, so the comparison tracks
reasonably well.

On the patched kernel, DELAY_DEQUEUE on/off causes less than 1%
difference in both preemption modes — compared to the ~50% regression
we observe on stock 6.12.81 with 0.15.06 (2.34M ops/s on 6.6.78 vs.
1.16M on 6.12.81, same hardware). That is a clear indication that 0089
is the key fix.

We also tested rpi-7.0.y (self-built, CONFIG_PREEMPT=y, 16k page size)
and 6.18.21. Here is the picture from our hardware (RPi5 C1-Stepping,
arm_freq=2400, stress-ng 0.15.06, 4 workers, 20 seconds):

Kernel pipe MB/s yield ns swapctx/s
-------------------------------------------------------
6.6.78-v8-16k+ 259.93 2072 2,060,126 ← reference
6.12.81-v8-16k+ 183.66 2164 2,081,527 ← DELAY_DEQUEUE=ON
6.12.81 ipc-perf 250.75 2310 2,070,523 ← DELAY_DEQUEUE=OFF
6.18.21-v8-16k+ 219.32 2073 2,089,853
7.0.0-v8-16k+ 206.43 2175 2,067,052

The DELAY_DEQUEUE=OFF result (250.75 MB/s) clearly confirms your
analysis: it recovers almost all of the regression versus 6.6 stock
(259.93 MB/s), leaving only ~4% gap. DELAY_DEQUEUE=ON on the same
kernel costs ~31%.

Interestingly, 7.0.0 still shows a significant regression versus
6.6.78 at the same frequency: −20% at 2400 MHz (206 vs 260 MB/s), −19%
at 2800 MHz OC. If patches 0047/0089 are already in 7.0, something
else may still be at play — or they have not yet landed in the
rpi-7.0.y tree.

Would it make sense to push 0047 and 0089 to the stable rpi-6.12.y
branch, or is the expectation that they land via the upstream LTS
route first?

Best regards,
Thomas Gebhardt

Am Di., 21. Apr. 2026 um 05:55 Uhr schrieb Mike Galbraith <efault@gmx.de>:
>
> On Mon, 2026-04-20 at 12:59 +0200, Tom Gebhardt wrote:
> > Hi Mike,
> >
> > thank you for testing — but I notice that your data only shows 6.12.75
> > with DELAY_DEQUEUE on/off. There is no 6.6 baseline in your results.
> > That is the comparison that matters: 6.6 vs 6.12 on the same hardware
> > with the same tool.
> >
> > There is also a version difference worth noting: you are running
> > stress-ng 0.21.00, while our measurements used stress-ng 0.15.06 —
> > which was the version available on Raspberry Pi OS Bookworm at the
> > time of the original report. The pipe stressor calculation changed
> > significantly between those versions.
> >
> > This creates an uncomfortable coincidence: a measurable scheduler
> > regression was introduced in 6.12 (DELAY_DEQUEUE), and around the same
> > time the standard measurement tool changed how it calculates the pipe
> > benchmark score. The result is that the regression becomes invisible
> > when comparing with the newer tool version — not because it was fixed,
> > but because the metric changed.
>
> Aha.
>
> > Would you be willing to run the 6.6 comparison with stress-ng 0.15.06
> > on your Pi 5?
>
> No need, your reported DELAY_DEQUEUE impact for 6.12.75+rpt-rpi-2712
> appeared with stress-ng 0.15.06.
>
> However, my local 6.12.y branch kernel, which has all eevdf fixes as
> well as Peter's still pending sched/ttwu series, still does NOT repro.
>
> git@homer:..git/raspberrypi-kernel> quilt applied|grep delay
> patches/eevdf/WIP/0043-sched-fair-Removed-unsued-cfs_rq.h_nr_delayed.patch
> patches/eevdf/WIP/0047-sched-fair-Do-not-try-to-migrate-delayed-dequeue-task.patch           <== ?
> patches/eevdf/WIP/0065-sched-Change-ttwu_runnable-vs-sched_delayed.patch
> patches/eevdf/WIP/0066-sched-Add-ttwu_queue-support-for-delayed-tasks.patch
> patches/eevdf/WIP/0089-sched-fair-Prevent-negative-lag-increase-during-delayed-dequeue.patch <== ?
>
>         -Mike
>
> aside: benchmark preemption model sensitivity magnitude poked me in the
> eye while testing, numbers below in case that's of interest to anyone.
>
> preempt=voluntary
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2566] setting to a 20 second run per stressor
> stress-ng: info:  [2566] dispatching hogs: 4 pipe
> stress-ng: metrc: [2566] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2566]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2566] pipe           45294217     20.00     22.71     57.23   2264640.83      566587.74
> stress-ng: metrc: [2566] miscellaneous metrics:
> stress-ng: metrc: [2566] pipe                 276.48 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2566] successful run completed in 20.01s
> rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2650] setting to a 20 second run per stressor
> stress-ng: info:  [2650] dispatching hogs: 4 pipe
> stress-ng: metrc: [2650] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2650]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2650] pipe           45379684     20.00     22.69     57.24   2268917.79      567687.53
> stress-ng: metrc: [2650] miscellaneous metrics:
> stress-ng: metrc: [2650] pipe                 276.99 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2650] successful run completed in 20.00s
> rpi5:..debug/sched # echo DELAY_DEQUEUE > features
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2721] setting to a 20 second run per stressor
> stress-ng: info:  [2721] dispatching hogs: 4 pipe
> stress-ng: metrc: [2721] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2721]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2721] pipe           45217909     20.00     21.81     58.13   2260821.48      565698.97
> stress-ng: metrc: [2721] miscellaneous metrics:
> stress-ng: metrc: [2721] pipe                 276.00 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2721] successful run completed in 20.00s
> rpi5:..debug/sched #
>
> preempt=lazy
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2070] setting to a 20 second run per stressor
> stress-ng: info:  [2070] dispatching hogs: 4 pipe
> stress-ng: metrc: [2070] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2070]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2070] pipe           32706095     20.00     17.88     62.02   1635263.85      409343.56
> stress-ng: metrc: [2070] miscellaneous metrics:
> stress-ng: metrc: [2070] pipe                 199.63 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2070] successful run completed in 20.00s
> rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2149] setting to a 20 second run per stressor
> stress-ng: info:  [2149] dispatching hogs: 4 pipe
> stress-ng: metrc: [2149] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2149]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2149] pipe           32226171     20.00     17.93     61.98   1611264.75      403306.31
> stress-ng: metrc: [2149] miscellaneous metrics:
> stress-ng: metrc: [2149] pipe                 196.70 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2149] successful run completed in 20.01s
> rpi5:..debug/sched # echo DELAY_DEQUEUE > features
> rpi5:..debug/sched # /usr/bin/stress-ng --pipe 4 --timeout 20s --metrics-brief
> stress-ng: info:  [2210] setting to a 20 second run per stressor
> stress-ng: info:  [2210] dispatching hogs: 4 pipe
> stress-ng: metrc: [2210] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
> stress-ng: metrc: [2210]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: metrc: [2210] pipe           32762333     20.00     18.08     61.86   1638067.42      409835.28
> stress-ng: metrc: [2210] miscellaneous metrics:
> stress-ng: metrc: [2210] pipe                 199.99 MB per sec pipe write rate (geometic mean of 4 instances)
> stress-ng: info:  [2210] successful run completed in 20.01s
>
>

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Mike Galbraith]

On Tue, 2026-04-21 at 14:46 +0200, Tom Gebhardt wrote:
> 
> The fact that your local 6.12.y branch with patches 0047 and 0089
> (Christian's "Prevent negative lag increase during delayed dequeue")
> does not reproduce the issue is very encouraging. It suggests that the
> downstream RPi kernel just hasn't picked up those fixes yet, which
> explains why we still see the regression on stock rpi-6.12.y.

Not sure.  Taking stress-ng 0.15.06 to trusty ole x86_64 i4790 desktop
box, since I see no excuse for it to be arch specific, the virgin
6.12.82 DELAY_DEQUEUE delta for a pure voluntary config (lowest
overhead) was -6%.  Adding all eevdf backports took it to -3%.  To me
that looks very much like a nothing-burger, ie routine bench jitter
when diddling *anything* in fastpaths.

The only clear signal has come from that pi5 6.12.75+rpt-rpi-2712 plus
stress-ng 0.15.06 combo.


	-Mike

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Tue, 2026-04-21 at 05:55 +0200, Mike Galbraith wrote:
> On Mon, 2026-04-20 at 12:59 +0200, Tom Gebhardt wrote:
> 
> 
> > Would you be willing to run the 6.6 comparison with stress-ng 0.15.06
> > on your Pi 5?
> 
> No need, your reported DELAY_DEQUEUE impact for 6.12.75+rpt-rpi-2712
> appeared with stress-ng 0.15.06.
> 
> However, my local 6.12.y branch kernel, which has all eevdf fixes as
> well as Peter's still pending sched/ttwu series, still does NOT repro.
> 
> git@homer:..git/raspberrypi-kernel> quilt applied|grep delay
> patches/eevdf/WIP/0043-sched-fair-Removed-unsued-cfs_rq.h_nr_delayed.patch
> patches/eevdf/WIP/0047-sched-fair-Do-not-try-to-migrate-delayed-dequeue-task.patch           <== ?
> patches/eevdf/WIP/0065-sched-Change-ttwu_runnable-vs-sched_delayed.patch
> patches/eevdf/WIP/0066-sched-Add-ttwu_queue-support-for-delayed-tasks.patch
> patches/eevdf/WIP/0089-sched-fair-Prevent-negative-lag-increase-during-delayed-dequeue.patch <== ?

I've finally finished slogging through my quilt backport battery, and
my dinky/cute rpi5 asserts that what prevents the local heavily patched
rpi-6.12-y@6.12.81 kernel from reproducing has nothing to do with
pointed at suspects, rather Peter's long pending ttwu series is
entirely responsible.

Validation enabling tarball attached.

	-Mike

[-- Attachment #2: peterz-queue-ttwu.tar.gz --]
[-- Type: application/x-compressed-tar, Size: 9155 bytes --]

Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5

[Peter Zijlstra]

On Thu, Apr 23, 2026 at 06:51:20AM +0200, Mike Galbraith wrote:
> On Tue, 2026-04-21 at 05:55 +0200, Mike Galbraith wrote:
> > On Mon, 2026-04-20 at 12:59 +0200, Tom Gebhardt wrote:
> > 
> > 
> > > Would you be willing to run the 6.6 comparison with stress-ng 0.15.06
> > > on your Pi 5?
> > 
> > No need, your reported DELAY_DEQUEUE impact for 6.12.75+rpt-rpi-2712
> > appeared with stress-ng 0.15.06.
> > 
> > However, my local 6.12.y branch kernel, which has all eevdf fixes as
> > well as Peter's still pending sched/ttwu series, still does NOT repro.
> > 
> > git@homer:..git/raspberrypi-kernel> quilt applied|grep delay
> > patches/eevdf/WIP/0043-sched-fair-Removed-unsued-cfs_rq.h_nr_delayed.patch
> > patches/eevdf/WIP/0047-sched-fair-Do-not-try-to-migrate-delayed-dequeue-task.patch           <== ?
> > patches/eevdf/WIP/0065-sched-Change-ttwu_runnable-vs-sched_delayed.patch
> > patches/eevdf/WIP/0066-sched-Add-ttwu_queue-support-for-delayed-tasks.patch
> > patches/eevdf/WIP/0089-sched-fair-Prevent-negative-lag-increase-during-delayed-dequeue.patch <== ?
> 
> I've finally finished slogging through my quilt backport battery, and
> my dinky/cute rpi5 asserts that what prevents the local heavily patched
> rpi-6.12-y@6.12.81 kernel from reproducing has nothing to do with
> pointed at suspects, rather Peter's long pending ttwu series is
> entirely responsible.

Humm, I'd almost forgotten about that. I suppose I'll have to go dust
that off again.