[REPORT] sched: ITMT + DELAY_DEQUEUE: woken task waits ~1 ms behind another task on the preferred CPU while allowed CPUs are idle

[REPORT] sched: ITMT + DELAY_DEQUEUE: woken task waits ~1 ms behind another task on the preferred CPU while allowed CPUs are idle

[Francesco Nigro]

Hi,

A woken thread waits behind another running thread while other CPUs it
is allowed to use are idle.

I see this on an AMD Ryzen 9 7950X running Fedora's 7.1.13 kernel, with
ITMT (AMD preferred cores) and DELAY_DEQUEUE enabled. Two busy threads
end up on the same CPU, the highest-ranked of the ones they are allowed
on. When one of them is woken, it is queued behind the other one and
starts one full CPU burst late (1 ms here). That happens to 0.08-0.36 %
of wakeups. Reproducer, scripts and raw results (commit 49b6d72):
https://github.com/franz1981/itmt-delay-dequeue-repro

Kernel and machine

  - Fedora 7.1.13-100.fc43.x86_64, untainted, Fedora's stock .config
    (at the link above): HZ=1000, PREEMPT_DYNAMIC (lazy), SCHED_MC_PRIO,
    SCHED_AUTOGROUP, FAIR_GROUP_SCHED, NO_HZ_FULL. Fedora's
    patch-7.1-redhat.patch touches nothing under kernel/sched/, nor
    itmt.c, amd-pstate or cpufreq. I cannot boot mainline or a patched
    kernel on this machine.
  - AMD Ryzen 9 7950X, SMT on, amd_pstate active with prefcore,
    sched_itmt_enabled=Y. CPUs pinned at 4300 MHz, boost off.
  - CPUs 4-7 have prefcore rankings 211, 216, 231, 221 and the same
    amd_pstate_highest_perf (166): CPU 6 is not faster than the others.

Reproducer

Two threads each run a calibrated 1 ms CPU burst and block on an eventfd.
A waker thread on CPU 2 busy-polls and writes each eventfd 10 us after
that thread's burst ends. A handoff (waker's timestamp before the write
to read() returning) of 0.5 ms or more counts as a stall.

    gcc -O2 -pthread -o asym_min asym_min.c
    taskset -c 4-7 ./asym_min 1000 10 30 2

asym_min.c (195 lines):
https://github.com/franz1981/itmt-delay-dequeue-repro/blob/main/asym_min.c

Results, 5 interleaved rounds of 30 s (results-final-min.txt):

    setting                      stalls per run         total
    defaults                     212  49  48  76 214    599 of 294616
    sched_itmt_enabled=0           1   0   0   0   0      1 of 295349
    NO_DELAY_DEQUEUE              21   9  11  26  14     81 of 295201
    defaults, SMT siblings
    20-23 also allowed            16  27  15  13   2     73 of 295215

The median handoff is 3 us in every setting.

Trace

stackcheck.bt records, per wakeup, whether se.sched_delayed was set at
try_to_wake_up() and whether select_task_rq_fair() ran. Two 30 s runs with
the defaults (results-final-min-trace.txt):

                                                    run 1    run 2
    worker wakeups                                  58592    58770
      delayed, no select_task_rq_fair()               340      174
    waits >= 0.5 ms                                   310      153
      on CPU 6 behind the other thread                288      137
      after a delayed wakeup, no CPU selection        280      139
      other thread moved to CPU 6 by
        active_load_balance_cpu_stop()                 30       12

In a third run (stackcheck-idle.bt), 3 of the 4 allowed CPUs were idle
when the waiting thread was woken, in all 102 waits >= 0.5 ms.

The code path in 7.1.13:

    fair.c:dequeue_entity()
        if (sched_feat(DELAY_DEQUEUE) && delay &&
            !entity_eligible(cfs_rq, se)) {
            ...
            set_delayed(se);
            return false;
        }

    core.c:try_to_wake_up()
        if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
            break;
        ...
        cpu = select_task_rq(p, p->wake_cpu, &wake_flags);

    core.c:ttwu_runnable()
        if (p->se.sched_delayed)
            enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);

Sequence, from the trace flags and the code above (A waits, B runs):

    CPU 6                                   CPUs 4, 5, 7
    -----                                   ------------
    A ends its burst and blocks while
    not eligible: stays queued, delayed
    B runs its 1 ms burst                   idle
    the waker writes A's eventfd;
    ttwu_runnable() re-queues A on CPU 6,
    no select_task_rq()                     idle
    B's burst ends, A runs ~1 ms late

A task alone on its cfs_rq is always eligible, so A can only be delayed
while B shares CPU 6. How the two first end up there is only partly
traced (active balance in 30 and 12 of the waits above).

Mainline (7.4-rc1) still re-queues a delayed task in place in
ttwu_runnable(). I have not run it.

Related, not tested here

  - Mike Galbraith, "sched/fair: Dequeue sched_delayed tasks when waking
    to a busy CPU" (tip sched/urgent, 2024-11-11; not in 7.1.13 or
    7.4-rc1) and its V2 thread:
    https://lore.kernel.org/all/bf4f50886c462ee1f33cc404843944fea4817616.camel@gmx.de/
  - Peter Zijlstra, "[PATCH v2 11/12] sched: Change ttwu_runnable() vs
    sched_delayed" (2025-07-02, not merged):
    https://lore.kernel.org/all/20250702121159.535226098@infradead.org/
  - "sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on
    Raspberry Pi 5" (2026-04), a different symptom, no ITMT:
    https://lore.kernel.org/all/20260423080453.GQ3126523@noisy.programming.kicks-ass.net/

Questions

  1. Is this cost expected when a delayed task is woken on a busy CPU
     while other allowed CPUs are idle? Here the busy CPU is not faster
     than the idle ones, and each stall costs one full burst.
  2. Is the ttwu_runnable() change in the threads above still planned,
     and would it be expected to cover this case?
  3. What else would be useful to trace or measure on this machine?

Notes

  - Tested on this machine only; Intel ITMT is untested. Which settings
    matter elsewhere is a guess: the allowed CPUs relative to the
    preferred-core ranking, SMT siblings, the waker CPU. scan.sh at the
    link above tries several CPU sets without root.
  - asym_stack2.c at the link above is an earlier, longer version of the
    reproducer. It showed the same pattern at about 3x the stall rate
    (results-final.txt); I don't know why the rates differ.

Tool use: an AI coding assistant (Claude Code, Claude Opus 5.5) wrote
the scripts, ran the measurements on this machine and drafted this
report; Codex helped with earlier analysis. I reviewed the method and
the numbers against the raw files and can answer for them.

Thanks,
Franz

Re: [REPORT] sched: ITMT + DELAY_DEQUEUE: woken task waits ~1 ms behind another task on the preferred CPU while allowed CPUs are idle

[Mike Galbraith]

On Tue, 2026-11-03 at 16:46 +0100, Francesco Nigro wrote:
> 
> Related, not tested here
> 
>   - Mike Galbraith, "sched/fair: Dequeue sched_delayed tasks when waking
>     to a busy CPU" (tip sched/urgent, 2024-11-11; not in 7.1.13 or
>     7.4-rc1) and its V2 thread:
>     https://lore.kernel.org/all/bf4f50886c462ee1f33cc404843944fea4817616.camel@gmx.de/

Mmm, that's a be careful what you wish for thing.  I flushed that one
down the toilet because while lowest possible service latency always
sounds super sexy, it has down sides. Rabid migration does not do great
things for throughput, for instance, that patch slagged the throughput
gain for tbench that put DELAY_DEQUEUE equipped kernels back on par
with pre-EEVDF kernels.  Not a keeper.

	-Mike