Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Steven Rostedt <rostedt@goodmis.org>]

On Fri, 14 Aug 2026 16:12:35 -0500
Martin King <martin3kx@gmail.com> wrote:

> Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
> PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
> non-PREEMPT_RT desktop running a DAW.

Thanks for the report.

> 
> Symptom
> -------
> Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
> 2-3 minutes of use) while the machine is otherwise idle and responsive.
> 
> Mechanism (as measured, not inferred)
> -------------------------------------
> REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
> worker threads. During a stall:
> 
>   - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
>   - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
>     and is enqueued but never scheduled
>   - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:
> 
>         reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
>         (during a 3403ms dropout)
> 
>   - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
>     thread does meaningful work (largest run was 7.2ms across 3.4s)
>   - the machine stays responsive throughout; unrelated SCHED_OTHER work
>     proceeds at 84-94% of normal rate with 0-1ms runqueue wait
>   - all 16 cores keep taking timer interrupts at normal rates
> 
> So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
> cores idle. With push disabled nothing hands it to an idle CPU.
> 
> Evidence
> --------------------------------
>   kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
>   7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
> 23/265/434 per min
>   7.1.3   on             39 min   0                     0
>   7.1.8   on             42 min   0                     0
> 
> The third row differs from the first ONLY in the sched feature: Identical
> kernel, toolchain, GPU driver, project and audio configuration. At the
> observed broken rate ~13 dropouts were expected in that window; zero were
> seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
> cycles at quantum 256 / 48kHz).
> 
> Workaround
> ----------
>   echo RT_PUSH_IPI > /sys/kernel/debug/sched/features
> 
> Applied at boot via a systemd unit.
> 
> System
> ------
>   AMD Ryzen 7 5800X (8c/16t), Arch Linux
>   kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
>   PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
>   CONFIG_PREEMPT_RT not set

Unfortunately, I'm going to be traveling next week and I may not be able to
look into this until I get back. I don't see anything that the normal (non
PUSH) logic would fail in this case. Thus I'm thinking it simply has a bug
that has been hiding because we've been using the PUSH logic.

I may be asking for some traces here when I get back to find out the exact
cause. Use the workaround for now.

The RT_PUSH_IPI was added because of a "thundering herd" of IPIs when all
but one CPU dropped its priority and the one still running had more than
one RT task on it. That would cause all the other CPUs to send an IPI to
that CPU to do a pull, but only would would get it. The RT_PUSH_IPI was
created to pass around a single IPI to do the push for all CPUs wanting a
new RT task to run.

The above use case doesn't look like it is having that. Thus, I'm thinking
something else broke.

-- Steve