Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Steven Rostedt <rostedt@goodmis.org>]

On Thu, 15 Oct 2026 14:19:15 +0800
Chaosheng Qu <chaosheng.qu.linux@gmail.com> wrote:

> Hi Steve, Martin,

Hi Chaosheng,

Thanks for the in-depth analysis!

> 
> I do not have Martin's workload, so this is not a reproduction of the reported
> audio dropouts. What I have is a narrower synthetic experiment that may help
> locate the non-PUSH failure Steve suspected. Sending data, not conclusions, and
> no patch yet.
> 
> The short version: with RT_PUSH_IPI off, a PI-boosted task that cannot run on
> its own CPU can be stuck in a state where the pull side cannot reach it and the
> push side was never queued, until the task holding the CPU schedules out on its
> own. Two things have to line up for that, and one of them (affinity) is in my
> first attempt to get this wrong, so I have included the counter-example below.
> 
> Setup
> -----
> 
> QEMU/KVM, 4 vCPU, non-PREEMPT_RT, 7.3-rc7 (24a19b953e2a). The feature is
> toggled at runtime via /sys/kernel/debug/sched/features. Four SCHED_FIFO
> threads:
> 
>   cpu1: L  FIFO 10  owner of PI mutex M, affinity {cpu1,cpu3}
>   cpu1: X  FIFO 99  hog, busy for the whole dwell
>   cpu2: W  FIFO 90  waiter, blocks on M     <- deliberately outside L's mask
>   cpu3: R  FIFO 60  spare, spinning
> 
> W blocking on M boosts L from 10 to 90; the trace confirms it:
> 
>   waiter-114 [002] d..3. 52.286120: sched_pi_setprio: comm=owner pid=115 oldprio=89 newprio=9

After reading the above, I can see the issue immediately.

> 
> L then cannot run on cpu1 because X holds it, and its only way out is to be
> moved to cpu3. cpu3's spare runs at FIFO 60, which matters: before the boost L
> has prio 89, and find_lock_lowest_rq() requires the target's
> highest_prio.curr to be strictly higher priority than the task being moved, so
> cpu3 is not a legal target until the boost happens. That way the opportunity is
> created by the boost itself and not by some unrelated migration.
> 
> Measurements, dwell 1500 ms
> ---------------------------
> 
>   RT_PUSH_IPI   n   resumed <100 ms   owner resume median   min / max
>   off (default) 8   0/8               1499.6 ms             1146.1 / 1502.6 ms
>   on            6   6/6                 0.8 ms                0.6 / 5.4 ms
>   off, no boost 2   0/2               1499.2 ms            1499.1 / 1499.2 ms
> 
> In the 8 "off" runs, 7 never migrated at all: L stayed on cpu1 and started only
> once X finished its dwell. The 8th moved to cpu3 5 ms before X exited, which is
> a by-product of the hog leaving rather than a rescue, so I used a fixed 100 ms
> threshold for "timely" rather than "eventually moved". With the dwell raised to
> 4000 ms the waiter was blocked for 2.26 s before L finally moved.
> 
> The third row is a control with PTHREAD_PRIO_NONE instead of PRIO_INHERIT, to
> check that the opportunity really comes from the boost. Without the boost L
> stays at prio 89, cpu3's curr is 39, 39 > 89 is false, and so there is no legal
> migration target at all. Both feature settings look the same there, which is
> the expected result.
> 
> The affinity counter-example
> ----------------------------
> 
> My first version gave L affinity {cpu1,cpu2}, that is, including the CPU whose
> priority drops when W blocks. In that shape both settings rescued L in 0.5-6.2
> ms and the two paths do not diverge at all. The reason is that cpu2 can pull L
> inline, and task_is_pushable() only requires
> cpumask_test_cpu(cpu, &p->cpus_mask) for the pulling CPU. The split only shows
> up once that CPU is removed from L's mask.
> 
> So this is a fairly narrow condition: PI boost, RT overload on the owner's CPU,
> and affinity that excludes the CPU that just went idle. I do not know whether
> Martin's livefx workers hit all three. The trace in his report has the worker
> boosted and enqueued but never scheduled while other CPUs idle, and nothing in
> his description rules it out, but I cannot confirm it from the data in the
> report.
> 
> Where the non-PUSH path appears to lose it
> ------------------------------------------
> 
> Both sides have to miss the boosted task.
> 
> 1. The pull side cannot reach it. In the non-PUSH branch, pull_rt_task() walks
>    the overloaded CPUs and calls pick_highest_pushable_task(src_rq, this_cpu),
>    which requires cpumask_test_cpu(cpu, &p->cpus_mask). L's mask excludes cpu2,
>    so cpu2's loop finds nothing and returns empty-handed.
> 
> 2. The push side was never queued. rt_queue_push_tasks() is only called from
>    set_next_task_rt() and switched_to_rt(). prio_changed_rt() queues a pull when
>    the running task's priority drops, but it does not queue a push. The boost
>    arrives through rt_mutex_setprio() -> sched_change_end() ->
>    prio_changed_rt(). L is not the current task on cpu1 (X is), so it takes the
>    non-current branch, which only ends up doing resched_curr().
> 
> The result is that after the boost, cpu1 will not queue a push until X itself
> schedules out, and cpu2 cannot pull L because of the affinity check.
> 
> The two paths are separable in the trace by the hardirq marker:
> 
>   # RT_PUSH_IPI on - the ipi runs the push in hardirq context on cpu1
>   hog-122 [001] d.h2. 53.803264: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
> 
>   # off - no ipi; the push happens on cpu1's ordinary scheduling path,
>   # and in this run only 5 ms before X left on its own
>   hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
>   hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X
> 
> One more observation, in case it is useful: prio_changed_rt() does not appear to
> have ever queued a push. I walked that function's history and did not find a
> revision that did. That would be consistent with Steve's guess that this is a
> long-standing bug that the PUSH path happened to cover, rather than something
> dd29c017aed6 introduced.

Exactly. The issue that you show here appears to be that the boosting of
the RT task should trigger another analysis of RT migration. Which we do
not do.

> 
> What I am not proposing
> -----------------------
> 
> I am not suggesting flipping the RT_PUSH_IPI default. The thundering-herd
> reason for keeping it off outside PREEMPT_RT is clear, and trading a regression
> for that is not a fix.
> 
> If this is too far from the real workload to be worth chasing, or if there is a
> specific trace you would rather have, tell me and I will either drop it or
> collect that instead. The probe and the full logs are small; happy to send them
> to anyone who wants to rerun it.

I think this is a legitimate bug and should be fix. We need to make the
prio_changed_rt() aware of the rto mask and trigger a push if needed.

-- Steve