[REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Martin King]

Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
non-PREEMPT_RT desktop running a DAW.

Symptom
-------
Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
2-3 minutes of use) while the machine is otherwise idle and responsive.

Mechanism (as measured, not inferred)
-------------------------------------
REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
worker threads. During a stall:

  - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
  - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
    and is enqueued but never scheduled
  - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:

        reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
        (during a 3403ms dropout)

  - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
    thread does meaningful work (largest run was 7.2ms across 3.4s)
  - the machine stays responsive throughout; unrelated SCHED_OTHER work
    proceeds at 84-94% of normal rate with 0-1ms runqueue wait
  - all 16 cores keep taking timer interrupts at normal rates

So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
cores idle. With push disabled nothing hands it to an idle CPU.

Evidence
--------------------------------
  kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
  7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
23/265/434 per min
  7.1.3   on             39 min   0                     0
  7.1.8   on             42 min   0                     0

The third row differs from the first ONLY in the sched feature: Identical
kernel, toolchain, GPU driver, project and audio configuration. At the
observed broken rate ~13 dropouts were expected in that window; zero were
seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
cycles at quantum 256 / 48kHz).

Workaround
----------
  echo RT_PUSH_IPI > /sys/kernel/debug/sched/features

Applied at boot via a systemd unit.

System
------
  AMD Ryzen 7 5800X (8c/16t), Arch Linux
  kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
  PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
  CONFIG_PREEMPT_RT not set

Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Steven Rostedt]

On Fri, 14 Aug 2026 16:12:35 -0500
Martin King <martin3kx@gmail.com> wrote:

> Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
> PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
> non-PREEMPT_RT desktop running a DAW.

Thanks for the report.

> 
> Symptom
> -------
> Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
> 2-3 minutes of use) while the machine is otherwise idle and responsive.
> 
> Mechanism (as measured, not inferred)
> -------------------------------------
> REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
> worker threads. During a stall:
> 
>   - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
>   - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
>     and is enqueued but never scheduled
>   - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:
> 
>         reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
>         (during a 3403ms dropout)
> 
>   - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
>     thread does meaningful work (largest run was 7.2ms across 3.4s)
>   - the machine stays responsive throughout; unrelated SCHED_OTHER work
>     proceeds at 84-94% of normal rate with 0-1ms runqueue wait
>   - all 16 cores keep taking timer interrupts at normal rates
> 
> So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
> cores idle. With push disabled nothing hands it to an idle CPU.
> 
> Evidence
> --------------------------------
>   kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
>   7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
> 23/265/434 per min
>   7.1.3   on             39 min   0                     0
>   7.1.8   on             42 min   0                     0
> 
> The third row differs from the first ONLY in the sched feature: Identical
> kernel, toolchain, GPU driver, project and audio configuration. At the
> observed broken rate ~13 dropouts were expected in that window; zero were
> seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
> cycles at quantum 256 / 48kHz).
> 
> Workaround
> ----------
>   echo RT_PUSH_IPI > /sys/kernel/debug/sched/features
> 
> Applied at boot via a systemd unit.
> 
> System
> ------
>   AMD Ryzen 7 5800X (8c/16t), Arch Linux
>   kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
>   PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
>   CONFIG_PREEMPT_RT not set

Unfortunately, I'm going to be traveling next week and I may not be able to
look into this until I get back. I don't see anything that the normal (non
PUSH) logic would fail in this case. Thus I'm thinking it simply has a bug
that has been hiding because we've been using the PUSH logic.

I may be asking for some traces here when I get back to find out the exact
cause. Use the workaround for now.

The RT_PUSH_IPI was added because of a "thundering herd" of IPIs when all
but one CPU dropped its priority and the one still running had more than
one RT task on it. That would cause all the other CPUs to send an IPI to
that CPU to do a pull, but only would would get it. The RT_PUSH_IPI was
created to pass around a single IPI to do the push for all CPUs wanting a
new RT task to run.

The above use case doesn't look like it is having that. Thus, I'm thinking
something else broke.

-- Steve

Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Thorsten Leemhuis]

On 8/15/26 00:36, Steven Rostedt wrote:
> On Fri, 14 Aug 2026 16:12:35 -0500
> Martin King <martin3kx@gmail.com> wrote:
> 
>> Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
>> PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
>> non-PREEMPT_RT desktop running a DAW.
> 
> Thanks for the report.

This made it to my list of tracked regression -- but it looks like there
was no progress for a while. Is that the case? Steven, did that maybe
fall through the cracks due to the travel you mentioned? Or was this
resolved meanwhile and I just missed it?

Ciao, Thorsten

>> Symptom
>> -------
>> Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
>> 2-3 minutes of use) while the machine is otherwise idle and responsive.
>>
>> Mechanism (as measured, not inferred)
>> -------------------------------------
>> REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
>> worker threads. During a stall:
>>
>>   - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
>>   - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
>>     and is enqueued but never scheduled
>>   - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:
>>
>>         reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
>>         (during a 3403ms dropout)
>>
>>   - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
>>     thread does meaningful work (largest run was 7.2ms across 3.4s)
>>   - the machine stays responsive throughout; unrelated SCHED_OTHER work
>>     proceeds at 84-94% of normal rate with 0-1ms runqueue wait
>>   - all 16 cores keep taking timer interrupts at normal rates
>>
>> So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
>> cores idle. With push disabled nothing hands it to an idle CPU.
>>
>> Evidence
>> --------------------------------
>>   kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
>>   7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
>> 23/265/434 per min
>>   7.1.3   on             39 min   0                     0
>>   7.1.8   on             42 min   0                     0
>>
>> The third row differs from the first ONLY in the sched feature: Identical
>> kernel, toolchain, GPU driver, project and audio configuration. At the
>> observed broken rate ~13 dropouts were expected in that window; zero were
>> seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
>> cycles at quantum 256 / 48kHz).
>>
>> Workaround
>> ----------
>>   echo RT_PUSH_IPI > /sys/kernel/debug/sched/features
>>
>> Applied at boot via a systemd unit.
>>
>> System
>> ------
>>   AMD Ryzen 7 5800X (8c/16t), Arch Linux
>>   kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
>>   PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
>>   CONFIG_PREEMPT_RT not set
> 
> Unfortunately, I'm going to be traveling next week and I may not be able to
> look into this until I get back. I don't see anything that the normal (non
> PUSH) logic would fail in this case. Thus I'm thinking it simply has a bug
> that has been hiding because we've been using the PUSH logic.
> 
> I may be asking for some traces here when I get back to find out the exact
> cause. Use the workaround for now.
> 
> The RT_PUSH_IPI was added because of a "thundering herd" of IPIs when all
> but one CPU dropped its priority and the one still running had more than
> one RT task on it. That would cause all the other CPUs to send an IPI to
> that CPU to do a pull, but only would would get it. The RT_PUSH_IPI was
> created to pass around a single IPI to do the push for all CPUs wanting a
> new RT task to run.
> 
> The above use case doesn't look like it is having that. Thus, I'm thinking
> something else broke.
> 
> -- Steve
> 

Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Steven Rostedt]

On Thu, 3 Sep 2026 13:27:59 +0200
Thorsten Leemhuis <regressions@leemhuis.info> wrote:

> This made it to my list of tracked regression -- but it looks like there
> was no progress for a while. Is that the case? Steven, did that maybe
> fall through the cracks due to the travel you mentioned? Or was this
> resolved meanwhile and I just missed it?

No, I was traveling during this and now I'm way behind in other work. I'm
not sure when I'll get to look at it.

Feel free to keep pinging me about it ;-)

-- Steve

Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Chaosheng Qu]

Hi Steve, Martin,

I do not have Martin's workload, so this is not a reproduction of the reported
audio dropouts. What I have is a narrower synthetic experiment that may help
locate the non-PUSH failure Steve suspected. Sending data, not conclusions, and
no patch yet.

The short version: with RT_PUSH_IPI off, a PI-boosted task that cannot run on
its own CPU can be stuck in a state where the pull side cannot reach it and the
push side was never queued, until the task holding the CPU schedules out on its
own. Two things have to line up for that, and one of them (affinity) is in my
first attempt to get this wrong, so I have included the counter-example below.

Setup
-----

QEMU/KVM, 4 vCPU, non-PREEMPT_RT, 7.3-rc7 (24a19b953e2a). The feature is
toggled at runtime via /sys/kernel/debug/sched/features. Four SCHED_FIFO
threads:

  cpu1: L  FIFO 10  owner of PI mutex M, affinity {cpu1,cpu3}
  cpu1: X  FIFO 99  hog, busy for the whole dwell
  cpu2: W  FIFO 90  waiter, blocks on M     <- deliberately outside L's mask
  cpu3: R  FIFO 60  spare, spinning

W blocking on M boosts L from 10 to 90; the trace confirms it:

  waiter-114 [002] d..3. 52.286120: sched_pi_setprio: comm=owner pid=115 oldprio=89 newprio=9

L then cannot run on cpu1 because X holds it, and its only way out is to be
moved to cpu3. cpu3's spare runs at FIFO 60, which matters: before the boost L
has prio 89, and find_lock_lowest_rq() requires the target's
highest_prio.curr to be strictly higher priority than the task being moved, so
cpu3 is not a legal target until the boost happens. That way the opportunity is
created by the boost itself and not by some unrelated migration.

Measurements, dwell 1500 ms
---------------------------

  RT_PUSH_IPI   n   resumed <100 ms   owner resume median   min / max
  off (default) 8   0/8               1499.6 ms             1146.1 / 1502.6 ms
  on            6   6/6                 0.8 ms                0.6 / 5.4 ms
  off, no boost 2   0/2               1499.2 ms            1499.1 / 1499.2 ms

In the 8 "off" runs, 7 never migrated at all: L stayed on cpu1 and started only
once X finished its dwell. The 8th moved to cpu3 5 ms before X exited, which is
a by-product of the hog leaving rather than a rescue, so I used a fixed 100 ms
threshold for "timely" rather than "eventually moved". With the dwell raised to
4000 ms the waiter was blocked for 2.26 s before L finally moved.

The third row is a control with PTHREAD_PRIO_NONE instead of PRIO_INHERIT, to
check that the opportunity really comes from the boost. Without the boost L
stays at prio 89, cpu3's curr is 39, 39 > 89 is false, and so there is no legal
migration target at all. Both feature settings look the same there, which is
the expected result.

The affinity counter-example
----------------------------

My first version gave L affinity {cpu1,cpu2}, that is, including the CPU whose
priority drops when W blocks. In that shape both settings rescued L in 0.5-6.2
ms and the two paths do not diverge at all. The reason is that cpu2 can pull L
inline, and task_is_pushable() only requires
cpumask_test_cpu(cpu, &p->cpus_mask) for the pulling CPU. The split only shows
up once that CPU is removed from L's mask.

So this is a fairly narrow condition: PI boost, RT overload on the owner's CPU,
and affinity that excludes the CPU that just went idle. I do not know whether
Martin's livefx workers hit all three. The trace in his report has the worker
boosted and enqueued but never scheduled while other CPUs idle, and nothing in
his description rules it out, but I cannot confirm it from the data in the
report.

Where the non-PUSH path appears to lose it
------------------------------------------

Both sides have to miss the boosted task.

1. The pull side cannot reach it. In the non-PUSH branch, pull_rt_task() walks
   the overloaded CPUs and calls pick_highest_pushable_task(src_rq, this_cpu),
   which requires cpumask_test_cpu(cpu, &p->cpus_mask). L's mask excludes cpu2,
   so cpu2's loop finds nothing and returns empty-handed.

2. The push side was never queued. rt_queue_push_tasks() is only called from
   set_next_task_rt() and switched_to_rt(). prio_changed_rt() queues a pull when
   the running task's priority drops, but it does not queue a push. The boost
   arrives through rt_mutex_setprio() -> sched_change_end() ->
   prio_changed_rt(). L is not the current task on cpu1 (X is), so it takes the
   non-current branch, which only ends up doing resched_curr().

The result is that after the boost, cpu1 will not queue a push until X itself
schedules out, and cpu2 cannot pull L because of the affinity check.

The two paths are separable in the trace by the hardirq marker:

  # RT_PUSH_IPI on - the ipi runs the push in hardirq context on cpu1
  hog-122 [001] d.h2. 53.803264: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3

  # off - no ipi; the push happens on cpu1's ordinary scheduling path,
  # and in this run only 5 ms before X left on its own
  hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
  hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X

One more observation, in case it is useful: prio_changed_rt() does not appear to
have ever queued a push. I walked that function's history and did not find a
revision that did. That would be consistent with Steve's guess that this is a
long-standing bug that the PUSH path happened to cover, rather than something
dd29c017aed6 introduced.

What I am not proposing
-----------------------

I am not suggesting flipping the RT_PUSH_IPI default. The thundering-herd
reason for keeping it off outside PREEMPT_RT is clear, and trading a regression
for that is not a fix.

If this is too far from the real workload to be worth chasing, or if there is a
specific trace you would rather have, tell me and I will either drop it or
collect that instead. The probe and the full logs are small; happy to send them
to anyone who wants to rerun it.

Thanks,
Chaosheng

--
Chaosheng Qu

Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Steven Rostedt]

On Thu, 15 Oct 2026 14:19:15 +0800
Chaosheng Qu <chaosheng.qu.linux@gmail.com> wrote:

> Hi Steve, Martin,

Hi Chaosheng,

Thanks for the in-depth analysis!

> 
> I do not have Martin's workload, so this is not a reproduction of the reported
> audio dropouts. What I have is a narrower synthetic experiment that may help
> locate the non-PUSH failure Steve suspected. Sending data, not conclusions, and
> no patch yet.
> 
> The short version: with RT_PUSH_IPI off, a PI-boosted task that cannot run on
> its own CPU can be stuck in a state where the pull side cannot reach it and the
> push side was never queued, until the task holding the CPU schedules out on its
> own. Two things have to line up for that, and one of them (affinity) is in my
> first attempt to get this wrong, so I have included the counter-example below.
> 
> Setup
> -----
> 
> QEMU/KVM, 4 vCPU, non-PREEMPT_RT, 7.3-rc7 (24a19b953e2a). The feature is
> toggled at runtime via /sys/kernel/debug/sched/features. Four SCHED_FIFO
> threads:
> 
>   cpu1: L  FIFO 10  owner of PI mutex M, affinity {cpu1,cpu3}
>   cpu1: X  FIFO 99  hog, busy for the whole dwell
>   cpu2: W  FIFO 90  waiter, blocks on M     <- deliberately outside L's mask
>   cpu3: R  FIFO 60  spare, spinning
> 
> W blocking on M boosts L from 10 to 90; the trace confirms it:
> 
>   waiter-114 [002] d..3. 52.286120: sched_pi_setprio: comm=owner pid=115 oldprio=89 newprio=9

After reading the above, I can see the issue immediately.

> 
> L then cannot run on cpu1 because X holds it, and its only way out is to be
> moved to cpu3. cpu3's spare runs at FIFO 60, which matters: before the boost L
> has prio 89, and find_lock_lowest_rq() requires the target's
> highest_prio.curr to be strictly higher priority than the task being moved, so
> cpu3 is not a legal target until the boost happens. That way the opportunity is
> created by the boost itself and not by some unrelated migration.
> 
> Measurements, dwell 1500 ms
> ---------------------------
> 
>   RT_PUSH_IPI   n   resumed <100 ms   owner resume median   min / max
>   off (default) 8   0/8               1499.6 ms             1146.1 / 1502.6 ms
>   on            6   6/6                 0.8 ms                0.6 / 5.4 ms
>   off, no boost 2   0/2               1499.2 ms            1499.1 / 1499.2 ms
> 
> In the 8 "off" runs, 7 never migrated at all: L stayed on cpu1 and started only
> once X finished its dwell. The 8th moved to cpu3 5 ms before X exited, which is
> a by-product of the hog leaving rather than a rescue, so I used a fixed 100 ms
> threshold for "timely" rather than "eventually moved". With the dwell raised to
> 4000 ms the waiter was blocked for 2.26 s before L finally moved.
> 
> The third row is a control with PTHREAD_PRIO_NONE instead of PRIO_INHERIT, to
> check that the opportunity really comes from the boost. Without the boost L
> stays at prio 89, cpu3's curr is 39, 39 > 89 is false, and so there is no legal
> migration target at all. Both feature settings look the same there, which is
> the expected result.
> 
> The affinity counter-example
> ----------------------------
> 
> My first version gave L affinity {cpu1,cpu2}, that is, including the CPU whose
> priority drops when W blocks. In that shape both settings rescued L in 0.5-6.2
> ms and the two paths do not diverge at all. The reason is that cpu2 can pull L
> inline, and task_is_pushable() only requires
> cpumask_test_cpu(cpu, &p->cpus_mask) for the pulling CPU. The split only shows
> up once that CPU is removed from L's mask.
> 
> So this is a fairly narrow condition: PI boost, RT overload on the owner's CPU,
> and affinity that excludes the CPU that just went idle. I do not know whether
> Martin's livefx workers hit all three. The trace in his report has the worker
> boosted and enqueued but never scheduled while other CPUs idle, and nothing in
> his description rules it out, but I cannot confirm it from the data in the
> report.
> 
> Where the non-PUSH path appears to lose it
> ------------------------------------------
> 
> Both sides have to miss the boosted task.
> 
> 1. The pull side cannot reach it. In the non-PUSH branch, pull_rt_task() walks
>    the overloaded CPUs and calls pick_highest_pushable_task(src_rq, this_cpu),
>    which requires cpumask_test_cpu(cpu, &p->cpus_mask). L's mask excludes cpu2,
>    so cpu2's loop finds nothing and returns empty-handed.
> 
> 2. The push side was never queued. rt_queue_push_tasks() is only called from
>    set_next_task_rt() and switched_to_rt(). prio_changed_rt() queues a pull when
>    the running task's priority drops, but it does not queue a push. The boost
>    arrives through rt_mutex_setprio() -> sched_change_end() ->
>    prio_changed_rt(). L is not the current task on cpu1 (X is), so it takes the
>    non-current branch, which only ends up doing resched_curr().
> 
> The result is that after the boost, cpu1 will not queue a push until X itself
> schedules out, and cpu2 cannot pull L because of the affinity check.
> 
> The two paths are separable in the trace by the hardirq marker:
> 
>   # RT_PUSH_IPI on - the ipi runs the push in hardirq context on cpu1
>   hog-122 [001] d.h2. 53.803264: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
> 
>   # off - no ipi; the push happens on cpu1's ordinary scheduling path,
>   # and in this run only 5 ms before X left on its own
>   hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
>   hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X
> 
> One more observation, in case it is useful: prio_changed_rt() does not appear to
> have ever queued a push. I walked that function's history and did not find a
> revision that did. That would be consistent with Steve's guess that this is a
> long-standing bug that the PUSH path happened to cover, rather than something
> dd29c017aed6 introduced.

Exactly. The issue that you show here appears to be that the boosting of
the RT task should trigger another analysis of RT migration. Which we do
not do.

> 
> What I am not proposing
> -----------------------
> 
> I am not suggesting flipping the RT_PUSH_IPI default. The thundering-herd
> reason for keeping it off outside PREEMPT_RT is clear, and trading a regression
> for that is not a fix.
> 
> If this is too far from the real workload to be worth chasing, or if there is a
> specific trace you would rather have, tell me and I will either drop it or
> collect that instead. The probe and the full logs are small; happy to send them
> to anyone who wants to rerun it.

I think this is a legitimate bug and should be fix. We need to make the
prio_changed_rt() aware of the rto mask and trigger a push if needed.

-- Steve

[PATCH] sched/rt: queue a push when a queued RT task is boosted

[Chaosheng Qu]

Hi Steve, Peter, Martin,

Two things: a correction to my previous message, and a patch that follows
from your suggestion.

First, the correction. In my last mail I quoted this trace pair to argue
that the non-PUSH push only happens when the hog leaves:

  hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
  hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X

The second line is wrong. It does not appear in any of my logs; I
assembled it from a different run and mislabelled the timestamps. The
first line is real (aff-multi.log, the 4 s dwell run). It does not change
the conclusion - in the 1.5 s dwell runs the owner still resumes at a
median of 1499.6 ms and 0/8 meet a 100 ms bound - but the quoted trace was
not what I said it was, and I would rather correct that now than have it
found later.

Second, the instrumented follow-up you asked for. I put kprobes on
pull_rt_task(), push_rt_task(), push_rt_tasks(), prio_changed_rt(),
enqueue_pushable_task(), pick_next_pushable_task() and
find_lock_lowest_rq(), on the same 4-vCPU non-PREEMPT_RT config. The
picture is tighter than what I described before:

- pull is reached, and immediately. On the boost, waiter blocks, cpu2
  goes idle, and pull_rt_task(cpu2) runs between 5 and 99 us later. It
  comes back empty. The reason it comes back empty is the affinity test in
  task_is_pushable(): pick_highest_pushable_task(src_rq=cpu1,
  this_cpu=cpu2) requires cpu2 to be in the boosted task's cpus_mask. In
  my scenario the mask is {1,3}, so it is not. When I widen the mask to
  {1,2} the very same code pulls the owner to cpu2 within 2 us, so the
  affinity test is the whole difference.
- push is not queued at the boost. prio_changed_rt() runs, and nothing
  queues a push from it. I kept the full trace for three clean runs: in
  all three the first balancer event after the boost is pull_rt_task(cpu2)
  at +5..+99 us, there is no push_rt_task()/push_rt_tasks() for the whole
  1.5 s dwell, and the owner never migrates - it just waits the hog out.
  Being precise, OFF is not "never pushes": in other runs a push does
  appear on cpu1, but it is entered from the hog's own scheduling path
  (the trace attributes it to the hog thread), not from the boost, and it
  lands whenever that path next happens to run - which is why the few
  runs that do escape land anywhere between 62 ms and 1.2 s. Across all my
  OFF runs (n=17) only one met a 100 ms bound; the median is 1498.5 ms.

So the non-PUSH path loses the task because both sides miss it: pull
cannot select it (affinity), and push is not requested at the boost. That
matches what you wrote - the boost should trigger another analysis of RT
migration, and we do not do that.

The patch below does exactly that: when a queued, non-current RT task's
priority is raised, ask its rq for a push. The push selects a target from
the task's own cpus_mask via find_lowest_rq(), which is the selection the
pull side could not make. rt_mutex_setprio() already ends with
__balance_callbacks(), so the callback runs inside the same rq->lock
section, on the same path as the existing rt_queue_pull_task().

A/B on the same tree, same config, same probe, only this patch different:

  OFF, mask {1,3}       before: 1/17 timely, median 1498.5 ms
                        after: 13/13 timely, median 4.8 ms (0.6-59.6)
  no boost (control)    before: 0/2 migrated, median 1498.2 ms
                        after: 0/3 migrated, median 1499.2 ms
  ON (RT_PUSH_IPI)      before: 3/3, median 3.06 ms
                        after: 5/5, median 0.72 ms

The control row is the one I would look at: with no boost there is no
legal target (cpu3's spare at FIFO 60 is more urgent than the owner at
FIFO 10), so the new push finds nothing and I did not observe any
migration, same as before the patch. The patch only acts when a real
migration opportunity exists.

With the patch, the mechanism is visible in trace:

  waiter-109 [002] 51.667482: prio_changed_rt
  waiter-109 [002] 51.667695: push_rt_tasks
  waiter-109 [002] 51.667739: pick_next_pushable_task
  waiter-109 [002] 51.667955: find_lock_lowest_rq
  waiter-109 [002] 51.669068: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3

Caveats, since this is a first cut: measured only on a KASAN +
PROVE_LOCKING debug config, only non-PREEMPT_RT, only this synthetic
topology, not on Martin's REAPER workload. I did compile-test the UP
build (build_policy.o, CONFIG_SMP=n) and it builds; the PREEMPT_RT
combination I could not build here because this tree has no RT patch set,
so it is untested.
One scope note: prio_changed_rt() is reached from sched_change_end() for
ordinary sched_setscheduler()/sched_setattr() priority changes as well,
not just PI boosts, so this would also queue a push when a queued RT task
is raised by userspace. That case looks symmetric to me but I have not
A/B'd it; say the word if you would rather gate this on pi_top_task. I
have not flipped any default and I am not proposing to.

One more thing, in case it is useful, and it may be the real reason
nobody has touched this: the prio_changed_rt() push omission is not
recent. I walked the function through 15 revisions back to 2007 and none
of them queued a push, so this looks like it has been there since the
non-PUSH fallback was introduced, which is consistent with your guess
that the PUSH path was covering for it.

Happy to respin, narrow it, or drop it if the direction is wrong.

Thanks,
Chaosheng


A PI boost can leave a queued RT task stuck on a runqueue it cannot run
on. The pull side cannot help: pull_rt_task() on another CPU only picks
tasks whose affinity includes that CPU, so when the boosted task's
cpus_mask excludes the CPU that just went idle it is never selected. The
push side is never asked either: prio_changed_rt() only queues a pull.

In a four-vCPU non-PREEMPT_RT guest with RT_PUSH_IPI off, a FIFO-10 task
holding a PI mutex, boosted to FIFO 90 by a waiter on another CPU while a
FIFO-99 hog keeps its own CPU busy, stayed queued for the whole 1.5 s
dwell while a FIFO-60 spare sat idle on a CPU that was in its mask. With
RT_PUSH_IPI on the same setup resumed in under 6 ms.

Queue a push from prio_changed_rt() when a queued, non-current RT task has
its priority raised. The push selects its target from the task's own
cpus_mask, which is exactly the selection the pulling CPU cannot make.
RT_MUTEXES already arranges for __balance_callbacks() to run inside the
same rq->lock section, so this is the same delivery path as the existing
rt_queue_pull_task().

With no boost present there is no legal target and the push finds
nothing, so behaviour is unchanged.

Signed-off-by: Chaosheng Qu <chaosheng.qu.linux@gmail.com>
---
 kernel/sched/rt.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 85303add7..13d505470 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2499,6 +2499,17 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
 		 */
 		if (p->prio < rq->donor->prio)
 			resched_curr(rq);
+
+		/*
+		 * A boost can create a migration opportunity that the pull
+		 * side cannot reach: pull_rt_task() on another CPU only picks
+		 * tasks whose affinity includes that CPU, so a task that gets
+		 * boosted while its CPUs are busy can stay queued until it
+		 * becomes current. Queue a push on this rq instead, which
+		 * selects a target from the task's own affinity.
+		 */
+		if (p->prio < oldprio && rq->rt.overloaded)
+			rt_queue_push_tasks(rq);
 	}
 }
 
-- 
2.43.0

Re: [PATCH] sched/rt: queue a push when a queued RT task is boosted

[Kayra Cizmeci]

On Sat, Oct 17, 2026 at 08:14:23PM +0800, Chaosheng Qu wrote:
> Hi Steve, Peter, Martin,
> 
> Two things: a correction to my previous message, and a patch that follows
> from your suggestion.
> 
> First, the correction. In my last mail I quoted this trace pair to argue
> that the non-PUSH push only happens when the hog leaves:
> 
>   hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
>   hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X
> 
> The second line is wrong. It does not appear in any of my logs; I
> assembled it from a different run and mislabelled the timestamps. The
> first line is real (aff-multi.log, the 4 s dwell run). It does not change
> the conclusion - in the 1.5 s dwell runs the owner still resumes at a
> median of 1499.6 ms and 0/8 meet a 100 ms bound - but the quoted trace was
> not what I said it was, and I would rather correct that now than have it
> found later.
> 
> Second, the instrumented follow-up you asked for. I put kprobes on
> pull_rt_task(), push_rt_task(), push_rt_tasks(), prio_changed_rt(),
> enqueue_pushable_task(), pick_next_pushable_task() and
> find_lock_lowest_rq(), on the same 4-vCPU non-PREEMPT_RT config. The
> picture is tighter than what I described before:
> 
> - pull is reached, and immediately. On the boost, waiter blocks, cpu2
>   goes idle, and pull_rt_task(cpu2) runs between 5 and 99 us later. It
>   comes back empty. The reason it comes back empty is the affinity test in
>   task_is_pushable(): pick_highest_pushable_task(src_rq=cpu1,
>   this_cpu=cpu2) requires cpu2 to be in the boosted task's cpus_mask. In
>   my scenario the mask is {1,3}, so it is not. When I widen the mask to
>   {1,2} the very same code pulls the owner to cpu2 within 2 us, so the
>   affinity test is the whole difference.
> - push is not queued at the boost. prio_changed_rt() runs, and nothing
>   queues a push from it. I kept the full trace for three clean runs: in
>   all three the first balancer event after the boost is pull_rt_task(cpu2)
>   at +5..+99 us, there is no push_rt_task()/push_rt_tasks() for the whole
>   1.5 s dwell, and the owner never migrates - it just waits the hog out.
>   Being precise, OFF is not "never pushes": in other runs a push does
>   appear on cpu1, but it is entered from the hog's own scheduling path
>   (the trace attributes it to the hog thread), not from the boost, and it
>   lands whenever that path next happens to run - which is why the few
>   runs that do escape land anywhere between 62 ms and 1.2 s. Across all my
>   OFF runs (n=17) only one met a 100 ms bound; the median is 1498.5 ms.
> 
> So the non-PUSH path loses the task because both sides miss it: pull
> cannot select it (affinity), and push is not requested at the boost. That
> matches what you wrote - the boost should trigger another analysis of RT
> migration, and we do not do that.
> 
> The patch below does exactly that: when a queued, non-current RT task's
> priority is raised, ask its rq for a push. The push selects a target from
> the task's own cpus_mask via find_lowest_rq(), which is the selection the
> pull side could not make. rt_mutex_setprio() already ends with
> __balance_callbacks(), so the callback runs inside the same rq->lock
> section, on the same path as the existing rt_queue_pull_task().
> 
> A/B on the same tree, same config, same probe, only this patch different:
> 
>   OFF, mask {1,3}       before: 1/17 timely, median 1498.5 ms
>                         after: 13/13 timely, median 4.8 ms (0.6-59.6)
>   no boost (control)    before: 0/2 migrated, median 1498.2 ms
>                         after: 0/3 migrated, median 1499.2 ms
>   ON (RT_PUSH_IPI)      before: 3/3, median 3.06 ms
>                         after: 5/5, median 0.72 ms
> 
> The control row is the one I would look at: with no boost there is no
> legal target (cpu3's spare at FIFO 60 is more urgent than the owner at
> FIFO 10), so the new push finds nothing and I did not observe any
> migration, same as before the patch. The patch only acts when a real
> migration opportunity exists.
> 
> With the patch, the mechanism is visible in trace:
> 
>   waiter-109 [002] 51.667482: prio_changed_rt
>   waiter-109 [002] 51.667695: push_rt_tasks
>   waiter-109 [002] 51.667739: pick_next_pushable_task
>   waiter-109 [002] 51.667955: find_lock_lowest_rq
>   waiter-109 [002] 51.669068: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
> 
> Caveats, since this is a first cut: measured only on a KASAN +
> PROVE_LOCKING debug config, only non-PREEMPT_RT, only this synthetic
> topology, not on Martin's REAPER workload. I did compile-test the UP
> build (build_policy.o, CONFIG_SMP=n) and it builds; the PREEMPT_RT
> combination I could not build here because this tree has no RT patch set,
> so it is untested.
> One scope note: prio_changed_rt() is reached from sched_change_end() for
> ordinary sched_setscheduler()/sched_setattr() priority changes as well,
> not just PI boosts, so this would also queue a push when a queued RT task
> is raised by userspace. That case looks symmetric to me but I have not
> A/B'd it; say the word if you would rather gate this on pi_top_task. I
> have not flipped any default and I am not proposing to.
> 
> One more thing, in case it is useful, and it may be the real reason
> nobody has touched this: the prio_changed_rt() push omission is not
> recent. I walked the function through 15 revisions back to 2007 and none
> of them queued a push, so this looks like it has been there since the
> non-PUSH fallback was introduced, which is consistent with your guess
> that the PUSH path was covering for it.
> 
> Happy to respin, narrow it, or drop it if the direction is wrong.
> 
> Thanks,
> Chaosheng
> 
> 
> A PI boost can leave a queued RT task stuck on a runqueue it cannot run
> on. The pull side cannot help: pull_rt_task() on another CPU only picks
> tasks whose affinity includes that CPU, so when the boosted task's
> cpus_mask excludes the CPU that just went idle it is never selected. The
> push side is never asked either: prio_changed_rt() only queues a pull.
> 
> In a four-vCPU non-PREEMPT_RT guest with RT_PUSH_IPI off, a FIFO-10 task
> holding a PI mutex, boosted to FIFO 90 by a waiter on another CPU while a
> FIFO-99 hog keeps its own CPU busy, stayed queued for the whole 1.5 s
> dwell while a FIFO-60 spare sat idle on a CPU that was in its mask. With
> RT_PUSH_IPI on the same setup resumed in under 6 ms.
> 
> Queue a push from prio_changed_rt() when a queued, non-current RT task has
> its priority raised. The push selects its target from the task's own
> cpus_mask, which is exactly the selection the pulling CPU cannot make.
> RT_MUTEXES already arranges for __balance_callbacks() to run inside the
> same rq->lock section, so this is the same delivery path as the existing
> rt_queue_pull_task().
> 
> With no boost present there is no legal target and the push finds
> nothing, so behaviour is unchanged.
> 
> Signed-off-by: Chaosheng Qu <chaosheng.qu.linux@gmail.com>


IDK about the patch itself but,

This long long explanation should go below '---'. Not to the commit
message.

And isn't this Steven's suggestion? So I think we need a
Suggested-by.


Thanks,

Kayra

Re: [PATCH] sched/rt: queue a push when a queued RT task is boosted

[Chaosheng Qu]

Thanks. The long explanation is below the --- in the patch; the prose above
it is the short changelog. I added the Suggested-by and shortened the
changelog. The commit is now 04de902d6a4e8a6ce6471b97d73f869c5753dc61 on top
of 1d08a07fd492.

The patch is attached.

Thanks,
Chaosheng

[PATCH] sched/rt: queue a push when a queued RT task is boosted

[Chaosheng Qu]

A PI boost can leave a queued RT task stuck on a CPU where it cannot run.
The pull side only considers tasks whose affinity includes the CPU that
became idle, and prio_changed_rt() does not queue a push. If the boosted
task's mask excludes the idle CPU, neither path can move it.

In a four-vCPU non-PREEMPT_RT guest with RT_PUSH_IPI off, a FIFO-10 task
holding a PI mutex, boosted to FIFO 90 by a waiter on another CPU while
a FIFO-99 hog kept its CPU busy, stayed queued for the whole 1.5 s dwell.
A FIFO-60 spare was idle on a CPU in its mask. With RT_PUSH_IPI on, the
same setup resumed in under 6 ms.

Queue a push from prio_changed_rt() when a queued, non-current RT task
with more than one CPU has its priority raised. The push selects a target
from the task's cpus_mask, which is what the pulling CPU cannot do.
rt_mutex_setprio() calls __balance_callbacks() after prio_changed_rt(),
so the callback runs in the same rq->lock section as the existing
rt_queue_pull_task().

With no boost present the push finds no legal target.

Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Chaosheng Qu <chaosheng.qu.linux@gmail.com>
---
 kernel/sched/rt.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 85303add7..dc2775261 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2499,6 +2499,18 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
 		 */
 		if (p->prio < rq->donor->prio)
 			resched_curr(rq);
+
+		/*
+		 * A boost can create a migration opportunity that the pull
+		 * side cannot reach: pull_rt_task() on another CPU only picks
+		 * tasks whose affinity includes that CPU, so a task that gets
+		 * boosted while its CPUs are busy can stay queued until it
+		 * becomes current. Queue a push on this rq instead, which
+		 * selects a target from the task's own affinity.
+		 */
+		if (p->nr_cpus_allowed > 1 && p->prio < oldprio &&
+		    rq->rt.overloaded)
+			rt_queue_push_tasks(rq);
 	}
 }
 
-- 
2.43.0