Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Chaosheng Qu <chaosheng.qu.linux@gmail.com>]

Hi Steve, Martin,

I do not have Martin's workload, so this is not a reproduction of the reported
audio dropouts. What I have is a narrower synthetic experiment that may help
locate the non-PUSH failure Steve suspected. Sending data, not conclusions, and
no patch yet.

The short version: with RT_PUSH_IPI off, a PI-boosted task that cannot run on
its own CPU can be stuck in a state where the pull side cannot reach it and the
push side was never queued, until the task holding the CPU schedules out on its
own. Two things have to line up for that, and one of them (affinity) is in my
first attempt to get this wrong, so I have included the counter-example below.

Setup
-----

QEMU/KVM, 4 vCPU, non-PREEMPT_RT, 7.3-rc7 (24a19b953e2a). The feature is
toggled at runtime via /sys/kernel/debug/sched/features. Four SCHED_FIFO
threads:

  cpu1: L  FIFO 10  owner of PI mutex M, affinity {cpu1,cpu3}
  cpu1: X  FIFO 99  hog, busy for the whole dwell
  cpu2: W  FIFO 90  waiter, blocks on M     <- deliberately outside L's mask
  cpu3: R  FIFO 60  spare, spinning

W blocking on M boosts L from 10 to 90; the trace confirms it:

  waiter-114 [002] d..3. 52.286120: sched_pi_setprio: comm=owner pid=115 oldprio=89 newprio=9

L then cannot run on cpu1 because X holds it, and its only way out is to be
moved to cpu3. cpu3's spare runs at FIFO 60, which matters: before the boost L
has prio 89, and find_lock_lowest_rq() requires the target's
highest_prio.curr to be strictly higher priority than the task being moved, so
cpu3 is not a legal target until the boost happens. That way the opportunity is
created by the boost itself and not by some unrelated migration.

Measurements, dwell 1500 ms
---------------------------

  RT_PUSH_IPI   n   resumed <100 ms   owner resume median   min / max
  off (default) 8   0/8               1499.6 ms             1146.1 / 1502.6 ms
  on            6   6/6                 0.8 ms                0.6 / 5.4 ms
  off, no boost 2   0/2               1499.2 ms            1499.1 / 1499.2 ms

In the 8 "off" runs, 7 never migrated at all: L stayed on cpu1 and started only
once X finished its dwell. The 8th moved to cpu3 5 ms before X exited, which is
a by-product of the hog leaving rather than a rescue, so I used a fixed 100 ms
threshold for "timely" rather than "eventually moved". With the dwell raised to
4000 ms the waiter was blocked for 2.26 s before L finally moved.

The third row is a control with PTHREAD_PRIO_NONE instead of PRIO_INHERIT, to
check that the opportunity really comes from the boost. Without the boost L
stays at prio 89, cpu3's curr is 39, 39 > 89 is false, and so there is no legal
migration target at all. Both feature settings look the same there, which is
the expected result.

The affinity counter-example
----------------------------

My first version gave L affinity {cpu1,cpu2}, that is, including the CPU whose
priority drops when W blocks. In that shape both settings rescued L in 0.5-6.2
ms and the two paths do not diverge at all. The reason is that cpu2 can pull L
inline, and task_is_pushable() only requires
cpumask_test_cpu(cpu, &p->cpus_mask) for the pulling CPU. The split only shows
up once that CPU is removed from L's mask.

So this is a fairly narrow condition: PI boost, RT overload on the owner's CPU,
and affinity that excludes the CPU that just went idle. I do not know whether
Martin's livefx workers hit all three. The trace in his report has the worker
boosted and enqueued but never scheduled while other CPUs idle, and nothing in
his description rules it out, but I cannot confirm it from the data in the
report.

Where the non-PUSH path appears to lose it
------------------------------------------

Both sides have to miss the boosted task.

1. The pull side cannot reach it. In the non-PUSH branch, pull_rt_task() walks
   the overloaded CPUs and calls pick_highest_pushable_task(src_rq, this_cpu),
   which requires cpumask_test_cpu(cpu, &p->cpus_mask). L's mask excludes cpu2,
   so cpu2's loop finds nothing and returns empty-handed.

2. The push side was never queued. rt_queue_push_tasks() is only called from
   set_next_task_rt() and switched_to_rt(). prio_changed_rt() queues a pull when
   the running task's priority drops, but it does not queue a push. The boost
   arrives through rt_mutex_setprio() -> sched_change_end() ->
   prio_changed_rt(). L is not the current task on cpu1 (X is), so it takes the
   non-current branch, which only ends up doing resched_curr().

The result is that after the boost, cpu1 will not queue a push until X itself
schedules out, and cpu2 cannot pull L because of the affinity check.

The two paths are separable in the trace by the hardirq marker:

  # RT_PUSH_IPI on - the ipi runs the push in hardirq context on cpu1
  hog-122 [001] d.h2. 53.803264: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3

  # off - no ipi; the push happens on cpu1's ordinary scheduling path,
  # and in this run only 5 ms before X left on its own
  hog-113 [001] d..3. 50.615178: sched_migrate_task: comm=owner prio=9 orig_cpu=1 dest_cpu=3
  hog-131 [001] d..2. 70.614471: sched_switch: prev_comm=hog prev_state=X

One more observation, in case it is useful: prio_changed_rt() does not appear to
have ever queued a push. I walked that function's history and did not find a
revision that did. That would be consistent with Steve's guess that this is a
long-standing bug that the PUSH path happened to cover, rather than something
dd29c017aed6 introduced.

What I am not proposing
-----------------------

I am not suggesting flipping the RT_PUSH_IPI default. The thundering-herd
reason for keeping it off outside PREEMPT_RT is clear, and trading a regression
for that is not a fix.

If this is too far from the real workload to be worth chasing, or if there is a
specific trace you would rather have, tell me and I will either drop it or
collect that instead. The probe and the full logs are small; happy to send them
to anyone who wants to rerun it.

Thanks,
Chaosheng

--
Chaosheng Qu