Re: [RESEND RFC] sched_ext: Support RT task within sched_ext

[K Prateek Nayak <kprateek.nayak@amd.com>]

Hello Tengfei,

On 10/15/2026 6:17 PM, Tengfei Fan wrote:
> Sorry for resending this as plain text. The previous email was rejected
> by the sched-ext mailing list because it contained HTML.
> 
> Hi all,
> 
> We would like to discuss how RT tasks can best coordinate with sched_ext
> for cpu placement, cpu balance, and sched load reporting to schedutil
> governor. This RFC is intended to continue the discussion in email list
> from last LPC[1].
> 
> One of the options we would like to explore is allowing sched_ext to
> schedule some explicitly selected SCHED_FIFO/SCHED_RR tasks.
> 
> We received feedback from Changwoo that the issue is valid, and in
> LAVD sched-ext, they tried affining RT tasks to a limited set of CPUs as a
> workaround for the scheduler placement, but we do know that it is not a
> flexible enough solution.
> 
> We also had a discussion with Andrea Righi, that he acknowledged that
> he understands the scenario and current problem, and he would suggest
> not changing the current core scheduler, but instead using bpf ops to
> bring the RT tasks under the sched-ext control. We are trying to explore
> more on this path, and welcome to suggest or comment here.

This doing everything in sched-ext and eBPF attitude needs to change.
Sure it is good and all to ship quickly but you'll be left with a
sub optimal configuration forever and will always be workign around
that.

If you have some magic sauce to detect which scx threads are high
priority and matters for the KPI, why not just send a signal to a
userspace to demote the task temporarily to SCHED_OTHER and then
later move it back to RT when the situation allows?

You can always use a BPF Map to communicate the pids that need
promotion / demotion right? What is stopping you from doing this?
I thought that was the feedback from folks at LPC.

And if that is not possible, maybe you can convince the folks who
originally used the RT (most likely to get around some old issue)
that the custom scheduler is good enough to not hamper performance
and everything can now be a -20 nice task in sched-ext for
scheduler to know that it must be prioritized.

> LPC topic:
> [1] https://lpc.events/event/20/contributions/2561/
> 
> Slides:
> [2]
> https://lpc.events/event/20/contributions/2561/attachments/2025/4612/2026_LPC_Support_RT_tasks_within_sched-ext_.pdf
> 
> 
> Here is a text-based description of the main idea presented at LPC.
> If you already have that context, you can skip the section below.
> 
> 
> Background
> ----------
> 
> In most of the Android gaming, video playback, and livestreaming test
> scenarios, non-RT threads work together with RT threads to construct and
> consume a frame. Finishing the frame on time without frame jank is a
> key performance KPI. A balanced workload across CPUs with fewer
> unnecessary CPU frequency bumps is another key power KPI.
> 
> More specifically, taking a gaming scenario for example, UnityMain and
> UnityGfxDeviceW are the critical normal threads that have significantly
> higher load and usually run on the most performance-efficient cpus in
> gold/prime cluster. Other than those high load tasks, the other normal
> tasks mostly have similar workloads.
> 
> It has RT threads like the display HAL tasks, SurfaceFlinger, and the
> GPU related tasks. Those RT tasks have workloads comparable to those
> of other non-high-load normal tasks.
> 
> In our tests, sched_ext runs in switch-all mode, with
> SCX_OPS_SWITCH_PARTIAL unset. It schedules the non-RT tasks
> covered by this mode. SCHED_FIFO/SCHED_RR tasks still run in
> the native RT class.
> 
> Below, we call these non-RT tasks "SCX tasks".
> SCHED_DEADLINE tasks are not part of this proposal.
> 
> 
> Observed problems
> -----------------
> 
> cpu placement:
> 
> In our tests, RT tasks can run on CPUs used by important SCX tasks.
> This can cause the SCX tasks to be preempted, waiting longer to run,
> or lose cache locality.
> 
> Here is a diagram for easier understanding:
> 
>         RT task + important SCX task
>                     |
>                     v
>                  same CPU
>                     |
>                     v
>               CPU contention
>                     |
>         +-----------+-----------+
>         |           |           |
>    preemption   longer wait   cache impact
>         \           |           /
>          \          |          /
>           +---------+---------+
>                     |
>                     v
>        SCX work may finish later
> 
> RT tasks have higher priority, so the preemption itself is
> expected. However, from the whole scenario perspective, better CPU
> placement could reduce this interference when another suitable CPU
> is available.

Seems like a generic problem, no?

> Load balance:
> 
> RT tasks can be scheduled in the same cluster of non-high load scx
> tasks. This can also be a preferred placement decision. While the
> load balance within the cluster is also hard to maintain.
> 
> We sometimes see RT and SCX tasks on the same CPU while other CPUs
> can become idle very shortly. We need to check whether those newly
> idle CPUs can pull and run the scx tasks. While if RT tasks are also
> in the same scx scheduler, it is easier to make decisions from a load
> balance perspective.

This is also a problem with fair but fair scheduler seems to have
figured it out well - we have newidle balance to pull tasks and as
far as I know, SCX has ops.cpu_release() callaback that can do a
re-enqueue to a different.

Why is this not enough to move this high priority task away and
preempt something else on another CPU?

> sched util calculation:
> 
> We also see a difference in load tracking. Our implementation
> uses a frame-window model for SCX tasks, while the RT load signal
> in our setup uses PELT.
> 
> The SCX task load is flat and steady, while the RT task load
> sometimes spikes. An unnecessary CPU frequency bump up can be triggered
> by RT task load spikes.

Wasn't UCLAMP specifically designed to avoid those bumps if they
are unnecessary? RT can make sure you preempt first and UCLAMP
can ensure you don't boost frequency unnecessarily.

I think sched-util will request the highest frequency when an RT
task is running but that seems like a more generic problem to
solve.

> Why CPU affinity is not enough
> ------------------------------
> 
> We could use CPU affinity to keep RT and important SCX tasks
> on different CPUs. This could reduce the time they compete
> for the same CPU.
> 
> However, CPU affinity is fixed and strictly limits how tasks can move
> between CPUs. If the number of background tasks increases, CPUs perferred
> for critical SCX tasks may still have spare capacity. With fixed CPU
> affinity, non-critical SCX tasks cannot use those CPUs even when doing
> so would help balance the system load.
> 
> Ideally, the scheduler should be able to choose CPUs dynamically as
> the load changes, while still keeping tasks within their allowed CPU
> masks.
> 
> CPU affinity may still help, but it does not provide the
> flexibility we want here.

If you can consider cpu affinities an option, you can also consider
demoting them to SCHED_OTHER as an option no?

> Possible options
> ----------------
> 
> One option is to let sched_ext schedule some explicitly selected RT
> threads together with SCX tasks.

Who gets to decide this selection of threads?

> This could let one scheduler choose CPUs for both selected RT
> threads and SCX tasks.
> 
> For example, if other cpu is idle, or running a low priority scx task,
> or running a low load scx task, an RT task could use that CPU instead
> of a CPU running an important high load SCX task. This could reduce CPU
> contention for the frame important KPI and help balance RT and SCX tasks
> across CPUs.
> 
> Such placement should remain a preference rather than a fixed
> rule. It must respect the RT task's priority, timing needs, and
> allowed CPU mask. The other CPU must also have enough capacity.

I fail to see how "RT task's priority" is respected if you are
suggesting a SCHED_OTHER task can be prioritized over it.

Also how do you infer a timing needs of RT? From a traditional
theory perspective, if the task is of a higher priority, it
always preempts.

> It could also allow RT and SCX tasks to use the same utilization
> model. We would like to study whether this can make the load
> signal more stable and reduce late CPU frequency changes.
> 
> Another option is to keep all RT tasks in the native RT
> class, but provide enough information for the two schedulers
> to better coordinate CPU placement. However, this would also require
> placement and load synchronization between the RT scheduling class
> and the sched_ext class.

This can be a generic optimization too right? If RT task has a
choice to be scheduled on a bunch of CPU, maybe there are
heuristics that can be used using nice value, CPU load to better
place them.

> After the LPC discussion, we'd like to explore option 1 and use
> BPF-only hooks to achieve current needed RT tasks within scx class.

I don't like it one bit but what exactly are you planning to do
with this BPF op?

Direct RT task placement to another CPU? How is the RT scheduler
supposed to verify if that alternate placement doesn't violate the
RT guarantees? (And please don't say you'll use the ext's dispatch
path to do this) 
> One option is to have all RT tasks with a priority lower than the SCX
>  helper thread go into sched_ext.
> 
> Migration thread, sched-ext enable/disable threads and IRQ threads need
> to remain outside sched_ext. With this option, we will need to verify
> more common scenarios. We can explore the first option first.
> 
> 
> Open questions
> --------------
> 
> * Can sched deadline can used to optimize those scenarios?
> 
>   It is possible, of course, so we put it here, welcome to share the
>   possible configurations of which tasks goes to deadline. However, here
>   we discuss only the topic of plugging RT tasks into and out of
>   sched_ext for these scenarios. For different scenarios optimization,
>   there is of course no single solution. It still can have different
>   scheduler optimization to make a specific tasks a different sched-class.
> 
> * Other ideas?

You can always demote these tasks from the userspace bit of
sched-ext and there is no explanation in this entire write-up
that says why that is bad / insufficient.
> We would appreciate feedback on the problem, the possible
> directions, and what we should try first.
-- 
Thanks and Regards,
Prateek