Re: [RESEND RFC] sched_ext: Support RT task within sched_ext

[Tengfei Fan <tengfei.fan@oss.qualcomm.com>]

On 10/16/2026 11:42 AM, K Prateek Nayak wrote:
> Hello Tengfei,
>
> On 10/15/2026 6:17 PM, Tengfei Fan wrote:
>> Sorry for resending this as plain text. The previous email was rejected
>> by the sched-ext mailing list because it contained HTML.
>>
>> Hi all,
>>
>> We would like to discuss how RT tasks can best coordinate with sched_ext
>> for cpu placement, cpu balance, and sched load reporting to schedutil
>> governor. This RFC is intended to continue the discussion in email list
>> from last LPC[1].
>>
>> One of the options we would like to explore is allowing sched_ext to
>> schedule some explicitly selected SCHED_FIFO/SCHED_RR tasks.
>>
>> We received feedback from Changwoo that the issue is valid, and in
>> LAVD sched-ext, they tried affining RT tasks to a limited set of CPUs as a
>> workaround for the scheduler placement, but we do know that it is not a
>> flexible enough solution.
>>
>> We also had a discussion with Andrea Righi, that he acknowledged that
>> he understands the scenario and current problem, and he would suggest
>> not changing the current core scheduler, but instead using bpf ops to
>> bring the RT tasks under the sched-ext control. We are trying to explore
>> more on this path, and welcome to suggest or comment here.
> This doing everything in sched-ext and eBPF attitude needs to change.
> Sure it is good and all to ship quickly but you'll be left with a
> sub optimal configuration forever and will always be workign around
> that.
>
> If you have some magic sauce to detect which scx threads are high
> priority and matters for the KPI, why not just send a signal to a
> userspace to demote the task temporarily to SCHED_OTHER and then
> later move it back to RT when the situation allows?
>
> You can always use a BPF Map to communicate the pids that need
> promotion / demotion right? What is stopping you from doing this?
> I thought that was the feedback from folks at LPC.
>
> And if that is not possible, maybe you can convince the folks who
> originally used the RT (most likely to get around some old issue)
> that the custom scheduler is good enough to not hamper performance
> and everything can now be a -20 nice task in sched-ext for
> scheduler to know that it must be prioritized.


We are using sched_ext for the current experiments because it gives us a 
convenient way to test different ideas. If an idea works well, we would 
be happy to bring the results back to the community and discuss whether 
a more general solution makes sense.

Simply demoting these tasks to SCHED_OTHER does not fully capture what 
we need, since we still want these selected RT application threads to 
retain their RT priority information and behavior. For example, when 
there is a choice of CPUs, we would prefer them to preempt a less 
important SCX task rather than an important one, while still allowing 
them to preempt an important SCX task when there is no better choice.

Following Andrea's suggestion, we will first explore preserving the 
requested RT policy/priority of these selected application threads as 
metadata, making them non-RT, and using this metadata for scheduling 
decisions within sched_ext. We will evaluate whether this can provide 
the behavior we need, while keeping threads that require native RT 
behavior in the RT class.


>
>> LPC topic:
>> [1] https://lpc.events/event/20/contributions/2561/
>>
>> Slides:
>> [2]
>> https://lpc.events/event/20/contributions/2561/attachments/2025/4612/2026_LPC_Support_RT_tasks_within_sched-ext_.pdf
>>
>>
>> Here is a text-based description of the main idea presented at LPC.
>> If you already have that context, you can skip the section below.
>>
>>
>> Background
>> ----------
>>
>> In most of the Android gaming, video playback, and livestreaming test
>> scenarios, non-RT threads work together with RT threads to construct and
>> consume a frame. Finishing the frame on time without frame jank is a
>> key performance KPI. A balanced workload across CPUs with fewer
>> unnecessary CPU frequency bumps is another key power KPI.
>>
>> More specifically, taking a gaming scenario for example, UnityMain and
>> UnityGfxDeviceW are the critical normal threads that have significantly
>> higher load and usually run on the most performance-efficient cpus in
>> gold/prime cluster. Other than those high load tasks, the other normal
>> tasks mostly have similar workloads.
>>
>> It has RT threads like the display HAL tasks, SurfaceFlinger, and the
>> GPU related tasks. Those RT tasks have workloads comparable to those
>> of other non-high-load normal tasks.
>>
>> In our tests, sched_ext runs in switch-all mode, with
>> SCX_OPS_SWITCH_PARTIAL unset. It schedules the non-RT tasks
>> covered by this mode. SCHED_FIFO/SCHED_RR tasks still run in
>> the native RT class.
>>
>> Below, we call these non-RT tasks "SCX tasks".
>> SCHED_DEADLINE tasks are not part of this proposal.
>>
>>
>> Observed problems
>> -----------------
>>
>> cpu placement:
>>
>> In our tests, RT tasks can run on CPUs used by important SCX tasks.
>> This can cause the SCX tasks to be preempted, waiting longer to run,
>> or lose cache locality.
>>
>> Here is a diagram for easier understanding:
>>
>>          RT task + important SCX task
>>                      |
>>                      v
>>                   same CPU
>>                      |
>>                      v
>>                CPU contention
>>                      |
>>          +-----------+-----------+
>>          |           |           |
>>     preemption   longer wait   cache impact
>>          \           |           /
>>           \          |          /
>>            +---------+---------+
>>                      |
>>                      v
>>         SCX work may finish later
>>
>> RT tasks have higher priority, so the preemption itself is
>> expected. However, from the whole scenario perspective, better CPU
>> placement could reduce this interference when another suitable CPU
>> is available.
> Seems like a generic problem, no?


This is a generic placement problem. We are currently exploring it in 
the sched_ext context, as sched_ext provides a convenient way to 
experiment with different ideas. If we learn something useful from these 
experiments, we would be happy to share the results with the community 
and further discuss how this could be developed into a more general 
solution.


>
>> Load balance:
>>
>> RT tasks can be scheduled in the same cluster of non-high load scx
>> tasks. This can also be a preferred placement decision. While the
>> load balance within the cluster is also hard to maintain.
>>
>> We sometimes see RT and SCX tasks on the same CPU while other CPUs
>> can become idle very shortly. We need to check whether those newly
>> idle CPUs can pull and run the scx tasks. While if RT tasks are also
>> in the same scx scheduler, it is easier to make decisions from a load
>> balance perspective.
> This is also a problem with fair but fair scheduler seems to have
> figured it out well - we have newidle balance to pull tasks and as
> far as I know, SCX has ops.cpu_release() callaback that can do a
> re-enqueue to a different.
>
> Why is this not enough to move this high priority task away and
> preempt something else on another CPU?


For newidle balance, pulling a preempted SCX task to another CPU may add 
waiting and migration overhead and hurt cache locality. We would prefer 
to avoid this when better initial placement is possible.

We have not looked closely at ops.cpu_release() yet. We need to evaluate 
whether it is sufficient for this case.


>
>> sched util calculation:
>>
>> We also see a difference in load tracking. Our implementation
>> uses a frame-window model for SCX tasks, while the RT load signal
>> in our setup uses PELT.
>>
>> The SCX task load is flat and steady, while the RT task load
>> sometimes spikes. An unnecessary CPU frequency bump up can be triggered
>> by RT task load spikes.
> Wasn't UCLAMP specifically designed to avoid those bumps if they
> are unnecessary? RT can make sure you preempt first and UCLAMP
> can ensure you don't boost frequency unnecessarily.
>
> I think sched-util will request the highest frequency when an RT
> task is running but that seems like a more generic problem to
> solve.


In our experiments, we observed cases where the SCX workload utilization 
was already close to a frequency transition point, and some additional 
RT load could trigger a frequency bump.

We have not yet verified whether an appropriate uclamp configuration can 
avoid this unnecessary bump while still providing the performance needed 
by the RT task.

The RT/schedutil behavior itself is a generic issue. We are currently 
exploring it in the sched_ext context, as sched_ext provides a 
convenient way to experiment with different ideas. If we learn something 
useful from these experiments, we would be happy to share the results 
with the community and further discuss how this could be developed into 
a more general solution.


>
>> Why CPU affinity is not enough
>> ------------------------------
>>
>> We could use CPU affinity to keep RT and important SCX tasks
>> on different CPUs. This could reduce the time they compete
>> for the same CPU.
>>
>> However, CPU affinity is fixed and strictly limits how tasks can move
>> between CPUs. If the number of background tasks increases, CPUs perferred
>> for critical SCX tasks may still have spare capacity. With fixed CPU
>> affinity, non-critical SCX tasks cannot use those CPUs even when doing
>> so would help balance the system load.
>>
>> Ideally, the scheduler should be able to choose CPUs dynamically as
>> the load changes, while still keeping tasks within their allowed CPU
>> masks.
>>
>> CPU affinity may still help, but it does not provide the
>> flexibility we want here.
> If you can consider cpu affinities an option, you can also consider
> demoting them to SCHED_OTHER as an option no?


CPU affinity was not intended as the solution here.

Simply demoting these tasks to SCHED_OTHER would not fully capture what 
we need, since we still want to retain their requested RT 
policy/priority as information for scheduling decisions. Following 
Andrea's suggestion, we will explore keeping these attributes as 
metadata, making the selected application threads non-RT, and using the 
metadata within sched_ext to provide the placement behavior we need.


>
>> Possible options
>> ----------------
>>
>> One option is to let sched_ext schedule some explicitly selected RT
>> threads together with SCX tasks.
> Who gets to decide this selection of threads?


We don't have a final selection mechanism yet. For the initial 
evaluation of Andrea's suggestion, the selected application threads will 
be explicitly specified by userspace through a simple configuration, 
e.g. by PID. This is only intended for validating the approach; we have 
not settled on how the selection should be made in the final design.


>
>> This could let one scheduler choose CPUs for both selected RT
>> threads and SCX tasks.
>>
>> For example, if other cpu is idle, or running a low priority scx task,
>> or running a low load scx task, an RT task could use that CPU instead
>> of a CPU running an important high load SCX task. This could reduce CPU
>> contention for the frame important KPI and help balance RT and SCX tasks
>> across CPUs.
>>
>> Such placement should remain a preference rather than a fixed
>> rule. It must respect the RT task's priority, timing needs, and
>> allowed CPU mask. The other CPU must also have enough capacity.
> I fail to see how "RT task's priority" is respected if you are
> suggesting a SCHED_OTHER task can be prioritized over it.
>
> Also how do you infer a timing needs of RT? From a traditional
> theory perspective, if the task is of a higher priority, it
> always preempts.


We are not suggesting that a SCHED_OTHER task should have higher 
priority than an RT task. What we are describing is a placement 
preference when selecting a CPU for the RT task. For example, when 
multiple suitable CPUs are available, we would prefer the RT task to 
preempt a less important SCX task rather than an important one. If there 
is no better choice, however, the RT task should still be able to 
preempt the important SCX task.

The term "timing needs" was not well defined here. For 
SCHED_FIFO/SCHED_RR tasks, we do not have a separate timing requirement 
beyond the scheduling policy/priority, so this wording was misleading.


>
>> It could also allow RT and SCX tasks to use the same utilization
>> model. We would like to study whether this can make the load
>> signal more stable and reduce late CPU frequency changes.
>>
>> Another option is to keep all RT tasks in the native RT
>> class, but provide enough information for the two schedulers
>> to better coordinate CPU placement. However, this would also require
>> placement and load synchronization between the RT scheduling class
>> and the sched_ext class.
> This can be a generic optimization too right? If RT task has a
> choice to be scheduled on a bunch of CPU, maybe there are
> heuristics that can be used using nice value, CPU load to better
> place them.


This can also be a generic placement optimization. As mentioned above, 
we are currently exploring it in the sched_ext context, as sched_ext 
provides a convenient way to experiment with different ideas. If we 
learn something useful from these experiments, we would be happy to 
share the results with the community and further discuss how this could 
be developed into a more general solution.


>
>> After the LPC discussion, we'd like to explore option 1 and use
>> BPF-only hooks to achieve current needed RT tasks within scx class.
> I don't like it one bit but what exactly are you planning to do
> with this BPF op?
>
> Direct RT task placement to another CPU? How is the RT scheduler
> supposed to verify if that alternate placement doesn't violate the
> RT guarantees? (And please don't say you'll use the ext's dispatch
> path to do this)


The suggestion at LPC came from Andrea. We don't consider to directly 
place native RT tasks on another CPU through the sched_ext dispatch 
path. As discussed above, we will instead explore preserving the 
requested RT policy/priority of selected application threads as BPF 
metadata, making those threads non-RT, and using the metadata for 
scheduling decisions within sched_ext.


>> One option is to have all RT tasks with a priority lower than the SCX
>>   helper thread go into sched_ext.
>>
>> Migration thread, sched-ext enable/disable threads and IRQ threads need
>> to remain outside sched_ext. With this option, we will need to verify
>> more common scenarios. We can explore the first option first.
>>
>>
>> Open questions
>> --------------
>>
>> * Can sched deadline can used to optimize those scenarios?
>>
>>    It is possible, of course, so we put it here, welcome to share the
>>    possible configurations of which tasks goes to deadline. However, here
>>    we discuss only the topic of plugging RT tasks into and out of
>>    sched_ext for these scenarios. For different scenarios optimization,
>>    there is of course no single solution. It still can have different
>>    scheduler optimization to make a specific tasks a different sched-class.
>>
>> * Other ideas?
> You can always demote these tasks from the userspace bit of
> sched-ext and there is no explanation in this entire write-up
> that says why that is bad / insufficient.


As mentioned above, simply demoting these tasks to SCHED_OTHER does not 
fully capture what we need. We still want to retain the RT priority 
information of these selected application threads and use it for 
scheduling decisions. For example, when multiple suitable CPUs are 
available, we would prefer these tasks to preempt a less important SCX 
task rather than an important one. If there is no better choice, 
however, they should still be able to preempt an important SCX task.


>> We would appreciate feedback on the problem, the possible
>> directions, and what we should try first.