Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2026-10-26 09:05:42 [+0100], Juri Lelli wrote:
> Hi Sebastian,
Hi Juri,

> On 23/10/26 18:13, Sebastian Andrzej Siewior wrote:
> > Hi,
> > 
> > this is about the old RT throttling mechanism, the DL/fair server around
> > the RT tasks, the /proc/sys/kernel/sched_rt_* mechanism to control the
> > former and CONFIG_RT_GROUP_SCHED which never worked as it supposed to.
> > The current status looks like a mess and I'm not sure if there is a bug
> > somewhere leading to my headaches or if this is just not meant to be
> > working together. If so out with it.
> > 
> > tl;dr /proc/…/sched_rt_*, sched_rt_runtime_exceeded() probably
> > CONFIG_RT_GROUP_SCHED should be removed. A bit of docs should be added
> > for the fair server (or the right place).
> 
> Thanks for putting this recap together!
> 
> > Now. Details.
> 
> ...
> 
> > Without enabled RT_GROUP_SCHED the files can be set. Are they useful?
> > According to Documentation/scheduler/sched-deadline.rst it used for
> > "admission control" whatever that means. Anyway the fair server still
> > kicks if it is disabled. So I *think* this is just something else and
> > not part for the docs there.
> 
> Right, confusing, but for a root_domain comprising M CPUs, -deadline tasks can
> be created while the sum of their bandwidths stays below
> 
>   M * (sched_rt_runtime_us / sched_rt_period_us)

My understanding is that this is the runtime available to a DL task and
the length of a DL period (after the accounted runtime is reset). The
left over runtime (of runtime is less than period) is either buffer
(idle) or SCHED_OTHER runtime.
This defines how much runtime a DL task gets from the system, correct?

> and writing -1 disables the check altogether. That's the "admission control"
> the docs hand-wave about, and it's also why the knobs can't simply be deleted.
> As you noticed already, the servers are completely independent of these knobs:
> writing -1 does not disable the fair server, it only lifts the DL admission
> cap. In principle we could move this to a different interface that reads closer
> to the underlying purpose, but I'm not sure how much breakage it could cause.

That is my point. The rt_ runtime/ period is about DL properties and not
RT as in SCHED_FIFO/ RR as it once was. Both are set to one second as of
today and can not be changed with enabled CONFIG_RT_GROUP_SCHED.

Could we please move it to /sys/kernel/sched/ and rename it as dl
something? Is there any reason to keep as-is? There is
	Documentation/scheduler/sched-deadline.rst
	Documentation/arch/arm64/asymmetric-32bit.rst

referring to it in DL context. There is "ftracetest" writing
unconditionally -1 as a workaround, see commit 57c4cfd4a2eef
("ftrace/selftests: workaround cgroup RT scheduling issues").

If the runtime/ period arguments are fine then we could keep
it and make them only apply for the DL period.

Two files have been repurposed and if your intention as to disable the
RT throttler then you will be surprised.

> ...
> 
> > Anyway. Could this interface be moved to /sys/kernel/sched/fair_server
> > away from the debugfs part? debugfs is not for stable kernels/ ABI. Is
> > there a reason why this might be considered debugging and not final or
> > restricted from productive?
> 
> I think this was raised already in the past, but at the time maybe it
> was still to early to say if dl-servers will stick around. They now have
> got more users (sched-ext), so it might be a time we discuss a stable
> (non-debug) interface for it. Your suggestion doesn't look bad to me.

Thank you.

> > Just to summarize:
> > - Remove /proc/sys/kernel/sched_rt_*
> >   [ ] Yes [ ] No, because
> >               _____________________
> 
> Yes and no :). We need an interface to tweak admission control, but we
> could find a better/different one.

Okay.

> > - Remove CONFIG_RT_GROUP_SCHED
> >   [ ] Yes [ ] No, because
> >               _____________________
> 
> Luca already commented on this. I believe the plan is still to come up
> with a DEADLINE backed RT_GROUP_SCHED for Cgroups v2. If cleaner, an
> approach could be to first merge the removal of RT_GROUP_SCHED and then
> reimplement the whole lot as Yuri and Luca are proposing to do. The
> series is already somewhat doing that, but it could be made more
> explicit/clean I guess.

okay, then let me prepare a removal of this then.

> > - An additional trace sched event for fair server preemption
> >   [ ] Yes   [ ] No, it is silly to have
> 
> With 820725b0eb59f ("sched: Add deadline tracepoints") we already have
> some of that: sched_dl_{throttle,replenish,update,server_start,server_stop},
> which fire for fair/ext servers too (there's a type field, DL_SERVER_FAIR
> and friends). They of course require something like this [1] to be
> activated. We could add more if they are still not enough to get a clear
> picture about what's going on.

Hmm. You may want to merge Qais's current master to get it compiled
again ;) But then nothing hooks it up, so I just replaced with with
trace_printk() just to see what happens and got:

|       loop-time-1612    [000] d.h1.   154.121809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3999981 [ns]
|       loop-time-1612    [000] d.h1.   154.123812: sched_stat_runtime: comm=loop-time pid=1612 runtime=2003097 [ns]
|       loop-time-1612    [000] d.h1.   154.123812: replenish_dl_entity: trace_sched_dl_replenish_tp
^
|       loop-time-1612    [000] dN.2.   154.123814: sched_stat_runtime: comm=loop-time pid=1612 runtime=1783 [ns]
|       loop-time-1612    [000] d..2.   154.123815: sched_switch: prev_comm=loop-time prev_pid=1612 prev_prio=9 prev_state=R ==> next_comm=ksoftirqd/0 next_pid=14 next_prio=120
|     ksoftirqd/0-14      [000] d..2.   154.123817: sched_stat_runtime: comm=ksoftirqd/0 pid=14 runtime=3036 [ns]
|     ksoftirqd/0-14      [000] d..2.   154.123817: update_curr_dl_se: trace_sched_dl_update_tp
|     ksoftirqd/0-14      [000] d..2.   154.123818: dl_server_stop.part.0: trace_sched_dl_server_stop_tp
|     ksoftirqd/0-14      [000] d..2.   154.123819: sched_switch: prev_comm=ksoftirqd/0 prev_pid=14 prev_prio=120 prev_state=S ==> next_comm=loop-time next_pid=1612 next_prio=9
|       loop-time-1612    [000] d.h1.   154.125809: sched_stat_runtime: comm=loop-time pid=1612 runtime=1992016 [ns]
|       loop-time-1612    [000] d.h1.   154.129809: sched_stat_runtime: comm=loop-time pid=1612 runtime=4000522 [ns]
|       loop-time-1612    [000] d.h1.   154.133809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3999631 [ns]
|       loop-time-1612    [000] d.h2.   154.133810: sched_waking: comm=ksoftirqd/0 pid=14 prio=120 target_cpu=000
|       loop-time-1612    [000] d.h3.   154.133811: sched_stat_runtime: comm=loop-time pid=1612 runtime=1463 [ns]
|       loop-time-1612    [000] d.h3.   154.133811: dl_server_start: trace_sched_dl_server_start_tp
|       loop-time-1612    [000] dNh3.   154.133811: sched_wakeup: comm=ksoftirqd/0 pid=14 prio=120 target_cpu=000
|       loop-time-1612    [000] d..2.   154.133812: sched_switch: prev_comm=loop-time prev_pid=1612 prev_prio=9 prev_state=R ==> next_comm=ksoftirqd/0 next_pid=14 next_prio=120
^
|     ksoftirqd/0-14      [000] d..2.   154.133812: sched_stat_runtime: comm=ksoftirqd/0 pid=14 runtime=2054 [ns]
|     ksoftirqd/0-14      [000] d..2.   154.133812: update_curr_dl_se: trace_sched_dl_update_tp
|     ksoftirqd/0-14      [000] d..2.   154.133813: dl_server_stop.part.0: trace_sched_dl_server_stop_tp
|     ksoftirqd/0-14      [000] d..2.   154.133814: sched_switch: prev_comm=ksoftirqd/0 prev_pid=14 prev_prio=120 prev_state=S ==> next_comm=loop-time next_pid=1612 next_prio=9
|       loop-time-1612    [000] d.h1.   154.137809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3996495 [ns]
|       loop-time-1612    [000] d.h2.   154.137810: sched_waking: comm=ksoftirqd/0 pid=14 prio=120 target_cpu=000

Comparing the first and the second preemption point, it is different.
The first looks like a direct consequence of the fair_server and second
probably because of the left over budget. This reflects the
implementation and I understand if you don't want to expose this
directly as-is. There is also this where the N flag just vanishes:
|       loop-time-1612    [000] d.h1.   159.029809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3997426 [ns]
|       loop-time-1612    [000] d.h2.   159.029811: sched_waking: comm=ksoftirqd/0 pid=14 prio=120 target_cpu=000
|       loop-time-1612    [000] d.h3.   159.029811: sched_stat_runtime: comm=loop-time pid=1612 runtime=1543 [ns]
|       loop-time-1612    [000] d.h3.   159.029811: dl_server_start: trace_sched_dl_server_start_tp
|       loop-time-1612    [000] d.h3.   159.029811: enqueue_dl_entity: trace_sched_dl_replenish_tp
|       loop-time-1612    [000] dNh3.   159.029811: sched_wakeup: comm=ksoftirqd/0 pid=14 prio=120 target_cpu=000
|       loop-time-1612    [000] d.h1.   159.033809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3998449 [ns]
|       loop-time-1612    [000] d.h1.   159.037809: sched_stat_runtime: comm=loop-time pid=1612 runtime=4000072 [ns]
|       loop-time-1612    [000] d.h1.   159.041809: sched_stat_runtime: comm=loop-time pid=1612 runtime=3999901 [ns]

I assume the bandwidth was replenished and then loop-time could remain
on the CPU. It would probably help if I would weld have dump some DL
details along…
I'm not sure if it would be better to add some DL details along or
extend sched_switch with a scheduling reason.

There are also no trace events for DL tasks if they exceed their
requested runtime, right? I *think* there is a signal (or can be) sent
if the DL task runs for too long. 
The sched_pelt* events didn't contain any bandwidth information as far
as I can tell (you just pointed me there as an example how those trace
points could be utilized I suppose).

> > - Fair server documentation needs to go to
> >   [ ] It is fine where it is
> >   [ ] To ____________________
> 
> Not fine where it is: the servers (fair & ext) work independently of
> RT_GROUP_SCHED and deserve their own description (defaults 50ms/1s per
> CPU, what they guarantee for fair tasks under RT pressure, how to tune
> or disable them). Either a new section in sched-deadline.rst (they are
> DL servers after all) or a small standalone file works for me.

Okay. Let me look into this.

> Thanks!
> Juri
> 
> 1 - https://github.com/jlelli/sched_tp/commits/deadline-tp/
>     from Qais' https://github.com/qais-yousef/sched_tp
> 

Sebastian