[RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Sebastian Andrzej Siewior]

Hi,

this is about the old RT throttling mechanism, the DL/fair server around
the RT tasks, the /proc/sys/kernel/sched_rt_* mechanism to control the
former and CONFIG_RT_GROUP_SCHED which never worked as it supposed to.
The current status looks like a mess and I'm not sure if there is a bug
somewhere leading to my headaches or if this is just not meant to be
working together. If so out with it.

tl;dr /proc/…/sched_rt_*, sched_rt_runtime_exceeded() probably
CONFIG_RT_GROUP_SCHED should be removed. A bit of docs should be added
for the fair server (or the right place).

Now. Details.

The "old" RT throttler was meant to throttle RT task(s) that went rogue.
The idea was to throttle the RT tasks so that the admin, using
SCHED_OTHER, could step in and kill the misbehaving task.
This, as simple as it sounds, is problematic on a PREEMPT_RT system
because all the drivers are running also as SCHED_FIFO meaning
throttling the RT class resulted also in not servicing any HW. So an
admin can't interact with the system via SSH/ network or keyboard/ USB.
What often did help was the fact that the RT application did realise its
overstepping and contained themself.
As a result real-time applications under PREEMPT_RT disable the RT
throttler because from time to time they do have legitimate cases where
the RT class runs > 1s and the throttler just breaks everything. So
hush.
Some people did not learn about the RT-throttler because at some point
one CPU could run 100% real-time tasks if there were other CPUs
available (the argument was that the SCHED_OTHER workload could be moved
there).

The throttler used to be present unconditionally. Since commit
5f6bd380c7bdb ("sched/rt: Remove default bandwidth control") (v6.12) it
is hidden behind CONFIG_RT_GROUP_SCHED. Without CONFIG_RT_GROUP_SCHED
defined it does not exist, yet the proc files are there. Not sure if
they do much.

With CONFIG_RT_GROUP_SCHED enabled assigning SCHED_FIFO-1 priority to a
user task is kind of broken as it returns with -EPERM due to

| #ifdef CONFIG_RT_GROUP_SCHED
|                 /*
|                  * Do not allow real-time tasks into groups that have no runtime
|                  * assigned.
|                  */
|                 if (rt_group_sched_enabled() &&
|                                 rt_bandwidth_enabled() && rt_policy(policy) &&
|                                 task_group(p)->rt_bandwidth.rt_runtime == 0 &&
|                                 !task_group_is_autogroup(task_group(p))) {
|                         retval = -EPERM;
|                         pr_err("%s(%d)\n", __func__, __LINE__);
|                         goto unlock;
|                 }
| #endif /* CONFIG_RT_GROUP_SCHED */

rt_bandwidth.rt_runtime is zero and auto-group (which is SCHED_OTHER
anyway) is off. I did not find a way to enable one of these knobs. The
cgroup file (rt_runtime_us) that _should_ set rt_bandwidth.rt_runtime is
CGroup v1 and not available (since we do v2 these days). I don't see how
to bypass this. Unless rt_group_sched_enabled() is disabled I see no
hope.
Oh wait, this was exactly mentioned in commit 277e0909754e9 ("sched:
Bypass bandwitdh checks with runtime disabled RT_GROUP_SCHED").

Once I passed these gates I got stuck at the next barrier. Since commit
c2e390197ad13 ("sched/rt: Update default bandwidth for real-time tasks
to ONE") (v7.2) the value for sysctl_sched_rt_runtime is at 1s, same as
sysctl_sched_rt_period. (I am not going to mention that the
documentation in Documentation/scheduler/sched-rt-group.rst is kind of
outdated.)

This means the throttler does nothing since period can not exceed
the runtime. I'm not sure why exactly but I can't lower the value
because then tg_rt_schedulable() will complain in

|         /*
|          * Nobody can have more than the global setting allows.
|          */
|         if (total > to_ratio(global_rt_period(), global_rt_runtime()))
|                 return -EINVAL;

so that one second remains. It can be disabled (via -1). One can revert
the commit and have the throttler is working again (but more later).
Once at 0.95s it can go up to 1s and down again but it can't go below
its initial value. This does not look as it is working as designed.

Without enabled RT_GROUP_SCHED the files can be set. Are they useful?
According to Documentation/scheduler/sched-deadline.rst it used for
"admission control" whatever that means. Anyway the fair server still
kicks if it is disabled. So I *think* this is just something else and
not part for the docs there.

With RT_GROUP_SCHED enabled the two proc files can't change their
content. Plus it all kind of works just for cgroup-v1 and for v2 it
breaks RT applications. And it never worked AFAIK. So from my point of
view CONFIG_RT_GROUP_SCHED did not serve any good and can go.

Moving on. The old throttler did warn one before throttling real-time
tasks so the risk of a heart attack was limited but still there. The
sched* trace events do not show this: the RT task goes away and the CPU
might go idle in the end. But we did have the printk().
On the other hand the fair-server decides that it is time for a
SCHED_OTHER task, the scheduler obeys and we switch to SCHED_OTHER and
then back to SCHED_FIFO. Okay, more efficient (as no CPU in idle). No
hint what goes on.

In order to get to the non-throttling behavior one needs to
	echo 0 > /sys/kernel/debug/sched/fair_server/cpuX/runtime

There is a crumble of documentation in sched-rt-group.rst which went
in via commit b1536481c81fb ("sched/rt: Update limit of sched_rt sysctl
in documentation") but it has nothing to do with CONFIG_RT_GROUP_SCHED
and the fair server is 100% fine working without it. I don't think it is
the right place for this.

Side note: Remember me asking for a sched-event in sched_tick() if
sched-lazy is promoted to an actual scheduling request? Well, this is
what the fair-server preemption looks like with sched event enabled:

|       loop-time-1295    [000] d.h1.   339.006215: sched_stat_runtime: comm=loop-time pid=1295 runtime=3999761 [ns]
|       loop-time-1295    [000] d.h1.   339.006482: sched_stat_runtime: comm=loop-time pid=1295 runtime=267302 [ns]
|       loop-time-1295    [000] d.h1.   339.008953: sched_stat_runtime: comm=loop-time pid=1295 runtime=2471124 [ns]
|       loop-time-1295    [000] dN.2.   339.008955: sched_stat_runtime: comm=loop-time pid=1295 runtime=1152 [ns]
|       loop-time-1295    [000] d..2.   339.008956: sched_switch: prev_comm=loop-time prev_pid=1295 prev_prio=9 prev_state=R ==> next_comm=kworker/u4:4 next_pid=1148 next_prio=120
|    kworker/u4:4-1148    [000] d..21   339.008959: sched_waking: comm=sshd-session pid=1256 prio=120 target_cpu=000

suddenly Need-resched is set and we schedule to lower prio task. Not a
problem if you know about this… Or have the timer events enabled at
which point dl_task_timer will give you a hint. But we have LLMs so…

Anyway. Could this interface be moved to /sys/kernel/sched/fair_server
away from the debugfs part? debugfs is not for stable kernels/ ABI. Is
there a reason why this might be considered debugging and not final or
restricted from productive?

Just to summarize:
- Remove /proc/sys/kernel/sched_rt_*
  [ ] Yes [ ] No, because
              _____________________
- Remove CONFIG_RT_GROUP_SCHED
  [ ] Yes [ ] No, because
              _____________________

- An additional trace sched event for fair server preemption
  [ ] Yes   [ ] No, it is silly to have

- Fair server documentation needs to go to
  [ ] It is fine where it is
  [ ] To ____________________

Sebastian

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Tejun Heo]

Hello, Sebastian.

On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior wrote:
> - Remove CONFIG_RT_GROUP_SCHED
>   [ ] Yes [ ] No, because
>               _____________________

RT_GROUP_SCHED in its current form isn't widely useful. No major distro
kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu, openSUSE,
Arch and the Android GKI defconfig, PREEMPT_RT flavors included). I have
no objection to removing it.

There was an attempt to make it more useful: the Hierarchical Constant
Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni. It
replaces RT_GROUP_SCHED with per-cgroup deadline servers and a cgroup2
interface. The last posting was RFC v6 in June:

https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/

The effort seems to have stalled.

Thanks.

--
tejun

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[luca abeni]

Hi all,

On Fri, 23 Oct 2026 07:16:10 -1000
Tejun Heo <tj@kernel.org> wrote:

> Hello, Sebastian.
> 
> On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior
> wrote:
> > - Remove CONFIG_RT_GROUP_SCHED
> >   [ ] Yes [ ] No, because
> >               _____________________  
> 
> RT_GROUP_SCHED in its current form isn't widely useful. No major
> distro kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu,
> openSUSE, Arch and the Android GKI defconfig, PREEMPT_RT flavors
> included). I have no objection to removing it.
> 
> There was an attempt to make it more useful: the Hierarchical Constant
> Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni.
> It replaces RT_GROUP_SCHED with per-cgroup deadline servers and a
> cgroup2 interface. The last posting was RFC v6 in June:
> 
> https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/
> 
> The effort seems to have stalled.

Yuri is working on an updated version of the patchset... If everything
goes well, a new RFC should arrive in few weeks :)


			Luca

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Sebastian Andrzej Siewior]

On 2026-10-23 07:16:10 [-1000], Tejun Heo wrote:
> Hello, Sebastian.
Hi Tejun,

> On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior wrote:
> > - Remove CONFIG_RT_GROUP_SCHED
> >   [ ] Yes [ ] No, because
> >               _____________________
> 
> RT_GROUP_SCHED in its current form isn't widely useful. No major distro
> kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu, openSUSE,
> Arch and the Android GKI defconfig, PREEMPT_RT flavors included). I have
> no objection to removing it.

Perfect.

> There was an attempt to make it more useful: the Hierarchical Constant
> Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni. It
> replaces RT_GROUP_SCHED with per-cgroup deadline servers and a cgroup2
> interface. The last posting was RFC v6 in June:
> 
> https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/

Yes, I am aware of this. It replaces RT_GROUP_SCHED and does not rely on
it (as far as I remember).

> The effort seems to have stalled.
> 
> Thanks.

Sebastian

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Juri Lelli]

Hi Sebastian,

On 23/10/26 18:13, Sebastian Andrzej Siewior wrote:
> Hi,
> 
> this is about the old RT throttling mechanism, the DL/fair server around
> the RT tasks, the /proc/sys/kernel/sched_rt_* mechanism to control the
> former and CONFIG_RT_GROUP_SCHED which never worked as it supposed to.
> The current status looks like a mess and I'm not sure if there is a bug
> somewhere leading to my headaches or if this is just not meant to be
> working together. If so out with it.
> 
> tl;dr /proc/…/sched_rt_*, sched_rt_runtime_exceeded() probably
> CONFIG_RT_GROUP_SCHED should be removed. A bit of docs should be added
> for the fair server (or the right place).

Thanks for putting this recap together!

> Now. Details.

...

> Without enabled RT_GROUP_SCHED the files can be set. Are they useful?
> According to Documentation/scheduler/sched-deadline.rst it used for
> "admission control" whatever that means. Anyway the fair server still
> kicks if it is disabled. So I *think* this is just something else and
> not part for the docs there.

Right, confusing, but for a root_domain comprising M CPUs, -deadline tasks can
be created while the sum of their bandwidths stays below

  M * (sched_rt_runtime_us / sched_rt_period_us)

and writing -1 disables the check altogether. That's the "admission control"
the docs hand-wave about, and it's also why the knobs can't simply be deleted.
As you noticed already, the servers are completely independent of these knobs:
writing -1 does not disable the fair server, it only lifts the DL admission
cap. In principle we could move this to a different interface that reads closer
to the underlying purpose, but I'm not sure how much breakage it could cause.

...

> Anyway. Could this interface be moved to /sys/kernel/sched/fair_server
> away from the debugfs part? debugfs is not for stable kernels/ ABI. Is
> there a reason why this might be considered debugging and not final or
> restricted from productive?

I think this was raised already in the past, but at the time maybe it
was still to early to say if dl-servers will stick around. They now have
got more users (sched-ext), so it might be a time we discuss a stable
(non-debug) interface for it. Your suggestion doesn't look bad to me.

> Just to summarize:
> - Remove /proc/sys/kernel/sched_rt_*
>   [ ] Yes [ ] No, because
>               _____________________

Yes and no :). We need an interface to tweak admission control, but we
could find a better/different one.

> - Remove CONFIG_RT_GROUP_SCHED
>   [ ] Yes [ ] No, because
>               _____________________

Luca already commented on this. I believe the plan is still to come up
with a DEADLINE backed RT_GROUP_SCHED for Cgroups v2. If cleaner, an
approach could be to first merge the removal of RT_GROUP_SCHED and then
reimplement the whole lot as Yuri and Luca are proposing to do. The
series is already somewhat doing that, but it could be made more
explicit/clean I guess.

> - An additional trace sched event for fair server preemption
>   [ ] Yes   [ ] No, it is silly to have

With 820725b0eb59f ("sched: Add deadline tracepoints") we already have
some of that: sched_dl_{throttle,replenish,update,server_start,server_stop},
which fire for fair/ext servers too (there's a type field, DL_SERVER_FAIR
and friends). They of course require something like this [1] to be
activated. We could add more if they are still not enough to get a clear
picture about what's going on.

> - Fair server documentation needs to go to
>   [ ] It is fine where it is
>   [ ] To ____________________

Not fine where it is: the servers (fair & ext) work independently of
RT_GROUP_SCHED and deserve their own description (defaults 50ms/1s per
CPU, what they guarantee for fair tasks under RT pressure, how to tune
or disable them). Either a new section in sched-deadline.rst (they are
DL servers after all) or a small standalone file works for me.

Thanks!
Juri

1 - https://github.com/jlelli/sched_tp/commits/deadline-tp/
    from Qais' https://github.com/qais-yousef/sched_tp