[RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Sebastian Andrzej Siewior]

Hi,

this is about the old RT throttling mechanism, the DL/fair server around
the RT tasks, the /proc/sys/kernel/sched_rt_* mechanism to control the
former and CONFIG_RT_GROUP_SCHED which never worked as it supposed to.
The current status looks like a mess and I'm not sure if there is a bug
somewhere leading to my headaches or if this is just not meant to be
working together. If so out with it.

tl;dr /proc/…/sched_rt_*, sched_rt_runtime_exceeded() probably
CONFIG_RT_GROUP_SCHED should be removed. A bit of docs should be added
for the fair server (or the right place).

Now. Details.

The "old" RT throttler was meant to throttle RT task(s) that went rogue.
The idea was to throttle the RT tasks so that the admin, using
SCHED_OTHER, could step in and kill the misbehaving task.
This, as simple as it sounds, is problematic on a PREEMPT_RT system
because all the drivers are running also as SCHED_FIFO meaning
throttling the RT class resulted also in not servicing any HW. So an
admin can't interact with the system via SSH/ network or keyboard/ USB.
What often did help was the fact that the RT application did realise its
overstepping and contained themself.
As a result real-time applications under PREEMPT_RT disable the RT
throttler because from time to time they do have legitimate cases where
the RT class runs > 1s and the throttler just breaks everything. So
hush.
Some people did not learn about the RT-throttler because at some point
one CPU could run 100% real-time tasks if there were other CPUs
available (the argument was that the SCHED_OTHER workload could be moved
there).

The throttler used to be present unconditionally. Since commit
5f6bd380c7bdb ("sched/rt: Remove default bandwidth control") (v6.12) it
is hidden behind CONFIG_RT_GROUP_SCHED. Without CONFIG_RT_GROUP_SCHED
defined it does not exist, yet the proc files are there. Not sure if
they do much.

With CONFIG_RT_GROUP_SCHED enabled assigning SCHED_FIFO-1 priority to a
user task is kind of broken as it returns with -EPERM due to

| #ifdef CONFIG_RT_GROUP_SCHED
|                 /*
|                  * Do not allow real-time tasks into groups that have no runtime
|                  * assigned.
|                  */
|                 if (rt_group_sched_enabled() &&
|                                 rt_bandwidth_enabled() && rt_policy(policy) &&
|                                 task_group(p)->rt_bandwidth.rt_runtime == 0 &&
|                                 !task_group_is_autogroup(task_group(p))) {
|                         retval = -EPERM;
|                         pr_err("%s(%d)\n", __func__, __LINE__);
|                         goto unlock;
|                 }
| #endif /* CONFIG_RT_GROUP_SCHED */

rt_bandwidth.rt_runtime is zero and auto-group (which is SCHED_OTHER
anyway) is off. I did not find a way to enable one of these knobs. The
cgroup file (rt_runtime_us) that _should_ set rt_bandwidth.rt_runtime is
CGroup v1 and not available (since we do v2 these days). I don't see how
to bypass this. Unless rt_group_sched_enabled() is disabled I see no
hope.
Oh wait, this was exactly mentioned in commit 277e0909754e9 ("sched:
Bypass bandwitdh checks with runtime disabled RT_GROUP_SCHED").

Once I passed these gates I got stuck at the next barrier. Since commit
c2e390197ad13 ("sched/rt: Update default bandwidth for real-time tasks
to ONE") (v7.2) the value for sysctl_sched_rt_runtime is at 1s, same as
sysctl_sched_rt_period. (I am not going to mention that the
documentation in Documentation/scheduler/sched-rt-group.rst is kind of
outdated.)

This means the throttler does nothing since period can not exceed
the runtime. I'm not sure why exactly but I can't lower the value
because then tg_rt_schedulable() will complain in

|         /*
|          * Nobody can have more than the global setting allows.
|          */
|         if (total > to_ratio(global_rt_period(), global_rt_runtime()))
|                 return -EINVAL;

so that one second remains. It can be disabled (via -1). One can revert
the commit and have the throttler is working again (but more later).
Once at 0.95s it can go up to 1s and down again but it can't go below
its initial value. This does not look as it is working as designed.

Without enabled RT_GROUP_SCHED the files can be set. Are they useful?
According to Documentation/scheduler/sched-deadline.rst it used for
"admission control" whatever that means. Anyway the fair server still
kicks if it is disabled. So I *think* this is just something else and
not part for the docs there.

With RT_GROUP_SCHED enabled the two proc files can't change their
content. Plus it all kind of works just for cgroup-v1 and for v2 it
breaks RT applications. And it never worked AFAIK. So from my point of
view CONFIG_RT_GROUP_SCHED did not serve any good and can go.

Moving on. The old throttler did warn one before throttling real-time
tasks so the risk of a heart attack was limited but still there. The
sched* trace events do not show this: the RT task goes away and the CPU
might go idle in the end. But we did have the printk().
On the other hand the fair-server decides that it is time for a
SCHED_OTHER task, the scheduler obeys and we switch to SCHED_OTHER and
then back to SCHED_FIFO. Okay, more efficient (as no CPU in idle). No
hint what goes on.

In order to get to the non-throttling behavior one needs to
	echo 0 > /sys/kernel/debug/sched/fair_server/cpuX/runtime

There is a crumble of documentation in sched-rt-group.rst which went
in via commit b1536481c81fb ("sched/rt: Update limit of sched_rt sysctl
in documentation") but it has nothing to do with CONFIG_RT_GROUP_SCHED
and the fair server is 100% fine working without it. I don't think it is
the right place for this.

Side note: Remember me asking for a sched-event in sched_tick() if
sched-lazy is promoted to an actual scheduling request? Well, this is
what the fair-server preemption looks like with sched event enabled:

|       loop-time-1295    [000] d.h1.   339.006215: sched_stat_runtime: comm=loop-time pid=1295 runtime=3999761 [ns]
|       loop-time-1295    [000] d.h1.   339.006482: sched_stat_runtime: comm=loop-time pid=1295 runtime=267302 [ns]
|       loop-time-1295    [000] d.h1.   339.008953: sched_stat_runtime: comm=loop-time pid=1295 runtime=2471124 [ns]
|       loop-time-1295    [000] dN.2.   339.008955: sched_stat_runtime: comm=loop-time pid=1295 runtime=1152 [ns]
|       loop-time-1295    [000] d..2.   339.008956: sched_switch: prev_comm=loop-time prev_pid=1295 prev_prio=9 prev_state=R ==> next_comm=kworker/u4:4 next_pid=1148 next_prio=120
|    kworker/u4:4-1148    [000] d..21   339.008959: sched_waking: comm=sshd-session pid=1256 prio=120 target_cpu=000

suddenly Need-resched is set and we schedule to lower prio task. Not a
problem if you know about this… Or have the timer events enabled at
which point dl_task_timer will give you a hint. But we have LLMs so…

Anyway. Could this interface be moved to /sys/kernel/sched/fair_server
away from the debugfs part? debugfs is not for stable kernels/ ABI. Is
there a reason why this might be considered debugging and not final or
restricted from productive?

Just to summarize:
- Remove /proc/sys/kernel/sched_rt_*
  [ ] Yes [ ] No, because
              _____________________
- Remove CONFIG_RT_GROUP_SCHED
  [ ] Yes [ ] No, because
              _____________________

- An additional trace sched event for fair server preemption
  [ ] Yes   [ ] No, it is silly to have

- Fair server documentation needs to go to
  [ ] It is fine where it is
  [ ] To ____________________

Sebastian

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Tejun Heo]

Hello, Sebastian.

On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior wrote:
> - Remove CONFIG_RT_GROUP_SCHED
>   [ ] Yes [ ] No, because
>               _____________________

RT_GROUP_SCHED in its current form isn't widely useful. No major distro
kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu, openSUSE,
Arch and the Android GKI defconfig, PREEMPT_RT flavors included). I have
no objection to removing it.

There was an attempt to make it more useful: the Hierarchical Constant
Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni. It
replaces RT_GROUP_SCHED with per-cgroup deadline servers and a cgroup2
interface. The last posting was RFC v6 in June:

https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/

The effort seems to have stalled.

Thanks.

--
tejun

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[luca abeni]

Hi all,

On Fri, 23 Oct 2026 07:16:10 -1000
Tejun Heo <tj@kernel.org> wrote:

> Hello, Sebastian.
> 
> On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior
> wrote:
> > - Remove CONFIG_RT_GROUP_SCHED
> >   [ ] Yes [ ] No, because
> >               _____________________  
> 
> RT_GROUP_SCHED in its current form isn't widely useful. No major
> distro kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu,
> openSUSE, Arch and the Android GKI defconfig, PREEMPT_RT flavors
> included). I have no objection to removing it.
> 
> There was an attempt to make it more useful: the Hierarchical Constant
> Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni.
> It replaces RT_GROUP_SCHED with per-cgroup deadline servers and a
> cgroup2 interface. The last posting was RFC v6 in June:
> 
> https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/
> 
> The effort seems to have stalled.

Yuri is working on an updated version of the patchset... If everything
goes well, a new RFC should arrive in few weeks :)


			Luca

Re: [RFC] RT throttling, DL/ fair server, RT_GROUP_SCHED and everything

[Sebastian Andrzej Siewior]

On 2026-10-23 07:16:10 [-1000], Tejun Heo wrote:
> Hello, Sebastian.
Hi Tejun,

> On Fri, Oct 23, 2026 at 06:13:48PM +0200, Sebastian Andrzej Siewior wrote:
> > - Remove CONFIG_RT_GROUP_SCHED
> >   [ ] Yes [ ] No, because
> >               _____________________
> 
> RT_GROUP_SCHED in its current form isn't widely useful. No major distro
> kernel enables it (Fedora, RHEL/CentOS Stream, Debian, Ubuntu, openSUSE,
> Arch and the Android GKI defconfig, PREEMPT_RT flavors included). I have
> no objection to removing it.

Perfect.

> There was an attempt to make it more useful: the Hierarchical Constant
> Bandwidth Server (HCBS) patchset from Yuri Andriaccio and Luca Abeni. It
> replaces RT_GROUP_SCHED with per-cgroup deadline servers and a cgroup2
> interface. The last posting was RFC v6 in June:
> 
> https://lore.kernel.org/all/20260608121546.69910-1-yurand2000@gmail.com/

Yes, I am aware of this. It replaces RT_GROUP_SCHED and does not rely on
it (as far as I remember).

> The effort seems to have stalled.
> 
> Thanks.

Sebastian