Re: [RFC PATCH v7 00/29] Hierarchical Constant Bandwidth Server

[Yuri Andriaccio <yurand2000@gmail.com>]

Hello Tejun,

Thanks for taking a look. I'm answering some of your comments first and at the
end I'm proposing a summary of changes that will appear in the next RFC to
then "stabilize" the user interface.



> The new name still doesn't follow the cgroup2 naming convention.

Thanks for pointing this out, it was a mistake on my end. This will be fixed
in the next RFC.

> The file prints a "CPU runtime=R period=P" line for each possible CPU.

We will condense the displaying for same runtime/period CPUs and omit zeroed
ones.



> Switching between "max" and a budget is refused while the cgroup or its
> "max" descendants have RT tasks. That's okay as a restriction for now.
> Ideally, the switch wouldn't be restricted as long as the budget is
> sufficient, but that can be left for the future.

The restriction is in place to simplify the initial implementation: switching
between "max" and non-"max" requires to migrate all tasks from one cgroup's
runqueues to another cgroup ones, and could potentially be an expensive
operation if there are lots of tasks. If you think the restriction is
acceptable (for now), we will keep this unchanged in hte next RFC, and fix it
later.



> Can we separate cpu.rt.max from cpuset?

Maybe we misunderstood, but in some previous discussion it was asked us to
integrate the `cpuset` and `cpu` controllers somehow. Your proposal of having
the `cpu` and `cpuset` controllers orthogonal simplifies things, but it is
similar to the previous (cgroup-v1-based) implementation, or before the
integration patch itself (i.e. just allow the per-CPU overrides but remove the
tighter coupling). What is the final verdict on this?



> On the per-CPU budgets themselves, they don't match cpu.max, and giving
> a budget per CPU is a bit odd when the consumption isn't per CPU. The
> threads move around. Maybe it can't be avoided, but I'd like to see a
> discussion of the rationale behind the design choice and some
> exploration of alternatives.

I'm not sure if I'm understanding correctly here, but the budget for
`cpu.rt.max`, differently from `cpu.max`, is meant to be per-CPU and not
global. Sure the initial patches have a single setting that is the same on all
CPUs, but it was meant to be a per-CPU limit that is the same across all cores.
Notice that threads move around, but we prevent migrating them to cores with
zero runtime (hence, there is some overlap of functionality between cpusets
and rt-cgroups).

The rationale behind this is that, from a real-time theoretical pov, it is
necessary to know how much budget and how it is distributed on the CPUs, as
changing the number of CPUs or the budget distribution radically changes the
schedulability outputs. The most complete and flexible interface is therefore
having per-CPU budgets.

This is also necessary to prevent over-provisioning and allowing partitioning
of the total bandwidth:

- If implementing solution 1, a request of 10% of budget reserves 10% of
  runtime on all cores, regardless of `cpuset` settings. Of course a `cpuset`
  mask that is smaller than the total allows limiting the tasks on a subset
  of CPUs, yet the budget on the unused cores is still allocated, and therefore
  wasted. This is a case of over-provisioning.
- On the other hand, allowing per-CPU controls allows to potentially partition
  the total bandwidth in the CPU hierarchy, for example reserving some 10%
  runtime on some cores, while still allowing other cgroups to make use of the
  unused 10% bandwidth on the other cores.

From this limited discussion it appears to us that solution 2, which is
similar to what we already had but without the "default" case, is the most
suitable. Solutions 1 and 3 are more or less what already exists before
the `cpuset` integration patches.



> AFAICS, disabling the cpuset controller above a budgeted cgroup moves it to
> the ancestor's cpuset and changes its effective CPUs without going through
> the lockout at all.

You are right, we tried to prevent changes in the `cpuset` when an active
rt-cgroup was involved but this was clearly missed out.

The lockout is in place because, in the current integration, it is necessary
to rerun the bandwdith admission test on every `cpuset` change, which greatly
complicates the code. Nonetheless, a more permissive integration still needs
to couple updates in both controllers and run some admission tests.



> Skipping the boost when the target has no runtime is fine as long
> as nothing stalls.

For PI and zero-budget cgroups, if this is an acceptable solution, it seems
the most straightforward one: disallow switching to RT for zeroed cgroups and
disable zeroing the budget of any parent which has active RT tasks.

- - -

As mentioned at the beginning, here is a summary of what the interface
would be after the discussed points:

- `cpu.rt.max` and `cpu.rt.max.effective.local` files. When displaying the
  per-CPU settings, zero runtimes are skipped and same runtime/period cpus
  are condensed in the same cpuset.

- `cpu.rt.max` only accepts per-CPU settings, so the global "default" is
  removed. Removing all overrides means the cgroup is a "max" cgroup, while
  just a single override means that the cgroup has its own budget.
  + This new interface disallows zero-runtime overrides, as a zero override
    is equal to an empty override when there is at least one non-zero override.
    As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just
    having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?

- Switching between `max` and non-`max` is still disallowed for active
  real-time cgroups (to be left for the future).

- `cpu` and `cpuset` controllers are decoupled, as it was before the `cpuset`
  integration patch. The user/admin is responsible for inconsistencies between
  the two controllers (i.e. empty intersection between active server in the
  rt-cgroup and the allowed cpuset just disallows executing RT tasks entirely).

- zero "internal" runtime cgroups will disallow RT tasks and prevent
  priority boosting. Admission controls will disallow zeroing the "internal"
  budget of a parent which has active tasks running.

Hopefully we can agree on the ideas posted here. If you have any doubts or
better suggestions let us know.

Thanks,
Yuri