Re: [PATCH v5 18/29] sched/deadline: Move a task's bandwidth only where room was made

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

Hi Waiman,

On 2026-11-13 12:33, Waiman Long wrote:
> Right, top_cpuset is the exception here. Boot isolated CPUs are not
> allowed in a root partition. They are only allowed in isolated
> partitions. So root partition other than the top one cannot span more
> than one root domain.

Agreed for partition roots. The case I mean is a cpuset that is not a
partition root. A member cpuset below the top cpuset can list the
isolated CPUs, and gets them as effective CPUs. A in my last mail was
one, and B below is another.

> Are you referring to the case that shrinking the cpuset CPU count to
> reduce the total DL bandwidth will fail with EBUSY when it is less
> than what the existing DL tasks require?

No. That check is about capacity, and the planned next version leaves
it as it is. The two new refusals are about the root domain that
counts a task: the one whose total_bw holds its bandwidth. They apply
when a change leaves a deadline task no CPU in that root domain. There
is room everywhere in the two examples below.

Both are on tip sched/urgent, in a 5-CPU guest booted with
isolcpus=domain,1,2. CPUs 0, 3 and 4 are in the housekeeping root
domain. CPUs 1 and 2 are in the default root domain. B is a member
cpuset of CPUs 1-3:

   # echo 1-3 > B/cpuset.cpus
   # cat B/cpuset.cpus.effective
   1-3
   # cat B/cpuset.cpus.partition
   member

T is a task of the top cpuset, pinned to CPUs 0, 3 and 4 by
sched_setaffinity(). It calls sched_setattr() for 10% and is admitted.
I then move it into B:

   # echo $T > B/cgroup.procs

CPU 3 is the one CPU that B and the affinity of T share, so T is
allowed on CPU 3 only. The housekeeping root domain counts it. The
tables give total_bw of each root domain, from
/sys/kernel/debug/sched/debug. "Before" is the fair servers alone, and
T adds 104857.

1. A write to cpuset.cpus of B:

   # echo 1-2 > B/cpuset.cpus

   CPUs    before    T in B    after the write
   0,3,4   157284    262141    157284
   1,2     104856    104856    104856

The write is accepted. T is now allowed on CPUs 1-2 and is on CPU 1.
No root domain counts its 10%.

2. A partition root Q, next to B, takes CPU 3:

   # echo 3 > Q/cpuset.cpus
   # echo root > Q/cpuset.cpus.partition
   # cat Q/cpuset.cpus.partition
   root
   # cat B/cpuset.cpus.effective
   1-2

   CPUs    before    T in B    after Q is a partition root
   0,4     157284    262141    104856
   3       157284    262141     52428
   1,2     104856    104856    209713

CPUs 0, 3 and 4 are one root domain until Q is a partition root. Q is
valid. T is allowed on CPUs 1-2 and is on CPU 1. The rebuild counts it
in the default root domain, which was never asked for room.

With the planned next version:
- the write of case 1 fails with EBUSY, and B keeps CPUs 1-3;
- Q of case 2 is invalid, and cpuset.cpus.partition reads "root invalid
   (Deadline task would leave its root domain)".

T then stays on CPU 3, and total_bw keeps the values of the column
"T in B". A move of T to another cpuset still takes it to the other
root domain: the attach asks for room there.

> We already have pending cpuset patch that will disallow taking the
> last housekeeping CPU from the top_cpuset partition.

Which patch is it? I would like to test the series on top of it. From
your description it covers another case: in case 2 above, the top
cpuset keeps the housekeeping CPUs 0 and 4.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com