[PATCHSET v3 sched_ext/for-7.4] sched_ext: Report a sub-scheduler's ecaps on attach, detach and cpu hotplug

[Tejun Heo <tj@kernel.org>]

Hello,

v3 (resched from irq work):
- Reschedule the cpu for its queued ecaps syncs from hard irq work on that
  cpu instead of a kick. dispatch_one() processes the syncs only at its
  start, and a kick that lands while the pick has the rq lock dropped is
  cleared with the pick (#3, #5, sashiko).
- Fix the SCX_RQ_ONLINE comments (#4) and why no report mark is pending at a
  suspend down (#6), and update comments and descriptions.

v2 (detach and cpu hotplug):
- Report the caps on detach as well as attach, and deliver regardless of
  bypass, which drops the replay at unbypass (#3).
- Drop the sync_pcpu_ecaps() factoring. Add the cid_pshard() factoring and
  free a sub-scheduler a grace period after discarding its syncs (#1, #2).
- Deliver only while the root has the cpu online, report the loss when a
  cpu goes offline, and leave suspend and hibernation out (#4-#6).

v2: https://lore.kernel.org/r/20261017164556.4136807-1-tj@kernel.org
v1: https://lore.kernel.org/r/20261009094356.2852794-1-tj@kernel.org

A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(),
but the grants its parent makes when it attaches are reported only after its
enable bypass is lifted, which is also when its tasks are handed to it. So
it gets its first ops.enqueue() calls before it has been notified of a
single cid and has to hold the tasks somewhere until the grants arrive.

The losses aren't reported either. The caps go away on detach and when a cpu
goes offline without a delivery to the sub-scheduler or its parent. The
reseed of a cpu coming back can also run their ops before the root's
ops.cpu_online().

With this patchset, a sub-scheduler's ecaps change only with the deliveries
that report them. The attach-time caps are reported before bypass lifts, the
loss on detach before any task goes back to the parent, and a cpu's loss and
regain around the root's ops.cpu_offline() and ops.cpu_online(). Suspend
and hibernation are left out: a cpu they take down keeps the caps last
delivered, and the resume reports only the net change.

Verified with sub-schedulers attaching and detaching with tasks queued and
caps granted and revoked, through cpu hotplug with the races widened by
debug patches and grants flooding a cpu as it goes down, on PREEMPT_RT as
well, and across suspend. The protocol between the reports, the per-cpu
syncs, cpu hotplug, suspend and the free path was model checked with TLC
for small cpu counts.

Based on sched_ext/for-7.4 (9c0af54cb49c).

This patchset contains the following 6 patches.

 0001 sched_ext: Factor out cid_pshard()
 0002 sched_ext: Free a sub-scheduler a grace period after discarding its ecaps syncs
 0003 sched_ext: Report a sub-scheduler's ecaps on attach and detach
 0004 sched_ext: Deliver ecaps only while the root has the cpu online
 0005 sched_ext: Report the ecaps loss when a cpu goes offline
 0006 sched_ext: Leave suspend and hibernation out of the ecaps reports

The patches are organized as follows:

- 0001-0002 are prep. 0002 stops the free path from relying on a bypassing
  sched's syncs calling no op.

- 0003 reports the caps on attach and detach.

- 0004-0006 cover cpu hotplug.

The patchset is also available in the following git branch:

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git sub-attach-ecaps-v3

diffstat follows. Thanks.

 kernel/sched/ext/ext.c         | 157 ++++++++---
 kernel/sched/ext/internal.h    |  71 ++++-
 kernel/sched/ext/sub.c         | 625 +++++++++++++++++++++++++++++++----------
 kernel/sched/ext/sub.h         |  10 +-
 kernel/sched/sched.h           |  17 +-
 tools/sched_ext/scx_qmap.bpf.c |  25 +-
 6 files changed, 683 insertions(+), 222 deletions(-)

--
tejun