Re: [REPORT] sched_ext: sub-scheduler tasks are severely under-scheduled vs root-owned tasks

[Tao Cui <cui.tao@linux.dev>]

[-- Attachment #1: Type: text/plain, Size: 5628 bytes --]

Hello Tejun,

在 2026/10/8 17:10, Tejun Heo 写道:
> Hello, Tao.
> 
> On Thu, Oct 08, 2026 at 04:36:49PM +0800, Tao Cui wrote:
>> A pair of minimal cid-form test
>> schedulers: a parent that grants SCX_CAP_PERF | SCX_CAP_ENQ_IMMED over
>> a cgroup to a child, and the child writing cpuperf target 1 from
>> ops.tick().
> 
> Can you explain exactly what you tested and how, including the scheduler
> source and complete reproduction steps? This isn't a useful bug report
> without that information.
> 

Sure thing, and thanks for pointing out what was missing. Attached
are the complete scheduler sources together with the exact setup and
reproduction steps.

The scheduler is a single binary (scx_k2) with parent/root and
child/sub modes. The parent grants SCX_CAP_PERF | SCX_CAP_ENQ_IMMED
to the child over a cgroup, while the child only writes cpuperf
target 1 from ops.tick(). Both schedulers print their counters every
2s.

Environment:

- x86_64 KVM guest, QEMU "-enable-kvm -cpu host -M pc -smp 4 -m 4G",
  CONFIG_HZ_1000, default NO_HZ idle, cgroup2, everything as root.
- Kernel: your for-7.4 tree at f55a5ae74ca6 (Merge branch
  'for-7.3-fixes' into for-7.4), plus cda36da11910 (sched_ext: Clear
  a sub-scheduler's caps before ops.sub_detach()).

Reproduce (one round):

  1. Build scx_k2 and scx_perfprobe (recipe below).

  2. Mount cgroup2 if not already mounted:
     mount -t cgroup2 none /sys/fs/cgroup

  3. Enable the CPU controller:
     echo +cpu > /sys/fs/cgroup/cgroup.subtree_control

  4. Create the test cgroup:
     mkdir -p /sys/fs/cgroup/ss/a

  5. Start the parent and let it come up:
     ./scx_k2 -g /sys/fs/cgroup/ss/a > root.log 2>&1 &
     sleep 2

  6. Start the spinner pinned to CPU 0:
     taskset 0x1 sh -c 'while :; do :; done' &

  7. Move it into the cgroup before attaching the child:
     echo $! > /sys/fs/cgroup/ss/a/cgroup.procs

  8. Start a root-owned spinner pinned to CPU 2:
     taskset 0x4 sh -c 'while :; do :; done' &

  9. Attach the child and let it settle:
     ./scx_k2 -s /sys/fs/cgroup/ss/a > sub.log 2>&1 &
     sleep 2

  10. Wait 6s.

  11. Record the spinner's utime+stime from /proc/<pid>/stat before
      and after the window, and inspect the counter lines in
      root.log and sub.log.

The attached repro-subsched-starve.sh automates the setup and
reports the result:

  sh repro-subsched-starve.sh before 1

Expected: the child owns the spinner and it continues running at
full duty cycle, accumulating approximately 600 CPU ticks over 6s
(utime+stime; the guest's USER_HZ is 100). While the task runs,
setcnt increases at roughly 100/s in our healthy rounds.

Actual: the child claims the task, and its ops.tick() initially
services it, but setcnt then stops increasing and the task receives
little or no CPU time - 0-91 ticks per 6s window across our runs,
most rounds 0. In affected rounds, subdisp0 stays at 0 throughout:
the child's ops.dispatch() is never invoked for its CID.

Counters (both logs):

  setcnt    - child ops.tick() invocations
  rootticks - parent ops.tick() on CPU 0
  disp0     - parent's ops.dispatch() invocations for CID 0
  subdisp0  - child's ops.dispatch() invocations for CID 0
  dsqdepth  - parent SHARED DSQ depth, sampled on tick

Attachments:

  scx_k2.bpf.c              parent and child schedulers in one BPF
                            object
  scx_k2.c                  userspace loader and counter printer
  scx_perfprobe.bpf.c       sched_tick kprobe sampling
                            rq->scx.cpuperf_target per CPU
  scx_perfprobe.c           sampler printer
  repro-subsched-starve.sh  one-round runner, accepting
                            [before|after] [hog]

Build (from tools/sched_ext, with the four sources dropped in):

  make $(pwd)/build/include/scx_k2.bpf.skel.h \
       $(pwd)/build/include/scx_perfprobe.bpf.skel.h

  gcc -g -O2 -pthread -Ibuild/include -I../../include/generated \
      -I../../tools/lib -I../../tools/include -I../../tools/include/uapi \
      -Iinclude -o scx_k2 scx_k2.c build/obj/libbpf/libbpf.a \
      -lelf -lz -lzstd -llzma -lpthread

  gcc -g -O2 -pthread -Ibuild/include -I../../include/generated \
      -I../../tools/lib -I../../tools/include -I../../tools/include/uapi \
      -Iinclude -o scx_perfprobe scx_perfprobe.c build/obj/libbpf/libbpf.a \
      -lelf -lz -lzstd -llzma -lpthread

Follow-up testing:

We repeated the tests across 11 boots (89 rounds, covering both
attach orders, with and without the root hog). Every round on 10
boots starved (74 rounds total); on the remaining boot, all 15
rounds ran at full duty cycle with normal child dispatch. The
healthy boot used the same binary build as one of the starving
boots, suggesting that the behavior is boot-dependent rather than
determined by the attach order or binary build. We haven't
identified what accounts for the difference yet. Note that this also
differs from my earlier mail, where order (b) appeared healthy; on
this tree, starvation occurs with both attach orders.

In an A/B test, having the parent kick the task's CPU from
ops.enqueue() with scx_bpf_kick_cid() and hand its dispatch turn to
the child with scx_bpf_sub_dispatch() prevents the starvation.

One possibility is that, on a NO_HZ idle CPU, no balance is
triggered for the child's DSQ, so the child's ops.dispatch() is
never called. This is still a hypothesis; the A/B result suggests
the dispatch/kick path is worth investigating.

I'll keep digging into the per-boot difference. If it doesn't
reproduce on your side, please let me know what differs (kernel
config, topology, or anything else).

Thanks,
Tao

> Thanks.
> 

[-- Attachment #2: repro-subsched-starve.sh --]
[-- Type: application/x-shellscript, Size: 2131 bytes --]

[-- Attachment #3: scx_k2.bpf.c --]
[-- Type: text/x-csrc, Size: 6572 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * scx_k2 - minimal cid-form scheduler pair for sub-scheduler attach
 * testing (single binary, two modes).
 *
 * Root mode (default): private FIFO DSQ; when a sub-scheduler attaches,
 * grants grant_caps (default SCX_CAP_ENQ_IMMED|SCX_CAP_PERF, full cid
 * mask) over the cgroup given with -g. Never writes cpuperf itself.
 * -R <ticks> issues scx_bpf_sub_revoke() of SCX_CAP_PERF on tick N
 * (0 = off).
 *
 * Sub mode (-s): ops.tick() writes perf_target via
 * scx_bpf_cidperf_set(this_cid).
 *
 * cid-form notes: must use SCX_OPS_CID_DEFINE/OPEN; every program must
 * touch the arena (k2_arena_touch) or bpf_prog_arena() comes back NULL
 * and programs get rejected; sub_cgroup_id must be written to both
 * rodata and struct_ops; the minimal cap set is ENQ_IMMED; a private
 * DSQ is required.
 */
#include <scx/common.bpf.h>

char _license[] SEC("license") = "GPL";

const volatile u64 grant_cgid;	/* root: cgid to grant/revoke caps over */
const volatile u64 grant_caps;	/* root: cap bits to grant */
const volatile u64 revoke_at;	/* root: revoke PERF on tick N (0 = off) */
const volatile u32 restore_on_detach; /* root: parent writes neutral values on sub_detach */
const volatile u64 sub_cgroup_id; /* sub mode: dual-written with ops->sub_cgroup_id */
const volatile u32 is_sub;	/* 1: sub mode */
const volatile u32 perf_target;	/* sub: target written every tick */

UEI_DEFINE(uei);

#define SHARED_DSQ 0

enum {
	K2_GRANT_RET = 0,	/* sub_grant return value (errno stored as 8192-errno) */
	K2_REVOKE_DONE = 1,	/* number of revokes performed */
	K2_SETCNT = 2,		/* number of sub tick writes */
	K2_SET_ERR = 3,		/* last cidperf_set return value (errno as 8192-errno) */
	K2_ROOTTICKS = 4,	/* root ticks on cpu0 (ownership discriminator) */
	K2_DSQDEPTH = 7,	/* SHARED_DSQ depth sampled in root tick (starvation probe) */
	K2_DISP0 = 8,		/* ROOT's cid0 dispatch invocations */
	K2_DISP0_MOVE = 9,	/* ROOT's cid0 successful move_to_local */
	K2_SUBDISP0 = 10,	/* SUB's cid0 dispatch invocations (core routing signal) */
	K2_SUBDISP0_MOVE = 11,	/* SUB's cid0 successful move_to_local */
	K2_CNT_NR = 12,
};

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__uint(max_entries, K2_CNT_NR);
	__type(key, u32);
	__type(value, u64);
} counters SEC(".maps");

/* cid-form requires an arena map (verified by the kernel) */
struct {
	__uint(type, BPF_MAP_TYPE_ARENA);
	__uint(map_flags, BPF_F_MMAPABLE);
	__uint(max_entries, 1 << 16);	/* pages */
} arena SEC(".maps");

struct scx_cmask __arena cap_mask;
u64 __arena k2_ticks;

static void bump(u32 idx)
{
	u64 *cnt = bpf_map_lookup_elem(&counters, &idx);

	if (cnt)
		(*cnt)++;
}

static void setcnt(u32 idx, s64 v)
{
	u64 val = v < 0 ? 8192 + (-v) : v;

	bpf_map_update_elem(&counters, &idx, &val, BPF_ANY);
}

/* every scx program must touch the arena, or bpf_prog_arena() == NULL -> EINVAL */
u64 __arena k2_arena_sink;
static void arena_touch(void)
{
	k2_arena_sink += cap_mask.alloc_words;
}

s32 BPF_STRUCT_OPS(k2_select_cid, struct task_struct *p,
		   s32 prev_cid, u64 wake_flags)
{
	arena_touch();
	return prev_cid;
}

void BPF_STRUCT_OPS(k2_enqueue, struct task_struct *p, u64 enq_flags)
{
	arena_touch();
	scx_bpf_dsq_insert(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
}

void BPF_STRUCT_OPS(k2_dispatch, s32 cid, struct task_struct *prev)
{
	arena_touch();

	if (is_sub) {
		/* the sub's own dispatch turn: consume our SHARED DSQ */
		if (cid == 0)
			bump(K2_SUBDISP0);
		if (scx_bpf_dsq_move_to_local(SHARED_DSQ, 0))
			bump(K2_SUBDISP0_MOVE);
		return;
	}

	/*
	 * Root's dispatch turn: consume our own SHARED DSQ. (A/B tested:
	 * handing the turn down to the sub first via
	 * scx_bpf_sub_dispatch() does not rescue the claimed task.)
	 */

	if (cid == 0) {
		bump(K2_DISP0);
		if (scx_bpf_dsq_move_to_local(SHARED_DSQ, 0))
			bump(K2_DISP0_MOVE);
	} else {
		scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
	}
}

/* root: grant caps when a sub attaches (full cid mask; test harness
 * deliberately over-grants) */
s32 BPF_STRUCT_OPS(k2_sub_attach, struct scx_sub_attach_args *args)
{
	u32 nr;
	s32 ret;

	arena_touch();
	if (!grant_cgid || !grant_caps)
		return 0;

	nr = scx_bpf_nr_cids();
	cmask_init(&cap_mask, 0, nr);
	cmask_set_range(&cap_mask, 0, nr);
	ret = scx_bpf_sub_grant(grant_cgid, grant_caps, &cap_mask, NULL);
	setcnt(K2_GRANT_RET, ret);
	return 0;
}

/* parent-side restore per the last-writer-wins ownership model: once
 * the sub is inert (caps cleared), the parent writes back neutral values */
void BPF_STRUCT_OPS(k2_sub_detach, struct scx_sub_detach_args *args)
{
	u32 i, nr;

	arena_touch();
	if (!restore_on_detach)
		return;

	nr = scx_bpf_nr_cids();
	bpf_for(i, 0, nr) {
		if (restore_on_detach == 2 || i == 0)
			scx_bpf_cidperf_set(i, SCX_CPUPERF_ONE);
	}
}

/* root: revoke PERF on tick revoke_at; sub: write target every tick */
void BPF_STRUCT_OPS(k2_tick, struct task_struct *p)
{
	s32 cid;

	if (is_sub) {
		cid = scx_bpf_this_cid();
		if (cid >= 0)
			setcnt(K2_SET_ERR,
			       scx_bpf_cidperf_set(cid, perf_target));
		bump(K2_SETCNT);
		return;
	}

	/* count only cpu0 root ticks: discriminates who owns the cpu0
	 * spinner (root ticks mean the spinner runs as a root task) */
	if (bpf_get_smp_processor_id() == 0)
		bump(K2_ROOTTICKS);
	setcnt(K2_DSQDEPTH, scx_bpf_dsq_nr_queued(SHARED_DSQ));
	arena_touch();
	k2_ticks++;
	if (revoke_at && k2_ticks == revoke_at) {
		u32 nr = scx_bpf_nr_cids();

		cmask_init(&cap_mask, 0, nr);
		cmask_set_range(&cap_mask, 0, nr);
		scx_bpf_sub_revoke(grant_cgid, SCX_CAP_PERF, &cap_mask);
		bump(K2_REVOKE_DONE);
	}
}

void BPF_STRUCT_OPS(k2_cid_online, s32 cpu)
{
	arena_touch();
}

void BPF_STRUCT_OPS(k2_cid_offline, s32 cpu)
{
	arena_touch();
}

s32 BPF_STRUCT_OPS(k2_init_task, struct task_struct *p,
		   struct scx_init_task_args *args)
{
	arena_touch();
	return 0;
}

s32 BPF_STRUCT_OPS_SLEEPABLE(k2_init)
{
	asm volatile("" :: "r"(&arena));	/* make the verifier claim the arena map */
	return scx_bpf_create_dsq(SHARED_DSQ, -1);
}

void BPF_STRUCT_OPS(k2_exit, struct scx_exit_info *ei)
{
	UEI_RECORD(uei, ei);
}

SCX_OPS_CID_DEFINE(k2_ops,
		   .select_cid		= (void *)k2_select_cid,
		   .enqueue		= (void *)k2_enqueue,
		   .dispatch		= (void *)k2_dispatch,
		   .sub_attach		= (void *)k2_sub_attach,
		   .sub_detach		= (void *)k2_sub_detach,
		   .tick		= (void *)k2_tick,
		   .cid_online		= (void *)k2_cid_online,
		   .cid_offline		= (void *)k2_cid_offline,
		   .init_task		= (void *)k2_init_task,
		   .init		= (void *)k2_init,
		   .exit		= (void *)k2_exit,
		   .name		= "k2");

[-- Attachment #4: scx_k2.c --]
[-- Type: text/x-csrc, Size: 4006 bytes --]

// SPDX-License-Identifier: GPL-2.0
/* scx_k2 userspace. Usage:
 *   root mode: scx_k2 [-g <sub-cgroup-dir>] [-c <caps>] [-R <ticks>] [-p <perf>]
 *   sub  mode: scx_k2 -s <my-cgroup-dir> [-p <perf>]
 * Prints all counters every 2s; SIGTERM exits (triggers sub disable =>
 * sweep path).
 */
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
#include <scx/common.h>

/* Mirror of kernel/sched/ext/types.h's arena cap mask: the arena global
 * cap_mask is mirrored to userspace via the skeleton (must be defined
 * before the skel include). The bits[] flexible array is omitted and a
 * __pad field pads the struct to the 16 bytes BTF reports (FAM alignment
 * padding) - userspace never accesses the object, and a FAM would
 * trigger -Wgnu-variable-sized-type-not-at-end. */
struct scx_cmask {
	u32 base;
	u32 nr_cids;
	u32 alloc_words;
	u32 __pad;
};

#include "scx_k2.bpf.skel.h"

static bool verbose;
static volatile int exit_req;

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
	if (level == LIBBPF_DEBUG && !verbose)
		return 0;
	return vfprintf(stderr, format, args);
}

static void sig_handler(int sig)
{
	exit_req = 1;
}

static const char *cnt_name(u32 i)
{
	static const char *names[] = { "grant_ret", "revoke_done", "setcnt", "set_err", "rootticks", "detach", "detach_ok", "dsqdepth", "disp0", "moved0", "subdisp0", "submoved0" };

	return names[i];
}

static u64 read_cnt(struct scx_k2 *skel, u32 idx)
{
	u64 v = 0;

	bpf_map_lookup_elem(bpf_map__fd(skel->maps.counters), &idx, &v);
	return v;
}

int main(int argc, char **argv)
{
	struct scx_k2 *skel;
	struct bpf_link *link;
	__s32 opt;
	u64 sub_cgid = 0, grant_cgid = 0, grant_caps = 0, revoke_at = 0;
	u32 restore_on_detach = 0;
	u32 perf_target = 1;

	libbpf_set_print(libbpf_print_fn);
	signal(SIGINT, sig_handler);
	signal(SIGTERM, sig_handler);

	skel = SCX_OPS_CID_OPEN(k2_ops, scx_k2);
	while ((opt = getopt(argc, argv, "s:g:c:R:e:p:v")) != -1) {
		struct stat st;

		switch (opt) {
		case 's':	/* sub mode, arg = own cgroup */
			if (stat(optarg, &st)) { perror(optarg); return 1; }
			sub_cgid = st.st_ino;
			break;
		case 'g':	/* root mode, arg = sub's cgroup */
			if (stat(optarg, &st)) { perror(optarg); return 1; }
			grant_cgid = st.st_ino;
			break;
		case 'c':
			grant_caps = strtoull(optarg, NULL, 0);
			break;
		case 'R':
			revoke_at = strtoull(optarg, NULL, 0);
			break;
		case 'e':	/* on sub_detach parent restores neutral values (1=cpu0 only, 2=all cids) */
			restore_on_detach = (u32)strtoul(optarg, NULL, 0);
			break;
		case 'p':
			perf_target = (u32)strtoul(optarg, NULL, 0);
			break;
		case 'v':
			verbose = true;
			break;
		default:
			fprintf(stderr, "scx_k2: -s <cg>|-g <cg> [-c caps] [-R ticks] [-e 1|2] [-p perf]\n");
			return opt != 'h';
		}
	}

	if (sub_cgid) {
		skel->struct_ops.k2_ops->sub_cgroup_id = sub_cgid;
		skel->rodata->sub_cgroup_id = sub_cgid;
		skel->rodata->is_sub = 1;
	}
	skel->rodata->grant_cgid = grant_cgid;
	/* no SCX_CAP_* enum in the userspace headers (kernel internal.h):
	 * ENQ_IMMED=BIT(0), PERF=BIT(3) */
	skel->rodata->grant_caps = grant_caps ? grant_caps : 0x9;
	skel->rodata->revoke_at = revoke_at;
	skel->rodata->restore_on_detach = restore_on_detach;
	skel->rodata->perf_target = perf_target;

	SCX_OPS_LOAD(skel, k2_ops, scx_k2, uei);
	link = SCX_OPS_ATTACH(skel, k2_ops, scx_k2);
	fprintf(stderr, "[k2] up: %s grant_cgid=%llu caps=0x%llx revoke_at=%llu perf=%u\n",
		sub_cgid ? "sub" : "root",
		(unsigned long long)grant_cgid,
		(unsigned long long)skel->rodata->grant_caps,
		(unsigned long long)revoke_at, perf_target);

	while (!exit_req) {
		u32 i;

		sleep(2);
		for (i = 0; i < 12; i++)
			fprintf(stderr, "[k2] %s=%llu%s", cnt_name(i),
				(unsigned long long)read_cnt(skel, i),
				i == 3 ? "\n" : " ");
	}

	bpf_link__destroy(link);
	scx_k2__destroy(skel);
	fprintf(stderr, "[k2] exited cleanly\n");
	return 0;
}

[-- Attachment #5: scx_perfprobe.bpf.c --]
[-- Type: text/x-csrc, Size: 1058 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * scx_perfprobe - samples per-CPU rq->scx.cpuperf_target (the value
 * schedutil consumes). Kprobes sched_tick and overwrites this CPU's
 * bucket on every tick (~1000/s at HZ=1000).
 */
#include <scx/common.bpf.h>

char _license[] SEC("license") = "GPL";

#define K2_MAX_CPUS 128

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__uint(max_entries, K2_MAX_CPUS);
	__type(key, u32);
	__type(value, u64);
} targets SEC(".maps");

/* per-CPU sched_tick hits: tells whether a CPU is really running ticks */
struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__uint(max_entries, K2_MAX_CPUS);
	__type(key, u32);
	__type(value, u64);
} tickcnt SEC(".maps");

SEC("kprobe/sched_tick")
int BPF_KPROBE(k2_sample_tick)
{
	struct rq *rq = bpf_this_cpu_ptr(&runqueues);
	/* map value is u64: read the target into a u64 so we don't pass
	 * a u32 address as an 8-byte value */
	u64 t = rq->scx.cpuperf_target;
	u32 cpu = bpf_get_smp_processor_id();

	if (cpu < K2_MAX_CPUS)
		bpf_map_update_elem(&targets, &cpu, &t, BPF_ANY);
	return 0;
}

[-- Attachment #6: scx_perfprobe.c --]
[-- Type: text/x-csrc, Size: 1647 bytes --]

// SPDX-License-Identifier: GPL-2.0
/* scx_perfprobe userspace - prints one line per second with every
 * CPU's rq->scx.cpuperf_target. */
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include <errno.h>
#include <bpf/bpf.h>
#include <scx/common.h>
#include "scx_perfprobe.bpf.skel.h"

#define K2_MAX_CPUS 128

static volatile int exit_req;

static void sig_handler(int sig)
{
	exit_req = 1;
}

int main(int argc, char **argv)
{
	struct scx_perfprobe *skel;
	struct bpf_link *link;
	long online = sysconf(_SC_NPROCESSORS_ONLN);
	unsigned long sec = 0;

	signal(SIGINT, sig_handler);
	signal(SIGTERM, sig_handler);

	skel = scx_perfprobe__open();
	if (!skel) {
		fprintf(stderr, "perfprobe: open failed\n");
		return 1;
	}
	if (scx_perfprobe__load(skel)) {
		fprintf(stderr, "perfprobe: load failed\n");
		return 1;
	}
	link = bpf_program__attach(skel->progs.k2_sample_tick);
	if (libbpf_get_error(link)) {
		fprintf(stderr, "perfprobe: attach failed\n");
		return 1;
	}

	while (!exit_req) {
		char line[1024];
		int n = 0, cpu;

		n += snprintf(line + n, sizeof(line) - n, "[T+%lus]", ++sec);
		for (cpu = 0; cpu < online && cpu < K2_MAX_CPUS; cpu++) {
			u64 v = 0;
			u32 key = cpu;

			bpf_map_lookup_elem(bpf_map__fd(skel->maps.targets),
					    &key, &v);
			u64 tk = 0;
			bpf_map_lookup_elem(bpf_map__fd(skel->maps.tickcnt),
					    &key, &tk);
			n += snprintf(line + n, sizeof(line) - n,
				      " c%d=%llu/t%llu", cpu,
				      (unsigned long long)v,
				      (unsigned long long)tk);
		}
		printf("%s\n", line);
		fflush(stdout);
		sleep(1);
	}

	bpf_link__destroy(link);
	scx_perfprobe__destroy(skel);
	return 0;
}