* [PATCHSET v4 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-15 9:00 Andrea Righi
2026-09-15 9:00 ` [PATCH 1/2] " Andrea Righi
2026-09-15 9:00 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 2 replies; 13+ messages in thread
From: Andrea Righi @ 2026-09-15 9:00 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags. They
expire the running sched_ext task's slice while allowing the scheduling boundary
to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_SLICE_EXPIRY selects lazy slice expiry by default. Schedulers can
override that default in either direction for individual tasks through
scx_bpf_task_set_slice_expiry(). The helper preserves sub-scheduler task
ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU, ensuring forward progress even when the
target's tick has stopped.
The selftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task slice-expiry overrides, invalid
flags and lazy preemption of infinite-slice tasks on NO_HZ_FULL CPUs.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- tools/testing/selftests/sched_ext/runner -t kick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
$ vng -a "preempt=lazy nohz_full=8-15" -- tools/testing/selftests/sched_ext/runner -t nohz_tick
nohz_tick:
CPU 8 received 7 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
Changes in v4:
- Preserve SCX_KICK_WAIT synchronization when it is combined with
SCX_KICK_PREEMPT_LAZY (Sashiko).
- Restore the scheduler tick dependency when ops.tick() lazily expires an
infinite slice, ensuring the request is promoted by a later tick (Sashiko).
- kselftest: verify that the deferred SCX_KICK_WAIT callback runs (Sashiko).
- kselftest: keep the kick test controller off the victim CPU and skip cleanly
when a second allowed CPU is unavailable (Sashiko).
- kselftest: select the nohz_tick controller from allowed housekeeping CPUs
outside the NOHZ_FULL mask and skip when the required topology is
unavailable.
- Link to v3: https://lore.kernel.org/r/20260914144620.2032614-1-arighi@nvidia.com
Changes in v3:
- Avoid redundant rq clock updates when restoring the tick dependency and use
the pinned rq lock helpers for lazy kick delivery (Sashiko).
- Set PR_SET_PDEATHSIG for the indefinitely spinning kick test workers so they
don't survive unexpected test-runner exit (Sashiko).
- Wait for a delta in nr_lazy_victim_running instead of an absolute count that
may include reschedules from the previous phase (Sashiko).
- Increase the nohz_tick scheduler watchdog and phase timeouts to tolerate
scheduling delays in oversubscribed test VMs.
- Link to v2: https://lore.kernel.org/r/20260914084955.1798562-1-arighi@nvidia.com
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any sched_ext
callback (preserving sub-scheduler task ownership).
- Restore the scheduler tick dependency for lazy enqueue and kick preemption of
infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve their
precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY, immediate preemption, WAIT
and plain kicks take precedence over lazy preemption (Tejun Heo).
- Clear both preemption masks together when processing or skipping a preemption
request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests, per-task
expiry overrides and NO_HZ_FULL coverage.
- Link to v1: https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 153 +++++-
kernel/sched/ext/internal.h | 39 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 13 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 3 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 6 +
tools/sched_ext/include/scx/enums.autogen.h | 2 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 5 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 182 +++++++
tools/testing/selftests/sched_ext/kick.c | 608 ++++++++++++++++++++++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 56 +-
tools/testing/selftests/sched_ext/nohz_tick.c | 249 ++++++++-
14 files changed, 1273 insertions(+), 67 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/kick.c
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 1/2] sched_ext: Add lazy preemption support 2026-09-15 9:00 [PATCHSET v4 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-15 9:00 ` Andrea Righi 2026-09-15 9:00 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 1 sibling, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-15 9:00 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, sched-ext, linux-kernel The fair scheduling class can request lazy rescheduling, deferring an in-kernel scheduling boundary until returning to user space or until the next scheduler tick. sched_ext only exposes immediate preemption, preventing BPF schedulers from making the same trade-off. Add SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY. Both expire the current sched_ext task slice but request lazy rescheduling. Immediate preemption, WAIT and plain kicks take precedence when requests are combined, while a lazy enqueue to a non-local DSQ retains the head-insertion semantics of SCX_ENQ_PREEMPT. Add SCX_OPS_LAZY_SLICE_EXPIRY as the default expiry policy for newly enabled tasks and initialize it before ops.enable(). Add scx_bpf_task_set_slice_expiry() so the owning scheduler can override the policy per task from any callback while preserving sub-scheduler task ownership boundaries. Bypass continues to force immediate expiry. Restore the scheduler tick dependency before lazily rescheduling a task whose infinite slice allowed a NO_HZ_FULL CPU to stop its tick. Accumulate kick requests independently and resolve precedence while holding the target rq lock. Reject unknown kick flags and invalid SCX_KICK_IDLE combinations. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- include/linux/sched/ext.h | 8 + kernel/sched/ext/ext.c | 153 ++++++++++++++---- kernel/sched/ext/internal.h | 39 ++++- kernel/sched/ext/sub.c | 15 +- tools/sched_ext/include/scx/compat.bpf.h | 13 ++ .../sched_ext/include/scx/enum_defs.autogen.h | 3 + .../sched_ext/include/scx/enums.autogen.bpf.h | 6 + tools/sched_ext/include/scx/enums.autogen.h | 2 + .../sched_ext/include/scx/enums_abi.autogen.h | 5 +- 9 files changed, 204 insertions(+), 40 deletions(-) diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 8de69843c2150..685f1846aa386 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -249,6 +249,14 @@ struct sched_ext_entity { */ u64 dsq_vtime; + /* + * If set, depletion of this task's slice at the scheduler tick requests + * lazy instead of immediate rescheduling. Initialized from + * %SCX_OPS_LAZY_SLICE_EXPIRY immediately before ops.enable() and may be + * modified afterwards with scx_bpf_task_set_slice_expiry(). + */ + bool slice_expires_lazy; + /* * Out-of-band slice request from scx_bpf_task_set_slice() when the * caller does not hold the rq lock, applied under the rq lock at the diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index 4080902a528cd..83752c076094c 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -383,11 +383,11 @@ static bool rq_is_open(struct rq *rq, u64 enq_flags) return true; /* - * %SCX_ENQ_PREEMPT clears $curr's slice if on SCX and kicks dispatch, - * so allow it to avoid spuriously triggering reenq on a combined + * The preemption flags clear $curr's slice if on SCX and kick dispatch, + * so allow them to avoid spuriously triggering reenq on a combined * PREEMPT|IMMED insertion. */ - if (enq_flags & SCX_ENQ_PREEMPT) { + if (enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) { struct task_struct *curr = rq->curr; /* @@ -1515,6 +1515,24 @@ static void call_task_dequeue(struct scx_sched *sch, struct rq *rq, p->scx.flags &= ~SCX_TASK_IN_CUSTODY; } +/* + * A task with an infinite slice may be running with its tick stopped. Lazy + * rescheduling doesn't send an IPI, so restore the tick dependency first to + * guarantee that the lazy request is promoted by a real scheduler tick. + */ +static void scx_resched_curr_lazy(struct rq *rq) +{ + if (rq->scx.flags & SCX_RQ_CAN_STOP_TICK) { + rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK; + if (rq->clock_update_flags < RQCF_UPDATED) + update_rq_clock(rq); + update_other_load_avgs(rq); + sched_update_tick_dependency(rq); + } + + resched_curr_lazy(rq); +} + static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq, struct scx_dispatch_q *dsq, struct task_struct *p, u64 enq_flags) @@ -1577,12 +1595,16 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq, if (rq->scx.flags & SCX_RQ_IN_DISPATCH) return; - if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr && + if ((enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) && p != rq->curr && rq->curr->sched_class == &ext_sched_class) { - if (likely(scx_set_task_slice(rq->curr, 0))) - resched_curr(rq); - else + if (likely(scx_set_task_slice(rq->curr, 0))) { + if (enq_flags & SCX_ENQ_PREEMPT) + resched_curr(rq); + else + scx_resched_curr_lazy(rq); + } else { __scx_add_event(sch, SCX_EV_SLICE_DENIED, 1); + } } } @@ -1672,7 +1694,7 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq, scx_error(sch, "DSQ ID 0x%016llx already had PRIQ-enqueued tasks", dsq->id); - if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT)) { + if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) { /* new task inserted at head - use fastpath */ if (dsq_insert_head(dsq, p) && !(dsq->id & SCX_DSQ_FLAG_BUILTIN)) rcu_assign_pointer(dsq->first_task, p); @@ -2386,7 +2408,7 @@ void scx_move_local_task_to_local_dsq(struct scx_sched *sch, struct task_struct WARN_ON_ONCE(p->scx.holding_cpu >= 0); - if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT)) + if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) dsq_insert_head(dst_dsq, p); else list_add_tail(&p->scx.dsq_list.node, &dst_dsq->list); @@ -3804,8 +3826,14 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued) else if (SCX_HAS_OP(sch, tick)) SCX_CALL_OP_TASK(sch, tick, rq, curr); - if (!curr->scx.slice) - resched_curr(rq); + if (!curr->scx.slice) { + /* the slice can't be trusted while bypassing */ + if (READ_ONCE(curr->scx.slice_expires_lazy) && + !scx_bypassing(sch, cpu_of(rq))) + scx_resched_curr_lazy(rq); + else + resched_curr(rq); + } } #ifdef CONFIG_EXT_GROUP_SCHED @@ -3921,6 +3949,7 @@ static void __scx_enable_task(struct scx_sched *sch, struct task_struct *p) weight = sched_prio_to_weight[p->static_prio - MAX_RT_PRIO]; p->scx.weight = sched_weight_to_cgroup(weight); + p->scx.slice_expires_lazy = sch->ops.flags & SCX_OPS_LAZY_SLICE_EXPIRY; if (SCX_HAS_OP(sch, enable)) SCX_CALL_OP_TASK(sch, enable, rq, p); @@ -5371,6 +5400,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work) free_cpumask_var(pcpu->cpus_to_kick); free_cpumask_var(pcpu->cpus_to_kick_if_idle); free_cpumask_var(pcpu->cpus_to_preempt); + free_cpumask_var(pcpu->cpus_to_preempt_lazy); free_cpumask_var(pcpu->cpus_to_wait); exit_dsq(scx_bypass_dsq(sch, cpu)); @@ -6883,6 +6913,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s, if (!cpumask_empty(pcpu->cpus_to_preempt)) scx_dump_line(&ns, " cpus_to_preempt: %*pb", cpumask_pr_args(pcpu->cpus_to_preempt)); + if (!cpumask_empty(pcpu->cpus_to_preempt_lazy)) + scx_dump_line(&ns, " preempt_lazy : %*pb", + cpumask_pr_args(pcpu->cpus_to_preempt_lazy)); if (!cpumask_empty(pcpu->cpus_to_wait)) scx_dump_line(&ns, " cpus_to_wait : %*pb", cpumask_pr_args(pcpu->cpus_to_wait)); @@ -7200,6 +7233,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, if (!zalloc_cpumask_var_node(&pcpu->cpus_to_kick, GFP_KERNEL, node) || !zalloc_cpumask_var_node(&pcpu->cpus_to_kick_if_idle, GFP_KERNEL, node) || !zalloc_cpumask_var_node(&pcpu->cpus_to_preempt, GFP_KERNEL, node) || + !zalloc_cpumask_var_node(&pcpu->cpus_to_preempt_lazy, GFP_KERNEL, node) || !zalloc_cpumask_var_node(&pcpu->cpus_to_wait, GFP_KERNEL, node)) { ret = -ENOMEM; goto err_free_pcpu; @@ -7335,6 +7369,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, free_cpumask_var(pcpu->cpus_to_kick); free_cpumask_var(pcpu->cpus_to_kick_if_idle); free_cpumask_var(pcpu->cpus_to_preempt); + free_cpumask_var(pcpu->cpus_to_preempt_lazy); free_cpumask_var(pcpu->cpus_to_wait); } for_each_possible_cpu(cpu) { @@ -7953,7 +7988,7 @@ static bool bpf_scx_is_valid_access(int off, int size, return btf_ctx_access(off, size, type, prog, info); } -/* common to both forms: only scx.disallow is writable */ +/* common to both forms: only the fields below are writable */ static int bpf_scx_btf_struct_access_common(const struct bpf_reg_state *reg, int off, int size) { @@ -7964,7 +7999,6 @@ static int bpf_scx_btf_struct_access_common(const struct bpf_reg_state *reg, off >= offsetof(struct task_struct, scx.disallow) && off + size <= offsetofend(struct task_struct, scx.disallow)) return SCALAR_VALUE; - return -EACCES; } @@ -8459,13 +8493,23 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r { struct rq *rq = cpu_rq(cpu); struct scx_rq *this_scx = &this_rq->scx; + struct rq_flags rf; const struct sched_class *cur_class; bool should_wait = false; bool kickable; - unsigned long flags; + bool preempt, preempt_lazy, wait, immediate; - raw_spin_rq_lock_irqsave(rq, flags); + rq_lock_irqsave(rq, &rf); cur_class = rq->curr->sched_class; + preempt = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt); + preempt_lazy = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt_lazy); + wait = cpumask_test_cpu(cpu, pcpu->cpus_to_wait); + /* + * Immediate preemption, waiting and a plain kick take precedence over + * lazy preemption. The lazy request still clears the slice, so all + * accumulated requests are served. + */ + immediate = preempt || wait || cpumask_test_cpu(cpu, pcpu->cpus_to_kick); /* * During CPU hotplug, a CPU may depend on kicking itself to make @@ -8479,19 +8523,23 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r !sched_class_above(cur_class, &ext_sched_class); if (kickable && !scx_missing_caps(pcpu->sch, cpu, SCX_CAP_BASE)) { - if (cpumask_test_cpu(cpu, pcpu->cpus_to_preempt)) { + if (preempt || preempt_lazy) { if (cur_class == &ext_sched_class) { u64 caps = scx_caps_for_preempt(pcpu->sch, rq, 0); - if (unlikely(scx_missing_caps(pcpu->sch, cpu, caps))) + if (unlikely(scx_missing_caps(pcpu->sch, cpu, caps))) { __scx_add_event(pcpu->sch, SCX_EV_SUB_PREEMPT_DENIED, 1); - else if (unlikely(!scx_set_task_slice(rq->curr, 0))) + /* degrade to a plain, immediate kick */ + immediate = true; + } else if (unlikely(!scx_set_task_slice(rq->curr, 0))) { __scx_add_event(pcpu->sch, SCX_EV_SLICE_DENIED, 1); + } } cpumask_clear_cpu(cpu, pcpu->cpus_to_preempt); + cpumask_clear_cpu(cpu, pcpu->cpus_to_preempt_lazy); } - if (cpumask_test_cpu(cpu, pcpu->cpus_to_wait)) { + if (wait) { if (cur_class == &ext_sched_class) { cpumask_set_cpu(cpu, this_scx->cpus_to_sync); ksyncs[cpu] = rq->scx.kick_sync; @@ -8500,17 +8548,21 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r cpumask_clear_cpu(cpu, pcpu->cpus_to_wait); } - resched_curr(rq); + if (immediate) + resched_curr(rq); + else + scx_resched_curr_lazy(rq); } else { /* a kickable cpu was skipped solely for the missing caps */ if (kickable) __scx_add_event(pcpu->sch, SCX_EV_SUB_KICK_DENIED, 1); cpumask_clear_cpu(cpu, pcpu->cpus_to_preempt); + cpumask_clear_cpu(cpu, pcpu->cpus_to_preempt_lazy); cpumask_clear_cpu(cpu, pcpu->cpus_to_wait); } scx_rq_lock_drop(rq); - raw_spin_rq_unlock_irqrestore(rq, flags); + rq_unlock_irqrestore(rq, &rf); return should_wait; } @@ -8566,6 +8618,14 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work) cpumask_clear_cpu(cpu, pcpu->cpus_to_kick_if_idle); } + /* + * kick_one_cpu() clears the lazy bit of every cpu it visited + * above; visit the remaining requests which contain lazy + * preemption, see scx_kick_cpu(). + */ + for_each_cpu(cpu, pcpu->cpus_to_preempt_lazy) + should_wait |= kick_one_cpu(cpu, pcpu, this_rq, ksyncs); + for_each_cpu(cpu, pcpu->cpus_to_kick_if_idle) { kick_one_cpu_if_idle(cpu, pcpu, this_rq); cpumask_clear_cpu(cpu, pcpu->cpus_to_kick_if_idle); @@ -9531,6 +9591,31 @@ __bpf_kfunc bool scx_bpf_task_set_dsq_vtime(struct task_struct *p, u64 vtime, return true; } +/** + * scx_bpf_task_set_slice_expiry - Set task's slice expiry policy + * @p: task of interest + * @lazy: whether slice expiry should request lazy rescheduling + * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs + * + * Choose whether depletion of @p's slice at the scheduler tick requests lazy + * or immediate rescheduling. @p must be on the calling scheduler. + * + * Return %true on success, %false if @p is not on the calling scheduler. + */ +__bpf_kfunc bool scx_bpf_task_set_slice_expiry(struct task_struct *p, bool lazy, + const struct bpf_prog_aux *aux) +{ + struct scx_sched *sch; + + guard(rcu)(); + sch = scx_prog_sched(aux); + if (unlikely(!sch || !scx_task_on_sched(sch, p))) + return false; + + WRITE_ONCE(p->scx.slice_expires_lazy, lazy); + return true; +} + void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags) { struct scx_sched_pcpu *pcpu; @@ -9539,6 +9624,16 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags) if (!scx_kf_allowed_ctx(sch)) return; + if (unlikely(flags & ~(SCX_KICK_IDLE | SCX_KICK_PREEMPT | SCX_KICK_WAIT | + SCX_KICK_PREEMPT_LAZY))) { + scx_error(sch, "invalid kick flags 0x%llx", flags); + return; + } + if (unlikely((flags & SCX_KICK_IDLE) && + (flags & (SCX_KICK_PREEMPT | SCX_KICK_PREEMPT_LAZY | SCX_KICK_WAIT)))) { + scx_error(sch, "PREEMPT/WAIT cannot be used with SCX_KICK_IDLE"); + return; + } local_irq_save(irq_flags); @@ -9564,9 +9659,6 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags) if (flags & SCX_KICK_IDLE) { struct rq *target_rq = cpu_rq(cpu); - if (unlikely(flags & (SCX_KICK_PREEMPT | SCX_KICK_WAIT))) - scx_error(sch, "PREEMPT/WAIT cannot be used with SCX_KICK_IDLE"); - if (raw_spin_rq_trylock(target_rq)) { if (can_skip_idle_kick(target_rq)) { scx_rq_lock_drop(target_rq); @@ -9578,12 +9670,15 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags) } cpumask_set_cpu(cpu, pcpu->cpus_to_kick_if_idle); } else { - cpumask_set_cpu(cpu, pcpu->cpus_to_kick); - + /* Accumulate requests and resolve their precedence at delivery. */ if (flags & SCX_KICK_PREEMPT) cpumask_set_cpu(cpu, pcpu->cpus_to_preempt); + if (flags & SCX_KICK_PREEMPT_LAZY) + cpumask_set_cpu(cpu, pcpu->cpus_to_preempt_lazy); if (flags & SCX_KICK_WAIT) cpumask_set_cpu(cpu, pcpu->cpus_to_wait); + if (!(flags & SCX_KICK_PREEMPT_LAZY)) + cpumask_set_cpu(cpu, pcpu->cpus_to_kick); } if (list_empty(&pcpu->to_kick_node)) @@ -9623,8 +9718,9 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux * cid-addressed equivalent of scx_bpf_kick_cpu(). An invalid @cid aborts the * scheduler via scx_cid_to_cpu(). Caps are enforced on the delivery path: a * kick is dropped if the caller lacks baseline access on @cid, and a - * %SCX_KICK_PREEMPT degrades to a plain reschedule if the caller lacks - * %SCX_CAP_PREEMPT for a task outside its subtree. + * %SCX_KICK_PREEMPT or %SCX_KICK_PREEMPT_LAZY request degrades to a plain + * reschedule if the caller lacks %SCX_CAP_PREEMPT for a task outside its + * subtree. */ __bpf_kfunc void scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) { @@ -10691,6 +10787,7 @@ __bpf_kfunc_end_defs(); BTF_KFUNCS_START(scx_kfunc_ids_any) BTF_ID_FLAGS(func, scx_bpf_task_set_slice, KF_IMPLICIT_ARGS | KF_RCU); BTF_ID_FLAGS(func, scx_bpf_task_set_dsq_vtime, KF_IMPLICIT_ARGS | KF_RCU); +BTF_ID_FLAGS(func, scx_bpf_task_set_slice_expiry, KF_IMPLICIT_ARGS | KF_RCU); BTF_ID_FLAGS(func, scx_bpf_kick_cpu, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_kick_cid, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued, KF_IMPLICIT_ARGS) diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h index e1eb3a0d456cb..d73b10d692d21 100644 --- a/kernel/sched/ext/internal.h +++ b/kernel/sched/ext/internal.h @@ -215,6 +215,19 @@ enum scx_ops_flags { */ SCX_OPS_TID_TO_TASK = 1LLU << 8, + /* + * If set, tasks default to requesting lazy rescheduling when their slice + * runs out at the tick, the way fair.c expires a slice from update_curr(). + * The default is copied to p->scx.slice_expires_lazy immediately before + * ops.enable(), after which scx_bpf_task_set_slice_expiry() may override + * it per task. + * A task in user space still reschedules on the way back from the tick; a + * task in the kernel runs on to its next return to user space or to the + * next tick, which promotes the request. No effect on kernels without + * lazy preemption. Rescheduling while disabling stays immediate. + */ + SCX_OPS_LAZY_SLICE_EXPIRY = 1LLU << 9, + SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE | SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING | @@ -223,7 +236,8 @@ enum scx_ops_flags { SCX_OPS_SWITCH_PARTIAL | SCX_OPS_BUILTIN_IDLE_PER_NODE | SCX_OPS_ALWAYS_ENQ_IMMED | - SCX_OPS_TID_TO_TASK, + SCX_OPS_TID_TO_TASK | + SCX_OPS_LAZY_SLICE_EXPIRY, /* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */ __SCX_OPS_INTERNAL_MASK = 0xffLLU << 56, @@ -1327,6 +1341,7 @@ struct scx_sched_pcpu { cpumask_var_t cpus_to_kick; cpumask_var_t cpus_to_kick_if_idle; cpumask_var_t cpus_to_preempt; + cpumask_var_t cpus_to_preempt_lazy; cpumask_var_t cpus_to_wait; struct list_head to_kick_node; @@ -1407,15 +1422,16 @@ struct scx_sched_pnode { * the allocation pattern. * * ENQ_IMMED insert an IMMED task onto the cid's local DSQ - * - kick the cid's cpu (except SCX_KICK_PREEMPT) + * - kick the cid's cpu (except SCX_KICK_PREEMPT and + * SCX_KICK_PREEMPT_LAZY) * * ENQ insert any task onto the cid's local DSQ (implies ENQ_IMMED) * * PREEMPT preempt any task running on the cid regardless of the owning * sched (implies ENQ). Preempting a task in the sched's own subtree * doesn't require any cap. - * - SCX_ENQ_PREEMPT inserts - * - SCX_KICK_PREEMPT kicks + * - SCX_ENQ_PREEMPT and SCX_ENQ_PREEMPT_LAZY inserts + * - SCX_KICK_PREEMPT and SCX_KICK_PREEMPT_LAZY kicks * * PERF control the cid's cpu power/perf management state, currently the * cpufreq target set through scx_bpf_cidperf_set(). Hardware @@ -1685,6 +1701,15 @@ enum scx_enq_flags { */ SCX_ENQ_PREEMPT = 1LLU << 32, + /* + * Like %SCX_ENQ_PREEMPT, but request lazy rescheduling. The current + * task's slice is still cleared immediately so that the next scheduling + * boundary observes the new ordering. %SCX_ENQ_PREEMPT takes precedence + * if both are specified. Implies %SCX_ENQ_HEAD, which is all it means on + * a non-local DSQ, as with %SCX_ENQ_PREEMPT. + */ + SCX_ENQ_PREEMPT_LAZY = 1LLU << 35, + /* * Only allowed on local DSQs. Guarantees that the task either gets * on the CPU immediately and stays on it, or gets reenqueued back @@ -1811,6 +1836,12 @@ enum scx_kick_flags { * is not on SCX. */ SCX_KICK_WAIT = 1LLU << 2, + + /* + * Like %SCX_KICK_PREEMPT, but request lazy rescheduling. If combined + * with %SCX_KICK_PREEMPT or %SCX_KICK_WAIT, rescheduling is immediate. + */ + SCX_KICK_PREEMPT_LAZY = 1LLU << 3, }; enum scx_tg_flags { diff --git a/kernel/sched/ext/sub.c b/kernel/sched/ext/sub.c index 380a5653dc529..a88b87614b55e 100644 --- a/kernel/sched/ext/sub.c +++ b/kernel/sched/ext/sub.c @@ -686,9 +686,10 @@ void scx_rescue_init(struct rq *rq) * rescue is enabled, or @rq's reject DSQ after recording the reenq reason on * @p. * - * %SCX_ENQ_IMMED, %SCX_ENQ_PREEMPT and %SCX_ENQ_HEAD are cleared when diverting - * to rescue or reject. %SCX_ENQ_PREEMPT is also cleared on a fallback - * migration-disabled admission. + * %SCX_ENQ_IMMED, %SCX_ENQ_PREEMPT, %SCX_ENQ_PREEMPT_LAZY and %SCX_ENQ_HEAD + * are cleared when diverting to rescue or reject. %SCX_ENQ_PREEMPT and + * %SCX_ENQ_PREEMPT_LAZY are also cleared on a fallback migration-disabled + * admission. * * Bypass doesn't need special-casing as a bypassing sched's tasks are enqueued * to and run by its nearest non-bypassing ancestor. If root is bypassing, it @@ -709,7 +710,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r * On a remote activation the scheduling sched (@asch) differs from * @p's owner (@sch). Check caps against the scheduling sched. */ - if (*enq_flags & SCX_ENQ_PREEMPT) + if (*enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) needed |= scx_caps_for_preempt(asch, rq, *enq_flags); missing = scx_missing_caps(asch, cpu_of(rq), needed); @@ -726,7 +727,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r if (unlikely(!scx_rq_online(rq) || is_migration_disabled(p) || p->migration_pending)) { __scx_add_event(sch, SCX_EV_SUB_FORCED_ADMIT, 1); - *enq_flags &= ~SCX_ENQ_PREEMPT; + *enq_flags &= ~(SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY); return &rq->scx.local_dsq; } @@ -735,8 +736,8 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r * or HEAD - a diversion has no priority and IMMED is not allowed on * non-local DSQs. Strip the enq and task flags along with the slice. */ - *enq_flags &= ~(SCX_ENQ_IMMED | SCX_ENQ_PREEMPT | SCX_ENQ_HEAD | - SCX_ENQ_APPLY_SLICE | SCX_ENQ_SLICE_DFL); + *enq_flags &= ~(SCX_ENQ_IMMED | SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY | + SCX_ENQ_HEAD | SCX_ENQ_APPLY_SLICE | SCX_ENQ_SLICE_DFL); p->scx.flags &= ~SCX_TASK_IMMED; /* the enqueuer opted for rescue instead of rejection and reenqueue */ diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index 6944221f96cc0..4a9bf1a4bb4a8 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -403,6 +403,19 @@ static inline void scx_bpf_task_set_dsq_vtime(struct task_struct *p, u64 vtime) p->scx.dsq_vtime = vtime; } +/* + * v7.4: scx_bpf_task_set_slice_expiry() added to enforce sub-scheduler task + * ownership. Preserve until v7.7. + */ +bool scx_bpf_task_set_slice_expiry___new(struct task_struct *p, bool lazy) __ksym __weak; + +static inline bool scx_bpf_task_set_slice_expiry(struct task_struct *p, bool lazy) +{ + if (bpf_ksym_exists(scx_bpf_task_set_slice_expiry___new)) + return scx_bpf_task_set_slice_expiry___new(p, lazy); + return false; +} + /* * v7.1: New scx_bpf_dsq_reenq() that allows re-enqueues on more DSQs. This * will eventually deprecate scx_bpf_reenqueue_local(). diff --git a/tools/sched_ext/include/scx/enum_defs.autogen.h b/tools/sched_ext/include/scx/enum_defs.autogen.h index 63b6b14b19bd4..f1554c22071ff 100644 --- a/tools/sched_ext/include/scx/enum_defs.autogen.h +++ b/tools/sched_ext/include/scx/enum_defs.autogen.h @@ -85,6 +85,7 @@ #define HAVE_SCX_ENQ_HEAD #define HAVE_SCX_ENQ_CPU_SELECTED #define HAVE_SCX_ENQ_PREEMPT +#define HAVE_SCX_ENQ_PREEMPT_LAZY #define HAVE_SCX_ENQ_IMMED #define HAVE_SCX_ENQ_RESCUE #define HAVE_SCX_ENQ_REENQ @@ -148,6 +149,7 @@ #define HAVE_SCX_KF_ALLOW_SELECT_CPU #define HAVE_SCX_KICK_IDLE #define HAVE_SCX_KICK_PREEMPT +#define HAVE_SCX_KICK_PREEMPT_LAZY #define HAVE_SCX_KICK_WAIT #define HAVE_SCX_OPI_BEGIN #define HAVE_SCX_OPI_NORMAL_BEGIN @@ -164,6 +166,7 @@ #define HAVE_SCX_OPS_BUILTIN_IDLE_PER_NODE #define HAVE_SCX_OPS_ALWAYS_ENQ_IMMED #define HAVE_SCX_OPS_TID_TO_TASK +#define HAVE_SCX_OPS_LAZY_SLICE_EXPIRY #define HAVE_SCX_OPS_ALL_FLAGS #define HAVE___SCX_OPS_INTERNAL_MASK #define HAVE_SCX_OPS_HAS_CPU_PREEMPT diff --git a/tools/sched_ext/include/scx/enums.autogen.bpf.h b/tools/sched_ext/include/scx/enums.autogen.bpf.h index 7268131010de3..2cec24beb2d80 100644 --- a/tools/sched_ext/include/scx/enums.autogen.bpf.h +++ b/tools/sched_ext/include/scx/enums.autogen.bpf.h @@ -109,6 +109,9 @@ const volatile u64 __SCX_KICK_IDLE __weak; const volatile u64 __SCX_KICK_PREEMPT __weak; #define SCX_KICK_PREEMPT __SCX_KICK_PREEMPT +const volatile u64 __SCX_KICK_PREEMPT_LAZY __weak; +#define SCX_KICK_PREEMPT_LAZY __SCX_KICK_PREEMPT_LAZY + const volatile u64 __SCX_KICK_WAIT __weak; #define SCX_KICK_WAIT __SCX_KICK_WAIT @@ -121,6 +124,9 @@ const volatile u64 __SCX_ENQ_HEAD __weak; const volatile u64 __SCX_ENQ_PREEMPT __weak; #define SCX_ENQ_PREEMPT __SCX_ENQ_PREEMPT +const volatile u64 __SCX_ENQ_PREEMPT_LAZY __weak; +#define SCX_ENQ_PREEMPT_LAZY __SCX_ENQ_PREEMPT_LAZY + const volatile u64 __SCX_ENQ_IMMED __weak; #define SCX_ENQ_IMMED __SCX_ENQ_IMMED diff --git a/tools/sched_ext/include/scx/enums.autogen.h b/tools/sched_ext/include/scx/enums.autogen.h index e616326545172..dd762ee0ddb36 100644 --- a/tools/sched_ext/include/scx/enums.autogen.h +++ b/tools/sched_ext/include/scx/enums.autogen.h @@ -40,10 +40,12 @@ SCX_ENUM_SET(skel, scx_ent_dsq_flags, SCX_TASK_DSQ_ON_PRIQ); \ SCX_ENUM_SET(skel, scx_kick_flags, SCX_KICK_IDLE); \ SCX_ENUM_SET(skel, scx_kick_flags, SCX_KICK_PREEMPT); \ + SCX_ENUM_SET(skel, scx_kick_flags, SCX_KICK_PREEMPT_LAZY); \ SCX_ENUM_SET(skel, scx_kick_flags, SCX_KICK_WAIT); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_WAKEUP); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_HEAD); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_PREEMPT); \ + SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_PREEMPT_LAZY); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_IMMED); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_RESCUE); \ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_REENQ); \ diff --git a/tools/sched_ext/include/scx/enums_abi.autogen.h b/tools/sched_ext/include/scx/enums_abi.autogen.h index d53899764f5ac..672c02a497d14 100644 --- a/tools/sched_ext/include/scx/enums_abi.autogen.h +++ b/tools/sched_ext/include/scx/enums_abi.autogen.h @@ -97,6 +97,7 @@ static const struct __scx_enum_abi_val __scx_enum_abi_vals[] { "scx_enq_flags", "SCX_ENQ_HEAD", 0x10000LLU }, { "scx_enq_flags", "SCX_ENQ_CPU_SELECTED", 0x100000LLU }, { "scx_enq_flags", "SCX_ENQ_PREEMPT", 0x100000000LLU }, + { "scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", 0x800000000LLU }, { "scx_enq_flags", "SCX_ENQ_IMMED", 0x200000000LLU }, { "scx_enq_flags", "SCX_ENQ_RESCUE", 0x400000000LLU }, { "scx_enq_flags", "SCX_ENQ_REENQ", 0x10000000000LLU }, @@ -160,6 +161,7 @@ static const struct __scx_enum_abi_val __scx_enum_abi_vals[] { "scx_kf_allow_flags", "SCX_KF_ALLOW_SELECT_CPU", 0x20LLU }, { "scx_kick_flags", "SCX_KICK_IDLE", 0x1LLU }, { "scx_kick_flags", "SCX_KICK_PREEMPT", 0x2LLU }, + { "scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", 0x8LLU }, { "scx_kick_flags", "SCX_KICK_WAIT", 0x4LLU }, { "scx_opi", "SCX_OPI_BEGIN", 0x0LLU }, { "scx_opi", "SCX_OPI_NORMAL_BEGIN", 0x0LLU }, @@ -176,7 +178,8 @@ static const struct __scx_enum_abi_val __scx_enum_abi_vals[] { "scx_ops_flags", "SCX_OPS_BUILTIN_IDLE_PER_NODE", 0x40LLU }, { "scx_ops_flags", "SCX_OPS_ALWAYS_ENQ_IMMED", 0x80LLU }, { "scx_ops_flags", "SCX_OPS_TID_TO_TASK", 0x100LLU }, - { "scx_ops_flags", "SCX_OPS_ALL_FLAGS", 0x1ffLLU }, + { "scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", 0x200LLU }, + { "scx_ops_flags", "SCX_OPS_ALL_FLAGS", 0x3ffLLU }, { "scx_ops_flags", "__SCX_OPS_INTERNAL_MASK", 0xff00000000000000LLU }, { "scx_ops_flags", "SCX_OPS_HAS_CPU_PREEMPT", 0x100000000000000LLU }, { "scx_ops_state", "SCX_OPSS_NONE", 0x0LLU }, -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-15 9:00 [PATCHSET v4 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-15 9:00 ` [PATCH 1/2] " Andrea Righi @ 2026-09-15 9:00 ` Andrea Righi 1 sibling, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-15 9:00 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF and the current task slice, while the stopping callback verifies that the task reaches a scheduling boundary. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the expiry default in both directions through scx_bpf_task_set_slice_expiry(). Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy enqueue and kick requests restart a stopped tick and make forward progress. Keep invalid kick flag coverage and skip modes not exposed by the running kernel. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 182 ++++++ tools/testing/selftests/sched_ext/kick.c | 608 ++++++++++++++++++ .../selftests/sched_ext/nohz_tick.bpf.c | 56 +- tools/testing/selftests/sched_ext/nohz_tick.c | 249 ++++++- 5 files changed, 1069 insertions(+), 27 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..5a64e19246004 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 challenger_pid; +const volatile s32 target_cpu; +const volatile s32 slice_expiry_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; +u64 nr_wait_callbacks; +bool enq_preempt_issued; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (p->pid == challenger_pid && + state == KICK_STATE_QUEUED) { + enq_preempt_issued = true; + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + return; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +} + +static void set_slice_expiry_override(struct task_struct *p) +{ + if (p->pid == victim_pid && slice_expiry_override >= 0) + scx_bpf_task_set_slice_expiry(p, slice_expiry_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_slice_expiry_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + if (scenario == ENQ_BOTH && !enq_preempt_issued) + return 0; + + slice_at_resched = BPF_CORE_READ(task, scx.slice); + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +SEC("fentry/kick_sync_wait_bal_cb") +int BPF_PROG(kick_wait_callback, struct rq *rq) +{ + if (scenario == KICK_LAZY_WAIT && + BPF_CORE_READ(rq, cpu) == target_cpu) + __sync_fetch_and_add(&nr_wait_callbacks, 1); + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..bd347c5051814 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,608 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <errno.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#include "kick.bpf.skel.h" +#include "scx_test.h" + +#define WAIT_LOOPS 3000 + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +struct victim { + pid_t pid; + int start_fd; +}; + +struct observation { + u64 slice_before; + u64 slice_at_resched; + u64 nr_wait_callbacks; + s32 resched_tif; +}; + +struct kick_ctx { + cpu_set_t original_mask; + int target_cpu; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static enum scx_test_status setup_controller(void **ctx_ptr) +{ + struct kick_ctx *ctx; + cpu_set_t controller_mask; + int cpu, controller_cpu = -1; + + ctx = calloc(1, sizeof(*ctx)); + SCX_FAIL_IF(!ctx, "Failed to allocate context"); + if (sched_getaffinity(0, sizeof(ctx->original_mask), + &ctx->original_mask)) { + free(ctx); + SCX_FAIL("Failed to get affinity (%d)", errno); + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &ctx->original_mask)) + continue; + if (controller_cpu < 0) { + controller_cpu = cpu; + } else { + ctx->target_cpu = cpu; + break; + } + } + if (cpu == CPU_SETSIZE) { + printf("SKIP: two allowed CPUs are required\n"); + free(ctx); + return SCX_TEST_SKIP; + } + + CPU_ZERO(&controller_mask); + CPU_SET(controller_cpu, &controller_mask); + if (sched_setaffinity(0, sizeof(controller_mask), &controller_mask)) { + free(ctx); + SCX_FAIL("Failed to pin controller to CPU %d (%d)", + controller_cpu, errno); + } + + *ctx_ptr = ctx; + return SCX_TEST_PASS; +} + +static void cleanup_controller(void *ctx_ptr) +{ + struct kick_ctx *ctx = ctx_ptr; + + sched_setaffinity(0, sizeof(ctx->original_mask), &ctx->original_mask); + free(ctx); +} + +static struct victim spawn_victim(int cpu) +{ + struct victim victim = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return victim; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return victim; + } + + victim.pid = fork(); + if (!victim.pid) { + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (victim.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return victim; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(victim.pid, SIGKILL); + waitpid(victim.pid, NULL, 0); + victim.pid = -1; + return victim; + } + close(ready[0]); + victim.start_fd = start[1]; + return victim; +} + +static void stop_victim(struct victim *victim) +{ + if (victim->start_fd >= 0) + close(victim->start_fd); + if (victim->pid > 0) { + kill(victim->pid, SIGKILL); + waitpid(victim->pid, NULL, 0); + } +} + +static bool start_victim(struct victim *victim) +{ + char byte = 1; + + if (write(victim->start_fd, &byte, 1) != 1) + return false; + close(victim->start_fd); + victim->start_fd = -1; + return true; +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(struct kick_ctx *ctx, u32 scenario, + u64 ops_flags, + s32 slice_expiry_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim challenger = { .pid = -1, .start_fd = -1 }; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + + victim = spawn_victim(cpu); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = spawn_victim(cpu); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->challenger_pid = challenger.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->slice_expiry_override = slice_expiry_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!start_victim(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!start_victim(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->nr_wait_callbacks = skel->bss->nr_wait_callbacks; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + stop_victim(&challenger); + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static int active_lazy_mode(void) +{ + char buf[128]; + FILE *file; + + file = fopen("/sys/kernel/debug/sched/preempt", "r"); + if (!file) + return -1; + if (!fgets(buf, sizeof(buf), file)) { + fclose(file); + return -1; + } + fclose(file); + + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(full)")) + return 0; + return -1; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY")) { + printf("SKIP: SCX_OPS_LAZY_SLICE_EXPIRY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + return setup_lazy(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &obs); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + enum scx_test_status status; + int lazy_mode; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY, 0, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY_THEN_IMMEDIATE, 0, -1, + &lazy_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + status = trace_one(ctx, KICK_IMMEDIATE_THEN_LAZY, 0, -1, + &immediate_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + status = trace_one(ctx, KICK_PLAIN_THEN_LAZY, 0, -1, + &plain_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + status = trace_one(ctx, KICK_LAZY_THEN_PLAIN, 0, -1, + &lazy_then_plain); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + status = trace_one(ctx, KICK_BOTH, 0, -1, &both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + status = trace_one(ctx, KICK_LAZY_WAIT, 0, -1, &lazy_wait); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + SCX_GT(lazy_wait.nr_wait_callbacks, 0); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + status = trace_one(ctx, ENQ_BOTH, 0, -1, &enq_both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_SLICE_EXPIRY, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + enum scx_test_status status; + u64 lazy_flag; + int lazy_mode; + bool found; + + found = __COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", + &lazy_flag); + SCX_ASSERT(found); + status = trace_one(ctx, TICK_EXPIRY, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + status = trace_one(ctx, TICK_EXPIRY, 0, 1, &force_lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_lazy)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, 0, + &force_immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_immediate)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(struct kick_ctx *ctx, u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + int i; + + victim = spawn_victim(cpu); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !start_victim(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + enum scx_test_status status; + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; + scenario <= INVALID_KICK_UNKNOWN; scenario++) { + status = invalid_one(ctx, scenario); + SCX_EQ(status, SCX_TEST_PASS); + } + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, + .cleanup = cleanup_controller, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..96ba1524ca384 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -9,10 +9,23 @@ char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; UEI_DEFINE(uei); @@ -24,9 +37,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,15 +71,22 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() == test_cpu && + phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); } @@ -61,5 +103,5 @@ struct sched_ext_ops nohz_tick_ops = { .tick = (void *)nohz_tick_tick, .exit = (void *)nohz_tick_exit, .name = "nohz_tick", - .timeout_ms = 1000U, + .timeout_ms = 5000U, }; diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..acc6510e239ad 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -28,12 +28,26 @@ #endif #define MIN_FINITE_TICKS 3 -#define PHASE_TIMEOUT_MS 1000 +#define PHASE_TIMEOUT_MS 5000 struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool lazy_supported; +}; + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +struct gated_worker { + pid_t pid; + int start_fd; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -47,20 +61,21 @@ static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) return -1; } -static int find_nohz_full_cpu(const cpu_set_t *allowed) +static int read_nohz_full_mask(cpu_set_t *mask) { char buf[4096], *cur, *end; FILE *file; + int ret = 0; file = fopen("/sys/devices/system/cpu/nohz_full", "r"); if (!file) - return -1; + return -errno; if (!fgets(buf, sizeof(buf), file)) { - fclose(file); - return -1; + ret = ferror(file) ? -errno : -EINVAL; + goto out; } - fclose(file); + CPU_ZERO(mask); cur = buf; while (*cur) { long first, last; @@ -73,25 +88,30 @@ static int find_nohz_full_cpu(const cpu_set_t *allowed) errno = 0; first = strtol(cur, &end, 10); - if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) - return -1; + if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) { + ret = -EINVAL; + goto out; + } cur = end; last = first; if (*cur == '-') { cur++; errno = 0; last = strtol(cur, &end, 10); - if (errno || end == cur || last < first) - return -1; + if (errno || end == cur || last < first) { + ret = -EINVAL; + goto out; + } cur = end; } - cpu = first_allowed_cpu(allowed, first, last); - if (cpu >= 0) - return cpu; + for (cpu = first; cpu <= last && cpu < CPU_SETSIZE; cpu++) + CPU_SET(cpu, mask); } - return -1; +out: + fclose(file); + return ret; } static pid_t start_worker(int cpu) @@ -132,6 +152,87 @@ static void stop_worker(pid_t pid) waitpid(pid, NULL, 0); } +static struct gated_worker spawn_gated_worker(int cpu) +{ + struct gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +static bool start_gated_worker(struct gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +static void stop_gated_worker(struct gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + stop_worker(worker->pid); + worker->pid = -1; +} + static int pause_worker(pid_t pid) { int status; @@ -162,8 +263,9 @@ static bool wait_for_counter(const u64 *counter, u64 value, int timeout_ms) static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; - cpu_set_t controller_mask; - int cpu; + cpu_set_t controller_mask, nohz_full_mask; + u64 enum_value; + int cpu, i, ret; ctx = calloc(1, sizeof(*ctx)); SCX_FAIL_IF(!ctx, "Failed to allocate context"); @@ -173,15 +275,27 @@ static enum scx_test_status setup(void **ctx_ptr) SCX_FAIL("Failed to get affinity (%d)", errno); } - cpu = find_nohz_full_cpu(&ctx->original_mask); - if (cpu < 0) { + ret = read_nohz_full_mask(&nohz_full_mask); + if (ret) { + fprintf(stderr, "SKIP: failed to read NOHZ_FULL mask (%d)\n", ret); + free(ctx); + return SCX_TEST_SKIP; + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &ctx->original_mask) && + CPU_ISSET(cpu, &nohz_full_mask)) + break; + if (cpu == CPU_SETSIZE) { fprintf(stderr, "SKIP: no allowed NOHZ_FULL CPU\n"); free(ctx); return SCX_TEST_SKIP; } controller_mask = ctx->original_mask; - CPU_CLR(cpu, &controller_mask); + for (i = 0; i < CPU_SETSIZE; i++) + if (CPU_ISSET(i, &nohz_full_mask)) + CPU_CLR(i, &controller_mask); if (CPU_COUNT(&controller_mask) == 0) { fprintf(stderr, "SKIP: no housekeeping CPU available\n"); free(ctx); @@ -189,6 +303,13 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + ctx->lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -220,11 +341,15 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; u64 finite_running; u64 finite_ticks; + u64 victim_running; int ret; link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops); @@ -260,7 +385,8 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); @@ -308,7 +434,87 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->lazy_supported) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + usleep(100000); + + if (!start_gated_worker(&challenger) || + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&challenger); + stop_gated_worker(&victim); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + trigger = spawn_gated_worker(ctx->housekeeping_cpu); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + victim_running = __atomic_load_n(&skel->bss->nr_lazy_victim_running, + __ATOMIC_RELAXED); + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, + victim_running + 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + usleep(100000); + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!start_gated_worker(&trigger) || + !wait_for_counter(&skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +527,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); stop_worker(finite_worker); stop_worker(inf_worker); if (link) -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCHSET v7 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-18 17:13 Andrea Righi
2026-09-18 17:13 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2026-09-18 17:13 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, Cheng-Yang Chou, Tao Cui, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags.
They expire the running sched_ext task's slice while allowing the scheduling
boundary to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_RESCHED selects lazy rescheduling on slice expiry by default.
Schedulers can override that persistent policy in either direction for
individual tasks through scx_bpf_task_set_lazy_resched(). Unlike the one-shot
SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY operations, the kfunc does not
request rescheduling itself; it controls what happens when a later scheduler
tick finds the task's slice depleted. A compatibility wrapper makes the call a
no-op on older kernels that lack the kfunc. The helper preserves sub-scheduler
task ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU. The reschedule request is set first so
that the dependency update's IPI can serve it on return to user space, while the
restored tick guarantees eventual promotion otherwise.
The selftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task lazy rescheduling policy
overrides, invalid flags, and lazy preemption of infinite-slice tasks on
NO_HZ_FULL CPUs. The NO_HZ_FULL cases verify that the tick has stopped before
issuing the lazy request.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- \
tools/testing/selftests/sched_ext/runner -t kick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
$ vng --cpus 4 -a "nohz_full=1-3" -- \
tools/testing/selftests/sched_ext/runner -t nohz_tick
nohz_tick:
CPU 1 received 6 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
The mode-sensitive kick_lazy and kick_tick cases also passed after switching
/sys/kernel/debug/sched/preempt between lazy and full. Tejun Heo additionally
reported successful vng runs in lazy and full modes, with nohz_full on CPUs 2-3
and on CPUs 1-3.
Tao Cui tested v6 in a 4-CPU KVM guest. Lazy and immediate wakeup preemption
both averaged 59 usecs, compared to 60 usecs for CFS, over 2,000 iterations.
Lazy slice expiry reduced the context-switch rate of six busy hogs from about
1,000/s with CFS to about 215/s.
Changes in v7:
- Fold normal and lazy kick processing into one CPU loop, avoiding a second rq
lock and clearing idle-kick state for lazy-only targets (Tejun Heo).
- Rename scx_bpf_task_set_slice_expiry() to scx_bpf_task_set_lazy_resched(),
together with its task field and default ops flag (Tejun Heo).
- Clarify that scx_bpf_task_set_lazy_resched() sets a persistent per-task
policy, while SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY are one-shot
requests.
- Add SCX_KICK_ALL_FLAGS and simplify validation of SCX_KICK_IDLE combinations
(Tejun Heo).
- Fix the compatibility-wrapper comment and separate the ops flag documentation
paragraphs (Tejun Heo).
- Share test enums between BPF and userspace, consolidate gated-worker helpers
in util.c and remove an unused generated configuration include (Tejun Heo).
- Detect static preemption modes through /proc/config.gz when debugfs does not
expose sched/preempt, and skip mode-sensitive assertions when the active mode
cannot be determined (Tejun Heo).
- Restrict coalescing, per-task override and lazy NO_HZ_FULL checks to kernels
running with lazy preemption enabled (Tejun Heo).
- Stop infinite-slice victims before their challengers and check the scheduler
exit state while polling (Tejun Heo).
- Run the NO_HZ_FULL lazy phases with SCX_OPS_LAZY_RESCHED so that a remote
tick cannot turn the test request into immediate preemption (Tejun Heo).
- Link to v6:
https://lore.kernel.org/r/20260917070751.3617935-1-arighi@nvidia.com
Changes in v6:
- Pack slice_expires_lazy next to scx.disallow instead of growing
task_struct by eight bytes (Tejun Heo).
- Correct the SCX_OPS_LAZY_SLICE_EXPIRY documentation for
PREEMPT_DYNAMIC and bypass (Tejun Heo).
- Set the lazy reschedule request before restoring the tick dependency,
so the dependency update's IPI can serve it on return to user space
(Tejun Heo).
- Explain the pinned rq locking and the newly rejected unknown kick
flags in the patch description (Tejun Heo).
- Regenerate the enum headers in definition order (Tejun Heo).
- kselftest: use SCX_OPS_ENQ_LAST to drive kick-test victims through
ops.stopping() (Tejun Heo).
- kselftest: poll for the SCX_KICK_WAIT callback after ops.stopping()
(Tejun Heo).
- kselftest: skip trace-based tests when their BTF targets are
unavailable (Tejun Heo).
- kselftest: verify that the NO_HZ_FULL tick has stopped before testing
lazy enqueue and kick requests (Tejun Heo).
- Link to v5:
https://lore.kernel.org/r/20260915194611.2674127-1-arighi@nvidia.com
Changes in v5:
- kselftest: detect static lazy preemption when CONFIG_PREEMPT_DYNAMIC
is disabled and the sched/preempt debugfs file is unavailable
(Cheng-Yang Chou).
- Link to v4:
https://lore.kernel.org/r/20260915090127.2321020-1-arighi@nvidia.com
Changes in v4:
- Preserve SCX_KICK_WAIT synchronization when it is combined with
SCX_KICK_PREEMPT_LAZY (Sashiko).
- Restore the scheduler tick dependency when ops.tick() lazily expires
an infinite slice, ensuring the request is promoted by a later tick
(Sashiko).
- kselftest: verify that the deferred SCX_KICK_WAIT callback runs
(Sashiko).
- kselftest: keep the kick test controller off the victim CPU and skip
cleanly when a second allowed CPU is unavailable (Sashiko).
- kselftest: select the nohz_tick controller from allowed housekeeping
CPUs outside the NOHZ_FULL mask and skip when the required topology
is unavailable.
- Link to v3:
https://lore.kernel.org/r/20260914144620.2032614-1-arighi@nvidia.com
Changes in v3:
- Avoid redundant rq clock updates when restoring the tick dependency
and use the pinned rq lock helpers for lazy kick delivery (Sashiko).
- Set PR_SET_PDEATHSIG for the indefinitely spinning kick test workers
so they don't survive unexpected test-runner exit (Sashiko).
- Wait for a delta in nr_lazy_victim_running instead of an absolute
count that may include reschedules from the previous phase (Sashiko).
- Increase the nohz_tick scheduler watchdog and phase timeouts to
tolerate scheduling delays in oversubscribed test VMs.
- Link to v2:
https://lore.kernel.org/r/20260914084955.1798562-1-arighi@nvidia.com
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any
sched_ext callback while preserving sub-scheduler task ownership.
- Restore the scheduler tick dependency for lazy enqueue and kick
preemption of infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve
their precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY; immediate
preemption, WAIT and plain kicks take precedence over lazy preemption
(Tejun Heo).
- Clear both preemption masks together when processing or skipping a
preemption request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests,
per-task expiry overrides and NO_HZ_FULL coverage.
- Link to v1:
https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 151 +++++-
kernel/sched/ext/internal.h | 44 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 9 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 4 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 9 +
tools/sched_ext/include/scx/enums.autogen.h | 3 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 6 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 170 +++++++
tools/testing/selftests/sched_ext/kick.c | 531 +++++++++++++++++++++
tools/testing/selftests/sched_ext/kick_test.h | 29 ++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 57 ++-
tools/testing/selftests/sched_ext/nohz_tick.c | 220 ++++++++-
tools/testing/selftests/sched_ext/nohz_tick_test.h | 13 +
tools/testing/selftests/sched_ext/util.c | 144 ++++++
tools/testing/selftests/sched_ext/util.h | 13 +
18 files changed, 1357 insertions(+), 70 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/kick.c
create mode 100644 tools/testing/selftests/sched_ext/kick_test.h
create mode 100644 tools/testing/selftests/sched_ext/nohz_tick_test.h
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-18 17:13 [PATCHSET v7 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-18 17:13 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-18 17:13 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, Cheng-Yang Chou, Tao Cui, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF after the current task's slice is cleared. SCX_OPS_ENQ_LAST drives the victim through ops.stopping() without relying on an unrelated wakeup. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the persistent policy in both directions through scx_bpf_task_set_lazy_resched(), and add invalid kick flag coverage. Extend the NO_HZ_FULL test with infinite-slice victims. Wait until the scheduler tick is observed to stop, then verify that lazy enqueue and kick requests restart it and make forward progress. Move the controller off all NO_HZ_FULL CPUs and increase the phase and watchdog timeouts to tolerate scheduling delays in oversubscribed test VMs. Detect both dynamic and static kernel preemption modes and skip mode-sensitive assertions when the active mode is unavailable. Skip trace-based cases when their kernel BTF targets are unavailable, while allowing the independent invalid-flag cases to run. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 170 ++++++ tools/testing/selftests/sched_ext/kick.c | 531 ++++++++++++++++++ tools/testing/selftests/sched_ext/kick_test.h | 29 + .../selftests/sched_ext/nohz_tick.bpf.c | 57 +- tools/testing/selftests/sched_ext/nohz_tick.c | 220 +++++++- .../selftests/sched_ext/nohz_tick_test.h | 13 + tools/testing/selftests/sched_ext/util.c | 144 +++++ tools/testing/selftests/sched_ext/util.h | 13 + 9 files changed, 1145 insertions(+), 33 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c create mode 100644 tools/testing/selftests/sched_ext/kick_test.h create mode 100644 tools/testing/selftests/sched_ext/nohz_tick_test.h diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..cb71905d975e4 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +#include "kick_test.h" + +char _license[] SEC("license") = "GPL"; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 challenger_pid; +const volatile s32 target_cpu; +const volatile s32 lazy_resched_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; +u64 nr_wait_callbacks; +bool enq_preempt_issued; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (p->pid == challenger_pid && + state == KICK_STATE_QUEUED) { + enq_preempt_issued = true; + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + goto kick_last; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +kick_last: + if (enq_flags & SCX_ENQ_LAST) + scx_bpf_kick_cpu(target_cpu, SCX_KICK_IDLE); +} + +static void set_lazy_resched_override(struct task_struct *p) +{ + if (p->pid == victim_pid && lazy_resched_override >= 0) + scx_bpf_task_set_lazy_resched(p, lazy_resched_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_lazy_resched_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + u64 slice; + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + if (scenario == ENQ_BOTH && !enq_preempt_issued) + return 0; + + slice = BPF_CORE_READ(task, scx.slice); + if (slice) + return 0; + + slice_at_resched = slice; + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +SEC("fentry/kick_sync_wait_bal_cb") +int BPF_PROG(kick_wait_callback, struct rq *rq) +{ + if (scenario == KICK_LAZY_WAIT && + BPF_CORE_READ(rq, cpu) == target_cpu) + __sync_fetch_and_add(&nr_wait_callbacks, 1); + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .flags = SCX_OPS_ENQ_LAST, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..0c7d14194211a --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,531 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <errno.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#include "kick.bpf.skel.h" +#include "kick_test.h" +#include "scx_test.h" +#include "util.h" + +#define WAIT_LOOPS 3000 + +struct observation { + u64 slice_before; + u64 slice_at_resched; + u64 nr_wait_callbacks; + s32 resched_tif; +}; + +struct kick_ctx { + cpu_set_t original_mask; + int target_cpu; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static enum scx_test_status setup_controller(void **ctx_ptr) +{ + struct kick_ctx *ctx; + cpu_set_t controller_mask; + int cpu, controller_cpu = -1; + + ctx = calloc(1, sizeof(*ctx)); + SCX_FAIL_IF(!ctx, "Failed to allocate context"); + if (sched_getaffinity(0, sizeof(ctx->original_mask), + &ctx->original_mask)) { + free(ctx); + SCX_FAIL("Failed to get affinity (%d)", errno); + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &ctx->original_mask)) + continue; + if (controller_cpu < 0) { + controller_cpu = cpu; + } else { + ctx->target_cpu = cpu; + break; + } + } + if (cpu == CPU_SETSIZE) { + printf("SKIP: two allowed CPUs are required\n"); + free(ctx); + return SCX_TEST_SKIP; + } + + CPU_ZERO(&controller_mask); + CPU_SET(controller_cpu, &controller_mask); + if (sched_setaffinity(0, sizeof(controller_mask), &controller_mask)) { + free(ctx); + SCX_FAIL("Failed to pin controller to CPU %d (%d)", + controller_cpu, errno); + } + + *ctx_ptr = ctx; + return SCX_TEST_PASS; +} + +static void cleanup_controller(void *ctx_ptr) +{ + struct kick_ctx *ctx = ctx_ptr; + + sched_setaffinity(0, sizeof(ctx->original_mask), &ctx->original_mask); + free(ctx); +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static bool wait_for_counter(struct kick *skel, const u64 *counter, u64 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(counter, __ATOMIC_ACQUIRE) >= wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(struct kick_ctx *ctx, u32 scenario, + u64 ops_flags, + s32 lazy_resched_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct scx_test_gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct scx_test_gated_worker victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + + victim = scx_test_spawn_gated_worker(cpu, false); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = scx_test_spawn_gated_worker(cpu, false); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->challenger_pid = challenger.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->lazy_resched_override = lazy_resched_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (scenario != KICK_LAZY_WAIT) + bpf_program__set_autoload(skel->progs.kick_wait_callback, + false); + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!scx_test_start_gated_worker(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!scx_test_start_gated_worker(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + if (scenario == KICK_LAZY_WAIT && + !wait_for_counter(skel, &skel->bss->nr_wait_callbacks, 1)) { + SCX_ERR("WAIT callback did not run"); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->nr_wait_callbacks = skel->bss->nr_wait_callbacks; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + scx_test_stop_gated_worker(&victim); + scx_test_stop_gated_worker(&challenger); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static bool trace_target_supported(const char *name, + enum bpf_attach_type attach_type) +{ + return libbpf_find_vmlinux_btf_id(name, attach_type) > 0; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + if (!trace_target_supported("__resched_curr", BPF_TRACE_FEXIT)) { + printf("SKIP: __resched_curr is not available in BTF\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_RESCHED")) { + printf("SKIP: SCX_OPS_LAZY_RESCHED is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + if (!trace_target_supported("kick_sync_wait_bal_cb", BPF_TRACE_FENTRY)) { + printf("SKIP: kick_sync_wait_bal_cb is not available in BTF\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &obs); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + enum scx_test_status status; + int lazy_mode = scx_test_preempt_lazy_mode(); + + if (lazy_mode < 0) { + printf("SKIP: running kernel preemption mode is unavailable\n"); + return SCX_TEST_SKIP; + } + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY, 0, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + enum scx_test_status status; + + if (scx_test_preempt_lazy_mode() != 1) { + printf("SKIP: kick coalescing requires active lazy preemption\n"); + return SCX_TEST_SKIP; + } + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY_THEN_IMMEDIATE, 0, -1, + &lazy_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + status = trace_one(ctx, KICK_IMMEDIATE_THEN_LAZY, 0, -1, + &immediate_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + status = trace_one(ctx, KICK_PLAIN_THEN_LAZY, 0, -1, + &plain_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + status = trace_one(ctx, KICK_LAZY_THEN_PLAIN, 0, -1, + &lazy_then_plain); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + status = trace_one(ctx, KICK_BOTH, 0, -1, &both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + status = trace_one(ctx, KICK_LAZY_WAIT, 0, -1, &lazy_wait); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + SCX_GT(lazy_wait.nr_wait_callbacks, 0); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + status = trace_one(ctx, ENQ_BOTH, 0, -1, &enq_both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_RESCHED, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + enum scx_test_status status; + u64 lazy_flag; + int lazy_mode = scx_test_preempt_lazy_mode(); + bool found; + + if (lazy_mode < 0) { + printf("SKIP: running kernel preemption mode is unavailable\n"); + return SCX_TEST_SKIP; + } + + found = __COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_RESCHED", + &lazy_flag); + SCX_ASSERT(found); + status = trace_one(ctx, TICK_EXPIRY, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + if (lazy_mode > 0) { + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + + status = trace_one(ctx, TICK_EXPIRY, 0, 1, &force_lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_lazy)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, 0, + &force_immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_immediate)); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + } else { + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + } + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(struct kick_ctx *ctx, u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct scx_test_gated_worker victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + int i; + + victim = scx_test_spawn_gated_worker(cpu, false); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + bpf_program__set_autoload(skel->progs.kick_need_resched, false); + bpf_program__set_autoload(skel->progs.kick_wait_callback, false); + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !scx_test_start_gated_worker(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + scx_test_stop_gated_worker(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + enum scx_test_status status; + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; + scenario <= INVALID_KICK_UNKNOWN; scenario++) { + status = invalid_one(ctx, scenario); + SCX_EQ(status, SCX_TEST_PASS); + } + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, + .cleanup = cleanup_controller, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/kick_test.h b/tools/testing/selftests/sched_ext/kick_test.h new file mode 100644 index 0000000000000..7d3c608c6b630 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick_test.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __KICK_TEST_H__ +#define __KICK_TEST_H__ + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +#endif /* __KICK_TEST_H__ */ diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..5fec5673fbf26 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -6,13 +6,22 @@ */ #include <scx/common.bpf.h> +#include "nohz_tick_test.h" + char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; +u64 nr_lazy_ticks; UEI_DEFINE(uei); @@ -24,9 +33,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,16 +67,28 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() != test_cpu) + return; + + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); + else if ((phase == NOHZ_PHASE_LAZY_ENQ || + phase == NOHZ_PHASE_LAZY_KICK) && p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_ticks, 1); } void BPF_STRUCT_OPS(nohz_tick_exit, struct scx_exit_info *ei) @@ -61,5 +104,5 @@ struct sched_ext_ops nohz_tick_ops = { .tick = (void *)nohz_tick_tick, .exit = (void *)nohz_tick_exit, .name = "nohz_tick", - .timeout_ms = 1000U, + .timeout_ms = 5000U, }; diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..53f1e8d92508b 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -21,19 +21,24 @@ #include <scx/common.h> #include "nohz_tick.bpf.skel.h" +#include "nohz_tick_test.h" #include "scx_test.h" +#include "util.h" #ifndef SCHED_EXT #define SCHED_EXT 7 #endif #define MIN_FINITE_TICKS 3 -#define PHASE_TIMEOUT_MS 1000 +#define PHASE_TIMEOUT_MS 5000 +#define TICK_STOP_STABLE_MS 100 struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool test_lazy; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -47,20 +52,21 @@ static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) return -1; } -static int find_nohz_full_cpu(const cpu_set_t *allowed) +static int read_nohz_full_mask(cpu_set_t *mask) { char buf[4096], *cur, *end; FILE *file; + int ret = 0; file = fopen("/sys/devices/system/cpu/nohz_full", "r"); if (!file) - return -1; + return -errno; if (!fgets(buf, sizeof(buf), file)) { - fclose(file); - return -1; + ret = ferror(file) ? -errno : -EINVAL; + goto out; } - fclose(file); + CPU_ZERO(mask); cur = buf; while (*cur) { long first, last; @@ -73,25 +79,30 @@ static int find_nohz_full_cpu(const cpu_set_t *allowed) errno = 0; first = strtol(cur, &end, 10); - if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) - return -1; + if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) { + ret = -EINVAL; + goto out; + } cur = end; last = first; if (*cur == '-') { cur++; errno = 0; last = strtol(cur, &end, 10); - if (errno || end == cur || last < first) - return -1; + if (errno || end == cur || last < first) { + ret = -EINVAL; + goto out; + } cur = end; } - cpu = first_allowed_cpu(allowed, first, last); - if (cpu >= 0) - return cpu; + for (cpu = first; cpu <= last && cpu < CPU_SETSIZE; cpu++) + CPU_SET(cpu, mask); } - return -1; +out: + fclose(file); + return ret; } static pid_t start_worker(int cpu) @@ -146,24 +157,54 @@ static int pause_worker(pid_t pid) return 0; } -static bool wait_for_counter(const u64 *counter, u64 value, int timeout_ms) +static bool wait_for_counter(struct nohz_tick *skel, const u64 *counter, + u64 value, int timeout_ms) { int elapsed; for (elapsed = 0; elapsed < timeout_ms; elapsed++) { if (__atomic_load_n(counter, __ATOMIC_RELAXED) >= value) return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; usleep(1000); } return false; } +static bool wait_for_tick_stop(struct nohz_tick *skel, const u64 *counter, + int timeout_ms) +{ + u64 prev = __atomic_load_n(counter, __ATOMIC_RELAXED); + int elapsed, stable = 0; + + for (elapsed = 0; elapsed < timeout_ms; elapsed++) { + u64 curr; + + usleep(1000); + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + curr = __atomic_load_n(counter, __ATOMIC_RELAXED); + if (curr == prev) { + if (++stable >= TICK_STOP_STABLE_MS) + return true; + } else { + prev = curr; + stable = 0; + } + } + + return false; +} + static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; - cpu_set_t controller_mask; - int cpu; + cpu_set_t controller_mask, nohz_full_mask; + bool lazy_supported; + u64 enum_value, lazy_ops_flag; + int cpu, i, lazy_mode, ret; ctx = calloc(1, sizeof(*ctx)); SCX_FAIL_IF(!ctx, "Failed to allocate context"); @@ -173,15 +214,27 @@ static enum scx_test_status setup(void **ctx_ptr) SCX_FAIL("Failed to get affinity (%d)", errno); } - cpu = find_nohz_full_cpu(&ctx->original_mask); - if (cpu < 0) { + ret = read_nohz_full_mask(&nohz_full_mask); + if (ret) { + fprintf(stderr, "SKIP: failed to read NOHZ_FULL mask (%d)\n", ret); + free(ctx); + return SCX_TEST_SKIP; + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &ctx->original_mask) && + CPU_ISSET(cpu, &nohz_full_mask)) + break; + if (cpu == CPU_SETSIZE) { fprintf(stderr, "SKIP: no allowed NOHZ_FULL CPU\n"); free(ctx); return SCX_TEST_SKIP; } controller_mask = ctx->original_mask; - CPU_CLR(cpu, &controller_mask); + for (i = 0; i < CPU_SETSIZE; i++) + if (CPU_ISSET(i, &nohz_full_mask)) + CPU_CLR(i, &controller_mask); if (CPU_COUNT(&controller_mask) == 0) { fprintf(stderr, "SKIP: no housekeeping CPU available\n"); free(ctx); @@ -189,6 +242,23 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_RESCHED", + &lazy_ops_flag); + lazy_mode = scx_test_preempt_lazy_mode(); + ctx->test_lazy = lazy_supported && lazy_mode > 0; + if (lazy_supported && lazy_mode < 0) + fprintf(stderr, + "SKIP: kernel preemption mode unavailable; skipping lazy NOHZ_FULL phases\n"); + else if (lazy_supported && !lazy_mode) + fprintf(stderr, + "SKIP: lazy preemption inactive; skipping lazy NOHZ_FULL phases\n"); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -199,6 +269,8 @@ static enum scx_test_status setup(void **ctx_ptr) ctx->skel->rodata->test_cpu = cpu; ctx->skel->struct_ops.nohz_tick_ops->flags |= SCX_OPS_SWITCH_PARTIAL | SCX_OPS_ENQ_LAST; + if (lazy_supported) + ctx->skel->struct_ops.nohz_tick_ops->flags |= lazy_ops_flag; if (nohz_tick__load(ctx->skel)) { nohz_tick__destroy(ctx->skel); free(ctx); @@ -220,11 +292,15 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct scx_test_gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct scx_test_gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct scx_test_gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; u64 finite_running; u64 finite_ticks; + u64 victim_running; int ret; link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops); @@ -241,7 +317,7 @@ static enum scx_test_status run(void *ctx_ptr) SCX_ERR("Failed to start infinite-slice worker (%d)", errno); goto out; } - if (!wait_for_counter(&skel->bss->nr_inf_running, 1, + if (!wait_for_counter(skel, &skel->bss->nr_inf_running, 1, PHASE_TIMEOUT_MS)) { SCX_ERR("Infinite-slice worker was not scheduled"); goto out; @@ -260,18 +336,19 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); goto out; } - if (!wait_for_counter(&skel->bss->nr_finite_running, 1, + if (!wait_for_counter(skel, &skel->bss->nr_finite_running, 1, PHASE_TIMEOUT_MS)) { SCX_ERR("Finite-slice worker was not scheduled"); goto out; } - if (!wait_for_counter(&skel->bss->nr_finite_ticks, MIN_FINITE_TICKS, + if (!wait_for_counter(skel, &skel->bss->nr_finite_ticks, MIN_FINITE_TICKS, PHASE_TIMEOUT_MS)) { SCX_ERR("Finite-slice worker received only %llu scheduler ticks", (unsigned long long)skel->bss->nr_finite_ticks); @@ -295,12 +372,12 @@ static enum scx_test_status run(void *ctx_ptr) SCX_ERR("Failed to start second finite-slice worker (%d)", errno); goto out; } - if (!wait_for_counter(&skel->bss->nr_finite_running, + if (!wait_for_counter(skel, &skel->bss->nr_finite_running, finite_running + 1, PHASE_TIMEOUT_MS)) { SCX_ERR("Second finite-slice worker was not scheduled"); goto out; } - if (!wait_for_counter(&skel->bss->nr_finite_ticks, + if (!wait_for_counter(skel, &skel->bss->nr_finite_ticks, finite_ticks + MIN_FINITE_TICKS, PHASE_TIMEOUT_MS)) { SCX_ERR("Second finite-slice worker received only %llu scheduler ticks", @@ -308,7 +385,95 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->test_lazy) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = scx_test_spawn_gated_worker(ctx->test_cpu, true); + challenger = scx_test_spawn_gated_worker(ctx->test_cpu, true); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!scx_test_start_gated_worker(&victim) || + !wait_for_counter(skel, &skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + if (!wait_for_tick_stop(skel, &skel->bss->nr_lazy_ticks, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Tick did not stop before lazy enqueue"); + goto out; + } + + if (!scx_test_start_gated_worker(&challenger) || + !wait_for_counter(skel, &skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + scx_test_stop_gated_worker(&victim); + scx_test_stop_gated_worker(&challenger); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = scx_test_spawn_gated_worker(ctx->test_cpu, true); + challenger = scx_test_spawn_gated_worker(ctx->test_cpu, true); + trigger = scx_test_spawn_gated_worker(ctx->housekeeping_cpu, true); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + victim_running = __atomic_load_n(&skel->bss->nr_lazy_victim_running, + __ATOMIC_RELAXED); + if (!scx_test_start_gated_worker(&victim) || + !wait_for_counter(skel, &skel->bss->nr_lazy_victim_running, + victim_running + 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!scx_test_start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + if (!wait_for_tick_stop(skel, &skel->bss->nr_lazy_ticks, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Tick did not stop before lazy kick"); + goto out; + } + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!scx_test_start_gated_worker(&trigger) || + !wait_for_counter(skel, &skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + scx_test_stop_gated_worker(&trigger); + scx_test_stop_gated_worker(&victim); + scx_test_stop_gated_worker(&challenger); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +486,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + scx_test_stop_gated_worker(&trigger); + scx_test_stop_gated_worker(&victim); + scx_test_stop_gated_worker(&challenger); stop_worker(finite_worker); stop_worker(inf_worker); if (link) diff --git a/tools/testing/selftests/sched_ext/nohz_tick_test.h b/tools/testing/selftests/sched_ext/nohz_tick_test.h new file mode 100644 index 0000000000000..b122d723cd8b4 --- /dev/null +++ b/tools/testing/selftests/sched_ext/nohz_tick_test.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __NOHZ_TICK_TEST_H__ +#define __NOHZ_TICK_TEST_H__ + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +#endif /* __NOHZ_TICK_TEST_H__ */ diff --git a/tools/testing/selftests/sched_ext/util.c b/tools/testing/selftests/sched_ext/util.c index 2111329ed2893..5703c8afd5e2f 100644 --- a/tools/testing/selftests/sched_ext/util.c +++ b/tools/testing/selftests/sched_ext/util.c @@ -5,10 +5,21 @@ */ #include <errno.h> #include <fcntl.h> +#include <sched.h> +#include <signal.h> #include <stdio.h> #include <stdlib.h> #include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> #include <unistd.h> +#include <zlib.h> + +#include "util.h" + +#ifndef SCHED_EXT +#define SCHED_EXT 7 +#endif /* Returns read len on success, or -errno on failure. */ static ssize_t read_text(const char *path, char *buf, size_t max_len) @@ -69,3 +80,136 @@ int file_write_long(const char *path, long val) return 0; } + +struct scx_test_gated_worker scx_test_spawn_gated_worker(int cpu, + bool set_sched_ext) +{ + struct scx_test_gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (set_sched_ext && sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +bool scx_test_start_gated_worker(struct scx_test_gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +void scx_test_stop_gated_worker(struct scx_test_gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + if (worker->pid > 0) { + kill(worker->pid, SIGKILL); + waitpid(worker->pid, NULL, 0); + } + worker->pid = -1; + worker->start_fd = -1; +} + +static int config_preempt_lazy_mode(void) +{ + bool dynamic = false, lazy = false, immediate = false; + char buf[128]; + gzFile file; + + file = gzopen("/proc/config.gz", "r"); + if (!file) + return -1; + + while (gzgets(file, buf, sizeof(buf))) { + if (!strcmp(buf, "CONFIG_PREEMPT_DYNAMIC=y\n")) + dynamic = true; + else if (!strcmp(buf, "CONFIG_PREEMPT_LAZY=y\n")) + lazy = true; + else if (!strcmp(buf, "CONFIG_PREEMPT_NONE=y\n") || + !strcmp(buf, "CONFIG_PREEMPT_VOLUNTARY=y\n") || + !strcmp(buf, "CONFIG_PREEMPT=y\n") || + !strcmp(buf, "CONFIG_PREEMPT_RT=y\n")) + immediate = true; + } + gzclose(file); + + if (dynamic) + return -1; + if (lazy) + return 1; + if (immediate) + return 0; + return -1; +} + +int scx_test_preempt_lazy_mode(void) +{ + char buf[128]; + + if (read_text("/sys/kernel/debug/sched/preempt", buf, sizeof(buf)) > 0) { + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(none)") || strstr(buf, "(voluntary)") || + strstr(buf, "(full)")) + return 0; + } + + return config_preempt_lazy_mode(); +} diff --git a/tools/testing/selftests/sched_ext/util.h b/tools/testing/selftests/sched_ext/util.h index 681cec04b4395..6a9bd92d42a3f 100644 --- a/tools/testing/selftests/sched_ext/util.h +++ b/tools/testing/selftests/sched_ext/util.h @@ -7,7 +7,20 @@ #ifndef __SCX_TEST_UTIL_H__ #define __SCX_TEST_UTIL_H__ +#include <stdbool.h> +#include <sys/types.h> + +struct scx_test_gated_worker { + pid_t pid; + int start_fd; +}; + long file_read_long(const char *path); int file_write_long(const char *path, long val); +struct scx_test_gated_worker scx_test_spawn_gated_worker(int cpu, + bool set_sched_ext); +bool scx_test_start_gated_worker(struct scx_test_gated_worker *worker); +void scx_test_stop_gated_worker(struct scx_test_gated_worker *worker); +int scx_test_preempt_lazy_mode(void); #endif // __SCX_TEST_UTIL_H__ -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCHSET v6 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-17 7:02 Andrea Righi
2026-09-17 7:02 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2026-09-17 7:02 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, Cheng-Yang Chou, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags. They
expire the running sched_ext task's slice while allowing the scheduling boundary
to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_SLICE_EXPIRY selects lazy slice expiry by default. Schedulers can
override that default in either direction for individual tasks through
scx_bpf_task_set_slice_expiry(). The helper preserves sub-scheduler task
ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU. The reschedule request is set first so
that the dependency update's IPI can serve it on return to user space, while the
restored tick guarantees eventual promotion otherwise.
The selftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task slice-expiry overrides, invalid
flags, and lazy preemption of infinite-slice tasks on NO_HZ_FULL CPUs. The
NO_HZ_FULL cases verify that the tick has stopped before issuing the lazy
request.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- \
tools/testing/selftests/sched_ext/runner -t kick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
$ vng --cpus 4 -a "nohz_full=1-3" -- \
tools/testing/selftests/sched_ext/runner -t nohz_tick
nohz_tick:
CPU 1 received 6 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
The mode-sensitive kick_lazy and kick_tick cases also passed after switching
/sys/kernel/debug/sched/preempt between lazy and full.
Changes in v6:
- Pack slice_expires_lazy next to scx.disallow instead of growing task_struct
by eight bytes (Tejun Heo).
- Correct the SCX_OPS_LAZY_SLICE_EXPIRY documentation for PREEMPT_DYNAMIC and
bypass (Tejun Heo).
- Set the lazy reschedule request before restoring the tick dependency, so the
dependency update's IPI can serve it on return to user space (Tejun Heo).
- Explain the pinned rq locking and the newly rejected unknown kick flags in
the patch description (Tejun Heo).
- Regenerate the enum headers in definition order (Tejun Heo).
- kselftest: use SCX_OPS_ENQ_LAST to drive kick-test victims through
ops.stopping() (Tejun Heo).
- kselftest: poll for the SCX_KICK_WAIT callback after ops.stopping()
(Tejun Heo).
- kselftest: skip trace-based tests when their BTF targets are unavailable
(Tejun Heo).
- kselftest: verify that the NO_HZ_FULL tick has stopped before testing lazy
enqueue and kick requests (Tejun Heo).
- Link to v5: https://lore.kernel.org/r/20260915194611.2674127-1-arighi@nvidia.com
Changes in v5:
- kselftest: detect static lazy preemption when CONFIG_PREEMPT_DYNAMIC is
disabled and the sched/preempt debugfs file is unavailable (Cheng-Yang Chou).
- Link to v4: https://lore.kernel.org/r/20260915090127.2321020-1-arighi@nvidia.com
Changes in v4:
- Preserve SCX_KICK_WAIT synchronization when it is combined with
SCX_KICK_PREEMPT_LAZY (Sashiko).
- Restore the scheduler tick dependency when ops.tick() lazily expires an
infinite slice, ensuring the request is promoted by a later tick (Sashiko).
- kselftest: verify that the deferred SCX_KICK_WAIT callback runs (Sashiko).
- kselftest: keep the kick test controller off the victim CPU and skip cleanly
when a second allowed CPU is unavailable (Sashiko).
- kselftest: select the nohz_tick controller from allowed housekeeping CPUs
outside the NOHZ_FULL mask and skip when the required topology is
unavailable.
- Link to v3: https://lore.kernel.org/r/20260914144620.2032614-1-arighi@nvidia.com
Changes in v3:
- Avoid redundant rq clock updates when restoring the tick dependency and use
the pinned rq lock helpers for lazy kick delivery (Sashiko).
- Set PR_SET_PDEATHSIG for the indefinitely spinning kick test workers so they
don't survive unexpected test-runner exit (Sashiko).
- Wait for a delta in nr_lazy_victim_running instead of an absolute count that
may include reschedules from the previous phase (Sashiko).
- Increase the nohz_tick scheduler watchdog and phase timeouts to tolerate
scheduling delays in oversubscribed test VMs.
- Link to v2: https://lore.kernel.org/r/20260914084955.1798562-1-arighi@nvidia.com
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any sched_ext
callback (preserving sub-scheduler task ownership).
- Restore the scheduler tick dependency for lazy enqueue and kick preemption of
infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve their
precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY, immediate preemption, WAIT
and plain kicks take precedence over lazy preemption (Tejun Heo).
- Clear both preemption masks together when processing or skipping a preemption
request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests, per-task
expiry overrides and NO_HZ_FULL coverage.
- Link to v1: https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 152 ++++-
kernel/sched/ext/internal.h | 40 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 17 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 3 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 6 +
tools/sched_ext/include/scx/enums.autogen.h | 2 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 5 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 191 +++++++
tools/testing/selftests/sched_ext/kick.c | 655 ++++++++++++++++++++++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 62 +-
tools/testing/selftests/sched_ext/nohz_tick.c | 280 ++++++++-
14 files changed, 1372 insertions(+), 65 deletions(-)
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-17 7:02 [PATCHSET v6 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-17 7:02 ` Andrea Righi 2026-09-17 19:19 ` Tejun Heo 0 siblings, 1 reply; 13+ messages in thread From: Andrea Righi @ 2026-09-17 7:02 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, Cheng-Yang Chou, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF after the current task's slice is cleared. SCX_OPS_ENQ_LAST drives the victim through ops.stopping() without relying on an unrelated wakeup. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the expiry default in both directions through scx_bpf_task_set_slice_expiry(), and add invalid kick flag coverage. Extend the NO_HZ_FULL test with infinite-slice victims. Wait until the scheduler tick is observed to stop, then verify that lazy enqueue and kick requests restart it and make forward progress. Move the controller off all NO_HZ_FULL CPUs and increase the phase and watchdog timeouts to tolerate scheduling delays in oversubscribed test VMs. Skip trace-based cases when their kernel BTF targets are unavailable, while allowing the independent invalid-flag cases to run. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 191 +++++ tools/testing/selftests/sched_ext/kick.c | 655 ++++++++++++++++++ .../selftests/sched_ext/nohz_tick.bpf.c | 62 +- tools/testing/selftests/sched_ext/nohz_tick.c | 280 +++++++- 5 files changed, 1162 insertions(+), 27 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..7d4eb0919f099 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,191 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 challenger_pid; +const volatile s32 target_cpu; +const volatile s32 slice_expiry_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; +u64 nr_wait_callbacks; +bool enq_preempt_issued; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (p->pid == challenger_pid && + state == KICK_STATE_QUEUED) { + enq_preempt_issued = true; + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + goto kick_last; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +kick_last: + if (enq_flags & SCX_ENQ_LAST) + scx_bpf_kick_cpu(target_cpu, SCX_KICK_IDLE); +} + +static void set_slice_expiry_override(struct task_struct *p) +{ + if (p->pid == victim_pid && slice_expiry_override >= 0) + scx_bpf_task_set_slice_expiry(p, slice_expiry_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_slice_expiry_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + u64 slice; + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + if (scenario == ENQ_BOTH && !enq_preempt_issued) + return 0; + + slice = BPF_CORE_READ(task, scx.slice); + if (slice) + return 0; + + slice_at_resched = slice; + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +SEC("fentry/kick_sync_wait_bal_cb") +int BPF_PROG(kick_wait_callback, struct rq *rq) +{ + if (scenario == KICK_LAZY_WAIT && + BPF_CORE_READ(rq, cpu) == target_cpu) + __sync_fetch_and_add(&nr_wait_callbacks, 1); + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .flags = SCX_OPS_ENQ_LAST, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..f64baa948d9d5 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,655 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <errno.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#ifdef HAVE_GENHDR +#include "autoconf.h" +#endif + +#include "kick.bpf.skel.h" +#include "scx_test.h" + +#define WAIT_LOOPS 3000 + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +struct victim { + pid_t pid; + int start_fd; +}; + +struct observation { + u64 slice_before; + u64 slice_at_resched; + u64 nr_wait_callbacks; + s32 resched_tif; +}; + +struct kick_ctx { + cpu_set_t original_mask; + int target_cpu; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static enum scx_test_status setup_controller(void **ctx_ptr) +{ + struct kick_ctx *ctx; + cpu_set_t controller_mask; + int cpu, controller_cpu = -1; + + ctx = calloc(1, sizeof(*ctx)); + SCX_FAIL_IF(!ctx, "Failed to allocate context"); + if (sched_getaffinity(0, sizeof(ctx->original_mask), + &ctx->original_mask)) { + free(ctx); + SCX_FAIL("Failed to get affinity (%d)", errno); + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &ctx->original_mask)) + continue; + if (controller_cpu < 0) { + controller_cpu = cpu; + } else { + ctx->target_cpu = cpu; + break; + } + } + if (cpu == CPU_SETSIZE) { + printf("SKIP: two allowed CPUs are required\n"); + free(ctx); + return SCX_TEST_SKIP; + } + + CPU_ZERO(&controller_mask); + CPU_SET(controller_cpu, &controller_mask); + if (sched_setaffinity(0, sizeof(controller_mask), &controller_mask)) { + free(ctx); + SCX_FAIL("Failed to pin controller to CPU %d (%d)", + controller_cpu, errno); + } + + *ctx_ptr = ctx; + return SCX_TEST_PASS; +} + +static void cleanup_controller(void *ctx_ptr) +{ + struct kick_ctx *ctx = ctx_ptr; + + sched_setaffinity(0, sizeof(ctx->original_mask), &ctx->original_mask); + free(ctx); +} + +static struct victim spawn_victim(int cpu) +{ + struct victim victim = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return victim; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return victim; + } + + victim.pid = fork(); + if (!victim.pid) { + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (victim.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return victim; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(victim.pid, SIGKILL); + waitpid(victim.pid, NULL, 0); + victim.pid = -1; + return victim; + } + close(ready[0]); + victim.start_fd = start[1]; + return victim; +} + +static void stop_victim(struct victim *victim) +{ + if (victim->start_fd >= 0) + close(victim->start_fd); + if (victim->pid > 0) { + kill(victim->pid, SIGKILL); + waitpid(victim->pid, NULL, 0); + } +} + +static bool start_victim(struct victim *victim) +{ + char byte = 1; + + if (write(victim->start_fd, &byte, 1) != 1) + return false; + close(victim->start_fd); + victim->start_fd = -1; + return true; +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static bool wait_for_counter(struct kick *skel, const u64 *counter, u64 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(counter, __ATOMIC_ACQUIRE) >= wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(struct kick_ctx *ctx, u32 scenario, + u64 ops_flags, + s32 slice_expiry_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim challenger = { .pid = -1, .start_fd = -1 }; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + + victim = spawn_victim(cpu); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = spawn_victim(cpu); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->challenger_pid = challenger.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->slice_expiry_override = slice_expiry_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (scenario != KICK_LAZY_WAIT) + bpf_program__set_autoload(skel->progs.kick_wait_callback, + false); + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!start_victim(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!start_victim(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + if (scenario == KICK_LAZY_WAIT && + !wait_for_counter(skel, &skel->bss->nr_wait_callbacks, 1)) { + SCX_ERR("WAIT callback did not run"); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->nr_wait_callbacks = skel->bss->nr_wait_callbacks; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + stop_victim(&challenger); + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static int active_lazy_mode(void) +{ + char buf[128]; + FILE *file; + + file = fopen("/sys/kernel/debug/sched/preempt", "r"); + if (!file) + return -1; + if (!fgets(buf, sizeof(buf), file)) { + fclose(file); + return -1; + } + fclose(file); + + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(none)") || strstr(buf, "(voluntary)") || + strstr(buf, "(full)")) + return 0; + return -1; +} + +static bool trace_target_supported(const char *name, + enum bpf_attach_type attach_type) +{ + return libbpf_find_vmlinux_btf_id(name, attach_type) > 0; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + if (!trace_target_supported("__resched_curr", BPF_TRACE_FEXIT)) { + printf("SKIP: __resched_curr is not available in BTF\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY")) { + printf("SKIP: SCX_OPS_LAZY_SLICE_EXPIRY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + if (!trace_target_supported("kick_sync_wait_bal_cb", BPF_TRACE_FENTRY)) { + printf("SKIP: kick_sync_wait_bal_cb is not available in BTF\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &obs); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + enum scx_test_status status; + int lazy_mode; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY, 0, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY_THEN_IMMEDIATE, 0, -1, + &lazy_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + status = trace_one(ctx, KICK_IMMEDIATE_THEN_LAZY, 0, -1, + &immediate_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + status = trace_one(ctx, KICK_PLAIN_THEN_LAZY, 0, -1, + &plain_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + status = trace_one(ctx, KICK_LAZY_THEN_PLAIN, 0, -1, + &lazy_then_plain); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + status = trace_one(ctx, KICK_BOTH, 0, -1, &both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + status = trace_one(ctx, KICK_LAZY_WAIT, 0, -1, &lazy_wait); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + SCX_GT(lazy_wait.nr_wait_callbacks, 0); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + status = trace_one(ctx, ENQ_BOTH, 0, -1, &enq_both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_SLICE_EXPIRY, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + enum scx_test_status status; + u64 lazy_flag; + int lazy_mode; + bool found; + + found = __COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", + &lazy_flag); + SCX_ASSERT(found); + status = trace_one(ctx, TICK_EXPIRY, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + status = trace_one(ctx, TICK_EXPIRY, 0, 1, &force_lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_lazy)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, 0, + &force_immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_immediate)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(struct kick_ctx *ctx, u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + int i; + + victim = spawn_victim(cpu); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + bpf_program__set_autoload(skel->progs.kick_need_resched, false); + bpf_program__set_autoload(skel->progs.kick_wait_callback, false); + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !start_victim(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + enum scx_test_status status; + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; + scenario <= INVALID_KICK_UNKNOWN; scenario++) { + status = invalid_one(ctx, scenario); + SCX_EQ(status, SCX_TEST_PASS); + } + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, + .cleanup = cleanup_controller, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..3f45233acc464 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -9,10 +9,24 @@ char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; +u64 nr_lazy_ticks; UEI_DEFINE(uei); @@ -24,9 +38,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,16 +72,28 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() != test_cpu) + return; + + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); + else if ((phase == NOHZ_PHASE_LAZY_ENQ || + phase == NOHZ_PHASE_LAZY_KICK) && p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_ticks, 1); } void BPF_STRUCT_OPS(nohz_tick_exit, struct scx_exit_info *ei) @@ -61,5 +109,5 @@ struct sched_ext_ops nohz_tick_ops = { .tick = (void *)nohz_tick_tick, .exit = (void *)nohz_tick_exit, .name = "nohz_tick", - .timeout_ms = 1000U, + .timeout_ms = 5000U, }; diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..7b3042b4794d3 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -28,12 +28,27 @@ #endif #define MIN_FINITE_TICKS 3 -#define PHASE_TIMEOUT_MS 1000 +#define PHASE_TIMEOUT_MS 5000 +#define TICK_STOP_STABLE_MS 100 struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool lazy_supported; +}; + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +struct gated_worker { + pid_t pid; + int start_fd; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -47,20 +62,21 @@ static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) return -1; } -static int find_nohz_full_cpu(const cpu_set_t *allowed) +static int read_nohz_full_mask(cpu_set_t *mask) { char buf[4096], *cur, *end; FILE *file; + int ret = 0; file = fopen("/sys/devices/system/cpu/nohz_full", "r"); if (!file) - return -1; + return -errno; if (!fgets(buf, sizeof(buf), file)) { - fclose(file); - return -1; + ret = ferror(file) ? -errno : -EINVAL; + goto out; } - fclose(file); + CPU_ZERO(mask); cur = buf; while (*cur) { long first, last; @@ -73,25 +89,30 @@ static int find_nohz_full_cpu(const cpu_set_t *allowed) errno = 0; first = strtol(cur, &end, 10); - if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) - return -1; + if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) { + ret = -EINVAL; + goto out; + } cur = end; last = first; if (*cur == '-') { cur++; errno = 0; last = strtol(cur, &end, 10); - if (errno || end == cur || last < first) - return -1; + if (errno || end == cur || last < first) { + ret = -EINVAL; + goto out; + } cur = end; } - cpu = first_allowed_cpu(allowed, first, last); - if (cpu >= 0) - return cpu; + for (cpu = first; cpu <= last && cpu < CPU_SETSIZE; cpu++) + CPU_SET(cpu, mask); } - return -1; +out: + fclose(file); + return ret; } static pid_t start_worker(int cpu) @@ -132,6 +153,87 @@ static void stop_worker(pid_t pid) waitpid(pid, NULL, 0); } +static struct gated_worker spawn_gated_worker(int cpu) +{ + struct gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +static bool start_gated_worker(struct gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +static void stop_gated_worker(struct gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + stop_worker(worker->pid); + worker->pid = -1; +} + static int pause_worker(pid_t pid) { int status; @@ -159,11 +261,34 @@ static bool wait_for_counter(const u64 *counter, u64 value, int timeout_ms) return false; } +static bool wait_for_tick_stop(const u64 *counter, int timeout_ms) +{ + u64 prev = __atomic_load_n(counter, __ATOMIC_RELAXED); + int elapsed, stable = 0; + + for (elapsed = 0; elapsed < timeout_ms; elapsed++) { + u64 curr; + + usleep(1000); + curr = __atomic_load_n(counter, __ATOMIC_RELAXED); + if (curr == prev) { + if (++stable >= TICK_STOP_STABLE_MS) + return true; + } else { + prev = curr; + stable = 0; + } + } + + return false; +} + static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; - cpu_set_t controller_mask; - int cpu; + cpu_set_t controller_mask, nohz_full_mask; + u64 enum_value; + int cpu, i, ret; ctx = calloc(1, sizeof(*ctx)); SCX_FAIL_IF(!ctx, "Failed to allocate context"); @@ -173,15 +298,27 @@ static enum scx_test_status setup(void **ctx_ptr) SCX_FAIL("Failed to get affinity (%d)", errno); } - cpu = find_nohz_full_cpu(&ctx->original_mask); - if (cpu < 0) { + ret = read_nohz_full_mask(&nohz_full_mask); + if (ret) { + fprintf(stderr, "SKIP: failed to read NOHZ_FULL mask (%d)\n", ret); + free(ctx); + return SCX_TEST_SKIP; + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &ctx->original_mask) && + CPU_ISSET(cpu, &nohz_full_mask)) + break; + if (cpu == CPU_SETSIZE) { fprintf(stderr, "SKIP: no allowed NOHZ_FULL CPU\n"); free(ctx); return SCX_TEST_SKIP; } controller_mask = ctx->original_mask; - CPU_CLR(cpu, &controller_mask); + for (i = 0; i < CPU_SETSIZE; i++) + if (CPU_ISSET(i, &nohz_full_mask)) + CPU_CLR(i, &controller_mask); if (CPU_COUNT(&controller_mask) == 0) { fprintf(stderr, "SKIP: no housekeeping CPU available\n"); free(ctx); @@ -189,6 +326,13 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + ctx->lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -220,11 +364,15 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; u64 finite_running; u64 finite_ticks; + u64 victim_running; int ret; link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops); @@ -260,7 +408,8 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); @@ -308,7 +457,95 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->lazy_supported) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + if (!wait_for_tick_stop(&skel->bss->nr_lazy_ticks, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Tick did not stop before lazy enqueue"); + goto out; + } + + if (!start_gated_worker(&challenger) || + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&challenger); + stop_gated_worker(&victim); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + trigger = spawn_gated_worker(ctx->housekeeping_cpu); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + victim_running = __atomic_load_n(&skel->bss->nr_lazy_victim_running, + __ATOMIC_RELAXED); + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, + victim_running + 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + if (!wait_for_tick_stop(&skel->bss->nr_lazy_ticks, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Tick did not stop before lazy kick"); + goto out; + } + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!start_gated_worker(&trigger) || + !wait_for_counter(&skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +558,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); stop_worker(finite_worker); stop_worker(inf_worker); if (link) -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-17 7:02 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi @ 2026-09-17 19:19 ` Tejun Heo 2026-09-18 6:30 ` Andrea Righi 0 siblings, 1 reply; 13+ messages in thread From: Tejun Heo @ 2026-09-17 19:19 UTC (permalink / raw) To: Andrea Righi Cc: Tejun Heo, David Vernet, Changwoo Min, Emil Tsalapatis, Emil Tsalapatis, David Dai, Cheng-Yang Chou, sched-ext, linux-kernel Hello, Andrea. This is an AI review. The series was built and the tests pass under vng here in lazy and full modes, with nohz_full on 2-3 and on 1-3. Three things worth fixing and some leftovers. On Thu, Sep 17, 2026 at 09:02:53AM +0200, Andrea Righi wrote: > +enum kick_scenario { exit_test.h and hotplug_test.h share such enums between the BPF and user sides. The same would work here and for nohz_phase. run_invalid() and is_trace_scenario() depend on the two copies staying in order. > +#ifdef HAVE_GENHDR > +#include "autoconf.h" > +#endif Nothing uses CONFIG_ in kick.c anymore. > +static struct victim spawn_victim(int cpu) Same as spawn_gated_worker() in nohz_tick.c except for sched_setscheduler(). util.c could hold one copy. > + file = fopen("/sys/kernel/debug/sched/preempt", "r"); > + if (!file) > + return -1; That file only exists with PREEMPT_DYNAMIC, and needs debugfs mounted. Every static kernel takes the INFO path and passes without comparing, including static PREEMPT_LAZY, which is the default on arm64, riscv, powerpc, s390 and loongarch where PREEMPT_DYNAMIC defaults off. uname can't tell either, as PREEMPT_LAZY selects PREEMPT_BUILD. /proc/config.gz covers static kernels when it's there. Otherwise skipping would be more honest than passing. > + SCX_EQ(both.resched_tif, immediate.resched_tif); > + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); Outside lazy mode the lazy TIF is the immediate one, so these and the override checks in run_tick() hold whatever kick_one_cpu() decided. Maybe skip or note it when the mode isn't lazy. > + case NOHZ_PHASE_LAZY_ENQ: > + case NOHZ_PHASE_LAZY_KICK: > + dsq_id = SCX_DSQ_LOCAL; > + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; The victim gets an infinite slice on every enqueue, including the one after the lazy preemption puts it back. In the kick phase the challenger is stopped first; if its slice runs out before the SIGKILL lands, it's re-enqueued behind the victim and never runs again, waitpid() blocks, and the 5s watchdog aborts the scheduler, which then fails the test at check_exit. The enqueue phase is safe as the challenger re-inserts at the head. Stopping the victim first avoids it. ENQ_BOTH in kick.c has the same shape with the global DSQ and the default 30s watchdog, a stall rather than a failure there. > +static bool wait_for_tick_stop(const u64 *counter, int timeout_ms) Neither this nor wait_for_counter() checks uei.kind, so a watchdog abort mid-phase reports as no progress. > + if (!start_gated_worker(&challenger) || > + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, > + PHASE_TIMEOUT_MS)) { > + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); This passes even without the tick dependency restore. sched_tick_remote() runs task_tick_scx() once a second on the tick-stopped CPU, and as nohz_tick_ops doesn't set SCX_OPS_LAZY_SLICE_EXPIRY that path does resched_curr() with an IPI, so the challenger runs within a second either way. Setting the flag on nohz_tick_ops keeps the remote tick lazy, so a missing restore shows up as the timeout. Both phases only test anything in lazy mode. Thanks. -- tejun ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-17 19:19 ` Tejun Heo @ 2026-09-18 6:30 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-18 6:30 UTC (permalink / raw) To: Tejun Heo Cc: David Vernet, Changwoo Min, Emil Tsalapatis, Emil Tsalapatis, David Dai, Cheng-Yang Chou, sched-ext, linux-kernel Hi Tejun, On Thu, Sep 17, 2026 at 09:19:07AM -1000, Tejun Heo wrote: > Hello, Andrea. > > This is an AI review. The series was built and the tests pass under vng here > in lazy and full modes, with nohz_full on 2-3 and on 1-3. Three things worth > fixing and some leftovers. > > On Thu, Sep 17, 2026 at 09:02:53AM +0200, Andrea Righi wrote: > > +enum kick_scenario { > > exit_test.h and hotplug_test.h share such enums between the BPF and user > sides. The same would work here and for nohz_phase. run_invalid() and > is_trace_scenario() depend on the two copies staying in order. > > > +#ifdef HAVE_GENHDR > > +#include "autoconf.h" > > +#endif > > Nothing uses CONFIG_ in kick.c anymore. > > > +static struct victim spawn_victim(int cpu) > > Same as spawn_gated_worker() in nohz_tick.c except for sched_setscheduler(). > util.c could hold one copy. > > > + file = fopen("/sys/kernel/debug/sched/preempt", "r"); > > + if (!file) > > + return -1; > > That file only exists with PREEMPT_DYNAMIC, and needs debugfs mounted. Every > static kernel takes the INFO path and passes without comparing, including > static PREEMPT_LAZY, which is the default on arm64, riscv, powerpc, s390 and > loongarch where PREEMPT_DYNAMIC defaults off. uname can't tell either, as > PREEMPT_LAZY selects PREEMPT_BUILD. /proc/config.gz covers static kernels > when it's there. Otherwise skipping would be more honest than passing. > > > + SCX_EQ(both.resched_tif, immediate.resched_tif); > > + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); > > Outside lazy mode the lazy TIF is the immediate one, so these and the > override checks in run_tick() hold whatever kick_one_cpu() decided. Maybe > skip or note it when the mode isn't lazy. > > > + case NOHZ_PHASE_LAZY_ENQ: > > + case NOHZ_PHASE_LAZY_KICK: > > + dsq_id = SCX_DSQ_LOCAL; > > + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; > > The victim gets an infinite slice on every enqueue, including the one after > the lazy preemption puts it back. In the kick phase the challenger is stopped > first; if its slice runs out before the SIGKILL lands, it's re-enqueued > behind the victim and never runs again, waitpid() blocks, and the 5s watchdog > aborts the scheduler, which then fails the test at check_exit. The enqueue > phase is safe as the challenger re-inserts at the head. Stopping the victim > first avoids it. ENQ_BOTH in kick.c has the same shape with the global DSQ > and the default 30s watchdog, a stall rather than a failure there. > > > +static bool wait_for_tick_stop(const u64 *counter, int timeout_ms) > > Neither this nor wait_for_counter() checks uei.kind, so a watchdog abort > mid-phase reports as no progress. > > > + if (!start_gated_worker(&challenger) || > > + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, > > + PHASE_TIMEOUT_MS)) { > > + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); > > This passes even without the tick dependency restore. sched_tick_remote() > runs task_tick_scx() once a second on the tick-stopped CPU, and as > nohz_tick_ops doesn't set SCX_OPS_LAZY_SLICE_EXPIRY that path does > resched_curr() with an IPI, so the challenger runs within a second either > way. Setting the flag on nohz_tick_ops keeps the remote tick lazy, so a > missing restore shows up as the timeout. Both phases only test anything in > lazy mode. Agree to all of the above. I'll send a new version with all of these fixed. Thanks, -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCHSET v5 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-15 19:45 Andrea Righi
2026-09-15 19:45 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2026-09-15 19:45 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, Cheng-Yang Chou, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags. They
expire the running sched_ext task's slice while allowing the scheduling boundary
to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_SLICE_EXPIRY selects lazy slice expiry by default. Schedulers can
override that default in either direction for individual tasks through
scx_bpf_task_set_slice_expiry(). The helper preserves sub-scheduler task
ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU, ensuring forward progress even when the
target's tick has stopped.
The selftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task slice-expiry overrides, invalid
flags, and lazy preemption of infinite-slice tasks on NO_HZ_FULL CPUs.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- tools/testing/selftests/sched_ext/runner -t kick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
$ vng -a "preempt=lazy nohz_full=8-15" -- tools/testing/selftests/sched_ext/runner -t nohz_tick
nohz_tick:
CPU 8 received 7 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
Changes in v5:
- kselftest: detect static lazy preemption when CONFIG_PREEMPT_DYNAMIC is
disabled and the sched/preempt debugfs file is unavailable (Cheng-Yang Chou).
- Link to v4: https://lore.kernel.org/r/20260915090127.2321020-1-arighi@nvidia.com
Changes in v4:
- Preserve SCX_KICK_WAIT synchronization when it is combined with
SCX_KICK_PREEMPT_LAZY (Sashiko).
- Restore the scheduler tick dependency when ops.tick() lazily expires an
infinite slice, ensuring the request is promoted by a later tick (Sashiko).
- kselftest: verify that the deferred SCX_KICK_WAIT callback runs (Sashiko).
- kselftest: keep the kick test controller off the victim CPU and skip cleanly
when a second allowed CPU is unavailable (Sashiko).
- kselftest: select the nohz_tick controller from allowed housekeeping CPUs
outside the NOHZ_FULL mask and skip when the required topology is
unavailable.
- Link to v3: https://lore.kernel.org/r/20260914144620.2032614-1-arighi@nvidia.com
Changes in v3:
- Avoid redundant rq clock updates when restoring the tick dependency and use
the pinned rq lock helpers for lazy kick delivery (Sashiko).
- Set PR_SET_PDEATHSIG for the indefinitely spinning kick test workers so they
don't survive unexpected test-runner exit (Sashiko).
- Wait for a delta in nr_lazy_victim_running instead of an absolute count that
may include reschedules from the previous phase (Sashiko).
- Increase the nohz_tick scheduler watchdog and phase timeouts to tolerate
scheduling delays in oversubscribed test VMs.
- Link to v2: https://lore.kernel.org/r/20260914084955.1798562-1-arighi@nvidia.com
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any sched_ext
callback (preserving sub-scheduler task ownership).
- Restore the scheduler tick dependency for lazy enqueue and kick preemption of
infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve their
precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY, immediate preemption, WAIT
and plain kicks take precedence over lazy preemption (Tejun Heo).
- Clear both preemption masks together when processing or skipping a preemption
request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests, per-task
expiry overrides and NO_HZ_FULL coverage.
- Link to v1: https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 153 +++++-
kernel/sched/ext/internal.h | 39 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 13 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 3 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 6 +
tools/sched_ext/include/scx/enums.autogen.h | 2 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 5 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 182 +++++++
tools/testing/selftests/sched_ext/kick.c | 617 ++++++++++++++++++++++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 56 +-
tools/testing/selftests/sched_ext/nohz_tick.c | 249 ++++++++-
14 files changed, 1282 insertions(+), 67 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/kick.c
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-15 19:45 [PATCHSET v5 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-15 19:45 ` Andrea Righi 2026-09-16 21:03 ` Tejun Heo 0 siblings, 1 reply; 13+ messages in thread From: Andrea Righi @ 2026-09-15 19:45 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, Cheng-Yang Chou, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF and the current task slice, while the stopping callback verifies that the task reaches a scheduling boundary. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the expiry default in both directions through scx_bpf_task_set_slice_expiry(). Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy enqueue and kick requests restart a stopped tick and make forward progress. Keep invalid kick flag coverage and skip modes not exposed by the running kernel. Cc: Cheng-Yang Chou <yphbchou0911@gmail.com> Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 182 ++++++ tools/testing/selftests/sched_ext/kick.c | 617 ++++++++++++++++++ .../selftests/sched_ext/nohz_tick.bpf.c | 56 +- tools/testing/selftests/sched_ext/nohz_tick.c | 249 ++++++- 5 files changed, 1078 insertions(+), 27 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..5a64e19246004 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 challenger_pid; +const volatile s32 target_cpu; +const volatile s32 slice_expiry_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; +u64 nr_wait_callbacks; +bool enq_preempt_issued; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (p->pid == challenger_pid && + state == KICK_STATE_QUEUED) { + enq_preempt_issued = true; + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + return; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +} + +static void set_slice_expiry_override(struct task_struct *p) +{ + if (p->pid == victim_pid && slice_expiry_override >= 0) + scx_bpf_task_set_slice_expiry(p, slice_expiry_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_slice_expiry_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + if (scenario == ENQ_BOTH && !enq_preempt_issued) + return 0; + + slice_at_resched = BPF_CORE_READ(task, scx.slice); + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +SEC("fentry/kick_sync_wait_bal_cb") +int BPF_PROG(kick_wait_callback, struct rq *rq) +{ + if (scenario == KICK_LAZY_WAIT && + BPF_CORE_READ(rq, cpu) == target_cpu) + __sync_fetch_and_add(&nr_wait_callbacks, 1); + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..118cc374e6b82 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,617 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <errno.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#ifdef HAVE_GENHDR +#include "autoconf.h" +#endif + +#include "kick.bpf.skel.h" +#include "scx_test.h" + +#define WAIT_LOOPS 3000 + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +struct victim { + pid_t pid; + int start_fd; +}; + +struct observation { + u64 slice_before; + u64 slice_at_resched; + u64 nr_wait_callbacks; + s32 resched_tif; +}; + +struct kick_ctx { + cpu_set_t original_mask; + int target_cpu; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static enum scx_test_status setup_controller(void **ctx_ptr) +{ + struct kick_ctx *ctx; + cpu_set_t controller_mask; + int cpu, controller_cpu = -1; + + ctx = calloc(1, sizeof(*ctx)); + SCX_FAIL_IF(!ctx, "Failed to allocate context"); + if (sched_getaffinity(0, sizeof(ctx->original_mask), + &ctx->original_mask)) { + free(ctx); + SCX_FAIL("Failed to get affinity (%d)", errno); + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &ctx->original_mask)) + continue; + if (controller_cpu < 0) { + controller_cpu = cpu; + } else { + ctx->target_cpu = cpu; + break; + } + } + if (cpu == CPU_SETSIZE) { + printf("SKIP: two allowed CPUs are required\n"); + free(ctx); + return SCX_TEST_SKIP; + } + + CPU_ZERO(&controller_mask); + CPU_SET(controller_cpu, &controller_mask); + if (sched_setaffinity(0, sizeof(controller_mask), &controller_mask)) { + free(ctx); + SCX_FAIL("Failed to pin controller to CPU %d (%d)", + controller_cpu, errno); + } + + *ctx_ptr = ctx; + return SCX_TEST_PASS; +} + +static void cleanup_controller(void *ctx_ptr) +{ + struct kick_ctx *ctx = ctx_ptr; + + sched_setaffinity(0, sizeof(ctx->original_mask), &ctx->original_mask); + free(ctx); +} + +static struct victim spawn_victim(int cpu) +{ + struct victim victim = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return victim; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return victim; + } + + victim.pid = fork(); + if (!victim.pid) { + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (victim.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return victim; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(victim.pid, SIGKILL); + waitpid(victim.pid, NULL, 0); + victim.pid = -1; + return victim; + } + close(ready[0]); + victim.start_fd = start[1]; + return victim; +} + +static void stop_victim(struct victim *victim) +{ + if (victim->start_fd >= 0) + close(victim->start_fd); + if (victim->pid > 0) { + kill(victim->pid, SIGKILL); + waitpid(victim->pid, NULL, 0); + } +} + +static bool start_victim(struct victim *victim) +{ + char byte = 1; + + if (write(victim->start_fd, &byte, 1) != 1) + return false; + close(victim->start_fd); + victim->start_fd = -1; + return true; +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(struct kick_ctx *ctx, u32 scenario, + u64 ops_flags, + s32 slice_expiry_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim challenger = { .pid = -1, .start_fd = -1 }; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + + victim = spawn_victim(cpu); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = spawn_victim(cpu); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->challenger_pid = challenger.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->slice_expiry_override = slice_expiry_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!start_victim(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!start_victim(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->nr_wait_callbacks = skel->bss->nr_wait_callbacks; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + stop_victim(&challenger); + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static int active_lazy_mode(void) +{ + char buf[128]; + FILE *file; + + file = fopen("/sys/kernel/debug/sched/preempt", "r"); + if (!file) { +#if defined(CONFIG_PREEMPT_LAZY) && !defined(CONFIG_PREEMPT_DYNAMIC) + return 1; +#else + return -1; +#endif + } + if (!fgets(buf, sizeof(buf), file)) { + fclose(file); + return -1; + } + fclose(file); + + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(full)")) + return 0; + return -1; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_controller(ctx); +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY")) { + printf("SKIP: SCX_OPS_LAZY_SLICE_EXPIRY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + return setup_lazy(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &obs); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + enum scx_test_status status; + int lazy_mode; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY, 0, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + enum scx_test_status status; + + status = trace_one(ctx, KICK_IMMEDIATE, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, KICK_LAZY_THEN_IMMEDIATE, 0, -1, + &lazy_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + status = trace_one(ctx, KICK_IMMEDIATE_THEN_LAZY, 0, -1, + &immediate_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + status = trace_one(ctx, KICK_PLAIN_THEN_LAZY, 0, -1, + &plain_first); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + status = trace_one(ctx, KICK_LAZY_THEN_PLAIN, 0, -1, + &lazy_then_plain); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + status = trace_one(ctx, KICK_BOTH, 0, -1, &both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + status = trace_one(ctx, KICK_LAZY_WAIT, 0, -1, &lazy_wait); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + SCX_GT(lazy_wait.nr_wait_callbacks, 0); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + status = trace_one(ctx, ENQ_BOTH, 0, -1, &enq_both); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_SLICE_EXPIRY, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + enum scx_test_status status; + u64 lazy_flag; + int lazy_mode; + bool found; + + found = __COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", + &lazy_flag); + SCX_ASSERT(found); + status = trace_one(ctx, TICK_EXPIRY, 0, -1, &immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, -1, &lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy)); + status = trace_one(ctx, TICK_EXPIRY, 0, 1, &force_lazy); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_lazy)); + status = trace_one(ctx, TICK_EXPIRY, lazy_flag, 0, + &force_immediate); + SCX_EQ(status, SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&force_immediate)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(struct kick_ctx *ctx, u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = ctx->target_cpu; + int i; + + victim = spawn_victim(cpu); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !start_victim(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + enum scx_test_status status; + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; + scenario <= INVALID_KICK_UNKNOWN; scenario++) { + status = invalid_one(ctx, scenario); + SCX_EQ(status, SCX_TEST_PASS); + } + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, + .cleanup = cleanup_controller, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, + .cleanup = cleanup_controller, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..96ba1524ca384 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -9,10 +9,23 @@ char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; UEI_DEFINE(uei); @@ -24,9 +37,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,15 +71,22 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() == test_cpu && + phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); } @@ -61,5 +103,5 @@ struct sched_ext_ops nohz_tick_ops = { .tick = (void *)nohz_tick_tick, .exit = (void *)nohz_tick_exit, .name = "nohz_tick", - .timeout_ms = 1000U, + .timeout_ms = 5000U, }; diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..acc6510e239ad 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -28,12 +28,26 @@ #endif #define MIN_FINITE_TICKS 3 -#define PHASE_TIMEOUT_MS 1000 +#define PHASE_TIMEOUT_MS 5000 struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool lazy_supported; +}; + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +struct gated_worker { + pid_t pid; + int start_fd; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -47,20 +61,21 @@ static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) return -1; } -static int find_nohz_full_cpu(const cpu_set_t *allowed) +static int read_nohz_full_mask(cpu_set_t *mask) { char buf[4096], *cur, *end; FILE *file; + int ret = 0; file = fopen("/sys/devices/system/cpu/nohz_full", "r"); if (!file) - return -1; + return -errno; if (!fgets(buf, sizeof(buf), file)) { - fclose(file); - return -1; + ret = ferror(file) ? -errno : -EINVAL; + goto out; } - fclose(file); + CPU_ZERO(mask); cur = buf; while (*cur) { long first, last; @@ -73,25 +88,30 @@ static int find_nohz_full_cpu(const cpu_set_t *allowed) errno = 0; first = strtol(cur, &end, 10); - if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) - return -1; + if (errno || end == cur || first < 0 || first >= CPU_SETSIZE) { + ret = -EINVAL; + goto out; + } cur = end; last = first; if (*cur == '-') { cur++; errno = 0; last = strtol(cur, &end, 10); - if (errno || end == cur || last < first) - return -1; + if (errno || end == cur || last < first) { + ret = -EINVAL; + goto out; + } cur = end; } - cpu = first_allowed_cpu(allowed, first, last); - if (cpu >= 0) - return cpu; + for (cpu = first; cpu <= last && cpu < CPU_SETSIZE; cpu++) + CPU_SET(cpu, mask); } - return -1; +out: + fclose(file); + return ret; } static pid_t start_worker(int cpu) @@ -132,6 +152,87 @@ static void stop_worker(pid_t pid) waitpid(pid, NULL, 0); } +static struct gated_worker spawn_gated_worker(int cpu) +{ + struct gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +static bool start_gated_worker(struct gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +static void stop_gated_worker(struct gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + stop_worker(worker->pid); + worker->pid = -1; +} + static int pause_worker(pid_t pid) { int status; @@ -162,8 +263,9 @@ static bool wait_for_counter(const u64 *counter, u64 value, int timeout_ms) static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; - cpu_set_t controller_mask; - int cpu; + cpu_set_t controller_mask, nohz_full_mask; + u64 enum_value; + int cpu, i, ret; ctx = calloc(1, sizeof(*ctx)); SCX_FAIL_IF(!ctx, "Failed to allocate context"); @@ -173,15 +275,27 @@ static enum scx_test_status setup(void **ctx_ptr) SCX_FAIL("Failed to get affinity (%d)", errno); } - cpu = find_nohz_full_cpu(&ctx->original_mask); - if (cpu < 0) { + ret = read_nohz_full_mask(&nohz_full_mask); + if (ret) { + fprintf(stderr, "SKIP: failed to read NOHZ_FULL mask (%d)\n", ret); + free(ctx); + return SCX_TEST_SKIP; + } + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &ctx->original_mask) && + CPU_ISSET(cpu, &nohz_full_mask)) + break; + if (cpu == CPU_SETSIZE) { fprintf(stderr, "SKIP: no allowed NOHZ_FULL CPU\n"); free(ctx); return SCX_TEST_SKIP; } controller_mask = ctx->original_mask; - CPU_CLR(cpu, &controller_mask); + for (i = 0; i < CPU_SETSIZE; i++) + if (CPU_ISSET(i, &nohz_full_mask)) + CPU_CLR(i, &controller_mask); if (CPU_COUNT(&controller_mask) == 0) { fprintf(stderr, "SKIP: no housekeeping CPU available\n"); free(ctx); @@ -189,6 +303,13 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + ctx->lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -220,11 +341,15 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; u64 finite_running; u64 finite_ticks; + u64 victim_running; int ret; link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops); @@ -260,7 +385,8 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); @@ -308,7 +434,87 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->lazy_supported) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + usleep(100000); + + if (!start_gated_worker(&challenger) || + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&challenger); + stop_gated_worker(&victim); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + trigger = spawn_gated_worker(ctx->housekeeping_cpu); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + victim_running = __atomic_load_n(&skel->bss->nr_lazy_victim_running, + __ATOMIC_RELAXED); + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, + victim_running + 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + usleep(100000); + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!start_gated_worker(&trigger) || + !wait_for_counter(&skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +527,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); stop_worker(finite_worker); stop_worker(inf_worker); if (link) -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-15 19:45 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi @ 2026-09-16 21:03 ` Tejun Heo 0 siblings, 0 replies; 13+ messages in thread From: Tejun Heo @ 2026-09-16 21:03 UTC (permalink / raw) To: Andrea Righi Cc: Tejun Heo, David Vernet, Changwoo Min, Emil Tsalapatis, Cheng-Yang Chou, sched-ext, linux-kernel Hello, Andrea. This is an AI review. The tests pass here under vng in lazy, full and none modes. The timing below was measured with the target CPU on nohz_full. On Tue, Sep 15, 2026 at 09:45:13PM +0200, Andrea Righi wrote: > +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) > +{ > + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) > + state = KICK_STATE_DONE; > +} In the kick and tick scenarios the victim is the only runnable task on the target CPU, so when its slice is cleared the kernel keeps running it with a refilled slice and ops.stopping() is never called. DONE only arrives when some other task pinned to that CPU wakes up. With the target CPU on nohz_full here, the first such wakeup was a kworker 2.86s after the victim started, against the 3s wait. Without nohz_full the scenarios still spend about half a second each waiting for one. SCX_OPS_ENQ_LAST with the SCX_ENQ_LAST kick as in nohz_tick.bpf.c would put the victim through ops.stopping() on its own. > +SEC("fexit/__resched_curr") > +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) > +{ > + struct task_struct *task = BPF_CORE_READ(rq, curr); > + > + if (!task || BPF_CORE_READ(task, pid) != victim_pid || > + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) > + return 0; This latches the first resched of the victim whatever the cause. A higher class wakeup on the target CPU inside the window, which is the whole 20ms slice in TICK_EXPIRY, records a full slice and fails observation_valid(). Skipping observations with a non-zero slice would be more robust. > + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); > + SCX_GT(lazy_wait.nr_wait_callbacks, 0); DONE is set from ops.stopping() before the context switch and the balance callback runs after it, so this read can race the increment. Poll for it. > + file = fopen("/sys/kernel/debug/sched/preempt", "r"); > + if (!file) { > +#if defined(CONFIG_PREEMPT_LAZY) && !defined(CONFIG_PREEMPT_DYNAMIC) > + return 1; > +#else > + return -1; > +#endif This reflects the build tree, not the running kernel. Also, none and voluntary return -1 and skip the comparison, but the TIFs must match in those modes too. > + skel->rodata->scenario = scenario; > + if (kick__load(skel)) > + goto out; If __resched_curr() isn't in BTF, this fails too and kick_invalid FAILs along with the other four although it doesn't use the tracer. Skipping when the target is missing would be better. > + if (!start_gated_worker(&victim) || > + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, > + PHASE_TIMEOUT_MS)) { > + SCX_ERR("Lazy-enqueue victim was not scheduled"); > + goto out; > + } > + usleep(100000); Neither lazy phase checks that the tick actually stopped before the request, so they pass with the tick running the whole time. > Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy > enqueue and kick requests restart a stopped tick and make forward > progress. Keep invalid kick flag coverage and skip modes not exposed by > the running kernel. kick.c is new, nothing is kept. The controller mask and timeout changes to nohz_tick aren't mentioned either. Thanks. -- tejun ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCHSET v3 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-14 14:44 Andrea Righi
2026-09-14 14:44 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2026-09-14 14:44 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags. They
expire the running sched_ext task's slice while allowing the scheduling boundary
to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_SLICE_EXPIRY selects lazy slice expiry by default. Schedulers can
override that default in either direction for individual tasks through
scx_bpf_task_set_slice_expiry(). The helper preserves sub-scheduler task
ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU, ensuring forward progress even when the
target's tick has stopped.
The selftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task slice-expiry overrides, invalid
flags, and lazy preemption of infinite-slice tasks on NO_HZ_FULL CPUs.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- tools/testing/selftests/sched_ext/runner -t kick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
$ vng -a "preempt=lazy nohz_full=8-15" -- tools/testing/selftests/sched_ext/runner -t nohz_tick
nohz_tick:
CPU 8 received 7 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
Changes in v3:
- Avoid redundant rq clock updates when restoring the tick dependency and use
the pinned rq lock helpers for lazy kick delivery (Sashiko).
- Set PR_SET_PDEATHSIG for the indefinitely spinning kick test workers so they
don't survive unexpected test-runner exit (Sashiko).
- Wait for a delta in nr_lazy_victim_running instead of an absolute count that
may include reschedules from the previous phase (Sashiko).
- Increase the nohz_tick scheduler watchdog and phase timeouts to tolerate
scheduling delays in oversubscribed test VMs.
- Link to v2: https://lore.kernel.org/r/20260914084955.1798562-1-arighi@nvidia.com
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any sched_ext
callback (preserving sub-scheduler task ownership).
- Restore the scheduler tick dependency for lazy enqueue and kick preemption of
infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve their
precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY, immediate preemption, WAIT
and plain kicks take precedence over lazy preemption (Tejun Heo).
- Clear both preemption masks together when processing or skipping a preemption
request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests, per-task
expiry overrides and NO_HZ_FULL coverage.
- Link to v1: https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 153 +++++--
kernel/sched/ext/internal.h | 39 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 13 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 3 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 6 +
tools/sched_ext/include/scx/enums.autogen.h | 2 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 5 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 166 +++++++
tools/testing/selftests/sched_ext/kick.c | 530 ++++++++++++++++++++++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 56 ++-
tools/testing/selftests/sched_ext/nohz_tick.c | 195 +++++++-
14 files changed, 1143 insertions(+), 49 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/kick.c
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-14 14:44 [PATCHSET v3 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-14 14:44 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-14 14:44 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF and the current task slice, while the stopping callback verifies that the task reaches a scheduling boundary. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the expiry default in both directions through scx_bpf_task_set_slice_expiry(). Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy enqueue and kick requests restart a stopped tick and make forward progress. Keep invalid kick flag coverage and skip modes not exposed by the running kernel. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 166 ++++++ tools/testing/selftests/sched_ext/kick.c | 530 ++++++++++++++++++ .../selftests/sched_ext/nohz_tick.bpf.c | 56 +- tools/testing/selftests/sched_ext/nohz_tick.c | 195 ++++++- 5 files changed, 939 insertions(+), 9 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..03ddc44faf5c4 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 target_cpu; +const volatile s32 slice_expiry_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (state == KICK_STATE_QUEUED) { + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + return; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +} + +static void set_slice_expiry_override(struct task_struct *p) +{ + if (p->pid == victim_pid && slice_expiry_override >= 0) + scx_bpf_task_set_slice_expiry(p, slice_expiry_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_slice_expiry_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + + slice_at_resched = BPF_CORE_READ(task, scx.slice); + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..b7af2683c2728 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,530 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <string.h> +#include <sys/prctl.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#include "kick.bpf.skel.h" +#include "scx_test.h" + +#define WAIT_LOOPS 3000 + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +struct victim { + pid_t pid; + int start_fd; +}; + +struct observation { + u64 slice_before; + u64 slice_at_resched; + s32 resched_tif; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static int pick_target_cpu(void) +{ + cpu_set_t mask; + int cpu; + + if (sched_getaffinity(0, sizeof(mask), &mask)) + return -1; + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &mask)) + return cpu; + + return -1; +} + +static struct victim spawn_victim(int cpu) +{ + struct victim victim = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return victim; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return victim; + } + + victim.pid = fork(); + if (!victim.pid) { + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (victim.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return victim; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(victim.pid, SIGKILL); + waitpid(victim.pid, NULL, 0); + victim.pid = -1; + return victim; + } + close(ready[0]); + victim.start_fd = start[1]; + return victim; +} + +static void stop_victim(struct victim *victim) +{ + if (victim->start_fd >= 0) + close(victim->start_fd); + if (victim->pid > 0) { + kill(victim->pid, SIGKILL); + waitpid(victim->pid, NULL, 0); + } +} + +static bool start_victim(struct victim *victim) +{ + char byte = 1; + + if (write(victim->start_fd, &byte, 1) != 1) + return false; + close(victim->start_fd); + victim->start_fd = -1; + return true; +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(u32 scenario, u64 ops_flags, + s32 slice_expiry_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim challenger = { .pid = -1, .start_fd = -1 }; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = pick_target_cpu(); + + if (cpu < 0) { + SCX_ERR("No available CPU"); + return SCX_TEST_FAIL; + } + victim = spawn_victim(cpu); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = spawn_victim(cpu); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->slice_expiry_override = slice_expiry_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!start_victim(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!start_victim(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + stop_victim(&challenger); + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static int active_lazy_mode(void) +{ + char buf[128]; + FILE *file; + + file = fopen("/sys/kernel/debug/sched/preempt", "r"); + if (!file) + return -1; + if (!fgets(buf, sizeof(buf), file)) { + fclose(file); + return -1; + } + fclose(file); + + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(full)")) + return 0; + return -1; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + return SCX_TEST_PASS; +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY")) { + printf("SKIP: SCX_OPS_LAZY_SLICE_EXPIRY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + return setup_lazy(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &obs), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + int lazy_mode; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY, 0, -1, &lazy), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + SCX_ASSERT(observation_valid(&lazy)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_THEN_IMMEDIATE, 0, -1, &lazy_first), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_IMMEDIATE_THEN_LAZY, 0, -1, &immediate_first), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + SCX_EQ(trace_one(KICK_PLAIN_THEN_LAZY, 0, -1, &plain_first), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_THEN_PLAIN, 0, -1, &lazy_then_plain), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + SCX_EQ(trace_one(KICK_BOTH, 0, -1, &both), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_WAIT, 0, -1, &lazy_wait), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + SCX_EQ(trace_one(ENQ_BOTH, 0, -1, &enq_both), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_SLICE_EXPIRY, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + u64 lazy_flag; + int lazy_mode; + + SCX_ASSERT(__COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", &lazy_flag)); + SCX_EQ(trace_one(TICK_EXPIRY, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, lazy_flag, -1, &lazy), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, 0, 1, &force_lazy), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, lazy_flag, 0, &force_immediate), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + SCX_ASSERT(observation_valid(&lazy)); + SCX_ASSERT(observation_valid(&force_lazy)); + SCX_ASSERT(observation_valid(&force_immediate)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = pick_target_cpu(); + int i; + + if (cpu < 0) + return SCX_TEST_FAIL; + victim = spawn_victim(cpu); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !start_victim(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; scenario <= INVALID_KICK_UNKNOWN; scenario++) + SCX_EQ(invalid_one(scenario), SCX_TEST_PASS); + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..96ba1524ca384 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -9,10 +9,23 @@ char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; UEI_DEFINE(uei); @@ -24,9 +37,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,15 +71,22 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() == test_cpu && + phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); } @@ -61,5 +103,5 @@ struct sched_ext_ops nohz_tick_ops = { .tick = (void *)nohz_tick_tick, .exit = (void *)nohz_tick_exit, .name = "nohz_tick", - .timeout_ms = 1000U, + .timeout_ms = 5000U, }; diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..6fd74fd629b85 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -28,12 +28,26 @@ #endif #define MIN_FINITE_TICKS 3 -#define PHASE_TIMEOUT_MS 1000 +#define PHASE_TIMEOUT_MS 5000 struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool lazy_supported; +}; + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +struct gated_worker { + pid_t pid; + int start_fd; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -132,6 +146,87 @@ static void stop_worker(pid_t pid) waitpid(pid, NULL, 0); } +static struct gated_worker spawn_gated_worker(int cpu) +{ + struct gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +static bool start_gated_worker(struct gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +static void stop_gated_worker(struct gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + stop_worker(worker->pid); + worker->pid = -1; +} + static int pause_worker(pid_t pid) { int status; @@ -163,6 +258,7 @@ static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; cpu_set_t controller_mask; + u64 enum_value; int cpu; ctx = calloc(1, sizeof(*ctx)); @@ -189,6 +285,13 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + ctx->lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -220,11 +323,15 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; u64 finite_running; u64 finite_ticks; + u64 victim_running; int ret; link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops); @@ -260,7 +367,8 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); @@ -308,7 +416,87 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->lazy_supported) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + usleep(100000); + + if (!start_gated_worker(&challenger) || + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&challenger); + stop_gated_worker(&victim); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + trigger = spawn_gated_worker(ctx->housekeeping_cpu); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + victim_running = __atomic_load_n(&skel->bss->nr_lazy_victim_running, + __ATOMIC_RELAXED); + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, + victim_running + 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + usleep(100000); + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!start_gated_worker(&trigger) || + !wait_for_counter(&skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +509,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); stop_worker(finite_worker); stop_worker(inf_worker); if (link) -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCHSET v2 sched_ext/for-7.4] sched_ext: Add lazy preemption support
@ 2026-09-14 8:47 Andrea Righi
2026-09-14 8:47 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2026-09-14 8:47 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min
Cc: Emil Tsalapatis, sched-ext, linux-kernel
The fair scheduling class can defer a scheduling boundary through lazy
rescheduling, but sched_ext currently exposes only immediate preemption to BPF
schedulers.
This series adds lazy variants of the enqueue and kick preemption flags. They
expire the running sched_ext task's slice while allowing the scheduling boundary
to be deferred until a return to user space or the next scheduler tick.
SCX_OPS_LAZY_SLICE_EXPIRY selects lazy slice expiry by default. Schedulers can
override that default in either direction for individual tasks through
scx_bpf_task_set_slice_expiry(). The helper preserves sub-scheduler task
ownership boundaries. Bypass continues to force immediate rescheduling.
Lazy enqueue and kick requests also restore the scheduler tick when targeting an
infinite-slice task on a NO_HZ_FULL CPU, ensuring forward progress even when the
target's tick has stopped.
The kselftests cover immediate and lazy preemption, coalescing and precedence
between kick modes, enqueue preemption, per-task slice-expiry overrides, invalid
flags and lazy preemption of infinite-slice tasks on NO_HZ_FULL CPUs.
Tested in separate virtme-ng runs, covering preempt=lazy and nohz_full:
$ vng -a "preempt=lazy" -- tools/testing/selftests/sched_ext/runner -t kick
$ vng -a "preempt=lazy nohz_full=8-15" -- tools/testing/selftests/sched_ext/runner -t nohz_tick
kick:
PASSED: 6
SKIPPED: 0
FAILED: 0
nohz_tick:
CPU 8 received 7 finite-slice ticks
PASSED: 1
SKIPPED: 0
FAILED: 0
Changes in v2:
- Make lazy slice expiry a per-task property (Tejun Heo).
- Add scx_bpf_task_set_slice_expiry() for per-task overrides from any sched_ext
callback (preserving sub-scheduler task ownership).
- Restore the scheduler tick dependency for lazy enqueue and kick preemption of
infinite-slice tasks on NO_HZ_FULL CPUs (Tejun Heo).
- Accumulate immediate and lazy kick requests independently and resolve their
precedence in kick_one_cpu() (Tejun Heo).
- Allow PREEMPT|PREEMPT_LAZY and WAIT|PREEMPT_LAZY, immediate preemption, WAIT
and plain kicks take precedence over lazy preemption (Tejun Heo).
- Clear both preemption masks together when processing or skipping a preemption
request (Tejun Heo).
- Extend the kselftests with combined and ordered kick requests, per-task
expiry overrides and NO_HZ_FULL coverage.
- Link to v1: https://lore.kernel.org/r/20260911195800.974364-1-arighi@nvidia.com
Andrea Righi (2):
sched_ext: Add lazy preemption support
selftests/sched_ext: Add lazy preemption tests
include/linux/sched/ext.h | 8 +
kernel/sched/ext/ext.c | 146 +++++-
kernel/sched/ext/internal.h | 39 +-
kernel/sched/ext/sub.c | 15 +-
tools/sched_ext/include/scx/compat.bpf.h | 13 +
tools/sched_ext/include/scx/enum_defs.autogen.h | 3 +
tools/sched_ext/include/scx/enums.autogen.bpf.h | 6 +
tools/sched_ext/include/scx/enums.autogen.h | 2 +
tools/sched_ext/include/scx/enums_abi.autogen.h | 5 +-
tools/testing/selftests/sched_ext/Makefile | 1 +
tools/testing/selftests/sched_ext/kick.bpf.c | 166 +++++++
tools/testing/selftests/sched_ext/kick.c | 526 ++++++++++++++++++++++
tools/testing/selftests/sched_ext/nohz_tick.bpf.c | 54 ++-
tools/testing/selftests/sched_ext/nohz_tick.c | 189 +++++++-
14 files changed, 1129 insertions(+), 44 deletions(-)
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-14 8:47 [PATCHSET v2 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi @ 2026-09-14 8:47 ` Andrea Righi 2026-09-14 14:55 ` Cheng-Yang Chou 0 siblings, 1 reply; 13+ messages in thread From: Andrea Righi @ 2026-09-14 8:47 UTC (permalink / raw) To: Tejun Heo, David Vernet, Changwoo Min Cc: Emil Tsalapatis, sched-ext, linux-kernel Add trace-based coverage for immediate and lazy preemption. An fexit probe on __resched_curr() records the requested TIF and the current task slice, while the stopping callback verifies that the task reaches a scheduling boundary. Exercise kick requests individually, in both orders and combined in one call. Verify that immediate preemption and WAIT take precedence over lazy preemption, and that combining immediate and lazy enqueue preemption follows the same rule. Also test overriding the expiry default in both directions through scx_bpf_task_set_slice_expiry(). Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy enqueue and kick requests restart a stopped tick and make forward progress. Keep invalid kick flag coverage and skip modes not exposed by the running kernel. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/kick.bpf.c | 166 ++++++ tools/testing/selftests/sched_ext/kick.c | 526 ++++++++++++++++++ .../selftests/sched_ext/nohz_tick.bpf.c | 54 +- tools/testing/selftests/sched_ext/nohz_tick.c | 189 ++++++- 5 files changed, 929 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/kick.bpf.c create mode 100644 tools/testing/selftests/sched_ext/kick.c diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile index 5f5dd9ab903ae..c4ec9b21a4c2a 100644 --- a/tools/testing/selftests/sched_ext/Makefile +++ b/tools/testing/selftests/sched_ext/Makefile @@ -172,6 +172,7 @@ auto-test-targets := \ exit \ hotplug \ init_enable_count \ + kick \ maximal \ maybe_null \ minimal \ diff --git a/tools/testing/selftests/sched_ext/kick.bpf.c b/tools/testing/selftests/sched_ext/kick.bpf.c new file mode 100644 index 0000000000000..03ddc44faf5c4 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.bpf.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +const volatile u32 scenario; +const volatile s32 victim_pid; +const volatile s32 target_cpu; +const volatile s32 slice_expiry_override = -1; + +u32 state; +u64 slice_before; +u64 slice_at_resched; +s32 resched_tif; + +UEI_DEFINE(uei); + +static bool is_trace_scenario(void) +{ + return scenario <= TICK_EXPIRY; +} + +void BPF_STRUCT_OPS(kick_enqueue, struct task_struct *p, u64 enq_flags) +{ + switch (scenario) { + case ENQ_BOTH: + if (p->pid == victim_pid) { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_INF, + enq_flags); + } else if (state == KICK_STATE_QUEUED) { + scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + enq_flags | SCX_ENQ_PREEMPT | + SCX_ENQ_PREEMPT_LAZY); + } else { + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, + enq_flags); + } + return; + case INVALID_KICK_IDLE: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_PREEMPT_LAZY | SCX_KICK_IDLE); + break; + case INVALID_KICK_UNKNOWN: + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 1LLU << 63); + break; + } + + scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); +} + +static void set_slice_expiry_override(struct task_struct *p) +{ + if (p->pid == victim_pid && slice_expiry_override >= 0) + scx_bpf_task_set_slice_expiry(p, slice_expiry_override); +} + +void BPF_STRUCT_OPS(kick_running, struct task_struct *p) +{ + if (!is_trace_scenario() || p->pid != victim_pid) + return; + set_slice_expiry_override(p); + if (__sync_val_compare_and_swap(&state, KICK_STATE_ARMED, KICK_STATE_QUEUED) != + KICK_STATE_ARMED) + return; + + slice_before = p->scx.slice; + + switch (scenario) { + case KICK_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_IMMEDIATE: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + break; + case KICK_IMMEDIATE_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_PLAIN_THEN_LAZY: + scx_bpf_kick_cpu(target_cpu, 0); + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_THEN_PLAIN: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY); + scx_bpf_kick_cpu(target_cpu, 0); + break; + case KICK_BOTH: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT | + SCX_KICK_PREEMPT_LAZY); + break; + case KICK_LAZY_WAIT: + scx_bpf_kick_cpu(target_cpu, SCX_KICK_PREEMPT_LAZY | + SCX_KICK_WAIT); + break; + case ENQ_BOTH: + /* The userspace controller wakes a competing task. */ + break; + case TICK_EXPIRY: + /* no kick: the slice runs out at the tick */ + break; + } +} + +SEC("fexit/__resched_curr") +int BPF_PROG(kick_need_resched, struct rq *rq, int tif) +{ + struct task_struct *task = BPF_CORE_READ(rq, curr); + + if (!task || BPF_CORE_READ(task, pid) != victim_pid || + BPF_CORE_READ(rq, cpu) != target_cpu || state != KICK_STATE_QUEUED) + return 0; + + slice_at_resched = BPF_CORE_READ(task, scx.slice); + resched_tif = tif; + state = KICK_STATE_RESCHED; + return 0; +} + +void BPF_STRUCT_OPS(kick_stopping, struct task_struct *p, bool runnable) +{ + if (p->pid == victim_pid && state == KICK_STATE_RESCHED) + state = KICK_STATE_DONE; +} + +void BPF_STRUCT_OPS(kick_exit, struct scx_exit_info *ei) +{ + UEI_RECORD(uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops kick_ops = { + .enqueue = (void *)kick_enqueue, + .running = (void *)kick_running, + .stopping = (void *)kick_stopping, + .exit = (void *)kick_exit, + .name = "kick", +}; diff --git a/tools/testing/selftests/sched_ext/kick.c b/tools/testing/selftests/sched_ext/kick.c new file mode 100644 index 0000000000000..3fa112a1de607 --- /dev/null +++ b/tools/testing/selftests/sched_ext/kick.c @@ -0,0 +1,526 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + */ +#define _GNU_SOURCE +#include <bpf/bpf.h> +#include <linux/sched.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <string.h> +#include <sys/wait.h> +#include <unistd.h> +#include <scx/common.h> + +#include "kick.bpf.skel.h" +#include "scx_test.h" + +#define WAIT_LOOPS 3000 + +enum kick_scenario { + KICK_IMMEDIATE, + KICK_LAZY, + KICK_LAZY_THEN_IMMEDIATE, + KICK_IMMEDIATE_THEN_LAZY, + KICK_PLAIN_THEN_LAZY, + KICK_LAZY_THEN_PLAIN, + KICK_BOTH, + KICK_LAZY_WAIT, + ENQ_BOTH, + TICK_EXPIRY, + INVALID_KICK_IDLE, + INVALID_KICK_UNKNOWN, +}; + +enum kick_state { + KICK_STATE_IDLE, + KICK_STATE_ARMED, + KICK_STATE_QUEUED, + KICK_STATE_RESCHED, + KICK_STATE_DONE, +}; + +struct victim { + pid_t pid; + int start_fd; +}; + +struct observation { + u64 slice_before; + u64 slice_at_resched; + s32 resched_tif; +}; + +static bool enum_supported(const char *type, const char *name) +{ + u64 value; + + return __COMPAT_read_enum(type, name, &value); +} + +static int pick_target_cpu(void) +{ + cpu_set_t mask; + int cpu; + + if (sched_getaffinity(0, sizeof(mask), &mask)) + return -1; + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, &mask)) + return cpu; + + return -1; +} + +static struct victim spawn_victim(int cpu) +{ + struct victim victim = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + char byte = 1; + + if (pipe(ready)) + return victim; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return victim; + } + + victim.pid = fork(); + if (!victim.pid) { + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (victim.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return victim; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(victim.pid, SIGKILL); + waitpid(victim.pid, NULL, 0); + victim.pid = -1; + return victim; + } + close(ready[0]); + victim.start_fd = start[1]; + return victim; +} + +static void stop_victim(struct victim *victim) +{ + if (victim->start_fd >= 0) + close(victim->start_fd); + if (victim->pid > 0) { + kill(victim->pid, SIGKILL); + waitpid(victim->pid, NULL, 0); + } +} + +static bool start_victim(struct victim *victim) +{ + char byte = 1; + + if (write(victim->start_fd, &byte, 1) != 1) + return false; + close(victim->start_fd); + victim->start_fd = -1; + return true; +} + +static bool wait_for_state(struct kick *skel, u32 wanted) +{ + int i; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (__atomic_load_n(&skel->bss->state, __ATOMIC_ACQUIRE) == wanted) + return true; + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) + return false; + usleep(1000); + } + return false; +} + +static enum scx_test_status trace_one(u32 scenario, u64 ops_flags, + s32 slice_expiry_override, + struct observation *obs) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim challenger = { .pid = -1, .start_fd = -1 }; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = pick_target_cpu(); + + if (cpu < 0) { + SCX_ERR("No available CPU"); + return SCX_TEST_FAIL; + } + victim = spawn_victim(cpu); + if (victim.pid < 0) { + SCX_ERR("Failed to spawn victim"); + return SCX_TEST_FAIL; + } + if (scenario == ENQ_BOTH) { + challenger = spawn_victim(cpu); + if (challenger.pid < 0) { + SCX_ERR("Failed to spawn enqueue challenger"); + goto out; + } + } + + skel = kick__open(); + if (!skel) { + SCX_ERR("Failed to open scenario %u", scenario); + goto out; + } + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + skel->rodata->victim_pid = victim.pid; + skel->rodata->target_cpu = cpu; + skel->rodata->slice_expiry_override = slice_expiry_override; + skel->struct_ops.kick_ops->flags |= ops_flags; + if (kick__load(skel)) { + SCX_ERR("Failed to load scenario %u", scenario); + goto out; + } + + bpf_map__set_autoattach(skel->maps.kick_ops, false); + if (kick__attach(skel)) { + SCX_ERR("Failed to attach __resched_curr tracer"); + goto out; + } + skel->bss->state = KICK_STATE_ARMED; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link) { + SCX_ERR("Failed to attach scenario %u", scenario); + goto out; + } + if (!start_victim(&victim)) { + SCX_ERR("Failed to start victim"); + goto out; + } + if (scenario == ENQ_BOTH) { + if (!wait_for_state(skel, KICK_STATE_QUEUED)) { + SCX_ERR("Enqueue scenario did not arm"); + goto out; + } + if (!start_victim(&challenger)) { + SCX_ERR("Failed to start enqueue challenger"); + goto out; + } + } + if (!wait_for_state(skel, KICK_STATE_DONE)) { + SCX_ERR("Scenario %u stopped in state %u, exit kind %d", scenario, skel->bss->state, + skel->data->uei.kind); + goto out; + } + + obs->slice_before = skel->bss->slice_before; + obs->slice_at_resched = skel->bss->slice_at_resched; + obs->resched_tif = skel->bss->resched_tif; + ret = SCX_TEST_PASS; +out: + stop_victim(&challenger); + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static bool observation_valid(const struct observation *obs) +{ + return obs->slice_before > 0 && !obs->slice_at_resched; +} + +static int active_lazy_mode(void) +{ + char buf[128]; + FILE *file; + + file = fopen("/sys/kernel/debug/sched/preempt", "r"); + if (!file) + return -1; + if (!fgets(buf, sizeof(buf), file)) { + fclose(file); + return -1; + } + fclose(file); + + if (strstr(buf, "(lazy)")) + return 1; + if (strstr(buf, "(full)")) + return 0; + return -1; +} + +static enum scx_test_status setup_immediate(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT")) { + printf("SKIP: SCX_KICK_PREEMPT is not supported\n"); + return SCX_TEST_SKIP; + } + return SCX_TEST_PASS; +} + +static enum scx_test_status setup_lazy(void **ctx) +{ + if (!enum_supported("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY")) { + printf("SKIP: SCX_KICK_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_immediate(ctx); +} + +static enum scx_test_status setup_tick(void **ctx) +{ + if (!enum_supported("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY")) { + printf("SKIP: SCX_OPS_LAZY_SLICE_EXPIRY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_coalesce(void **ctx) +{ + if (!enum_supported("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY")) { + printf("SKIP: SCX_ENQ_PREEMPT_LAZY is not supported\n"); + return SCX_TEST_SKIP; + } + return setup_lazy(ctx); +} + +static enum scx_test_status setup_invalid(void **ctx) +{ + return setup_lazy(ctx); +} + +static enum scx_test_status run_immediate(void *ctx) +{ + struct observation obs; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &obs), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&obs)); + return SCX_TEST_PASS; +} + +static enum scx_test_status run_lazy(void *ctx) +{ + struct observation immediate, lazy; + int lazy_mode; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY, 0, -1, &lazy), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + SCX_ASSERT(observation_valid(&lazy)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy mode used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status run_coalesce(void *ctx) +{ + struct observation immediate, lazy_first, immediate_first; + struct observation plain_first, lazy_then_plain; + struct observation both, lazy_wait, enq_both; + + SCX_EQ(trace_one(KICK_IMMEDIATE, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_THEN_IMMEDIATE, 0, -1, &lazy_first), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_IMMEDIATE_THEN_LAZY, 0, -1, &immediate_first), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&lazy_first)); + SCX_ASSERT(observation_valid(&immediate_first)); + SCX_EQ(lazy_first.resched_tif, immediate.resched_tif); + SCX_EQ(immediate_first.resched_tif, immediate.resched_tif); + + /* + * A plain kick in the same batch doesn't clear the slice by itself. + * The lazy preemption must still expire it, and the plain kick must + * still reschedule immediately, whichever came first. + */ + SCX_EQ(trace_one(KICK_PLAIN_THEN_LAZY, 0, -1, &plain_first), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_THEN_PLAIN, 0, -1, &lazy_then_plain), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&plain_first)); + SCX_ASSERT(observation_valid(&lazy_then_plain)); + SCX_EQ(plain_first.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_then_plain.resched_tif, immediate.resched_tif); + + /* Immediate kick and WAIT both take precedence over lazy preemption. */ + SCX_EQ(trace_one(KICK_BOTH, 0, -1, &both), SCX_TEST_PASS); + SCX_EQ(trace_one(KICK_LAZY_WAIT, 0, -1, &lazy_wait), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&both)); + SCX_ASSERT(observation_valid(&lazy_wait)); + SCX_EQ(both.resched_tif, immediate.resched_tif); + SCX_EQ(lazy_wait.resched_tif, immediate.resched_tif); + + /* Immediate enqueue preemption likewise takes precedence over lazy. */ + SCX_EQ(trace_one(ENQ_BOTH, 0, -1, &enq_both), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&enq_both)); + SCX_EQ(enq_both.resched_tif, immediate.resched_tif); + return SCX_TEST_PASS; +} + +/* + * A slice running out at the tick reschedules immediately by default and + * lazily with SCX_OPS_LAZY_SLICE_EXPIRY, the way fair.c expires a slice. + */ +static enum scx_test_status run_tick(void *ctx) +{ + struct observation immediate, lazy, force_lazy, force_immediate; + u64 lazy_flag; + int lazy_mode; + + SCX_ASSERT(__COMPAT_read_enum("scx_ops_flags", "SCX_OPS_LAZY_SLICE_EXPIRY", &lazy_flag)); + SCX_EQ(trace_one(TICK_EXPIRY, 0, -1, &immediate), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, lazy_flag, -1, &lazy), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, 0, 1, &force_lazy), SCX_TEST_PASS); + SCX_EQ(trace_one(TICK_EXPIRY, lazy_flag, 0, &force_immediate), SCX_TEST_PASS); + SCX_ASSERT(observation_valid(&immediate)); + SCX_ASSERT(observation_valid(&lazy)); + SCX_ASSERT(observation_valid(&force_lazy)); + SCX_ASSERT(observation_valid(&force_immediate)); + + lazy_mode = active_lazy_mode(); + if (lazy_mode > 0) + SCX_FAIL_IF(lazy.resched_tif == immediate.resched_tif, + "Lazy slice expiry used immediate TIF %d", lazy.resched_tif); + else if (!lazy_mode) + SCX_EQ(lazy.resched_tif, immediate.resched_tif); + else + printf("INFO: preemption mode unavailable; lazy TIF was %d, immediate TIF was %d\n", + lazy.resched_tif, immediate.resched_tif); + SCX_EQ(force_lazy.resched_tif, lazy.resched_tif); + SCX_EQ(force_immediate.resched_tif, immediate.resched_tif); + + return SCX_TEST_PASS; +} + +static enum scx_test_status invalid_one(u32 scenario) +{ + struct bpf_link *ops_link = NULL; + struct kick *skel = NULL; + struct victim victim; + enum scx_test_status ret = SCX_TEST_FAIL; + int cpu = pick_target_cpu(); + int i; + + if (cpu < 0) + return SCX_TEST_FAIL; + victim = spawn_victim(cpu); + if (victim.pid < 0) + return SCX_TEST_FAIL; + + skel = kick__open(); + if (!skel) + goto out; + SCX_ENUM_INIT(skel); + skel->rodata->scenario = scenario; + if (kick__load(skel)) + goto out; + ops_link = bpf_map__attach_struct_ops(skel->maps.kick_ops); + if (!ops_link || !start_victim(&victim)) + goto out; + + for (i = 0; i < WAIT_LOOPS; i++) { + if (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR)) { + ret = SCX_TEST_PASS; + break; + } + usleep(1000); + } +out: + stop_victim(&victim); + if (ops_link) + bpf_link__destroy(ops_link); + if (skel) + kick__destroy(skel); + return ret; +} + +static enum scx_test_status run_invalid(void *ctx) +{ + u32 scenario; + + for (scenario = INVALID_KICK_IDLE; scenario <= INVALID_KICK_UNKNOWN; scenario++) + SCX_EQ(invalid_one(scenario), SCX_TEST_PASS); + return SCX_TEST_PASS; +} + +static struct scx_test kick_immediate = { + .name = "kick_immediate", + .description = "Trace immediate kick slice expiration and rescheduling", + .setup = setup_immediate, + .run = run_immediate, +}; + +static struct scx_test kick_lazy = { + .name = "kick_lazy", + .description = "Trace lazy kick slice expiration and rescheduling", + .setup = setup_lazy, + .run = run_lazy, +}; + +static struct scx_test kick_coalesce = { + .name = "kick_coalesce", + .description = "Verify lazy preemption coalesces with immediate requests", + .setup = setup_coalesce, + .run = run_coalesce, +}; + +static struct scx_test kick_tick = { + .name = "kick_tick", + .description = "Trace slice expiry at the tick, immediate and lazy", + .setup = setup_tick, + .run = run_tick, +}; + +static struct scx_test kick_invalid = { + .name = "kick_invalid", + .description = "Verify invalid kick flag combinations fail", + .setup = setup_invalid, + .run = run_invalid, +}; + +__attribute__((constructor)) +static void register_kick_tests(void) +{ + scx_test_register(&kick_immediate); + scx_test_register(&kick_lazy); + scx_test_register(&kick_coalesce); + scx_test_register(&kick_tick); + scx_test_register(&kick_invalid); +} diff --git a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c index 6998c5dd6bcb9..fc89079d19e79 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.bpf.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.bpf.c @@ -9,10 +9,23 @@ char _license[] SEC("license") = "GPL"; const volatile s32 test_cpu; -bool finite_phase; +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +u32 phase; +s32 victim_pid; +s32 challenger_pid; +s32 trigger_pid; u64 nr_inf_running; u64 nr_finite_running; u64 nr_finite_ticks; +u64 nr_lazy_victim_running; +u64 nr_lazy_enq_running; +u64 nr_lazy_kick_running; UEI_DEFINE(uei); @@ -24,9 +37,31 @@ s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu, void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags) { - u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF; + u64 slice; + u64 dsq_id = SCX_DSQ_GLOBAL; + + switch (phase) { + case NOHZ_PHASE_INF: + slice = SCX_SLICE_INF; + break; + case NOHZ_PHASE_FINITE: + slice = 1000000ULL; + break; + case NOHZ_PHASE_LAZY_ENQ: + case NOHZ_PHASE_LAZY_KICK: + dsq_id = SCX_DSQ_LOCAL; + slice = p->pid == victim_pid ? SCX_SLICE_INF : SCX_SLICE_DFL; + if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + enq_flags |= SCX_ENQ_PREEMPT_LAZY; + break; + default: + slice = SCX_SLICE_DFL; + break; + } - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags); + scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags); + if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == trigger_pid) + scx_bpf_kick_cpu(test_cpu, SCX_KICK_PREEMPT_LAZY); if (enq_flags & SCX_ENQ_LAST) scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE); } @@ -36,15 +71,22 @@ void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p) if (bpf_get_smp_processor_id() != test_cpu) return; - if (finite_phase) + if (phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_running, 1); - else + else if (phase == NOHZ_PHASE_INF) __sync_fetch_and_add(&nr_inf_running, 1); + else if (p->pid == victim_pid) + __sync_fetch_and_add(&nr_lazy_victim_running, 1); + else if (phase == NOHZ_PHASE_LAZY_ENQ && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_enq_running, 1); + else if (phase == NOHZ_PHASE_LAZY_KICK && p->pid == challenger_pid) + __sync_fetch_and_add(&nr_lazy_kick_running, 1); } void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p) { - if (bpf_get_smp_processor_id() == test_cpu && finite_phase) + if (bpf_get_smp_processor_id() == test_cpu && + phase == NOHZ_PHASE_FINITE) __sync_fetch_and_add(&nr_finite_ticks, 1); } diff --git a/tools/testing/selftests/sched_ext/nohz_tick.c b/tools/testing/selftests/sched_ext/nohz_tick.c index 028f54391c2ca..a5266a5412935 100644 --- a/tools/testing/selftests/sched_ext/nohz_tick.c +++ b/tools/testing/selftests/sched_ext/nohz_tick.c @@ -34,6 +34,20 @@ struct nohz_tick_ctx { struct nohz_tick *skel; cpu_set_t original_mask; int test_cpu; + int housekeeping_cpu; + bool lazy_supported; +}; + +enum nohz_phase { + NOHZ_PHASE_INF, + NOHZ_PHASE_FINITE, + NOHZ_PHASE_LAZY_ENQ, + NOHZ_PHASE_LAZY_KICK, +}; + +struct gated_worker { + pid_t pid; + int start_fd; }; static int first_allowed_cpu(const cpu_set_t *mask, int first, int last) @@ -132,6 +146,87 @@ static void stop_worker(pid_t pid) waitpid(pid, NULL, 0); } +static struct gated_worker spawn_gated_worker(int cpu) +{ + struct gated_worker worker = { .pid = -1, .start_fd = -1 }; + int ready[2], start[2]; + pid_t parent = getpid(); + char byte = 1; + + if (pipe(ready)) + return worker; + if (pipe(start)) { + close(ready[0]); + close(ready[1]); + return worker; + } + + worker.pid = fork(); + if (!worker.pid) { + struct sched_param param = {}; + cpu_set_t mask; + + close(ready[0]); + close(start[1]); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) + _exit(1); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + _exit(1); + if (sched_setscheduler(0, SCHED_EXT, ¶m)) + _exit(1); + if (write(ready[1], &byte, 1) != 1) + _exit(1); + close(ready[1]); + if (read(start[0], &byte, 1) != 1) + _exit(1); + close(start[0]); + for (;;) + asm volatile("" ::: "memory"); + } + if (worker.pid < 0) { + close(ready[0]); + close(ready[1]); + close(start[0]); + close(start[1]); + return worker; + } + + close(ready[1]); + close(start[0]); + if (read(ready[0], &byte, 1) != 1) { + close(ready[0]); + close(start[1]); + kill(worker.pid, SIGKILL); + waitpid(worker.pid, NULL, 0); + worker.pid = -1; + return worker; + } + close(ready[0]); + worker.start_fd = start[1]; + return worker; +} + +static bool start_gated_worker(struct gated_worker *worker) +{ + char byte = 1; + + if (write(worker->start_fd, &byte, 1) != 1) + return false; + close(worker->start_fd); + worker->start_fd = -1; + return true; +} + +static void stop_gated_worker(struct gated_worker *worker) +{ + if (worker->start_fd >= 0) + close(worker->start_fd); + stop_worker(worker->pid); + worker->pid = -1; +} + static int pause_worker(pid_t pid) { int status; @@ -163,6 +258,7 @@ static enum scx_test_status setup(void **ctx_ptr) { struct nohz_tick_ctx *ctx; cpu_set_t controller_mask; + u64 enum_value; int cpu; ctx = calloc(1, sizeof(*ctx)); @@ -189,6 +285,13 @@ static enum scx_test_status setup(void **ctx_ptr) } ctx->test_cpu = cpu; + ctx->housekeeping_cpu = first_allowed_cpu(&controller_mask, 0, + CPU_SETSIZE - 1); + ctx->lazy_supported = + __COMPAT_read_enum("scx_enq_flags", "SCX_ENQ_PREEMPT_LAZY", + &enum_value) && + __COMPAT_read_enum("scx_kick_flags", "SCX_KICK_PREEMPT_LAZY", + &enum_value); ctx->skel = nohz_tick__open(); if (!ctx->skel) { free(ctx); @@ -220,6 +323,9 @@ static enum scx_test_status run(void *ctx_ptr) struct nohz_tick_ctx *ctx = ctx_ptr; struct nohz_tick *skel = ctx->skel; struct bpf_link *link = NULL; + struct gated_worker victim = { .pid = -1, .start_fd = -1 }; + struct gated_worker challenger = { .pid = -1, .start_fd = -1 }; + struct gated_worker trigger = { .pid = -1, .start_fd = -1 }; enum scx_test_status status = SCX_TEST_FAIL; pid_t finite_worker = -1; pid_t inf_worker = -1; @@ -260,7 +366,8 @@ static enum scx_test_status run(void *ctx_ptr) /* * The next EXT task receives a finite slice and must restart the tick. */ - __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE); + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_FINITE, + __ATOMIC_RELEASE); finite_worker = start_worker(ctx->test_cpu); if (finite_worker < 0) { SCX_ERR("Failed to start finite-slice worker (%d)", errno); @@ -308,7 +415,84 @@ static enum scx_test_status run(void *ctx_ptr) finite_ticks)); goto out; } + stop_worker(finite_worker); + finite_worker = -1; + stop_worker(inf_worker); + inf_worker = -1; + if (!ctx->lazy_supported) + goto check_exit; + + /* + * A lazy local enqueue must restart the tick after clearing the slice of + * an infinite-slice task on a full-dynticks CPU. + */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_ENQ, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + if (victim.pid < 0 || challenger.pid < 0) { + SCX_ERR("Failed to spawn lazy-enqueue workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-enqueue victim was not scheduled"); + goto out; + } + usleep(100000); + + if (!start_gated_worker(&challenger) || + !wait_for_counter(&skel->bss->nr_lazy_enq_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy enqueue made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&challenger); + stop_gated_worker(&victim); + + /* Repeat with a lazy kick delivered from a housekeeping CPU. */ + __atomic_store_n(&skel->bss->phase, NOHZ_PHASE_LAZY_KICK, + __ATOMIC_RELEASE); + victim = spawn_gated_worker(ctx->test_cpu); + challenger = spawn_gated_worker(ctx->test_cpu); + trigger = spawn_gated_worker(ctx->housekeeping_cpu); + if (victim.pid < 0 || challenger.pid < 0 || trigger.pid < 0) { + SCX_ERR("Failed to spawn lazy-kick workers"); + goto out; + } + skel->bss->victim_pid = victim.pid; + skel->bss->challenger_pid = challenger.pid; + skel->bss->trigger_pid = trigger.pid; + if (!start_gated_worker(&victim) || + !wait_for_counter(&skel->bss->nr_lazy_victim_running, 2, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy-kick victim was not scheduled"); + goto out; + } + if (!start_gated_worker(&challenger)) { + SCX_ERR("Failed to start lazy-kick challenger"); + goto out; + } + usleep(100000); + if (__atomic_load_n(&skel->bss->nr_lazy_kick_running, + __ATOMIC_RELAXED)) { + SCX_ERR("Lazy-kick challenger ran before the kick"); + goto out; + } + if (!start_gated_worker(&trigger) || + !wait_for_counter(&skel->bss->nr_lazy_kick_running, 1, + PHASE_TIMEOUT_MS)) { + SCX_ERR("Lazy kick made no progress on CPU %d", ctx->test_cpu); + goto out; + } + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); +check_exit: if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) { SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)", (unsigned long long)skel->data->uei.kind, @@ -321,6 +505,9 @@ static enum scx_test_status run(void *ctx_ptr) (unsigned long long)skel->bss->nr_finite_ticks); status = SCX_TEST_PASS; out: + stop_gated_worker(&trigger); + stop_gated_worker(&challenger); + stop_gated_worker(&victim); stop_worker(finite_worker); stop_worker(inf_worker); if (link) -- 2.55.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-14 8:47 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi @ 2026-09-14 14:55 ` Cheng-Yang Chou 2026-09-15 13:35 ` Andrea Righi 0 siblings, 1 reply; 13+ messages in thread From: Cheng-Yang Chou @ 2026-09-14 14:55 UTC (permalink / raw) To: Andrea Righi Cc: Tejun Heo, David Vernet, Changwoo Min, Emil Tsalapatis, sched-ext, linux-kernel, Ching-Chun Huang, Chia-Ping Tsai, chengyang.chou Hi Andrea, On Mon, Sep 14, 2026 at 10:47:46AM +0200, Andrea Righi wrote: > Add trace-based coverage for immediate and lazy preemption. An fexit > probe on __resched_curr() records the requested TIF and the current task > slice, while the stopping callback verifies that the task reaches a > scheduling boundary. > > Exercise kick requests individually, in both orders and combined in one > call. Verify that immediate preemption and WAIT take precedence over > lazy preemption, and that combining immediate and lazy enqueue > preemption follows the same rule. Also test overriding the expiry > default in both directions through scx_bpf_task_set_slice_expiry(). > > Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy > enqueue and kick requests restart a stopped tick and make forward > progress. Keep invalid kick flag coverage and skip modes not exposed by > the running kernel. > > Signed-off-by: Andrea Righi <arighi@nvidia.com> [...] Didn't have time to go through the whole series, but I applied it on for-next and built with PREEMPT_LAZY=y and PREEMPT_DYNAMIC=n: INFO: preemption mode unavailable; lazy TIF was 5, immediate TIF was 4 ok 2 kick_lazy # INFO: preemption mode unavailable; lazy TIF was 5, immediate TIF was 4 ok 4 kick_tick # PASSED: 6 > +static int active_lazy_mode(void) > +{ > + char buf[128]; > + FILE *file; > + > + file = fopen("/sys/kernel/debug/sched/preempt", "r"); > + if (!file) > + return -1; sched_init_debug() only creates that file under CONFIG_PREEMPT_DYNAMIC, and PREEMPT_LAZY can be built without it ("select PREEMPT_BUILD if !PREEMPT_DYNAMIC"), so the two tests pass without comparing the TIFs. Would something like this work? if (!file) { #if defined(CONFIG_PREEMPT_LAZY) && !defined(CONFIG_PREEMPT_DYNAMIC) return 1; #else return -1; #endif } Thanks. > + if (!fgets(buf, sizeof(buf), file)) { > + fclose(file); > + return -1; > + } > + fclose(file); > + > + if (strstr(buf, "(lazy)")) > + return 1; > + if (strstr(buf, "(full)")) > + return 0; > + return -1; > +} > + -- Cheers, Cheng-Yang ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests 2026-09-14 14:55 ` Cheng-Yang Chou @ 2026-09-15 13:35 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2026-09-15 13:35 UTC (permalink / raw) To: Cheng-Yang Chou Cc: Tejun Heo, David Vernet, Changwoo Min, Emil Tsalapatis, sched-ext, linux-kernel, Ching-Chun Huang, Chia-Ping Tsai, chengyang.chou On Mon, Sep 14, 2026 at 10:55:56PM +0800, Cheng-Yang Chou wrote: > Hi Andrea, > > On Mon, Sep 14, 2026 at 10:47:46AM +0200, Andrea Righi wrote: > > Add trace-based coverage for immediate and lazy preemption. An fexit > > probe on __resched_curr() records the requested TIF and the current task > > slice, while the stopping callback verifies that the task reaches a > > scheduling boundary. > > > > Exercise kick requests individually, in both orders and combined in one > > call. Verify that immediate preemption and WAIT take precedence over > > lazy preemption, and that combining immediate and lazy enqueue > > preemption follows the same rule. Also test overriding the expiry > > default in both directions through scx_bpf_task_set_slice_expiry(). > > > > Extend the NO_HZ_FULL test with infinite-slice victims. Verify that lazy > > enqueue and kick requests restart a stopped tick and make forward > > progress. Keep invalid kick flag coverage and skip modes not exposed by > > the running kernel. > > > > Signed-off-by: Andrea Righi <arighi@nvidia.com> > > [...] > > Didn't have time to go through the whole series, but I applied it on > for-next and built with PREEMPT_LAZY=y and PREEMPT_DYNAMIC=n: > > INFO: preemption mode unavailable; lazy TIF was 5, immediate TIF was 4 > ok 2 kick_lazy # > INFO: preemption mode unavailable; lazy TIF was 5, immediate TIF was 4 > ok 4 kick_tick # > PASSED: 6 > > > +static int active_lazy_mode(void) > > +{ > > + char buf[128]; > > + FILE *file; > > + > > + file = fopen("/sys/kernel/debug/sched/preempt", "r"); > > + if (!file) > > + return -1; > > sched_init_debug() only creates that file under CONFIG_PREEMPT_DYNAMIC, > and PREEMPT_LAZY can be built without it ("select PREEMPT_BUILD if > !PREEMPT_DYNAMIC"), so the two tests pass without comparing the TIFs. > > Would something like this work? > > if (!file) { > #if defined(CONFIG_PREEMPT_LAZY) && !defined(CONFIG_PREEMPT_DYNAMIC) > return 1; > #else > return -1; > #endif Ah yes, makes sense, we can include autoconf.h and check that. I'll include this fix in the next version. Thanks, -Andrea > } > > Thanks. > > > + if (!fgets(buf, sizeof(buf), file)) { > > + fclose(file); > > + return -1; > > + } > > + fclose(file); > > + > > + if (strstr(buf, "(lazy)")) > > + return 1; > > + if (strstr(buf, "(full)")) > > + return 0; > > + return -1; > > +} > > + > > -- > Cheers, > Cheng-Yang ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2026-09-18 17:16 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-09-15 9:00 [PATCHSET v4 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-15 9:00 ` [PATCH 1/2] " Andrea Righi 2026-09-15 9:00 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi -- strict thread matches above, loose matches on Subject: below -- 2026-09-18 17:13 [PATCHSET v7 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-18 17:13 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 2026-09-17 7:02 [PATCHSET v6 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-17 7:02 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 2026-09-17 19:19 ` Tejun Heo 2026-09-18 6:30 ` Andrea Righi 2026-09-15 19:45 [PATCHSET v5 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-15 19:45 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 2026-09-16 21:03 ` Tejun Heo 2026-09-14 14:44 [PATCHSET v3 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-14 14:44 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 2026-09-14 8:47 [PATCHSET v2 sched_ext/for-7.4] sched_ext: Add lazy preemption support Andrea Righi 2026-09-14 8:47 ` [PATCH 2/2] selftests/sched_ext: Add lazy preemption tests Andrea Righi 2026-09-14 14:55 ` Cheng-Yang Chou 2026-09-15 13:35 ` Andrea Righi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®