* [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-18 3:11 ` John Stultz
2026-07-16 13:20 ` [PATCH 02/11] sched: Add helper to block retained proxy donors Andrea Righi
` (9 subsequent siblings)
10 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Proxy execution separates the scheduling context in rq->donor from the
physical execution context in rq->curr. sched_can_stop_tick() checks the
latter for CFS bandwidth constraints and only does so when nr_running is
one.
A retained proxy donor keeps both the donor and mutex owner queued. The
check therefore misses a constrained FAIR donor and may stop the tick
while its runtime still needs to be enforced.
Check the selected donor instead and remove the nr_running restriction.
The donor being a queued FAIR task is sufficient to require bandwidth
accounting regardless of other runnable tasks.
Fixes: af0c8b2bf67b ("sched: Split scheduler and execution contexts")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/r/20260713164807.E5ED21F00A3A@smtp.kernel.org
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/core.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 3cc6fb1d20547..f47bcfb25033a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1409,11 +1409,8 @@ static void nohz_csd_func(void *info)
#endif /* CONFIG_NO_HZ_COMMON */
#ifdef CONFIG_NO_HZ_FULL
-static inline bool __need_bw_check(struct rq *rq, struct task_struct *p)
+static inline bool __need_bw_check(struct task_struct *p)
{
- if (rq->nr_running != 1)
- return false;
-
if (p->sched_class != &fair_sched_class)
return false;
@@ -1462,14 +1459,17 @@ bool sched_can_stop_tick(struct rq *rq)
return false;
/*
- * If there is one task and it has CFS runtime bandwidth constraints
- * and it's on the cpu now we don't want to stop the tick.
+ * If the selected scheduling context has CFS runtime bandwidth
+ * constraints, don't stop the tick. With proxy execution, rq->donor is
+ * the selected scheduling context while rq->curr is the task physically
+ * executing on its behalf.
+ *
* This check prevents clearing the bit if a newly enqueued task here is
- * dequeued by migrating while the constrained task continues to run.
+ * dequeued by migrating while the constrained donor continues to run.
* E.g. going from 2->1 without going through pick_next_task().
*/
- if (__need_bw_check(rq, rq->curr)) {
- if (cfs_task_bw_constrained(rq->curr))
+ if (__need_bw_check(rq->donor)) {
+ if (cfs_task_bw_constrained(rq->donor))
return false;
}
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor
2026-07-16 13:20 ` [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor Andrea Righi
@ 2026-07-18 3:11 ` John Stultz
0 siblings, 0 replies; 23+ messages in thread
From: John Stultz @ 2026-07-18 3:11 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Thu, Jul 16, 2026 at 6:22 AM Andrea Righi <arighi@nvidia.com> wrote:
>
> Proxy execution separates the scheduling context in rq->donor from the
> physical execution context in rq->curr. sched_can_stop_tick() checks the
> latter for CFS bandwidth constraints and only does so when nr_running is
> one.
>
> A retained proxy donor keeps both the donor and mutex owner queued. The
> check therefore misses a constrained FAIR donor and may stop the tick
> while its runtime still needs to be enforced.
>
> Check the selected donor instead and remove the nr_running restriction.
> The donor being a queued FAIR task is sufficient to require bandwidth
> accounting regardless of other runnable tasks.
>
> Fixes: af0c8b2bf67b ("sched: Split scheduler and execution contexts")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Link: https://lore.kernel.org/r/20260713164807.E5ED21F00A3A@smtp.kernel.org
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
Acked-by: John Stultz <jstultz@google.com>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 02/11] sched: Add helper to block retained proxy donors
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
2026-07-16 13:20 ` [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 13:20 ` [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions Andrea Righi
` (8 subsequent siblings)
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Scheduler ownership changes may need to turn a mutex-blocked task
retained on the runqueue for proxy execution back into a normally
blocked task.
Add sched_proxy_block_task() to perform that transition while holding
p->pi_lock and the task's runqueue lock. Move an active donor off the
CPU first, and complete any delayed dequeue so a following sched_change
cannot preserve and re-enqueue the task.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/core.c | 39 +++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 6 ++++++
2 files changed, 45 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f47bcfb25033a..e7827cffc2f0c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6764,6 +6764,45 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
block_task(rq, donor, state);
}
+/*
+ * Remove a retained proxy donor before changing its scheduler ownership.
+ * The caller holds p->pi_lock, so p cannot wake and migrate after block_task()
+ * drops it from the runqueue.
+ *
+ * Unlike the regular schedule() path, this must leave @p fully dequeued.
+ * DELAY_DEQUEUE may keep a blocked fair task queued with sched_delayed set,
+ * which would let the following sched_change preserve and re-enqueue it under
+ * the new scheduler. Complete an existing or newly-created delayed dequeue
+ * before returning.
+ */
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
+{
+ unsigned long state = READ_ONCE(p->__state);
+
+ lockdep_assert_held(&p->pi_lock);
+ lockdep_assert_rq_held(rq);
+
+ if (!p->is_blocked || !task_on_rq_queued(p))
+ return;
+ if (WARN_ON_ONCE(state == TASK_RUNNING))
+ return;
+
+ if (task_current_donor(rq, p)) {
+ proxy_resched_idle(rq);
+ /* Kick the execution context if @rq is remote. */
+ resched_curr(rq);
+ }
+
+ if (!p->se.sched_delayed)
+ block_task(rq, p, state);
+ if (p->se.sched_delayed)
+ dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED |
+ DEQUEUE_NOCLOCK);
+
+ WARN_ON_ONCE(task_on_rq_queued(p));
+ WARN_ON_ONCE(p->se.sched_delayed);
+}
+
static inline void proxy_release_rq_lock(struct rq *rq, struct rq_flags *rf)
__releases(__rq_lockp(rq))
{
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 64d79e9efc3d1..437c4f68c9e53 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2477,6 +2477,12 @@ static inline bool task_is_blocked(struct task_struct *p)
return !!p->blocked_on;
}
+#ifdef CONFIG_SCHED_PROXY_EXEC
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p);
+#else
+static inline void sched_proxy_block_task(struct rq *rq, struct task_struct *p) {}
+#endif
+
static inline int task_on_cpu(struct rq *rq, struct task_struct *p)
{
return p->on_cpu;
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
2026-07-16 13:20 ` [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor Andrea Righi
2026-07-16 13:20 ` [PATCH 02/11] sched: Add helper to block retained proxy donors Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-18 3:16 ` John Stultz
2026-07-16 13:20 ` [PATCH 04/11] sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors Andrea Righi
` (7 subsequent siblings)
10 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Proxy execution retains mutex-blocked donors on the runqueue so their
scheduling context can execute a lock owner. sched_ext cannot safely
retain such donors unless the BPF scheduler explicitly participates in
their admission and ordering.
Make sched_ext reject retained donors by default. Force blocked EXT
tasks through the regular block path in schedule(), and fully deactivate
a retained donor before sched_setscheduler(), PI de-boosting, or global
activation moves it into the EXT class. These hooks establish the safe
default that a later opt-in can relax.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/core.c | 5 ++++-
kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++++
kernel/sched/ext/ext.h | 6 ++++++
kernel/sched/syscalls.c | 3 +++
4 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e7827cffc2f0c..ddc832035854b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7173,7 +7173,8 @@ static void __sched notrace __schedule(int sched_mode)
* task_is_blocked() will always be false).
*/
try_to_block_task(rq, prev, &prev_state,
- !task_is_blocked(prev));
+ !task_is_blocked(prev) ||
+ !scx_allow_proxy_exec(prev));
switch_count = &prev->nvcsw;
}
@@ -7726,6 +7727,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
if (prev_class != next_class)
queue_flag |= DEQUEUE_CLASS;
+ scx_prepare_setscheduler(p, next_class);
+
scoped_guard (sched_change, p, queue_flag) {
/*
* Boosting condition are:
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index b3b8cf95e0f7a..cd43283e54d8d 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -24,6 +24,46 @@
DEFINE_RAW_SPINLOCK(scx_sched_lock);
+bool scx_allow_proxy_exec(const struct task_struct *p)
+{
+ return p->sched_class != &ext_sched_class;
+}
+
+/*
+ * Called after sched_setscheduler() validation and immediately before
+ * sched_change_begin(), with @p's pi and rq locks held.
+ */
+void scx_prepare_setscheduler(struct task_struct *p,
+ const struct sched_class *next_class)
+{
+ lockdep_assert_held(&p->pi_lock);
+ lockdep_assert_rq_held(task_rq(p));
+
+ /*
+ * Retained proxy donors need admission only when entering EXT. A PI
+ * boost moves an EXT task to RT/DL and may keep it queued; the matching
+ * de-boost moves it back to EXT and therefore falls through below.
+ */
+ if (p->sched_class == next_class || next_class != &ext_sched_class)
+ return;
+
+ sched_proxy_block_task(task_rq(p), p);
+}
+
+/*
+ * Called with @p's pi and rq locks held immediately before
+ * sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock
+ * is updated only once.
+ */
+static void scx_prepare_task_sched_change(struct task_struct *p)
+{
+ lockdep_assert_held(&p->pi_lock);
+ lockdep_assert_rq_held(task_rq(p));
+
+ update_rq_clock(task_rq(p));
+ sched_proxy_block_task(task_rq(p), p);
+}
+
/*
* NOTE: sched_ext is in the process of growing multiple scheduler support and
* scx_root usage is in a transitional state. Naked dereferences are safe if the
@@ -7303,6 +7343,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
if (old_class != new_class)
queue_flags |= DEQUEUE_CLASS;
+ if (new_class == &ext_sched_class) {
+ scx_prepare_task_sched_change(p);
+ queue_flags |= DEQUEUE_NOCLOCK;
+ }
scoped_guard (sched_change, p, queue_flags) {
set_task_slice(p, READ_ONCE(sch->slice_dfl));
diff --git a/kernel/sched/ext/ext.h b/kernel/sched/ext/ext.h
index 0b7fc46aee08c..d708abf2c3bb8 100644
--- a/kernel/sched/ext/ext.h
+++ b/kernel/sched/ext/ext.h
@@ -18,8 +18,11 @@ bool scx_can_stop_tick(struct rq *rq);
void scx_rq_activate(struct rq *rq);
void scx_rq_deactivate(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
+void scx_prepare_setscheduler(struct task_struct *p,
+ const struct sched_class *next_class);
bool task_should_scx(int policy);
bool scx_allow_ttwu_queue(const struct task_struct *p);
+bool scx_allow_proxy_exec(const struct task_struct *p);
void init_sched_ext_class(void);
static inline u32 scx_cpuperf_target(s32 cpu)
@@ -52,8 +55,11 @@ static inline bool scx_can_stop_tick(struct rq *rq) { return true; }
static inline void scx_rq_activate(struct rq *rq) {}
static inline void scx_rq_deactivate(struct rq *rq) {}
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
+static inline void scx_prepare_setscheduler(struct task_struct *p,
+ const struct sched_class *next_class) {}
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
+static inline bool scx_allow_proxy_exec(const struct task_struct *p) { return true; }
static inline void init_sched_ext_class(void) {}
#endif /* CONFIG_SCHED_CLASS_EXT */
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b215b0ead9a60..2bbba3dc8c890 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -678,6 +678,9 @@ int __sched_setscheduler(struct task_struct *p,
if (prev_class != next_class)
queue_flags |= DEQUEUE_CLASS;
+ if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS))
+ scx_prepare_setscheduler(p, next_class);
+
scoped_guard (sched_change, p, queue_flags) {
if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions
2026-07-16 13:20 ` [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions Andrea Righi
@ 2026-07-18 3:16 ` John Stultz
0 siblings, 0 replies; 23+ messages in thread
From: John Stultz @ 2026-07-18 3:16 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
>
> Proxy execution retains mutex-blocked donors on the runqueue so their
> scheduling context can execute a lock owner. sched_ext cannot safely
> retain such donors unless the BPF scheduler explicitly participates in
> their admission and ordering.
>
> Make sched_ext reject retained donors by default. Force blocked EXT
> tasks through the regular block path in schedule(), and fully deactivate
> a retained donor before sched_setscheduler(), PI de-boosting, or global
> activation moves it into the EXT class. These hooks establish the safe
> default that a later opt-in can relax.
>
> This is a preparatory change to support proxy execution with sched_ext.
>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> ---
> kernel/sched/core.c | 5 ++++-
> kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++++
> kernel/sched/ext/ext.h | 6 ++++++
> kernel/sched/syscalls.c | 3 +++
> 4 files changed, 57 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index e7827cffc2f0c..ddc832035854b 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7173,7 +7173,8 @@ static void __sched notrace __schedule(int sched_mode)
> * task_is_blocked() will always be false).
> */
> try_to_block_task(rq, prev, &prev_state,
> - !task_is_blocked(prev));
> + !task_is_blocked(prev) ||
> + !scx_allow_proxy_exec(prev));
> switch_count = &prev->nvcsw;
> }
This reminds me that since we no longer re-use try_to_block_task()
elsewhere, we can probably push the checks down into the function and
get rid of teh should_block argument.
But this is a minor cleanup for later, not an issue with this patch.
This looks good to me!
Acked-by: John Stultz <jstultz@google.com>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 04/11] sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (2 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 13:20 ` [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task() Andrea Righi
` (6 subsequent siblings)
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
With proxy-exec, pick_next_task() can return a task with p->is_blocked
set (a proxy donor). put_prev_set_next_task() then calls
set_next_task_scx() on this "ghost" task even though the task only
provides scheduling context and never actually runs.
Calling ops.running() for such a donor produces a spurious running
event. Simply suppressing ops.running() is not sufficient because the
following put_prev_task_scx() would still invoke ops.stopping(),
resulting in an unpaired stopping event.
Introduce SCX_TASK_RUN_TRACKED to track whether a task entered an
ops.running()/stopping() session. Set and clear the flag independently
of ops.running() and ops.stopping(), as the callbacks are independently
optional. Invoke ops.running() only for non-blocked tasks and invoke
ops.stopping() only after entering a tracked session. This keeps the
callbacks paired for proxy donors while preserving stopping
notifications for schedulers which only implement ops.stopping().
This is a preparatory change for enabling proxy execution together with
sched_ext. The explicit running-state tracking is also required by later
donor-based accounting: it prevents an EXT owner executing for a non-EXT
donor from being treated as the active EXT scheduling context when it is
dequeued.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
include/linux/sched/ext.h | 2 ++
kernel/sched/ext/ext.c | 33 +++++++++++++++++++++++++--------
2 files changed, 27 insertions(+), 8 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 803da0f1e5092..5da12d6562b8e 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -103,6 +103,8 @@ enum scx_ent_flags {
SCX_TASK_SUB_INIT = 1 << 4, /* task being initialized for a sub sched */
SCX_TASK_IMMED = 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
+ SCX_TASK_RUN_TRACKED = 1 << 6, /* task is in an ops.running()/stopping() session */
+
/*
* Bits 8 to 10 are used to carry task state:
*
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index cd43283e54d8d..ecef4cd9be1fa 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -2209,9 +2209,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
* information meaningful to the BPF scheduler and can be suppressed by
* skipping the callbacks if the task is !QUEUED.
*/
- if (SCX_HAS_OP(sch, stopping) && task_current(rq, p)) {
- update_curr_scx(rq);
- SCX_CALL_OP_TASK(sch, stopping, rq, p, false);
+ if (task_current(rq, p) &&
+ (p->scx.flags & SCX_TASK_RUN_TRACKED)) {
+ if (SCX_HAS_OP(sch, stopping)) {
+ update_curr_scx(rq);
+ SCX_CALL_OP_TASK(sch, stopping, rq, p, false);
+ }
+ p->scx.flags &= ~SCX_TASK_RUN_TRACKED;
}
if (SCX_HAS_OP(sch, quiescent) && !task_on_rq_migrating(p))
@@ -2913,9 +2917,17 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
p->se.exec_start = rq_clock_task(rq);
- /* see dequeue_task_scx() on why we skip when !QUEUED */
- if (SCX_HAS_OP(sch, running) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP_TASK(sch, running, rq, p);
+ /*
+ * See dequeue_task_scx() for why we skip when !QUEUED. A blocked proxy
+ * donor is also skipped because it provides scheduling context but never
+ * runs itself.
+ */
+ if ((p->scx.flags & SCX_TASK_QUEUED) && !p->is_blocked) {
+ if (SCX_HAS_OP(sch, running))
+ SCX_CALL_OP_TASK(sch, running, rq, p);
+
+ p->scx.flags |= SCX_TASK_RUN_TRACKED;
+ }
clr_task_runnable(p, true);
@@ -3021,8 +3033,13 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
update_curr_scx(rq);
/* see dequeue_task_scx() on why we skip when !QUEUED */
- if (SCX_HAS_OP(sch, stopping) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP_TASK(sch, stopping, rq, p, true);
+ if ((p->scx.flags & SCX_TASK_QUEUED) &&
+ (p->scx.flags & SCX_TASK_RUN_TRACKED)) {
+ if (SCX_HAS_OP(sch, stopping))
+ SCX_CALL_OP_TASK(sch, stopping, rq, p, true);
+
+ p->scx.flags &= ~SCX_TASK_RUN_TRACKED;
+ }
if (p->scx.flags & SCX_TASK_QUEUED) {
set_task_runnable(rq, p);
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (3 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 04/11] sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 21:29 ` Tejun Heo
2026-07-16 13:20 ` [PATCH 06/11] sched_ext: Split curr|donor references properly Andrea Righi
` (5 subsequent siblings)
10 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
When pulling a task from a non-local DSQ, scx_consume_dispatch_q()
filters candidates without holding the task's source rq lock;
consume_remote_task() then unlinks the selected task, drops the DSQ and
destination rq locks and acquires the source rq lock before migrating
it.
Whether the task may be remotely migrated can change across this
handoff. In particular, proxy execution allows a task to execute as
rq->curr under another task's scheduling context while its own
scheduling context remains on a DSQ. The task can therefore become
on-CPU or migration disabled without dequeueing its DSQ entry or
clearing holding_cpu.
With proxy execution enabled, a mutex-intensive workload such as
stress-ng --pipeherd 0 can easily trigger this race when a donor becomes
active on its source rq during the unlocked window. Migrating the active
execution context can make it resume with IRQ and preemption state from
the wrong scheduling path, triggering sleeping-while-atomic warnings and
subsequent lockdep corruption.
Fix this by splitting the validation into two helpers:
task_can_run_on_remote_rq() remains the preliminary lockless filter used
while scanning a DSQ; task_can_move_from_locked_rq() is used at actual
migration points, it requires the task's source rq lock, rejects the
task if it is still on-CPU and then applies the existing destination
checks.
After consume_remote_task() has switched to the source rq lock, run the
locked check again immediately before move_remote_task_to_local_dsq().
If the check fails, the task has already been removed from its original
DSQ, so cancel the transfer and enqueue it on the global DSQ. Use a
non-enforcing check, because the failure results from a kernel-side
race, not an invalid BPF placement request.
Deferring this check until the source rq is locked both closes the
migration race and ensures the task remains discoverable by an idle
CPU.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/ext/ext.c | 56 +++++++++++++++++++++++++++++++++++-------
1 file changed, 47 insertions(+), 9 deletions(-)
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index ecef4cd9be1fa..08773f0aca5a6 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -2364,8 +2364,10 @@ static void move_remote_task_to_local_dsq(struct scx_sched *sch,
* - The BPF scheduler is bypassed while the rq is offline and we can always say
* no to the BPF scheduler initiated migrations while offline.
*
- * The caller must ensure that @p and @rq are on different CPUs.
- * If enforce == true, caller must hold @p's rq lock.
+ * The caller must ensure that @p and @rq are on different CPUs. If @enforce is
+ * true, report violations attributable to BPF-directed migrations. The caller
+ * must hold @p's rq lock to avoid reporting a transient race as a scheduler
+ * error.
*/
static bool task_can_run_on_remote_rq(struct scx_sched *sch,
struct task_struct *p, struct rq *rq,
@@ -2373,11 +2375,6 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
{
s32 cpu = cpu_of(rq);
- /*
- * To prevent races with @p still running on its old CPU while switching
- * out, make sure we're holding @p's rq lock so as not to risk
- * erroneously killing the BPF scheduler.
- */
if (enforce)
lockdep_assert_rq_held(task_rq(p));
@@ -2424,6 +2421,25 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
return true;
}
+/*
+ * Final migration check with @p's rq locked. Unlike the lockless DSQ scan,
+ * this can safely reject transient execution state without losing a kick.
+ */
+static bool task_can_move_from_locked_rq(struct scx_sched *sch,
+ struct task_struct *p, struct rq *dst_rq,
+ bool enforce)
+{
+ struct rq *src_rq = task_rq(p);
+
+ lockdep_assert_rq_held(src_rq);
+
+ /* @p may be rq->curr under another task's proxy scheduling context. */
+ if (task_on_cpu(src_rq, p))
+ return false;
+
+ return task_can_run_on_remote_rq(sch, p, dst_rq, enforce);
+}
+
/**
* unlink_dsq_and_switch_rq_lock() - Unlink task and switch to its rq lock
* @p: target task
@@ -2481,6 +2497,28 @@ static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq,
struct scx_dispatch_q *dsq, struct rq *src_rq)
{
if (unlink_dsq_and_switch_rq_lock(p, dsq, this_rq, src_rq)) {
+ /*
+ * Whether @p may be migrated to @this_rq may have changed while
+ * switching rq locks. This is a kernel-side race, not an invalid BPF
+ * placement request, so don't abort the scheduler on failure. Fall
+ * back to the global DSQ, where normal consumption filters can select
+ * a valid destination without forcing the task onto the source local
+ * DSQ.
+ */
+ if (unlikely(!task_can_move_from_locked_rq(sch, p, this_rq, false))) {
+ p->scx.dsq = NULL;
+ p->scx.holding_cpu = -1;
+ scx_dispatch_enqueue(sch, src_rq,
+ find_global_dsq(sch, task_cpu(p)), p,
+ enq_flags | SCX_ENQ_CLEAR_OPSS |
+ SCX_ENQ_GDSQ_FALLBACK);
+ if (sched_class_above(p->sched_class,
+ src_rq->donor->sched_class))
+ resched_curr(src_rq);
+ switch_rq_lock(src_rq, this_rq);
+ return false;
+ }
+
move_remote_task_to_local_dsq(sch, p, enq_flags, src_rq, this_rq);
return true;
} else {
@@ -2519,7 +2557,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,
if (dst_dsq->id == SCX_DSQ_LOCAL) {
dst_rq = container_of(dst_dsq, struct rq, scx.local_dsq);
if (src_rq != dst_rq &&
- unlikely(!task_can_run_on_remote_rq(sch, p, dst_rq, true))) {
+ unlikely(!task_can_move_from_locked_rq(sch, p, dst_rq, true))) {
dst_dsq = find_global_dsq(sch, task_cpu(p));
dst_rq = src_rq;
enq_flags |= SCX_ENQ_GDSQ_FALLBACK;
@@ -2682,7 +2720,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
p->scx.holding_cpu = -1;
scx_dispatch_enqueue(sch, dst_rq, &dst_rq->scx.local_dsq, p,
enq_flags);
- } else if (unlikely(!task_can_run_on_remote_rq(sch, p, dst_rq, true))) {
+ } else if (unlikely(!task_can_move_from_locked_rq(sch, p, dst_rq, true))) {
p->scx.holding_cpu = -1;
fallback = true;
scx_dispatch_enqueue(sch, src_rq, find_global_dsq(sch, task_cpu(p)),
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
2026-07-16 13:20 ` [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task() Andrea Righi
@ 2026-07-16 21:29 ` Tejun Heo
2026-07-16 21:38 ` Tejun Heo
0 siblings, 1 reply; 23+ messages in thread
From: Tejun Heo @ 2026-07-16 21:29 UTC (permalink / raw)
To: Andrea Righi
Cc: David Vernet, Changwoo Min, John Stultz, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
Hello,
On Thu, Jul 16, 2026 at 03:20:40PM +0200, Andrea Righi wrote:
> When pulling a task from a non-local DSQ, scx_consume_dispatch_q()
> filters candidates without holding the task's source rq lock;
> consume_remote_task() then unlinks the selected task, drops the DSQ and
> destination rq locks and acquires the source rq lock before migrating
> it.
>
> Whether the task may be remotely migrated can change across this
> handoff. In particular, proxy execution allows a task to execute as
> rq->curr under another task's scheduling context while its own
> scheduling context remains on a DSQ. The task can therefore become
> on-CPU or migration disabled without dequeueing its DSQ entry or
> clearing holding_cpu.
I don't think this can happen w/o proxy execution. The task is enqueued and
can't be dequeued or start running without grabbing the DSQ lock. Before the
DSQ lock is dropped, p->scx.holding_cpu is set and if that hasn't been
cleared, we know that nothing else has touched while we were switching
locks.
If this no longer holds w/ proxy execution, adding further conditions are
fine but let's make it explicitly for and confined to proxy execution
enabled cases. The logic there is already subtle and I want to avoid adding
misleading checks.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
2026-07-16 21:29 ` Tejun Heo
@ 2026-07-16 21:38 ` Tejun Heo
2026-07-17 6:35 ` Andrea Righi
0 siblings, 1 reply; 23+ messages in thread
From: Tejun Heo @ 2026-07-16 21:38 UTC (permalink / raw)
To: Andrea Righi
Cc: David Vernet, Changwoo Min, John Stultz, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Thu, Jul 16, 2026 at 11:29:37AM -1000, Tejun Heo wrote:
> If this no longer holds w/ proxy execution, adding further conditions are
> fine but let's make it explicitly for and confined to proxy execution
> enabled cases. The logic there is already subtle and I want to avoid adding
> misleading checks.
Thought a bit more about it and the only thing that can change are the
running state and migration disabled, right? So, maybe just test those again
after the lock switch?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
2026-07-16 21:38 ` Tejun Heo
@ 2026-07-17 6:35 ` Andrea Righi
0 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-17 6:35 UTC (permalink / raw)
To: Tejun Heo
Cc: David Vernet, Changwoo Min, John Stultz, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
Hi Tejun,
On Thu, Jul 16, 2026 at 11:38:34AM -1000, Tejun Heo wrote:
> On Thu, Jul 16, 2026 at 11:29:37AM -1000, Tejun Heo wrote:
> > If this no longer holds w/ proxy execution, adding further conditions are
> > fine but let's make it explicitly for and confined to proxy execution
> > enabled cases. The logic there is already subtle and I want to avoid adding
> > misleading checks.
>
> Thought a bit more about it and the only thing that can change are the
> running state and migration disabled, right? So, maybe just test those again
> after the lock switch?
Right, without proxy execution, the DSQ lock and holding_cpu handshake guarantee
that with an unchanged holding_cpu the task could not have been dequeued or
started running during the rq-lock handoff.
Proxy execution is the exception: a task can start physically executing as a
lock owner while its own scheduling context remains on the DSQ, allowing its
on-CPU or migration-disabled state to change without clearing holding_cpu.
I'll leave the existing validation unchanged and, only when proxy execution is
enabled, recheck the running and migration-disabled state after switching to the
source rq lock.
Thanks for taking a look!
-Andrea
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 06/11] sched_ext: Split curr|donor references properly
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (4 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task() Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 13:20 ` [PATCH 07/11] sched_ext: Handle blocked donor migration with proxy execution Andrea Righi
` (4 subsequent siblings)
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
With proxy execution, the task selected by the scheduler and the task
physically executing can differ. A blocked mutex waiter donates its
scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T
[donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class,
priority, and runtime budget, while O supplies the execution context: O
is the task whose code physically executes. T is a competing runnable
task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR
can be replaced with a higher scheduling class such as RT or deadline
without changing the class interaction described here. The possible
combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes
with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because
EXT is below FAIR. Once T stops, BPF can dispatch D and O executes
with D's EXT priority and runtime budget. If T becomes runnable
again, it preempts the D/O proxy execution.
3. D is EXT, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
4. D is EXT, O is FAIR, T is FAIR
D cannot boost O above T because EXT is below FAIR. O and T
continue competing under FAIR. Once O releases M, D wakes and
resumes normal EXT scheduling.
5. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes
with D's FAIR priority and runtime budget, while T waits in EXT.
D is not visible to the BPF scheduler.
6. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is
selected, O executes with D's FAIR priority and runtime budget.
D is not visible to the BPF scheduler.
7. D is FAIR, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
8. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable
as a blocked proxy donor. When CFS selects D, O executes using D's
FAIR scheduling context. When CFS selects O, O executes using its
own FAIR context, and when CFS selects T, T executes normally. D
is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the
scheduler-selected task which supplies the scheduling context, rather
than rq->curr, the task whose code physically executes. Without proxy
execution they are the same task.
On nohz_full CPUs, a blocked proxy donor must retain the scheduler tick
even when it has an infinite slice. Otherwise, a full dynticks CPU could
stop the tick while rq->curr and rq->donor differ, violating assumptions
made by the remote NOHZ tick path.
This is a conservative compromise that keeps the change local to
sched_ext, at the cost of a periodic tick while a blocked proxy donor is
selected. Allowing blocked proxy donors to run tickless would require
making the core scheduler's remote tick handling aware that rq->curr and
rq->donor can differ.
Moreover, extend scx_dump_state() to report both contexts. Each CPU
record now includes a donor= line. If an EXT donor differs from
rq->curr, also emit its detailed task record. The existing '*' marker
continues to identify rq->curr, while the donor= line identifies the
otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still
depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext
cannot be enabled together. The scheduling changes are therefore
preparatory. A later patch removes this restriction.
Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/ext/ext.c | 93 +++++++++++++++++++++++++++---------------
kernel/sched/ext/sub.h | 8 ++--
2 files changed, 64 insertions(+), 37 deletions(-)
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 08773f0aca5a6..c924af485224f 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -1333,20 +1333,27 @@ static void apply_task_slice_oob(struct rq *rq, struct task_struct *p)
static void update_curr_scx(struct rq *rq)
{
- struct task_struct *curr = rq->curr;
+ struct task_struct *donor;
s64 delta_exec;
+ /*
+ * update_curr_scx() is selected through rq->donor->sched_class, not
+ * rq->curr->sched_class, so @donor is always an EXT task here. If an EXT
+ * owner executes for a FAIR donor, FAIR's update_curr() runs instead.
+ */
+ donor = rq->donor;
+
/* apply even on 0 delta_exec, callers may still act on the slice */
- apply_task_slice_oob(rq, curr);
+ apply_task_slice_oob(rq, donor);
delta_exec = update_curr_common(rq);
if (unlikely(delta_exec <= 0))
return;
- if (curr->scx.slice != SCX_SLICE_INF) {
- curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
- if (!curr->scx.slice)
- touch_core_sched(rq, curr);
+ if (donor->scx.slice != SCX_SLICE_INF) {
+ donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
+ if (!donor->scx.slice)
+ touch_core_sched(rq, donor);
}
dl_server_update(&rq->ext_server, delta_exec);
@@ -1516,9 +1523,9 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,
if (rq->scx.flags & SCX_RQ_IN_BALANCE)
return;
- if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
- rq->curr->sched_class == &ext_sched_class) {
- set_task_slice(rq->curr, 0);
+ if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->donor &&
+ rq->donor->sched_class == &ext_sched_class) {
+ set_task_slice(rq->donor, 0);
resched_curr(rq);
}
}
@@ -2732,7 +2739,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
}
/* if the destination CPU is idle, wake it up */
- if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class))
+ if (!fallback && sched_class_above(p->sched_class,
+ dst_rq->donor->sched_class))
resched_curr(dst_rq);
}
@@ -2943,6 +2951,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
{
struct scx_sched *sch = scx_task_sched(p);
+ bool can_stop_tick;
if (p->scx.flags & SCX_TASK_QUEUED) {
/*
@@ -2971,6 +2980,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
/* apply any pending out-of-band slice request before the tick decision */
apply_task_slice_oob(rq, p);
+ can_stop_tick = p->scx.slice == SCX_SLICE_INF && !p->is_blocked;
/*
* @p is getting newly scheduled or got kicked after someone updated its
@@ -2981,7 +2991,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
* nohz. In the future, we might want to add a mechanism to update
* load_avgs periodically on tick-stopped CPUs.
*/
- if (p->scx.slice == SCX_SLICE_INF) {
+ if (can_stop_tick) {
if (!(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
/*
* Bypass mode always assigns finite slices, so @p
@@ -3002,7 +3012,8 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
/*
* @rq still references the outgoing scheduling context. A finite
- * slice is sufficient by itself to require the tick.
+ * slice or a blocked proxy donor is sufficient by itself to require
+ * the tick.
*/
if (tick_nohz_full_cpu(cpu_of(rq)))
tick_nohz_dep_set_cpu(cpu_of(rq), TICK_DEP_BIT_SCHED);
@@ -3177,7 +3188,7 @@ static struct task_struct *first_local_task(struct rq *rq)
static struct task_struct *
do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
{
- struct task_struct *prev = rq->curr;
+ struct task_struct *prev = rq->donor;
bool keep_prev;
struct task_struct *p;
@@ -3537,9 +3548,9 @@ void scx_tick(struct rq *rq)
update_other_load_avgs(rq);
}
-static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+static void task_tick_scx(struct rq *rq, struct task_struct *donor, int queued)
{
- struct scx_sched *sch = scx_task_sched(curr);
+ struct scx_sched *sch = scx_task_sched(donor);
update_curr_scx(rq);
@@ -3548,13 +3559,13 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
* we can't trust the slice management or ops.core_sched_before().
*/
if (scx_bypassing(sch, cpu_of(rq))) {
- set_task_slice(curr, 0);
- touch_core_sched(rq, curr);
+ set_task_slice(donor, 0);
+ touch_core_sched(rq, donor);
} else if (SCX_HAS_OP(sch, tick)) {
- SCX_CALL_OP_TASK(sch, tick, rq, curr);
+ SCX_CALL_OP_TASK(sch, tick, rq, donor);
}
- if (!curr->scx.slice)
+ if (!donor->scx.slice)
resched_curr(rq);
}
@@ -4165,16 +4176,16 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
}
/*
- * The revoke that scheduled this scan may have raced the pick: curr
+ * The revoke that scheduled this scan may have raced the pick: donor
* may be a now-capless task, either one that kept running or one
* promoted off the local DSQ between the ecaps sync and this scan.
* Zero the slice to evict it. The enqueue gate blocks new capless
* inserts, so no later pick can slip through after the scan.
*/
if ((reenq_flags & SCX_REENQ_CAP_REVOKE) &&
- rq->curr->sched_class == &ext_sched_class &&
- scx_task_reenq_on_cap_revoke(rq, rq->curr)) {
- set_task_slice(rq->curr, 0);
+ rq->donor->sched_class == &ext_sched_class &&
+ scx_task_reenq_on_cap_revoke(rq, rq->donor)) {
+ set_task_slice(rq->donor, 0);
resched_curr(rq);
}
@@ -4360,14 +4371,14 @@ static void run_deferred(struct rq *rq)
#ifdef CONFIG_NO_HZ_FULL
bool scx_can_stop_tick(struct rq *rq)
{
- struct task_struct *p = rq->curr;
+ struct task_struct *p = rq->donor;
struct scx_sched *sch = scx_task_sched(p);
if (p->sched_class != &ext_sched_class)
return true;
/*
- * @rq->curr may still reference an outgoing EXT task after it has been
+ * @rq->donor may still reference an outgoing EXT task after it has been
* dequeued. If no EXT tasks are accounted on @rq, ignore its stale
* slice state. If another task is dispatched from a DSQ,
* set_next_task_scx() will update the dependency for the incoming task.
@@ -4381,7 +4392,8 @@ bool scx_can_stop_tick(struct rq *rq)
/*
* @rq can dispatch from different DSQs, so we can't tell whether it
* needs the tick or not by looking at nr_running. Allow stopping ticks
- * iff the BPF scheduler indicated so. See set_next_task_scx().
+ * iff set_next_task_scx() determined that the selected scheduling context
+ * can run tickless.
*/
return rq->scx.flags & SCX_RQ_CAN_STOP_TICK;
}
@@ -6480,6 +6492,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
+ dump_line(&ns, " donor=%s[%d] class=%ps",
+ rq->donor->comm, rq->donor->pid,
+ rq->donor->sched_class);
if (!cpumask_empty(pcpu->cpus_to_kick))
dump_line(&ns, " cpus_to_kick : %*pb",
cpumask_pr_args(pcpu->cpus_to_kick));
@@ -6523,6 +6538,10 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
if (rq->curr->sched_class == &ext_sched_class &&
(dump_all_tasks || scx_task_on_sched(sch, rq->curr)))
scx_dump_task(sch, s, dctx, rq, rq->curr, '*');
+ if (rq->donor != rq->curr &&
+ rq->donor->sched_class == &ext_sched_class &&
+ (dump_all_tasks || scx_task_on_sched(sch, rq->donor)))
+ scx_dump_task(sch, s, dctx, rq, rq->donor, ' ');
list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node)
if (dump_all_tasks || scx_task_on_sched(sch, p))
@@ -8035,7 +8054,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r
unsigned long flags;
raw_spin_rq_lock_irqsave(rq, flags);
- cur_class = rq->curr->sched_class;
+ cur_class = rq->donor->sched_class;
/*
* During CPU hotplug, a CPU may depend on kicking itself to make
@@ -8052,7 +8071,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r
if (cur_class == &ext_sched_class) {
if (likely(!scx_missing_caps(pcpu->sch, cpu,
scx_caps_for_preempt(pcpu->sch, rq))))
- set_task_slice(rq->curr, 0);
+ set_task_slice(rq->donor, 0);
else
__scx_add_event(pcpu->sch,
SCX_EV_SUB_PREEMPT_DENIED, 1);
@@ -9864,12 +9883,16 @@ __bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask)
}
/**
- * scx_bpf_task_running - Is task currently running?
+ * scx_bpf_task_running - Is task the current scheduling context?
* @p: task of interest
+ *
+ * Under proxy execution, this reports the donor rather than the task whose
+ * code is physically executing. ops.running() continues to report when the
+ * task itself starts executing.
*/
__bpf_kfunc bool scx_bpf_task_running(const struct task_struct *p)
{
- return task_rq(p)->curr == p;
+ return rcu_access_pointer(task_rq(p)->donor) == p;
}
/**
@@ -9926,10 +9949,14 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux)
}
/**
- * scx_bpf_cpu_curr - Return remote CPU's curr task
+ * scx_bpf_cpu_curr - Return remote CPU's current scheduling context
* @cpu: CPU of interest
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
+ * Under proxy execution, this returns the donor, which supplies the scheduling
+ * policy and runtime budget, rather than the task whose code is physically
+ * executing. ops.running() continues to report physical task execution.
+ *
* Callers must hold RCU read lock (KF_RCU).
*/
__bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_aux *aux)
@@ -9945,7 +9972,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_
if (!scx_cpu_valid(sch, cpu, NULL))
return NULL;
- return rcu_dereference(cpu_rq(cpu)->curr);
+ return rcu_dereference(cpu_rq(cpu)->donor);
}
/**
@@ -9969,7 +9996,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cid_curr(s32 cid, const struct bpf_prog_
cpu = scx_cid_to_cpu(sch, cid);
if (cpu < 0)
return NULL;
- return rcu_dereference(cpu_rq(cpu)->curr);
+ return rcu_dereference(cpu_rq(cpu)->donor);
}
/**
diff --git a/kernel/sched/ext/sub.h b/kernel/sched/ext/sub.h
index 625d7ce334aa8..a164e6b2f2562 100644
--- a/kernel/sched/ext/sub.h
+++ b/kernel/sched/ext/sub.h
@@ -141,14 +141,14 @@ static inline u64 scx_caps_for_task(struct task_struct *p)
return SCX_CAP_ENQ;
}
-/* the cap @sch needs to preempt @rq's current task, 0 if none */
+/* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */
static inline u64 scx_caps_for_preempt(struct scx_sched *sch, struct rq *rq)
{
- struct task_struct *curr = rq->curr;
+ struct task_struct *donor = rq->donor;
/* a non-ext task can't be preempted by ext, own-subtree needs no cap */
- if (curr->sched_class != &ext_sched_class ||
- scx_is_descendant(scx_task_sched(curr), sch))
+ if (donor->sched_class != &ext_sched_class ||
+ scx_is_descendant(scx_task_sched(donor), sch))
return 0;
return SCX_CAP_PREEMPT;
}
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH 07/11] sched_ext: Handle blocked donor migration with proxy execution
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (5 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 06/11] sched_ext: Split curr|donor references properly Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 13:20 ` [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers Andrea Righi
` (3 subsequent siblings)
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
From: John Stultz <jstultz@google.com>
With proxy execution enabled, mutex-blocked donors stay runnable so
their scheduling context can execute the lock owner.
sched_ext may normally relocate a queued blocked donor. set_task_cpu()
updates both task_cpu() and wake_cpu, making the destination rq the
donor's new callback home. This remains valid if the donor was previously
proxy-migrated: the BPF scheduler is intentionally selecting a new return
destination, which the next proxy migration will preserve. The usual
migration-disabled, affinity, and rq-online checks apply.
An active donor cannot be moved normally because the source rq still
references it through rq->donor for scheduling-class operations and
runtime accounting. Moving it would associate the task with a different
rq while leaving those source-rq references in place. The core proxy
migration path avoids this by switching the rq donor to idle before moving
the scheduling context.
Check active-donor state in task_can_move_from_locked_rq(), where the
source rq is held and the result is authoritative. Keep it out of the
preliminary lockless DSQ filter so a donor transition during the rq-lock
handoff is resolved by the locked recheck rather than consuming a dispatch
kick and skipping the task.
Allow normal sched_ext migration of blocked donors unless the donor is
active on its rq. In that case, defer migration until it is switched out
or leave it to the proxy machinery.
Keep blocked donors on the local DSQ when they are put so they remain
visible to the proxy pick path.
Signed-off-by: John Stultz <jstultz@google.com>
Co-developed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
kernel/sched/ext/ext.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index c924af485224f..02e3ea7add16f 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -2444,6 +2444,10 @@ static bool task_can_move_from_locked_rq(struct scx_sched *sch,
if (task_on_cpu(src_rq, p))
return false;
+ /* Don't move a donor while its source rq still references it. */
+ if (p->is_blocked && rcu_access_pointer(src_rq->donor) == p)
+ return false;
+
return task_can_run_on_remote_rq(sch, p, dst_rq, enforce);
}
@@ -3093,6 +3097,22 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
if (p->scx.flags & SCX_TASK_QUEUED) {
set_task_runnable(rq, p);
+ /*
+ * Mutex-blocked donors stay queued on the runqueue under proxy
+ * execution, but the donor never runs as itself, proxy-exec
+ * walks the blocked_on chain on the next __schedule() and runs
+ * the lock owner in its place.
+ *
+ * Put the donor on the local DSQ directly so pick_next_task()
+ * can still see it. find_proxy_task() will either run the chain
+ * owner or deactivate the donor so the wakeup path can return it
+ * and let BPF make a new dispatch decision once it is unblocked.
+ */
+ if (p->is_blocked) {
+ scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
+ goto switch_class;
+ }
+
/*
* If @p has slice left and is being put, @p is getting
* preempted by a higher priority scheduler class or core-sched
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (6 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 07/11] sched_ext: Handle blocked donor migration with proxy execution Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-18 6:16 ` John Stultz
2026-07-16 13:20 ` [PATCH 09/11] sched_ext: Add selftest for blocked donor admission Andrea Righi
` (2 subsequent siblings)
10 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Proxy execution keeps a blocked donor runnable so its scheduling context
can execute the mutex owner. Dispatching sched_ext donors on a local
DSQ bypasses the BPF scheduler ordering policy and can give donors
more CPU priority than intended to perform the proxy execution handoff.
Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks
owned by schedulers without the flag block normally. Schedulers with the
flag receive blocked donors through ops.enqueue() with SCX_ENQ_BLOCKED
set in enq_flags and can apply their own admission policy.
From a high-level perspective, the resulting flow for a BPF scheduler
with SCX_OPS_ENQ_BLOCKED is:
D ------ blocked on -----> M ------ owned by -----> O
[donor] [mutex] [owner]
|
| ops.enqueue(D, SCX_ENQ_BLOCKED)
| BPF dispatches D to CPUi
v
+-----------------+
| CPUi local DSQ |
+-------+---------+
|
| pick_next_task() selects D
v
+-----------------+
| proxy exec | move D to O's rq
+-------+---------+
|
| run O using D's scheduling context
v
rq->curr = O
rq->donor = D
|
| O releases M
v
+-----------------+
| proxy exec | return D to CPUi via wakeup
+-----------------+
A proxy migration preserves the donor's wake_cpu while moving its
scheduling context to the owner's rq, so task_cpu() differs from wake_cpu
until the donor wakes. Prevent BPF-directed migrations from pulling a
blocked donor off this proxy rq. Otherwise each blocked re-enqueue may
move the donor back to its callback rq only for proxy execution to move
it to the owner again.
Scheduler ownership can change after a donor has already blocked. Since
sched_change preserves queued state, handle both iterator-driven moves
to root or sub-schedulers and sched_setscheduler() transitions into EXT.
Before entering a scheduler without the flag, deactivate the retained
donor. It remains on the mutex wait path and wakes normally when the
mutex becomes available.
The global sched_ext enable state can change while __schedule() holds
the runqueue lock. Once an EXT task has an assigned scheduler, consult
it directly so the task cannot retain a donor during the transition
unless the scheduler opted in. Tasks without an assigned scheduler keep
the generic proxy execution behavior.
The preparation helpers require and assert that p->pi_lock and p's rq
lock are held before updating rq state.
The donor starts associated with its original CPU. A BPF scheduler may
dispatch it directly into that CPU local DSQ to let the core resolve the
mutex owner and execute it with the donor scheduling context.
A normal sleeper also has p->is_blocked set until after
wakeup_preempt(). Exclude full wakeup activations from SCX_ENQ_BLOCKED
and carry a transient task flag from enqueue_task_scx() to
wakeup_preempt_scx(). Retained donor wakeups skip enqueue_task_scx(), so
they remain distinguishable without changing core scheduler wake flags.
Reschedule a retained donor when its mutex wakes it so ops.dispatch()
can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED
override the exiting and migration-disabled enqueue fallbacks so
opted-in schedulers receive all eligible donor requests.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
include/linux/sched/ext.h | 1 +
kernel/sched/ext/ext.c | 109 ++++++++++++++----
kernel/sched/ext/internal.h | 23 +++-
kernel/sched/ext/sub.c | 12 +-
tools/sched_ext/include/scx/compat.h | 1 +
.../sched_ext/include/scx/enum_defs.autogen.h | 1 +
.../sched_ext/include/scx/enums.autogen.bpf.h | 3 +
tools/sched_ext/include/scx/enums.autogen.h | 1 +
8 files changed, 127 insertions(+), 24 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 5da12d6562b8e..82d47a046a4f5 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -104,6 +104,7 @@ enum scx_ent_flags {
SCX_TASK_IMMED = 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
SCX_TASK_RUN_TRACKED = 1 << 6, /* task is in an ops.running()/stopping() session */
+ SCX_TASK_ENQ_WAKEUP = 1 << 7, /* wakeup enqueue awaiting wakeup_preempt() */
/*
* Bits 8 to 10 are used to carry task state:
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 02e3ea7add16f..3456be01d5548 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -26,7 +26,18 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
bool scx_allow_proxy_exec(const struct task_struct *p)
{
- return p->sched_class != &ext_sched_class;
+ struct scx_sched *sch;
+
+ if (p->sched_class != &ext_sched_class)
+ return true;
+
+ /*
+ * scx_enabled() may change while __schedule() holds only @p's rq lock.
+ * Once @p is associated with a scheduler, use that scheduler's policy
+ * even while the global enable state is transitioning.
+ */
+ sch = scx_task_sched(p);
+ return !sch || (sch->ops.flags & SCX_OPS_ENQ_BLOCKED);
}
/*
@@ -36,6 +47,8 @@ bool scx_allow_proxy_exec(const struct task_struct *p)
void scx_prepare_setscheduler(struct task_struct *p,
const struct sched_class *next_class)
{
+ struct scx_sched *sch;
+
lockdep_assert_held(&p->pi_lock);
lockdep_assert_rq_held(task_rq(p));
@@ -47,7 +60,13 @@ void scx_prepare_setscheduler(struct task_struct *p,
if (p->sched_class == next_class || next_class != &ext_sched_class)
return;
- sched_proxy_block_task(task_rq(p), p);
+ sch = scx_task_sched(p);
+ if (WARN_ON_ONCE(!sch))
+ return;
+
+ /* Block retained donors that the incoming scheduler cannot manage. */
+ if (!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
+ sched_proxy_block_task(task_rq(p), p);
}
/*
@@ -55,13 +74,16 @@ void scx_prepare_setscheduler(struct task_struct *p,
* sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock
* is updated only once.
*/
-static void scx_prepare_task_sched_change(struct task_struct *p)
+void scx_prepare_task_sched_change(struct task_struct *p, struct scx_sched *sch)
{
lockdep_assert_held(&p->pi_lock);
lockdep_assert_rq_held(task_rq(p));
update_rq_clock(task_rq(p));
- sched_proxy_block_task(task_rq(p), p);
+
+ /* Block retained donors that the incoming scheduler cannot manage. */
+ if (!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
+ sched_proxy_block_task(task_rq(p), p);
}
/*
@@ -1914,6 +1936,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
struct scx_sched *sch = scx_task_sched(p);
struct task_struct **ddsp_taskp;
struct scx_dispatch_q *dsq;
+ bool enq_blocked;
unsigned long qseq;
WARN_ON_ONCE(!(p->scx.flags & SCX_TASK_QUEUED));
@@ -1946,15 +1969,22 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
goto direct;
+ /* %SCX_OPS_ENQ_BLOCKED takes precedence over the fallbacks below. */
+ enq_blocked = (sch->ops.flags & SCX_OPS_ENQ_BLOCKED) &&
+ p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP);
+ if (enq_blocked)
+ enq_flags |= SCX_ENQ_BLOCKED;
+
/* see %SCX_OPS_ENQ_EXITING */
- if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
+ if (!enq_blocked && !(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
unlikely(p->flags & PF_EXITING)) {
__scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
goto local;
}
/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */
- if (!(sch->ops.flags & SCX_OPS_ENQ_MIGRATION_DISABLED) &&
+ if (!enq_blocked &&
+ !(sch->ops.flags & SCX_OPS_ENQ_MIGRATION_DISABLED) &&
is_migration_disabled(p)) {
__scx_add_event(sch, SCX_EV_ENQ_SKIP_MIGRATION_DISABLED, 1);
goto local;
@@ -2062,8 +2092,16 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
int sticky_cpu = p->scx.sticky_cpu;
u64 enq_flags = core_enq_flags | rq->scx.remote_activate_enq_flags;
- if (enq_flags & ENQUEUE_WAKEUP)
+ /*
+ * p->is_blocked is cleared after wakeup_preempt(), so remember whether
+ * this is a full wakeup activation until that callback is reached.
+ */
+ if (enq_flags & ENQUEUE_WAKEUP) {
rq->scx.flags |= SCX_RQ_IN_WAKEUP;
+ p->scx.flags |= SCX_TASK_ENQ_WAKEUP;
+ } else {
+ p->scx.flags &= ~SCX_TASK_ENQ_WAKEUP;
+ }
/*
* Restoring a running task will be immediately followed by
@@ -2268,11 +2306,24 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_fl
{
/*
* Preemption between SCX tasks is implemented by resetting the victim
- * task's slice to 0 and triggering reschedule on the target CPU.
- * Nothing to do.
+ * task's slice to 0 and triggering reschedule on the target CPU. A
+ * mutex-blocked task is kept queued for proxy execution, so its wakeup
+ * doesn't go through enqueue_task_scx(). If the BPF scheduler manages
+ * blocked donors, reschedule explicitly so that it can reconsider a
+ * donor it declined to dispatch while blocked.
*/
- if (p->sched_class == &ext_sched_class)
+ if (p->sched_class == &ext_sched_class) {
+ bool enq_wakeup = p->scx.flags & SCX_TASK_ENQ_WAKEUP;
+
+ p->scx.flags &= ~SCX_TASK_ENQ_WAKEUP;
+ if (!enq_wakeup && p->is_blocked) {
+ struct scx_sched *sch = scx_task_sched(p);
+
+ if (sch && (sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
+ resched_curr(rq);
+ }
return;
+ }
/*
* Getting preempted by a higher-priority class. Reenqueue IMMED tasks.
@@ -2387,6 +2438,19 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
WARN_ON_ONCE(task_cpu(p) == cpu);
+ /*
+ * A blocked donor may be moved normally to select a new callback rq.
+ * set_task_cpu() updates wake_cpu and makes the destination rq its new
+ * callback home.
+ *
+ * proxy_set_task_cpu() instead preserves wake_cpu when moving a donor to
+ * its lock owner's CPU. Keep such a donor on the proxy rq until it wakes;
+ * otherwise normal BPF placement may repeatedly pull it back to its
+ * callback rq only for proxy execution to move it to the owner again.
+ */
+ if (p->is_blocked && task_cpu(p) != p->wake_cpu)
+ return false;
+
/*
* If @p has migration disabled, @p->cpus_ptr is updated to contain only
* the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2957,6 +3021,8 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
struct scx_sched *sch = scx_task_sched(p);
bool can_stop_tick;
+ p->scx.flags &= ~SCX_TASK_ENQ_WAKEUP;
+
if (p->scx.flags & SCX_TASK_QUEUED) {
/*
* Core-sched might decide to execute @p before it is
@@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
set_task_runnable(rq, p);
/*
- * Mutex-blocked donors stay queued on the runqueue under proxy
- * execution, but the donor never runs as itself, proxy-exec
- * walks the blocked_on chain on the next __schedule() and runs
- * the lock owner in its place.
+ * Mutex-blocked donors only stay queued when their BPF scheduler
+ * enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since
+ * scx_allow_proxy_exec(), so @p's scheduler association cannot have
+ * changed and @sch must be non-NULL with the flag set.
*
- * Put the donor on the local DSQ directly so pick_next_task()
- * can still see it. find_proxy_task() will either run the chain
- * owner or deactivate the donor so the wakeup path can return it
- * and let BPF make a new dispatch decision once it is unblocked.
+ * Delegate admission to the BPF scheduler.
*/
if (p->is_blocked) {
- scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
+ WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
+ scx_do_enqueue_task(rq, p, 0, -1);
goto switch_class;
}
@@ -7056,6 +7120,11 @@ int scx_validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)
return -EINVAL;
}
+ if ((ops->flags & SCX_OPS_ENQ_BLOCKED) && !ops->enqueue) {
+ scx_error(sch, "SCX_OPS_ENQ_BLOCKED requires ops.enqueue() to be implemented");
+ return -EINVAL;
+ }
+
/*
* SCX_OPS_TID_TO_TASK is enabled by the root scheduler. A sub-sched
* may set it to declare a dependency; reject if the root hasn't
@@ -7438,7 +7507,7 @@ static void scx_root_enable_workfn(struct kthread_work *work)
if (old_class != new_class)
queue_flags |= DEQUEUE_CLASS;
if (new_class == &ext_sched_class) {
- scx_prepare_task_sched_change(p);
+ scx_prepare_task_sched_change(p, sch);
queue_flags |= DEQUEUE_NOCLOCK;
}
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index 4f4130f0d1211..5cbc2808cf365 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -213,6 +213,19 @@ enum scx_ops_flags {
*/
SCX_OPS_TID_TO_TASK = 1LLU << 8,
+ /*
+ * If set, mutex-blocked tasks remain runnable as proxy donors and are
+ * passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler controls
+ * when donors are dispatched and whether they should preempt other work.
+ *
+ * If clear, mutex-blocked tasks are removed from the runqueue normally
+ * and cannot donate their scheduling context through proxy execution.
+ *
+ * For blocked donors, this flag takes precedence over
+ * %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED.
+ */
+ SCX_OPS_ENQ_BLOCKED = 1LLU << 9,
+
SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE |
SCX_OPS_ENQ_LAST |
SCX_OPS_ENQ_EXITING |
@@ -221,7 +234,8 @@ enum scx_ops_flags {
SCX_OPS_SWITCH_PARTIAL |
SCX_OPS_BUILTIN_IDLE_PER_NODE |
SCX_OPS_ALWAYS_ENQ_IMMED |
- SCX_OPS_TID_TO_TASK,
+ SCX_OPS_TID_TO_TASK |
+ SCX_OPS_ENQ_BLOCKED,
/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
__SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
@@ -1614,6 +1628,12 @@ enum scx_enq_flags {
*/
SCX_ENQ_LAST = 1LLU << 41,
+ /*
+ * The task is blocked on a mutex and is being kept runnable as a proxy
+ * donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set.
+ */
+ SCX_ENQ_BLOCKED = 1LLU << 42,
+
/* high 8 bits are internal */
__SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
@@ -1909,6 +1929,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);
void scx_task_iter_unlock(struct scx_task_iter *iter);
void scx_task_iter_stop(struct scx_task_iter *iter);
struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter);
+void scx_prepare_task_sched_change(struct task_struct *p, struct scx_sched *sch);
void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p);
void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
int sticky_cpu);
diff --git a/kernel/sched/ext/sub.c b/kernel/sched/ext/sub.c
index 3cc6d2633f73c..b1c464061d394 100644
--- a/kernel/sched/ext/sub.c
+++ b/kernel/sched/ext/sub.c
@@ -780,7 +780,9 @@ static void scx_fail_parent(struct scx_sched *sch,
if (scx_task_on_sched(parent, p))
continue;
- scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+ scx_prepare_task_sched_change(p, parent);
+ scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+ DEQUEUE_NOCLOCK) {
scx_disable_and_exit_task(sch, p);
scx_set_task_sched(p, parent);
}
@@ -872,7 +874,9 @@ void scx_sub_disable(struct scx_sched *sch)
continue;
}
- scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+ scx_prepare_task_sched_change(p, parent);
+ scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+ DEQUEUE_NOCLOCK) {
/*
* $p is initialized for $parent and still attached to
* @sch. Disable and exit for @sch, switch over to
@@ -1207,7 +1211,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)
if (!(p->scx.flags & SCX_TASK_SUB_INIT))
continue;
- scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+ scx_prepare_task_sched_change(p, sch);
+ scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+ DEQUEUE_NOCLOCK) {
/*
* $p must be either READY or ENABLED. If ENABLED,
* __scx_disabled_and_exit_task() first disables and
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index 23d9ef3e4c9d2..8d5606e465080 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)
#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
+#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
#define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
diff --git a/tools/sched_ext/include/scx/enum_defs.autogen.h b/tools/sched_ext/include/scx/enum_defs.autogen.h
index da4b459820fdd..79b31eb7db7cb 100644
--- a/tools/sched_ext/include/scx/enum_defs.autogen.h
+++ b/tools/sched_ext/include/scx/enum_defs.autogen.h
@@ -55,6 +55,7 @@
#define HAVE_SCX_ENQ_IMMED
#define HAVE_SCX_ENQ_REENQ
#define HAVE_SCX_ENQ_LAST
+#define HAVE_SCX_ENQ_BLOCKED
#define HAVE___SCX_ENQ_INTERNAL_MASK
#define HAVE_SCX_ENQ_CLEAR_OPSS
#define HAVE_SCX_ENQ_DSQ_PRIQ
diff --git a/tools/sched_ext/include/scx/enums.autogen.bpf.h b/tools/sched_ext/include/scx/enums.autogen.bpf.h
index dafccbb6b69d2..7efe7b9346b49 100644
--- a/tools/sched_ext/include/scx/enums.autogen.bpf.h
+++ b/tools/sched_ext/include/scx/enums.autogen.bpf.h
@@ -130,6 +130,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;
const volatile u64 __SCX_ENQ_LAST __weak;
#define SCX_ENQ_LAST __SCX_ENQ_LAST
+const volatile u64 __SCX_ENQ_BLOCKED __weak;
+#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED
+
const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;
#define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
diff --git a/tools/sched_ext/include/scx/enums.autogen.h b/tools/sched_ext/include/scx/enums.autogen.h
index bbd4901f4fce3..f8fbeb7fbf95b 100644
--- a/tools/sched_ext/include/scx/enums.autogen.h
+++ b/tools/sched_ext/include/scx/enums.autogen.h
@@ -47,6 +47,7 @@
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_IMMED); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_REENQ); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_LAST); \
+ SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_BLOCKED); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_CLEAR_OPSS); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_DSQ_PRIQ); \
SCX_ENUM_SET(skel, scx_deq_flags, SCX_DEQ_SCHED_CHANGE); \
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
2026-07-16 13:20 ` [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers Andrea Righi
@ 2026-07-18 6:16 ` John Stultz
2026-07-18 6:50 ` John Stultz
0 siblings, 1 reply; 23+ messages in thread
From: John Stultz @ 2026-07-18 6:16 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
> @@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
> set_task_runnable(rq, p);
>
> /*
> - * Mutex-blocked donors stay queued on the runqueue under proxy
> - * execution, but the donor never runs as itself, proxy-exec
> - * walks the blocked_on chain on the next __schedule() and runs
> - * the lock owner in its place.
> + * Mutex-blocked donors only stay queued when their BPF scheduler
> + * enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since
> + * scx_allow_proxy_exec(), so @p's scheduler association cannot have
> + * changed and @sch must be non-NULL with the flag set.
> *
> - * Put the donor on the local DSQ directly so pick_next_task()
> - * can still see it. find_proxy_task() will either run the chain
> - * owner or deactivate the donor so the wakeup path can return it
> - * and let BPF make a new dispatch decision once it is unblocked.
> + * Delegate admission to the BPF scheduler.
> */
> if (p->is_blocked) {
> - scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
> + WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
> + scx_do_enqueue_task(rq, p, 0, -1);
> goto switch_class;
> }
>
So this isn't a blocker for your patches, but just as a heads up: when
applying my sleeping owner handling changes (even just patches 4-7
from my v30 submission[1]), I managed to trip the above WARN_ON, when
running the test-ww_mutex driver under the scx_pair (usually right as
scx_pair loads).
[ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at
put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527
[ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11
Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116
PREEMPT(full)
[ 5818.979484] Tainted: [W]=WARN
[ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[ 5818.984225] Workqueue: test-ww_mutex test_cycle_work
[ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms
[ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550
[ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de
48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0
ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00
00 00
[ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046
[ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990
[ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390
[ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000
[ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000
[ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000
[ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000)
knlGS:0000000000000000
[ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0
[ 5819.021163] Call Trace:
[ 5819.022168] <TASK>
[ 5819.023010] __schedule+0x15c4/0x2460
[ 5819.024504] ? find_held_lock+0x2b/0x80
[ 5819.025981] ? lock_release+0x191/0x310
[ 5819.027503] schedule+0x3d/0x130
[ 5819.028772] schedule_preempt_disabled+0x18/0x30
[ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0
[ 5819.032492] ? schedule_timeout+0xca/0x130
[ 5819.034095] ? test_cycle_work+0x15d/0x340
[ 5819.035686] ? ww_mutex_lock+0x3c/0xb0
[ 5819.037094] ww_mutex_lock+0x3c/0xb0
[ 5819.038494] test_cycle_work+0x15d/0x340
[ 5819.039949] process_one_work+0x20c/0x5e0
[ 5819.041631] ? lock_is_held_type+0xcd/0x130
[ 5819.043277] worker_thread+0x1a4/0x360
[ 5819.044717] ? __pfx_worker_thread+0x10/0x10
[ 5819.046392] kthread+0x103/0x130
[ 5819.047650] ? __pfx_kthread+0x10/0x10
[ 5819.049074] ret_from_fork+0x27c/0x340
[ 5819.050525] ? __pfx_kthread+0x10/0x10
[ 5819.051990] ret_from_fork_asm+0x1a/0x30
[ 5819.053488] </TASK>
[ 5819.054371] irq event stamp: 38
[ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>]
_raw_spin_unlock_irqrestore+0x50/0x60
[ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>]
__schedule+0xb95/0x2460
[ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>]
__irq_exit_rcu+0xe0/0x150
[ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>]
__irq_exit_rcu+0xe0/0x150
[ 5819.068013] ---[ end trace 0000000000000000 ]---
It seems to be coming from the proxy_resched_idle() call in the "if
(!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {" case in
find_proxy_task(), prior to proxy_enqueue_on_owner() calling
block_task().
I'll have to dig a bit more next week on this, as I'm not yet seeing
whats going wrong here.
Anyway, I'm very eager to see your changes move forward!
thanks
-john
[1] https://lore.kernel.org/lkml/20260701214615.3773339-1-jstultz@google.com/
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
2026-07-18 6:16 ` John Stultz
@ 2026-07-18 6:50 ` John Stultz
0 siblings, 0 replies; 23+ messages in thread
From: John Stultz @ 2026-07-18 6:50 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Fri, Jul 17, 2026 at 11:16 PM John Stultz <jstultz@google.com> wrote:
>
> On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
> > @@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
> > set_task_runnable(rq, p);
> >
> > /*
> > - * Mutex-blocked donors stay queued on the runqueue under proxy
> > - * execution, but the donor never runs as itself, proxy-exec
> > - * walks the blocked_on chain on the next __schedule() and runs
> > - * the lock owner in its place.
> > + * Mutex-blocked donors only stay queued when their BPF scheduler
> > + * enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since
> > + * scx_allow_proxy_exec(), so @p's scheduler association cannot have
> > + * changed and @sch must be non-NULL with the flag set.
> > *
> > - * Put the donor on the local DSQ directly so pick_next_task()
> > - * can still see it. find_proxy_task() will either run the chain
> > - * owner or deactivate the donor so the wakeup path can return it
> > - * and let BPF make a new dispatch decision once it is unblocked.
> > + * Delegate admission to the BPF scheduler.
> > */
> > if (p->is_blocked) {
> > - scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
> > + WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
> > + scx_do_enqueue_task(rq, p, 0, -1);
> > goto switch_class;
> > }
> >
>
> So this isn't a blocker for your patches, but just as a heads up: when
> applying my sleeping owner handling changes (even just patches 4-7
> from my v30 submission[1]), I managed to trip the above WARN_ON, when
> running the test-ww_mutex driver under the scx_pair (usually right as
> scx_pair loads).
>
> [ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at
> put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527
> [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11
> Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116
> PREEMPT(full)
> [ 5818.979484] Tainted: [W]=WARN
> [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS 1.17.0-debian-1.17.0-1 04/01/2014
> [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work
> [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms
> [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550
> [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de
> 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0
> ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00
> 00 00
> [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046
> [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990
> [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390
> [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000
> [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000
> [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000
> [ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000)
> knlGS:0000000000000000
> [ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0
> [ 5819.021163] Call Trace:
> [ 5819.022168] <TASK>
> [ 5819.023010] __schedule+0x15c4/0x2460
> [ 5819.024504] ? find_held_lock+0x2b/0x80
> [ 5819.025981] ? lock_release+0x191/0x310
> [ 5819.027503] schedule+0x3d/0x130
> [ 5819.028772] schedule_preempt_disabled+0x18/0x30
> [ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0
> [ 5819.032492] ? schedule_timeout+0xca/0x130
> [ 5819.034095] ? test_cycle_work+0x15d/0x340
> [ 5819.035686] ? ww_mutex_lock+0x3c/0xb0
> [ 5819.037094] ww_mutex_lock+0x3c/0xb0
> [ 5819.038494] test_cycle_work+0x15d/0x340
> [ 5819.039949] process_one_work+0x20c/0x5e0
> [ 5819.041631] ? lock_is_held_type+0xcd/0x130
> [ 5819.043277] worker_thread+0x1a4/0x360
> [ 5819.044717] ? __pfx_worker_thread+0x10/0x10
> [ 5819.046392] kthread+0x103/0x130
> [ 5819.047650] ? __pfx_kthread+0x10/0x10
> [ 5819.049074] ret_from_fork+0x27c/0x340
> [ 5819.050525] ? __pfx_kthread+0x10/0x10
> [ 5819.051990] ret_from_fork_asm+0x1a/0x30
> [ 5819.053488] </TASK>
> [ 5819.054371] irq event stamp: 38
> [ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>]
> _raw_spin_unlock_irqrestore+0x50/0x60
> [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>]
> __schedule+0xb95/0x2460
> [ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>]
> __irq_exit_rcu+0xe0/0x150
> [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>]
> __irq_exit_rcu+0xe0/0x150
> [ 5819.068013] ---[ end trace 0000000000000000 ]---
>
> It seems to be coming from the proxy_resched_idle() call in the "if
> (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {" case in
> find_proxy_task(), prior to proxy_enqueue_on_owner() calling
> block_task().
>
> I'll have to dig a bit more next week on this, as I'm not yet seeing
> whats going wrong here.
Ok, my tired theory is something like:
1) You have a task A that that is_blocked waiting on a sleeping owner
B. It gets enqueued onto that owner and waits.
2) We start scx_pair, and scx_prepare_task_sched_change() calls
sched_proxy_block_task(), which bails because !task_on_rq_queued()
3) B wakes up, and that causes us to activate_blocked_waiters(), which
re-adds A (with is_blocked still set) to the rq
- this is problematic because now we have is_blocked tasks on the
sched_ext DSQ where its not allowed.
4) A is selected as a donor, and maybe B is sleeping again, so we call
put_prev_set_next() and hti the warning because we see A is is_blocked
when the sched_ext scheduler doesn't support it.
I'll work to prove this out a bit further next week. I suspect we'll
need something somewhere between activate_blocked_waiters() ->
scx_do_enqueue_task() to skip enquing of is_blocked tasks when
SCX_OPS_ENQ_BLOCKED isn't set.
thanks
-john
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 09/11] sched_ext: Add selftest for blocked donor admission
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (7 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-16 13:20 ` [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support Andrea Righi
2026-07-16 13:20 ` [PATCH 11/11] sched: Allow enabling proxy exec with sched_ext Andrea Righi
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
SCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy
donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor
admission requests. Add selftest coverage for this interface.
Exercise a priority inversion using a weighted-vruntime BPF scheduler.
A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it,
and nice 0 CPU contenders, one per allowed CPU, keep the system busy.
Test both a same-CPU topology and a cross-CPU topology with the donor
and owner on different CPUs.
Treat blocked donors according to the normal BPF ordering policy and
assign the default slice on every enqueue, as for other tasks. Run each
CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and
then enabled, count blocked-donor enqueues by CPU and report average
mutex hold and wait times. Reject enqueues carrying both
SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED, as full wakeups are not retained
donor admissions.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the
selftest config selects. Access to the kernel mutex is provided via a
loadable kernel module, built through TEST_GEN_MODS_DIR and managed by
the test.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
tools/testing/selftests/sched_ext/.gitignore | 4 +
tools/testing/selftests/sched_ext/Makefile | 2 +
tools/testing/selftests/sched_ext/config | 2 +
.../selftests/sched_ext/enq_blocked.bpf.c | 116 +++
.../testing/selftests/sched_ext/enq_blocked.c | 908 ++++++++++++++++++
.../testing/selftests/sched_ext/enq_blocked.h | 27 +
.../selftests/sched_ext/test_modules/Makefile | 13 +
.../test_modules/scx_enq_blocked_test.c | 193 ++++
8 files changed, 1265 insertions(+)
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
diff --git a/tools/testing/selftests/sched_ext/.gitignore b/tools/testing/selftests/sched_ext/.gitignore
index ae5491a114c09..54a1fd2af713d 100644
--- a/tools/testing/selftests/sched_ext/.gitignore
+++ b/tools/testing/selftests/sched_ext/.gitignore
@@ -4,3 +4,7 @@
!Makefile
!.gitignore
!config
+!test_modules/
+!test_modules/scx_enq_blocked_test.c
+!test_modules/Makefile
+test_modules/*.mod.c
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
index 3cfe90e0f34fa..51a16b3d32d9b 100644
--- a/tools/testing/selftests/sched_ext/Makefile
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch
include ../../../scripts/Makefile.include
TEST_GEN_PROGS := runner
+TEST_GEN_MODS_DIR := test_modules
# override lib.mk's default rules
OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
auto-test-targets := \
create_dsq \
dequeue \
+ enq_blocked \
enq_last_no_enq_fails \
ddsp_bogus_dsq_fail \
ddsp_vtimelocal_fail \
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
index aa901b05c8ad6..affa3cf33470a 100644
--- a/tools/testing/selftests/sched_ext/config
+++ b/tools/testing/selftests/sched_ext/config
@@ -6,3 +6,5 @@ CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_BTF=y
+CONFIG_EXPERT=y
+CONFIG_SCHED_PROXY_EXEC=y
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.bpf.c b/tools/testing/selftests/sched_ext/enq_blocked.bpf.c
new file mode 100644
index 0000000000000..212690bf4e07b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.bpf.c
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Verify that SCX_OPS_ENQ_BLOCKED passes blocked proxy donors through
+ * ops.enqueue() and record whether callbacks occur on the donor or owner CPU.
+ */
+
+#include <scx/common.bpf.h>
+
+#define SHARED_DSQ 0
+
+char _license[] SEC("license") = "GPL";
+
+s32 donor_pid;
+s32 donor_cpu = -1;
+s32 owner_cpu = -1;
+u64 nr_blocked_enqueues;
+u64 nr_blocked_enqueues_donor_cpu;
+u64 nr_blocked_enqueues_owner_cpu;
+u64 nr_blocked_enqueues_other_cpu;
+u64 nr_blocked_wakeups;
+static u64 vtime_now;
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(enq_blocked_select_cpu,
+ struct task_struct *p, s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ u64 vtime = p->scx.dsq_vtime;
+
+ if (enq_flags & SCX_ENQ_BLOCKED) {
+ int cpu = scx_bpf_task_cpu(p);
+
+ if (enq_flags & SCX_ENQ_WAKEUP)
+ __sync_fetch_and_add(&nr_blocked_wakeups, 1);
+
+ if (p->pid == donor_pid) {
+ __sync_fetch_and_add(&nr_blocked_enqueues, 1);
+ if (cpu == donor_cpu)
+ __sync_fetch_and_add(&nr_blocked_enqueues_donor_cpu, 1);
+ else if (cpu == owner_cpu)
+ __sync_fetch_and_add(&nr_blocked_enqueues_owner_cpu, 1);
+ else
+ __sync_fetch_and_add(&nr_blocked_enqueues_other_cpu, 1);
+ }
+ }
+
+ /* Limit the amount of budget an idling task can accumulate. */
+ if (time_before(vtime, vtime_now - SCX_SLICE_DFL))
+ vtime = vtime_now - SCX_SLICE_DFL;
+
+ scx_bpf_dsq_insert_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime,
+ enq_flags);
+ scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_IDLE);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_running, struct task_struct *p)
+{
+ if (time_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_stopping, struct task_struct *p, bool runnable)
+{
+ u64 delta = scale_by_task_weight_inverse(p,
+ SCX_SLICE_DFL - p->scx.slice);
+
+ scx_bpf_task_set_dsq_vtime(p, p->scx.dsq_vtime + delta);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_enable, struct task_struct *p)
+{
+ scx_bpf_task_set_dsq_vtime(p, vtime_now);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(enq_blocked_init)
+{
+ int ret;
+
+ ret = scx_bpf_create_dsq(SHARED_DSQ, -1);
+ if (ret) {
+ scx_bpf_error("failed to create DSQ %d (%d)", SHARED_DSQ, ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_blocked_ops = {
+ .select_cpu = (void *)enq_blocked_select_cpu,
+ .enqueue = (void *)enq_blocked_enqueue,
+ .dispatch = (void *)enq_blocked_dispatch,
+ .running = (void *)enq_blocked_running,
+ .stopping = (void *)enq_blocked_stopping,
+ .enable = (void *)enq_blocked_enable,
+ .init = (void *)enq_blocked_init,
+ .exit = (void *)enq_blocked_exit,
+ .name = "enq_blocked",
+};
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.c b/tools/testing/selftests/sched_ext/enq_blocked.c
new file mode 100644
index 0000000000000..9798b89f6d305
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.c
@@ -0,0 +1,908 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Exercise a priority inversion with the owner and donor first pinned to the
+ * same CPU, then with each on a different CPU. A high-priority donor blocks on
+ * a mutex held by a low-priority owner while one medium-priority contender per
+ * available CPU keeps the system busy. A weighted-vruntime BPF scheduler runs
+ * both CPU placement configurations with SCX_OPS_ENQ_BLOCKED first disabled
+ * and then enabled. The test validates blocked-donor admission and reports the
+ * average mutex hold and wait times, plus their enabled-minus-disabled deltas,
+ * for each configuration. The timing data is informational.
+ *
+ * CONFIG_SCHED_PROXY_EXEC=y is required to exercise the proxy-execution paths.
+ */
+#define _GNU_SOURCE
+
+#include <bpf/bpf.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/resource.h>
+#include <sys/syscall.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "enq_blocked.bpf.skel.h"
+#include "enq_blocked.h"
+#include "scx_test.h"
+
+#define MODULE_NAME "scx_enq_blocked_test"
+#define MODULE_FILE "test_modules/" MODULE_NAME ".ko"
+#define DEVICE_PATH "/dev/scx_enq_blocked"
+#define WAIT_STEP_US 1000
+#define WAIT_TIMEOUT_MS 2000
+#define NR_WARMUP_TRIALS 1
+#define NR_MEASURED_TRIALS 10
+#define NR_TRIALS (NR_WARMUP_TRIALS + NR_MEASURED_TRIALS)
+#define JOIN_TIMEOUT_MS ((NR_TRIALS + 1) * WAIT_TIMEOUT_MS)
+#define OWNER_NICE 19
+#define DONOR_NICE -20
+#define CONTENDER_NICE 0
+
+struct thread_ctx {
+ atomic_bool start_donor;
+ atomic_bool abort;
+ atomic_bool stop_contender;
+ atomic_bool measurement_ready;
+ atomic_int donor_pid;
+ atomic_int donor_completed;
+ int fd;
+ int donor_cpu;
+ int owner_cpu;
+};
+
+struct contender_ctx {
+ struct thread_ctx *thread_ctx;
+ atomic_int status;
+ int cpu;
+};
+
+struct run_result {
+ struct enq_blocked_stats stats;
+ u64 nr_blocked_enqueues;
+ u64 nr_blocked_enqueues_donor_cpu;
+ u64 nr_blocked_enqueues_owner_cpu;
+ u64 nr_blocked_enqueues_other_cpu;
+ u64 nr_blocked_wakeups;
+};
+
+static bool parse_bool(const char *value, bool *result)
+{
+ if (!strcasecmp(value, "1") || !strcasecmp(value, "y") ||
+ !strcasecmp(value, "yes") || !strcasecmp(value, "on") ||
+ !strcasecmp(value, "true")) {
+ *result = true;
+ return true;
+ }
+
+ if (!strcasecmp(value, "0") || !strcasecmp(value, "n") ||
+ !strcasecmp(value, "no") || !strcasecmp(value, "off") ||
+ !strcasecmp(value, "false")) {
+ *result = false;
+ return true;
+ }
+
+ return false;
+}
+
+static bool cmdline_bool(const char *name, bool default_value)
+{
+ char cmdline[4096], *newline, *saveptr = NULL, *token;
+ size_t name_len = strlen(name);
+ bool value = default_value;
+ FILE *file;
+
+ file = fopen("/proc/cmdline", "r");
+ if (!file)
+ return default_value;
+
+ if (!fgets(cmdline, sizeof(cmdline), file)) {
+ fclose(file);
+ return default_value;
+ }
+ fclose(file);
+ newline = strchr(cmdline, '\n');
+ if (newline)
+ *newline = '\0';
+
+ for (token = strtok_r(cmdline, " ", &saveptr); token;
+ token = strtok_r(NULL, " ", &saveptr)) {
+ bool parsed;
+
+ if (strncmp(token, name, name_len) || token[name_len] != '=')
+ continue;
+ if (parse_bool(token + name_len + 1, &parsed))
+ value = parsed;
+ }
+
+ return value;
+}
+
+static int module_path(char *path, size_t size)
+{
+ ssize_t len;
+ char *slash;
+
+ len = readlink("/proc/self/exe", path, size - 1);
+ if (len < 0)
+ return -errno;
+ path[len] = '\0';
+
+ slash = strrchr(path, '/');
+ if (!slash)
+ return -EINVAL;
+ *slash = '\0';
+
+ if (snprintf(slash, size - (slash - path), "/%s", MODULE_FILE) >=
+ size - (slash - path))
+ return -ENAMETOOLONG;
+
+ return 0;
+}
+
+static int load_test_module(bool *loaded_here)
+{
+ char path[PATH_MAX];
+ int fd, err;
+
+ err = module_path(path, sizeof(path));
+ if (err)
+ return err;
+
+ fd = open(path, O_RDONLY | O_CLOEXEC);
+ if (fd < 0)
+ return -errno;
+
+ if (syscall(SYS_finit_module, fd, "", 0)) {
+ err = errno;
+ close(fd);
+ if (err == EEXIST)
+ return 0;
+ return -err;
+ }
+
+ close(fd);
+ *loaded_here = true;
+ return 0;
+}
+
+static void unload_test_module(bool loaded_here)
+{
+ if (loaded_here && syscall(SYS_delete_module, MODULE_NAME, O_NONBLOCK))
+ SCX_ERR("Failed to unload %s (%d)", MODULE_NAME, errno);
+}
+
+static int pin_to_cpu(int cpu)
+{
+ cpu_set_t mask;
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu, &mask);
+ return sched_setaffinity(0, sizeof(mask), &mask) ? errno : 0;
+}
+
+static int select_test_cpus(bool cross_cpu, cpu_set_t *mask, int *donor_cpu,
+ int *owner_cpu)
+{
+ int cpu, first = -1;
+
+ if (sched_getaffinity(0, sizeof(*mask), mask))
+ return -errno;
+
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (!CPU_ISSET(cpu, mask))
+ continue;
+ if (first < 0) {
+ first = cpu;
+ if (!cross_cpu)
+ break;
+ } else {
+ *donor_cpu = first;
+ *owner_cpu = cpu;
+ return 0;
+ }
+ }
+
+ if (first < 0)
+ return -ENODEV;
+ if (cross_cpu)
+ return -EAGAIN;
+
+ *donor_cpu = first;
+ *owner_cpu = first;
+ return 0;
+}
+
+static int set_nice(int nice)
+{
+ return setpriority(PRIO_PROCESS, 0, nice) ? errno : 0;
+}
+
+static bool wait_for_pid(atomic_int *pid)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (atomic_load_explicit(pid, memory_order_acquire) > 0)
+ return true;
+ usleep(WAIT_STEP_US);
+ }
+
+ return false;
+}
+
+static int wait_for_contenders(struct contender_ctx *contenders,
+ size_t nr_contenders)
+{
+ size_t i, nr_ready;
+ int status, waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ nr_ready = 0;
+ for (i = 0; i < nr_contenders; i++) {
+ status = atomic_load_explicit(&contenders[i].status,
+ memory_order_acquire);
+ if (status < 0)
+ return status;
+ if (status > 0)
+ nr_ready++;
+ }
+ if (nr_ready == nr_contenders)
+ return 1;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static int wait_for_donor_state(struct thread_ctx *ctx, int expected)
+{
+ int state, waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ state = ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_DONOR_STATE);
+ if (state == expected)
+ return state;
+ if (state < 0 && errno != ENOENT)
+ return -errno;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static bool wait_for_donor(struct thread_ctx *ctx, int trial)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (atomic_load_explicit(&ctx->donor_completed,
+ memory_order_acquire) >= trial)
+ return true;
+ if (atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ return false;
+ usleep(WAIT_STEP_US);
+ }
+
+ return false;
+}
+
+static bool wait_for_measurement(struct thread_ctx *ctx)
+{
+ while (!atomic_load_explicit(&ctx->measurement_ready,
+ memory_order_acquire) &&
+ !atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ sched_yield();
+
+ return !atomic_load_explicit(&ctx->abort, memory_order_relaxed);
+}
+
+static void *contender_fn(void *arg)
+{
+ struct contender_ctx *contender = arg;
+ struct thread_ctx *ctx = contender->thread_ctx;
+ int err;
+
+ err = pin_to_cpu(contender->cpu);
+ if (!err)
+ err = set_nice(CONTENDER_NICE);
+ atomic_store_explicit(&contender->status, err ? -err : 1,
+ memory_order_release);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ while (!atomic_load_explicit(&ctx->stop_contender,
+ memory_order_relaxed))
+ ;
+
+ return NULL;
+}
+
+static void *owner_fn(void *arg)
+{
+ struct thread_ctx *ctx = arg;
+ int err, i;
+
+ err = pin_to_cpu(ctx->owner_cpu);
+ if (err)
+ return (void *)(uintptr_t)err;
+ err = set_nice(OWNER_NICE);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ for (i = 0; i < NR_TRIALS; i++) {
+ if (ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_OWNER))
+ return (void *)(uintptr_t)errno;
+ if (!wait_for_donor(ctx, i + 1))
+ return (void *)(uintptr_t)ETIMEDOUT;
+
+ if (i + 1 == NR_WARMUP_TRIALS && !wait_for_measurement(ctx))
+ return NULL;
+ }
+
+ return NULL;
+}
+
+static int run_donor_trial(struct thread_ctx *ctx)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (!ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_DONOR))
+ return 0;
+ if (errno != EAGAIN)
+ return -errno;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static void *donor_fn(void *arg)
+{
+ struct thread_ctx *ctx = arg;
+ int err, i;
+
+ err = pin_to_cpu(ctx->donor_cpu);
+ if (err)
+ return (void *)(uintptr_t)err;
+ err = set_nice(DONOR_NICE);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ atomic_store_explicit(&ctx->donor_pid, syscall(SYS_gettid),
+ memory_order_release);
+ while (!atomic_load_explicit(&ctx->start_donor, memory_order_acquire) &&
+ !atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ sched_yield();
+
+ if (atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ return NULL;
+
+ for (i = 0; i < NR_TRIALS; i++) {
+ err = run_donor_trial(ctx);
+ if (err)
+ return (void *)(uintptr_t)-err;
+ atomic_store_explicit(&ctx->donor_completed, i + 1,
+ memory_order_release);
+ }
+
+ return NULL;
+}
+
+static void print_avg_time(const char *name, u64 total_ns, u64 samples)
+{
+ u64 avg_ns = samples ? total_ns / samples : 0;
+
+ printf(" %s_avg_ns=%llu (%llu.%03llu ms, samples=%llu)\n", name,
+ (unsigned long long)avg_ns,
+ (unsigned long long)(avg_ns / 1000000),
+ (unsigned long long)((avg_ns / 1000) % 1000),
+ (unsigned long long)samples);
+}
+
+static void print_avg_delta(const char *name, u64 disabled_total,
+ u64 disabled_samples, u64 enabled_total,
+ u64 enabled_samples)
+{
+ u64 disabled_avg, enabled_avg;
+ s64 delta_ns;
+ double delta_pct;
+
+ if (!disabled_samples || !enabled_samples)
+ return;
+
+ disabled_avg = disabled_total / disabled_samples;
+ enabled_avg = enabled_total / enabled_samples;
+ delta_ns = (s64)enabled_avg - (s64)disabled_avg;
+ delta_pct = disabled_avg ? 100.0 * delta_ns / disabled_avg : 0.0;
+
+ printf(" %s_delta_ns=%+lld (%+.2f%%)\n", name,
+ (long long)delta_ns, delta_pct);
+}
+
+static int join_thread(pthread_t thread, const struct timespec *deadline,
+ int *thread_err)
+{
+ void *result;
+ int err;
+
+ err = pthread_timedjoin_np(thread, &result, deadline);
+ if (err)
+ return err;
+
+ *thread_err = (int)(uintptr_t)result;
+ return 0;
+}
+
+static void set_join_deadline(struct timespec *deadline)
+{
+ clock_gettime(CLOCK_REALTIME, deadline);
+ deadline->tv_sec += JOIN_TIMEOUT_MS / 1000;
+ deadline->tv_nsec += (JOIN_TIMEOUT_MS % 1000) * 1000000;
+ if (deadline->tv_nsec >= 1000000000) {
+ deadline->tv_sec++;
+ deadline->tv_nsec -= 1000000000;
+ }
+}
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_blocked *skel;
+ u64 flag;
+
+ skel = enq_blocked__open();
+ SCX_FAIL_IF(!skel, "Failed to open skel");
+ SCX_ENUM_INIT(skel);
+
+ flag = SCX_OPS_ENQ_BLOCKED;
+ if (!flag) {
+ enq_blocked__destroy(skel);
+ fprintf(stderr, "SKIP: SCX_OPS_ENQ_BLOCKED is unavailable\n");
+ return SCX_TEST_SKIP;
+ }
+
+ enq_blocked__destroy(skel);
+ *ctx = NULL;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_one(bool enq_blocked, bool cross_cpu,
+ struct run_result *result)
+{
+ struct enq_blocked *skel;
+ struct thread_ctx thread_ctx = {};
+ struct contender_ctx *contender_ctxs = NULL;
+ struct bpf_link *link = NULL;
+ pthread_t owner, donor, *contenders = NULL;
+ struct timespec join_deadline;
+ cpu_set_t allowed_cpus;
+ bool module_loaded = false;
+ bool owner_started = false, donor_started = false;
+ bool join_timed_out = false;
+ bool proxy_enabled;
+ enum scx_test_status status = SCX_TEST_PASS;
+ int cpu, donor_pid, donor_state, err, thread_err;
+ size_t i, nr_contenders, nr_contenders_started = 0;
+ size_t nr_contenders_joined = 0;
+ u64 nr_blocked, nr_blocked_donor_cpu, nr_blocked_owner_cpu;
+ u64 nr_blocked_other_cpu, nr_blocked_wakeups;
+ struct enq_blocked_stats stats;
+
+ err = select_test_cpus(cross_cpu, &allowed_cpus, &thread_ctx.donor_cpu,
+ &thread_ctx.owner_cpu);
+ if (err == -EAGAIN) {
+ fprintf(stderr, "SKIP: cross-CPU case requires two allowed CPUs\n");
+ return SCX_TEST_SKIP;
+ }
+ if (err) {
+ SCX_ERR("Failed to select test CPUs (%d)", -err);
+ return SCX_TEST_FAIL;
+ }
+ nr_contenders = CPU_COUNT(&allowed_cpus);
+ contenders = calloc(nr_contenders, sizeof(*contenders));
+ contender_ctxs = calloc(nr_contenders, sizeof(*contender_ctxs));
+ if (!contenders || !contender_ctxs) {
+ SCX_ERR("Failed to allocate %zu contender threads", nr_contenders);
+ status = SCX_TEST_FAIL;
+ goto out_contenders;
+ }
+
+ skel = enq_blocked__open();
+ if (!skel) {
+ SCX_ERR("Failed to open skel");
+ status = SCX_TEST_FAIL;
+ goto out_contenders;
+ }
+ SCX_ENUM_INIT(skel);
+ skel->struct_ops.enq_blocked_ops->flags =
+ SCX_OPS_ENQ_LAST |
+ (enq_blocked ? SCX_OPS_ENQ_BLOCKED : 0);
+ if (enq_blocked__load(skel)) {
+ SCX_ERR("Failed to load skel");
+ status = SCX_TEST_FAIL;
+ goto out_skel;
+ }
+
+ proxy_enabled = cmdline_bool("sched_proxy_exec", true);
+
+ err = load_test_module(&module_loaded);
+ if (err == -EPERM || err == -ENOENT) {
+ fprintf(stderr, "SKIP: cannot load mutex fixture (%d)\n", -err);
+ status = SCX_TEST_SKIP;
+ goto out_skel;
+ }
+ if (err) {
+ SCX_ERR("Failed to load mutex fixture (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out_skel;
+ }
+
+ thread_ctx.fd = open(DEVICE_PATH, O_RDONLY | O_CLOEXEC);
+ if (thread_ctx.fd < 0) {
+ SCX_ERR("Failed to open %s (%d)", DEVICE_PATH, errno);
+ status = SCX_TEST_FAIL;
+ goto out_module;
+ }
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_RESET_STATS)) {
+ SCX_ERR("Failed to reset mutex statistics (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out_fd;
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_PREP_ATTACH)) {
+ SCX_ERR("Failed to prepare scheduler attachment (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out_fd;
+ }
+
+ err = pthread_create(&owner, NULL, owner_fn, &thread_ctx);
+ if (err) {
+ SCX_ERR("Failed to create owner thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ owner_started = true;
+
+ err = pthread_create(&donor, NULL, donor_fn, &thread_ctx);
+ if (err) {
+ SCX_ERR("Failed to create donor thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ donor_started = true;
+
+ if (!wait_for_pid(&thread_ctx.donor_pid)) {
+ SCX_ERR("Timed out waiting for donor thread");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ donor_pid = atomic_load_explicit(&thread_ctx.donor_pid,
+ memory_order_acquire);
+ skel->bss->donor_pid = donor_pid;
+ skel->data->donor_cpu = thread_ctx.donor_cpu;
+ skel->data->owner_cpu = thread_ctx.owner_cpu;
+ atomic_store_explicit(&thread_ctx.start_donor, true,
+ memory_order_release);
+
+ donor_state = ENQ_BLOCKED_DONOR_SLEEPING;
+ if (proxy_enabled)
+ donor_state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ err = wait_for_donor_state(&thread_ctx, donor_state);
+ if (err < 0) {
+ SCX_ERR("Donor did not block before scheduler attachment (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_blocked_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ donor_state = ENQ_BLOCKED_DONOR_SLEEPING;
+ if (proxy_enabled && enq_blocked)
+ donor_state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ err = wait_for_donor_state(&thread_ctx, donor_state);
+ if (err < 0) {
+ SCX_ERR("Unexpected donor state after scheduler attachment (%d)",
+ -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_ATTACH_DONE)) {
+ SCX_ERR("Failed to complete scheduler attachment (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ i = 0;
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (!CPU_ISSET(cpu, &allowed_cpus))
+ continue;
+
+ contender_ctxs[i].thread_ctx = &thread_ctx;
+ contender_ctxs[i].cpu = cpu;
+ atomic_init(&contender_ctxs[i].status, 0);
+ err = pthread_create(&contenders[i], NULL, contender_fn,
+ &contender_ctxs[i]);
+ if (err) {
+ SCX_ERR("Failed to create contender for CPU %d (%d)",
+ cpu, err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ nr_contenders_started++;
+ i++;
+ }
+
+ err = wait_for_contenders(contender_ctxs, nr_contenders);
+ if (err != 1) {
+ SCX_ERR("Contender threads failed (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ /*
+ * The first trial spans scheduler attachment and validates the state
+ * transition, but including it would skew scheduling latency. Exclude it
+ * from both the mutex and BPF enqueue measurements.
+ */
+ if (!wait_for_donor(&thread_ctx, NR_WARMUP_TRIALS)) {
+ SCX_ERR("Timed out waiting for warm-up trial");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_RESET_STATS)) {
+ SCX_ERR("Failed to reset mutex statistics after warm-up (%d)",
+ errno);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ skel->bss->nr_blocked_enqueues = 0;
+ skel->bss->nr_blocked_enqueues_donor_cpu = 0;
+ skel->bss->nr_blocked_enqueues_owner_cpu = 0;
+ skel->bss->nr_blocked_enqueues_other_cpu = 0;
+ skel->bss->nr_blocked_wakeups = 0;
+ atomic_store_explicit(&thread_ctx.measurement_ready, true,
+ memory_order_release);
+
+out:
+ ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_ATTACH_DONE);
+ if (status != SCX_TEST_PASS) {
+ atomic_store_explicit(&thread_ctx.abort, true, memory_order_release);
+ atomic_store_explicit(&thread_ctx.start_donor, true,
+ memory_order_release);
+ atomic_store_explicit(&thread_ctx.measurement_ready, true,
+ memory_order_release);
+ }
+
+ set_join_deadline(&join_deadline);
+ if (donor_started) {
+ err = join_thread(donor, &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for donor thread");
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join donor thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ } else {
+ donor_started = false;
+ if (thread_err) {
+ SCX_ERR("Donor thread failed (%d)", thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+ if (!join_timed_out && owner_started) {
+ err = join_thread(owner, &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for owner thread");
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join owner thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ } else {
+ owner_started = false;
+ if (thread_err) {
+ SCX_ERR("Owner thread failed (%d)", thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+ atomic_store_explicit(&thread_ctx.stop_contender, true,
+ memory_order_release);
+ for (i = 0; !join_timed_out && i < nr_contenders_started; i++) {
+ err = join_thread(contenders[i], &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for contender on CPU %d",
+ contender_ctxs[i].cpu);
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join contender on CPU %d (%d)",
+ contender_ctxs[i].cpu, err);
+ status = SCX_TEST_FAIL;
+ } else {
+ nr_contenders_joined++;
+ if (thread_err) {
+ SCX_ERR("Contender on CPU %d failed (%d)",
+ contender_ctxs[i].cpu, thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+
+ /* Restore the fair scheduler before waiting for any stranded thread. */
+ if (join_timed_out) {
+ atomic_store_explicit(&thread_ctx.abort, true,
+ memory_order_release);
+ if (link) {
+ bpf_link__destroy(link);
+ link = NULL;
+ }
+ if (donor_started)
+ pthread_join(donor, NULL);
+ if (owner_started)
+ pthread_join(owner, NULL);
+ for (i = nr_contenders_joined;
+ i < nr_contenders_started; i++)
+ pthread_join(contenders[i], NULL);
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_GET_STATS, &stats)) {
+ SCX_ERR("Failed to read mutex statistics (%d)", errno);
+ status = SCX_TEST_FAIL;
+ } else {
+ result->stats = stats;
+ printf("\n[topology=%s SCX_OPS_ENQ_BLOCKED=%s]\n",
+ cross_cpu ? "cross-cpu" : "same-cpu",
+ enq_blocked ? "enabled" : "disabled");
+ printf(" proxy_exec=%s\n",
+ proxy_enabled ? "enabled" : "disabled");
+ printf(" donor_cpu=%d\n", thread_ctx.donor_cpu);
+ printf(" owner_cpu=%d\n", thread_ctx.owner_cpu);
+ printf(" nr_contenders=%zu\n", nr_contenders);
+ printf(" measured_trials=%d\n", NR_MEASURED_TRIALS);
+ printf(" owner_nice=%d\n", OWNER_NICE);
+ printf(" donor_nice=%d\n", DONOR_NICE);
+ printf(" contender_nice=%d\n", CONTENDER_NICE);
+ print_avg_time("mutex_hold", stats.hold_time_ns, stats.nr_holds);
+ print_avg_time("mutex_wait", stats.wait_time_ns, stats.nr_waits);
+ if (stats.nr_holds != NR_MEASURED_TRIALS ||
+ stats.nr_waits != NR_MEASURED_TRIALS) {
+ SCX_ERR("Expected %d measured trials, got %llu holds and %llu waits",
+ NR_MEASURED_TRIALS,
+ (unsigned long long)stats.nr_holds,
+ (unsigned long long)stats.nr_waits);
+ status = SCX_TEST_FAIL;
+ }
+ }
+
+ nr_blocked = skel->bss->nr_blocked_enqueues;
+ nr_blocked_donor_cpu = skel->bss->nr_blocked_enqueues_donor_cpu;
+ nr_blocked_owner_cpu = skel->bss->nr_blocked_enqueues_owner_cpu;
+ nr_blocked_other_cpu = skel->bss->nr_blocked_enqueues_other_cpu;
+ nr_blocked_wakeups = skel->bss->nr_blocked_wakeups;
+ result->nr_blocked_enqueues = nr_blocked;
+ result->nr_blocked_enqueues_donor_cpu = nr_blocked_donor_cpu;
+ result->nr_blocked_enqueues_owner_cpu = nr_blocked_owner_cpu;
+ result->nr_blocked_enqueues_other_cpu = nr_blocked_other_cpu;
+ result->nr_blocked_wakeups = nr_blocked_wakeups;
+ printf(" nr_blocked_enqueues=%llu\n",
+ (unsigned long long)nr_blocked);
+ printf(" nr_blocked_enqueues_donor_cpu=%llu\n",
+ (unsigned long long)nr_blocked_donor_cpu);
+ printf(" nr_blocked_enqueues_owner_cpu=%llu\n",
+ (unsigned long long)nr_blocked_owner_cpu);
+ printf(" nr_blocked_enqueues_other_cpu=%llu\n",
+ (unsigned long long)nr_blocked_other_cpu);
+ printf(" nr_blocked_wakeups=%llu\n",
+ (unsigned long long)nr_blocked_wakeups);
+ if (status == SCX_TEST_PASS) {
+ if (enq_blocked && proxy_enabled && !nr_blocked) {
+ SCX_ERR("ops.enqueue() did not receive the blocked donor");
+ status = SCX_TEST_FAIL;
+ } else if ((!enq_blocked || !proxy_enabled) && nr_blocked) {
+ SCX_ERR("ops.enqueue() unexpectedly received %llu blocked donors",
+ (unsigned long long)nr_blocked);
+ status = SCX_TEST_FAIL;
+ } else if (nr_blocked_wakeups) {
+ SCX_ERR("Ordinary wakeups received %llu blocked enqueue flags",
+ (unsigned long long)nr_blocked_wakeups);
+ status = SCX_TEST_FAIL;
+ } else if (nr_blocked_other_cpu) {
+ SCX_ERR("Blocked donor had %llu enqueues on unexpected CPUs",
+ (unsigned long long)nr_blocked_other_cpu);
+ status = SCX_TEST_FAIL;
+ }
+ }
+
+ if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) {
+ SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)",
+ (unsigned long long)skel->data->uei.kind,
+ (long long)skel->data->uei.exit_code);
+ status = SCX_TEST_FAIL;
+ }
+
+ if (link)
+ bpf_link__destroy(link);
+out_fd:
+ close(thread_ctx.fd);
+out_module:
+ unload_test_module(module_loaded);
+out_skel:
+ enq_blocked__destroy(skel);
+out_contenders:
+ free(contender_ctxs);
+ free(contenders);
+ return status;
+}
+
+static enum scx_test_status run_topology(bool cross_cpu)
+{
+ struct run_result disabled = {}, enabled = {};
+ enum scx_test_status status;
+
+ status = run_one(false, cross_cpu, &disabled);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ status = run_one(true, cross_cpu, &enabled);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ printf("\n[topology=%s delta: enabled - disabled]\n",
+ cross_cpu ? "cross-cpu" : "same-cpu");
+ print_avg_delta("mutex_hold", disabled.stats.hold_time_ns,
+ disabled.stats.nr_holds, enabled.stats.hold_time_ns,
+ enabled.stats.nr_holds);
+ print_avg_delta("mutex_wait", disabled.stats.wait_time_ns,
+ disabled.stats.nr_waits, enabled.stats.wait_time_ns,
+ enabled.stats.nr_waits);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ enum scx_test_status status;
+
+ (void)ctx;
+
+ status = run_topology(false);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ status = run_topology(true);
+ if (status == SCX_TEST_SKIP)
+ return SCX_TEST_PASS;
+
+ return status;
+}
+
+struct scx_test enq_blocked = {
+ .name = "enq_blocked",
+ .description = "Verify proxy donor admission under CPU-wide contention",
+ .setup = setup,
+ .run = run,
+};
+
+REGISTER_SCX_TEST(&enq_blocked)
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.h b/tools/testing/selftests/sched_ext/enq_blocked.h
new file mode 100644
index 0000000000000..68094fb06c875
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */
+#ifndef __ENQ_BLOCKED_H
+#define __ENQ_BLOCKED_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct enq_blocked_stats {
+ __u64 hold_time_ns;
+ __u64 wait_time_ns;
+ __u64 nr_holds;
+ __u64 nr_waits;
+};
+
+#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1)
+#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2)
+#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3)
+#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats)
+#define ENQ_BLOCKED_IOCTL_PREP_ATTACH _IO('s', 5)
+#define ENQ_BLOCKED_IOCTL_ATTACH_DONE _IO('s', 6)
+#define ENQ_BLOCKED_IOCTL_DONOR_STATE _IO('s', 7)
+
+#define ENQ_BLOCKED_DONOR_SLEEPING (1U << 0)
+#define ENQ_BLOCKED_DONOR_ON_RQ (1U << 1)
+
+#endif /* __ENQ_BLOCKED_H */
diff --git a/tools/testing/selftests/sched_ext/test_modules/Makefile b/tools/testing/selftests/sched_ext/test_modules/Makefile
new file mode 100644
index 0000000000000..a0e9e9401ead6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_modules/Makefile
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(if $(O),$(O),$(realpath ../../../../..))
+
+obj-m += scx_enq_blocked_test.o
+
+all:
+ +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+ +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
diff --git a/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c b/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
new file mode 100644
index 0000000000000..e352301d9e274
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Kernel mutex fixture for the sched_ext SCX_OPS_ENQ_BLOCKED selftest.
+ */
+
+#include <linux/atomic.h>
+#include <linux/fs.h>
+#include <linux/jiffies.h>
+#include <linux/ktime.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+
+#include "../enq_blocked.h"
+
+#define DONOR_WAIT_TIMEOUT msecs_to_jiffies(2000)
+#define ATTACH_WAIT_TIMEOUT msecs_to_jiffies(10000)
+#define MUTEX_HOLD_TIME msecs_to_jiffies(200)
+
+static DEFINE_MUTEX(test_mutex);
+static DEFINE_SPINLOCK(donor_lock);
+static struct task_struct *donor_task;
+static atomic_t owner_ready = ATOMIC_INIT(0);
+static atomic_t donor_started = ATOMIC_INIT(0);
+static atomic_t attach_pending = ATOMIC_INIT(0);
+static atomic_t attach_done = ATOMIC_INIT(0);
+static atomic64_t hold_time_ns = ATOMIC64_INIT(0);
+static atomic64_t wait_time_ns = ATOMIC64_INIT(0);
+static atomic64_t nr_holds = ATOMIC64_INIT(0);
+static atomic64_t nr_waits = ATOMIC64_INIT(0);
+
+static long run_owner(void)
+{
+ unsigned long timeout;
+ u64 start_ns;
+ long ret = 0;
+
+ atomic_set(&donor_started, 0);
+ mutex_lock(&test_mutex);
+ start_ns = ktime_get_ns();
+ atomic_set(&owner_ready, 1);
+
+ timeout = jiffies + DONOR_WAIT_TIMEOUT;
+ while (!atomic_read(&donor_started)) {
+ if (time_after(jiffies, timeout)) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+ cond_resched();
+ }
+ if (atomic_xchg(&attach_pending, 0)) {
+ timeout = jiffies + ATTACH_WAIT_TIMEOUT;
+ while (!atomic_read(&attach_done)) {
+ if (time_after(jiffies, timeout)) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+ cond_resched();
+ }
+ }
+
+ /* Keep yielding while the donor blocks on test_mutex. */
+ timeout = jiffies + MUTEX_HOLD_TIME;
+ while (time_before(jiffies, timeout))
+ cond_resched();
+
+out:
+ atomic_set(&owner_ready, 0);
+ atomic64_add(ktime_get_ns() - start_ns, &hold_time_ns);
+ atomic64_inc(&nr_holds);
+ mutex_unlock(&test_mutex);
+ return ret;
+}
+
+static long run_donor(void)
+{
+ unsigned long flags;
+ u64 start_ns;
+
+ if (!atomic_read(&owner_ready))
+ return -EAGAIN;
+
+ get_task_struct(current);
+ spin_lock_irqsave(&donor_lock, flags);
+ WARN_ON_ONCE(donor_task);
+ donor_task = current;
+ spin_unlock_irqrestore(&donor_lock, flags);
+
+ atomic_set(&donor_started, 1);
+ start_ns = ktime_get_ns();
+ mutex_lock(&test_mutex);
+
+ spin_lock_irqsave(&donor_lock, flags);
+ donor_task = NULL;
+ spin_unlock_irqrestore(&donor_lock, flags);
+ put_task_struct(current);
+
+ atomic64_add(ktime_get_ns() - start_ns, &wait_time_ns);
+ atomic64_inc(&nr_waits);
+ mutex_unlock(&test_mutex);
+ return 0;
+}
+
+static long get_donor_state(void)
+{
+ struct task_struct *task;
+ unsigned long flags;
+ long state = 0;
+
+ spin_lock_irqsave(&donor_lock, flags);
+ task = donor_task;
+ if (task)
+ get_task_struct(task);
+ spin_unlock_irqrestore(&donor_lock, flags);
+ if (!task)
+ return -ENOENT;
+
+ if (READ_ONCE(task->__state) != TASK_RUNNING)
+ state |= ENQ_BLOCKED_DONOR_SLEEPING;
+ if (READ_ONCE(task->on_rq))
+ state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ put_task_struct(task);
+ return state;
+}
+
+static void reset_stats(void)
+{
+ atomic64_set(&hold_time_ns, 0);
+ atomic64_set(&wait_time_ns, 0);
+ atomic64_set(&nr_holds, 0);
+ atomic64_set(&nr_waits, 0);
+}
+
+static long get_stats(unsigned long arg)
+{
+ struct enq_blocked_stats stats = {
+ .hold_time_ns = atomic64_read(&hold_time_ns),
+ .wait_time_ns = atomic64_read(&wait_time_ns),
+ .nr_holds = atomic64_read(&nr_holds),
+ .nr_waits = atomic64_read(&nr_waits),
+ };
+
+ return copy_to_user((void __user *)arg, &stats, sizeof(stats)) ?
+ -EFAULT : 0;
+}
+
+static long enq_blocked_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ switch (cmd) {
+ case ENQ_BLOCKED_IOCTL_OWNER:
+ return run_owner();
+ case ENQ_BLOCKED_IOCTL_DONOR:
+ return run_donor();
+ case ENQ_BLOCKED_IOCTL_RESET_STATS:
+ reset_stats();
+ return 0;
+ case ENQ_BLOCKED_IOCTL_GET_STATS:
+ return get_stats(arg);
+ case ENQ_BLOCKED_IOCTL_PREP_ATTACH:
+ atomic_set(&attach_done, 0);
+ atomic_set(&attach_pending, 1);
+ return 0;
+ case ENQ_BLOCKED_IOCTL_ATTACH_DONE:
+ atomic_set(&attach_done, 1);
+ return 0;
+ case ENQ_BLOCKED_IOCTL_DONOR_STATE:
+ return get_donor_state();
+ default:
+ return -EINVAL;
+ }
+}
+
+static const struct file_operations enq_blocked_fops = {
+ .owner = THIS_MODULE,
+ .unlocked_ioctl = enq_blocked_ioctl,
+};
+
+static struct miscdevice enq_blocked_device = {
+ .minor = MISC_DYNAMIC_MINOR,
+ .name = "scx_enq_blocked",
+ .fops = &enq_blocked_fops,
+ .mode = 0600,
+};
+
+module_misc_device(enq_blocked_device);
+MODULE_AUTHOR("Andrea Righi <arighi@nvidia.com>");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("sched_ext blocked donor test module");
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (8 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 09/11] sched_ext: Add selftest for blocked donor admission Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
2026-07-18 2:28 ` John Stultz
2026-07-16 13:20 ` [PATCH 11/11] sched: Allow enabling proxy exec with sched_ext Andrea Righi
10 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Add a -B option to opt scx_qmap into queueing mutex-blocked tasks for
proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear
and mutex waiters block normally. With -B, blocked donors are passed to
qmap_enqueue() with SCX_ENQ_BLOCKED.
When scx_qmap receives a blocked donor, dispatch it directly to the
local DSQ of its current cid with a fresh slice and SCX_ENQ_PREEMPT.
This places the donor at the head of the DSQ and requests an immediate
reschedule, allowing the core proxy-exec path to run the mutex owner
using the donor's scheduling context as soon as the donor is selected.
The blocked policy is intentionally unfair and can strongly prioritize
tasks using contended mutexes, but scx_qmap is a demo scheduler and such
aggressive behavior makes proxy-exec support easy to observe. Count
these dispatch attempts in nr_enq_blocked and report their per-interval
delta.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
tools/sched_ext/scx_qmap.bpf.c | 52 ++++++++++++++++++++++------------
tools/sched_ext/scx_qmap.c | 13 +++++++--
tools/sched_ext/scx_qmap.h | 1 +
3 files changed, 45 insertions(+), 21 deletions(-)
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 09aee49120c28..ab14325a07b6d 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -451,6 +451,40 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
*/
taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
+ /*
+ * If the task was re-enqueued due to the CPU being preempted by a
+ * higher priority scheduling class, just re-enqueue the task directly
+ * on the global DSQ. As we want another CPU to pick it up, find and
+ * kick an idle cid.
+ */
+ if (enq_flags & SCX_ENQ_REENQ) {
+ taskc->force_local = false;
+ scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
+ cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
+ &qa.idle_cids.mask,
+ &qa.self_cids.mask, 0);
+ if (cid < scx_bpf_nr_cids())
+ scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
+ return;
+ }
+
+ /*
+ * Insert a blocked mutex donor at the head of its current cid's local
+ * DSQ with a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate
+ * reschedule. Once selected, the core proxy-exec path can immediately
+ * run the mutex owner using the donor's scheduling context.
+ *
+ * This policy is intentionally unfair and can strongly prioritize tasks
+ * using contended mutexes; scx_qmap is a demonstration scheduler and
+ * this behavior makes proxy-exec support easy to observe.
+ */
+ if (enq_flags & SCX_ENQ_BLOCKED) {
+ __sync_fetch_and_add(&qa.nr_enq_blocked, 1);
+ scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | scx_bpf_task_cid(p),
+ slice_ns, enq_flags | SCX_ENQ_PREEMPT);
+ return;
+ }
+
/*
* A node with children delegates most cids. A task of ours that can run
* on none of our self cids (e.g. a per-NUMA kthread pinned to delegated
@@ -537,24 +571,6 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
- /*
- * If the task was re-enqueued due to the CPU being preempted by a
- * higher priority scheduling class, just re-enqueue the task directly
- * on the global DSQ. As we want another CPU to pick it up, find and
- * kick an idle cid.
- */
- if (enq_flags & SCX_ENQ_REENQ) {
- s32 cid;
-
- scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
- cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
- &qa.idle_cids.mask,
- &qa.self_cids.mask, 0);
- if (cid < scx_bpf_nr_cids())
- scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
- return;
- }
-
/* Queue on the selected FIFO. */
qmap_fifo_enqueue(&qa.fifos[idx], taskc);
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index dda3ddf5b7494..bdc8c4a5e178f 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -51,7 +51,7 @@ const char help_fmt[] =
"See the top-of-file comment in .bpf.c for the design.\n"
"\n"
"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
-" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n"
+" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-B]\n"
" [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
@@ -70,6 +70,7 @@ const char help_fmt[] =
" -S Suppress qmap-specific debug dump\n"
" -p Switch only tasks on SCHED_EXT policy instead of all\n"
" -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n"
+" -B Turn on SCX_OPS_ENQ_BLOCKED\n"
" -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n"
" -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n"
" -i SEC Stats and weight-refresh interval, seconds (default 5)\n"
@@ -185,6 +186,7 @@ struct hier_prev {
u64 nr_dsps[MAX_SUB_SCHEDS];
u64 nr_reenq_cap;
u64 nr_reenq_immed;
+ u64 nr_enq_blocked;
u64 nr_inject_attempts;
};
@@ -267,13 +269,15 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg
}
format_cid_ranges(qa, CID_SHARED, ranges, sizeof(ranges));
- printf("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n",
+ printf("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu\n",
(unsigned long long)qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr,
(unsigned long long)(qa->nr_reenq_cap - prev->nr_reenq_cap),
(unsigned long long)(qa->nr_reenq_immed - prev->nr_reenq_immed),
+ (unsigned long long)(qa->nr_enq_blocked - prev->nr_enq_blocked),
(unsigned long long)(qa->nr_inject_attempts - prev->nr_inject_attempts));
prev->nr_reenq_cap = qa->nr_reenq_cap;
prev->nr_reenq_immed = qa->nr_reenq_immed;
+ prev->nr_enq_blocked = qa->nr_enq_blocked;
prev->nr_inject_attempts = qa->nr_inject_attempts;
printf("hier : %-4s %10s %4s %6s %8s %s\n",
@@ -336,7 +340,7 @@ int main(int argc, char **argv)
skel->rodata->slice_ns = __COMPAT_ENUM_OR_ZERO("scx_public_consts", "SCX_SLICE_DFL");
skel->rodata->max_tasks = 16384;
- while ((opt = getopt(argc, argv, "s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:l:b:N:PMHc:d:D:SpIBF:C:i:R:J:vh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -398,6 +402,9 @@ int main(int argc, char **argv)
skel->rodata->always_enq_immed = true;
skel->struct_ops.qmap_ops->flags |= SCX_OPS_ALWAYS_ENQ_IMMED;
break;
+ case 'B':
+ skel->struct_ops.qmap_ops->flags |= SCX_OPS_ENQ_BLOCKED;
+ break;
case 'F':
skel->rodata->immed_stress_nth = strtoul(optarg, NULL, 0);
break;
diff --git a/tools/sched_ext/scx_qmap.h b/tools/sched_ext/scx_qmap.h
index 87642d21fce38..c706d91a6349c 100644
--- a/tools/sched_ext/scx_qmap.h
+++ b/tools/sched_ext/scx_qmap.h
@@ -173,6 +173,7 @@ struct qmap_arena {
/* bpf -> userspace: stats */
u64 nr_reenq_cap; /* SCX_TASK_REENQ_CAP bounces */
u64 nr_reenq_immed; /* SCX_TASK_REENQ_IMMED bounces */
+ u64 nr_enq_blocked; /* SCX_ENQ_BLOCKED dispatches */
u64 nr_inject_attempts; /* fault-injection: dispatches to an unheld cid */
u32 inject_mode; /* fault-injection mode (QMAP_INJ_*) */
};
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
2026-07-16 13:20 ` [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support Andrea Righi
@ 2026-07-18 2:28 ` John Stultz
2026-07-18 5:47 ` Andrea Righi
0 siblings, 1 reply; 23+ messages in thread
From: John Stultz @ 2026-07-18 2:28 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
>
> Add a -B option to opt scx_qmap into queueing mutex-blocked tasks for
> proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear
> and mutex waiters block normally. With -B, blocked donors are passed to
> qmap_enqueue() with SCX_ENQ_BLOCKED.
>
> When scx_qmap receives a blocked donor, dispatch it directly to the
> local DSQ of its current cid with a fresh slice and SCX_ENQ_PREEMPT.
> This places the donor at the head of the DSQ and requests an immediate
> reschedule, allowing the core proxy-exec path to run the mutex owner
> using the donor's scheduling context as soon as the donor is selected.
Hey Andrea!
Thanks again for your great work on all this, I'm working to test your
changes with my full proxy-exec tree, but before that I caught one
issue here.
I was excited to try this out, but when building and running this from
your kernel tree in a VM, (who's kernel is running from the same
kernel tree, built on the host), I get the following error (with or
without -B). The other scx_ schedulers in tools/sched_ext/ seem to
load and run ok. Maybe is there something I'm forgetting to do here?
thanks
-john
# ./build/bin/scx_qmap -B
libbpf: prog 'qmap_dequeue': BPF program load failed: -EACCES
libbpf: prog 'qmap_dequeue': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64
deq_flags) @ scx_qmap.bpf.c:584
0: (79) r2 = *(u64 *)(r1 +0)
func 'dequeue' arg0 has btf_id 142670 type STRUCT 'task_struct'
1: R1=ctx() R2=trusted_ptr_task_struct()
1: (79) r4 = *(u64 *)(r1 +8) ; R1=ctx() R4=scalar()
; __sync_fetch_and_add(&qa.nr_dequeued, 1); @ scx_qmap.bpf.c:588
2: (18) r1 = 0x10000fff7000 ; R1=scalar()
4: (18) r3 = 0x10000fff7000 ; R3=scalar()
6: (07) r3 += 32 ; R3=scalar()
7: (bf) r5 = addr_space_cast(r3, 0, 1) ; R3=scalar() R5=arena
8: (b7) r3 = 1 ; R3=1
9: (b7) r0 = 1 ; R0=1
10: (db) lock *(u64 *)(r5 +0) += r0 ; R0=1 R5=arena
11: (18) r5 = 0x100000000 ; R5=0x100000000
; if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC) @ scx_qmap.bpf.c:589
13: (5f) r4 &= r5 ;
R4=scalar(smin=smin32=0,smax=umax=0x100000000,smax32=umax32=0,var_off=(0x0;
0x100000000))
14: (15) if r4 == 0x0 goto pc+3 ; R4=0x100000000
; __sync_fetch_and_add(&qa.nr_core_sched_execed, 1); @ scx_qmap.bpf.c:590
15: (07) r1 += 48 ; R1=scalar()
16: (bf) r1 = addr_space_cast(r1, 0, 1) ; R1=arena
17: (db) lock *(u64 *)(r1 +0) += r3 ;
; QMAP_TOUCH_ARENA(); @ scx_qmap.bpf.c:268
18: (18) r1 = 0xffff888104842c00 ; R1=map_ptr(map=arena,ks=0,vs=0)
; v = bpf_task_storage_get(&task_ctx_stor, p, 0, 0); @ scx_qmap.bpf.c:270
20: (18) r1 = 0xffff88810c333600 ; R1=map_ptr(map=task_ctx_stor,ks=4,vs=8)
22: (b7) r3 = 0 ; R3=0
23: (b7) r4 = 0 ; R4=0
24: (85) call bpf_task_storage_get#156 ;
R0=map_value_or_null(id=1,map=task_ctx_stor,ks=4,vs=8)
; if (!v || !v->taskc) @ scx_qmap.bpf.c:271
25: (15) if r0 == 0x0 goto pc+76 ;
R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8)
26: (79) r7 = *(u64 *)(r0 +0) ;
R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8) R7=scalar()
27: (15) if r7 == 0x0 goto pc+74 ; R7=scalar(umin=1)
28: (bf) r9 = addr_space_cast(r7, 0, 1) ; R7=scalar(umin=1) R9=arena
; if (taskc && taskc->fifo) { @ scx_qmap.bpf.c:593
29: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
30: (15) if r8 == 0x0 goto pc+71 ; R8=scalar(umin=1)
; if (taskc->highpri) @ scx_qmap.bpf.c:594
31: (71) r1 = *(u8 *)(r9 +45) ;
R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0;
0xff)) R9=arena
32: (56) if w1 != 0x1 goto pc+9 ; R1=1
33: (b7) r1 = 1 ; R1=1
; __sync_fetch_and_sub(&qa.nr_highpri_queued, 1); @ scx_qmap.bpf.c:595
34: (87) r1 = -r1 ; R1=-1
35: (18) r2 = 0x10000fff7000 ; R2=scalar()
37: (07) r2 += 88 ; R2=scalar()
38: (bf) r2 = addr_space_cast(r2, 0, 1) ; R2=arena
39: (db) lock *(u64 *)(r2 +0) += r1 ; R1=-1 R2=arena
; struct qmap_fifo __arena *fifo = taskc->fifo; @ scx_qmap.bpf.c:320
40: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
; if (!fifo) @ scx_qmap.bpf.c:323
41: (15) if r8 == 0x0 goto pc+60 ; R8=scalar(umin=1)
; lock = qa_q_lock(fifo->idx); @ scx_qmap.bpf.c:326
42: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(umin=1)
43: (61) r1 = *(u32 *)(r2 +16) ;
R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
R2=arena
; switch (qid) { @ scx_qmap.bpf.c:129
44: (66) if w1 s> 0x1 goto pc+8 ;
R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
0xffffffff))
45: (18) r6 = 0xffff88810c3339f8 ;
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
47: (16) if w1 == 0x0 goto pc+17 ;
R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
0xffffffff))
48: (16) if w1 == 0x1 goto pc+1 50: R2=arena R7=scalar(umin=1)
R8=scalar(umin=1) R9=arena R10=fp0
; switch (qid) { @ scx_qmap.bpf.c:129
50: (18) r6 = 0xffff88810c333bf8 ;
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
52: (05) goto pc+12
65: (7b) *(u64 *)(r10 -16) = r2 ; R2=arena R10=fp0 fp-16=arena
; if (bpf_res_spin_lock(lock)) { @ scx_qmap.bpf.c:212
66: (bf) r1 = r6 ;
R1=map_value(map=.data.qa_q_lock,ks=4,vs=4)
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
67: (85) call bpf_res_spin_lock#85478 ; R0=0
68: (16) if w0 == 0x0 goto pc+9 ; R0=0
; if (taskc->fifo != fifo) { @ scx_qmap.bpf.c:331
78: (79) r1 = *(u64 *)(r9 +24) ; R1=scalar() R9=arena
79: (1d) if r1 == r8 goto pc+1 81:
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) R7=scalar(umin=1)
R8=scalar(umin=1) R9=arena R10=fp0 fp-16=arena
; bpf_res_spin_unlock(lock); @ scx_qmap.bpf.c:332
81: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
; if (taskc->q_next) @ scx_qmap.bpf.c:336
82: (79) r1 = *(u64 *)(r9 +8) ; R1=scalar() R9=arena
83: (15) if r1 == 0x0 goto pc+4 ; R1=scalar(umin=1)
84: (bf) r3 = addr_space_cast(r1, 0, 1) ; R1=scalar(umin=1) R3=arena
; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
85: (7b) *(u64 *)(r3 +16) = r2 ; R2=scalar() R3=arena
; if (taskc->q_prev) @ scx_qmap.bpf.c:340
86: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
87: (05) goto pc+2
; fifo->tail = taskc->q_prev; @ scx_qmap.bpf.c:339
90: (07) r7 += 8 ; R7=scalar()
; if (taskc->q_prev) @ scx_qmap.bpf.c:340
91: (15) if r2 == 0x0 goto pc+2 ; R2=scalar(umin=1)
92: (07) r2 += 8 ; R2=scalar()
93: (bf) r8 = r2 ; R8=scalar(id=2)
94: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(id=2)
95: (7b) *(u64 *)(r2 +0) = r1 ; R1=scalar(umin=1) R2=arena
96: (b7) r1 = 0 ; R1=0
; taskc->q_prev = NULL; @ scx_qmap.bpf.c:345
97: (7b) *(u64 *)(r7 +16) = r1
R7 invalid mem access 'scalar'
processed 71 insns (limit 1000000) max_states_per_insn 1 total_states
5 peak_states 5 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'qmap_dequeue': failed to load: -EACCES
libbpf: failed to load object 'scx_qmap'
libbpf: failed to load BPF skeleton 'scx_qmap': -EACCES
[SCX_BUG] scx_qmap.c:497 (Permission denied)
Failed to load skel
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
2026-07-18 2:28 ` John Stultz
@ 2026-07-18 5:47 ` Andrea Righi
2026-07-18 6:04 ` John Stultz
0 siblings, 1 reply; 23+ messages in thread
From: Andrea Righi @ 2026-07-18 5:47 UTC (permalink / raw)
To: John Stultz
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
Hi John,
On Fri, Jul 17, 2026 at 07:28:49PM -0700, John Stultz wrote:
> On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
> >
> > Add a -B option to opt scx_qmap into queueing mutex-blocked tasks for
> > proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear
> > and mutex waiters block normally. With -B, blocked donors are passed to
> > qmap_enqueue() with SCX_ENQ_BLOCKED.
> >
> > When scx_qmap receives a blocked donor, dispatch it directly to the
> > local DSQ of its current cid with a fresh slice and SCX_ENQ_PREEMPT.
> > This places the donor at the head of the DSQ and requests an immediate
> > reschedule, allowing the core proxy-exec path to run the mutex owner
> > using the donor's scheduling context as soon as the donor is selected.
>
> Hey Andrea!
>
> Thanks again for your great work on all this, I'm working to test your
> changes with my full proxy-exec tree, but before that I caught one
> issue here.
>
> I was excited to try this out, but when building and running this from
> your kernel tree in a VM, (who's kernel is running from the same
> kernel tree, built on the host), I get the following error (with or
> without -B). The other scx_ schedulers in tools/sched_ext/ seem to
> load and run ok. Maybe is there something I'm forgetting to do here?
>
> thanks
> -john
>
> # ./build/bin/scx_qmap -B
> libbpf: prog 'qmap_dequeue': BPF program load failed: -EACCES
> libbpf: prog 'qmap_dequeue': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
> ; void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64
> deq_flags) @ scx_qmap.bpf.c:584
> 0: (79) r2 = *(u64 *)(r1 +0)
> func 'dequeue' arg0 has btf_id 142670 type STRUCT 'task_struct'
> 1: R1=ctx() R2=trusted_ptr_task_struct()
> 1: (79) r4 = *(u64 *)(r1 +8) ; R1=ctx() R4=scalar()
> ; __sync_fetch_and_add(&qa.nr_dequeued, 1); @ scx_qmap.bpf.c:588
> 2: (18) r1 = 0x10000fff7000 ; R1=scalar()
> 4: (18) r3 = 0x10000fff7000 ; R3=scalar()
> 6: (07) r3 += 32 ; R3=scalar()
> 7: (bf) r5 = addr_space_cast(r3, 0, 1) ; R3=scalar() R5=arena
> 8: (b7) r3 = 1 ; R3=1
> 9: (b7) r0 = 1 ; R0=1
> 10: (db) lock *(u64 *)(r5 +0) += r0 ; R0=1 R5=arena
> 11: (18) r5 = 0x100000000 ; R5=0x100000000
> ; if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC) @ scx_qmap.bpf.c:589
> 13: (5f) r4 &= r5 ;
> R4=scalar(smin=smin32=0,smax=umax=0x100000000,smax32=umax32=0,var_off=(0x0;
> 0x100000000))
> 14: (15) if r4 == 0x0 goto pc+3 ; R4=0x100000000
> ; __sync_fetch_and_add(&qa.nr_core_sched_execed, 1); @ scx_qmap.bpf.c:590
> 15: (07) r1 += 48 ; R1=scalar()
> 16: (bf) r1 = addr_space_cast(r1, 0, 1) ; R1=arena
> 17: (db) lock *(u64 *)(r1 +0) += r3 ;
> ; QMAP_TOUCH_ARENA(); @ scx_qmap.bpf.c:268
> 18: (18) r1 = 0xffff888104842c00 ; R1=map_ptr(map=arena,ks=0,vs=0)
> ; v = bpf_task_storage_get(&task_ctx_stor, p, 0, 0); @ scx_qmap.bpf.c:270
> 20: (18) r1 = 0xffff88810c333600 ; R1=map_ptr(map=task_ctx_stor,ks=4,vs=8)
> 22: (b7) r3 = 0 ; R3=0
> 23: (b7) r4 = 0 ; R4=0
> 24: (85) call bpf_task_storage_get#156 ;
> R0=map_value_or_null(id=1,map=task_ctx_stor,ks=4,vs=8)
> ; if (!v || !v->taskc) @ scx_qmap.bpf.c:271
> 25: (15) if r0 == 0x0 goto pc+76 ;
> R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8)
> 26: (79) r7 = *(u64 *)(r0 +0) ;
> R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8) R7=scalar()
> 27: (15) if r7 == 0x0 goto pc+74 ; R7=scalar(umin=1)
> 28: (bf) r9 = addr_space_cast(r7, 0, 1) ; R7=scalar(umin=1) R9=arena
> ; if (taskc && taskc->fifo) { @ scx_qmap.bpf.c:593
> 29: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
> 30: (15) if r8 == 0x0 goto pc+71 ; R8=scalar(umin=1)
> ; if (taskc->highpri) @ scx_qmap.bpf.c:594
> 31: (71) r1 = *(u8 *)(r9 +45) ;
> R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0;
> 0xff)) R9=arena
> 32: (56) if w1 != 0x1 goto pc+9 ; R1=1
> 33: (b7) r1 = 1 ; R1=1
> ; __sync_fetch_and_sub(&qa.nr_highpri_queued, 1); @ scx_qmap.bpf.c:595
> 34: (87) r1 = -r1 ; R1=-1
> 35: (18) r2 = 0x10000fff7000 ; R2=scalar()
> 37: (07) r2 += 88 ; R2=scalar()
> 38: (bf) r2 = addr_space_cast(r2, 0, 1) ; R2=arena
> 39: (db) lock *(u64 *)(r2 +0) += r1 ; R1=-1 R2=arena
> ; struct qmap_fifo __arena *fifo = taskc->fifo; @ scx_qmap.bpf.c:320
> 40: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
> ; if (!fifo) @ scx_qmap.bpf.c:323
> 41: (15) if r8 == 0x0 goto pc+60 ; R8=scalar(umin=1)
> ; lock = qa_q_lock(fifo->idx); @ scx_qmap.bpf.c:326
> 42: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(umin=1)
> 43: (61) r1 = *(u32 *)(r2 +16) ;
> R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
> R2=arena
> ; switch (qid) { @ scx_qmap.bpf.c:129
> 44: (66) if w1 s> 0x1 goto pc+8 ;
> R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
> 0xffffffff))
> 45: (18) r6 = 0xffff88810c3339f8 ;
> R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
> 47: (16) if w1 == 0x0 goto pc+17 ;
> R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
> 0xffffffff))
> 48: (16) if w1 == 0x1 goto pc+1 50: R2=arena R7=scalar(umin=1)
> R8=scalar(umin=1) R9=arena R10=fp0
> ; switch (qid) { @ scx_qmap.bpf.c:129
> 50: (18) r6 = 0xffff88810c333bf8 ;
> R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
> 52: (05) goto pc+12
> 65: (7b) *(u64 *)(r10 -16) = r2 ; R2=arena R10=fp0 fp-16=arena
> ; if (bpf_res_spin_lock(lock)) { @ scx_qmap.bpf.c:212
> 66: (bf) r1 = r6 ;
> R1=map_value(map=.data.qa_q_lock,ks=4,vs=4)
> R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
> 67: (85) call bpf_res_spin_lock#85478 ; R0=0
> 68: (16) if w0 == 0x0 goto pc+9 ; R0=0
> ; if (taskc->fifo != fifo) { @ scx_qmap.bpf.c:331
> 78: (79) r1 = *(u64 *)(r9 +24) ; R1=scalar() R9=arena
> 79: (1d) if r1 == r8 goto pc+1 81:
> R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) R7=scalar(umin=1)
> R8=scalar(umin=1) R9=arena R10=fp0 fp-16=arena
> ; bpf_res_spin_unlock(lock); @ scx_qmap.bpf.c:332
> 81: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
> ; if (taskc->q_next) @ scx_qmap.bpf.c:336
> 82: (79) r1 = *(u64 *)(r9 +8) ; R1=scalar() R9=arena
> 83: (15) if r1 == 0x0 goto pc+4 ; R1=scalar(umin=1)
> 84: (bf) r3 = addr_space_cast(r1, 0, 1) ; R1=scalar(umin=1) R3=arena
> ; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
> 85: (7b) *(u64 *)(r3 +16) = r2 ; R2=scalar() R3=arena
> ; if (taskc->q_prev) @ scx_qmap.bpf.c:340
> 86: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
> ; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
> 87: (05) goto pc+2
> ; fifo->tail = taskc->q_prev; @ scx_qmap.bpf.c:339
> 90: (07) r7 += 8 ; R7=scalar()
> ; if (taskc->q_prev) @ scx_qmap.bpf.c:340
> 91: (15) if r2 == 0x0 goto pc+2 ; R2=scalar(umin=1)
> 92: (07) r2 += 8 ; R2=scalar()
> 93: (bf) r8 = r2 ; R8=scalar(id=2)
> 94: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(id=2)
> 95: (7b) *(u64 *)(r2 +0) = r1 ; R1=scalar(umin=1) R2=arena
> 96: (b7) r1 = 0 ; R1=0
> ; taskc->q_prev = NULL; @ scx_qmap.bpf.c:345
> 97: (7b) *(u64 *)(r7 +16) = r1
> R7 invalid mem access 'scalar'
This looks like something related to BPF arena pointers and incorrectly
generated BPF code (the other schedulers don't use BPF arenas, so that'd explain
why you only see this with scx_qmap).
Which version of clang are you using?
Thanks,
-Andrea
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
2026-07-18 5:47 ` Andrea Righi
@ 2026-07-18 6:04 ` John Stultz
0 siblings, 0 replies; 23+ messages in thread
From: John Stultz @ 2026-07-18 6:04 UTC (permalink / raw)
To: Andrea Righi
Cc: Tejun Heo, David Vernet, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak, Christian Loehle, David Dai, Koba Ko, Aiqun Yu,
Shuah Khan, sched-ext, linux-kernel
On Fri, Jul 17, 2026 at 10:47 PM Andrea Righi <arighi@nvidia.com> wrote:
> This looks like something related to BPF arena pointers and incorrectly
> generated BPF code (the other schedulers don't use BPF arenas, so that'd explain
> why you only see this with scx_qmap).
>
> Which version of clang are you using?
Debian clang version 19.1.7 (3+b1)
Running in a debian 13 vm. Though I'm using gcc normally for my
builds (but I see the tools/sched_ext/ build does use clang explicitly
for some things).
thanks
-john
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 11/11] sched: Allow enabling proxy exec with sched_ext
2026-07-16 13:20 [PATCHSET v7 sched_ext/for-7.3] sched: Make proxy execution compatible with sched_ext Andrea Righi
` (9 preceding siblings ...)
2026-07-16 13:20 ` [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support Andrea Righi
@ 2026-07-16 13:20 ` Andrea Righi
10 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-16 13:20 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
Now that sched_ext can handle proxy donors and BPF schedulers can opt in
to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the
!SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options
to be enabled together.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
init/Kconfig | 2 --
1 file changed, 2 deletions(-)
diff --git a/init/Kconfig b/init/Kconfig
index 5230d4879b1c8..0817e62266e03 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC
bool "Proxy Execution"
# Avoid some build failures w/ PREEMPT_RT until it can be fixed
depends on !PREEMPT_RT
- # Need to investigate how to inform sched_ext of split contexts
- depends on !SCHED_CLASS_EXT
# Not particularly useful until we get to multi-rq proxying
depends on EXPERT
help
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 09/11] sched_ext: Add selftest for blocked donor admission
2026-07-15 20:54 [PATCHSET v6 sched_ext/for-7.3] sched: Make proxy execution compatible " Andrea Righi
@ 2026-07-15 20:54 ` Andrea Righi
0 siblings, 0 replies; 23+ messages in thread
From: Andrea Righi @ 2026-07-15 20:54 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Changwoo Min, John Stultz
Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Christian Loehle, David Dai,
Koba Ko, Aiqun Yu, Shuah Khan, sched-ext, linux-kernel
SCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy
donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor
admission requests. Add selftest coverage for this interface.
Exercise a priority inversion using a weighted-vruntime BPF scheduler.
A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it,
and nice 0 CPU contenders, one per allowed CPU, keep the system busy.
Test both a same-CPU topology and a cross-CPU topology with the donor
and owner on different CPUs.
Treat blocked donors according to the normal BPF ordering policy and
assign the default slice on every enqueue, as for other tasks. Run each
CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and
then enabled, count blocked-donor enqueues by CPU and report average
mutex hold and wait times. Reject enqueues carrying both
SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED, as full wakeups are not retained
donor admissions.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the
selftest config selects. Access to the kernel mutex is provided via a
loadable kernel module, built through TEST_GEN_MODS_DIR and managed by
the test.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
tools/testing/selftests/sched_ext/.gitignore | 4 +
tools/testing/selftests/sched_ext/Makefile | 2 +
tools/testing/selftests/sched_ext/config | 2 +
.../selftests/sched_ext/enq_blocked.bpf.c | 116 +++
.../testing/selftests/sched_ext/enq_blocked.c | 908 ++++++++++++++++++
.../testing/selftests/sched_ext/enq_blocked.h | 27 +
.../selftests/sched_ext/test_modules/Makefile | 13 +
.../test_modules/scx_enq_blocked_test.c | 193 ++++
8 files changed, 1265 insertions(+)
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
diff --git a/tools/testing/selftests/sched_ext/.gitignore b/tools/testing/selftests/sched_ext/.gitignore
index ae5491a114c09..54a1fd2af713d 100644
--- a/tools/testing/selftests/sched_ext/.gitignore
+++ b/tools/testing/selftests/sched_ext/.gitignore
@@ -4,3 +4,7 @@
!Makefile
!.gitignore
!config
+!test_modules/
+!test_modules/scx_enq_blocked_test.c
+!test_modules/Makefile
+test_modules/*.mod.c
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
index 3cfe90e0f34fa..51a16b3d32d9b 100644
--- a/tools/testing/selftests/sched_ext/Makefile
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch
include ../../../scripts/Makefile.include
TEST_GEN_PROGS := runner
+TEST_GEN_MODS_DIR := test_modules
# override lib.mk's default rules
OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
auto-test-targets := \
create_dsq \
dequeue \
+ enq_blocked \
enq_last_no_enq_fails \
ddsp_bogus_dsq_fail \
ddsp_vtimelocal_fail \
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
index aa901b05c8ad6..affa3cf33470a 100644
--- a/tools/testing/selftests/sched_ext/config
+++ b/tools/testing/selftests/sched_ext/config
@@ -6,3 +6,5 @@ CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_BTF=y
+CONFIG_EXPERT=y
+CONFIG_SCHED_PROXY_EXEC=y
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.bpf.c b/tools/testing/selftests/sched_ext/enq_blocked.bpf.c
new file mode 100644
index 0000000000000..212690bf4e07b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.bpf.c
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Verify that SCX_OPS_ENQ_BLOCKED passes blocked proxy donors through
+ * ops.enqueue() and record whether callbacks occur on the donor or owner CPU.
+ */
+
+#include <scx/common.bpf.h>
+
+#define SHARED_DSQ 0
+
+char _license[] SEC("license") = "GPL";
+
+s32 donor_pid;
+s32 donor_cpu = -1;
+s32 owner_cpu = -1;
+u64 nr_blocked_enqueues;
+u64 nr_blocked_enqueues_donor_cpu;
+u64 nr_blocked_enqueues_owner_cpu;
+u64 nr_blocked_enqueues_other_cpu;
+u64 nr_blocked_wakeups;
+static u64 vtime_now;
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(enq_blocked_select_cpu,
+ struct task_struct *p, s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ u64 vtime = p->scx.dsq_vtime;
+
+ if (enq_flags & SCX_ENQ_BLOCKED) {
+ int cpu = scx_bpf_task_cpu(p);
+
+ if (enq_flags & SCX_ENQ_WAKEUP)
+ __sync_fetch_and_add(&nr_blocked_wakeups, 1);
+
+ if (p->pid == donor_pid) {
+ __sync_fetch_and_add(&nr_blocked_enqueues, 1);
+ if (cpu == donor_cpu)
+ __sync_fetch_and_add(&nr_blocked_enqueues_donor_cpu, 1);
+ else if (cpu == owner_cpu)
+ __sync_fetch_and_add(&nr_blocked_enqueues_owner_cpu, 1);
+ else
+ __sync_fetch_and_add(&nr_blocked_enqueues_other_cpu, 1);
+ }
+ }
+
+ /* Limit the amount of budget an idling task can accumulate. */
+ if (time_before(vtime, vtime_now - SCX_SLICE_DFL))
+ vtime = vtime_now - SCX_SLICE_DFL;
+
+ scx_bpf_dsq_insert_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime,
+ enq_flags);
+ scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_IDLE);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_running, struct task_struct *p)
+{
+ if (time_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_stopping, struct task_struct *p, bool runnable)
+{
+ u64 delta = scale_by_task_weight_inverse(p,
+ SCX_SLICE_DFL - p->scx.slice);
+
+ scx_bpf_task_set_dsq_vtime(p, p->scx.dsq_vtime + delta);
+}
+
+void BPF_STRUCT_OPS(enq_blocked_enable, struct task_struct *p)
+{
+ scx_bpf_task_set_dsq_vtime(p, vtime_now);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(enq_blocked_init)
+{
+ int ret;
+
+ ret = scx_bpf_create_dsq(SHARED_DSQ, -1);
+ if (ret) {
+ scx_bpf_error("failed to create DSQ %d (%d)", SHARED_DSQ, ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(enq_blocked_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_blocked_ops = {
+ .select_cpu = (void *)enq_blocked_select_cpu,
+ .enqueue = (void *)enq_blocked_enqueue,
+ .dispatch = (void *)enq_blocked_dispatch,
+ .running = (void *)enq_blocked_running,
+ .stopping = (void *)enq_blocked_stopping,
+ .enable = (void *)enq_blocked_enable,
+ .init = (void *)enq_blocked_init,
+ .exit = (void *)enq_blocked_exit,
+ .name = "enq_blocked",
+};
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.c b/tools/testing/selftests/sched_ext/enq_blocked.c
new file mode 100644
index 0000000000000..9798b89f6d305
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.c
@@ -0,0 +1,908 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Exercise a priority inversion with the owner and donor first pinned to the
+ * same CPU, then with each on a different CPU. A high-priority donor blocks on
+ * a mutex held by a low-priority owner while one medium-priority contender per
+ * available CPU keeps the system busy. A weighted-vruntime BPF scheduler runs
+ * both CPU placement configurations with SCX_OPS_ENQ_BLOCKED first disabled
+ * and then enabled. The test validates blocked-donor admission and reports the
+ * average mutex hold and wait times, plus their enabled-minus-disabled deltas,
+ * for each configuration. The timing data is informational.
+ *
+ * CONFIG_SCHED_PROXY_EXEC=y is required to exercise the proxy-execution paths.
+ */
+#define _GNU_SOURCE
+
+#include <bpf/bpf.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/resource.h>
+#include <sys/syscall.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "enq_blocked.bpf.skel.h"
+#include "enq_blocked.h"
+#include "scx_test.h"
+
+#define MODULE_NAME "scx_enq_blocked_test"
+#define MODULE_FILE "test_modules/" MODULE_NAME ".ko"
+#define DEVICE_PATH "/dev/scx_enq_blocked"
+#define WAIT_STEP_US 1000
+#define WAIT_TIMEOUT_MS 2000
+#define NR_WARMUP_TRIALS 1
+#define NR_MEASURED_TRIALS 10
+#define NR_TRIALS (NR_WARMUP_TRIALS + NR_MEASURED_TRIALS)
+#define JOIN_TIMEOUT_MS ((NR_TRIALS + 1) * WAIT_TIMEOUT_MS)
+#define OWNER_NICE 19
+#define DONOR_NICE -20
+#define CONTENDER_NICE 0
+
+struct thread_ctx {
+ atomic_bool start_donor;
+ atomic_bool abort;
+ atomic_bool stop_contender;
+ atomic_bool measurement_ready;
+ atomic_int donor_pid;
+ atomic_int donor_completed;
+ int fd;
+ int donor_cpu;
+ int owner_cpu;
+};
+
+struct contender_ctx {
+ struct thread_ctx *thread_ctx;
+ atomic_int status;
+ int cpu;
+};
+
+struct run_result {
+ struct enq_blocked_stats stats;
+ u64 nr_blocked_enqueues;
+ u64 nr_blocked_enqueues_donor_cpu;
+ u64 nr_blocked_enqueues_owner_cpu;
+ u64 nr_blocked_enqueues_other_cpu;
+ u64 nr_blocked_wakeups;
+};
+
+static bool parse_bool(const char *value, bool *result)
+{
+ if (!strcasecmp(value, "1") || !strcasecmp(value, "y") ||
+ !strcasecmp(value, "yes") || !strcasecmp(value, "on") ||
+ !strcasecmp(value, "true")) {
+ *result = true;
+ return true;
+ }
+
+ if (!strcasecmp(value, "0") || !strcasecmp(value, "n") ||
+ !strcasecmp(value, "no") || !strcasecmp(value, "off") ||
+ !strcasecmp(value, "false")) {
+ *result = false;
+ return true;
+ }
+
+ return false;
+}
+
+static bool cmdline_bool(const char *name, bool default_value)
+{
+ char cmdline[4096], *newline, *saveptr = NULL, *token;
+ size_t name_len = strlen(name);
+ bool value = default_value;
+ FILE *file;
+
+ file = fopen("/proc/cmdline", "r");
+ if (!file)
+ return default_value;
+
+ if (!fgets(cmdline, sizeof(cmdline), file)) {
+ fclose(file);
+ return default_value;
+ }
+ fclose(file);
+ newline = strchr(cmdline, '\n');
+ if (newline)
+ *newline = '\0';
+
+ for (token = strtok_r(cmdline, " ", &saveptr); token;
+ token = strtok_r(NULL, " ", &saveptr)) {
+ bool parsed;
+
+ if (strncmp(token, name, name_len) || token[name_len] != '=')
+ continue;
+ if (parse_bool(token + name_len + 1, &parsed))
+ value = parsed;
+ }
+
+ return value;
+}
+
+static int module_path(char *path, size_t size)
+{
+ ssize_t len;
+ char *slash;
+
+ len = readlink("/proc/self/exe", path, size - 1);
+ if (len < 0)
+ return -errno;
+ path[len] = '\0';
+
+ slash = strrchr(path, '/');
+ if (!slash)
+ return -EINVAL;
+ *slash = '\0';
+
+ if (snprintf(slash, size - (slash - path), "/%s", MODULE_FILE) >=
+ size - (slash - path))
+ return -ENAMETOOLONG;
+
+ return 0;
+}
+
+static int load_test_module(bool *loaded_here)
+{
+ char path[PATH_MAX];
+ int fd, err;
+
+ err = module_path(path, sizeof(path));
+ if (err)
+ return err;
+
+ fd = open(path, O_RDONLY | O_CLOEXEC);
+ if (fd < 0)
+ return -errno;
+
+ if (syscall(SYS_finit_module, fd, "", 0)) {
+ err = errno;
+ close(fd);
+ if (err == EEXIST)
+ return 0;
+ return -err;
+ }
+
+ close(fd);
+ *loaded_here = true;
+ return 0;
+}
+
+static void unload_test_module(bool loaded_here)
+{
+ if (loaded_here && syscall(SYS_delete_module, MODULE_NAME, O_NONBLOCK))
+ SCX_ERR("Failed to unload %s (%d)", MODULE_NAME, errno);
+}
+
+static int pin_to_cpu(int cpu)
+{
+ cpu_set_t mask;
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu, &mask);
+ return sched_setaffinity(0, sizeof(mask), &mask) ? errno : 0;
+}
+
+static int select_test_cpus(bool cross_cpu, cpu_set_t *mask, int *donor_cpu,
+ int *owner_cpu)
+{
+ int cpu, first = -1;
+
+ if (sched_getaffinity(0, sizeof(*mask), mask))
+ return -errno;
+
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (!CPU_ISSET(cpu, mask))
+ continue;
+ if (first < 0) {
+ first = cpu;
+ if (!cross_cpu)
+ break;
+ } else {
+ *donor_cpu = first;
+ *owner_cpu = cpu;
+ return 0;
+ }
+ }
+
+ if (first < 0)
+ return -ENODEV;
+ if (cross_cpu)
+ return -EAGAIN;
+
+ *donor_cpu = first;
+ *owner_cpu = first;
+ return 0;
+}
+
+static int set_nice(int nice)
+{
+ return setpriority(PRIO_PROCESS, 0, nice) ? errno : 0;
+}
+
+static bool wait_for_pid(atomic_int *pid)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (atomic_load_explicit(pid, memory_order_acquire) > 0)
+ return true;
+ usleep(WAIT_STEP_US);
+ }
+
+ return false;
+}
+
+static int wait_for_contenders(struct contender_ctx *contenders,
+ size_t nr_contenders)
+{
+ size_t i, nr_ready;
+ int status, waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ nr_ready = 0;
+ for (i = 0; i < nr_contenders; i++) {
+ status = atomic_load_explicit(&contenders[i].status,
+ memory_order_acquire);
+ if (status < 0)
+ return status;
+ if (status > 0)
+ nr_ready++;
+ }
+ if (nr_ready == nr_contenders)
+ return 1;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static int wait_for_donor_state(struct thread_ctx *ctx, int expected)
+{
+ int state, waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ state = ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_DONOR_STATE);
+ if (state == expected)
+ return state;
+ if (state < 0 && errno != ENOENT)
+ return -errno;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static bool wait_for_donor(struct thread_ctx *ctx, int trial)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (atomic_load_explicit(&ctx->donor_completed,
+ memory_order_acquire) >= trial)
+ return true;
+ if (atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ return false;
+ usleep(WAIT_STEP_US);
+ }
+
+ return false;
+}
+
+static bool wait_for_measurement(struct thread_ctx *ctx)
+{
+ while (!atomic_load_explicit(&ctx->measurement_ready,
+ memory_order_acquire) &&
+ !atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ sched_yield();
+
+ return !atomic_load_explicit(&ctx->abort, memory_order_relaxed);
+}
+
+static void *contender_fn(void *arg)
+{
+ struct contender_ctx *contender = arg;
+ struct thread_ctx *ctx = contender->thread_ctx;
+ int err;
+
+ err = pin_to_cpu(contender->cpu);
+ if (!err)
+ err = set_nice(CONTENDER_NICE);
+ atomic_store_explicit(&contender->status, err ? -err : 1,
+ memory_order_release);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ while (!atomic_load_explicit(&ctx->stop_contender,
+ memory_order_relaxed))
+ ;
+
+ return NULL;
+}
+
+static void *owner_fn(void *arg)
+{
+ struct thread_ctx *ctx = arg;
+ int err, i;
+
+ err = pin_to_cpu(ctx->owner_cpu);
+ if (err)
+ return (void *)(uintptr_t)err;
+ err = set_nice(OWNER_NICE);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ for (i = 0; i < NR_TRIALS; i++) {
+ if (ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_OWNER))
+ return (void *)(uintptr_t)errno;
+ if (!wait_for_donor(ctx, i + 1))
+ return (void *)(uintptr_t)ETIMEDOUT;
+
+ if (i + 1 == NR_WARMUP_TRIALS && !wait_for_measurement(ctx))
+ return NULL;
+ }
+
+ return NULL;
+}
+
+static int run_donor_trial(struct thread_ctx *ctx)
+{
+ int waited_ms;
+
+ for (waited_ms = 0; waited_ms < WAIT_TIMEOUT_MS; waited_ms++) {
+ if (!ioctl(ctx->fd, ENQ_BLOCKED_IOCTL_DONOR))
+ return 0;
+ if (errno != EAGAIN)
+ return -errno;
+ usleep(WAIT_STEP_US);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static void *donor_fn(void *arg)
+{
+ struct thread_ctx *ctx = arg;
+ int err, i;
+
+ err = pin_to_cpu(ctx->donor_cpu);
+ if (err)
+ return (void *)(uintptr_t)err;
+ err = set_nice(DONOR_NICE);
+ if (err)
+ return (void *)(uintptr_t)err;
+
+ atomic_store_explicit(&ctx->donor_pid, syscall(SYS_gettid),
+ memory_order_release);
+ while (!atomic_load_explicit(&ctx->start_donor, memory_order_acquire) &&
+ !atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ sched_yield();
+
+ if (atomic_load_explicit(&ctx->abort, memory_order_relaxed))
+ return NULL;
+
+ for (i = 0; i < NR_TRIALS; i++) {
+ err = run_donor_trial(ctx);
+ if (err)
+ return (void *)(uintptr_t)-err;
+ atomic_store_explicit(&ctx->donor_completed, i + 1,
+ memory_order_release);
+ }
+
+ return NULL;
+}
+
+static void print_avg_time(const char *name, u64 total_ns, u64 samples)
+{
+ u64 avg_ns = samples ? total_ns / samples : 0;
+
+ printf(" %s_avg_ns=%llu (%llu.%03llu ms, samples=%llu)\n", name,
+ (unsigned long long)avg_ns,
+ (unsigned long long)(avg_ns / 1000000),
+ (unsigned long long)((avg_ns / 1000) % 1000),
+ (unsigned long long)samples);
+}
+
+static void print_avg_delta(const char *name, u64 disabled_total,
+ u64 disabled_samples, u64 enabled_total,
+ u64 enabled_samples)
+{
+ u64 disabled_avg, enabled_avg;
+ s64 delta_ns;
+ double delta_pct;
+
+ if (!disabled_samples || !enabled_samples)
+ return;
+
+ disabled_avg = disabled_total / disabled_samples;
+ enabled_avg = enabled_total / enabled_samples;
+ delta_ns = (s64)enabled_avg - (s64)disabled_avg;
+ delta_pct = disabled_avg ? 100.0 * delta_ns / disabled_avg : 0.0;
+
+ printf(" %s_delta_ns=%+lld (%+.2f%%)\n", name,
+ (long long)delta_ns, delta_pct);
+}
+
+static int join_thread(pthread_t thread, const struct timespec *deadline,
+ int *thread_err)
+{
+ void *result;
+ int err;
+
+ err = pthread_timedjoin_np(thread, &result, deadline);
+ if (err)
+ return err;
+
+ *thread_err = (int)(uintptr_t)result;
+ return 0;
+}
+
+static void set_join_deadline(struct timespec *deadline)
+{
+ clock_gettime(CLOCK_REALTIME, deadline);
+ deadline->tv_sec += JOIN_TIMEOUT_MS / 1000;
+ deadline->tv_nsec += (JOIN_TIMEOUT_MS % 1000) * 1000000;
+ if (deadline->tv_nsec >= 1000000000) {
+ deadline->tv_sec++;
+ deadline->tv_nsec -= 1000000000;
+ }
+}
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_blocked *skel;
+ u64 flag;
+
+ skel = enq_blocked__open();
+ SCX_FAIL_IF(!skel, "Failed to open skel");
+ SCX_ENUM_INIT(skel);
+
+ flag = SCX_OPS_ENQ_BLOCKED;
+ if (!flag) {
+ enq_blocked__destroy(skel);
+ fprintf(stderr, "SKIP: SCX_OPS_ENQ_BLOCKED is unavailable\n");
+ return SCX_TEST_SKIP;
+ }
+
+ enq_blocked__destroy(skel);
+ *ctx = NULL;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_one(bool enq_blocked, bool cross_cpu,
+ struct run_result *result)
+{
+ struct enq_blocked *skel;
+ struct thread_ctx thread_ctx = {};
+ struct contender_ctx *contender_ctxs = NULL;
+ struct bpf_link *link = NULL;
+ pthread_t owner, donor, *contenders = NULL;
+ struct timespec join_deadline;
+ cpu_set_t allowed_cpus;
+ bool module_loaded = false;
+ bool owner_started = false, donor_started = false;
+ bool join_timed_out = false;
+ bool proxy_enabled;
+ enum scx_test_status status = SCX_TEST_PASS;
+ int cpu, donor_pid, donor_state, err, thread_err;
+ size_t i, nr_contenders, nr_contenders_started = 0;
+ size_t nr_contenders_joined = 0;
+ u64 nr_blocked, nr_blocked_donor_cpu, nr_blocked_owner_cpu;
+ u64 nr_blocked_other_cpu, nr_blocked_wakeups;
+ struct enq_blocked_stats stats;
+
+ err = select_test_cpus(cross_cpu, &allowed_cpus, &thread_ctx.donor_cpu,
+ &thread_ctx.owner_cpu);
+ if (err == -EAGAIN) {
+ fprintf(stderr, "SKIP: cross-CPU case requires two allowed CPUs\n");
+ return SCX_TEST_SKIP;
+ }
+ if (err) {
+ SCX_ERR("Failed to select test CPUs (%d)", -err);
+ return SCX_TEST_FAIL;
+ }
+ nr_contenders = CPU_COUNT(&allowed_cpus);
+ contenders = calloc(nr_contenders, sizeof(*contenders));
+ contender_ctxs = calloc(nr_contenders, sizeof(*contender_ctxs));
+ if (!contenders || !contender_ctxs) {
+ SCX_ERR("Failed to allocate %zu contender threads", nr_contenders);
+ status = SCX_TEST_FAIL;
+ goto out_contenders;
+ }
+
+ skel = enq_blocked__open();
+ if (!skel) {
+ SCX_ERR("Failed to open skel");
+ status = SCX_TEST_FAIL;
+ goto out_contenders;
+ }
+ SCX_ENUM_INIT(skel);
+ skel->struct_ops.enq_blocked_ops->flags =
+ SCX_OPS_ENQ_LAST |
+ (enq_blocked ? SCX_OPS_ENQ_BLOCKED : 0);
+ if (enq_blocked__load(skel)) {
+ SCX_ERR("Failed to load skel");
+ status = SCX_TEST_FAIL;
+ goto out_skel;
+ }
+
+ proxy_enabled = cmdline_bool("sched_proxy_exec", true);
+
+ err = load_test_module(&module_loaded);
+ if (err == -EPERM || err == -ENOENT) {
+ fprintf(stderr, "SKIP: cannot load mutex fixture (%d)\n", -err);
+ status = SCX_TEST_SKIP;
+ goto out_skel;
+ }
+ if (err) {
+ SCX_ERR("Failed to load mutex fixture (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out_skel;
+ }
+
+ thread_ctx.fd = open(DEVICE_PATH, O_RDONLY | O_CLOEXEC);
+ if (thread_ctx.fd < 0) {
+ SCX_ERR("Failed to open %s (%d)", DEVICE_PATH, errno);
+ status = SCX_TEST_FAIL;
+ goto out_module;
+ }
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_RESET_STATS)) {
+ SCX_ERR("Failed to reset mutex statistics (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out_fd;
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_PREP_ATTACH)) {
+ SCX_ERR("Failed to prepare scheduler attachment (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out_fd;
+ }
+
+ err = pthread_create(&owner, NULL, owner_fn, &thread_ctx);
+ if (err) {
+ SCX_ERR("Failed to create owner thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ owner_started = true;
+
+ err = pthread_create(&donor, NULL, donor_fn, &thread_ctx);
+ if (err) {
+ SCX_ERR("Failed to create donor thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ donor_started = true;
+
+ if (!wait_for_pid(&thread_ctx.donor_pid)) {
+ SCX_ERR("Timed out waiting for donor thread");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ donor_pid = atomic_load_explicit(&thread_ctx.donor_pid,
+ memory_order_acquire);
+ skel->bss->donor_pid = donor_pid;
+ skel->data->donor_cpu = thread_ctx.donor_cpu;
+ skel->data->owner_cpu = thread_ctx.owner_cpu;
+ atomic_store_explicit(&thread_ctx.start_donor, true,
+ memory_order_release);
+
+ donor_state = ENQ_BLOCKED_DONOR_SLEEPING;
+ if (proxy_enabled)
+ donor_state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ err = wait_for_donor_state(&thread_ctx, donor_state);
+ if (err < 0) {
+ SCX_ERR("Donor did not block before scheduler attachment (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_blocked_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ donor_state = ENQ_BLOCKED_DONOR_SLEEPING;
+ if (proxy_enabled && enq_blocked)
+ donor_state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ err = wait_for_donor_state(&thread_ctx, donor_state);
+ if (err < 0) {
+ SCX_ERR("Unexpected donor state after scheduler attachment (%d)",
+ -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_ATTACH_DONE)) {
+ SCX_ERR("Failed to complete scheduler attachment (%d)", errno);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ i = 0;
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (!CPU_ISSET(cpu, &allowed_cpus))
+ continue;
+
+ contender_ctxs[i].thread_ctx = &thread_ctx;
+ contender_ctxs[i].cpu = cpu;
+ atomic_init(&contender_ctxs[i].status, 0);
+ err = pthread_create(&contenders[i], NULL, contender_fn,
+ &contender_ctxs[i]);
+ if (err) {
+ SCX_ERR("Failed to create contender for CPU %d (%d)",
+ cpu, err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ nr_contenders_started++;
+ i++;
+ }
+
+ err = wait_for_contenders(contender_ctxs, nr_contenders);
+ if (err != 1) {
+ SCX_ERR("Contender threads failed (%d)", -err);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+
+ /*
+ * The first trial spans scheduler attachment and validates the state
+ * transition, but including it would skew scheduling latency. Exclude it
+ * from both the mutex and BPF enqueue measurements.
+ */
+ if (!wait_for_donor(&thread_ctx, NR_WARMUP_TRIALS)) {
+ SCX_ERR("Timed out waiting for warm-up trial");
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_RESET_STATS)) {
+ SCX_ERR("Failed to reset mutex statistics after warm-up (%d)",
+ errno);
+ status = SCX_TEST_FAIL;
+ goto out;
+ }
+ skel->bss->nr_blocked_enqueues = 0;
+ skel->bss->nr_blocked_enqueues_donor_cpu = 0;
+ skel->bss->nr_blocked_enqueues_owner_cpu = 0;
+ skel->bss->nr_blocked_enqueues_other_cpu = 0;
+ skel->bss->nr_blocked_wakeups = 0;
+ atomic_store_explicit(&thread_ctx.measurement_ready, true,
+ memory_order_release);
+
+out:
+ ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_ATTACH_DONE);
+ if (status != SCX_TEST_PASS) {
+ atomic_store_explicit(&thread_ctx.abort, true, memory_order_release);
+ atomic_store_explicit(&thread_ctx.start_donor, true,
+ memory_order_release);
+ atomic_store_explicit(&thread_ctx.measurement_ready, true,
+ memory_order_release);
+ }
+
+ set_join_deadline(&join_deadline);
+ if (donor_started) {
+ err = join_thread(donor, &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for donor thread");
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join donor thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ } else {
+ donor_started = false;
+ if (thread_err) {
+ SCX_ERR("Donor thread failed (%d)", thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+ if (!join_timed_out && owner_started) {
+ err = join_thread(owner, &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for owner thread");
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join owner thread (%d)", err);
+ status = SCX_TEST_FAIL;
+ } else {
+ owner_started = false;
+ if (thread_err) {
+ SCX_ERR("Owner thread failed (%d)", thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+ atomic_store_explicit(&thread_ctx.stop_contender, true,
+ memory_order_release);
+ for (i = 0; !join_timed_out && i < nr_contenders_started; i++) {
+ err = join_thread(contenders[i], &join_deadline, &thread_err);
+ if (err == ETIMEDOUT) {
+ SCX_ERR("Timed out waiting for contender on CPU %d",
+ contender_ctxs[i].cpu);
+ join_timed_out = true;
+ status = SCX_TEST_FAIL;
+ } else if (err) {
+ SCX_ERR("Failed to join contender on CPU %d (%d)",
+ contender_ctxs[i].cpu, err);
+ status = SCX_TEST_FAIL;
+ } else {
+ nr_contenders_joined++;
+ if (thread_err) {
+ SCX_ERR("Contender on CPU %d failed (%d)",
+ contender_ctxs[i].cpu, thread_err);
+ status = SCX_TEST_FAIL;
+ }
+ }
+ }
+
+ /* Restore the fair scheduler before waiting for any stranded thread. */
+ if (join_timed_out) {
+ atomic_store_explicit(&thread_ctx.abort, true,
+ memory_order_release);
+ if (link) {
+ bpf_link__destroy(link);
+ link = NULL;
+ }
+ if (donor_started)
+ pthread_join(donor, NULL);
+ if (owner_started)
+ pthread_join(owner, NULL);
+ for (i = nr_contenders_joined;
+ i < nr_contenders_started; i++)
+ pthread_join(contenders[i], NULL);
+ }
+
+ if (ioctl(thread_ctx.fd, ENQ_BLOCKED_IOCTL_GET_STATS, &stats)) {
+ SCX_ERR("Failed to read mutex statistics (%d)", errno);
+ status = SCX_TEST_FAIL;
+ } else {
+ result->stats = stats;
+ printf("\n[topology=%s SCX_OPS_ENQ_BLOCKED=%s]\n",
+ cross_cpu ? "cross-cpu" : "same-cpu",
+ enq_blocked ? "enabled" : "disabled");
+ printf(" proxy_exec=%s\n",
+ proxy_enabled ? "enabled" : "disabled");
+ printf(" donor_cpu=%d\n", thread_ctx.donor_cpu);
+ printf(" owner_cpu=%d\n", thread_ctx.owner_cpu);
+ printf(" nr_contenders=%zu\n", nr_contenders);
+ printf(" measured_trials=%d\n", NR_MEASURED_TRIALS);
+ printf(" owner_nice=%d\n", OWNER_NICE);
+ printf(" donor_nice=%d\n", DONOR_NICE);
+ printf(" contender_nice=%d\n", CONTENDER_NICE);
+ print_avg_time("mutex_hold", stats.hold_time_ns, stats.nr_holds);
+ print_avg_time("mutex_wait", stats.wait_time_ns, stats.nr_waits);
+ if (stats.nr_holds != NR_MEASURED_TRIALS ||
+ stats.nr_waits != NR_MEASURED_TRIALS) {
+ SCX_ERR("Expected %d measured trials, got %llu holds and %llu waits",
+ NR_MEASURED_TRIALS,
+ (unsigned long long)stats.nr_holds,
+ (unsigned long long)stats.nr_waits);
+ status = SCX_TEST_FAIL;
+ }
+ }
+
+ nr_blocked = skel->bss->nr_blocked_enqueues;
+ nr_blocked_donor_cpu = skel->bss->nr_blocked_enqueues_donor_cpu;
+ nr_blocked_owner_cpu = skel->bss->nr_blocked_enqueues_owner_cpu;
+ nr_blocked_other_cpu = skel->bss->nr_blocked_enqueues_other_cpu;
+ nr_blocked_wakeups = skel->bss->nr_blocked_wakeups;
+ result->nr_blocked_enqueues = nr_blocked;
+ result->nr_blocked_enqueues_donor_cpu = nr_blocked_donor_cpu;
+ result->nr_blocked_enqueues_owner_cpu = nr_blocked_owner_cpu;
+ result->nr_blocked_enqueues_other_cpu = nr_blocked_other_cpu;
+ result->nr_blocked_wakeups = nr_blocked_wakeups;
+ printf(" nr_blocked_enqueues=%llu\n",
+ (unsigned long long)nr_blocked);
+ printf(" nr_blocked_enqueues_donor_cpu=%llu\n",
+ (unsigned long long)nr_blocked_donor_cpu);
+ printf(" nr_blocked_enqueues_owner_cpu=%llu\n",
+ (unsigned long long)nr_blocked_owner_cpu);
+ printf(" nr_blocked_enqueues_other_cpu=%llu\n",
+ (unsigned long long)nr_blocked_other_cpu);
+ printf(" nr_blocked_wakeups=%llu\n",
+ (unsigned long long)nr_blocked_wakeups);
+ if (status == SCX_TEST_PASS) {
+ if (enq_blocked && proxy_enabled && !nr_blocked) {
+ SCX_ERR("ops.enqueue() did not receive the blocked donor");
+ status = SCX_TEST_FAIL;
+ } else if ((!enq_blocked || !proxy_enabled) && nr_blocked) {
+ SCX_ERR("ops.enqueue() unexpectedly received %llu blocked donors",
+ (unsigned long long)nr_blocked);
+ status = SCX_TEST_FAIL;
+ } else if (nr_blocked_wakeups) {
+ SCX_ERR("Ordinary wakeups received %llu blocked enqueue flags",
+ (unsigned long long)nr_blocked_wakeups);
+ status = SCX_TEST_FAIL;
+ } else if (nr_blocked_other_cpu) {
+ SCX_ERR("Blocked donor had %llu enqueues on unexpected CPUs",
+ (unsigned long long)nr_blocked_other_cpu);
+ status = SCX_TEST_FAIL;
+ }
+ }
+
+ if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) {
+ SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)",
+ (unsigned long long)skel->data->uei.kind,
+ (long long)skel->data->uei.exit_code);
+ status = SCX_TEST_FAIL;
+ }
+
+ if (link)
+ bpf_link__destroy(link);
+out_fd:
+ close(thread_ctx.fd);
+out_module:
+ unload_test_module(module_loaded);
+out_skel:
+ enq_blocked__destroy(skel);
+out_contenders:
+ free(contender_ctxs);
+ free(contenders);
+ return status;
+}
+
+static enum scx_test_status run_topology(bool cross_cpu)
+{
+ struct run_result disabled = {}, enabled = {};
+ enum scx_test_status status;
+
+ status = run_one(false, cross_cpu, &disabled);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ status = run_one(true, cross_cpu, &enabled);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ printf("\n[topology=%s delta: enabled - disabled]\n",
+ cross_cpu ? "cross-cpu" : "same-cpu");
+ print_avg_delta("mutex_hold", disabled.stats.hold_time_ns,
+ disabled.stats.nr_holds, enabled.stats.hold_time_ns,
+ enabled.stats.nr_holds);
+ print_avg_delta("mutex_wait", disabled.stats.wait_time_ns,
+ disabled.stats.nr_waits, enabled.stats.wait_time_ns,
+ enabled.stats.nr_waits);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ enum scx_test_status status;
+
+ (void)ctx;
+
+ status = run_topology(false);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ status = run_topology(true);
+ if (status == SCX_TEST_SKIP)
+ return SCX_TEST_PASS;
+
+ return status;
+}
+
+struct scx_test enq_blocked = {
+ .name = "enq_blocked",
+ .description = "Verify proxy donor admission under CPU-wide contention",
+ .setup = setup,
+ .run = run,
+};
+
+REGISTER_SCX_TEST(&enq_blocked)
diff --git a/tools/testing/selftests/sched_ext/enq_blocked.h b/tools/testing/selftests/sched_ext/enq_blocked.h
new file mode 100644
index 0000000000000..68094fb06c875
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_blocked.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */
+#ifndef __ENQ_BLOCKED_H
+#define __ENQ_BLOCKED_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct enq_blocked_stats {
+ __u64 hold_time_ns;
+ __u64 wait_time_ns;
+ __u64 nr_holds;
+ __u64 nr_waits;
+};
+
+#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1)
+#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2)
+#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3)
+#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats)
+#define ENQ_BLOCKED_IOCTL_PREP_ATTACH _IO('s', 5)
+#define ENQ_BLOCKED_IOCTL_ATTACH_DONE _IO('s', 6)
+#define ENQ_BLOCKED_IOCTL_DONOR_STATE _IO('s', 7)
+
+#define ENQ_BLOCKED_DONOR_SLEEPING (1U << 0)
+#define ENQ_BLOCKED_DONOR_ON_RQ (1U << 1)
+
+#endif /* __ENQ_BLOCKED_H */
diff --git a/tools/testing/selftests/sched_ext/test_modules/Makefile b/tools/testing/selftests/sched_ext/test_modules/Makefile
new file mode 100644
index 0000000000000..a0e9e9401ead6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_modules/Makefile
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(if $(O),$(O),$(realpath ../../../../..))
+
+obj-m += scx_enq_blocked_test.o
+
+all:
+ +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+ +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
diff --git a/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c b/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
new file mode 100644
index 0000000000000..e352301d9e274
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Kernel mutex fixture for the sched_ext SCX_OPS_ENQ_BLOCKED selftest.
+ */
+
+#include <linux/atomic.h>
+#include <linux/fs.h>
+#include <linux/jiffies.h>
+#include <linux/ktime.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+
+#include "../enq_blocked.h"
+
+#define DONOR_WAIT_TIMEOUT msecs_to_jiffies(2000)
+#define ATTACH_WAIT_TIMEOUT msecs_to_jiffies(10000)
+#define MUTEX_HOLD_TIME msecs_to_jiffies(200)
+
+static DEFINE_MUTEX(test_mutex);
+static DEFINE_SPINLOCK(donor_lock);
+static struct task_struct *donor_task;
+static atomic_t owner_ready = ATOMIC_INIT(0);
+static atomic_t donor_started = ATOMIC_INIT(0);
+static atomic_t attach_pending = ATOMIC_INIT(0);
+static atomic_t attach_done = ATOMIC_INIT(0);
+static atomic64_t hold_time_ns = ATOMIC64_INIT(0);
+static atomic64_t wait_time_ns = ATOMIC64_INIT(0);
+static atomic64_t nr_holds = ATOMIC64_INIT(0);
+static atomic64_t nr_waits = ATOMIC64_INIT(0);
+
+static long run_owner(void)
+{
+ unsigned long timeout;
+ u64 start_ns;
+ long ret = 0;
+
+ atomic_set(&donor_started, 0);
+ mutex_lock(&test_mutex);
+ start_ns = ktime_get_ns();
+ atomic_set(&owner_ready, 1);
+
+ timeout = jiffies + DONOR_WAIT_TIMEOUT;
+ while (!atomic_read(&donor_started)) {
+ if (time_after(jiffies, timeout)) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+ cond_resched();
+ }
+ if (atomic_xchg(&attach_pending, 0)) {
+ timeout = jiffies + ATTACH_WAIT_TIMEOUT;
+ while (!atomic_read(&attach_done)) {
+ if (time_after(jiffies, timeout)) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+ cond_resched();
+ }
+ }
+
+ /* Keep yielding while the donor blocks on test_mutex. */
+ timeout = jiffies + MUTEX_HOLD_TIME;
+ while (time_before(jiffies, timeout))
+ cond_resched();
+
+out:
+ atomic_set(&owner_ready, 0);
+ atomic64_add(ktime_get_ns() - start_ns, &hold_time_ns);
+ atomic64_inc(&nr_holds);
+ mutex_unlock(&test_mutex);
+ return ret;
+}
+
+static long run_donor(void)
+{
+ unsigned long flags;
+ u64 start_ns;
+
+ if (!atomic_read(&owner_ready))
+ return -EAGAIN;
+
+ get_task_struct(current);
+ spin_lock_irqsave(&donor_lock, flags);
+ WARN_ON_ONCE(donor_task);
+ donor_task = current;
+ spin_unlock_irqrestore(&donor_lock, flags);
+
+ atomic_set(&donor_started, 1);
+ start_ns = ktime_get_ns();
+ mutex_lock(&test_mutex);
+
+ spin_lock_irqsave(&donor_lock, flags);
+ donor_task = NULL;
+ spin_unlock_irqrestore(&donor_lock, flags);
+ put_task_struct(current);
+
+ atomic64_add(ktime_get_ns() - start_ns, &wait_time_ns);
+ atomic64_inc(&nr_waits);
+ mutex_unlock(&test_mutex);
+ return 0;
+}
+
+static long get_donor_state(void)
+{
+ struct task_struct *task;
+ unsigned long flags;
+ long state = 0;
+
+ spin_lock_irqsave(&donor_lock, flags);
+ task = donor_task;
+ if (task)
+ get_task_struct(task);
+ spin_unlock_irqrestore(&donor_lock, flags);
+ if (!task)
+ return -ENOENT;
+
+ if (READ_ONCE(task->__state) != TASK_RUNNING)
+ state |= ENQ_BLOCKED_DONOR_SLEEPING;
+ if (READ_ONCE(task->on_rq))
+ state |= ENQ_BLOCKED_DONOR_ON_RQ;
+ put_task_struct(task);
+ return state;
+}
+
+static void reset_stats(void)
+{
+ atomic64_set(&hold_time_ns, 0);
+ atomic64_set(&wait_time_ns, 0);
+ atomic64_set(&nr_holds, 0);
+ atomic64_set(&nr_waits, 0);
+}
+
+static long get_stats(unsigned long arg)
+{
+ struct enq_blocked_stats stats = {
+ .hold_time_ns = atomic64_read(&hold_time_ns),
+ .wait_time_ns = atomic64_read(&wait_time_ns),
+ .nr_holds = atomic64_read(&nr_holds),
+ .nr_waits = atomic64_read(&nr_waits),
+ };
+
+ return copy_to_user((void __user *)arg, &stats, sizeof(stats)) ?
+ -EFAULT : 0;
+}
+
+static long enq_blocked_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ switch (cmd) {
+ case ENQ_BLOCKED_IOCTL_OWNER:
+ return run_owner();
+ case ENQ_BLOCKED_IOCTL_DONOR:
+ return run_donor();
+ case ENQ_BLOCKED_IOCTL_RESET_STATS:
+ reset_stats();
+ return 0;
+ case ENQ_BLOCKED_IOCTL_GET_STATS:
+ return get_stats(arg);
+ case ENQ_BLOCKED_IOCTL_PREP_ATTACH:
+ atomic_set(&attach_done, 0);
+ atomic_set(&attach_pending, 1);
+ return 0;
+ case ENQ_BLOCKED_IOCTL_ATTACH_DONE:
+ atomic_set(&attach_done, 1);
+ return 0;
+ case ENQ_BLOCKED_IOCTL_DONOR_STATE:
+ return get_donor_state();
+ default:
+ return -EINVAL;
+ }
+}
+
+static const struct file_operations enq_blocked_fops = {
+ .owner = THIS_MODULE,
+ .unlocked_ioctl = enq_blocked_ioctl,
+};
+
+static struct miscdevice enq_blocked_device = {
+ .minor = MISC_DYNAMIC_MINOR,
+ .name = "scx_enq_blocked",
+ .fops = &enq_blocked_fops,
+ .mode = 0600,
+};
+
+module_misc_device(enq_blocked_device);
+MODULE_AUTHOR("Andrea Righi <arighi@nvidia.com>");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("sched_ext blocked donor test module");
--
2.55.0
^ permalink raw reply [flat|nested] 23+ messages in thread