* [PATCH sched_ext/for-7.4-fixes 0/2] sched_ext: Fix two gaps in dl_server protection against RT starvation
@ 2026-11-22 18:04 Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 1/2] sched_ext: Attach fair_server before tasks fall back to fair on disable Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 2/2] sched_ext: Start ext_server on every enqueue Hang Zhengyang via B4 Relay
0 siblings, 2 replies; 3+ messages in thread
From: Hang Zhengyang via B4 Relay @ 2026-11-22 18:04 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Andrea Righi, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak
Cc: Joel Fernandes, sched-ext, linux-kernel, stable, Hang Zhengyang
These patches fix two cases where a dl_server should protect tasks from
an RT task but isn't running. The starved tasks then hold up the
scheduler disable. It doesn't finish until the RT task stops.
Patch 1: in full mode, the disable attaches fair_server only at the end.
Tasks move back to fair long before that.
Patch 2: a CPU's ext_server restarts only when the first sched_ext task
is enqueued there. If it stops while tasks are still queued, nothing
restarts it.
Both came out of one stress test (scx_simple, CPU hotplug, stress-ng,
SCHED_FIFO spinners). The patches don't depend on each other.
Tested on sched_ext/for-7.4-fixes (8e156b9218f9) in an 8 vCPU KVM guest
with PREEMPT, PREEMPT_RCU and NO_HZ_FULL:
no patches patch 1 patch 2 both
patch 1 repro 37/300 stuck 0/300 - 0/300
same, KASAN+lockdep 3/100 stuck 0/100 - 0/100
stress, KASAN+lockdep hung, 4 min hung, 2-5 min 1/3 hung 6/7 ok [*]
stress, no debug - 2/2 ok - 3/3 ok
selftests - 38 ok, 2 skip - 38 ok, 2 skip
messaging [**] 1.849 s - - 1.821 s
Each stress run is 15 minutes.
[*] The 7th run stopped at 2.5 min. A fork stalled for 75 s with the
scheduler enabled, then went on. Not the disable hang. Cause not
found.
[**] perf bench sched messaging -g 8 -l 1000 under scx_simple, mean of
15 runs
With patch 2 alone, the one stress hang was the problem patch 1 fixes.
Patch 2's cost: during a scheduler reload, ext_server restarted and
stopped 238 times in 41 ms on one CPU. I also ran an RT spinner with
sched_ext tasks on the same CPU, under scx_simple and scx_central.
There, patch 2 changed neither the restart count nor the time the RT
task lost.
I didn't test on bare metal, or with other schedulers outside that cost
test. I didn't write selftests, because both hangs depend on timing.
I used Claude Code to find the issues, write the patches and run the
tests. I reran the reproducer and reviewed the patches.
---
Hang Zhengyang (2):
sched_ext: Attach fair_server before tasks fall back to fair on disable
sched_ext: Start ext_server on every enqueue
kernel/sched/ext/ext.c | 27 ++++++++++++++++++++++-----
1 file changed, 22 insertions(+), 5 deletions(-)
---
base-commit: 8e156b9218f9f1f8b866d8e253145afdb01cf289
change-id: 20261121-scx-dlsrv-7d44ef45820e
Best regards,
--
Hang Zhengyang <hang.zhengyang1010@gmail.com>
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH sched_ext/for-7.4-fixes 1/2] sched_ext: Attach fair_server before tasks fall back to fair on disable
2026-11-22 18:04 [PATCH sched_ext/for-7.4-fixes 0/2] sched_ext: Fix two gaps in dl_server protection against RT starvation Hang Zhengyang via B4 Relay
@ 2026-11-22 18:04 ` Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 2/2] sched_ext: Start ext_server on every enqueue Hang Zhengyang via B4 Relay
1 sibling, 0 replies; 3+ messages in thread
From: Hang Zhengyang via B4 Relay @ 2026-11-22 18:04 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Andrea Righi, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak
Cc: Joel Fernandes, sched-ext, linux-kernel, stable, Hang Zhengyang
From: Hang Zhengyang <hang.zhengyang1010@gmail.com>
In full mode, the enable path detaches fair_server, since no task runs
in the fair class. scx_root_disable() attaches it again only at the end,
in the dl_server_swap_bw() loop. But tasks fall back to fair well before
that. New forks and sched_setscheduler() calls do so once
scx_switching_all is cleared, and the switch loop moves all other tasks.
Until the swap, fair_server can't start, so an RT task that keeps a CPU
starves the fair tasks there.
Meanwhile, the disable path waits for RCU grace periods in
percpu_down_write() of scx_fork_rwsem and scx_cgroup_ops_rwsem. A grace
period can wait for a starved task, so the disable doesn't finish until
the RT task stops. The starved task can be a reader preempted under
PREEMPT_RCU, or the grace-period kthread itself. While the disable waits
in the scx_cgroup_lock() after the switch loop, scx_fork_rwsem and
cgroup_mutex are held, so forks wait too.
To reproduce, load scx_simple, pin SCHED_NORMAL tasks (/proc walkers,
stress-ng --fork) and a SCHED_FIFO spinner to CPU 1, and unload the
scheduler 50 ms after the spinner starts. The disable can then get
stuck in:
synchronize_rcu_normal+0x1da/0x1f0
rcu_sync_enter+0x56/0x140
scx_root_disable+0x230/0x420
kthread_worker_fn+0xf0/0x270
until the spinner stops.
Attach fair_server before tasks start to fall back, the same way the
enable path attaches ext_server before tasks switch to it. If the root
domain can't fit both servers, the swap attaches fair_server as before.
Fixes: e7b63427fdb4 ("sched_ext: Auto-register/unregister dl_server reservations")
Cc: stable@vger.kernel.org # v7.2+
Signed-off-by: Hang Zhengyang <hang.zhengyang1010@gmail.com>
---
kernel/sched/ext/ext.c | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index e8c30a21e03e..23294897dd6e 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -7576,6 +7576,21 @@ static void scx_root_disable(struct scx_sched *sch)
was_switched_all = scx_switched_all();
+ /*
+ * Tasks fall back to fair from here on. Attach fair_server now, or an
+ * RT task can starve them while we wait for RCU grace periods below.
+ * If its bandwidth doesn't fit, the swap at the end attaches it.
+ */
+ if (was_switched_all) {
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+
+ guard(rq_lock_irqsave)(rq);
+ update_rq_clock(rq);
+ dl_server_attach_bw(&rq->fair_server);
+ }
+ }
+
static_branch_disable(&__scx_switched_all);
WRITE_ONCE(scx_switching_all, false);
WRITE_ONCE(scx_task_switch_enabled, false);
@@ -7626,8 +7641,8 @@ static void scx_root_disable(struct scx_sched *sch)
/*
* Re-balance the dl_server bandwidth reservations: detach ext_server
- * (no more sched_ext tasks) and reinstate fair_server if it was
- * previously detached because we were running in full mode.
+ * (no more sched_ext tasks) and attach fair_server if the attach
+ * above failed.
*
* Unlike the enable path, this runs on a recovery path that cannot
* fail, so we use dl_server_swap_bw() to atomically free ext_server's
--
2.53.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH sched_ext/for-7.4-fixes 2/2] sched_ext: Start ext_server on every enqueue
2026-11-22 18:04 [PATCH sched_ext/for-7.4-fixes 0/2] sched_ext: Fix two gaps in dl_server protection against RT starvation Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 1/2] sched_ext: Attach fair_server before tasks fall back to fair on disable Hang Zhengyang via B4 Relay
@ 2026-11-22 18:04 ` Hang Zhengyang via B4 Relay
1 sibling, 0 replies; 3+ messages in thread
From: Hang Zhengyang via B4 Relay @ 2026-11-22 18:04 UTC (permalink / raw)
To: Tejun Heo, David Vernet, Andrea Righi, Changwoo Min, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider,
K Prateek Nayak
Cc: Joel Fernandes, sched-ext, linux-kernel, stable, Hang Zhengyang
From: Hang Zhengyang <hang.zhengyang1010@gmail.com>
Since commit a3a70caf7906 ("sched/deadline: Fix dl_server behaviour"),
ext_server stops when a pick finds no task, and a later wakeup starts
it again. But enqueue_task_scx() starts it only when
rq->scx.nr_running goes from 0 to 1.
For sched_ext, a pick can find no task while tasks are still counted on
the rq. During a disable, scx_claim_exit() sets sch->aborting. After
that, scx_consume_dispatch_q() no longer takes tasks from the
scheduler's DSQs. nr_running stays above 0, so no enqueue restarts the
server. An RT task that keeps the CPU then starves the sched_ext tasks
there.
A stress test hit this on a debug kernel. While CPU hotplug was
disabling scx_simple, the server on CPU 0 stopped with 3 tasks counted.
A SCHED_FIFO spinner then took the CPU with 15 tasks behind it. One of
them was the RCU grace-period kthread:
rcu: rcu_preempt kthread starved for 65011 jiffies! ... ->state=R ->cpu=0
The disable was waiting for a reader of scx_fork_rwsem and never
finished.
Call dl_server_start() on every enqueue. On a running server, it only
clears dl_defer_idle. On a stopped server that ran in this period, it
restarts the server at once and reschedules the CPU, even under an RT
task. So while picks find nothing, each enqueue costs one extra pick.
This lasts at most one server period (1 s by default).
Peter Zijlstra asked whether the server should start only when a task
reaches the local DSQ [1]. That is not enough. A task often gets to the
local DSQ only through this CPU's own pick, and that pick needs the
server to run.
This doesn't cover a task that another CPU dispatches to this CPU's
local DSQ.
Link: https://lore.kernel.org/all/20260202195035.GG1282955@noisy.programming.kicks-ass.net/ [1]
Fixes: cd959a356205 ("sched_ext: Add a DL server for sched_ext tasks")
Cc: stable@vger.kernel.org # v7.0+
Signed-off-by: Hang Zhengyang <hang.zhengyang1010@gmail.com>
---
kernel/sched/ext/ext.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 23294897dd6e..1031267508fd 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -2501,9 +2501,11 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
if (SCX_HAS_OP(sch, runnable) && !task_on_rq_migrating(p))
SCX_CALL_OP_TASK(sch, runnable, rq, p, enq_flags);
- /* Start dl_server if this is the first task being enqueued */
- if (rq->scx.nr_running == 1)
- dl_server_start(&rq->ext_server);
+ /*
+ * Start on every enqueue: a pick can stop the server while tasks
+ * counted here are out of its reach.
+ */
+ dl_server_start(&rq->ext_server);
scx_do_enqueue_task(rq, p, enq_flags, sticky_cpu);
out:
--
2.53.0
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-22 18:05 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-22 18:04 [PATCH sched_ext/for-7.4-fixes 0/2] sched_ext: Fix two gaps in dl_server protection against RT starvation Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 1/2] sched_ext: Attach fair_server before tasks fall back to fair on disable Hang Zhengyang via B4 Relay
2026-11-22 18:04 ` [PATCH sched_ext/for-7.4-fixes 2/2] sched_ext: Start ext_server on every enqueue Hang Zhengyang via B4 Relay
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®