Re: sched/deadline: total_bw leak when a task leaves SCHED_DEADLINE while not queued

[Juri Lelli <juri.lelli@redhat.com>]

Hello,

On 29/10/26 11:55, Tejun Heo wrote:
> Hello,
> 
> A SCHED_DEADLINE task that is switched to another policy while it isn't
> queued (sleeping, or stopped with SIGSTOP) leaves its bandwidth in the root
> domain's total_bw. The bandwidth stays after the task exits. The leak
> happens as follows.
> 
> 1. sched_dl_overflow() doesn't subtract the bandwidth of a task leaving
>    DEADLINE and leaves that to switched_from_dl().
> 2. switched_from_dl() calls task_non_contending() only for a queued task.
>    task_non_contending() releases the bandwidth right away if the 0-lag time
>    has passed and otherwise arms the inactive timer, and
>    inactive_task_timer() releases it once the task is no longer dl_task().
> 3. For a task that isn't queued, the bandwidth is released only if its
>    inactive timer is still pending from its dequeue. No timer is pending if
>    the task never ran as DEADLINE, or if it already fired while the task
>    slept. In the latter case the task was still DEADLINE when the timer
>    fired, so it only dropped running_bw. Either way, nothing releases the
>    bandwidth.
> 
> Repeating it fills total_bw up to the admission cap. After that,
> sched_setattr(SCHED_DEADLINE) and CPU offline fail with -EBUSY, and so does
> enabling a sched_ext scheduler, because attaching the ext dl_server
> bandwidth fails:
> 
>   sched_ext: failed to attach ext_server on CPU 0 (-16)
>   sched_ext: simple: scx_root_enable() failed (-16)
> 
> The reproducer below needs root and debugfs, and assumes x86-64 for the
> syscall number 314.
> 
>   import ctypes, os, signal, time
>   libc = ctypes.CDLL(None, use_errno=True)
>   class SchedAttr(ctypes.Structure):
>       _fields_ = [("size", ctypes.c_uint32), ("policy", ctypes.c_uint32),
>                   ("flags", ctypes.c_uint64), ("nice", ctypes.c_int32),
>                   ("prio", ctypes.c_uint32), ("runtime", ctypes.c_uint64),
>                   ("deadline", ctypes.c_uint64), ("period", ctypes.c_uint64)]
>   def setattr_(pid, policy, runtime=0, period=0):
>       a = SchedAttr(ctypes.sizeof(SchedAttr), policy, 0, 0, 0, runtime, period, period)
>       if libc.syscall(314, pid, ctypes.byref(a), 0):      # sched_setattr
>           raise OSError(ctypes.get_errno(), "sched_setattr")
>   def total_bw():
>       s = open("/sys/kernel/debug/sched/debug").read().split("dl_rq[0]:")[1]
>       return int(s.split("dl_bw->total_bw")[1].split(":")[1].split()[0])
>   def trial(name, body, stop=False):
>       pid = os.fork()
>       if pid == 0:
>           if body == "spin":
>               while True:
>                   pass
>           while True:
>               time.sleep(3600)
>       time.sleep(0.05)
>       bw0 = total_bw()
>       setattr_(pid, 6, runtime=2_000_000, period=10_000_000)  # DEADLINE 2ms/10ms
>       if stop:
>           os.kill(pid, signal.SIGSTOP)
>           time.sleep(0.02)
>       setattr_(pid, 0)                                         # back to OTHER
>       os.kill(pid, signal.SIGKILL)
>       os.waitpid(pid, 0)
>       time.sleep(0.05)
>       print(f"{name:24s} leaked total_bw {total_bw() - bw0:7d} (0.2 CPU = 209715)")
>   trial("sleeping dl>other", "sleep")
>   trial("stopped dl>other", "spin", stop=True)
>   trial("running dl>other", "spin")                            # control
> 
> On mainline 4dd16e31135e it prints:
> 
>   sleeping dl>other        leaked total_bw  209715 (0.2 CPU = 209715)
>   stopped dl>other         leaked total_bw  209715 (0.2 CPU = 209715)
>   running dl>other         leaked total_bw       0 (0.2 CPU = 209715)

Thanks for reporting and for the reproducer!

Does the below help?

Best,
Juri

---
 kernel/sched/deadline.c | 31 ++++++++++++++++++++++++-------
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c0ebdcde5fe5d..5a51368bfc70a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3530,14 +3530,31 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
 	dec_dl_tasks_cs(p);
 
 	if (!task_on_rq_queued(p)) {
-		/*
-		 * Inactive timer is armed. However, p is leaving DEADLINE and
-		 * might migrate away from this rq while continuing to run on
-		 * some other class. We need to remove its contribution from
-		 * this rq running_bw now, or sub_rq_bw (below) will complain.
-		 */
-		if (p->dl.dl_non_contending)
+		if (p->dl.dl_non_contending) {
+			/*
+			 * Inactive timer is armed. However, p is leaving
+			 * DEADLINE and might migrate away from this rq while
+			 * continuing to run on some other class. We need to
+			 * remove its contribution from this rq running_bw now,
+			 * or sub_rq_bw (below) will complain.
+			 *
+			 * inactive_task_timer() will find !dl_task(p) and
+			 * release total_bw via __dl_sub().
+			 */
 			sub_running_bw(&p->dl, &rq->dl);
+		} else {
+			/*
+			 * No pending inactive timer: the task never ran as
+			 * DEADLINE or the timer already fired while still
+			 * DEADLINE (only dropping running_bw at that time).
+			 * Nobody else will release total_bw, do it now.
+			 */
+			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+			raw_spin_lock(&dl_b->lock);
+			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+			raw_spin_unlock(&dl_b->lock);
+		}
 		sub_rq_bw(&p->dl, &rq->dl);
 	}