mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: juri.lelli@redhat.com, peterz@infradead.org, mingo@redhat.com,
	vincent.guittot@linaro.org
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	tglx@kernel.org, will@kernel.org, boqun@kernel.org,
	dietmar.eggemann@arm.com, rostedt@goodmis.org,
	bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
	kprateek.nayak@amd.com, luca.abeni@santannapisa.it,
	mathieu.poirier@linaro.org, longman@redhat.com,
	ridong.chen@linux.dev, tj@kernel.org, hannes@cmpxchg.org,
	mkoutny@suse.com, shuah@kernel.org, bigeasy@linutronix.de,
	clrkwllms@kernel.org, parri.andrea@gmail.com,
	m.szyprowski@samsung.com, kayracizmeci@gmail.com,
	weihengqiuu@gmail.com, linux-kernel@vger.kernel.org,
	cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org,
	linux-rt-devel@lists.linux.dev, stable@vger.kernel.org
Subject: [PATCH v10 38/40] sched/deadline: Boost an owner that cannot run on its own reservation
Date: Fri, 20 Nov 2026 15:42:15 -0500	[thread overview]
Message-ID: <297755d1d8dde37f65e12a4cd502c6332011f44c.1795207233.git.mathieu.desnoyers@efficios.com> (raw)
In-Reply-To: <cover.1795207232.git.mathieu.desnoyers@efficios.com>

rt_mutex_setprio() lets a deadline waiter boost the owner of the lock if
the waiter's deadline is earlier than the owner's.
Commit 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance
logic") added this test. A boosted owner is not throttled: when its
runtime runs out, it is refilled from the waiter's parameters.

The test assumes that the owner runs on its own deadline. Two kinds of
owners cannot:
- An owner that ran out of runtime is throttled until its next period.
- An owner with a constrained deadline, past its deadline and before its
  next period, is throttled at its next enqueue.
  Commit df8eac8cafce ("sched/deadline: Throttle a constrained deadline
  task activated after the deadline") does this.

Such an owner can still have the earlier deadline. A deadline that has
passed is always the earlier one. So the waiter does not boost such an
owner. It waits until the owner's next period: up to dl_period -
dl_deadline for a constrained owner. Before "sched/deadline: Reschedule
a running task throttled at its enqueue", a running owner in that gap
kept the CPU. It usually released the lock then. That patch now stops
it at once, so its waiters wait.

An owner past its deadline and not in that gap gets a new period at its
next enqueue. Its deadline is then now + dl_deadline, which can be
after the waiter's. The waiter does not boost it either.

Fix this by comparing the waiter with the deadline that the owner would
run on, in dl_waiter_boosts():
- A throttled owner, or one in that gap, is always boosted. The boost
  lasts until the owner's own next period.
- Another owner past its deadline is compared at now + dl_deadline.

dl_boost_start() saves the deadline, runtime and period start of an
owner boosted this way, and dl_deboost() puts them back. The time the
owner ran boosted was the donor's. If the owner blocked while boosted,
its inactive timer was armed from the donor's runtime and deadline.
dl_deboost() then works out its 0-lag time again. replenish_dl_entity()
no longer warns about a boosted owner with an expired deadline. It
starts a period of the donor.

The Sashiko AI review of this series found the gap case. The cases
pi_owner_asleep_in_gap and pi_owner_throttled of deadline_bw_test,
added later in this series, test the two kinds of owners. Both fail
without this patch. pi_owner_throttled_asleep tests the 0-lag time
worked out again at the deboost. It fails without this patch too.

Fixes: 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance logic")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
 include/linux/sched.h   |  12 +++++
 kernel/sched/core.c     |   9 +++-
 kernel/sched/deadline.c | 104 +++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h    |   3 ++
 4 files changed, 113 insertions(+), 15 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 244e57fa8c21..3282fc337715 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -662,6 +662,10 @@ struct sched_dl_entity {
 	s64				runtime;	/* Remaining runtime for this instance	*/
 	u64				deadline;	/* Absolute deadline for this instance	*/
 	u64				period_start;	/* Absolute start of this instance	*/
+	/* Saved by dl_boost_start(): see dl_boost_held */
+	u64				own_deadline;
+	s64				own_runtime;
+	u64				own_period_start;
 	unsigned int			flags;		/* Specifying the scheduler behaviour	*/
 
 	/*
@@ -720,6 +724,13 @@ struct sched_dl_entity {
 	 * @dl_new_params tells if sched_setattr() changed the runtime, the
 	 * deadline or the period. The next enqueue that is not boosted, or
 	 * the deboost, checks the runtime left against them, and clears it.
+	 *
+	 * @dl_boost_held tells if a waiter boosts the task because the task
+	 * could not run on its own reservation: it was throttled, or past its
+	 * deadline and before its next period. The boost then lasts until
+	 * the task's own next period at least. The deboost puts back the
+	 * deadline, runtime and period start saved in own_deadline,
+	 * own_runtime and own_period_start.
 	 */
 	unsigned int			dl_throttled      : 1;
 	unsigned int			dl_yielded        : 1;
@@ -735,6 +746,7 @@ struct sched_dl_entity {
 	unsigned int			dl_bw_attached    : 1;
 	unsigned int			dl_bw_oncpu       : 1;
 	unsigned int			dl_new_params     : 1;
+	unsigned int			dl_boost_held     : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0dce129b4146..cbe0dfe26de3 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7974,10 +7974,13 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 			if (!dl_prio(p->normal_prio) ||
 			    (pi_task && dl_prio(pi_task->prio) &&
 			     !(p->dl.dl_new_params && p->dl.pi_se != &p->dl) &&
-			     dl_entity_preempt(&pi_task->dl, &p->dl))) {
+			     dl_waiter_boosts(p, pi_task))) {
 				struct sched_dl_entity *old_donor = p->dl.pi_se;
 
 				dl_set_donor(&p->dl, pi_task);
+				if (dl_prio(p->normal_prio) && old_donor == &p->dl &&
+				    p->dl.pi_se != &p->dl)
+					dl_boost_start(&p->dl);
 				scope->flags |= ENQUEUE_REPLENISH;
 				if (!scope->queued && p->dl.pi_se != old_donor)
 					dl_boost_blocked(p);
@@ -7989,8 +7992,10 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 				 * Boost the task again then.
 				 */
 				if (pi_task && dl_prio(pi_task->prio) &&
-				    dl_entity_preempt(&pi_task->dl, &p->dl)) {
+				    dl_waiter_boosts(p, pi_task)) {
 					dl_set_donor(&p->dl, pi_task);
+					if (p->dl.pi_se != &p->dl)
+						dl_boost_start(&p->dl);
 					scope->flags |= ENQUEUE_REPLENISH;
 				}
 			}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index e9b3d3ec71d7..ff1c104f6afd 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -146,6 +146,10 @@ static inline bool dl_task_pi_blocked(struct task_struct *p)
  * first. Without a refill, the task keeps the runtime and the deadline
  * of its own reservation.
  *
+ * A boost that dl_boost_start() held puts back the deadline, runtime
+ * and period start it saved instead. The donor's reservation paid for
+ * the time the task ran boosted.
+ *
  * If sched_setattr() changed the parameters during the boost, check the
  * runtime left against them now, for the same reason. rt_mutex_setprio()
  * then compares the result with the top waiter. Leave that check to the
@@ -161,21 +165,32 @@ static inline bool dl_task_pi_blocked(struct task_struct *p)
 void dl_deboost(struct sched_dl_entity *dl_se)
 {
 	struct task_struct *p = dl_task_of(dl_se);
+	bool own = false;
 
+	/* Put back what dl_boost_start() saved. */
+	if (dl_se->dl_boost_held) {
+		dl_se->deadline = dl_se->own_deadline;
+		dl_se->runtime = dl_se->own_runtime;
+		dl_se->period_start = dl_se->own_period_start;
+		dl_se->dl_donor_refill = 0;
+		own = true;
+	}
 	if (dl_se->dl_donor_refill) {
 		dl_start_own_period(dl_se);
-		/*
-		 * A task asleep since the refill has its inactive timer armed
-		 * from the donor's runtime and deadline. The 0-lag time of the
-		 * new period is now: stop counting the task in running_bw
-		 * now. If the timer handler is already running, it does that.
-		 */
-		if (dl_se->dl_non_contending &&
-		    hrtimer_try_to_cancel(&dl_se->inactive_timer) == 1) {
-			dl_se->dl_non_contending = 0;
-			put_task_struct(dl_task_of(dl_se));
-			task_non_contending(dl_se, true);
-		}
+		own = true;
+	}
+	/*
+	 * A task that blocked while boosted has its inactive timer armed
+	 * from the donor's runtime and deadline. Cancel it, and work out the
+	 * 0-lag time again from the runtime and deadline the task gets back.
+	 * After a new period, that 0-lag time is now. If the timer handler
+	 * is already running, it releases the bandwidth.
+	 */
+	if (own && dl_se->dl_non_contending &&
+	    hrtimer_try_to_cancel(&dl_se->inactive_timer) == 1) {
+		dl_se->dl_non_contending = 0;
+		put_task_struct(p);
+		task_non_contending(dl_se, true);
 	}
 	dl_drop_boost(dl_se);
 
@@ -214,6 +229,62 @@ void dl_check_blocked_params(struct task_struct *p)
 		update_dl_entity(dl_se);
 	}
 }
+
+/*
+ * Return true if the deadline waiter @pi_task must boost the owner @p,
+ * which has a deadline policy of its own. Compare @pi_task with the
+ * deadline that @p would run on:
+ * - @p is throttled, or past its deadline and before its next period:
+ *   it cannot run on its own reservation. Boost it.
+ * - @p was boosted for that reason: keep the boost until its own next
+ *   period. After that period starts, compare with now + dl_deadline.
+ * - @p is past its deadline otherwise: its next enqueue starts a new
+ *   period. Compare with now + dl_deadline.
+ * - Else, compare with the deadline of @p.
+ * A special entity, such as a sugov kthread, always boosts @p.
+ *
+ * The caller holds the rq lock of @p and has updated the rq clock.
+ */
+bool dl_waiter_boosts(struct task_struct *p, struct task_struct *pi_task)
+{
+	struct sched_dl_entity *dl_se = &p->dl;
+	u64 now = rq_clock(task_rq(p));
+
+	if (dl_entity_is_special(&pi_task->dl))
+		return true;
+	if (dl_se->dl_boost_held) {
+		if (dl_time_before(now, dl_se->own_period_start + dl_se->dl_period))
+			return true;
+		return dl_time_before(pi_task->dl.deadline, now + dl_se->dl_deadline);
+	}
+	if (!is_dl_boosted(dl_se)) {
+		if (dl_se->dl_throttled || dl_in_constrained_gap(dl_se))
+			return true;
+		if (dl_time_before(dl_se->deadline, now))
+			return dl_time_before(pi_task->dl.deadline,
+					      now + dl_se->dl_deadline);
+	}
+	return dl_entity_preempt(&pi_task->dl, dl_se);
+}
+
+/*
+ * Call this when @dl_se gets its first donor, after dl_set_donor()
+ * installed it. If @dl_se cannot run on its own reservation, save its
+ * deadline, runtime and period start in own_*. Then set dl_boost_held.
+ * dl_deboost() puts them back. A task with no parameters of its own has
+ * nothing to save. The caller holds the rq lock and has updated the rq
+ * clock.
+ */
+void dl_boost_start(struct sched_dl_entity *dl_se)
+{
+	if (!dl_se->dl_runtime ||
+	    (!dl_se->dl_throttled && !dl_in_constrained_gap(dl_se)))
+		return;
+	dl_se->own_deadline = dl_se->deadline;
+	dl_se->own_runtime = dl_se->runtime;
+	dl_se->own_period_start = dl_se->period_start;
+	dl_se->dl_boost_held = 1;
+}
 #else /* !CONFIG_RT_MUTEXES: */
 static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
 {
@@ -1443,7 +1514,13 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 * entity.
 	 */
 	if (dl_time_before(dl_se->deadline, rq_clock(rq))) {
-		printk_deferred_once("sched: DL replenish lagged too much\n");
+		/*
+		 * dl_waiter_boosts() boosts an owner past its deadline. Its
+		 * deadline is then expected to be in the past: do not warn.
+		 * The new period takes the donor's parameters.
+		 */
+		if (!is_dl_boosted(dl_se))
+			printk_deferred_once("sched: DL replenish lagged too much\n");
 		replenish_dl_new_period(dl_se, rq);
 	}
 
@@ -5420,6 +5497,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer_armed		= 0;
 	dl_se->dl_bw_oncpu		= 0;
 	dl_se->dl_new_params		= 0;
+	dl_se->dl_boost_held		= 0;
 }
 
 void init_dl_entity(struct sched_dl_entity *dl_se)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4151364ca4ff..447aac4c0227 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -415,12 +415,15 @@ static inline void dl_clear_donor(struct sched_dl_entity *dl_se)
 extern void dl_deboost(struct sched_dl_entity *dl_se);
 extern void dl_boost_blocked(struct task_struct *p);
 extern void dl_check_blocked_params(struct task_struct *p);
+extern bool dl_waiter_boosts(struct task_struct *p, struct task_struct *pi_task);
+extern void dl_boost_start(struct sched_dl_entity *dl_se);
 
 /* End the boost of @dl_se. The caller holds the rq lock of its task. */
 static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
 {
 	dl_clear_donor(dl_se);
 	dl_se->dl_donor_refill = 0;
+	dl_se->dl_boost_held = 0;
 }
 #endif
 extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
-- 
2.47.3


  parent reply	other threads:[~2026-11-20 20:42 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 20:41 [PATCH v10 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 01/40] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 02/40] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 03/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 04/40] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 05/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 06/40] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 07/40] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 08/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 09/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 10/40] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 11/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 12/40] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 13/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 14/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 15/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 16/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 17/40] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 18/40] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:54   ` sashiko-bot
2026-11-20 20:41 ` [PATCH v10 19/40] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 20/40] sched/deadline: Do not truncate a bandwidth to 32 bits in __dl_add/sub() Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 21/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 22/40] rtmutex: Do not stop the chain walk at an owner whose top waiter changed Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 23/40] sched/deadline: Hold a reference on the deadline donor of a boosted task Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 24/40] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
2026-11-20 21:06   ` sashiko-bot
2026-11-20 20:42 ` [PATCH v10 25/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 26/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 27/40] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 28/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 29/40] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 30/40] sched/deadline: Start a period for a task requeued onto an rt_mutex Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 31/40] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 32/40] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 33/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 34/40] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 35/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 36/40] sched/deadline: Reschedule a running task throttled at its enqueue Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 37/40] sched/deadline: Keep the start of the period across a change of dl_deadline Mathieu Desnoyers
2026-11-20 20:42 ` Mathieu Desnoyers [this message]
2026-11-20 20:42 ` [PATCH v10 39/40] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 40/40] selftests/sched: Share the helpers of the deadline tests Mathieu Desnoyers

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=297755d1d8dde37f65e12a4cd502c6332011f44c.1795207233.git.mathieu.desnoyers@efficios.com \
    --to=mathieu.desnoyers@efficios.com \
    --cc=bigeasy@linutronix.de \
    --cc=boqun@kernel.org \
    --cc=bsegall@google.com \
    --cc=cgroups@vger.kernel.org \
    --cc=clrkwllms@kernel.org \
    --cc=dietmar.eggemann@arm.com \
    --cc=hannes@cmpxchg.org \
    --cc=juri.lelli@redhat.com \
    --cc=kayracizmeci@gmail.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=longman@redhat.com \
    --cc=luca.abeni@santannapisa.it \
    --cc=m.szyprowski@samsung.com \
    --cc=mathieu.poirier@linaro.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=mkoutny@suse.com \
    --cc=parri.andrea@gmail.com \
    --cc=peterz@infradead.org \
    --cc=ridong.chen@linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=shuah@kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tglx@kernel.org \
    --cc=tj@kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=weihengqiuu@gmail.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®