mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: juri.lelli@redhat.com, peterz@infradead.org, mingo@redhat.com,
	vincent.guittot@linaro.org
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	tglx@kernel.org, will@kernel.org, boqun@kernel.org,
	dietmar.eggemann@arm.com, rostedt@goodmis.org,
	bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
	kprateek.nayak@amd.com, luca.abeni@santannapisa.it,
	mathieu.poirier@linaro.org, longman@redhat.com,
	ridong.chen@linux.dev, tj@kernel.org, hannes@cmpxchg.org,
	mkoutny@suse.com, shuah@kernel.org, bigeasy@linutronix.de,
	clrkwllms@kernel.org, parri.andrea@gmail.com,
	m.szyprowski@samsung.com, kayracizmeci@gmail.com,
	weihengqiuu@gmail.com, linux-kernel@vger.kernel.org,
	cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org,
	linux-rt-devel@lists.linux.dev, stable@vger.kernel.org
Subject: [PATCH v10 23/40] sched/deadline: Hold a reference on the deadline donor of a boosted task
Date: Fri, 20 Nov 2026 15:42:00 -0500	[thread overview]
Message-ID: <19834911fdb02b3941d9021425e247c4b597b210.1795207232.git.mathieu.desnoyers@efficios.com> (raw)
In-Reply-To: <cover.1795207232.git.mathieu.desnoyers@efficios.com>

A task boosted by a SCHED_DEADLINE waiter runs on the parameters of a
donor, dl.pi_se. Since commit 2279f540ea7d ("sched/deadline: Fix
priority inheritance with multiple scheduling classes"),
rt_mutex_setprio() copies the donor of the top waiter to the owner. So
each owner along a chain points at the deadline task at its head.

rt_mutex_setprio() keeps the top waiter, pi_top_task, as a plain
pointer. The top waiter is blocked, so it cannot exit. The donor further
up the chain has no such guarantee. When it stops waiting, the owners
below it are fixed by chain walks. A walk can be preempted, and it runs
in the context of the task that started it, not the donor's.

M, O and P are SCHED_OTHER tasks. W is a SCHED_DEADLINE task:

  P         O              M                          W

  locks L3  locks L2       locks L1
            blocks on L3
                           blocks on L2
                                                      blocks on L1
                                                        M, O, P get W's parameters
                           times out on L2
                             O deboosted
                             walk from O: preempted
                                                      times out on L1
                                                        M not blocked: no walk
                                                      exits
  runs out of runtime
    refill reads P->dl.pi_se:
      W's freed task_struct

P would run on parameters read from freed memory, until M's walk
reaches it.

Setting up the deadline task needs CAP_SYS_NICE. The timeout of a
FUTEX_LOCK_PI needs no privilege. This was found by reading.

Fix this by holding a reference on the donor's task, for each task whose
donor is another task. dl_set_donor() gives a task the donor of its top
waiter. dl_clear_donor() ends the boost. Both move the reference.
put_task_struct() frees the task through call_rcu(), so it can run
under the rq lock. A late walk can still leave P on the parameters of
an exited donor for a while. P now reads them from valid memory.

dl_set_donor() reads the donor of the top waiter without the waiter's
locks. Another CPU can change that donor at the same time, and put the
last reference on the old one. So dl_set_donor() takes its reference
with tryget_task_struct(). If that fails, it reads the new donor
again.

put_task_struct() frees every task through call_rcu() since v6.17:
commit 8671bad873eb ("sched: Do not call __put_task_struct() on rt if
pi_blocked_on is set"). Before it, put_task_struct() frees the task at
once without PREEMPT_RT. So stable trees older than v6.17 need that
commit first.

__dl_clear_params() no longer resets the donor. switched_from_dl() can
call it on a boosted task, before the deboost drops the reference.
init_dl_entity() resets it for a new task, whose copy of the pointer
holds no reference.

No case tests the use after free: it needs a walk preempted at the right
step.

Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org # 6.16.x and older: 8671bad873eb: sched: Do not call __put_task_struct() on rt if pi_blocked_on is set
Cc: stable@vger.kernel.org
---
 kernel/sched/core.c     |  8 +++---
 kernel/sched/deadline.c | 12 ++++++---
 kernel/sched/sched.h    | 56 +++++++++++++++++++++++++++++++++++++++++
 kernel/sched/syscalls.c |  4 +--
 4 files changed, 70 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 3dd42581cc17..9831b4aa5235 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7969,19 +7969,19 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 			if (!dl_prio(p->normal_prio) ||
 			    (pi_task && dl_prio(pi_task->prio) &&
 			     dl_entity_preempt(&pi_task->dl, &p->dl))) {
-				p->dl.pi_se = pi_task->dl.pi_se;
+				dl_set_donor(&p->dl, pi_task);
 				scope->flags |= ENQUEUE_REPLENISH;
 			} else {
-				p->dl.pi_se = &p->dl;
+				dl_clear_donor(&p->dl);
 			}
 		} else if (rt_prio(prio)) {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				dl_clear_donor(&p->dl);
 			if (oldprio < prio)
 				scope->flags |= ENQUEUE_HEAD;
 		} else {
 			if (dl_prio(oldprio))
-				p->dl.pi_se = &p->dl;
+				dl_clear_donor(&p->dl);
 			if (rt_prio(oldprio))
 				p->rt.timeout = 0;
 		}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index cc5be00b2b0c..a566e9252a41 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -5080,10 +5080,6 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_defer_running		= 0;
 	dl_se->dl_defer_armed		= 0;
 	dl_se->dl_bw_oncpu		= 0;
-
-#ifdef CONFIG_RT_MUTEXES
-	dl_se->pi_se			= dl_se;
-#endif
 }
 
 void init_dl_entity(struct sched_dl_entity *dl_se)
@@ -5098,6 +5094,14 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
 	 */
 	dl_se->dl_bw = 0;
 	__dl_clear_params(dl_se);
+#ifdef CONFIG_RT_MUTEXES
+	/*
+	 * The copy points at the donor of the parent, without a reference.
+	 * Reset it here only: __dl_clear_params() can run on a boosted entity,
+	 * from switched_from_dl() before the deboost drops its reference.
+	 */
+	dl_se->pi_se = dl_se;
+#endif
 	dl_se->dl_bw_cpu = -1;
 	dl_se->dl_bw_resv_cpu = -1;
 	INIT_LIST_HEAD(&dl_se->dl_departed_node);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e2bb192f1ee6..859d782cc899 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -356,6 +356,62 @@ extern int  sched_dl_global_validate(void);
 extern void sched_dl_do_global(void);
 extern int  sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
 extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+#ifdef CONFIG_RT_MUTEXES
+/*
+ * Make @pi_se the deadline donor of @dl_se. The caller holds a reference
+ * on the task of @pi_se, unless @pi_se is @dl_se. Put the reference on
+ * the old donor.
+ */
+static inline void dl_replace_donor(struct sched_dl_entity *dl_se,
+				    struct sched_dl_entity *pi_se)
+{
+	struct sched_dl_entity *old = dl_se->pi_se;
+
+	WRITE_ONCE(dl_se->pi_se, pi_se);
+	if (old != dl_se)
+		put_task_struct(container_of(old, struct task_struct, dl));
+}
+
+/*
+ * Make the deadline donor of the waiter @pi_task the donor of @dl_se. A
+ * boosted entity holds a reference on the task of its donor. The donor
+ * is the deadline task at the head of the chain. It can stop waiting
+ * and exit before the chain walk reaches @dl_se: that walk can be
+ * preempted.
+ *
+ * The donor of @pi_task is read without the locks of @pi_task. Another
+ * CPU can change it meanwhile, and put the last reference on the old
+ * donor. That CPU stores the new donor first: read it again then. The
+ * old task_struct stays readable while the rq lock is held:
+ * put_task_struct() frees it through call_rcu(), and the rq lock holds
+ * off the grace period.
+ *
+ * The caller holds the rq lock and the pi_lock of the task of @dl_se.
+ */
+static inline void dl_set_donor(struct sched_dl_entity *dl_se,
+				struct task_struct *pi_task)
+{
+	struct sched_dl_entity *pi_se;
+
+	for (;;) {
+		pi_se = READ_ONCE(pi_task->dl.pi_se);
+		if (pi_se == dl_se->pi_se)
+			return;
+		if (pi_se == dl_se ||
+		    tryget_task_struct(container_of(pi_se, struct task_struct, dl)))
+			break;
+		/* Pairs with the ordering of refcount_dec_and_test(). */
+		smp_rmb();
+	}
+	dl_replace_donor(dl_se, pi_se);
+}
+
+/* Make @dl_se its own donor. The caller holds the rq lock of its task. */
+static inline void dl_clear_donor(struct sched_dl_entity *dl_se)
+{
+	dl_replace_donor(dl_se, dl_se);
+}
+#endif
 extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
 extern bool __checkparam_dl(const struct sched_attr *attr);
 extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 2032c37d15e9..0343c7831101 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -300,7 +300,7 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
 		struct task_struct *pi_task = rt_mutex_get_top_task(p);
 
 		if (pi_task) {
-			p->dl.pi_se = pi_task->dl.pi_se;
+			dl_set_donor(&p->dl, pi_task);
 			scope->flags |= ENQUEUE_REPLENISH;
 		}
 	}
@@ -310,7 +310,7 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
 	 * the class. Drop its boost then, as rt_mutex_setprio() does.
 	 */
 	if (!dl_prio(newprio))
-		p->dl.pi_se = &p->dl;
+		dl_clear_donor(&p->dl);
 }
 #else /* !CONFIG_RT_MUTEXES */
 static inline void __setscheduler_dl_pi(int newprio, int policy,
-- 
2.47.3


  parent reply	other threads:[~2026-11-20 20:42 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 20:41 [PATCH v10 00/40] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 01/40] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 02/40] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 03/40] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 04/40] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 05/40] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 06/40] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 07/40] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 08/40] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 09/40] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 10/40] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 11/40] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 12/40] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 13/40] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 14/40] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 15/40] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 16/40] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 17/40] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 18/40] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:54   ` sashiko-bot
2026-11-20 20:41 ` [PATCH v10 19/40] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 20/40] sched/deadline: Do not truncate a bandwidth to 32 bits in __dl_add/sub() Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 21/40] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
2026-11-20 20:41 ` [PATCH v10 22/40] rtmutex: Do not stop the chain walk at an owner whose top waiter changed Mathieu Desnoyers
2026-11-20 20:42 ` Mathieu Desnoyers [this message]
2026-11-20 20:42 ` [PATCH v10 24/40] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
2026-11-20 21:06   ` sashiko-bot
2026-11-20 20:42 ` [PATCH v10 25/40] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 26/40] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 27/40] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 28/40] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 29/40] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 30/40] sched/deadline: Start a period for a task requeued onto an rt_mutex Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 31/40] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 32/40] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 33/40] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 34/40] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 35/40] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 36/40] sched/deadline: Reschedule a running task throttled at its enqueue Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 37/40] sched/deadline: Keep the start of the period across a change of dl_deadline Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 38/40] sched/deadline: Boost an owner that cannot run on its own reservation Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 39/40] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
2026-11-20 20:42 ` [PATCH v10 40/40] selftests/sched: Share the helpers of the deadline tests Mathieu Desnoyers

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=19834911fdb02b3941d9021425e247c4b597b210.1795207232.git.mathieu.desnoyers@efficios.com \
    --to=mathieu.desnoyers@efficios.com \
    --cc=bigeasy@linutronix.de \
    --cc=boqun@kernel.org \
    --cc=bsegall@google.com \
    --cc=cgroups@vger.kernel.org \
    --cc=clrkwllms@kernel.org \
    --cc=dietmar.eggemann@arm.com \
    --cc=hannes@cmpxchg.org \
    --cc=juri.lelli@redhat.com \
    --cc=kayracizmeci@gmail.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=longman@redhat.com \
    --cc=luca.abeni@santannapisa.it \
    --cc=m.szyprowski@samsung.com \
    --cc=mathieu.poirier@linaro.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=mkoutny@suse.com \
    --cc=parri.andrea@gmail.com \
    --cc=peterz@infradead.org \
    --cc=ridong.chen@linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=shuah@kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tglx@kernel.org \
    --cc=tj@kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=weihengqiuu@gmail.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®