From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: juri.lelli@redhat.com, peterz@infradead.org, mingo@redhat.com,
vincent.guittot@linaro.org
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
tglx@kernel.org, will@kernel.org, boqun@kernel.org,
dietmar.eggemann@arm.com, rostedt@goodmis.org,
bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
kprateek.nayak@amd.com, luca.abeni@santannapisa.it,
mathieu.poirier@linaro.org, longman@redhat.com,
ridong.chen@linux.dev, tj@kernel.org, hannes@cmpxchg.org,
mkoutny@suse.com, shuah@kernel.org, bigeasy@linutronix.de,
clrkwllms@kernel.org, parri.andrea@gmail.com,
m.szyprowski@samsung.com, kayracizmeci@gmail.com,
weihengqiuu@gmail.com, linux-kernel@vger.kernel.org,
cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-rt-devel@lists.linux.dev, stable@vger.kernel.org
Subject: [PATCH v8 09/33] sched: Make deadline policy and affinity changes wait for a cpuset attach
Date: Thu, 19 Nov 2026 15:15:48 -0500 [thread overview]
Message-ID: <771cb2b59894986a695f6b26f1ae45d013327123.1795116790.git.mathieu.desnoyers@efficios.com> (raw)
In-Reply-To: <cover.1795116790.git.mathieu.desnoyers@efficios.com>
__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. Commit 710da3c8ea7d ("sched/core: Prevent race condition
between cpuset and __sched_setscheduler()") added that lock. It keeps
the cpuset code from changing the root domains while the deadline
admission test runs.
But a cpuset attach drops cpuset_mutex between cpuset_can_attach() and
cpuset_attach(). A sched_setattr() can run in between. Take a process
in the root cgroup. It has deadline threads, and a thread T that is not
a deadline task. The process is moved into a partition root P:
echo PID > P/cgroup.procs sched_setattr(T, SCHED_DEADLINE)
cpuset_can_attach()
lock cpuset_mutex
reserve room in P for the deadline
threads: T is not one of them
unlock cpuset_mutex
lock cpuset_mutex
admit T in the root domain it
is still in
unlock cpuset_mutex
cpuset_attach()
lock cpuset_mutex
move T to P, where nothing is
reserved for it
unlock cpuset_mutex
The attach still moves the bandwidth of T. It is freed in the root
domain T left, and P has no room reserved to count it. No root domain
counts it any more, and P can admit deadline tasks beyond its capacity.
In testing, P was a one-CPU partition root, which admits 95%. The
process had two sleeping deadline threads of 30% each, and
sched_setattr() made T a deadline task of 40%. The move and the
sched_setattr() both succeeded, with no room in P for the three. With
threads of 20% and a T of 30%, P then admitted five more tasks of 10%,
where it had room for two.
Fix this by taking cpuset_mutex in __sched_setscheduler() only when no
attach is in progress. cpuset_partition_write() and
cpuset_hotplug_update_tasks() already wait for an attach that way. A
deadline policy change then comes before the attach, and is part of its
reservation, or after it.
With this patch, and no room in P for the three, the two never both
succeed. If the sched_setattr() comes first, the move is refused with
EBUSY. If it comes after, T has its new CPUs and has not run on them.
The call is then refused with EPERM, as for any sleeping task moved to
another partition.
sched_setaffinity() on a deadline task can run in between as well. A
later patch of this series reserves the bandwidth of the task on one CPU
of its new affinity, and records that CPU with the task. It is
"cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to". A new affinity in between could leave that CPU out. cpuset_attach()
would then move no bandwidth, and the task would stay counted in the
root domain it left. So sched_setaffinity() waits the same way when the
task is a deadline task.
sched_setaffinity() tests the policy with no lock, before it decides to
wait. A sched_setattr() can make the task a deadline task right after
that test. So test the policy again under the runqueue lock, where the
mask changes. If the task is a deadline task there and the caller did
not wait, the call starts over, and waits. The new SCA_DL_NOWAIT flag
asks for that test. After a race with a cpuset update,
__sched_setaffinity() sets the mask a second time, within the cpuset.
That second change waits for an attach too, so that the test cannot
skip it. It reads the cpuset mask again after the wait, since the
attach can move the task.
For user-space, these calls now wait until a cpuset attach in progress
ends:
- sched_setattr() and sched_setscheduler() on a deadline task
- sched_setattr() that makes a task a deadline task
- sched_setaffinity() on a deadline task
The attach can be the move of any process. The wait is not
interruptible: a signal does not end it. Outside that window, the
results of these calls are unchanged.
Three cases, added to deadline_bw_test later in this series, test this.
move_became_deadline and move_became_deadline_no_room fail without this
patch. move_affinity_changed fails once the bandwidth is reserved on one
CPU, if sched_setaffinity() does not wait.
A fourth case, move_affinity_became_deadline, covers the second test of
the policy. Its window is a few instructions wide. A delay added after
the first test, for testing only, widens it. The case then fails without
the second test in 3 guests of 3. It passes with it.
Fixes: 710da3c8ea7d ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 11 ++++++++
kernel/sched/core.c | 10 +++++++
kernel/sched/sched.h | 2 ++
kernel/sched/syscalls.c | 59 +++++++++++++++++++++++++++++++++++------
5 files changed, 76 insertions(+), 8 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index db1cc2fc0324..5d0c01a25354 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -73,6 +73,7 @@ extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
extern void cpuset_update_active_cpus(void);
extern void cpuset_lock(void);
+extern void cpuset_lock_no_attach(void);
extern void cpuset_unlock(void);
extern void lockdep_assert_cpuset_lock_held(void);
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
@@ -192,6 +193,7 @@ static inline void cpuset_update_active_cpus(void)
}
static inline void cpuset_lock(void) { }
+static inline void cpuset_lock_no_attach(void) { }
static inline void cpuset_unlock(void) { }
static inline void lockdep_assert_cpuset_lock_held(void) { }
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 51b6be460794..ce9c57617f49 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -378,6 +378,17 @@ static void wait_attach_done_lock(void)
}
}
+/**
+ * cpuset_lock_no_attach - Acquire cpuset_mutex with no task attach in progress
+ *
+ * A task attach drops cpuset_mutex between cpuset_can_attach() and
+ * cpuset_attach(). Use this to keep a change out of that window.
+ */
+void cpuset_lock_no_attach(void)
+{
+ wait_attach_done_lock();
+}
+
static inline void check_insane_mems_config(nodemask_t *nodes)
{
if (!cpusets_insane_config() &&
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index bd5f3dfa50ac..c0a714dde425 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,16 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * sched_setaffinity() tested the policy with no lock, and did not
+ * wait for a cpuset attach in progress. @p became a deadline task
+ * since: let the caller wait and try again.
+ */
+ if ((ctx->flags & SCA_DL_NOWAIT) && task_has_dl_policy(p)) {
+ ret = -EAGAIN;
+ goto out;
+ }
+
/*
* Check under the same lock that __sched_setscheduler() holds for
* its admission test, so that p cannot become SCHED_DEADLINE
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 61c73c7d3f27..f4cd35811b96 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2975,6 +2975,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_DL_CHECK 0x10
/* The caller made room for the deadline bandwidth in the new root domain. */
#define SCA_DL_BW_RESERVED 0x20
+/* The caller did not wait for a cpuset attach: -EAGAIN for a deadline task. */
+#define SCA_DL_NOWAIT 0x40
extern void update_group_capacity(struct sched_domain *sd, int cpu);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 6562581721ed..ae595a4e7a8f 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -564,10 +564,15 @@ int __sched_setscheduler(struct task_struct *p,
* information. Do not take cpuset_mutex when pi is unset: the
* caller is normalize_rt_tasks(), which holds tasklist_lock and
* cannot sleep.
+ *
+ * Wait for a task attach in progress. cpuset_can_attach() reserved
+ * the bandwidth of the deadline tasks it saw, and cpuset_attach()
+ * moves them. A task that enters or leaves the class in between
+ * would not match that reservation.
*/
if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
- cpuset_lock();
+ cpuset_lock_no_attach();
dl_accounting_lock();
}
@@ -1246,6 +1251,23 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
cpuset_cpus_allowed(p, cpus_allowed);
if (!cpumask_subset(new_mask, cpus_allowed)) {
+ bool locked = false;
+
+ /*
+ * A sched_setattr() may have made @p a deadline task since
+ * the first call. SCA_DL_NOWAIT would then skip the second
+ * call. Wait for a cpuset attach in progress instead, as
+ * sched_setaffinity() does for a deadline task. That attach
+ * can move @p to another cpuset: read its mask again after
+ * the wait.
+ */
+ if (ctx->flags & SCA_DL_NOWAIT) {
+ ctx->flags &= ~SCA_DL_NOWAIT;
+ cpuset_lock_no_attach();
+ locked = true;
+ cpuset_cpus_allowed(p, cpus_allowed);
+ }
+
/*
* We must have raced with a concurrent cpuset update.
* Just reset the cpumask to the cpuset's cpus_allowed.
@@ -1274,6 +1296,8 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
*/
ctx->flags &= ~SCA_DL_CHECK;
__set_cpus_allowed_ptr(p, ctx);
+ if (locked)
+ cpuset_unlock();
retval = -EINVAL;
}
@@ -1289,6 +1313,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
struct affinity_context ac;
struct cpumask *user_mask;
int retval;
+ bool dl;
CLASS(find_get_task, p)(pid);
if (!p)
@@ -1318,13 +1343,31 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
return -ENOMEM;
}
- ac = (struct affinity_context){
- .new_mask = in_mask,
- .user_mask = user_mask,
- .flags = SCA_USER,
- };
-
- retval = __sched_setaffinity(p, &ac);
+ /*
+ * Wait for a cpuset attach in progress before changing the affinity
+ * of a deadline task. cpuset_can_attach() reserved the bandwidth of
+ * the task for the CPUs it moves to, and cpuset_attach() moves it
+ * there. A new affinity in between could change those CPUs.
+ *
+ * The policy is tested here with no lock. A task that becomes a
+ * deadline task after this test is found under its runqueue lock,
+ * with SCA_DL_NOWAIT. Wait and try again then.
+ */
+ for (;;) {
+ dl = task_has_dl_policy(p);
+ ac = (struct affinity_context){
+ .new_mask = in_mask,
+ .user_mask = user_mask,
+ .flags = SCA_USER | (dl ? 0 : SCA_DL_NOWAIT),
+ };
+ if (dl)
+ cpuset_lock_no_attach();
+ retval = __sched_setaffinity(p, &ac);
+ if (dl)
+ cpuset_unlock();
+ if (dl || retval != -EAGAIN)
+ break;
+ }
kfree(ac.user_mask);
return retval;
--
2.47.3
next prev parent reply other threads:[~2026-11-19 20:16 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 02/33] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 03/33] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-19 20:51 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 04/33] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 05/33] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-19 20:28 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 06/33] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 07/33] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 08/33] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-19 20:15 ` Mathieu Desnoyers [this message]
2026-11-19 20:15 ` [PATCH v8 10/33] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 11/33] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 12/33] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 13/33] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 14/33] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-19 20:53 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 15/33] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-19 20:55 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-19 20:22 ` Mathieu Desnoyers
2026-11-19 20:48 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 17/33] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-19 20:43 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 18/33] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:27 ` sashiko-bot
2026-11-19 20:15 ` [PATCH v8 19/33] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 20/33] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 21/33] rtmutex: Continue the walk when the owner's top waiter changes lock Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 22/33] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 23/33] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-19 20:53 ` sashiko-bot
2026-11-19 20:16 ` [PATCH v8 24/33] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 25/33] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 26/33] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-19 20:57 ` sashiko-bot
2026-11-19 20:16 ` [PATCH v8 27/33] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
2026-11-19 20:47 ` sashiko-bot
2026-11-19 20:16 ` [PATCH v8 28/33] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 29/33] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 30/33] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 31/33] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 32/33] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-19 20:59 ` sashiko-bot
2026-11-19 20:16 ` [PATCH v8 33/33] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=771cb2b59894986a695f6b26f1ae45d013327123.1795116790.git.mathieu.desnoyers@efficios.com \
--to=mathieu.desnoyers@efficios.com \
--cc=bigeasy@linutronix.de \
--cc=boqun@kernel.org \
--cc=bsegall@google.com \
--cc=cgroups@vger.kernel.org \
--cc=clrkwllms@kernel.org \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=juri.lelli@redhat.com \
--cc=kayracizmeci@gmail.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=longman@redhat.com \
--cc=luca.abeni@santannapisa.it \
--cc=m.szyprowski@samsung.com \
--cc=mathieu.poirier@linaro.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=parri.andrea@gmail.com \
--cc=peterz@infradead.org \
--cc=ridong.chen@linux.dev \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=stable@vger.kernel.org \
--cc=tglx@kernel.org \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=weihengqiuu@gmail.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®