From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: juri.lelli@redhat.com, peterz@infradead.org, mingo@redhat.com,
vincent.guittot@linaro.org
Cc: linux-kernel@vger.kernel.org,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
tglx@kernel.org, will@kernel.org, boqun@kernel.org,
dietmar.eggemann@arm.com, rostedt@goodmis.org,
bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
kprateek.nayak@amd.com, luca.abeni@santannapisa.it,
mathieu.poirier@linaro.org, longman@redhat.com,
ridong.chen@linux.dev, tj@kernel.org, hannes@cmpxchg.org,
mkoutny@suse.com, shuah@kernel.org, bigeasy@linutronix.de,
clrkwllms@kernel.org, parri.andrea@gmail.com,
m.szyprowski@samsung.com, kayracizmeci@gmail.com,
weihengqiuu@gmail.com, cgroups@vger.kernel.org,
linux-kselftest@vger.kernel.org, linux-rt-devel@lists.linux.dev,
stable@vger.kernel.org
Subject: [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth
Date: Mon, 16 Nov 2026 14:05:38 -0500 [thread overview]
Message-ID: <20261116190549.230119-27-mathieu.desnoyers@efficios.com> (raw)
In-Reply-To: <20261116190549.230119-1-mathieu.desnoyers@efficios.com>
The root domain that admits a deadline task counts its bandwidth. The
mask of the task can hold CPUs of another root domain. With
isolcpus="domain", the isolated CPUs are in the default root domain. A
cpuset that is not a partition root can hold isolated and housekeeping
CPUs.
The code that picks a CPU for a task does not look at the root domain
that counts it:
- select_fallback_rq() and dl_task_offline_migration() pick a CPU when
the CPU of the task goes offline.
- __set_cpus_allowed_ptr_locked() picks one when the mask changes.
- select_task_rq_dl() can leave a task on the CPU it slept on, which
can be in the other root domain.
- dl_task_rebuild_cpu() picks the CPU that a rebuild counts a task
with. It prefers the CPU the task is on, then a housekeeping CPU.
Each can take the task, or its bandwidth, to the other root domain
while its own root domain still has a CPU for it. The bandwidth then
moves with no admission test, and the root domain it arrives in can
exceed its limit.
Add dl_task_bw_span(), which returns the CPUs of the root domain that
counts a task. Make each of these places try one of those CPUs first.
After a rebuild the root domains are new, and the CPU recorded with the
bandwidth can be offline. So dl_task_rebuild_cpu() cannot use the root
domain. It looks at the kind of the CPU recorded, housekeeping or
isolated. It prefers a CPU of the mask of the same kind.
A task with no CPU left in its root domain still moves, and its
bandwidth with it. The next two patches refuse the changes that lead
to that.
The case hotplug_keeps_root_domain of deadline_bw_test, added later in
this series, tests this. A cpuset holds two isolated CPUs and two
housekeeping CPUs, H and K. A deadline task counted in the housekeeping
root domain sleeps on H. Then H goes offline. The task must wake up on
K. Without this patch it woke up on an isolated CPU: CPU 0, in a guest
with isolcpus=domain,0,1.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/core.c | 38 +++++++++++++++++++++++
kernel/sched/deadline.c | 69 +++++++++++++++++++++++++++++++++++++++--
kernel/sched/sched.h | 1 +
3 files changed, 106 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e9f80407f75b..dc270a23179d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3264,6 +3264,23 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
dest_cpu = ctx->dl_bw_cpu;
if (!task_on_rq_queued(p))
p->wake_cpu = dest_cpu;
+ } else {
+ const struct cpumask *span = dl_task_bw_span(p);
+ unsigned int cpu = nr_cpu_ids;
+
+ /*
+ * No room was made for @p in another root domain. Keep @p
+ * in the root domain that counts its bandwidth, if the new
+ * mask holds one of its CPUs.
+ */
+ if (span && !cpumask_test_cpu(dest_cpu, span))
+ cpu = cpumask_first_and_and(cpu_valid_mask,
+ ctx->new_mask, span);
+ if (cpu < nr_cpu_ids) {
+ dest_cpu = cpu;
+ if (!task_on_rq_queued(p))
+ p->wake_cpu = dest_cpu;
+ }
}
do_set_cpus_allowed(p, ctx);
@@ -3668,8 +3685,21 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
int nid = cpu_to_node(cpu);
const struct cpumask *nodemask = NULL;
enum { cpuset, possible, fail } state = cpuset;
+ const struct cpumask *span;
int dest_cpu;
+ /*
+ * Keep a deadline task in the root domain that counts its
+ * bandwidth, if one of its CPUs is allowed and online.
+ */
+ span = dl_task_bw_span(p);
+ if (span) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ return dest_cpu;
+ }
+ }
+
/*
* If the node that the CPU is on has been offlined, cpu_to_node()
* will return -1. There is no CPU on the node, and we should
@@ -3686,6 +3716,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
}
for (;;) {
+ /* The mask is wider now: look again in that root domain. */
+ if (span && state != cpuset) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ goto out;
+ }
+ }
+
/* Any allowed, online CPU? */
for_each_cpu(dest_cpu, p->cpus_ptr) {
if (!is_cpu_allowed(p, dest_cpu))
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 35f40185953a..5294546786f1 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -864,13 +864,19 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
later_rq = find_lock_later_rq(p, rq);
if (!later_rq) {
+ const struct cpumask *span = dl_task_bw_span(p);
int cpu;
/*
* If we cannot preempt any rq, fall back to pick any
- * online CPU:
+ * online CPU. Try the root domain that counts @p first.
*/
- cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
+ cpu = nr_cpu_ids;
+ if (span)
+ cpu = cpumask_first_and_and(cpu_active_mask,
+ p->cpus_ptr, span);
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
if (cpu >= nr_cpu_ids) {
/*
* Failed to find any suitable CPU.
@@ -2865,6 +2871,7 @@ static int
select_task_rq_dl(struct task_struct *p, int cpu, int flags)
{
struct task_struct *curr, *donor;
+ const struct cpumask *span;
bool select_rq;
struct rq *rq;
@@ -2905,6 +2912,20 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
dl_task_is_earliest_deadline(p, cpu_rq(target)))
cpu = target;
}
+
+ /*
+ * Keep @p in the root domain that counts its bandwidth. Its mask
+ * can hold CPUs of another root domain, with isolcpus="domain",
+ * and @p can sleep on one of them.
+ */
+ span = dl_task_bw_span(p);
+ if (span && !cpumask_test_cpu(cpu, span)) {
+ unsigned int in_span;
+
+ in_span = cpumask_first_and_and(span, p->cpus_ptr, cpu_active_mask);
+ if (in_span < nr_cpu_ids)
+ cpu = in_span;
+ }
rcu_read_unlock();
return cpu;
@@ -3669,6 +3690,29 @@ static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
return cpu;
}
+/*
+ * Return the CPUs of the root domain that counts the bandwidth of @p, or
+ * NULL if none does. The mask of @p can hold CPUs of another root domain,
+ * with isolcpus="domain". Where a CPU is picked for @p, a CPU of this
+ * root domain comes first: no other root domain admitted @p.
+ *
+ * The caller holds the pi_lock or the rq lock of @p. With interrupts
+ * off, the root domain is not freed.
+ */
+const struct cpumask *dl_task_bw_span(struct task_struct *p)
+{
+ int cpu;
+
+ if (!task_has_dl_policy(p) || dl_entity_is_special(&p->dl) ||
+ !p->dl.dl_bw)
+ return NULL;
+ cpu = READ_ONCE(p->dl.dl_bw_cpu);
+ if (cpu < 0 || !cpu_active(cpu) ||
+ READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen))
+ return NULL;
+ return cpu_rq(cpu)->rd->span;
+}
+
static void set_cpus_allowed_dl(struct task_struct *p,
struct affinity_context *ctx)
{
@@ -3903,6 +3947,27 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
*/
cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
cpu = task_cpu(p);
+ /*
+ * Keep the bandwidth where it was counted: with the housekeeping
+ * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
+ * domains are new, and the CPU recorded can be offline. So prefer a
+ * CPU of the mask of the same kind as the CPU recorded: housekeeping
+ * or not.
+ */
+ if (recorded >= 0) {
+ const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+ bool rec_hk = cpumask_test_cpu(recorded, hk);
+ unsigned int same;
+
+ if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
+ return cpu;
+ if (cpumask_test_cpu(recorded, msk))
+ return recorded;
+ for_each_cpu(same, msk) {
+ if (cpumask_test_cpu(same, hk) == rec_hk)
+ return same;
+ }
+ }
if (cpumask_test_cpu(cpu, msk))
return cpu;
if (cpumask_empty(msk)) {
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3cc287785b91..f80ea5541b3f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -371,6 +371,7 @@ extern bool __checkparam_dl(const struct sched_attr *attr);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
extern int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
extern int dl_bw_deactivate(int cpu);
+extern const struct cpumask *dl_task_bw_span(struct task_struct *p);
extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec);
/*
* SCHED_DEADLINE supports servers (nested scheduling) with the following
--
2.47.3
next prev parent reply other threads:[~2026-11-16 19:06 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 19:05 [PATCH v6 00/37] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 01/37] sched: Refuse SCHED_FLAG_KEEP_PARAMS that would change the policy or the class Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 02/37] sched/deadline: Do not admit bandwidth for parameters that are kept Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 03/37] sched: Let a task boosted by a deadline waiter change its policy Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 04/37] sched/deadline: Do not divide by zero active CPUs of a root domain Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 05/37] sched/deadline: Refuse dl_server debugfs writes while the rq is offline Mathieu Desnoyers
2026-11-16 19:30 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 06/37] sched: Take cpuset_mutex for a deadline policy change only with pi Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 07/37] sched/deadline: Do not lose a throttled task switched out and back Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 08/37] sched/deadline: Do not replenish a boost from cleared parameters Mathieu Desnoyers
2026-11-16 19:42 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 09/37] sched/deadline: Reset the deadline and the runtime with the parameters Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 10/37] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-16 19:38 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 11/37] sched/deadline: Release the bandwidth of a task that leaves the class asleep Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 12/37] sched/deadline: Reject SCHED_DEADLINE on exited tasks Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 13/37] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 14/37] sched/deadline: Fix this_bw when a task is changed during a remote wakeup Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 15/37] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 16/37] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-16 19:44 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 17/37] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 18/37] sched/deadline: Check SCHED_DEADLINE affinity under the rq lock Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 19/37] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 20/37] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 21/37] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-16 19:31 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 22/37] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 23/37] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-16 19:56 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 24/37] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-16 20:01 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 25/37] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-16 19:47 ` sashiko-bot
2026-11-16 19:05 ` Mathieu Desnoyers [this message]
2026-11-16 19:50 ` [PATCH v6 26/37] sched/deadline: Keep a task in the root domain that counts its bandwidth sashiko-bot
2026-11-16 19:05 ` [PATCH v6 27/37] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-16 19:51 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 28/37] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-16 20:08 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 29/37] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 30/37] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 31/37] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-16 20:12 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 32/37] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-16 20:08 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 33/37] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-16 20:04 ` sashiko-bot
2026-11-16 19:05 ` [PATCH v6 34/37] sched/debug: Show the deadline bandwidth of each runqueue Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 35/37] selftests/sched: Add a test of SCHED_DEADLINE policy round trips Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 36/37] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-16 19:05 ` [PATCH v6 37/37] selftests/sched: Add a test of SCHED_DEADLINE bandwidth accounting Mathieu Desnoyers
2026-11-16 19:49 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116190549.230119-27-mathieu.desnoyers@efficios.com \
--to=mathieu.desnoyers@efficios.com \
--cc=bigeasy@linutronix.de \
--cc=boqun@kernel.org \
--cc=bsegall@google.com \
--cc=cgroups@vger.kernel.org \
--cc=clrkwllms@kernel.org \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=juri.lelli@redhat.com \
--cc=kayracizmeci@gmail.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=longman@redhat.com \
--cc=luca.abeni@santannapisa.it \
--cc=m.szyprowski@samsung.com \
--cc=mathieu.poirier@linaro.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=parri.andrea@gmail.com \
--cc=peterz@infradead.org \
--cc=ridong.chen@linux.dev \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=stable@vger.kernel.org \
--cc=tglx@kernel.org \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=weihengqiuu@gmail.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®