From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Anna-Maria Behnsen <anna-maria@linutronix.de>,
Frederic Weisbecker <frederic@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>
Cc: <linux-kernel@vger.kernel.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>,
"Gautham R. Shenoy" <gautham.shenoy@amd.com>,
Swapnil Sapkal <swapnil.sapkal@amd.com>,
Shrikanth Hegde <sshegde@linux.ibm.com>,
Chen Yu <yu.c.chen@intel.com>
Subject: [RESEND RFC PATCH v2 13/29] sched/fair: Account idle cpus instead of busy cpus in sd->shared
Date: Mon, 8 Dec 2025 09:26:59 +0000 [thread overview]
Message-ID: <20251208092744.32737-13-kprateek.nayak@amd.com> (raw)
In-Reply-To: <20251208083602.31898-1-kprateek.nayak@amd.com>
Switch to keeping track of "sd->shared->nr_idle_cpus" instead of
"nr_busy_cpus". Since previous commit corrected the "sd->nohz_idle"
state during sched domain rebuild, the nr_idle_cpus will reflect the
correct number of idle CPUs.
The idle CPUs accounting will be used for nohz idle balance in the
subsequent commits.
Races are possible during hotplug / cpuset where "nr_idle_cpus" might
be incorrectly accounted if the CPU enters exits out of nohz idle state
between the read of "rq->nohz_tick_stopped" and the subsequent update of
"sd->nohz_idle" in the hotplug path but these inaccuracies are transient
and will be corrected when the CPU enters idle or receives a tick.
CPU0 (hotplug) CPU1 (exits nohz idle)
============== ======================
online()
if (rq->nohz_tick_stopped) /* True */
... rq->nohz_tick_stopped = 0
... set_cpu_sd_state_busy()
...
set_cpu_sd_state_idle()
These situations are rare and should not have any long-term effect on
the nohz idle balancing since there isn't a case where a nohz idle CPU
is not set on the mask - either the hotplug thread sees that
"rq->nohz_tick_stopped" is set or the CPU going idle sees the updated
sched_domain hierarchy.
After the conversion, all the bits that use "nr_idle_cpus" are already
guarded behind CONFIG_NO_HZ_COMMON which makes it convenient to put the
declaration behind CONFIG_NO_HZ_COMMON as well.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
include/linux/sched/topology.h | 4 +++-
kernel/sched/fair.c | 10 +++++-----
kernel/sched/topology.c | 1 -
3 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h
index fc3d89160513..15c61aed1b5c 100644
--- a/include/linux/sched/topology.h
+++ b/include/linux/sched/topology.h
@@ -65,7 +65,9 @@ struct sched_group;
struct sched_domain_shared {
atomic_t ref;
- atomic_t nr_busy_cpus;
+#ifdef CONFIG_NO_HZ_COMMON
+ atomic_t nr_idle_cpus;
+#endif
int has_idle_cores;
int nr_idle_scan;
};
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index de9e81eeb93d..fef3826a258f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12533,7 +12533,7 @@ static void nohz_balancer_kick(struct rq *rq)
* the others are - so just get a NOHZ balance going if it looks
* like this LLC domain has tasks we could move.
*/
- nr_busy = atomic_read(&sds->nr_busy_cpus);
+ nr_busy = per_cpu(sd_llc_size, cpu) - atomic_read(&sds->nr_idle_cpus);
if (nr_busy > 1) {
flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK;
goto unlock;
@@ -12562,7 +12562,7 @@ static void set_cpu_sd_state_busy(int cpu)
if (!xchg(&sd->nohz_idle, 0))
return;
- atomic_inc(&sd->shared->nr_busy_cpus);
+ atomic_dec(&sd->shared->nr_idle_cpus);
}
void nohz_balance_exit_idle(struct rq *rq)
@@ -12592,7 +12592,7 @@ static void set_cpu_sd_state_idle(int cpu)
if (xchg(&sd->nohz_idle, 1))
return;
- atomic_dec(&sd->shared->nr_busy_cpus);
+ atomic_inc(&sd->shared->nr_idle_cpus);
}
static void cpu_sd_exit_nohz_balance(struct rq *rq)
@@ -13075,7 +13075,7 @@ static void rq_online_fair(struct rq *rq)
update_runtime_enabled(rq);
- /* Fixup nr_busy_cpus and nohz stats. */
+ /* Fixup nr_idle_cpus and nohz stats. */
cpu_sd_reenter_nohz_balance(rq);
}
@@ -13089,7 +13089,7 @@ static void rq_offline_fair(struct rq *rq)
/* Ensure that we remove rq contribution to group share: */
clear_tg_offline_cfs_rqs(rq);
- /* Fixup nr_busy_cpus and nohz stats. */
+ /* Fixup nr_idle_cpus and nohz stats. */
cpu_sd_exit_nohz_balance(rq);
}
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index a212ae52cdac..6b14c7db3e35 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2656,7 +2656,6 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
int llc_id = cpumask_first(sched_domain_span(sd));
sd->shared = *per_cpu_ptr(d.sds, llc_id);
- atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight);
atomic_inc(&sd->shared->ref);
}
--
2.43.0
next prev parent reply other threads:[~2025-12-08 9:31 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-08 9:26 [RESEND RFC PATCH v2 00/29] sched/fair: Push-based load balancing K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 01/29] sched/fair: Simplify set_cpu_sd_state_*() with guards K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 02/29] sched/fair: Use rq->nohz_tick_stopped in update_nohz_stats() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 03/29] sched/topology: Optimize sd->shared allocation and assignment K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 04/29] sched/fair: Simplify the entry condition for update_idle_cpu_scan() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 05/29] sched/fair: Simplity SIS_UTIL handling in select_idle_cpu() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 06/29] cpumask: Introduce for_each_cpu_and_wrap() and bitfield helpers K Prateek Nayak
2025-12-12 21:03 ` Yury Norov
2025-12-08 9:26 ` [RESEND RFC PATCH v2 07/29] sched/fair: Use for_each_cpu_and_wrap() in select_idle_capacity() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 08/29] sched/fair: Use for_each_cpu_and_wrap() in select_idle_cpu() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 09/29] sched/fair: Rotate the CPU resposible for busy load balancing K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 10/29] sched/fair: Use xchg() to set sd->nohz_idle state K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 11/29] sched/topology: Attach new hierarchy in rq_attach_root() K Prateek Nayak
2025-12-08 9:26 ` [RESEND RFC PATCH v2 12/29] sched/fair: Fixup sd->nohz_idle state during hotplug / cpuset K Prateek Nayak
2025-12-08 9:26 ` K Prateek Nayak [this message]
2025-12-08 9:27 ` [RESEND RFC PATCH v2 14/29] sched/topology: Introduce fallback sd->shared assignment K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 15/29] sched/topology: Introduce percpu sd_nohz for nohz state tracking K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 16/29] sched/topology: Introduce "nohz_idle_cpus_mask" in sd->shared K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 17/29] sched/topology: Introduce "nohz_shared_list" to keep track of sd->shared K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 18/29] sched/fair: Reorder the barrier in nohz_balance_enter_idle() K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 19/29] sched/fair: Extract the main _nohz_idle_balance() loop into a helper K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 20/29] sched/fair: Convert find_new_ilb() to use nohz_shared_list K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 21/29] sched/fair: Introduce sched_asym_prefer_idle() for ILB kick K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 22/29] sched/fair: Convert sched_balance_nohz_idle() to use nohz_shared_list K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 23/29] sched/fair: Remove "nohz.idle_cpus_mask" K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 24/29] sched/fair: Optimize global "nohz.nr_cpus" tracking K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 25/29] sched/topology: Add basic debug information for "nohz_shared_list" K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 26/29] [EXPERIMENTAL] sched/fair: Add push task framework K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 27/29] [EXPERIMENTAL] sched/fair: Proactive idle balance using push mechanism K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 28/29] [EXPERIMENTAL] sched/fair: Add a local counter to rate limit task push K Prateek Nayak
2025-12-08 12:33 ` Christian Loehle
2025-12-08 17:35 ` K Prateek Nayak
2025-12-08 9:27 ` [RESEND RFC PATCH v2 29/29] [EXPERIMENTAL] sched/fair: Faster alternate for intra-NUMA newidle balance K Prateek Nayak
2025-12-08 14:04 ` [RESEND RFC PATCH v2 00/29] sched/fair: Push-based load balancing Shrikanth Hegde
2025-12-08 17:36 ` K Prateek Nayak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251208092744.32737-13-kprateek.nayak@amd.com \
--to=kprateek.nayak@amd.com \
--cc=anna-maria@linutronix.de \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=frederic@kernel.org \
--cc=gautham.shenoy@amd.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sshegde@linux.ibm.com \
--cc=swapnil.sapkal@amd.com \
--cc=tglx@linutronix.de \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®