From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A509C73C66 for ; Fri, 18 Aug 2023 15:30:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378190AbjHRP3y (ORCPT ); Fri, 18 Aug 2023 11:29:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378172AbjHRP32 (ORCPT ); Fri, 18 Aug 2023 11:29:28 -0400 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E07E2D65 for ; Fri, 18 Aug 2023 08:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1692372566; bh=AoHcTQEwU/NnvuXXlKDh/AgBCpzjE7XyPmh00eg/rfM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IZ+cn4ym1gZoMQN0AEQxq6XZ6mVfAh8fGg/yhIymJVuYuTOr9ImR7EoDHAUipzuCb 6w7y8hFf1qaOFllZ3An8n/7PoNcvZGkf0DcDqiJdg5HqUUfTlary4Lmy1/ttgMJe8g PcDPU/jwZINhNWkh5ankPIMmLsXxf9azvOQ9JnB3a51/+pR6n+RqaQ5z1+8lwsqzxP YUD8mzJhlzZsJddJbz4egupJvLqcCJAFrujwQ40hoBl4RIwVj8ltrJdupCWrStEyzG ggHKWauZ1l+CdihzXQuWbmL+MORZndr8m7gbK2L2UhM0K8Y2LPpdvD88bhU9qBsgWC PCtqe6IaBAWDg== Received: from thinkos.internal.efficios.com (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4RS5QG2g1rz1LYD; Fri, 18 Aug 2023 11:29:26 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Ingo Molnar , Valentin Schneider , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Vincent Guittot , Juri Lelli , Swapnil Sapkal , Aaron Lu , Julien Desfossez , x86@kernel.org Subject: [RFC PATCH 2/3] sched: Introduce cpus_share_l2c Date: Fri, 18 Aug 2023 11:30:26 -0400 Message-Id: <20230818153027.202017-2-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230818153027.202017-1-mathieu.desnoyers@efficios.com> References: <20230818153027.202017-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Introduce cpus_share_l2c to allow querying whether two logical CPUs share a common L2 cache. Considering a system like the AMD EPYC 9654 96-Core Processor, the L1 cache has a latency of 4-5 cycles, the L2 cache has a latency of at least 14ns, whereas the L3 cache has a latency of 50ns [1]. Compared to this, I measured the RAM accesses to a latency around 120ns on my system [2]. So L3 really is only 2.4x faster than RAM accesses. Therefore, with this relatively slow access speed compared to L2, the scheduler will benefit from only considering CPUs sharing an L2 cache for the purpose of using remote runqueue locking rather than queued wakeups. Link: https://en.wikichip.org/wiki/amd/microarchitectures/zen_4 [1] Link: https://github.com/ChipsandCheese/MemoryLatencyTest [2] Signed-off-by: Mathieu Desnoyers Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Vincent Guittot Cc: Juri Lelli Cc: Swapnil Sapkal Cc: Aaron Lu Cc: Julien Desfossez Cc: x86@kernel.org --- include/linux/sched/topology.h | 6 ++++++ kernel/sched/core.c | 8 ++++++++ kernel/sched/sched.h | 2 ++ kernel/sched/topology.c | 20 +++++++++++++++++--- 4 files changed, 33 insertions(+), 3 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7f9331f71260..c5fdee188bea 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -178,6 +178,7 @@ extern void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[], cpumask_var_t *alloc_sched_domains(unsigned int ndoms); void free_sched_domains(cpumask_var_t doms[], unsigned int ndoms); +bool cpus_share_l2c(int this_cpu, int that_cpu); bool cpus_share_llc(int this_cpu, int that_cpu); typedef const struct cpumask *(*sched_domain_mask_f)(int cpu); @@ -227,6 +228,11 @@ partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[], { } +static inline bool cpus_share_l2c(int this_cpu, int that_cpu) +{ + return true; +} + static inline bool cpus_share_llc(int this_cpu, int that_cpu) { return true; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d096ce815099..11e60a69ae31 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3904,6 +3904,14 @@ void wake_up_if_idle(int cpu) rcu_read_unlock(); } +bool cpus_share_l2c(int this_cpu, int that_cpu) +{ + if (this_cpu == that_cpu) + return true; + + return per_cpu(sd_l2c_id, this_cpu) == per_cpu(sd_l2c_id, that_cpu); +} + bool cpus_share_llc(int this_cpu, int that_cpu) { if (this_cpu == that_cpu) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 81ac605b9cd5..d93543db214c 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1828,6 +1828,8 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag) return sd; } +DECLARE_PER_CPU(int, sd_l2c_size); +DECLARE_PER_CPU(int, sd_l2c_id); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc); DECLARE_PER_CPU(int, sd_llc_size); DECLARE_PER_CPU(int, sd_llc_id); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 1ae2a0a1115a..41a41f730011 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -661,8 +661,11 @@ static void destroy_sched_domains(struct sched_domain *sd) * * Also keep a unique ID per domain (we use the first CPU number in * the cpumask of the domain), this allows us to quickly tell if - * two CPUs are in the same cache domain, see cpus_share_llc(). + * two CPUs are in the same cache domain, see cpus_share_l2c() and + * cpus_share_llc(). */ +DEFINE_PER_CPU(int, sd_l2c_size); +DEFINE_PER_CPU(int, sd_l2c_id); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); DEFINE_PER_CPU(int, sd_llc_id); @@ -674,10 +677,10 @@ DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity); static void update_top_cache_domain(int cpu) { + const struct cpumask *cluster_mask; struct sched_domain_shared *sds = NULL; struct sched_domain *sd; - int id = cpu; - int size = 1; + int id = cpu, size = 1, l2c_id, l2c_size; sd = highest_flag_domain(cpu, SD_SHARE_PKG_RESOURCES); if (sd) { @@ -686,6 +689,17 @@ static void update_top_cache_domain(int cpu) sds = sd->shared; } + cluster_mask = topology_cluster_cpumask(cpu); + l2c_size = cpumask_weight(cluster_mask); + if (l2c_size == 1) { + /* Fallback on using LLC. */ + l2c_size = size; + l2c_id = id; + } + l2c_id = cpumask_first(cluster_mask); + per_cpu(sd_l2c_id, cpu) = l2c_id; + per_cpu(sd_l2c_size, cpu) = l2c_size; + rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; -- 2.39.2