From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4822DC30654 for ; Thu, 17 Aug 2023 15:35:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352314AbjHQPeh (ORCPT ); Thu, 17 Aug 2023 11:34:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352305AbjHQPeS (ORCPT ); Thu, 17 Aug 2023 11:34:18 -0400 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61B1730C6 for ; Thu, 17 Aug 2023 08:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1692286452; bh=LYfwgFU7s5/MJa/0fx3LIMlynA7Gq4pgq2ulwVMJX50=; h=From:To:Cc:Subject:Date:From; b=KSFEhy5rg5rmKjYX+AYz6MizEmYjryQpwGZJMUyNJSuMLRgkXWBn5Gm9btiP4gle9 LnRUi+U/Cdutmhhy2BZ6oZG7rP0XwzLH6r6zbHn4GS9h0pmTod7XfdcFgLXE10d5G5 pQjJ8fJ9Q8QK0Sld74PJXFbAltUj6bO+FaDUjXkLzYyQrSnXpyIg6TUK4Afv4LF9J4 6vlJihIzn3wYwAmnakMUdcB4CKtm74Vn3Ui9BpnloylkVpyqlXAjyOgpecTMoDfdcE CjJfP5K6m+TYFo9Eyq+yO92dMs50nr6C1qjvfbq4J5mb952X1OKEOfJ36uUXzpeAl0 B6WdDBW1JhD3Q== Received: from thinkos.internal.efficios.com (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4RRTZD3GpJz1Ls2; Thu, 17 Aug 2023 11:34:12 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Ingo Molnar , Valentin Schneider , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Vincent Guittot , Juri Lelli , Swapnil Sapkal , Aaron Lu , x86@kernel.org Subject: [RFC PATCH 1/1] sched: ttwu_queue_cond: perform queued wakeups across different L2 caches Date: Thu, 17 Aug 2023 11:35:15 -0400 Message-Id: <20230817153515.143932-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Skipping queued wakeups for all logical CPUs sharing an LLC means that on a 192 cores AMD EPYC 9654 96-Core Processor (over 2 sockets), groups of 8 cores (16 hardware threads) end up grabbing runqueue locks of other runqueues within the same group for each wakeup, causing contention on the runqueue locks. Improve this by only considering hardware threads sharing an L2 cache as candidates for skipping use of the queued wakeups. This results in the following benchmark improvements: hackbench -g 32 -f 20 --threads --pipe -l 480000 -s 100 from 49s to 34s. (30% speedup) And similarly with perf bench: perf bench sched messaging -g 32 -p -t -l 100000 from 10.9s to 7.4s (32% speedup) This was developed as part of the investigation into a weird regression reported by AMD where adding a raw spinlock in the scheduler context switch accelerated hackbench. It turned out that changing this raw spinlock for a loop of 10000x cpu_relax within do_idle() had similar benefits. This patch achieves a similar effect without busy waiting nor changing anything about runqueue selection on wakeup. It considers that only hardware threads sharing an L2 cache should skip the queued try-to-wakeup and directly grab the target runqueue lock, rather than allowing all hardware threads sharing an LLC to do so. I would be interested to hear feedback about performance impact of this patch (improvement or regression) on other workloads and hardware, especially for Intel CPUs. One thing that we might want to empirically figure out from the topology is whether there is a maximum number of hardware threads within an LLC below which it would make sense to use the LLC rather than L2 as group within which queued wakeups can be skipped. [ Only tested on AMD CPUs so far. ] Link: https://lore.kernel.org/r/09e0f469-a3f7-62ef-75a1-e64cec2dcfc5@amd.com Link: https://lore.kernel.org/lkml/20230725193048.124796-1-mathieu.desnoyers@efficios.com/ Link: https://lore.kernel.org/lkml/20230810140635.75296-1-mathieu.desnoyers@efficios.com/ Signed-off-by: Mathieu Desnoyers Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Vincent Guittot Cc: Juri Lelli Cc: Swapnil Sapkal Cc: Aaron Lu Cc: x86@kernel.org --- arch/Kconfig | 6 ++++++ arch/x86/Kconfig | 1 + drivers/base/Kconfig | 1 + include/linux/sched/topology.h | 3 ++- kernel/sched/core.c | 26 +++++++++++++++++++++++--- 5 files changed, 33 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cad..e5aac1741712 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -340,6 +340,12 @@ config HAVE_ASM_MODVERSIONS to support the module versioning for symbols exported from assembly code. +config HAVE_CLUSTERGROUP + bool + help + This symbol should be selected by an architecture if it + implements CPU clustergroup. + config HAVE_REGS_AND_STACK_ACCESS_API bool help diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index cb1031018afa..07813a1a9a58 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -299,6 +299,7 @@ config X86 select FUNCTION_ALIGNMENT_4B imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE + select HAVE_CLUSTERGROUP config INSTRUCTION_DECODER def_bool y diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig index 2b8fd6bb7da0..408aaf7a4bd1 100644 --- a/drivers/base/Kconfig +++ b/drivers/base/Kconfig @@ -218,6 +218,7 @@ config DMA_FENCE_TRACE config GENERIC_ARCH_TOPOLOGY bool + select HAVE_CLUSTERGROUP help Enable support for architectures common topology code: e.g., parsing CPU capacity information from DT, usage of such information for diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 816df6cc444e..714386070463 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -178,7 +178,8 @@ extern void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[], cpumask_var_t *alloc_sched_domains(unsigned int ndoms); void free_sched_domains(cpumask_var_t doms[], unsigned int ndoms); -bool cpus_share_cache(int this_cpu, int that_cpu); +bool cpus_share_cluster(int this_cpu, int that_cpu); /* Share L2. */ +bool cpus_share_cache(int this_cpu, int that_cpu); /* Share LLC. */ typedef const struct cpumask *(*sched_domain_mask_f)(int cpu); typedef int (*sched_domain_flags_f)(void); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a68d1276bab0..ce3402b81e5e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3904,13 +3904,33 @@ void wake_up_if_idle(int cpu) rcu_read_unlock(); } +/* + * Query whether CPUs share LLC. + */ bool cpus_share_cache(int this_cpu, int that_cpu) +{ + return per_cpu(sd_llc_id, this_cpu) == per_cpu(sd_llc_id, that_cpu); +} + +#ifdef CONFIG_HAVE_CLUSTERGROUP +/* + * Query whether CPUs share L2 cache. + */ +bool cpus_share_cluster(int this_cpu, int that_cpu) { if (this_cpu == that_cpu) return true; - - return per_cpu(sd_llc_id, this_cpu) == per_cpu(sd_llc_id, that_cpu); + return cpumask_test_cpu(that_cpu, cpu_clustergroup_mask(this_cpu)); +} +#else +/* + * Fall-back on querying whether CPUs share LLC. + */ +bool cpus_share_cluster(int this_cpu, int that_cpu) +{ + return cpus_share_cache(this_cpu, that_cpu); } +#endif static inline bool ttwu_queue_cond(struct task_struct *p, int cpu) { @@ -3929,7 +3949,7 @@ static inline bool ttwu_queue_cond(struct task_struct *p, int cpu) * If the CPU does not share cache, then queue the task on the * remote rqs wakelist to avoid accessing remote data. */ - if (!cpus_share_cache(smp_processor_id(), cpu)) + if (!cpus_share_cluster(smp_processor_id(), cpu)) return true; if (cpu == smp_processor_id()) -- 2.39.2