From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010049.outbound.protection.outlook.com [52.101.61.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89E2A1391 for ; Mon, 2 Mar 2026 04:59:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.49 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772427574; cv=fail; b=avqkjxpaGmPMZzG/2j6H/ALHBlS4PJvaXZXEOnj/WBO1nMVXaDxLsEdGqgau3Y7NmbJtTI5LmIluHoYWRXmIkINvX+5A9t5a5ABPFhADOxPxPmOKtTgSE2bc9wxmjt5ZNob8HWpjjQulWCPCmHD4ychxyY8Nwt2AOjkDAKa3vG4= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772427574; c=relaxed/simple; bh=O6RkLtfuPMz6iDIhpaZ6RdhS6U+xzAta/DADf0UxuEw=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=VDvFi4hjsPPvYn7zP0S6iuOJA2qM3ostxjZlv/IBBlzIwkFq+3La92GAM50SN7GKcFrNrbsClIIfSVGTOReqW0kQiTQe/Z5mVc/QUPXpefVzm0p+uix0qquXhaiRLi7cOngZloGSCKBjksHYPJoaegmLcUx/SLwqZKw6j4jeFtQ= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Fkdjqp+s; arc=fail smtp.client-ip=52.101.61.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Fkdjqp+s" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jf5c7Ut84AMrTVNS48fe6b6SNXjh626QdSlCzVSu9zoCqTzeROS5xQy2pksFHktkPGEVLj+AeZsooDbJvUjEF9l3aVwSg3s74oYvVFkWex595jd4zARs+QH8J0xB1PDeHD45fzQ9k8XS/k0BrgZDaP6BYQymQDyRbu/GZ+ekGIE4IF5PeIYEb/jw9DW09wdN6f73+d8BVqkO0Mr2xS9z9tfPmdi0k8tjxKy0f1HPUjMhLdfG2LXRlVHaqZygoFDe7BLfiEGgi/kmbHDa+SrS/cSC/qt8B4tADCv6kvGoE2LtzmZg9nf0wYwJ22l3/z4DzH1ndmFUq3ZP/lJVII5bgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D/Z5Fnl1AhgZynTGQaQ4CM+8v3S3QR5rdbxxi3Uxnk4=; b=mV7azDmCOMGFuI4aKAaQa9uXJpNPdP39yeideoPqY2EHJXbiBVntVIkPV2OfE0jvPLxQ7ULq1ZRySG9E2qmGarwkqMN3NW5sLivPTeLQ6Cgi1TuaC91m+WJhPs1tMHZObSnWkaJBnd0KNozJYUNmg+jwbbej4PYl/BfJJJFVwwm5nkHXl5sreThi6AO5WDStrK82NjhfrlXaPoGKcdz8+UTUl7BeamxWStic5Kpd5K9VsPnxmUypw+nwDa2m21y2+OpheWlle/2Q+MI981k3zZKJdlQoRNdZFV+xZ69qTaHBaV0sF/3stWS3Q/7d3HUWQKgggHQVqygIf0nEB1mT8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/Z5Fnl1AhgZynTGQaQ4CM+8v3S3QR5rdbxxi3Uxnk4=; b=Fkdjqp+sQZFrZkgfMV0hm7z6gqrv7n4C73AHbl4J7P6AErFZxPmkTicDYEq2K8d+1jAEB+Hnn4G80PjAV06TzhYL5zExj3JRMpxpZ8N5u8+tfVrDfLYf1YKidL0tf6oaphjqmti3LnPdHZ2+i6icgNl5+ZT8l7Y5tf98QQsJtns= Received: from SJ0PR03CA0299.namprd03.prod.outlook.com (2603:10b6:a03:39e::34) by LV3PR12MB9401.namprd12.prod.outlook.com (2603:10b6:408:21c::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.18; Mon, 2 Mar 2026 04:59:30 +0000 Received: from SJ1PEPF000023D8.namprd21.prod.outlook.com (2603:10b6:a03:39e:cafe::49) by SJ0PR03CA0299.outlook.office365.com (2603:10b6:a03:39e::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9654.18 via Frontend Transport; Mon, 2 Mar 2026 04:59:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by SJ1PEPF000023D8.mail.protection.outlook.com (10.167.244.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9700.0 via Frontend Transport; Mon, 2 Mar 2026 04:59:29 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Sun, 1 Mar 2026 22:59:27 -0600 Received: from satlexmb08.amd.com (10.181.42.217) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Mar 2026 22:59:26 -0600 Received: from [10.136.33.80] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Sun, 1 Mar 2026 22:59:24 -0600 Message-ID: Date: Mon, 2 Mar 2026 10:29:23 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] futex: Dynamically allocate futex_queues depending on nr_node_ids To: Peter Zijlstra , Sebastian Andrzej Siewior CC: Thomas Gleixner , Ingo Molnar , , Darren Hart , "Davidlohr Bueso" , =?UTF-8?Q?Andr=C3=A9_Almeida?= References: <20260128101358.20954-1-kprateek.nayak@amd.com> <20260227144203.GJ1282955@noisy.programming.kicks-ass.net> <20260227161841.GH606826@noisy.programming.kicks-ass.net> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260227161841.GH606826@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF000023D8:EE_|LV3PR12MB9401:EE_ X-MS-Office365-Filtering-Correlation-Id: 2c0103cc-e52f-4350-2913-08de78187c7c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|376014; X-Microsoft-Antispam-Message-Info: E2Y1YGW4hKFULh8kpy57pdJCrGNjeBTxluo9OuM/Vvp0ZVclUutE4/KNs+tMQrBNNsxvdkHMRsZD02hDB0EjfNeHiJQj0AmKlH6tIuKsbiC9h+8G69vt78VD1MPTOnoCV/ABtZZobLMdtbRLAkWyX7cit0dOiyP0vIyxNRXUX8BDqekAdwFIoTRz1LU+V/4vlwPaJB7qWd74Gqs6SP5ytjBYbyFRoYBQymaRA1EIO2Qy3wiebiBDdxDMhVIIkKs2NgSoCAr9yYB86fErLE1zdSh5CBIYRUZuqbqlT3XwU1BE1TsRDaiVdzCp6LqA8cAzjuguA3yUfqCu4cGPZ8+lSJmcELmup4k+WoWn2EITiEQe86qkAqKn+c0oHt17yVfnbBF4xsIBobYlhJ6HBZ3XLk8JGTLnMYlVICNiwunnhwX4yEeVy0P0mHCxbcGDwNrV+NG9PVfYoDwweB7OL8EfkrRtoncZm48daVTzi48fFeUdAxeYbf6MYADyfJtZq+tnXZ8J+0WrxfaKRT06RvbVKqQsfixNp5dUWw+n0GO4Wmw5gVwBProybZ7EKLAzB1h6tur1snI8F3NJU/lSLnCARJ2TFfZl9Hl5XUGV3fBkjJmV9JA3IjfANQWObYQgmrTDrDxc/rCgAPuW5pgZsHdW5ax9krXiUx7U8qkOg2+iBU0SgEv+ucTCPWtC+JrRjF1ySZPRFvGVOmPzjZhdmbwdP505MnlWsvjv6Zt9Gnq28Xse/PowS1nW4Qkbwv8oGS8+6J0hsEPVkyzjN3Am8zoUjoLvoZ9+G5n597flMoLvueDartTnFJUUOS20OLxks2kKhrNEjSp5m4pRSQDU5F6Gyg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KzN1dz1dlH1oaJVVnxcIhW/qLHar0xLYIs5fv2hnPNLEDkjTet01kImmj7tTXqz+qS30UNuhucIE4pxjhDmUJMHGI8qm0OJfaqol9x8Oz2qEEoiYUV3G8RyIbbyTUr7IOQ9g3gr89JI8bdPzTm/JCcS85wG64IQtKtGN+g2F492NqLbT7gJ8T4J3pKum8H3QbReJd1HvfifECq1J5FXArfjU7cQMyW6ISZ87oi04CNbm5LXXn47nn4OuZj62P2AqZ9zzF6JA1Lmoo9EN/BM1ByPIfF7vLZvdnak7H9JrJ4k7CsisWUU5HwPCCafERi8E4xiWY/3fMYloqf/xs9i+0K7sIqg9KvaBvnpUKlYUHsPjPEZnQCApTeOazdhVK3hC//eq8TYoHcz1p08LyKsnEmjxllUcmbMRYNY8X28Be9hSjE1YzLeffq09kVg3slUD X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2026 04:59:29.4898 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2c0103cc-e52f-4350-2913-08de78187c7c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF000023D8.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9401 Hello Peter, Sebastian, On 2/27/2026 9:48 PM, Peter Zijlstra wrote: > On Fri, Feb 27, 2026 at 08:29:03PM +0530, K Prateek Nayak wrote: > >>> Both will result in at least one extra deref/cacheline for each futex >>> op, no? >> >> Ack but I was wondering if that penalty can be offset by the fact that >> we no longer need to look at "nr_node_ids" in a separate cacheline? >> >> I ran futex bench enough time before posting to come to conclusion that >> there isn't any noticeable regression - the numbers swung either ways >> and I just took one set for comparison. >> >> Sebastian and I have been having a more philosophical discussion on that >> CONFIG_NODES_SHIFT default but I guess as far as this patch is concerned, >> the conclusion is we want to avoid an extra dereference in the fast-path >> at the cost of a little bit extra space? > > Ooh, I just remebered, I've always wanted to apply Linus' runtime-const > stuff to the futex thing. I had no clue this existed! Nifty. > > Something like the below. But I'm not sure if it actually makes a > difference these days :/ > > But that can surely fix up the extra deref. > [..snip..] > @@ -1983,10 +1986,17 @@ static int __init futex_init(void) > hashsize = max(4, hashsize); > hashsize = roundup_pow_of_two(hashsize); > #endif > - futex_hashshift = ilog2(hashsize); > + __futex_mask = hashsize - 1; > + __futex_shift = ilog2(hashsize); > size = sizeof(struct futex_hash_bucket) * hashsize; > order = get_order(size); > > + void *__futex_queues = &__futex_data.queues; For __futex_queues, can we instead do: diff --git a/kernel/futex/core.c b/kernel/futex/core.c index 8b58d9035e3a..6cefa0629849 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -48,15 +48,6 @@ #include "futex.h" #include "../locking/rtmutex_common.h" -/* - * The base of the bucket array and its size are always used together - * (after initialization only in futex_hash()), so ensure that they - * reside in the same cacheline. - */ -static struct { - struct futex_hash_bucket *queues[MAX_NUMNODES]; -} __futex_data __read_mostly __aligned(2*sizeof(long)); - static u32 __futex_mask; static u32 __futex_shift; static struct futex_hash_bucket **__futex_queues; @@ -1991,12 +1982,14 @@ static int __init futex_init(void) size = sizeof(struct futex_hash_bucket) * hashsize; order = get_order(size); - void *__futex_queues = &__futex_data.queues; + __futex_queues = kcalloc(nr_node_ids, sizeof(*__futex_queues), GFP_KERNEL); runtime_const_init(shift, __futex_shift); runtime_const_init(mask, __futex_mask); runtime_const_init(ptr, __futex_queues); + BUG_ON(!futex_queues()); + for_each_node(n) { struct futex_hash_bucket *table; --- My machine didn't crash right away on running perf bench futex so I'm assuming this works? Sebastian, I haven't found any evidence of static data being interleaved across NUMA (at least on x86). Since kernel data is identity mapped, is it even possible for a static array to be allocated interleaved during boot unless the policy in the BIOS is set to interleaved? If the static allocation is same as a kcalloc(GFP_KERNEL) from NUMA standpoint, is the above feasible? > + > + runtime_const_init(shift, __futex_shift); > + runtime_const_init(mask, __futex_mask); > + runtime_const_init(ptr, __futex_queues); > + > for_each_node(n) { > struct futex_hash_bucket *table; > -- Thanks and Regards, Prateek