From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012071.outbound.protection.outlook.com [52.101.48.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C4752D3A75 for ; Fri, 27 Feb 2026 14:59:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.71 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772204365; cv=fail; b=NXJuG51HmT6QMIJ1io4ofMMsxyZPSPL1HFFl3BZ01xUJXM9ZuPjD6OfOBlO7U37APDOTOUvUHHh0ZJia33Kkz0brZ9p1C6xWUw/FeHKb5D1bPI6GOch4OJZOcRLFsHqGBdglXhtdxCQg5gw7/7XHyVh0ob3nQI653jDMK63moo8= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772204365; c=relaxed/simple; bh=PVznLZ4596tQCHqoiAk69ZlcXUm/tMdL2Wqbt7DUZYo=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=finulAY/bvbO4qGWPXKCxqp+NiSoNjd98UpplyubEjXL5a5iwmoDEHbrL166s/rizTiypi04aJeYyqPvOMW/TWuPJXn9oHeAQhQAUuiLCLAwG/aA6nWzsBAqZ7s0axD23wJkSWwS2U1Tdjzw221TeKfi+erfKRTCR+E3crnl7Dk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=IMnVu0dk; arc=fail smtp.client-ip=52.101.48.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="IMnVu0dk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FH7Y0mlEjQMCcle3Q5BMcFA5uDYVVOlUatkLvM5SM1+qGD4aIxafhCZm+ChAsSOxzaIChQz6ABm3U+EFwfRK+TChOTpV5zjB/MO8lvsxYHM+4RsVecjyGq+Tc8V9A5/68YZ9/2rx5nsd9NCg7sLvqdi1bAoGIRWlLDsLZlJZQcOTWWImtnRYcH5VtFZpPLVmR/XJBeRWtPAGbCDshyhfWBlA+8uygBvjh2A5qfh5YyNSXZfOcdbvs3GPCf2jsPL97Qr8n0tpr+v/kPP0UgUkhM207bodAdcuUi7mCga0Uifo26nQ4a+8Q1SPSkntsmTyeMb4ekYjh3LEPkT1ilkU3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ok7D+WqWqPJ4vbPfapmTU7bcipM+Z0MsZVDjTEyXPGo=; b=vdQFTFHEjZ1/28mQ6NRs0u7jLFBOVx89O1Z3CwwSbdDToBDYkopmT4/f0nwwDU8NyNeuhJQAfUwouuA2tdJBmawz75LXl7fqE0gjjyo05yhaxr6Lq/RudSGqg5I4x1F5uAOt3jCqmchEGYW85DEowFGBfBwOKBqpKuSiOT1LdjpcBuna+M4jfpQAwaYVSkzpqShVsUgTRCoY+Q6scvGfJQ//eH/tIhnqYUgFQq/fXJPTOsSsvM6VgTmTgymYvQ4VXH3Cmg9pVf1ARUhlmIHIg5tFO6XlsRK1GsBUzS0rQ24Bvc+vJBDLkcrnbPQPhA3vHfq2paViVwCbHHNsvhADzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ok7D+WqWqPJ4vbPfapmTU7bcipM+Z0MsZVDjTEyXPGo=; b=IMnVu0dkDY1Jcupec1nnVRjvm/3AS+zWCkY7qFRLsBscxwTSiTpfOfiGdexIL30BqMY5MgjnYgn0GmDp6K3TPeWR63aB9a9gZkdYdytIaLBzrFsjKbgiFHq5hpfcuDzbG/3wyJiXAt4PCP3zPZSFx7crdqRxGplgC6NBsKneSrM= Received: from BLAPR03CA0134.namprd03.prod.outlook.com (2603:10b6:208:32e::19) by PH0PR12MB8821.namprd12.prod.outlook.com (2603:10b6:510:28d::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.15; Fri, 27 Feb 2026 14:59:16 +0000 Received: from MN1PEPF0000ECD7.namprd02.prod.outlook.com (2603:10b6:208:32e:cafe::8c) by BLAPR03CA0134.outlook.office365.com (2603:10b6:208:32e::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.26 via Frontend Transport; Fri, 27 Feb 2026 14:58:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by MN1PEPF0000ECD7.mail.protection.outlook.com (10.167.242.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.16 via Frontend Transport; Fri, 27 Feb 2026 14:59:15 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 27 Feb 2026 08:59:15 -0600 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 27 Feb 2026 06:59:13 -0800 Received: from [172.31.184.125] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Fri, 27 Feb 2026 08:59:06 -0600 Message-ID: Date: Fri, 27 Feb 2026 20:29:03 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] futex: Dynamically allocate futex_queues depending on nr_node_ids To: Peter Zijlstra CC: Thomas Gleixner , Ingo Molnar , Sebastian Andrzej Siewior , , Darren Hart , "Davidlohr Bueso" , =?UTF-8?Q?Andr=C3=A9_Almeida?= References: <20260128101358.20954-1-kprateek.nayak@amd.com> <20260227144203.GJ1282955@noisy.programming.kicks-ass.net> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260227144203.GJ1282955@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD7:EE_|PH0PR12MB8821:EE_ X-MS-Office365-Filtering-Correlation-Id: a2a82de2-2496-48b4-e7ad-08de7610c672 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026|7142099003; X-Microsoft-Antispam-Message-Info: M6s+Ab/BVvzWhnBgdI6gkboq0RWChRpglzURtpYJXUM0rdAzyGv3dzLQPoTjc5catcZFnqMC+L+y2KRMZFgbmvFuds51Z6Ok8W3zsworuMMSP+lh/N3cnSH+C8bKXbmzR5NAPsfHj8UJh93duAi3sZ+tQKMkLjoKWHwDrPt7medTCsQLWJp5LEcMfvYzjIzCyLA+AalDqcgu64d22NhRCFYq7olhkejW0tdCGags5QWtf47TcYRVk4iZVLeuVoHUPrqoy06r4IDjnBQ8s9iiJVNSVRLoC2ySQMe0ArQdGgzuzJY/XQZZ/I4fI0m1EN+ctipi0Juox8xoCGL1S3fvppz0ArqcQ7KMylzGxyu+hch5B/UhmX9ZPCRTdsSK2VNXCPh7TrHAYT/V+KuvVlbEDx3rq7Obr01AVe1MZK+s+Suw1Fhffc03o7nFl0W9OXGZsS7vhRbPTUkGypsIjnNXgL8cuI58k0WSsoFq7s7cfNzWqODsVIAElMvNUptuYXMuqK1fQ76IeIxJ6quggds2ZluK4R5WFUiZhmx4gMLfoJ5R/Q0X1aLg1te+b5pjciN78iezvJBQOeaMclQ8UkMvx5eBMi4T6+qscsBGZtdPA6TazYim+VaVBx5wPA9gM2drtWBOMzGxHPGv7irCfL0hWeJ/Q51Htor5HreHHHFWtMjqIhtNRClwZETp7n3TPpyc3G31hQhanKt90wT1uufqNykwhXprZzHPWV1cfCrJ5kw+U3MoOzUp/e07kMHLprZRYqLgRXemRGxmsoncJVUIr5DRIyHaSh/uiEosIPfROpTahzN9rP3//tQl8jyRQIA9DQ5m8zYuzBs7cUokk6G2QQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026)(7142099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PUGI/0qKVOfSAWqzbW73aN//rP/zVBkfUC9IkEGKGdi6wx3zFoa9heHmtKWEwZCBJutXBSqfnDGSZ/WdWNGJ1vMZeTqjL9DLieDQTMPrpi3qg6Uzd7lPMQPw73SMxEJUe0r5dsFXHwpNrccndiEK88KtXpx8ZgJFc/AQoYxQ3XXL0x5Q3kLHVUu6Vxq95+ig5+YoZNgRREwfyKcszYnuURL/Q32e2S7PfQNjAOmV9UEK3sRDb/s8cJpqrYIcqYLe5eHIKCnSg1LiKjGkXmrm6PXCNKsC86KK3RcjqVVYwuFhKTn7+pdJzY3JjbvrQdnbpNn88XBLHVHeOS+aPP8XaAk0OyVbGwTt93qYu9Ji4vcTg+prY9/JLP50dgayS21SoIRiKfTnYYVE6uZFPKW7xo7OtHOt/UGaQqQd/HkUOgW46dVp69/R+eYRoC5iCAQE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2026 14:59:15.3717 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a2a82de2-2496-48b4-e7ad-08de7610c672 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD7.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8821 Hello Peter, On 2/27/2026 8:12 PM, Peter Zijlstra wrote: > On Wed, Jan 28, 2026 at 10:13:58AM +0000, K Prateek Nayak wrote: >> CONFIG_NODES_SHIFT (which influences MAX_NUMNODES) is often configured >> generously by distros while the actual number of possible NUMA nodes on >> most systems is often quite conservative. >> >> Instead of reserving MAX_NUMNODES worth of space for futex_queues, >> dynamically allocate it based on "nr_node_ids" at the time of >> futex_init(). >> >> "nr_node_ids" at the time of futex_init() is cached as "nr_futex_queues" >> to compensate for the extra dereference necessary to access the elements >> of futex_queues which ends up in a different cacheline now. >> >> Running 5 runs of perf bench futex showed no measurable impact for any >> variants on a dual socket 3rd generation AMD EPYC system (2 x 64C/128T): >> >> variant locking/futex base + patch %diff >> futex/hash 1220783.2 1333296.2 (9%) >> futex/wake 0.71186 0.72584 (2%) >> futex/wake-parallel 0.00624 0.00664 (6%) >> futex/requeue 0.25088 0.26102 (4%) >> futex/lock-pi 57.6 57.8 (0%) >> >> Note: futex/hash had noticeable run to run variance on test machine. >> >> "nr_node_ids" can rarely be larger than num_possible_nodes() but the >> additional space allows for simpler handling of node index in presence >> of sparse node_possible_map. >> >> Reported-by: Sebastian Andrzej Siewior >> Signed-off-by: K Prateek Nayak >> --- >> Sebastian, >> >> Does this work for your concerns with the large "MAX_NUMNODES" values on >> most distros? It does put the "queues" into a separate cacheline from >> the __futex_data. >> >> The other option is to dynamically allocate the entire __futex_data as: >> >> struct { >> unsigned long hashmask; >> unsigned int hashshift; >> unsigned int nr_queues; >> struct futex_hash_bucket *queues[] __counted_by(nr_queues); >> } *__futex_data __ro_after_init; >> >> with a variable length "queues" at the end if we want to ensure >> everything ends up in the same cacheline but all the __futex_data >> member access would then be pointer dereferencing which might not be >> ideal. >> >> Thoughts? > > Both will result in at least one extra deref/cacheline for each futex > op, no? Ack but I was wondering if that penalty can be offset by the fact that we no longer need to look at "nr_node_ids" in a separate cacheline? I ran futex bench enough time before posting to come to conclusion that there isn't any noticeable regression - the numbers swung either ways and I just took one set for comparison. Sebastian and I have been having a more philosophical discussion on that CONFIG_NODES_SHIFT default but I guess as far as this patch is concerned, the conclusion is we want to avoid an extra dereference in the fast-path at the cost of a little bit extra space? -- Thanks and Regards, Prateek