From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAC78448CEF; Wed, 29 Jul 2026 09:55:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785318915; cv=none; b=a4gutSWdfmpKETaMWrcCMRIYjFmAxQ0e6pxfxr4iuevKiRjEO3G//4yctINkWRGZz6++BVwMk2lI0Q3oXcfDCPOf4Mbojvl8QwhIQFUPxUzepV62pdRm5puxg9yN6e2/wlDCS765nLzs18nFExXLY7taAjU//Mu5tFTpJkzQ3ec= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785318915; c=relaxed/simple; bh=li1hCEdsiJONstS/2Ml852HBXPmnq/4mc1M0ysWl9lo=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=njW04ZbO1TdTUTqYx5/6lU4X31morsSUHonRDtkrAld91iy8r+NRYkUBiTXDo4EYIxnALRL3ItdTGvP3P9ZUDZZgOQIy52yCkxTmFS0lcqZMUX0NkQjzO8wQZ4X4UwujoGXiv5tZiIQk6jT94iz5EfgLjwyH3URydEMEOBJ2mQ0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=D7z1iYNI; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Y7noXZg9; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="D7z1iYNI"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Y7noXZg9" Date: Wed, 29 Jul 2026 09:55:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785318909; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yzJa7dd30lf53/sIWBdxL6FY+ASx+B/7sbutVnQw8Tg=; b=D7z1iYNIifgT/Bkn+JAwCDb6FsVIZsVNUkCLMXlSBHUaRNpoGY1y5+USF4RzB9aK082jOl BFgmGDy6JDpknzkEs8gH2qqsHexrk+l8f4jbqWSkBglAt/9w+QfJ/spgwecLt6Zik2vf1Y SarZjsjfuEwMuyaIEVNYslihRmB8LCdOOoYn4qG5wJxfdaH3ntyTIyevg7j9cbtRBHGyJK +4hAKD3BYzukWVW3fFpZn2RM4f4xRy4fmVJx+fPB+jMOHJN8rmDdk+6l0IC6w9PbJ4z4cZ UkkyQ/sH+lAiLHDxOtCpMaGR4KmV13j/HMCHLj/T3n6TPDoLsAlFhqttiQ+2ZA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785318909; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yzJa7dd30lf53/sIWBdxL6FY+ASx+B/7sbutVnQw8Tg=; b=Y7noXZg9eoYfN1BP6Y+3lWb1CKDHkheBhf424l88kt7KSje0q7uBB6WSbAR8sICIlA0uRv n3vppK1Pan5PW+Bw== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] futex: Use runtime constants for __futex_hash() hot path Cc: "Peter Zijlstra (Intel)" , Sebastian Andrzej Siewior , K Prateek Nayak , Charlie Jenkins , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260728052540.4728-9-kprateek.nayak@amd.com> References: <20260728052540.4728-9-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <178531890808.112641.6335881614564690088.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the locking/core branch of tip: Commit-ID: 45ac711e2f14833534a57f785f17e20873b1ba21 Gitweb: https://git.kernel.org/tip/45ac711e2f14833534a57f785f17e20873b= 1ba21 Author: Peter Zijlstra AuthorDate: Tue, 28 Jul 2026 05:25:40=20 Committer: Peter Zijlstra CommitterDate: Tue, 28 Jul 2026 13:07:16 +02:00 futex: Use runtime constants for __futex_hash() hot path Runtime constify the read-only after init data __futex_shift(shift_32), __futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash() hot path to avoid referencing global variable. This also allows __futex_queues to be allocated dynamically to "nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES (1 << CONFIG_NODES_SHIFT) worth of slots upfront. Runtime constants are initialized before their first access and runtime_const_init() provides necessary barrier to ensure subsequent accesses are not reordered against their initialization. No functional changes intended. perf bench futex on a 3rd Gen EPYC (2 x 64C/128T): +----------------+-----------+-----------+-----------+--------------+ | Benchmark | Kernel 1 | Kernel 2 | Unit | % Improvement| | | (avg/5) | (avg/5) | | (K2 vs K1) | +----------------+-----------+-----------+-----------+--------------+ | Wake-parallel | 0.01614 | 0.00456 | ms | +71.75% | | Requeue | 0.26394 | 0.24644 | ms | +6.63% | | Lock-pi | 34.0 | 57.2 | ops/sec | +68.24% | +----------------+-----------+-----------+-----------+--------------+ Performance testing on a 144-thread Intel(R) Xeon(R) CPU E7-8890 v3 (4 NUMA n= odes): +-------------------------------------------------------------+ | perf bench futex hash -b 0 | +----------------------+------------+------------+------------+ | Configuration | As-is | Patched | Delta | +----------------------+------------+------------+------------+ | 1 thread, 1 futex | 6,449,632 | 6,532,004 | +1.28% | | 144 threads, 1024 fx | 2,111,486 | 2,139,685 | +1.34% | +----------------------+------------+------------+------------+i [ prateek: Dynamically allocate __futex_queues, mark the global data __ro_after_init since they are constified after futex_init(). ] Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sebastian Andrzej Siewior # MAX_NUMNODES= bloat Signed-off-by: K Prateek Nayak Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Charlie Jenkins Tested-by: Charlie Jenkins Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kick= s-ass.net Link: https://patch.msgid.link/20260728052540.4728-9-kprateek.nayak@amd.com --- include/asm-generic/vmlinux.lds.h | 5 ++- kernel/futex/core.c | 44 +++++++++++++++++------------- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.= lds.h index 5659f4b..5320790 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -970,7 +970,10 @@ RUNTIME_CONST(ptr, __dentry_cache) \ RUNTIME_CONST(ptr, __names_cache) \ RUNTIME_CONST(ptr, __filp_cache) \ - RUNTIME_CONST(ptr, __bfilp_cache) + RUNTIME_CONST(ptr, __bfilp_cache) \ + RUNTIME_CONST(shift, __futex_shift) \ + RUNTIME_CONST(mask, __futex_mask) \ + RUNTIME_CONST(ptr, __futex_queues) =20 /* Alignment must be consistent with (kunit_suite *) in include/kunit/test.h= */ #define KUNIT_TABLE() \ diff --git a/kernel/futex/core.c b/kernel/futex/core.c index 179b26e..3bf6cb7 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -48,23 +48,19 @@ =20 #include =20 +#include + #include "futex.h" #include "../locking/rtmutex_common.h" =20 -/* - * The base of the bucket array and its size are always used together - * (after initialization only in futex_hash()), so ensure that they - * reside in the same cacheline. - */ -static struct { - unsigned long hashmask; - unsigned int hashshift; - struct futex_hash_bucket *queues[MAX_NUMNODES]; -} __futex_data __read_mostly __aligned(2*sizeof(long)); +static u32 __futex_mask __ro_after_init; +static u32 __futex_shift __ro_after_init; +static struct futex_hash_bucket **__futex_queues __ro_after_init; =20 -#define futex_hashmask (__futex_data.hashmask) -#define futex_hashshift (__futex_data.hashshift) -#define futex_queues (__futex_data.queues) +static __always_inline struct futex_hash_bucket **futex_queues(void) +{ + return runtime_const_ptr(__futex_queues); +} =20 struct futex_private_hash { int state; @@ -395,13 +391,13 @@ __futex_hash(union futex_key *key, struct futex_private= _hash *fph, struct futex_ * NOTE: this isn't perfectly uniform, but it is fast and * handles sparse node masks. */ - node =3D (hash >> futex_hashshift) % nr_node_ids; + node =3D runtime_const_shift_right_32(hash, __futex_shift) % nr_node_ids; if (!node_possible(node)) { node =3D find_next_bit_wrap(node_possible_map.bits, nr_node_ids, node); } } =20 - return &futex_queues[node][hash & futex_hashmask]; + return &futex_queues()[node][runtime_const_mask_32(hash, __futex_mask)]; } =20 /** @@ -1922,7 +1918,7 @@ int futex_hash_allocate_default(void) * 16 <=3D threads * 4 <=3D global hash size */ buckets =3D roundup_pow_of_two(4 * threads); - buckets =3D clamp(buckets, 16, futex_hashmask + 1); + buckets =3D clamp(buckets, 16, __futex_mask + 1); =20 if (current_buckets >=3D buckets) return 0; @@ -2020,10 +2016,21 @@ static int __init futex_init(void) hashsize =3D max(4, hashsize); hashsize =3D roundup_pow_of_two(hashsize); #endif - futex_hashshift =3D ilog2(hashsize); + __futex_mask =3D hashsize - 1; + __futex_shift =3D ilog2(hashsize); size =3D sizeof(struct futex_hash_bucket) * hashsize; order =3D get_order(size); =20 + __futex_queues =3D kcalloc(nr_node_ids, sizeof(*__futex_queues), GFP_KERNEL= ); + + runtime_const_init(shift, __futex_shift); + runtime_const_init(mask, __futex_mask); + runtime_const_init(ptr, __futex_queues); + + barrier(); + + BUG_ON(!futex_queues()); + for_each_node(n) { struct futex_hash_bucket *table; =20 @@ -2037,10 +2044,9 @@ static int __init futex_init(void) for (i =3D 0; i < hashsize; i++) futex_hash_bucket_init(&table[i]); =20 - futex_queues[n] =3D table; + futex_queues()[n] =3D table; } =20 - futex_hashmask =3D hashsize - 1; pr_info("futex hash table entries: %lu (%lu bytes on %d NUMA nodes, total %= lu KiB, %s).\n", hashsize, size, num_possible_nodes(), size * num_possible_nodes() / 1024, order > MAX_PAGE_ORDER ? "vmalloc" : "linear");