From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E40363B4EB7 for ; Tue, 28 Jul 2026 12:39:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785242386; cv=none; b=fK0y5j+vjGdbo6HhVIPDyFL9kReul5YMj23Ou7VX7WQl2R+hJV284OTqXtMT58c0suOThn+Beaw4/Z3osX5cDrYZKc3rWIei1HTV0QEwoH+AwMsEH9b14XwGRXCvGi7vrqkfYiK1IiYSNoVxN1gSRKWBJliXV1LRItOku63YFdU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785242386; c=relaxed/simple; bh=w1+YIJTVAP5jT45iufkHCyc6hDcCQv6aD2oTFAUaN5Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kvyN4tVLxOMYXxy+yfdE3yAy7HG4xADpWMDqw1dPgDL5PJQNBWFbbjB/9McWoW02gS4HrSKVv3it2HVM9Y5zPf/mH8A5zIsd6aXUcBOdywpp6Mn0/JCPkkN96cs/AChTHkcw4WTRI5yl/xy+lTvLGcGu+NJar0jaEbnhpLiyrTg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=e9MZfImJ; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="e9MZfImJ" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 261191BC0; Tue, 28 Jul 2026 05:39:39 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5D6063F86F; Tue, 28 Jul 2026 05:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785242383; bh=w1+YIJTVAP5jT45iufkHCyc6hDcCQv6aD2oTFAUaN5Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e9MZfImJJQjWew8KtLyIaQAjM1i7qvWrGcWwiEJo7mFFetOKMVHhSr8m1GJVNL/5p 68fnOIzemnhv1lsqLD0DGiLMejQMPynp0l5qVVkMCGJlx+0zOUCn3TFcpK/rtdY74q Y11YbY0E6vZi19/g5ivDqd99VbfEOemRgJ0jk4BQ= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ada.coupriediaz@arm.com, ardb@kernel.org, catalin.marinas@arm.com, hca@linux.ibm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, maz@kernel.org, peterz@infradead.org, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org, yang@os.amperecomputing.com Subject: [RFC PATCH 01/13] arm64: preempt: Simplify and optimize __preempt_count_dec_and_test() Date: Tue, 28 Jul 2026 13:38:47 +0100 Message-Id: <20260728123859.2911495-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260728123859.2911495-1-mark.rutland@arm.com> References: <20260728123859.2911495-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit In arm64's __preempt_count_dec_and_test(), the final conditional load of 'ti->preempt_count' is always executed in the common case. The conditional load leads to unfortunate code generation for preempt_enable[_notrace](), and it would be better to unconditionally load 'ti->preempt_count', as described below. On arm64, struct thread_info contains the following union: | union { | u64 preempt_count; | struct { | u32 count; | u32 need_resched; | } preempt; | }; Note: 'need_resched' is encoded so that '0' means a reschedule is needed, and '1' means a reschedule is NOT needed. The core logic of __preempt_count_dec_and_test() is: | static inline bool __preempt_count_dec_and_test(void) | { | struct thread_info *ti = current_thread_info(); | u64 pc = READ_ONCE(ti->preempt_count); | | WRITE_ONCE(ti->preempt.count, --pc); | | return !pc || !READ_ONCE(ti->preempt_count); | } The '!pc' condition can only be true when both: * The initial value of 'need_resched' was 0, meaning that a reschedule is needed. This should be rare. * The initial value of 'count' was exactly 1. This cannot be true for a nested preempt_disable() ... preempt_enable() sequence. Hence in common cases, '!pc' will be false, and it's necessary to execute the final READ_ONCE(ti->preempt_count). This results in a conditional branch in the common case, as can be seen when __preempt_count_dec_and_test() is outlined: | : | mrs x2, sp_el0 | ldr x1, [x2, #8] | mov w0, #0x1 | sub x1, x1, #0x1 | str w1, [x2, #8] | cbz x1, 1f | ldr x0, [x2, #8] | cmp x0, #0x0 | cset w0, eq // eq = none | 1: ret It would be better to avoid the special case for 'pc == 0', and to always load 'ti->preempt_count' after decrementing 'ti->preempt.count'. For the common cases this will remove a conditional branch. For the rare cases where preemption is needed initially, this only adds a single load, whose cost should be dominated by other factors. Remove the special case for 'pc == 0', and always load the combined 'ti->preempt_count' after decrementing 'ti->preempt.count'. The removal of the conditional branch helps with code generation, as the compiler can more easily move a dependent slow path out-of-line, as demonstrated with the following compiled with GCC 15.2.0: | void outline_preempt_enable_notrace(void) | { | preempt_enable_notrace(); | } Before this patch: | : | mrs x1, sp_el0 | ldr x0, [x1, #8] | sub x0, x0, #0x1 | str w0, [x1, #8] | cbz x0, 1f | ldr x0, [x1, #8] | cbnz x0, 2f | 1: paciasp | stp x29, x30, [sp, #-16]! | mov x29, sp | bl preempt_schedule_notrace | ldp x29, x30, [sp], #16 | autiasp | ret | 2: ret After this patch: | : | mrs x0, sp_el0 | ldr w1, [x0, #8] | sub w1, w1, #0x1 | str w1, [x0, #8] | ldr x0, [x0, #8] | cbz x0, 1f | ret | 1: paciasp | stp x29, x30, [sp, #-16]! | mov x29, sp | bl preempt_schedule_notrace | ldp x29, x30, [sp], #16 | autiasp | ret Signed-off-by: Mark Rutland Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Marc Zyngier Cc: Peter Zijlstra Cc: Vladimir Murzin Cc: Will Deacon Cc: Yang Shi --- arch/arm64/include/asm/preempt.h | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h index 932ea4b620428..ca2ad1a8db095 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -55,30 +55,23 @@ static inline void __preempt_count_sub(int val) WRITE_ONCE(current_thread_info()->preempt.count, pc); } -static inline bool __preempt_count_dec_and_test(void) -{ - struct thread_info *ti = current_thread_info(); - u64 pc = READ_ONCE(ti->preempt_count); - - /* Update only the count field, leaving need_resched unchanged */ - WRITE_ONCE(ti->preempt.count, --pc); - - /* - * If we wrote back all zeroes, then we're preemptible and in - * need of a reschedule. Otherwise, we need to reload the - * preempt_count in case the need_resched flag was cleared by an - * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE - * pair. - */ - return !pc || !READ_ONCE(ti->preempt_count); -} - static inline bool should_resched(int preempt_offset) { u64 pc = READ_ONCE(current_thread_info()->preempt_count); return pc == preempt_offset; } +static inline bool __preempt_count_dec_and_test(void) +{ + /* + * We must load the combined 'prempt_count' after decrementing + * 'preempt.count' as an interrupt could modify 'need_resched' before + * __preempt_count_sub() writes back to 'preempt.count'. + */ + __preempt_count_sub(1); + return should_resched(0); +} + #ifdef CONFIG_PREEMPTION void preempt_schedule(void); -- 2.30.2