From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 44E474BC017 for ; Wed, 29 Jul 2026 14:25:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785335159; cv=none; b=QzMT99PJ16AwoN1dmu1Xj/7RPxsXqKxQ08COLyWN8xz1QIYcSC7IOLFPBmlYds8zBY9BSJS5mpGjLlvIAoD7MJDQ0Ox4WnTCey8wXCaMwcxGAU6L1SV9pfY3J7ih3nADgPBQaRWMEeZLEFfGsNzh7MYouRNz76Vdj8aCs6bMLp8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785335159; c=relaxed/simple; bh=ErNyR86wrn94FrcMH6ADSAiqZQIZlnlT9oJFEVr45DQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=K3hpn0zLV1A+V/WxfAeOrRgzDqaYEnJ/X2TgMAj4bQBtm58uO6M0oeoyWRx/i+mPXbu5g1e7rn+gVd7Xbf6upSP17UkhxbBGSnNpq6dZMn9378+siEbWW9IdSTfVfoHf/K3bvFQ5sERFNpFrWn26I3R2D6Qzf1V0c1idgaluPoc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=FI9FDrTJ; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="FI9FDrTJ" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8B1B61684; Wed, 29 Jul 2026 07:25:52 -0700 (PDT) Received: from [10.1.34.163] (e121487-lin.cambridge.arm.com [10.1.34.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 65AA93F66F; Wed, 29 Jul 2026 07:25:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785335156; bh=ErNyR86wrn94FrcMH6ADSAiqZQIZlnlT9oJFEVr45DQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=FI9FDrTJZHGX9+blzSa+Zifj27VVeZ3rI7OJZJI8DGZsK5zLZwOnMDNbgyaoqKCTb lrxc2iRf7XWqwqtLnP3wsDygP5FUgdQ5HkVKpJclAUiiGFkow2CMu0/SxHeRQ4Jk73 bkpLMSU9jGIJfydZoaIp0sn1y0Y8OA29YJnPJ/qc= Message-ID: Date: Wed, 29 Jul 2026 15:25:52 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH 06/13] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops To: Mark Rutland , linux-arm-kernel@lists.infradead.org Cc: ada.coupriediaz@arm.com, ardb@kernel.org, catalin.marinas@arm.com, hca@linux.ibm.com, linux-kernel@vger.kernel.org, maz@kernel.org, peterz@infradead.org, ruanjinjie@huawei.com, will@kernel.org, yang@os.amperecomputing.com References: <20260728123859.2911495-1-mark.rutland@arm.com> <20260728123859.2911495-7-mark.rutland@arm.com> Content-Language: en-GB From: Vladimir Murzin In-Reply-To: <20260728123859.2911495-7-mark.rutland@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 7/28/26 13:38, Mark Rutland wrote: > Currently arm64's this_cpu_*() ops transiently disable preemption in > order to guarantee that the address generation and memory access(es) > occur on the same CPU. > > Transiently disabling preemption can be expensive. When re-enabling > preemption it is necessary to make a conditional function call to > preempt_schedule[_notrace]() in order to handle the rare case that the > task needs to be rescheduled. The potential function call has a number > of negative effects on code generation (e.g. due to the need to create a > stack frame and spill registers), and the conditionality can result in > poor code generation and/or poor branch prediction. > > This patch adds infrastructure for a scheme where this_cpu_*() ops do > not need to transiently disable preemption, avoiding the negative > impacts described above. > > Each operation registers a critical section during which the exception > return code will adjust the offset and addresses if preemption occurs > mid-sequence. The critical section is registered/unregistered with a > small prologue and epilogue which encodes three distinct GPRRs (, > , ) into a new thread_info::pcp_gprs field: > > // Prologue. Enable fixups for and . > mrs , sp_el0 > mov , #__VAL_PCPU_GPRS(, , ) > strh , [, #TSK_TI_PCPU_GPRS] > > // Generate cpu-specific address > mrs , TPIDR_ELx > add , , > > // Perform access sequence > ldr , [] > > // Epilogue. Disable fixups > strh wzr, [, #TSK_TI_PCPU_GPRS] > > If an exception is taken from within the critical section, the exception > return code will adjust to be the current CPU's offset, and will > adjust to be ( + ). Distinct registers are used for > , , and , so that the fixup can be applied safely at any > point during the critical section. > > To ensure that this_cpu_*() operations within exception handlers work > correctly and do not corrupt state, thread_info::pcpu_gprs is saved > into a new pt_regs::pcpu_gprs field upon exception entry, and restored > upon exception return. > > Looking at a simple this_cpu_operation: > > | void outline_this_cpu_add_u64(u64 __percpu *p, u64 v) > | { > | this_cpu_add(*p, v); > | } > > Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as: > > | : > | paciasp > | stp x29, x30, [sp, #-16]! > | mrs x2, sp_el0 > | mov x29, sp > | ldr w3, [x2, #8] > | add w3, w3, #0x1 > | str w3, [x2, #8] > | mrs x3, tpidr_el1 > | add x0, x0, x3 > | 1: ldxr x5, [x0] > | add x5, x5, x1 > | stxr w4, x5, [x0] > | cbnz w4, 1b > | ldr x0, [x2, #8] > | sub x0, x0, #0x1 > | str w0, [x2, #8] > | cbz x0, 2f > | ldr x0, [x2, #8] > | cbnz x0, 3f > | 2: bl preempt_schedule_notrace > | 3: ldp x29, x30, [sp], #16 > | autiasp > | ret > > With the scheme added in this patch, this can be compiled as: > > | : > | mrs x2, sp_el0 > | mov x4, #0xc80 > | strh w4, [x2, #20] > | mrs x4, tpidr_el1 > | add x3, x0, x4 > | 1: ldxr x6, [x3] > | add x6, x6, x1 > | stxr w5, x6, [x3] > | cbnz w5, 1b > | strh wzr, [x2, #20] > | ret > > TODO: Save/restore the PCPU GPRs in __sdei_asm_handler(). This will > require some mechanical rework to the __sdei_asm_handler() assembly. > > Signed-off-by: Mark Rutland > Cc: Ada Couprie Diaz > Cc: Ard Biesheuvel > Cc: Catalin Marinas > Cc: Jinjie Ruan > Cc: Marc Zyngier > Cc: Peter Zijlstra > Cc: Vladimir Murzin > Cc: Will Deacon > Cc: Yang Shi > --- > arch/arm64/include/asm/percpu.h | 43 ++++++++++++++++++++++++++++ > arch/arm64/include/asm/ptrace.h | 5 ++++ > arch/arm64/include/asm/thread_info.h | 1 + > arch/arm64/kernel/asm-offsets.c | 2 ++ > arch/arm64/kernel/entry-common.c | 38 ++++++++++++++++++++++++ > arch/arm64/kernel/entry.S | 13 +++++++++ > 6 files changed, 102 insertions(+) > > diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h > index 98823c97d534c..0871dcc41d759 100644 > --- a/arch/arm64/include/asm/percpu.h > +++ b/arch/arm64/include/asm/percpu.h > @@ -5,10 +5,12 @@ > #ifndef __ASM_PERCPU_H > #define __ASM_PERCPU_H > > +#include > #include > > #include > #include > +#include > #include > #include > > @@ -51,6 +53,47 @@ static inline unsigned long __kern_my_cpu_offset(void) > return off; > } > > +#define PCPU_GPR_PCP GENMASK(4, 0) > +#define PCPU_GPR_OFF GENMASK(9, 5) > +#define PCPU_GPR_ADDR GENMASK(14, 10) > + > +#define __VAL_PCPU_GPRS(pcp, off, addr) \ > + "(" \ > + "(.L__gpr_num_" pcp " << 0) | " \ > + "(.L__gpr_num_" off " << 5) | " \ > + "(.L__gpr_num_" addr " << 10)" \ > + ")" Later in the patch, there is a comment stating that these registers are not expected to overlap. I can also see that early-clobber constraints are applied to the output registers later in the patch series, so the code is correct. However, would it be possible to add assertions that detect register overlap, perhaps something like: ".if (.L__gpr_num_" pcp "== .L__gpr_num_" off ") ||" \ " (.L__gpr_num_" pcp "== .L__gpr_num_" addr ") ||" \ " (.L__gpr_num_" off "== .L__gpr_num_" addr")" \ ".error "inline asm registers overlap" \ ".endif" \ or any other (better) way. Such assertions would serve both as documentation and as a strong guarantee that the registers are distinct. Cheers Vladimir