From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4C3F23A9DB1 for ; Tue, 28 Jul 2026 12:39:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785242383; cv=none; b=LDoC742n3d+1qd9VtbVUb0lBtqsc1cAkhyLAiDQ2UEvqXRCX9QTsmf3RVfcUsjQzP1yCFI7CHxY5UkVabM9p/XAUmqkLFizL8Rbp6AprNV01aZMUicYAd+VKMkcUr82IM0SXfOiCAQazvt9zp50XNNVr63HdaU5QNoB6QhNUNDM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785242383; c=relaxed/simple; bh=sDf8G/U2jwEJKijTrykFX6woe/52UXXgD/kVNcJr1/U=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=Lcnu4jzh4uHooxpoE8WDlaviahNTIT/Yql5M7Yyh1DUWwxbaIk41E4Hmtr+01dndW4KVhYsVdyDLX8UR309OCU3L7WU/sD3KCa+ooA7zGlANWpCAkLcmZsdJINsOmQUAB9T2f5thXfjtrFuuHYaEocaB5sdw5lX3dKU6vez/jV4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=DPgwmTFs; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="DPgwmTFs" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30EA61A9A; Tue, 28 Jul 2026 05:39:36 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 697093F86F; Tue, 28 Jul 2026 05:39:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785242380; bh=sDf8G/U2jwEJKijTrykFX6woe/52UXXgD/kVNcJr1/U=; h=From:To:Cc:Subject:Date:From; b=DPgwmTFsquV6gpRk55zrZqCBgLpjukPuAxCL2twrXEosv4V/rITab9Ie2Gcu526/t X+7hxlIFjzJyU/3f5gYxp7rSusuIUUibT3bunzETTx5tL8lAcOgHPXVgLe5mSTfHAh h6iQOn7l0sBug8+KxmkNilduMPMueysXHOuWPNvk= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ada.coupriediaz@arm.com, ardb@kernel.org, catalin.marinas@arm.com, hca@linux.ibm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, maz@kernel.org, peterz@infradead.org, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org, yang@os.amperecomputing.com Subject: [RFC PATCH 00/13] arm64: Preemptible this_cpu_*() operations Date: Tue, 28 Jul 2026 13:38:46 +0100 Message-Id: <20260728123859.2911495-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Currently arm64's this_cpu_*() ops transiently disable preemption in order to guarantee that the address generation and memory access(es) occur on the same CPU, which has some unfortunate overhead. In particular, re-enabling preemption requires a conditional call to preempt_schedule[_notrace](), which has a number of adverse impacts on code generation. This series reworks arm64's this_cpu_*() operations such that they do not need to disable preemption, avoiding the related overhead in the fast paths. Instead, the ope register a "PCPU GPR" critical section, during which which the entry code will apply a fixup to GPRs containing the percpu offset and the generated percpu address. The scheme is described in detail in patch 6, and is similar to the approach Heiko Carstens applied to s390 [1]. For example, an outlined this_cpu_read() goes from: | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x2, sp_el0 | mov x29, sp | ldr w3, [x2, #8] | add w3, w3, #0x1 | str w3, [x2, #8] | mrs x3, tpidr_el1 | str x1, [x0, x3] | ldr x0, [x2, #8] | sub x0, x0, #0x1 | str w0, [x2, #8] | cbz x0, 1f | ldr x0, [x2, #8] | cbnz x0, 2f | 1: bl preempt_schedule_notrace | 2: ldp x29, x30, [sp], #16 | autiasp | ret ... to: | : | mrs x2, sp_el0 | mov w3, #0x7c60 | strh w3, [x2, #20] | mrs x3, tpidr_el1 | str x1, [x0, x3] | strh wzr, [x2, #20] | ret Comparing defconfig before and after, the Image size is reduced by 64K, though .text is only reduced by ~7.5K: | [mark@lakrids:~/src/linux]% ls -al Image-defconfig-* | -rw-r--r-- 1 mark mark 41552384 Jul 28 11:07 Image-defconfig-after | -rw-r--r-- 1 mark mark 41617920 Jul 28 11:01 Image-defconfig-v7.2-rc4 | [mark@lakrids:~/src/linux]% ls -al vmlinux-defconfig-* | -rwxr-xr-x 1 mark mark 158239216 Jul 28 11:07 vmlinux-defconfig-after | -rwxr-xr-x 1 mark mark 158699800 Jul 28 11:01 vmlinux-defconfig-v7.2-rc4 | [mark@lakrids:~/src/linux]% size vmlinux-defconfig-* | text data bss dec hex filename | 20323522 21028511 730840 42082873 2822239 vmlinux-defconfig-after | 20331206 21028447 730840 42090493 2823ffd vmlinux-defconfig-v7.2-rc4 >From looking at the generated code, there are some reasonable savings we could make with a few changes to alternatives, notably: * Patching the read of TPIDR_ELx using a callback. This would remove the need for replacement instructions, saving ~12K for defconfig, and meaning real instructions in .text would be packed more densely. There's work underway to apply similar patching for ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE. * Allowing {relocation,veneer}-free replacement sequences to be placed in .altinst_rreplacement again. This would allow real instructions in .text to be packed more densely, with ~80K of .text moving into .altinstr_replacement where it would be freed after init. I've thrown togther a prototype [2]. I'm sending this as an RFC as: (1) I haven't had the chance to benchmark this. While I think the changes look good from inspection, we should obviously get some actual performance figures. (2) I haven't wired up everything for SDEI exception entry/return. This should only require some mechinical rework to the __sdei_asm_handler() assembly, and I've noted this as a TODO on patch 6. Patches 1 and 2 are complete as-is, and can be taken without the rest of the series. (3) I'm not sure whether this_cpu_read() should use a plain LDR, or use LDA[P]R when LTO is in use to match READ_ONCE(). Using LDA[P]R might require some refactoring of the existing READ_ONCE() code to share the logic. I've noted this as a question on patch 7. Thanks, Mark. [1] https://lore.kernel.org/linux-s390/20260526055702.1429061-1-hca@linux.ibm.com/ [2] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/alternatives/cleanup Mark Rutland (13): arm64: preempt: Simplify and optimize __preempt_count_dec_and_test() arm64: preempt: Treat should_resched() as unlikely arm64: ptrace: Always inline pt_regs_[read,write}_reg() arm64: percpu: Factor out percpu offset asm arm64: gpr-num: Add wxN aliases for wN registers arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops arm64: percpu: Implement preemptible read/write ops arm64: percpu: Implement preemptible void RMW ops arm64: percpu: Implement preemptible return RMW ops arm64: percpu: Implement preemptible XCHG ops arm64: percpu: Implement preemptible CMPXCHG ops arm64: percpu: Implement preemptible CMPXCHG128 ops arm64: percpu: Remove _pcp_protect*() wrappers arch/arm64/include/asm/gpr-num.h | 5 + arch/arm64/include/asm/percpu.h | 437 +++++++++++++++++++++------ arch/arm64/include/asm/preempt.h | 29 +- arch/arm64/include/asm/ptrace.h | 12 +- arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 2 + arch/arm64/kernel/entry-common.c | 38 +++ arch/arm64/kernel/entry.S | 13 + 8 files changed, 417 insertions(+), 120 deletions(-) -- 2.30.2