From: Yunhui Cui <cuiyunhui@bytedance.com>
To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu,
alex@ghiti.fr, dennis@kernel.org, tj@kernel.org, cl@gentwo.org,
oleg@redhat.com, ast@kernel.org, daniel@iogearbox.net,
andrii@kernel.org, eddyz87@gmail.com, memxor@gmail.com,
martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev,
jolsa@kernel.org, emil@etsalapatis.com, ihor.solodrai@linux.dev,
bjorn@kernel.org, pulehui@huawei.com, puranjay@kernel.org,
broonie@kernel.org, cuiyunhui@bytedance.com, dave@stgolabs.net,
david.laight.linux@gmail.com, jszhang@kernel.org,
tglx@kernel.org, conor.dooley@microchip.com, thuth@redhat.com,
ajones@ventanamicro.com, ben.dooks@codethink.co.uk,
rkrcmar@ventanamicro.com, samuel.holland@sifive.com,
zong.li@sifive.com, debug@rivosinc.com, seanwascoding@gmail.com,
andybnac@gmail.com, menglong8.dong@gmail.com,
cyrilbur@tenstorrent.com, wangruikang@iscas.ac.cn,
atishp@rivosinc.com, apatel@ventanamicro.com, arnd@arndb.de,
nathan@kernel.org, nick.desaulniers+lkml@gmail.com,
morbo@google.com, justinstitt@google.com,
qingfang.deng@siflower.com.cn, linux-riscv@lists.infradead.org,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
bpf@vger.kernel.org, linux-arch@vger.kernel.org,
llvm@lists.linux.dev
Subject: [PATCH v10 0/9] riscv: optimize percpu access
Date: Thu, 12 Nov 2026 18:41:40 +0800 [thread overview]
Message-ID: <20261112104149.38589-1-cuiyunhui@bytedance.com> (raw)
RISC-V has no asm/percpu.h, so this_cpu_*() read-modify-write
operations use generic fallbacks that disable local interrupts. Per-CPU
addressing also derives the current offset from the CPU number and
__per_cpu_offset[].
Cache the per-CPU offset in thread_info and implement preemptible this_cpu
operations using recoverable GPR critical sections. Each operation records
the registers used to form its per-CPU address so exception return can
repair them if the task resumes on a different CPU. Use AMOs for
native-width operations. Use Zabha for 8-bit and 16-bit operations when
available, or a 32-bit LR/SC fallback otherwise. Finally, select
HAVE_CMPXCHG_LOCAL so common per-CPU counter users can use
this_cpu_try_cmpxchg().
The recoverable-GPR scheme follows the arm64 design. Only the protocol is
common, however: RISC-V uses a cached offset in thread_info, its own GPR
encoding and exception entry code, and AMO/LR/SC operation sequences.
Factoring these pieces into generic code would share little beyond the
field layout while coupling architecture entry paths, so keep the
implementation under arch/riscv.
As with arm64's SDEI path, independent NMI-like entry paths must
participate in the entry-state protocol directly. Keep the save, clear and
restore operations in common RISC-V assembly macros so SBI Supervisor
Software Events (SSE) can preserve an interrupted per-CPU operation. SBI
SSE support is developed separately and is not a dependency of this
series.
With CONFIG_PREEMPT_LAZY=y, GCC 15, the same linux-next base and
configuration, an in-kernel microbenchmark pinned to one CPU ran each
operation 2,000,000 times for 15 rounds. The table reports the median
empty-loop-adjusted time; lower is better.
operation v9 (ns/op) v10 (ns/op) change
this_cpu_read(u64) 5.668 0.333 -94.1%
this_cpu_write(u64) 5.668 0.395 -93.0%
this_cpu_add(u64) 8.002 8.004 +0.0%
this_cpu_add_return(u64) 8.335 8.004 -4.0%
this_cpu_xchg(u64) 8.278 8.004 -3.3%
this_cpu_cmpxchg(u64) 18.713 16.133 -13.8%
With the same configuration, vmlinux .text decreased from 14,369,618 to
14,212,882 bytes (-156,736 bytes, -1.09%). percpu_counter_add_batch()
decreased from 330 to 228 bytes.
Changes since v9:
- Rework the this_cpu operations to remain preemptible using recoverable
GPR critical sections, as suggested by David Laight and Jisheng Zhang.
- Put the cached per-CPU offset first, then split the exception/fixup
infrastructure and operation classes into independently reviewable
patches.
- Keep GPR-state save and restore in reusable assembly macros for NMI-like
entry paths such as SBI SSE.
- Keep masked LR/SC cmpxchg operands XLEN-wide so RV64 high-byte and
high-halfword comparisons are not affected by LR.W sign extension.
- Force both halves of the AMOCAS.Q new value into fixed registers so the
implicit register pair remains valid when either half is zero.
- Use TYPEOF_UNQUAL() in wrappers and fully parenthesize this_cpu_write()
values.
Changes since v8:
- Drop the PIO helper patch, which has already been merged.
- Rename the first patch and update its changelog as suggested by
Davidlohr Bueso.
- Use value-returning AMOs instead of rd=x0 for CPU-local operations.
- Remove unnecessary memory clobbers from the new relaxed AMO and LR/SC
code.
- Use .option arch,+zabha for the Zabha inline asm, matching the
existing RISC-V cmpxchg implementation.
- Isolate arch_cmpxchg_relaxed() behind typed always-inline helpers to
avoid macro variable capture while preserving atomic instrumentation.
- Make the cmpxchg128 fallback use the already resolved per-CPU pointer,
avoiding the same macro variable capture.
- Copy the cached per-CPU offset from prev to next during context switch,
as suggested by Thomas Gleixner.
- Initialize secondary idle tasks in the common startup path, covering
parallel bringup and CPU hotplug, and remove the duplicate initialization
from smp_callin(), as reported by Sashiko.
- Select HAVE_CMPXCHG_LOCAL in a separate patch.
Yunhui Cui (9):
riscv: store percpu offset into thread_info
riscv: add infrastructure for preemptible this_cpu operations
riscv: implement preemptible this_cpu read and write operations
riscv: implement preemptible this_cpu void RMW operations
riscv: implement preemptible this_cpu add_return operations
riscv: implement preemptible this_cpu xchg operations
riscv: implement preemptible this_cpu cmpxchg operations
riscv: implement preemptible this_cpu cmpxchg128
riscv: select HAVE_CMPXCHG_LOCAL
arch/riscv/Kconfig | 1 +
arch/riscv/include/asm/asm.h | 25 +-
arch/riscv/include/asm/percpu.h | 701 +++++++++++++++++++++++++++
arch/riscv/include/asm/ptrace.h | 15 +
arch/riscv/include/asm/switch_to.h | 9 +
arch/riscv/include/asm/thread_info.h | 4 +-
arch/riscv/kernel/Makefile | 1 +
arch/riscv/kernel/asm-offsets.c | 4 +-
arch/riscv/kernel/entry-common.c | 32 ++
arch/riscv/kernel/entry.S | 10 +
arch/riscv/kernel/smpboot.c | 10 +-
arch/riscv/net/bpf_jit_comp64.c | 9 +-
12 files changed, 805 insertions(+), 16 deletions(-)
create mode 100644 arch/riscv/include/asm/percpu.h
create mode 100644 arch/riscv/kernel/entry-common.c
base-commit: ddfe51b0a0f5dc9e415cc2c8ad8db17532dd426b
--
2.39.5
next reply other threads:[~2026-11-12 10:42 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 10:41 Yunhui Cui [this message]
2026-11-12 10:41 ` [PATCH v10 1/9] riscv: store percpu offset into thread_info Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 2/9] riscv: add infrastructure for preemptible this_cpu operations Yunhui Cui
2026-11-12 12:02 ` David Laight
2026-11-12 10:41 ` [PATCH v10 3/9] riscv: implement preemptible this_cpu read and write operations Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 4/9] riscv: implement preemptible this_cpu void RMW operations Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 5/9] riscv: implement preemptible this_cpu add_return operations Yunhui Cui
2026-11-12 11:30 ` bot+bpf-ci
2026-11-12 12:09 ` David Laight
2026-11-12 10:41 ` [PATCH v10 6/9] riscv: implement preemptible this_cpu xchg operations Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 7/9] riscv: implement preemptible this_cpu cmpxchg operations Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 8/9] riscv: implement preemptible this_cpu cmpxchg128 Yunhui Cui
2026-11-12 10:41 ` [PATCH v10 9/9] riscv: select HAVE_CMPXCHG_LOCAL Yunhui Cui
2026-11-12 11:30 ` bot+bpf-ci
2026-11-12 12:10 ` [PATCH v10 0/9] riscv: optimize percpu access Jisheng Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112104149.38589-1-cuiyunhui@bytedance.com \
--to=cuiyunhui@bytedance.com \
--cc=ajones@ventanamicro.com \
--cc=alex@ghiti.fr \
--cc=andrii@kernel.org \
--cc=andybnac@gmail.com \
--cc=aou@eecs.berkeley.edu \
--cc=apatel@ventanamicro.com \
--cc=arnd@arndb.de \
--cc=ast@kernel.org \
--cc=atishp@rivosinc.com \
--cc=ben.dooks@codethink.co.uk \
--cc=bjorn@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=broonie@kernel.org \
--cc=cl@gentwo.org \
--cc=conor.dooley@microchip.com \
--cc=cyrilbur@tenstorrent.com \
--cc=daniel@iogearbox.net \
--cc=dave@stgolabs.net \
--cc=david.laight.linux@gmail.com \
--cc=debug@rivosinc.com \
--cc=dennis@kernel.org \
--cc=eddyz87@gmail.com \
--cc=emil@etsalapatis.com \
--cc=ihor.solodrai@linux.dev \
--cc=jolsa@kernel.org \
--cc=jszhang@kernel.org \
--cc=justinstitt@google.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-riscv@lists.infradead.org \
--cc=llvm@lists.linux.dev \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=menglong8.dong@gmail.com \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=nick.desaulniers+lkml@gmail.com \
--cc=oleg@redhat.com \
--cc=palmer@dabbelt.com \
--cc=pjw@kernel.org \
--cc=pulehui@huawei.com \
--cc=puranjay@kernel.org \
--cc=qingfang.deng@siflower.com.cn \
--cc=rkrcmar@ventanamicro.com \
--cc=samuel.holland@sifive.com \
--cc=seanwascoding@gmail.com \
--cc=song@kernel.org \
--cc=tglx@kernel.org \
--cc=thuth@redhat.com \
--cc=tj@kernel.org \
--cc=wangruikang@iscas.ac.cn \
--cc=yonghong.song@linux.dev \
--cc=zong.li@sifive.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®